// Limited guest access · no account for the first check
Fetch one public host's file and review captured User-agent, Allow, Disallow, Sitemap, and Crawl-delay records. The checker shows parser evidence, not a crawler-specific URL test.
> Target_URL
5 free checks · No signup · Free
Extract captured User-agent groups and their Allow and Disallow values.
List declared Sitemap values and flag values that are not absolute URLs.
Display named crawler tokens found in the file without verifying their owners or policies.
Note selected sensitive-looking paths because robots.txt is not access control.
Summarize a small set of response, record, and wildcard-root observations.
Review and copy up to 50 KB of text returned for the requested robots.txt URL.
Ways to go deeper with this site:
It requests /robots.txt for one submitted host, follows the fetch client's redirects, retains up to 50 KB of returned text, and extracts selected User-agent, Allow, Disallow, Sitemap, and numeric Crawl-delay records. It also reports a small set of parser heuristics.
No. It does not accept a separate path, choose the matching User-agent group, resolve the longest matching Allow or Disallow rule, normalize percent encoding, or emulate crawler-specific behavior. Test important URLs with the relevant service's current inspection tools.
No. It ignores unrecognized and malformed lines without a colon, does not report line numbers, and does not fully validate UTF-8, inline comments, special-character matching, group merging, product-token matching, crawler extensions, response headers, or files beyond the 50 KB capture.
No. It lists Sitemap values and checks whether they start with an HTTP scheme, but it does not fetch or parse sitemap files. It displays named User-agent values but does not maintain a verified crawler directory, inspect crawler traffic, or prove that a service honors the rules.
No. It asks conforming crawlers not to fetch matching URLs and is not an access-control mechanism. Google can sometimes index a disallowed URL discovered elsewhere. A noindex directive must be crawlable to be read; protect private resources with authentication and authorization.
An initial guest check is available without signing up. Guest access includes five shared uses across the Quick Tools; a Free account includes 15 Quick Tool uses per month, and Pro and Agency plans include unlimited uses.
Technical SEO Signals
Review crawl-captured metadata, directives, URL patterns, statuses, links, and JSON-LD presence
Sitemap Review
Review sitemap references, canonical-field coverage, noindex directives, fetch failures, and inbound links within a crawl sample
SEO Checklist
Review captured titles, descriptions, headings, canonicals, image alternatives, JSON-LD presence, Open Graph fields, and link responses
Redirect Mapper
Trace the entered URL and public scheme/www variants, then compare those HTTP responses with redirect and link evidence retained by the crawl
Enter a public website URL and the checker requests /robots.txt from that host. It displays up to 50 KB of captured text and extracts selected User-agent, Allow, Disallow, Sitemap, and numeric Crawl-delay records. It also notes a wildcard root restriction, relative Sitemap values, and selected path names that should not be treated as protected simply because they appear in a Disallow rule.
This parser does not test a page path, choose the group that applies to a named crawler, calculate the longest matching rule, interpret inline comments, validate encoding, report line numbers, or reproduce extensions used by individual services. It does not fetch a declared sitemap or determine whether blocked resources affect rendering. Before changing production rules, test important URLs with the relevant crawler's current tools and compare the result with your crawl intent.
Robots.txt asks conforming crawlers not to request selected URLs; it cannot enforce access, prevent discovery, or keep a page out of search results by itself. Use authentication and authorization for private resources. For search visibility, remember that a crawler must be able to fetch a page to read its noindex directive. Named search, commercial, and AI crawlers can publish different product tokens and policies, so verify current documentation for each service instead of assuming every crawler uses the same rules.