Robots.txt

A `robots.txt` file publishes Robots Exclusion Protocol rules that request how compliant automated clients may access URLs on a service. It is fetched from `/robots.txt`, and its scope is limited to the protocol, host, and port that serve it. Groups target user-agent tokens and contain `Allow` or `Disallow` path rules; supported crawlers may also read `Sitemap` locations. The protocol controls crawling, not access authorization or guaranteed search indexing.

Why it matters: Use `robots.txt` to manage compliant crawler traffic or avoid crawling low-value URL spaces, not to protect private information. Anyone can read the file, a non-compliant client can ignore it, and disallowed URLs may still appear in search results when discovered elsewhere. Use authentication or other access controls for confidential content, and use an index-control method such as `noindex` only on a URL the relevant crawler can fetch and read. Place a UTF-8 plain-text file at the exact root, group rules for the intended crawler, test path matching and case sensitivity, and verify separately for every subdomain, protocol, and non-default port. Before deployment, check that CSS, JavaScript, images, canonical pages, and intended crawlers remain accessible; monitor fetch errors and server logs after changes.