How to Create a robots.txt File: Complete Guide with Examples

The robots.txt file lives at the root of your domain (yourdomain.com/robots.txt) and tells web crawlers — including Googlebot, Bingbot, and hundreds of others — which pages or sections of your site they're allowed to visit. It's the first thing most crawlers check before visiting your site.

Critically, robots.txt is a directive to crawlers, not a security measure. It doesn't prevent pages from appearing in search results — it only prevents crawlers from fetching the content. A URL can be indexed and appear in Google search results even if robots.txt blocks the page, as long as the URL is linked from somewhere Google can reach. If you need to prevent a page from appearing in search results, use the noindex meta tag or X-Robots-Tag header.

robots.txt also doesn't apply to all crawlers. It applies to crawlers that choose to respect it — the major search engines do, but malicious scrapers and bots typically ignore it. Never rely on robots.txt to protect sensitive content.

A robots.txt file contains one or more blocks of rules. Each block starts with a User-agent line specifying which crawler the rules apply to, followed by Allow and Disallow lines specifying which paths are permitted or forbidden. User-agent: * applies to all crawlers.

The Disallow directive specifies paths the crawler should not access. Disallow: /admin/ blocks everything under /admin/. Disallow: / blocks the entire site (don't do this unintentionally — it's a common catastrophic mistake). An empty Disallow value means allow everything, which is the same as not having a Disallow line at all.

The Allow directive overrides a broader Disallow for specific sub-paths. If you want to block /private/ but allow /private/public-page.html, you use Disallow: /private/ followed by Allow: /private/public-page.html. Order matters: more specific rules take precedence, but the syntax is interpreted slightly differently by different crawlers — test carefully.

The purpose of robots.txt is to prevent crawlers from wasting their crawl budget on pages that have no value in search results — not to hide content or manipulate rankings. Block pages you don't want indexed (though noindex is more reliable), and pages that would be pointless for a search engine to crawl.

Common paths worth blocking: admin and CMS backend paths (/wp-admin/, /admin/, /backend/), search results pages with dynamic parameters, staging or development directories, internal user-only pages, printer-friendly versions that duplicate main content, and temporary or utility files.

Do not block: your CSS and JavaScript files (Google needs these to render and understand your pages), image directories (Google Images is a traffic source), your sitemap, or any page you want to appear in search results. Blocking CSS and JS is a surprisingly common mistake that can cause your pages to be rendered incorrectly by Googlebot.

WordPress generates a virtual robots.txt automatically if you don't create a physical file. The default virtual file allows everything. If you've installed an SEO plugin (Yoast SEO, Rank Math, All in One SEO), that plugin manages the robots.txt and lets you edit it from the WordPress admin. Always check your SEO plugin's robots.txt settings before manually creating a file — conflicts between the plugin's virtual file and a physical file can cause confusion.

A sensible WordPress robots.txt blocks the admin panel, search results, and login page while keeping the Sitemap reference. Block /wp-admin/, /?s=, and /wp-login.php, and add a Sitemap: directive pointing to your sitemap_index.xml.

Shopify manages its robots.txt through the Liquid template system. You can customize it by creating a robots.txt.liquid template in your theme. Shopify automatically adds your sitemap reference. The default Shopify robots.txt is reasonable — don't modify it without a specific reason, as Shopify's defaults are designed to allow indexing of all necessary content.

The most catastrophic robots.txt mistake is accidentally blocking your entire site. This happens during CMS migrations or platform changes where a staging configuration (Disallow: /) is deployed to production. Check your robots.txt every time you migrate platforms, change CMS, or launch after a rebuild. Google Search Console will alert you if it detects you're blocking Googlebot, but only after the damage has been done.

Other costly mistakes: disallowing your CSS and JavaScript directories (breaks Googlebot's ability to render your pages), blocking your sitemap (prevents Google from discovering its location), using a wildcard pattern that accidentally blocks more than intended, and having syntax errors that make the entire file invalid.

After any robots.txt change, use Google Search Console's robots.txt Tester tool to verify that the URLs you want crawled are allowed and the URLs you want blocked are disallowed. Also check the actual file at yourdomain.com/robots.txt from a browser to confirm what's live.

After creating or modifying your robots.txt, verify it using multiple tools. Google Search Console's URL Inspection tool shows whether a specific URL is blocked by robots.txt. The robots.txt Tester in Search Console lets you test any URL against the file and see which rule matched. For a full picture, a site crawl shows you which pages are actually being blocked versus indexed.

WebEnture's Sitemap Agent and Technical SEO Agent check your robots.txt for common errors, verify that important pages aren't accidentally blocked, confirm that your sitemap is referenced, and cross-reference disallowed paths against your sitemap to find conflicts. The SEO Agent provides a comprehensive view of how your crawl configuration affects your overall search visibility.