llms.txt: The New robots.txt That AI Crawlers Actually Read

llms.txt is a proposed standard (developed by Jeremy Howard and the fast.ai team) that provides a structured way for website owners to communicate with large language model crawlers. It's a Markdown file placed at the root of your domain (yourdomain.com/llms.txt) that describes what your site is about, what its most important pages are, and how AI systems should interpret your content.

The name is deliberately analogous to robots.txt, which tells search engine crawlers which pages to visit or avoid. llms.txt doesn't block or allow — it guides. It gives AI systems a curated map of your authoritative content so they don't have to discover it through crawling alone, and so they understand the context around your content.

The standard is emerging, not universally adopted. As of 2026, major AI tools including Anthropic, some configurations of Perplexity, and several AI web agents actively read llms.txt. Support is growing because it solves a real problem: AI systems face enormous amounts of low-quality web content and benefit from site owners marking their authoritative sources.

llms.txt is a plain Markdown file. The structure includes a top-level heading with your site or company name, a brief paragraph describing what your organization does, and then sections linking to your most important content with brief descriptions of what each page covers.

An llms.txt for a web optimization company might begin with the company name as an H1, a sentence describing the company, then H2 sections for 'Docs', 'Tools', 'Blog', and 'Reference' — each containing Markdown links to the most important pages in that category with one-sentence descriptions.

There's also an llms-full.txt convention for a more comprehensive version that includes the content of key pages inline, allowing AI systems to consume your most important content without additional crawl requests. Implementing both (a concise llms.txt and a more complete llms-full.txt) provides different AI tools with options depending on their approach.

llms.txt is newer than robots.txt and less universally supported. However, early adoption signals that you're thinking about AI discoverability, and the cost of implementation is low — creating the file takes 30–60 minutes for most sites. The potential upside is meaningful as AI search becomes a larger share of discovery traffic.

Perplexity has specifically mentioned that structured content and llms.txt help their crawlers find and understand site content faster. Given how quickly AI search is growing, establishing a presence in AI citation indexes now is likely more valuable than waiting until the standard is universal.

WebEnture's GEO AI Visibility Agent checks for the presence and quality of your llms.txt file, validates its structure against the current specification, and identifies content gaps — important pages that should be listed but aren't. It also checks robots.txt for unintentional blocks on AI crawlers.