Robots.txt

A text file at your domain root that tells search engine crawlers which pages or sections of your site they should or shouldn't crawl. It's advisory — crawlers can ignore it — but all major search engines respect it.

Why it matters: Robots.txt is one of the seo signals that shapes how search engines, AI tools, and real visitors experience your website. Getting it right removes friction, protects your rankings, and makes every other optimization work harder. Auditing it regularly is the fastest way to catch regressions before they cost you traffic or conversions.