AI crawlers, robots.txt and llms.txt

Two questions, one file

The Almanac frames the change clearly. Site owners used to ask whether a page should be indexed for search. Now they must also ask whether their content should be used to train AI models. These are distinct considerations with different business and ethical implications, it says, and robots.txt has become a dual-purpose control point: visibility in search on one side, protection against large-scale data harvesting on the other.

The first question is covered in robots.txt, noindex and the tags that control search. This guide covers the second, and what to do about the extra visitors that crawlers bring.

What real sites do

The Almanac counted the crawler names in robots.txt files. The catch-all rule for every crawler, *, was in 77% of files. Named AI crawlers have grown quickly from a small base:

The Almanac calls the rise of gptbot a roughly 55% increase and says claudebot nearly doubled. Tools that audit sites for search also show up: ahrefsbot is named in about 9% of files and mj12bot in about 7%. The Almanac says this reflects the effort to control SEO tools, which can generate significant crawl activity.

How to write the rules

MDN describes the format. A group starts with a User-agent line that names a crawler, followed by Disallow lines. Here is an example that lets everything in except three named AI crawlers:

# Everything not named below may crawl the whole site
User-agent: *
Disallow:

# Ask these AI crawlers not to fetch anything
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

I checked the example with Python's standard robots.txt parser. It said GPTBot, ClaudeBot and CCBot may not fetch a page, and that Googlebot, Bingbot and a crawler it had never heard of may. It also matched gptbot in lower case, in line with the Almanac's note that matching ignores case.

Points to keep in mind:

• Use each company's own documentation for its current crawler names. Names and purposes change, and this guide does not list them as a reference.
• Rules apply to cooperative crawlers. MDN stresses that only cooperative robots follow robots.txt, and that it does not stop bad actors. Do not use it to protect private pages.
• The file is public. Anyone can read it, including what you ask crawlers to avoid.
• *A crawler may not read `` the way you expect.** The Almanac notes that Google's AdsBot ignores it, and that Applebot falls back to Googlebot's rules before applying it. A rule meant for one crawler can reach another that borrows its rules.
• A mistake can hide your whole site. Disallow: / under User-agent: * blocks every crawler. See the robots.txt guide.
• A server error on the file can make crawlers treat the site as blocked.

llms.txt

The llms.txt file is a proposal from llmstxt.org, described as a way to give information that helps language models use a website at the time they answer a question. It is a text file in Markdown that holds a simplified version of the site. The Almanac measured it for the first time in 2025:

• It was on about 2% of sites, with 324,184 valid files found on mobile sites.
• 39.6% of them came from the All in One SEO plugin and 3.6% from Yoast SEO, judging by the comments those plugins leave. Many were produced by a plugin, not chosen by the owner.
• The Almanac says Google has often stated that it does not use llms.txt, and that no Google service currently does. It notes that Anthropic has published one for its own documentation.

So llms.txt is optional, unproven and cheap. If a plugin makes one, check what it says. If you write one, keep it short, accurate and up to date. Do not expect it to change your search results, and do not use it in place of good pages.

Make the content easy to read

The Almanac advises that all key information should be in the initial raw HTML, since AI crawlers do not render JavaScript, according to a source it cites. That is also the rule for search crawlers that read the first response, and for speed. See JavaScript and SEO. The same habits help every kind of reader:

• Clear headings and plain text. See titles, descriptions and headings.
• Structured data in the raw HTML. See structured data basics.
• Descriptive link text and a crawlable link structure. See internal linking.
• Alt text on images. See image SEO and speed.

The load crawlers add

Crawlers are visitors, and they cost your server work. The Almanac observes that SEO tools can generate significant crawl activity, and every new AI crawler adds to the traffic. Slow responses hurt crawlers and people alike. See server response time.

• Look at your server logs or CDN reports to see which crawlers visit and how often.
• Cache what you can, so that a crawler's request is cheap. See caching explained.
• Rate-limit abuse with the right status code. MDN says a 429 means the client has sent too many requests in a given time, and may carry a Retry-After header. A 503 with Retry-After signals a temporary problem.
• Block what ignores your rules at the CDN or firewall, not in robots.txt, which such crawlers ignore.

Choosing a policy

Review the file every few months, because crawler names and behavior keep changing.

Mistakes to avoid

• Blocking all crawlers while meaning to block a few.
• Treating robots.txt as a lock.
• A staging file with `Disallow: /` left on the live site.
• Copying a list of crawler names from the internet and never updating it.
• Hiding key content behind script.
• Believing `llms.txt` will change rankings.

Check your own page

Run the free audit and read the SEO findings, then open /robots.txt on your site. Check that the status is 200, that no group blocks the whole site by accident, and that every crawler you name is one you meant to name.

Sources

• SEO chapter, Web Almanac 2025: crawler names in robots.txt files, the dual role of the file, how crawlers read *, and the first measurement of llms.txt.
• Page Weight chapter, Web Almanac 2025: why key information belongs in the initial raw HTML.
• MDN: robots.txt, the robots.txt configuration guide, and the pages for 429 and 503.