AI Crawlers and llms.txt: How to Control What Models Read

AI crawlers are automated programmes that fetch web pages on behalf of AI companies, either to collect training data or to find sources for an answer. You control them mainly through robots.txt, the same file that has guided search engine crawlers for decades.

llms.txt is something different. It is a proposed file that summarises your site for language models, and it gets a lot of attention for something no major provider has confirmed using.

This guide explains who is crawling, how to set rules they follow, and where llms.txt fits.

Key takeaways

• AI crawlers fall into three groups: training crawlers, search crawlers and fetchers that act when a user asks a question.
• robots.txt is the control that named AI crawlers such as GPTBot, ClaudeBot and PerplexityBot say they respect.
• Blocking a training crawler and blocking a search crawler have different consequences, so decide on each separately.
• llms.txt is a proposed convention, not a standard, and major AI providers have not confirmed that they use it.
• Google-Extended controls use of your content for Gemini training and grounding, and it does not affect Google Search ranking or AI Overviews.

What are AI crawlers?

AI crawlers are bots that request your pages so an AI company can use the content. Each one identifies itself with a user agent, which is a name sent with every request and the name you use in robots.txt rules.

They do three different jobs.

• Training crawlers collect text that may be used to train future models.
• Search crawlers build an index so an AI tool can find and cite pages when it answers.
• User-triggered fetchers load a specific page because a person asked the tool to read it or to look something up.

Providers often run separate bots for each job. Here are some of the common names.

Names and purposes change, and new bots appear. Check each provider's own documentation before you rely on a rule.

Google-Extended deserves a note. It is not a separate crawler but a token you use in robots.txt. Blocking it tells Google not to use your content for Gemini training and grounding. It has no effect on Google Search ranking, and it does not remove you from Google AI Overviews, which are governed by normal search controls.

How do you control AI crawlers with robots.txt?

You add rules to the robots.txt file at the root of your domain, naming each crawler's user agent and the paths it may or may not fetch.

To block one training crawler from the whole site:

User-agent: GPTBot
Disallow: /

To allow a search crawler everywhere except a private area:

User-agent: OAI-SearchBot
Disallow: /account/

To opt out of Gemini training without touching Google Search:

User-agent: Google-Extended
Disallow: /

A few rules of the file are easy to get wrong.

• A crawler follows the most specific group that names it. If you have a group for GPTBot, it ignores your User-agent: * group entirely, so repeat any general rules you still want applied.
• The file must sit at the root, for example example.com/robots.txt, and each subdomain needs its own.
• robots.txt is a request, not a lock. Reputable crawlers follow it, but it does not stop a bot that chooses to ignore it. Truly private content needs a login.
• A block stops future fetching. It does not remove content from models that have already been trained.

Should you allow or block AI crawlers?

It depends on what you sell. If you want AI tools to recommend your brand, allow the search crawlers and fetchers at minimum. If your content is the product, you may have reasons to block training.

Think about it per group.

• Search crawlers and user-triggered fetchers. Blocking these makes it hard for the tool to read or cite your pages. For a business that wants to be found, that is usually a cost with no benefit.
• Training crawlers. Blocking these keeps your content out of future training sets from that provider. A publisher that licenses its archive may want that. A software company whose marketing pages explain its product probably wants models to learn what it does.

Whatever you choose, write it down as a policy. Otherwise someone will paste a "block all AI bots" snippet into the file a year from now and quietly remove you from the answers you spent months earning. If being recommended is the goal, pair this with the steps in how to get cited in AI answers.

What is llms.txt, and do you need one?

llms.txt is a proposed plain-text file, written in Markdown and placed at the root of your site, that gives language models a short summary of the site and links to its most useful pages. You do not need one, and it does not control anything.

The limits are important.

• It is a proposal, not a standard. No standards body has adopted it.
• Major AI providers have not confirmed that their crawlers or models read it.
• It is not an access control. It cannot block or allow anything. Only robots.txt and server-level rules do that.

A minimal file looks like this:

# Example Payroll

> Payroll software for small agencies that pay staff and contractors.

## Product

- [How it works](https://example.com/how-it-works): Overview of features
- [Pricing](https://example.com/pricing): Plans and what each includes

If your site is well organised, creating one takes an hour and does no harm. Just do not expect a measurable result, and do not let it take time from work that is known to matter: crawl access, clean HTML and clear writing.

How do you check which AI crawlers visit your site?

Look in your server or CDN logs for requests whose user agent matches the bots you care about. Analytics tools that rely on a script in the browser will not show them, because crawlers generally do not run those scripts.

1. Export recent access logs from your host, CDN or log service.
2. Filter by user agent for names such as GPTBot, ClaudeBot and PerplexityBot.
3. Check the status codes. A crawler you allow should be getting successful responses, not blocks or errors.
4. Check what they fetch. See whether they reach your important pages or waste requests on filters and parameters.
5. Verify the sender. A user agent is easy to fake. Several providers publish the IP ranges their bots use, so you can confirm a request is real.

Also check your firewall or bot protection settings. Some services block AI bots by default, which can override a robots.txt file that allows them. A regular technical SEO audit is a good place to repeat these checks.

Frequently asked questions

Does blocking AI crawlers hurt your Google rankings?

Blocking AI-specific crawlers such as GPTBot or ClaudeBot does not affect Google Search rankings. Blocking Google-Extended does not affect them either, because it only controls use of content for Gemini training and grounding. Blocking Googlebot is a different matter and would remove you from Google Search.

Is llms.txt an official standard?

No. llms.txt is a proposed convention, not a standard, and major AI providers have not confirmed that they use it. It is optional, and it has no power to allow or block access.

Do all AI crawlers obey robots.txt?

The named crawlers from major providers state that they respect robots.txt. The file is voluntary, though, so it cannot force compliance from a bot that ignores it. Use authentication or server-level blocking for content that must stay private.

Can you remove your content from an AI model that has already been trained?

Not through robots.txt. A robots.txt rule only affects future crawling, so it does not undo training that has already happened. Some providers offer their own request processes, which you would need to check with each one.

Crawler rules are a small file with large consequences, and they are easy to get wrong in both directions. If you want someone to review your robots.txt and logs before you change anything, start a conversation.