Do You Need llms.txt? AI Crawler Controls Explained

Direct answer: publish /llms.txt only if you maintain a documentation-heavy or structured knowledge site and can keep a curated machine-readable map accurate. For an ordinary service business, prioritize crawlable pages, internal links, a clean sitemap, correct structured data, and explicit crawler rules.

The distinction matters because llms.txt and robots.txt solve different problems. The llms.txt proposal describes a Markdown overview for use at inference time. Its own specification says it complements existing standards. It does not define access permission or an opt-out mechanism.

Use the control that matches the job

What llms.txt is

The proposal places a Markdown file at /llms.txt. It starts with one required H1, may include a short summary and explanatory text, and can group useful links under H2 headings. The specification also proposes optional Markdown versions of detailed pages.

That can be useful for a software library, API, public knowledge base, or complex documentation set. A maintainer can point an agent toward a stable quick start, current reference, examples, and known limitations without asking it to infer the hierarchy from navigation and decorative HTML.

The proposal does not specify how an AI provider must discover, fetch, or use the file. It explicitly leaves processing decisions to each application. That is why publishing the file is an experiment, not an access-control project.

Why most small-business sites do not need it

A typical service site has a homepage, service pages, a few decision guides, contact information, and perhaps a local profile. Those pages should already explain the business clearly in rendered HTML. Adding another summary file creates a second place where services, locations, prices, policies, and URLs can become stale.

Google's official guidance is especially direct. Pages need normal Search eligibility for AI Overviews and AI Mode, and Google says no new machine-readable file, AI text file, or special markup is required. Googlebot access, indexability, internal discovery, visible text, page experience, and accurate structured data remain the foundation.

Separate search retrieval from model training

There is no single universal "AI bot" switch. Vendors document separate agents for different jobs, and a site owner may want search visibility without allowing potential training use.

Google states that blocking Google-Extended does not affect inclusion or ranking in Google Search. OpenAI's publisher guidance similarly separates OAI-SearchBot from GPTBot. Anthropic documents three distinct agents so publishers can make separate choices.

Build the policy before editing robots.txt

Write the business decision first. Otherwise a copied rule can quietly block the exact search surface you intended to keep.

1. List the public content: decide which pages may be retrieved by search and user-directed agents.
2. Separate model development: decide whether to block training-specific agents where a vendor provides that control.
3. Protect private content properly: use authentication, authorization, network restrictions, or removal. Do not rely on crawler courtesy.
4. Inspect the whole delivery path: robots.txt, CDN, WAF, hosting firewall, rate limits, JavaScript challenges, and origin responses can disagree.
5. Record an owner and review date: crawler names, IP ranges, and product behavior change.

The AI search citation guide covers content eligibility and evidence after access is working. The technical SEO fixes guide covers infrastructure problems that ordinary content edits cannot solve.

A cautious robots.txt example

This example expresses one specific policy: allow documented search agents, block documented training controls, and leave normal search crawling open. It is not a universal recommendation. Read every vendor's current documentation and your existing robots file before adapting it.

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

This does not remove a page from search. A robots disallow can prevent a crawler from seeing a noindex directive on the blocked page. If removal is the goal, follow the search engine's documented removal or indexing controls instead of combining random directives.

It also does not create a hard security block. Robots rules are instructions for compliant agents. A scraper can ignore them, and a legitimate agent can still be stopped by an overly broad firewall rule.

Check the CDN and WAF separately

Allowing a crawler in robots.txt is only the first gate. OpenAI and Perplexity both tell publishers to check CDN, WAF, bot mitigation, response codes, and published IP ranges when access fails. Perplexity recommends combining the expected user agent with its current IP list rather than trusting the header alone.

A safe validation sequence is:

1. Request the live /robots.txt and confirm the deployed file matches the policy.
2. Request representative public URLs with the documented user-agent token.
3. Confirm a successful response, useful HTML, and no challenge page.
4. Inspect CDN and origin logs for 401, 403, 429, redirect loops, and unusually large responses.
5. Where the vendor publishes IP ranges, verify both the header and source range before creating an allow rule.
6. Recheck after deployment because caches and crawler processing can delay the observed change.

curl -I -A "OAI-SearchBot" https://example.com/important-page
curl -I -A "Claude-SearchBot" https://example.com/important-page
curl -I -A "PerplexityBot" https://example.com/important-page

A forged user-agent can pass this simple request test. Use it to find obvious blocks, not to authenticate crawler identity.

When an llms.txt test is worth maintaining

Consider a limited test when all of these conditions are true:

• the public site contains a large, stable documentation or knowledge corpus;
• the preferred entry points and current versions are difficult to infer from navigation alone;
• the file can be generated from the same source of truth as the documentation;
• every linked URL is public, canonical, useful, and maintained;
• the team can test named tools or agents instead of treating the file as a universal standard;
• removing the experiment would not damage ordinary crawling, indexing, or user navigation.

If you publish it, keep the file concise. Name the project, explain the scope, link only to preferred public resources, and record how it is generated. Do not paste your sitemap into Markdown or duplicate full pages into a second unmanaged content layer.

What not to do

• Do not use llms.txt as an opt-out file. Use each provider's documented crawler token and access controls.
• Do not block every AI-related token. A training control and a search bot may have different jobs.
• Do not list sensitive paths. Public crawler files are not a substitute for authentication.
• Do not create a stale second website map. Generate the file from maintained data or do not publish it.
• Do not expect a citation lift. Access and context create eligibility, not selection.
• Do not trust a user agent by itself. Use official IP data and logs when infrastructure policy depends on identity.
• Do not let the experiment delay basic SEO. Fix status codes, internal links, canonicals, sitemaps, rendered text, and page quality first.

FloxoLab's current decision

FloxoLab does not need llms.txt to make its guides eligible for Google AI features or ChatGPT Search. The current priorities are a crawlable public library, one canonical URL per task, accurate metadata, responsive media, contextual links, and explicit crawler access.

A generated file may become worth testing after the SEO guide library expands and the filterable hub is stable. If that happens, the file should be derived from the content manifest, not maintained as another hand-written index. The topic-cluster guide explains why one source of truth matters when a library grows.

Decision checklist

• The business goal is written before crawler rules are changed.
• Search retrieval and potential training are treated as separate decisions.
• Private content is protected with real access controls.
• robots.txt, CDN, WAF, and origin behavior agree.
• Official crawler documentation and IP sources were checked recently.
• Important public pages return useful HTML to allowed agents.
• An llms.txt experiment has a narrow purpose, owner, generator, and removal plan.
• No ranking or citation outcome is promised.

For most small businesses, the right answer is simple: skip llms.txt for now, make the real pages easy to crawl and understand, and control each crawler according to the job it performs.

Sources checked

Behavior and crawler names were verified from primary or official documentation on August 2, 2026.

• llms.txt proposal and format
• Google Search Central: AI features and your website
• Google: common crawlers and Google-Extended
• OpenAI: Publishers and Developers FAQ
• Anthropic: Claude crawler controls
• Perplexity: crawler and WAF documentation