How llms.txt Works (and What It Doesn't Do)

In this article

1. What is llms.txt?
2. How llms.txt works: the format
3. llms.txt vs robots.txt vs sitemap.xml
4. Who actually reads llms.txt in 2026?
5. Should a B2B company have an llms.txt?
6. How to create an llms.txt file in 6 steps
7. What about llms-full.txt and .md pages?
8. Common llms.txt mistakes
9. Frequently asked questions

What is llms.txt?

llms.txt is a proposed web standard for helping large language models understand a website. Jeremy Howard, co-founder of Answer.AI, published the proposal on September 3, 2024. His reasoning was that web pages are built for people. They're full of navigation, scripts and layout, and they're often too long for an AI tool to take in at once. A short Markdown file that says "here's who we are and here are the pages worth reading" gives an AI tool the context it needs without making it crawl and clean up a whole site.

The idea caught on quickly with developer documentation sites, where coding assistants need to find the right API page quickly. llmstxt.org notes that OpenAI, Anthropic and Google's Gemini team all publish llms.txt files for their own developer docs. Marketing sites followed, often because an SEO tool flagged the file as "missing".

The key point for business owners: llms.txt is a guide, not a rule. It doesn't grant or deny access to anything. Access is still controlled by robots.txt and your firewall (see our AI crawler field guide).

How llms.txt works: the format

The file lives at https://yourdomain.com/llms.txt and is written in Markdown. The spec defines these parts, in this order:

1. An H1 heading with the name of the site or company. This is the only required part.
2. A blockquote with a short summary: what you do, for whom and where.
3. Optional detail in paragraphs or bullet lists, such as key facts, terminology or how to read the rest of the file.
4. H2 sections of links. Each item is a Markdown link, optionally followed by a colon and a one-line note.
5. An "Optional" section (a convention) for secondary pages an agent can skip when it's short on context.

Example: llms.txt for a fictional industrial distributor

# Northfield Industrial Supply

> Industrial MRO distributor serving manufacturers in Ohio, Michigan and
> Indiana since 1987. 240 employees, three warehouses, same-day delivery
> within 100 miles of Toledo.

Key facts:
- Headquarters: Toledo, Ohio. Branches: Detroit, Michigan and Fort Wayne, Indiana.
- Certifications: ISO 9001:2015.
- Pricing: contract pricing for accounts; list prices shown online.

## Products
- [Bearings and power transmission](https://www.example.com/bearings): 12,000 SKUs from major brands
- [Safety and PPE](https://www.example.com/safety): gloves, eyewear, fall protection
- [Vending and inventory programs](https://www.example.com/vending): on-site vending for PPE and consumables

## Company
- [About Northfield](https://www.example.com/about): history, leadership, locations
- [Service area and delivery](https://www.example.com/delivery): delivery zones and cut-off times
- [Request a quote](https://www.example.com/quote)

## Optional
- [Blog](https://www.example.com/blog)
- [Careers](https://www.example.com/careers)

That's the whole format. An AI tool that reads this file learns, in about 200 words, what the company is, where it operates and which URL answers which question. You can see a live version in our own llms.txt, which includes a "Facts for AI assistants" section with our pricing, locations and service area.

What changed in version 2 (August 2026)

The spec was revised on August 10, 2026, after nearly two years of real-world use. The main changes:

• Markdown versions of pages. Sites can offer a clean Markdown copy of any page, either at the same URL with .md appended (page.html.md) or with the extension replaced (page.md).
• Discovery through link relations. Pages can point to their Markdown version with rel="alternate" type="text/markdown" and to the llms.txt that covers them with rel="describedby", in an HTML <link> element or an HTTP Link header. Agents no longer have to guess.
• Files in subfolders. A file covers the pages under its path, and the most specific file applies, so /docs/llms.txt can describe just the docs.
• A more realistic reading model. v2 assumes agents search or skim the file and then follow the links they need, rather than loading everything into context.

llms.txt vs robots.txt vs sitemap.xml

These three files are often mentioned together, but they do different jobs:

If you only have time for one of these, fix robots.txt first (here are four copy-paste templates). A perfect llms.txt does nothing if OAI-SearchBot or PerplexityBot is blocked from the pages it links to.

Who actually reads llms.txt in 2026?

This is where the hype and the data part ways.

Google: not for Search

Google's guide to optimizing for generative AI features (last updated July 10, 2026) is direct about it: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and "Google Search itself doesn't use them." The same page says it's "completely fine" to maintain llms.txt for other services. So the file has no effect on AI Overviews or AI Mode.

There's one twist. In May 2026 Google added an "Agentic Browsing" audit category to Chrome's Lighthouse tool, and it checks for an llms.txt file. Google's explanation is that without one, "agents may spend more time crawling the site to understand its high-level structure." That's about AI agents browsing your site, not about Search rankings.

AI crawlers: rarely

Ahrefs analyzed 137,210 domains in its Web Analytics product and published the results on June 15, 2026:

• 97% of llms.txt files received zero requests in May 2026.
• Among files that did get requests, SEO audit tools made the most (21.7%). AI bots together made 19.5%, led by OpenAI's GPTBot at 4.51%.
• AI bots never requested llms.txt on sites that didn't have one. They don't go looking for it.

AI citations: no measurable effect

SE Ranking studied 300,000 domains in November 2025. It found that 10.13% had an llms.txt file, with similar adoption for low-, mid- and high-traffic sites, and no correlation between having the file and being cited by AI. When SE Ranking removed the llms.txt factor from its model, the model's predictions got slightly better.

Should a B2B company have an llms.txt?

Our answer: yes, but spend about 30 minutes on it, not a project budget. Here's why it's still worth doing:

• It's a fact sheet you control. When an agent does read it, it gets your locations, certifications, service area and pricing model in your own words. That matters, because wrong locations, certifications and pricing are exactly what the AI accuracy check in our audits looks for.
• Agents are arriving. Browser agents and assistants that act for users (booking, comparing vendors, filling in RFQs) benefit from a clear map, and Lighthouse's Agentic Browsing audit now checks for one.
• It forces useful discipline. Picking your 20 most important URLs and describing each one in a line often reveals thin or outdated pages.
• It costs almost nothing and has no downside if the content matches your site.

What llms.txt won't do is make up for the things that do affect AI visibility: crawler access, content that's readable without JavaScript, clear and specific page copy, structured data, and mentions on third-party sites. Those are the parts of our AEO work that move the Agent Visibility Score.

How to create an llms.txt file in 6 steps

1. Choose 10 to 30 URLs. Start with the pages a buyer or an agent needs: core products or services, industries, locations, pricing or "how to buy", about and contact. Leave out tag pages, logins and old campaign pages.
2. Write the H1 and the summary. Use your exact legal or brand name. The blockquote should say what you do, for whom and where, in one or two sentences with no slogans.
3. Add key facts. List the details AI tools tend to get wrong: locations, service area, certifications, founding year, pricing model and what you don't do.
4. Group the links under H2 headings such as Products, Services, Industries and Company. Give each link a note that says what the page answers.
5. Publish it at the root as plain text. Upload it as /llms.txt and serve it as text/plain; charset=utf-8. On Netlify, add this to your _headers file:

   /llms.txt
     Content-Type: text/plain; charset=utf-8

   Then open the URL in a private window and check it returns the file with a 200 status, not your HTML error page.
6. Keep it current. Review it whenever prices, locations or key pages change, and at least every quarter. An out-of-date llms.txt that contradicts your site is worse than none.

Optional, if you want to follow v2 fully: add <link rel="describedby" href="/llms.txt"> to your page templates, and publish Markdown versions of your most important pages.

What about llms-full.txt and .md pages?

llms-full.txt is a widespread community convention, not part of the official spec. It's a single file with the full text of your key pages, so a tool can load everything in one request. It makes sense for developer documentation that people paste into coding assistants. For a mid-market B2B marketing site it's usually more maintenance than it's worth.

Markdown versions of pages (/services.md and so on) are now part of v2. They're worth considering if your pages depend heavily on JavaScript, but fixing server-side rendering helps every crawler, including Google and Bing, and is usually the better investment.

Common llms.txt mistakes

• Using it as a keyword dump. The file is read by machines that summarize, not by ranking algorithms. Stuffing it with keywords gains you nothing.
• Listing every URL. That's what sitemap.xml is for. llms.txt is meant to be selective.
• Contradicting the site. If llms.txt says "offices in 5 states" and the website says 4, you've created the kind of conflict that makes AI less confident about recommending you.
• Serving it as HTML or blocking it. Some platforms return a styled 404 page with a 200 status, and some firewalls block non-browser requests to .txt files. Check the raw response.
• Expecting it to control access. Disallowing a crawler in llms.txt does nothing. Use robots.txt for that.

Frequently asked questions

Free tool

Does your site have an llms.txt? Check in 10 seconds

The free LLM Readiness Check looks for llms.txt and tests robots.txt rules for 15 AI crawlers, JavaScript rendering, structured data and your sitemap.

Run the free check