What is llms.txt? The new standard for AI agents

For 30 years, the internet ran on a simple agreement called robots.txt. It was a “Do Not Enter” sign for clumsy search spiders.

In 2026, we’re dealing with readers, not just spiders.

AI agents (ChatGPT’s crawler, autonomous research bots, RAG pipelines) want to understand your content, not just index your links. The modern web makes that hard. It’s bloated with JavaScript, popups, cookie banners, and DOM structures that waste tokens and muddy context.

That’s the gap llms.txt is trying to fill. It’s a proposal to give AI agents what they actually want: clean, structured, markdown-formatted context about your site.

If you care about Generative Engine Optimization (GEO), shipping an llms.txt file is one of the highest-ROI technical changes available today.

Table of contents

• The problem with HTML in the AI age
• What is llms.txt exactly?
• The anatomy of the file
• How to implement llms.txt
• Tools to generate llms.txt
• The business case for clean context
• Robots.txt vs LLMs.txt
• Monitoring agent behavior

The problem with HTML in the AI age

To see why llms.txt is needed, look at how LLMs actually “read.”

A traditional crawler like Googlebot scans for links and keywords and ignores the visual chrome. An AI agent (say, a RAG pipeline) is trying to ingest information, and the modern web is hostile to ingestion.

The token tax of the modern web breaks down into three buckets:

1. Boilerplate. Headers, footers, and navbars repeat on every page. An AI reading 10 pages reads your navbar 10 times. That wastes context-window space and money.
2. DOM noise. <div>, <span>, class names, and inline scripts are gibberish to an LLM trying to answer a question.
3. Visual vs. semantic content. Popups can obscure content, “Read More” buttons hide it, and an AI can’t really “click” anything.

The result is hallucination. When an AI scrapes a JavaScript-heavy page, it often gets a fragmented mess and fills in the blanks itself. That’s when it invents your pricing, your features, or your history.

llms.txt solves this by giving knowledge its own dedicated endpoint.

What is llms.txt exactly?

The llms.txt proposal (popularized by Jeremy Howard) is a convention for placing a file at the root of your domain, e.g. yourdomain.com/llms.txt.

It does two things:

1. Acts as a map. It tells AI agents where to find the “AI-ready” version of your site.
2. Provides context. A concise summary of who you are and what you do, injected straight into the model’s prompt.

Think of it as a sitemap for robots that read. You explicitly list the pages that matter, and you point the agent at clean Markdown instead of HTML.

The anatomy of the file

The standard is simple. It typically lives at the root and points to a more comprehensive markdown file.

Example /llms.txt:

# cloro - AI Brand Monitoring Platform

> cloro is the leading platform for tracking brand visibility across Large Language Models (LLMs) like ChatGPT, Claude, and Perplexity.

## Key Pages

- [Pricing](/#pricing)

Key components:

• H1 title. States the entity name.
• Blockquote summary. A “system prompt” for your brand, often the first thing the AI reads. Make it count.
• Links. Direct pointers to markdown (.md or .txt) versions of your most critical content.

How to implement llms.txt

You don’t need a site redesign. You’re creating a shadow site of text files alongside the existing one.

Step 1: Create your shadow content

Convert your key pages into Markdown to strip the HTML noise.

Your pricing.html might be 50kb of code. The equivalent pricing.md should be 2kb of text.

Example pricing.md:

# Pricing Plans

## Hobby Plan

- Cost: $29/month
- Features: 500 queries, Daily updates.

## Business Plan

- Cost: $99/month
- Features: 5,000 queries, Hourly updates.

Step 2: Consolidate into llms-full.txt

Several proposals suggest a single large text file (llms-full.txt) containing all your core documentation concatenated together. RAG systems prefer fetching one file: fewer HTTP requests, and the model gets the full context in a single pass.

Step 3: Deploy the root file

Place llms.txt at your root. Make sure your server returns text/markdown or text/plain headers.

Step 4: Advertise it

Auto-discovery is still evolving. In the meantime, feed the URL manually to custom GPTs, Claude Projects, and other agents to “train” them on your documentation.

Tools to generate llms.txt

If writing these files by hand feels tedious, a few tools will crawl your site and produce the markdown structure for you.

• Keploy. One-click generator that scans a URL and builds the file. Fine for simple sites.
• Writesonic. Structured text generator aimed at LLM training and inference.
• Gushwork. More granular control over which site areas to include or exclude.
• Fibr AI. Generates a file with explicit permissions for bots like GPTBot and ClaudeBot.

These tools are useful for a first pass, but review the output by hand. The shadow content for your most critical pages needs to be accurate.

The business case for clean context

Why spend engineering hours on this?

1. Fewer hallucinations. Clean text drops the noise-to-signal ratio to near zero. The AI doesn’t get confused by your cookie banner and decide you sell cookies. It reads your markdown and knows you sell software.

2. Better citation authority. Perplexity and similar engines use RAG. If their scraper can parse your content faster and cheaper than a competitor’s heavy React app, you get the citation.

3. Token economy. A 128k context window shouldn’t burn 50k tokens on HTML boilerplate. Serving Markdown packs more of your useful content into the model’s working memory.

4. Future-proofing. OpenAI, Anthropic, and Google are all looking for ways to cut web scraping costs. Crawlers that find an llms.txt will likely prioritize it because it saves them compute.

Robots.txt vs LLMs.txt

These two files serve different masters.

Feature	robots.txt	llms.txt
Audience	Crawlers (Googlebot)	Agents (ChatGPT, Claude)
Function	Exclusion (Do not go here)	Inclusion (Read this first)
Format	Rules & Disallow paths	Markdown & Links
Goal	Indexing control	Context injection
Parsing	Machine logic	Semantic understanding

Don’t replace robots.txt. You still need it to block sensitive admin paths. llms.txt is an additive layer for the semantic web.

Monitoring agent behavior

Once llms.txt is live, how do you know it’s doing anything?

You need to track whether AI agents are hitting the file, and whether the data is showing up in their responses.

That’s where cloro fits in. Monitoring brand mentions lets you correlate the deployment of llms.txt with citation accuracy over time.

The feedback loop:

1. Deploy llms.txt.
2. Wait two weeks.
3. Check cloro for mention quality.
4. If hallucinations persist, refine the markdown descriptions.

The web is shifting from a library of documents to a training set for models. llms.txt is how you make sure your entry in that set is accurate and clean.