AI agents are visiting your website right now. They’re trying to understand what you do, what content you have, and where to find answers.
Most of them are struggling. They’re downloading entire pages, parsing HTML, guessing which links matter, and burning through context windows trying to make sense of your site structure.
There’s a better way. The llm.txt specification solves this problem with a single file that tells AI agents exactly what they need to know about your website.
What You’ll Learn
This guide covers everything you need to know about llm.txt files. You’ll understand what they are, why they matter, and how to create one for your website.
You’ll see the technical specification, real-world examples, and how llm.txt compares to older standards like robots.txt and sitemap.xml.
By the end, you’ll know how to make your website speak directly to AI agents.
What is llm.txt?
llm.txt is a machine-readable file that lives at the root of your website. It tells AI agents what your site is about and where to find important content.
Think of it as a curated introduction to your website, specifically designed for language models. While humans browse your navigation menu and click around, AI agents read your llm.txt file to understand your site’s structure instantly.
The specification is simple: a Markdown file at /llms.txt containing your site name, a brief description, and organized lists of your most important URLs.
The Problem llm.txt Solves
AI agents have a fundamental limitation: context windows. Even the largest language models can’t process entire websites at once.
When an AI agent visits your site without llm.txt, it has to guess. It might crawl your homepage, parse your navigation, follow random links, and hope it finds relevant content. This wastes tokens and time.
With llm.txt, you tell the agent exactly what matters. You curate the essential pages, organize them by topic, and provide context about what each page contains.
Who Created llm.txt?
The llm.txt specification emerged from the developer community as LLM-powered tools became more common. It’s an open standard, similar to how robots.txt became the de facto standard for controlling web crawlers.
The specification is hosted at llmstxt.org and continues to evolve based on real-world usage and feedback from developers building AI-powered tools.
Why Websites Need Machine-Readable Content
The web was built for humans. HTML includes navigation menus, sidebars, footers, advertisements, and design elements that help people browse visually.
AI agents don’t need any of that. They need structure, hierarchy, and clean content.
The Human Web vs The Machine Web
When you visit a website, you scan the page visually. You see the header, notice the menu, scroll past the hero image, and find the content you want.
An AI agent sees HTML tags, CSS classes, JavaScript functions, and nested div elements. It has to parse all of this to extract meaning.
Converting web pages to clean markdown removes the noise. It strips away the presentation layer and leaves only the semantic content. This is exactly what converting webpages to markdown does. It transforms human-readable HTML into machine-readable markdown.
The Cost of Unstructured Content
Every time an AI agent visits your site without llm.txt, it pays a cost in tokens. It might load your homepage (5,000 tokens), parse your about page (3,000 tokens), and scan your blog index (10,000 tokens) just to understand what you offer.
With llm.txt, that same understanding takes 500 tokens. You’ve pre-organized the information in exactly the format the agent needs.
This isn’t just efficiency. It’s accessibility. Websites with llm.txt files are more discoverable by AI-powered search tools, research agents, and automated systems that increasingly drive traffic and engagement. For a practical look at where tokens go and how to waste fewer of them, see how to reduce LLM token usage.
How llm.txt Works (Technical Specification)
The llm.txt specification uses Markdown as its format. This makes the file readable by both humans and machines while maintaining programmatic parsability.
File Location and Naming
Your llm.txt file must live at /llms.txt at the root of your domain. Note the plural “llms” in the filename.
https://yourwebsite.com/llms.txtYou can also place llm.txt files in subdirectories for specific sections of your site:
https://yourwebsite.com/docs/llms.txt
https://yourwebsite.com/blog/llms.txtRequired Elements
Only one element is required: an H1 heading with your project or site name.
# CopyMarkdownThat’s technically a valid llm.txt file. But you’ll want to include more to make it useful.
Recommended Structure
A complete llm.txt file follows this order:
- H1 heading: Your site or project name
- Blockquote: Brief summary with essential context
- Descriptive content: Paragraphs explaining what your site offers
- H2 sections: Organized lists of curated URLs
Here’s the basic structure:
# Site Name
> Brief description of what this site offers
Detailed explanation of the project, product, or content.
Multiple paragraphs can provide context and background.
## Core Documentation
- [Page Title](https://example.com/page): Description of what this page contains
- [Another Page](https://example.com/another): What readers will find here
## Blog Posts
- [Important Article](https://example.com/blog/article): Key insights from this post
- [Case Study](https://example.com/blog/case-study): What this example demonstrates
## Optional
- [Additional Resource](https://example.com/extra): Secondary content
- [Archive](https://example.com/archive): Historical informationThe Optional Section
The “Optional” section has special meaning. It signals to AI agents that this content is secondary and can be skipped if context is limited.
This lets you include comprehensive links while telling agents which ones matter most. When an agent needs to be selective, it reads everything before “Optional” and skips the rest.
Link Format Requirements
Every link in your file lists must follow this format:
- [Link Text](URL): Optional descriptionThe hyphen, brackets, parentheses, and colon are all required. The description after the colon is optional but recommended.
Avoid bare URLs or links without descriptions. The goal is to give AI agents context about what they’ll find before they visit the page.
Example: A Complete llm.txt File
Here’s what a real llm.txt file looks like for a developer tool:
# DevTools API
> A REST API for analyzing web performance metrics and generating optimization reports
DevTools API helps developers measure and improve website performance. It provides programmatic access to Chrome DevTools metrics, Lighthouse scores, and custom performance audits.
The API is designed for CI/CD integration, automated monitoring, and performance regression testing.
## Getting Started
- [Quick Start Guide](https://devtools-api.com/docs/quickstart): Set up your first API call in 5 minutes
- [Authentication](https://devtools-api.com/docs/auth): Generate API keys and configure security
- [Rate Limits](https://devtools-api.com/docs/limits): Understand usage tiers and quotas
## API Reference
- [Endpoints](https://devtools-api.com/docs/endpoints): Complete list of available API calls
- [Response Format](https://devtools-api.com/docs/responses): How to parse API responses
- [Error Handling](https://devtools-api.com/docs/errors): Common errors and solutions
## Use Cases
- [CI/CD Integration](https://devtools-api.com/guides/cicd): Run performance tests on every deploy
- [Monitoring Setup](https://devtools-api.com/guides/monitoring): Track metrics over time
- [Custom Audits](https://devtools-api.com/guides/audits): Build your own performance checks
## Optional
- [Changelog](https://devtools-api.com/changelog): Version history and updates
- [Community](https://devtools-api.com/community): Forums and discussion
- [Blog](https://devtools-api.com/blog): Updates and best practicesThis file tells AI agents everything they need to know: what the API does, where to find documentation, how to get started, and which resources are most important.
llm.txt vs robots.txt vs sitemap.xml
Websites already have machine-readable files. robots.txt controls crawler access, and sitemap.xml lists all pages for search engines.
llm.txt serves a different purpose. It’s not about access control or exhaustive indexing. It’s about curation and context.
robots.txt: Access Control
robots.txt tells crawlers what they can and can’t access:
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/This file is binary: allow or disallow. It doesn’t explain what pages contain or why they matter. It’s a bouncer at the door, not a guide inside.
sitemap.xml: Complete Indexing
sitemap.xml lists every page on your site for search engines:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/page1</loc>
<lastmod>2024-01-15</lastmod>
<priority>0.8</priority>
</url>
<url>
<loc>https://example.com/page2</loc>
<lastmod>2024-01-10</lastmod>
<priority>0.6</priority>
</url>
</urlset>Sitemaps are exhaustive. They include every page, even minor ones. They optimize for discovery, not comprehension.
For large sites, sitemaps can contain thousands of URLs. An AI agent with a limited context window can’t process that much information.
llm.txt: Curated Context
llm.txt is selective. You choose the most important pages and organize them by topic:
# Example Site
## Essential Pages
- [About](https://example.com/about): Company background and mission
- [Products](https://example.com/products): Full product catalog
- [Pricing](https://example.com/pricing): Plans and pricing details
## Documentation
- [Getting Started](https://example.com/docs/start): Quick start guide
- [API Reference](https://example.com/docs/api): Complete API documentationThis file is interpretable. It tells the agent not just where pages are, but what they contain and how they relate to each other.
When to Use Each
Use robots.txt to control what crawlers can access. This is about permissions and security.
Use sitemap.xml to help search engines discover all your pages. This is about SEO and indexing.
Use llm.txt to help AI agents understand your site efficiently. This is about comprehension and curation.
You need all three. They serve different purposes and complement each other.
How to Create an llm.txt File
Creating your llm.txt file takes three steps: identify your most important content, organize it by topic, and write clear descriptions.
Step 1: Audit Your Content
List every page that helps visitors understand what you offer or accomplish their goals. Include:
- Homepage and about page
- Product or service pages
- Documentation or guides
- Key blog posts or case studies
- Contact and support pages
Exclude utility pages like privacy policies, legal disclaimers, and navigation-only pages. Those don’t help AI agents understand your value.
Step 2: Organize by Topic
Group your pages into logical sections. Common sections include:
- Getting Started: Introductory content for new visitors
- Products/Services: What you offer
- Documentation: How-to guides and references
- Resources: Blog posts, case studies, examples
- Support: Help and contact information
Each section becomes an H2 heading in your llm.txt file.
Step 3: Write Descriptions
For each link, write a one-sentence description that answers “What will the AI agent learn from this page?”
Bad description:
- [API Reference](https://example.com/api): Read this for the APIGood description:
- [API Reference](https://example.com/api): Complete endpoint documentation with request/response examplesThe good description tells the agent exactly what information the page contains without visiting it.
Step 4: Add Context at the Top
Write your H1, blockquote, and opening paragraphs to give high-level context:
# Your Company Name
> One-sentence summary of what you do
Two or three paragraphs explaining your product, service, or mission.
Include your target audience and main use cases.This context helps AI agents decide if your site is relevant to their query before diving into specific pages.
Step 5: Create the File
Save your content as llms.txt (note the plural) and place it at your website’s root directory.
For static sites, put it in your public or root folder. For dynamic sites, configure your server to serve the file at /llms.txt.
Test it by visiting https://yourwebsite.com/llms.txt in a browser. You should see your Markdown content displayed as plain text.
Step 6: Keep It Updated
Your llm.txt file needs maintenance. When you publish important new content, add it to the appropriate section.
When you reorganize your site or deprecate pages, update your llm.txt to reflect the changes.
Maintaining llm.txt manually can become tedious for large sites with frequently updated content. Automated solutions are emerging to keep these files in sync with your site structure.
Real-World Examples of llm.txt
Several websites have already adopted llm.txt. Here’s how different types of sites implement the specification.
Example: Developer Documentation Site
A documentation site for a JavaScript framework:
# ReactFlow
> A customizable React component library for building node-based editors and interactive diagrams
ReactFlow helps developers build workflow editors, data visualizations, and interactive node graphs.
The library handles the complex parts like dragging, zooming, and connecting nodes while giving you
full control over styling and behavior.
## Core Concepts
- [Introduction](https://reactflow.dev/docs/introduction): What ReactFlow does and when to use it
- [Installation](https://reactflow.dev/docs/installation): Add ReactFlow to your project
- [Basic Example](https://reactflow.dev/docs/examples/basic): Your first node graph in 5 minutes
## Guides
- [Custom Nodes](https://reactflow.dev/docs/guides/custom-nodes): Build nodes with your own components
- [Styling](https://reactflow.dev/docs/guides/styling): Customize colors, shapes, and connections
- [State Management](https://reactflow.dev/docs/guides/state): Handle node data and updates
## API Reference
- [Components](https://reactflow.dev/docs/api/components): All ReactFlow components and props
- [Hooks](https://reactflow.dev/docs/api/hooks): React hooks for managing flow state
- [Types](https://reactflow.dev/docs/api/types): TypeScript type definitions
## Optional
- [Changelog](https://reactflow.dev/changelog): Version history
- [Community](https://reactflow.dev/community): Discord and GitHub discussionsThis llm.txt file helps AI coding assistants quickly find relevant documentation when developers ask questions about ReactFlow.
Example: SaaS Product Site
A project management tool:
# TaskManager Pro
> Team collaboration software for agile project management
TaskManager Pro combines sprint planning, task tracking, and team communication in one platform.
Designed for software teams practicing Scrum or Kanban methodologies.
## Product Overview
- [Features](https://taskmanager.com/features): Complete list of capabilities
- [Pricing](https://taskmanager.com/pricing): Plans for teams of all sizes
- [Integrations](https://taskmanager.com/integrations): Connect with Slack, GitHub, and more
## Getting Started
- [Quick Start](https://taskmanager.com/docs/quickstart): Set up your first project
- [Team Setup](https://taskmanager.com/docs/team-setup): Invite members and assign roles
- [First Sprint](https://taskmanager.com/docs/first-sprint): Plan and run your first sprint
## Use Cases
- [Software Teams](https://taskmanager.com/use-cases/software): How engineering teams use TaskManager
- [Remote Teams](https://taskmanager.com/use-cases/remote): Async collaboration features
- [Marketing Teams](https://taskmanager.com/use-cases/marketing): Campaign planning and tracking
## Optional
- [Blog](https://taskmanager.com/blog): Product updates and best practices
- [Case Studies](https://taskmanager.com/customers): How customers use the productThis llm.txt file helps AI agents answer questions like “What project management tools support agile methodologies?” or “How much does TaskManager cost?”
Example: Content-Heavy Blog
A marketing blog with hundreds of articles:
# Marketing Insights Blog
> Data-driven content marketing strategies and case studies
We publish weekly articles about content marketing, SEO, and audience growth.
Our posts combine research, real-world examples, and actionable tactics.
## Most Popular Posts
- [Content Distribution Guide](https://blog.com/content-distribution): How to amplify every piece you publish
- [SEO Fundamentals 2024](https://blog.com/seo-fundamentals): Modern approach to search optimization
- [Email List Building](https://blog.com/email-list-building): Convert readers to subscribers
## Recent Articles
- [AI Content Strategy](https://blog.com/ai-content-strategy): Using AI tools effectively
- [B2B Content Examples](https://blog.com/b2b-content-examples): What works in B2B marketing
- [Content Calendar Template](https://blog.com/content-calendar): Free template and guide
## Resources
- [Content Templates](https://blog.com/templates): Free templates for various content types
- [Tools We Use](https://blog.com/tools): Recommended marketing tools
- [Case Studies](https://blog.com/case-studies): Client success stories
## Optional
- [All Posts](https://blog.com/archive): Complete archive
- [About](https://blog.com/about): Our team and missionInstead of listing all hundreds of articles, this llm.txt file highlights the most valuable content. AI agents can find the best posts without processing the entire archive.
The Future: When Every Website Speaks to AI
We’re entering an era where AI agents are primary consumers of web content. They research products, compare services, answer questions, and complete tasks on behalf of humans.
These agents need structured, semantic content. The websites that provide it will become more discoverable, more quotable, and more useful.
Beyond Search Engines
Traditional SEO optimizes for Google’s algorithms. You write meta descriptions, use keywords, and build backlinks to rank higher in search results.
AI agent optimization is different. You’re not gaming an algorithm. You’re communicating directly with the agent. You’re saying “Here’s what I offer, here’s where to find details, and here’s why it matters.”
This shift from ranking pages to getting cited inside AI answers has a name and a playbook: the generative engine optimization playbook. llm.txt is one technical tactic inside that broader stack.
The shift has specific consequences for teams already running classical search programs; the cluster post on what actually changes in GEO vs SEO maps the overlap and the divergence dimension by dimension.
llm.txt is the first step in this direction. It makes your site legible to AI without changing your human-facing design.
The Complementary Proposal
The llm.txt specification includes a complementary recommendation: provide Markdown versions of your pages at the same URL with .md appended.
For example, if you have a page at:
https://example.com/guide.htmlYou would also serve a Markdown version at:
https://example.com/guide.html.mdThis gives AI agents access to clean, structured content without HTML parsing. The agent reads your llm.txt file to find relevant pages, then fetches the .md version for the full content.
This approach scales to any website size. Your llm.txt file curates the important pages, and Markdown versions provide the detailed content.
Adoption Will Be Gradual
Not every website needs llm.txt today. But as AI-powered search tools, research agents, and automated systems become more common, sites without machine-readable content will become less accessible.
Early adopters gain an advantage. They’re discoverable by AI agents that others miss. They’re quotable in AI-generated answers. They’re included in automated research and analysis. For a practical walkthrough of earning citations from ChatGPT, Perplexity, and AI Overviews, the companion post covers the six steps that turn quotability into measured citation share.
Creating your llm.txt file today prepares your website for this future. It’s a small investment that pays dividends as AI traffic grows.
What Comes Next
The llm.txt specification will evolve. Developers will propose extensions, new fields, and enhanced functionality based on real-world usage.
We might see standardized ways to indicate content freshness, specify update frequency, or provide structured data about specific page types.
The core principle will remain: give AI agents the context they need to understand your website efficiently.
The same convention is now playing out at the source-control layer. AI coding agents read a markdown file at the repository root for project context, with each tool reading its own filename: CLAUDE.md for Claude Code, AGENTS.md for OpenAI’s Codex, GEMINI.md for Gemini CLI. The idea is identical to llm.txt, only the consumer is a coding agent reading a repo instead of an assistant reading a live site. The cross-tool explainer lives in the breakdown of CLAUDE.md, AGENTS.md, and GEMINI.md.
Try CopyMarkdown to See How Your Website Converts
Want to see what your website looks like to an AI agent? CopyMarkdown converts any webpage to clean, LLM-ready markdown in one click.
The Chrome extension works on any page. Just click the icon and your page converts instantly. You’ll see exactly what an AI agent sees when it processes your content.
This helps you understand which pages need better structure, where information gets lost in HTML, and how to organize your llm.txt file for maximum clarity.
Try the CopyMarkdown Chrome extension to audit your site from an AI agent’s perspective. Then create your llm.txt file to make those agents more effective.