DEFINITIONS • AI SEARCH OPTIMIZATION
What is /llms.txt? The Technical Spec for AI Search Engines
/llms.txt is a web standard proposal defined at llmstxt.org that provides a plain-text Markdown manifest of your documentation, pricing, and system architecture at https://yourdomain.com/llms.txt.
The Problem /llms.txt Solves
Modern web apps run heavy client-side JavaScript. When an LLM crawler (such as GPTBot, ClaudeBot, or PerplexityBot) inspects a pricing page during real-time synthesis:
- It must run headless browser rendering or fall back to raw HTML extraction.
- HTML extraction garbles text from nested divs, client currency toggles, and CSS tables.
- Rather than risk hallucinating pricing in an answer, the model drops your tool from the comparison table.
/llms.txt removes this barrier. It feeds clean Markdown directly into the model's context window. For an audit checklist, see our guide on Auditing Your Website for AI Search Engines.
Comparing Root Manifests
| File Path | Target Consumer | Primary Role |
|---|---|---|
/robots.txt |
All Web Crawlers | Sets crawl permissions and disallow rules per user-agent under RFC 9309. |
/sitemap.xml |
Search Engines (Google, Bing) | Lists canonical page URLs and last-modified dates. |
/llms.txt |
LLM Answer Engines & Agents | Serves plain-text Markdown indexes for real-time RAG context (spec). |
Four Rules for AI Search Retrieval
- Origin TTFB Under 500ms: Live search pipelines run on strict sub-second timeouts. Put static pages on an edge CDN.
- Unblock Legitimate Bot Tokens: Verify
robots.txtdoes not disallowGPTBot,ClaudeBot, orPerplexityBot. - Server-Render JSON-LD: Put schema.org
SoftwareApplicationgraphs directly in<head>. - Host a Clean /llms.txt: Maintain a synchronized Markdown index at your root domain.
Audit Your Live Domain in Real Time
Our edge-streaming auditor tests your domain's live origin TTFB, robots.txt bot rules, JSON-LD schemas, and /llms.txt manifest: