llms.txt Implementation Guide 2026: How to Add the New AI-Engine-Friendly Standard

llms.txt Implementation Guide 2026: How to Add the New AI-Engine-Friendly Standard

llms.txt is the proposed new standard (drafted by Jeremy Howard, September 2024) that gives AI engines a curated, markdown-formatted map of your site — the way robots.txt does for traditional crawlers. As of May 2026, adoption is accelerating: Anthropic, Mintlify, Vercel, Stripe, and ~14 000 sites publish llms.txt files. CapstonAI’s Q1 2026 A/B cohort showed sites with a well-crafted llms.txt got +38% Perplexity citations and +24% ChatGPT citations vs. control. It’s not (yet) parsed by every engine, but the upside is asymmetric and the implementation cost is one afternoon. Here’s the complete how-to.

TL;DR: Implement llms.txt by: (1) creating /llms.txt at your root, (2) following the H1 + blockquote + H2 sections format, (3) listing your most important pages with one-line descriptions, (4) creating an extended /llms-full.txt with full content, (5) referencing it from robots.txt and HTML tag, (6) keeping it markdown-pure, (7) updating it quarterly as content evolves.

Free CapstonAI scan →    Pricing

The 9-step technical playbook

1. Step 1: Understand the llms.txt format. llms.txt is a markdown file at /llms.txt. Required structure: H1 (project name) → blockquote (one-paragraph summary) → optional details paragraphs → H2 sections grouping links → bullet list of [Page title](URL): one-line description. The simpler the better — AI engines parse this as a curated table of contents.
2. Step 2: Create your /llms.txt — minimal working example. Copy-paste starting template (replace with your content):

   # CapstonAI
   
   > CapstonAI is the AI search visibility platform helping B2B and D2C brands get cited by ChatGPT, Perplexity, Claude, and Google AI Overviews. We track citations across 4 engines, generate optimized content, and ship structured-data + schema improvements directly to WordPress and Shopify.
   
   Key context: we serve marketing teams at 50-500 person companies. Our flagship product is the citation tracking + content generation platform launched 2025.
   
   ## Product
   
   - [Platform overview](https://capston.ai/platform/): full product walkthrough
   - [Pricing](https://capston.ai/pricing-2026/): plans from Free to Enterprise
   - [WordPress plugin](https://capston.ai/wordpress-ai-seo-plugin/): auto-deploys schema and llms.txt
   - [Shopify app](https://capston.ai/shopify-ai-seo-app/): AI SEO for product pages
   
   ## Guides
   
   - [How to rank in Perplexity](https://capston.ai/how-to-rank-in-perplexity/): 9-step playbook
   - [How to get cited by Claude](https://capston.ai/how-to-get-cited-by-claude/): 8 tactics
   - [FAQ schema implementation](https://capston.ai/how-to-create-faq-schema-for-ai/): complete guide
   
   ## Optional
   
   - [Glossary](https://capston.ai/glossary-ai-search-geo-aeo-seo/): GEO, AEO, SEO terms
   - [Blog](https://capston.ai/blog/): weekly research and benchmarks

3. Step 3: Create a longer /llms-full.txt with full page content. The companion file /llms-full.txt embeds the full markdown of your most important pages — so an AI engine can read your entire knowledge base in one fetch. Generate it by: exporting your top 20-50 pages as markdown, concatenating with H1 separators. Mintlify, Apify, and CapstonAI auto-generate this from your sitemap.
4. Step 4: Reference llms.txt from robots.txt. Add at the bottom of your robots.txt:

   Sitemap: https://yourdomain.com/sitemap.xml
   
   # AI engine context files
   # https://llmstxt.org
   # llms.txt: https://yourdomain.com/llms.txt
   # llms-full.txt: https://yourdomain.com/llms-full.txt

   Note: the spec doesn’t require this, but it improves discoverability for crawlers that scan robots.txt for hints.

5. Step 5: Add a discovery tag in your . Help bots that don’t yet auto-fetch /llms.txt discover it from any page:

6. Step 6: Curate ruthlessly — quality over coverage. llms.txt is NOT your sitemap. It’s the 20-50 pages you’d want an AI to learn from if it could only read 20-50 things. Pick: pillar guides, pricing, comparison pages, glossary, key product pages. Skip: blog archives, tag pages, paginated lists, low-value content.
7. Step 7: Validate the file is served as text/markdown or text/plain. Test: curl -I https://yourdomain.com/llms.txt — Content-Type should be text/plain or text/markdown. WordPress needs a small functions.php snippet to serve .txt with correct headers. Cloudflare/Vercel handle this automatically if the file extension is .txt.
8. Step 8: Update quarterly + version it. When you ship new pillar content, launch products, or deprecate pages, refresh /llms.txt. Add a discreet # Last updated: 2026-05-01 at the top. Version-control it (commit to git like robots.txt) so changes are auditable.
9. Step 9: Monitor adoption + measure citation impact. Track llms.txt-attributed lift via your prompt panel: query Perplexity/ChatGPT/Claude on the topics covered in your llms.txt before and after deployment. CapstonAI A/B cohort saw measurable lift within 4-8 weeks on prompts mapped to listed pages.

Concrete case study

Real customer pattern (anonymized) showing the impact of this implementation over one quarter:

Common technical errors when implementing llms.txt

• Including every URL on the site (treating it like a sitemap). Defeats the purpose. llms.txt is a curated reading list. Cap at 50 links.
• Writing marketing copy in the blockquote summary. AI engines treat hype as noise. Write the summary as you’d brief a smart analyst in 60 seconds — concrete, factual, no superlatives.
• Forgetting Content-Type headers. If served as text/html or application/octet-stream, some parsers skip it. Verify with curl -I.
• Linking to gated/login-required pages. AI bots can’t fetch them. Only list publicly accessible pages.
• Never updating after the initial publish. Stale llms.txt = stale brand context in AI engines. Quarterly refresh minimum, monthly for fast-moving products.

FAQ — llms.txt

Do major AI engines actually parse llms.txt today?

Anthropic publicly references it. Perplexity has confirmed support. OpenAI and Google haven’t formally announced parsing but CapstonAI’s controlled tests show measurable citation lift on llms.txt-listed pages within 8 weeks. Treat it as high-upside / low-cost: ship it now.

What’s the difference between llms.txt and llms-full.txt?

llms.txt = a curated table of contents (~5-15 KB, just links + descriptions). llms-full.txt = the full text of those pages concatenated in markdown (~100-500 KB). The first is for discovery, the second for deep ingestion. Publish both.

Can plugins auto-generate llms.txt?

Yes. CapstonAI’s WordPress plugin auto-generates and updates llms.txt + llms-full.txt from your selected pages. Mintlify generates it for documentation sites. For Next.js/Astro, llmstxt-generator npm packages exist. Manual is fine too — it’s a flat markdown file.

Tools and related reading

• CapstonAI AI Citation Tracking (measure llms.txt impact)
• How to create FAQ schema for AI
• How to rank in Perplexity
• How to get cited by Claude
• CapstonAI WordPress plugin (auto-llms.txt)
• Glossary: AI Search, GEO, AEO, SEO

Ready to ship llms.txt the right way?

Free CapstonAI scan →

Last updated: May 2026. Sources: Schema.org documentation (https://schema.org/), Wikidata WikiProject Informatics, OpenAI bot documentation (platform.openai.com/docs/bots), Anthropic crawler documentation (anthropic.com/claudebot), Perplexity bot disclosure, Google-Extended documentation (developers.google.com/search/docs/crawling-indexing/google-common-crawlers), llmstxt.org (Howard et al., 2024), CapstonAI Q1 2026 cohort benchmark (86 customers, 24 800 LLM responses analyzed).