llms.txt and the Future of AI Discoverability

A new standard for talking to AI

The llms.txt file is an emerging specification that helps AI systems understand your website. Created by the team behind fast.ai and documented at llmstxt.org, it's a plain-text Markdown file at your website's root (/llms.txt) that provides a structured summary of what your site offers.

Think of it as robots.txt for the AI era. Where robots.txt tells crawlers what to access, llms.txt tells AI models what your site is about and where to find the important content.

Why llms.txt matters now

AI search engines — ChatGPT, Perplexity, Claude — generate answers by synthesizing information from multiple sources. When these models need to answer "what's the best website audit tool for agencies?", they look for clear, structured information they can cite confidently.

A well-crafted llms.txt gives them exactly that. Instead of forcing the model to parse your entire site and guess what matters, you provide a curated summary: product description, pricing, key features, and links to important pages.

The specification

The llms.txt file follows Markdown format with specific conventions:

• H1 heading — Your product/company name
• Blockquote — A one-sentence description
• H2 sections — Organized by topic (Product, Pricing, Blog, etc.)
• Bullet points with links — Key pages with short descriptions
• Optional section — Links that are helpful but not essential

What to include

For a SaaS product:

• Product name and one-line description
• Key features and capabilities
• Pricing tiers with specifics
• Links to documentation, blog, and tools
• FAQ answers
• API information (if applicable)

For a local business:

• Business name, location, and description
• Services offered with links
• Hours and contact information
• Service areas
• Common questions and answers

llms.txt vs llms-full.txt

The specification supports two files:

llms.txt — A concise directory (~50–100 lines). Lists key pages with links and one-line descriptions. AI models use this for quick orientation.

llms-full.txt — The complete documentation in a single file. Contains the full text of all important pages concatenated into one Markdown document. AI models and developer tools (like Cursor's feature) use this for deep context.