This documentation is also published as Markdown for efficient machine reading: the whole site is indexed at /llms.txt, and every page has a clean Markdown copy at the same URL with .md appended. These are generated from the same source and cost far fewer tokens to read than this rendered HTML.

Skip to main content

May 4, 2026

Built for LLMs — llms.txt support

By Phil Scott

When an LLM crawler reads your documentation, it parses rendered HTML — nav chrome, sidebars, highlighting spans — to recover the prose underneath. The llms.txt convention skips that by handing crawlers clean markdown directly. Pennington now generates it.

An index and a markdown copy of every page

During the static build, Pennington produces two things:

  • llms.txt — an index of the whole site, organized by your navigation tree. Drop an llms-header.txt in your content root and its contents become the index header. (Not llms.txt — a static file at that path would shadow the generated one.)
  • a co-located markdown copy of every page at <page>.md, beside its output folder — a front-matter-stripped copy an agent reaches by appending .md to the page URL (the home at /index.md).

The markdown is real markdown, not a degraded HTML dump. The converter handles highlighted code blocks, tabbed groups, and GFM alerts, so a code sample survives the round trip intact. The llms.txt how-to covers the setup.

Agents find it on their own

Generating the files isn't enough if nothing points to them. DocSite and BlogSite emit a <link rel="alternate" type="text/markdown"> tag in every content page's head, pointing at that page's co-located .md copy — the standard way to advertise an alternate representation. And because tools like WebFetch strip <head> before an agent sees it, Pennington also drops a paired hidden hint at the top of the <body>, so an agent reading the page the hard way still learns there's a clean copy.

Content that's only for agents

Two opt-in conventions let you write content for llms.txt without producing an HTML page. The first is a *.llms.md file. The second is WithLlmsTxtEntry on a MapGet endpoint, for serving the markdown dynamically:

csharp 
app.MapGet("/agents/architecture.md", () => Results.Text(overview, "text/markdown"))
   .WithLlmsTxtEntry("Architecture overview", "A high-level map for agents.");

Useful for an agent-oriented summary, or machine-readable detail you'd rather keep off the human-facing site.

llms-txtai-agents