Machines and search engines

Machines and search engines

Every published site serves the machine-readable surfaces below. You do not configure them.

For search engines

Path	What it is
/sitemap-index.xml	Every published page. Drafts, noindex pages and private sites are excluded.
/robots.txt	Generated. It points at the sitemap.

Each page carries a canonical URL, an Open Graph title, description and image, Twitter card tags, and breadcrumb structured data.

Warning

Check that the canonical URL names the hostname you actually publish on. If you connect a custom domain, the canonical must be that domain, or search engines index the subdomain instead of yours.

For readers who follow changes

Path	What it is
/rss.xml	The whole site.
/<section>.rss.xml	One section.

For agents and language models

Path	What it is
/llms.txt	A short index: what this site is, and its pages.
/llms-full.txt	The whole documentation as one text file.
/<path>.md	Any page as Markdown, at its own path plus .md.

The Markdown view returns what the published page says: variables resolved, snippets spliced in, drafts absent. A caller rendering your pages in their own product has neither a variable table nor a snippet store.

The MCP endpoint

A project can also expose an MCP server, so an agent reads your documentation with tools rather than by scraping. See Agents and MCP.

Private sites serve none of this

On a password or workspace site, the sitemap, llms.txt, llms-full.txt and the RSS feeds are not served, and every page is noindex.

Avoiding duplicate content

The Markdown views and llms-full.txt are for machines. They carry headers and canonical behaviour that keep them out of search results, so publishing them does not compete with your HTML pages.