Search engines have always had crawlers—automated programs that read your pages, parse your content, and use it to answer user queries. For two decades, SEO (Search Engine Optimization) was how you made sure those crawlers understood your site correctly.
Now there’s a new class of crawler: AI agents. Tools like ChatGPT, Claude, Perplexity, and Google’s AI Overviews don’t just index your content. They read it, synthesize it, and surface it in conversational answers. AEO (Answer Engine Optimization) is the emerging practice of structuring your content so these AI-powered tools can accurately find and represent it. Where SEO optimized for ranking, AEO optimizes for being understood.
The core challenge: HTML is noisy. Your pages are full of navigation menus, scripts, style attributes, and layout scaffolding that means nothing to an AI trying to understand what your page is about. What agents actually need is clean, structured text.
The .md URL Pattern#
SanityPress now serves a clean markdown version of any page just by appending .md to its URL:
- /about 👉 HTML page (for humans)
- /about.md 👉 Markdown (for agents)
- /blog/my-post 👉 HTML page
- /blog/my-post.md 👉 Markdown
Every markdown response includes YAML frontmatter with the page title and description, followed by the full page content stripped of all layout noise, with only the <main> content converted to clean markdown. For blog posts, the frontmatter also includes the author name, giving agents structured attribution alongside the content.
How It Works#
The implementation works from the rendered HTML output. No separate content pipeline, no duplicate data layer. When a .md request comes in:
- The handler self-fetches the original page's HTML
- Strips everything outside
<main>; no header, footer, or navigation - Converts the remaining HTML to markdown using
node-html-markdown - Prepends YAML frontmatter with structured metadata
- Returns
text/markdown
Because it's based on the live HTML output, it automatically reflects your actual content including dynamically fetched data. The blog index page, for instance, fetches the full post list from Sanity and renders it cleanly in markdown.
One extra detail: all relative URLs (images, internal links) are rewritten to absolute URLs in the output. An agent reading /about.md has no base URL to resolve /images/foo.jpg against, so the handler prefixes everything with the request host automatically.
Why This Matters for AEO#
AI agents increasingly browse the web directly. When an agent visits your site, having a clean machine-readable layer means it can request /about.md and immediately work with structured text rather than parsing through nav menus, footers, and CSS class noise.
Beyond direct agent browsing, clean markdown responses make your content far easier to use in AI-powered workflows: a Claude agent researching your docs, a RAG pipeline indexing your blog, or a custom tool built on top of your content.
As AEO becomes as standard a consideration as SEO, having this baked into your site—not bolted on afterward—is the kind of forward-looking detail that separates well-architected sites from the rest.
SanityPress ships with it ready to go.
Get started today! 🚀
