Layer 1 · Findability

SEO foundations for AI visibility

The least glamorous layer, and the one that silently zeroes out everything above it. Most “GEO isn't working” problems are actually crawl problems.

01Crawl access: check this today

The single most common cause of zero AI visibility is that AI crawlers cannot fetch the site. This is rarely deliberate. Cloudflare changed its default configuration to block AI bots in July 2025, which means a large number of sites switched themselves off without anyone making a decision.

The user agents that matter

These fall into three functional groups, and you may want different policies for each:

• Live answer fetchers — ChatGPT-User, Perplexity-User, Claude-User. These fetch a page because a human just asked a question. Blocking these directly removes you from answers.
• Search index builders — OAI-SearchBot, PerplexityBot, Claude-SearchBot. These build the retrieval indexes those answers draw from.
• Training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended. These feed model training. Allowing them affects long-run model memory rather than today's answers, and this is where legitimate publishers most often draw a line.

These groupings come from each vendor's own crawler documentation, which is the authoritative reference for what each agent does: OpenAI, Anthropic, Perplexity and Google each publish one.

A permissive robots.txt

# Allow everything that could cite us.
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

Then verify it worked. robots.txt is a request, not an enforcement mechanism, and it does nothing about a WAF rule or a rate limiter. Check server logs for actual 200 responses to those user agents, not just the absence of a Disallow line. Our own live robots.txt is a working example.

02Rendering: what the crawler actually receives

Googlebot has run a full rendering pipeline for years. Most AI crawlers have not caught up. If your article body is injected client-side, an AI fetcher may retrieve a page containing your nav, your footer, and nothing worth citing.

• Server-render or statically generate all primary content.
• Keep the answer in the initial HTML payload, not behind an accordion that loads on click.
• Content inside <details> elements is in the DOM and is generally fine; content fetched on interaction is not.
• Test with curl -A "GPTBot" https://example.com/page and read the output.

03Structure the crawler reads as meaning

Heading hierarchy is not decoration. Retrieval systems use it to decide chunk boundaries, so a broken hierarchy produces broken chunks.

• One <h1> per page, describing the page's single subject.
• <h2> for each major question or section — phrased the way a person would ask it where that reads naturally.
• No skipped levels. An <h4> directly under an <h2> makes nesting ambiguous.
• Semantic elements: <article>, <section>, <nav>, <table> with real <th scope> headers. Tables in particular are extracted well and quoted often.
• Descriptive link text. “Read our schema guide” carries meaning; “click here” carries none.

04Internal linking as a topic map

Internal links do two jobs here. They distribute crawl priority, and they tell a machine which concepts belong together. A hub page that links to twelve focused sub-pages, each linking back with descriptive anchor text, reads as a coherent topical cluster. Twelve orphaned posts do not.

Practical rules: every page reachable within three clicks of the homepage; every important page linked from at least three other pages; anchor text that names the target's subject; and a breadcrumb trail on every non-home page, marked up as BreadcrumbList.

05Speed and stability still count

An answer fetcher operating in real time has a timeout. A page that takes four seconds to return HTML may simply be dropped from consideration while a faster competitor is used. Core Web Vitals were built for user experience, but the underlying property — fast, stable, complete HTML on first response — matters more for machine retrieval than it ever did for ranking.

Foundation solid? Move up to answer engine optimization.