AI SEO audit: how to make your site visible to ChatGPT, Perplexity and AI Overviews

What an AI SEO audit is

A classic SEO audit asks whether a search engine will find a page. An AI SEO audit adds a second question: can an AI assistant fetch the text and cite it in an answer? Assistants run their own crawlers, follow their own access rules and prefer their own reading format.

Our audit canon gives this layer three groups: AI (11 checks on crawlers), LLMS (15 checks on llms.txt) and ENT (12 checks on entities and answer extraction). We covered the basic AI bot checks in the technical audit checklist. This guide goes deeper.

Data: how 46 sites treat AI bots

We picked 25 English-language SaaS and developer tools (Stripe, Vercel, Notion, Figma, Canva and others) and 25 large Russian sites. Four Russian sites blocked our script with anti-bot protection, which left 21. For each one we checked robots.txt, the /llms.txt file and the homepage response to the GPTBot user agent.

The takeaway: llms.txt is now standard among Western SaaS, but still rare on the Russian web. And few sites set access on purpose. 34 of 46 never mention AI bots at all, so the bots fall under the generic rules for everyone.

Step 1. robots.txt: split training from search

Large AI companies no longer run a single crawler. One collects text for model training, another builds the assistant search index, and a third fetches a page when a user asks for it in chat. Block the wrong one and you vanish from ChatGPT answers when all you meant to stop was training.

Figma and Canva do exactly this. GPTBot, ClaudeBot and CCBot are blocked, while search and user bots can reach public pages. If you want citations without feeding training sets, copy that setup. If you want maximum reach, allow them all.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
Disallow: /account/

Keep the names current. The hh.ru robots.txt still has a section for Claude-Web, a user agent Anthropic no longer uses, so that rule does nothing. Check AI-010 catches such leftovers: every key bot needs an explicit section with a deliberate policy.

Step 2. Make sure your WAF lets the bot in

robots.txt is only a request. The server makes the real call, and a WAF or CDN bot shield often sits in front of it. In our data the Canva homepage returns 200 to a normal browser and 403 to the GPTBot user agent. For Canva that is a policy choice, but on many sites the same thing happens through a default rule the owner never noticed.

One command tells you: request the homepage with the bot user agent and compare the status code with a normal request. If robots.txt allows the bot but the server answers 403 or a captcha page, the assistant cannot see you.

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://example.com/

In Cloudflare this lives in the AI crawler controls and Bot Fight Mode rules. If you want AI search visibility, allow at least the search bots there, or robots.txt will not help. In the canon this is check AI-007.

Step 3. llms.txt: a file, not a 404 page

llms.txt is a markdown file at the site root that briefly describes the project and links to its key pages. Jeremy Howard proposed the format in 2024. No major search engine officially uses it for ranking, but agents and IDE assistants read it, and it takes half an hour to make.

The most common mistake in our data is a fake llms.txt. On five Russian sites, /llms.txt opens a regular HTML page with a 200 status: a placeholder, the homepage or a SPA shell. For a crawler that is worse than an honest 404, because it gets junk instead of a summary. Checks LLMS-002 and LLMS-003 tell these cases apart.

• The first line is an H1 with the project name, followed by a one- or two-sentence blockquote summary
• Then H2 sections with link lists: a page and a short note after a colon
• Every link returns 200, with no redirects and no gated pages
• The file is built from site content at build time, not edited by hand, so it never drifts from the HTML

# App-Kit

> Free web tools: image compression and conversion, a 400-point SEO audit, text checks.

## Tools

- [SEO Audit](https://app-kit.dev/tools/seo-audit): a 0–100 site score and a fix list
- [Image Compressor](https://app-kit.dev/tools/image-compressor): JPG and PNG to WebP or AVIF

Step 4. Write answers that can be quoted

An assistant builds its answer from fragments. The easier it is to lift a complete thought from your page, the better the odds it cites you. The ENT group of our canon checks this semi-automatically. Here is the core of it.

• A direct answer in the first paragraph: what it is, how to do it, what you get
• Definitions in the form "X is…" for key terms (ENT-005)
• Comparison tables and step lists instead of walls of text (ENT-006)
• A Q&A block with the questions users actually ask (ENT-007)
• Key numbers and facts in the text, not in an image or only in a PDF (ENT-008, ENT-009)
• Content in the HTML on first load, without required JavaScript or login (AI-008)

Use one name for the company and product on every page and external profile. If the site says "App-Kit" but directories say "AppKit" or "App Kit", a model may not see them as one entity.

Step 5. No Google or Bing index, no AI search

AI Overviews draw on the regular Google index, built by Googlebot. Google-Extended has no effect on it, so opting out of Gemini training does not remove you from AI Overviews. You control snippets in Google AI answers with the same directives as regular snippets: nosnippet and max-snippet.

According to OpenAI, ChatGPT search relies on third-party search providers, with Bing as the main one. A site missing from Bing Webmaster Tools risks missing ChatGPT answers too. Set up both consoles, submit the sitemap and turn on IndexNow so Bing learns about new pages within minutes.

The AI SEO audit checklist in one table

As an example we ran the fast audit on figma.com. The AI group scored 70.5 out of 100: AI-001 and AI-003 failed because GPTBot and ClaudeBot are blocked. The LLMS group scored 0, since there is no llms.txt. For Figma, blocking training bots is a deliberate choice. The audit does not override it. It shows the cost so you can decide.

Run the same check on your own site: paste the URL into the form below. The audit runs every check from the table along with the rest of the canon for free, and shows which ones fail and how to fix them.

FAQ

Should I block GPTBot?

Only if you do not want your content used to train OpenAI models. It does not affect ChatGPT search visibility as long as OAI-SearchBot and ChatGPT-User stay allowed.

Does llms.txt affect Google rankings?

No, Google does not use the file. It serves agents and assistants that read the site directly, and it is cheap to make. A broken llms.txt that returns HTML is worse than none.

How do I get into AI Overviews?

There is no application. The page has to be in the Google index, open to Googlebot and give a short direct answer to the query. Blocking Google-Extended does not change eligibility.

Can I check all of this automatically?

The AI and LLMS groups are fully automated: our SEO audit runs them with the rest of the canon in about a minute. Through the MCP server, Claude, Cursor and VS Code call the same audit right from the chat.

Next: the audit and the docs