Quick answer
An AI search readiness audit checks three things: can AI systems reach your pages, can they read what is on them, and can you tell whether they are citing you.
Most of the work is ordinary SEO. Google says so directly in its own AI optimization guide: “From Google Search’s perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.” The same page says you do not need new machine-readable files, AI text files or special markup to appear in Google Search.
What is genuinely different in 2026 is the plumbing: a new class of crawlers with their own rules, a Cloudflare default that started blocking some of them on 15 September 2026, and new reporting in Search Console and Bing Webmaster Tools that did not exist a year ago.
Run this audit twice a year, or after any move to a new CDN, host or platform.
What changed in 2026
If your checklist was written in 2025, these eight items are now wrong or incomplete.
|
2025 assumption |
What is true now |
|
Search Console shows nothing about AI |
A Generative AI performance report launched 3 June 2026, rolled out worldwide 31 August 2026, covering AI Overviews, AI Mode and Discover. Impressions only, no clicks, CTR or queries |
|
You cannot opt out of AI Overviews without leaving Search |
Search Console now has an opt-out toggle for generative AI features, which Google states is not a ranking signal outside those features |
|
Block Google-Extended to stay out of AI Overviews |
Never true. Google-Extended governs Gemini training and grounding, and Google states it “does not impact a site’s inclusion in Google Search” |
|
Add FAQ schema for rich results |
FAQ rich results stopped appearing on 7 May 2026 and the documentation was removed on 15 June 2026 |
|
robots.txt is the whole control surface |
Cloudflare’s Content Signals, the IETF Content-Usage drafts and RSL licensing now sit alongside it. None is a ratified standard |
|
Your robots.txt reflects your actual policy |
From 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on pages that display ads, and migrated existing free customers to those defaults |
|
There is no AI citation reporting |
Bing Webmaster Tools added an AI Performance report in February 2026, extended in June 2026 with Citation Share |
|
AI traffic is invisible in analytics |
GA4 now has an AI Assistant channel for ChatGPT, Gemini, DeepSeek, Copilot and Grok, though it excludes Google’s own AI Overviews and AI Mode |
Step 1: Check which AI systems can reach you
Open yourdomain.com/robots.txt and read it against this table. The important thing most audits get wrong is that blocking a training crawler and blocking a search crawler have completely different consequences.
|
User agent |
Who |
What blocking it does |
|
GPTBot |
OpenAI |
Stops content being used to train foundation models. Does not remove you from ChatGPT search |
|
OAI-SearchBot |
OpenAI |
This is the one that costs visibility. Blocked sites “will not be shown in ChatGPT search answers” |
|
ChatGPT-User |
OpenAI |
User-triggered fetches. OpenAI states robots.txt rules may not apply |
|
OAI-AdsBot |
OpenAI |
Validates ChatGPT ad landing pages, not used for training |
|
ClaudeBot |
Anthropic |
Training crawler |
|
Claude-SearchBot |
Anthropic |
Improves search result quality. Blocking reduces citation chances |
|
Claude-User |
Anthropic |
User-triggered. Unlike OpenAI and Perplexity, Anthropic states all three honour robots.txt |
|
Googlebot |
|
Feeds Search, and therefore AI Overviews and AI Mode. Blocking removes you from both |
|
Google-Extended |
|
Gemini training and grounding only. No effect on Search inclusion or ranking |
|
PerplexityBot |
Perplexity |
Surfaces and links sites in Perplexity results |
|
Perplexity-User |
Perplexity |
User-triggered, and Perplexity states it “generally ignores robots.txt rules” |
|
Applebot-Extended |
Apple |
Generative AI training opt-out. Pages still appear in Siri, Spotlight and Safari results |
|
meta-externalagent, meta-webindexer |
Meta |
Training and Meta AI search indexing |
|
Amazonbot |
Amazon |
Products and services, may train Amazon AI models. Uses cached robots.txt for up to 30 days |
|
Bytespider |
ByteDance |
No official documentation, IP list or robots.txt statement exists. Only edge or server blocking is reliable |
Pass criteria: you have made a deliberate decision on each row, and the search-oriented agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) are allowed unless you have a reason not to be cited.
Then check the layer above robots.txt. This is new and it catches people out. If you are behind Cloudflare, open AI Crawl Control in the dashboard. Since 1 July 2026 Cloudflare groups crawlers as Search, Agent and Training, and from 15 September 2026 pages that display ads block Training and Agent by default. Your robots.txt can say one thing while the edge does another. Check server logs for which agents actually arrive.
One more honest note on blocking: every vendor frames it as forward looking. Disallowing a training crawler today does not remove content from models already trained.
Step 2: Check the snippet controls, not just the crawler rules
If your goal is to control how you appear rather than whether you are crawled, robots.txt is the wrong tool. The controls that work:
- nosnippet, max-snippet:[n] and data-nosnippet on specific elements. Bing has supported data-nosnippet since October 2025, and states marked content is still indexed but excluded from snippets and AI summaries, including Copilot answers.
- The Search Console opt-out toggle for generative AI features, if you want out of AI Overviews and AI Mode specifically. Expect to lose the traffic and impressions from those surfaces.
- noindex if the page should not be in Search at all.
Pass criteria: you are not using robots.txt to try to solve a snippet problem, and you know which pages, if any, carry data-nosnippet.
Step 3: Test whether your content survives without JavaScript
This is the single highest-impact technical check, and it takes two minutes.
Open a key page, view source, and search the raw HTML for a distinctive sentence from your main content. If it is not there, your content only exists after JavaScript runs.
Why it matters: network-level testing by Vercel and MERJ, published in December 2024 across three sites over roughly a month, found that none of the major AI crawlers execute JavaScript, specifically OpenAI’s OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic’s ClaudeBot, Meta’s external agent, ByteDance’s Bytespider and PerplexityBot. They fetch JavaScript files without running them. Google and Applebot were the exceptions, both rendering through browser infrastructure.
No vendor has announced a change since, and no comparable public test has been run, so treat that December 2024 study as the last rigorous evidence rather than a permanent law.
Pass criteria: headings, body copy, prices, product details and contact information all appear in the raw HTML. If they do not, server-render or statically render those pages.
Step 4: Check how the page is chunked
AI systems do not read a page top to bottom. Microsoft’s own guidance puts it plainly: “AI assistants don’t read a page top to bottom like a person would. They break content into smaller, usable pieces.”
That has three practical consequences:
- Each section must stand alone. A paragraph that only makes sense after reading the previous three will be quoted wrongly or skipped.
- Hidden content is risky. Microsoft warns that AI systems may not render content inside tabs or expandable menus. If it matters, keep it visible in the HTML.
- PDFs lose. They “often lack the structured signals (like headings and metadata) that HTML provides”. Anything important in a PDF should also exist as a web page.
Pass criteria: every H2 section answers one question in its first two sentences, tables carry real values rather than “contact us”, and nothing critical lives only in an accordion or a PDF.
Step 5: Audit the content itself
This is where most of the actual gain is, and it is unglamorous.
- Answer first, context second. Put the direct answer in the first paragraph under the heading, then explain.
- Facts over adjectives. Prices, timelines, specifications, eligibility rules, addresses. “Affordable and reliable” cannot be cited. “From S$3,500, delivered in six weeks” can.
- Dates on everything. Published and updated dates, visible on the page and in the markup.
- Named authors with real credentials, and an About page that says who runs the company. See our note on E-E-A-T and topical authority.
- Consistent entity details. Business name, address, phone and UEN should match across your site, Google Business Profile and directories. Contradictory details make a system less likely to use you as a source.
- Topical coverage. One deep page beats five shallow ones, and a cluster of connected pages beats one orphan. See topical authority.
Pass criteria: pick any five pages and check you could pull a citable, specific fact out of each within ten seconds.
Step 6: Check structured data, with realistic expectations
Google is explicit: “Structured data isn’t required for generative AI search, and there’s no special schema.org markup you need to add.” It then says to keep using it anyway, because it is how you qualify for rich results.
So schema is not a ticket into AI answers. It is still worth having, correctly.
- Organization, LocalBusiness, Product, Article, Breadcrumb and Review markup where relevant, matching the visible text. Google specifically asks you to make “sure your structured data matches the visible text on the page”.
- FAQPage is no longer a rich result. It stopped appearing on 7 May 2026. Keeping the markup is harmless, and Google notes other services may still use it, but nobody should be selling FAQ schema as a rich results play in 2026.
- Validate with the Rich Results Test and fix errors rather than adding more types.
Microsoft is slightly warmer on schema, describing it as code “that helps search engines and AI systems understand your content”, while still not claiming it is required for a Copilot citation.
Pass criteria: valid Organization and page-type markup, no invented ratings, no markup describing things that are not on the page.
Step 7: Decide on llms.txt with the facts in hand
A lot of agencies are selling /llms.txt as essential. Here is the state of it.
It is a proposal by Jeremy Howard, published in September 2024 and updated in August 2026: a Markdown file listing curated links for AI systems. It is not a standard, and no major AI vendor documents reading it.
Google’s position is unusually blunt. Its AI optimization guide says: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search”, that “Google Search itself doesn’t use them”, and that doing so “will neither harm nor help your site’s visibility or rankings in Google Search.” Gary Illyes said at a July 2025 Search Central event that Google does not support llms.txt and is not planning to.
Chrome’s Lighthouse muddies the picture by adding an llms.txt audit under an Agentic Browsing category, though even there a missing file is marked Not Applicable rather than failed, because providing it is optional.
Verdict: publishing one is cheap and harmless. Paying for it as an AI visibility service is not defensible. Do the crawler access, rendering and content work first.
Step 8: Check off-site signals
AI answers assemble from multiple sources. Pew’s analysis of AI summaries found 88% cited three or more sources, so the question is whether you are one of the three, not whether you rank first.
- Are you mentioned on sites that cover your category in Singapore: associations, directories, news, partner sites?
- Is your Google Business Profile complete, with categories, hours, services and recent reviews?
- Do third-party pages describe you with the same facts your site uses?
- Are there reviews with substance rather than star ratings alone?
Pass criteria: search your brand plus your main service in ChatGPT, Gemini and Perplexity. Note which sources they cite. Those sources are your off-site target list.
Step 9: Set up measurement properly
You cannot report on AI visibility with GA4 alone. Use three sources together.
Google Search Console, Generative AI report. Covers AI Overviews, AI Mode and generative AI features in Discover. Impressions only: how many times links to your site were shown in a generative AI feature. No clicks, no CTR, no query data, with Google saying more metrics will come. Dimensions available are pages, countries, dates and devices.
Bing Webmaster Tools, AI Performance report. Currently the most concrete free source of AI citation data. It reports Citations, the number of times your content appears as a source in AI-generated answers, and Grounding queries, the phrases the AI used when retrieving your content. Since June 2026 it also shows Intents, Topics, Compare, and Citation Share, your share of all citations shown for the same grounding query. Bing matters more than its search share suggests, because ChatGPT search leans on that index.
GA4, AI Assistant channel. Traffic where the medium matches ai-assistant and the referrer is a known assistant such as ChatGPT, Gemini, DeepSeek, Copilot or Grok. Two caveats: it excludes Google’s AI Overviews and AI Mode, which stay in Organic Search, and assistant traffic often arrives with no referrer at all and lands in Direct.
For the tracking setup, our guide to tracking traffic from AI Overviews walks through the GTM side, and AI Overview citations covers what gets cited.
Pass criteria: the GSC generative AI report is being checked monthly, Bing Webmaster Tools is verified and the AI Performance report is in your reporting, and GA4 shows the AI Assistant channel separately.
Step 10: The ordinary technical checks that still decide everything
None of this is AI-specific, which is the point.
- Indexation: are your important pages actually indexed? Check coverage in Search Console.
- Internal links: no orphan pages, and strong pages linking to the ones you want found.
- Speed and Core Web Vitals on mobile.
- Clean heading hierarchy, one H1, H2s in order.
- Canonicals and redirects that do not contradict each other.
- A sitemap that only lists live, indexable URLs.
Start with a technical website audit and the complete SEO audit checklist if you have not run one recently.
Score your site
Give one point for each pass.
|
# |
Check |
Pass when |
|
1 |
Crawler access |
A deliberate decision on each AI agent, search agents allowed |
|
2 |
Edge policy |
Cloudflare or CDN settings match your robots.txt intent |
|
3 |
Snippet controls |
Using data-nosnippet or GSC opt-out, not robots.txt, for appearance control |
|
4 |
Rendering |
Main content visible in raw HTML |
|
5 |
Chunking |
Sections stand alone, nothing critical hidden in tabs or PDFs |
|
6 |
Fact density |
Every key page has specific, citable facts |
|
7 |
Authorship and dates |
Named authors, visible published and updated dates |
|
8 |
Entity consistency |
Same business details everywhere |
|
9 |
Structured data |
Valid, matching visible content, no dead FAQ rich result claims |
|
10 |
Off-site presence |
Cited on third-party sites, complete Business Profile |
|
11 |
Measurement |
GSC generative AI report, Bing AI Performance, GA4 AI Assistant channel |
|
12 |
Technical health |
Indexed, linked, fast, correct canonicals |
10 to 12: in good shape, focus on content depth and off-site mentions. 6 to 9: typical for a well-maintained site, usually blocked by rendering or measurement gaps. Below 6: fix crawler access and rendering before spending anything on content.
What the data actually says about AI search
Useful for setting expectations with management, with the caveat that measurement methods differ wildly.
How often AI Overviews appear. Semrush, tracking over 10 million keywords, recorded 6.49% of keywords showing an AI Overview in January 2025, a peak near 24.61% in July 2025, and 15.69% by November 2025. Ahrefs, studying 25 million US keywords, found the share rising from 7.6% to 16.48% between February and May 2025. Pew found around 18% of Google searches by its panel in March 2025 produced an AI summary. Numbers above 40% circulate widely and are usually untraceable to a primary source.
Commercial queries are catching up. Semrush found AI Overviews grew 71% on commercial-intent searches between November 2025 and April 2026, and that Google Ads and AI Overviews now appear together on the same results page around twice as often as a year earlier.
The click impact. Pew Research Center, tracking 68,879 Google searches from 900 US adults in March 2025, found users clicked a traditional result in 8% of visits where an AI summary appeared, versus 15% where it did not, and clicked a link inside the summary in 1% of visits. Ahrefs, comparing 300,000 keywords in Search Console data between December 2023 and December 2025, reported a 58% lower average CTR for the top-ranking page when an AI Overview is present. Note that the widely quoted 34.5% figure is the earlier April 2025 version of that study.
For balance, Semrush’s before-and-after comparison on the same keywords found the zero-click rate fell slightly, from 33.75% to 31.53%, which suggests the causal story is messier than the headline numbers.
Singapore. The Reuters Institute Digital News Report 2026, fielded by YouGov across 48 markets in January and February 2026, puts weekly AI chatbot use for news at 11% in Singapore against a 10% global average, and found that only 4% of respondents overall say they always or often click through to the underlying sources from AI answers. Google AI Mode is live in Singapore in English.
The practical reading: AI answers are taking clicks from informational queries first, commercial queries are following, and being cited is worth pursuing for brand presence even when it does not produce a visit.
Frequently asked questions
Is AI search readiness different from SEO?
Mostly no. Google states that optimising for generative AI search is still SEO. The genuinely different parts are crawler access decisions, rendering, and measurement, which is why those are steps 1, 3 and 9 here.
Which AI crawlers should I allow?
Allow the search-oriented ones if you want to be cited: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended are a separate business decision, and blocking them does not remove you from search or AI answers.
Will blocking Google-Extended keep me out of AI Overviews?
No. Google states Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode are part of Search. Use the Search Console opt-out toggle if you genuinely want out.
Why does my page rank on Google but never get cited by ChatGPT?
Three usual causes: OAI-SearchBot is blocked at robots.txt or the edge, the content only exists after JavaScript runs, or the page has no specific facts worth quoting. Check in that order.
Do I need llms.txt?
No. It is harmless to publish and there is no evidence any major system reads it. Do not pay for it as a service.
Do I still need schema?
Yes, for rich results and clarity, not as an entry ticket to AI answers. Keep it accurate and matching the visible page.
Can I see whether AI engines cite my site?
Partly. Bing Webmaster Tools reports citations and citation share. Search Console reports impressions in generative AI features but no clicks or queries. GA4 shows assistant referral traffic in the AI Assistant channel, excluding Google’s own AI surfaces.
How often should I run this audit?
Twice a year, and immediately after changing CDN, host, platform or robots.txt. The Cloudflare default change on 15 September 2026 is a good example of why the edge needs re-checking even when your site did not change.
Run the audit with us
MediaPlus Digital runs AI search readiness audits for Singapore businesses alongside technical SEO, with 13 years in market, a team of 60 and more than 1,800 projects delivered. We will tell you when the answer is boring, because usually it is: fix rendering, add specifics, get the measurement in place.
See our AI SEO services in Singapore, or read GEO vs SEO for how the two disciplines fit together.
All vendor documentation, dates and statistics verified 16 September 2026 from primary sources. This area changes quickly, so re-check before acting on any single figure.