If your website runs on Cloudflare, a change arriving on 15 September 2026 could quietly affect how visible your content is inside AI tools like ChatGPT, Perplexity and Google’s AI answers. Cloudflare is moving from a blunt “block all AI bots” switch to a smarter system that sorts crawlers by intent. That is good news, but only if you set it up with your AI SEO goals in mind. Here is what is changing and what to do.
Quick answer: From 15 September 2026, Cloudflare classifies AI crawlers into three types, Search, Agent and Training, and by default blocks Training and Agent bots on ad-monetised pages while keeping Search allowed. If AI visibility matters to you, keep Search crawlers allowed, decide carefully on Agent bots, and check your settings before the deadline so you do not accidentally block bots you want.
What is changing
Cloudflare previously offered a single option to block AI bots outright. It is now replacing that with granular controls based on how a bot uses your content:
- Search crawlers power search and AI answers that cite sources and can send you referral traffic.
- Agent crawlers act on behalf of a user, for example an AI assistant completing a task or shopping.
- Training crawlers collect content to train AI models, with no citation or traffic back to you.
The new defaults (15 September 2026): on pages that show ads, Training and Agent bots are blocked by default, while Search is allowed. Cloudflare’s logic is that an ad signals a page meant for a human to see, so it keeps human attention as the goal and holds back bots that do not send visitors.
The mixed-use trap: around 36% of crawler activity now comes from bots that blend purposes, for example search and training in one crawler. Cloudflare applies the most restrictive rule to these, so if you block Training, a multi-purpose bot, potentially even Googlebot on ad pages, can be blocked too.
You stay in control: you can customise these categories or opt out entirely in your Cloudflare Security settings before the deadline. The controls are available on all plans, including Free.
Why this matters for AI SEO
AI SEO, sometimes called Generative Engine Optimisation, is about being found and cited inside AI answers, not just traditional search results. That makes crawler access a strategic decision, not just a security setting.
- Blocking everything hurts AI visibility. If you blanket-block AI bots, you remove your content from the AI tools your customers increasingly use to research and buy.
- Search crawlers are usually the ones you want. They feed AI answers that cite you and can drive referral traffic. Cloudflare keeps these allowed by default, which suits an AI SEO strategy.
- Agent bots are a judgement call. As “agentic” shopping grows, AI assistants that browse and buy on behalf of users could become a real traffic source, especially for ecommerce. Blocking them protects content but may cost you future visibility.
- Training bots give little back. They do not cite or link to you, so blocking them is usually fine unless you specifically want your brand represented in model training data.
The takeaway: AI SEO is no longer only about content and structure. It now includes deciding, deliberately, which machines are allowed to read you.
What to do before 15 September 2026
Use this quick checklist:
- Log into Cloudflare and open your Security settings. Find the AI crawler or bot controls.
- Keep Search crawlers allowed. This protects your presence in AI answers and search.
- Decide on Agent crawlers by business type. Ecommerce and lead-gen sites should consider allowing them as agentic traffic grows; content sites protecting monetised pages may block them.
- Choose your Training stance. Block to protect content, or allow if you want to be in training data. Either is valid.
- Watch the mixed-use risk. Check that blocking Training does not unintentionally block Googlebot or search crawlers on your ad pages.
- Verify with your logs. After changes, review server or Cloudflare logs to confirm the bots you want are getting through.
- Align it with your AI SEO plan. Crawler access should support your wider strategy for ranking and being cited by AI.
Frequently asked questions
Should I block AI bots on my website?
Not blindly. Block Training bots if you want to protect content, but keep Search crawlers allowed so you stay visible in AI answers and search. Treat Agent bots as a business decision.
Will these changes affect my Google rankings?
They can if a multi-purpose crawler like Googlebot is caught by a Training block on ad-monetised pages. Check your settings so search access stays open.
What is the difference between Search, Agent and Training crawlers?
Search crawlers feed search and cited AI answers, Agent crawlers act for a user in real time, and Training crawlers collect data to train AI models.
Do I need to do anything if I am happy with the current setup?
You can opt out of the new defaults in Cloudflare Security settings before 15 September 2026 to keep things as they are.
Get your site ready for AI search
Crawler controls are one piece of a bigger shift toward AI-driven search. If you want your content to rank and be cited across Google and AI tools, MediaPlus can help you get there. Explore our SEO services in Singapore and read our AI Search Readiness Audit guide, or talk to our team for an AI SEO plan built around how customers actually search today.
