Cloudflare just set a September 15 default to block AI training and agent crawlers on ad-supported pages. Here is what every service business owner needs to know before that date.
Ido Cohen · Published 2026-07-06 · SEO & Search
Cloudflare just drew a hard line in the AI crawl war, and September 15, 2026 is the date your website's AI search visibility could quietly change — without you touching a single setting. On July 1, Cloudflare announced new three-way controls that separate web crawlers into Search, Agent, and Training categories, and set a default that will block the latter two from ad-supported pages for all new and free-tier sites starting this fall. If you run a service business — a law firm, dental practice, HVAC company, med spa — and you have any ads on your website, this affects whether AI tools like ChatGPT, Perplexity, and Google's own AI Mode can read and cite your content going forward.
The old setup was binary: you could block all AI bots or allow all of them. That blunt instrument is gone.
Cloudflare now divides automated web traffic into three distinct crawler categories. According to Cloudflare's own announcement, Search crawlers collect or index content to answer questions later (think: traditional Googlebot behavior). Agent crawlers act in real time on a person's behalf — these are the bots that power ChatGPT's browsing mode, Gemini driving Chrome, and similar "go look that up for me" AI tasks. Training crawlers take your content to train or fine-tune a model — your words permanently absorbed into an AI's architecture.
As Help Net Security reported, "Cloudflare introduced new controls that let website owners manage AI traffic across three categories: Search, Agent, and Training," and the feature is available to all customers, including those on the free plan.
The new default that kicks in September 15 is specific and consequential: Training and Agent crawlers will be blocked on pages that display ads, while Search crawlers stay allowed. Cloudflare's stated logic, as published in their blog, is that "an ad is a signal that a website owner meant for a person to land there and see it — something monetizable that fuels the business."
Here is the twist that makes this story genuinely complicated for service businesses.
Googlebot is a mixed-use crawler. Google's main bot both indexes websites for search results and collects information to train Gemini and power AI features like AI Overviews and AI Mode. Engadget's coverage confirmed that "Google's main crawler, Googlebot, both indexes websites for the company's various search engines and collects information to train Gemini and power AI features."
Cloudflare's new rules apply the most restrictive policy to mixed-use crawlers. According to Help Net Security's reporting, "If a website blocks Training crawlers, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked even if Search crawlers are allowed."
Translation: if you're a new Cloudflare customer or a free-tier site owner and you don't opt out before September 15, your pages with ads could get blocked from Googlebot. That means no new content indexed for Google Search, no appearances in AI Overviews, no local pack updates — the whole SEO stack goes dark on those pages.
TechTimes reporting noted that Cloudflare's stated goal is "to pressure Google and other search providers to separate their crawlers structurally." That is a reasonable long-term policy goal. But you, the service business owner, are caught in the middle of a standoff between Cloudflare and Google while your appointment bookings depend on local search rankings.
Let's be direct: most service business websites display some form of advertising or monetization signals — Google Ads remarketing tags, third-party review widgets that serve ads, affiliate links in blog posts. If Cloudflare's system detects advertising on your pages and you're subject to the new defaults, the September 15 change could silently block the very crawlers that feed AI search.
There are three scenarios to understand:
Scenario 1 — You're a new Cloudflare customer after September 15. The Training and Agent bot block on ad pages is your starting default. You have to manually opt in to allow those crawlers. If you don't, your ad-supported pages won't be crawled by AI agents or training bots — including mixed-use bots like Googlebot.
Scenario 2 — You're an existing free-tier Cloudflare customer. According to TechCrunch, "These changes to the defaults will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers." You have until September 15 to review and opt out through your zone Security settings if you want to preserve the current behavior.
Scenario 3 — You're an existing paid Cloudflare customer who set up your site before September 15. You'll receive advance notification and can opt out. Your current settings won't change automatically — but ignoring the notification means missing a window to make an intentional choice.
The honest risk for a plumber, dentist, or real estate agent is not that your entire site disappears. It is that your blog posts, service pages, and FAQ content — the long-tail content that AI systems quote when someone asks "who's the best HVAC company in [city]" — stops being refreshed in those AI indexes. AI search citations rely on crawlers being able to reach your content regularly. Block those crawlers by default, and you fade from AI-generated answers over time.
Cloudflare also announced the evolution of its Pay Per Crawl marketplace into something called "Pay Per Use." As TechCrunch reported, this model "will allow publishers to charge AI companies when their content creates value, not just when it's fetched." Early partners include Ceramic.ai and You.com.
This is interesting for large media publishers who generate thousands of AI citations. For a local HVAC company or family dentistry practice, it is not relevant today. The economics of charging per AI use only make sense at scale — your plumbing blog is not going to generate meaningful royalty revenue from ChatGPT citations anytime soon. But it does matter directionally: the shift from "block or don't block" to "block, allow, or charge" means the web's relationship with AI crawlers is moving toward a negotiated access model. The service businesses that understand this early will be better positioned when local content licensing eventually becomes a real option.
Source: Cloudflare blog, July 1, 2026; Help Net Security, July 2, 2026.
One more detail worth flagging: Cloudflare's data shows that 36 percent of all crawler activity is driven by mixed-use crawlers that blend Search and Training functions, according to TechTimes. That means blocking Training bots catches a large share of traffic from crawlers you probably thought were "safe."
Even setting aside the Googlebot complication, Agent bots are separately blocked under the new defaults — and those are the bots that matter most for AI answer engines.
When someone asks ChatGPT "find me a good divorce attorney in Nashville," the Agent-type crawler is what goes and reads attorney websites in real time to populate that answer. When Perplexity surfaces a local med spa for a query about Botox pricing, an Agent crawler fetched that page. Blocking Agent crawlers on your ad-supported pages means those real-time AI tools can't read your content to answer user questions.
For service businesses trying to show up in AI-generated local recommendations — a trend that has been accelerating throughout 2026 — this is a meaningful visibility risk. The whole point of optimizing your site for AI answer engines is that crawlers can reach your content. If your Cloudflare defaults block agent bots by September 15, that optimization work becomes less effective.
Cloudflare CEO Matthew Prince framed the platform's position directly: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." That may be the right long-term policy. But your appointment calendar runs on short-term visibility.
You have 71 days before the September 15 defaults activate. Here is how to use them:
1. Find out if your site is on Cloudflare. Go to your website, open your browser's developer tools (F12), and look for a cf-ray header in the Network tab. If it's there, you're on Cloudflare. Alternatively, check with whoever manages your hosting or DNS.
2. Identify which of your pages carry ads. Any page with Google Ads tags, third-party ad networks, or monetized widgets could be flagged as an "ad page" under Cloudflare's detection. Audit your top service pages, location pages, and blog posts.
3. If you're on Cloudflare's Free tier or launching a new site, log into your Cloudflare dashboard and review the AI bot settings in Security → Bots before September 15. Decide consciously whether you want Search-only access or full access for all three crawler types. Don't let the default decide for you.
4. If you're on a paid Cloudflare plan, watch for the advance notification email and respond before the opt-out window closes. Paid plan sites set up before September 15 won't change automatically, but new sites you create after that date will default to the new settings.
5. Talk to your SEO or web agency this week. This is an infrastructure-level change that your marketing team needs to know about. If you outsource your site management, forward them this post and ask them to confirm your Cloudflare settings before September.
6. Consider whether you want Agent bots on your site at all. If you're actively trying to appear in ChatGPT and Perplexity results for local service queries, you want Agent crawlers allowed. If you're primarily optimized for traditional Google Search and don't want AI tools scraping your content, blocking them may be intentional. Make the choice deliberately — don't let a default make it for you.
---
Does this affect my Google Search rankings directly?
Yes, potentially. If Cloudflare's new defaults block mixed-use bots like Googlebot on your ad-supported pages, those pages won't be re-indexed by Google until you adjust your settings. This could gradually hurt rankings for pages that rely on regular crawl updates, like service area pages or blog posts covering seasonal topics. Review your Cloudflare settings before September 15 to ensure Googlebot can still reach the pages you care most about.
My website doesn't run display ads — am I safe from the new defaults?
Cloudflare is using "displays ads" as the trigger for the new blocking defaults, meaning the September 15 changes primarily affect pages Cloudflare detects as ad-supported. If your site has no advertising — no Google Ads tags, no third-party ad widgets, no monetized banners — the default block on Agent and Training bots likely won't apply to your pages. That said, it's still worth logging into your Cloudflare dashboard and verifying, because Cloudflare's ad detection is automated and may flag things you don't expect.
What is the difference between an Agent bot and a Search bot, and why does it matter for my business?
A Search bot crawls your site to build an index that gets queried later — this is how traditional Google Search works. An Agent bot visits your site in real time to answer a specific user question right now — this is how ChatGPT's browsing mode, Perplexity, and similar AI tools work when they pull up fresh local information. For service businesses trying to appear in AI-generated answers to queries like "find a plumber near me open Saturday," Agent bots are the ones doing the work. Blocking them by default means those AI tools can't read your pages to generate those answers.
What is Pay Per Use, and should I sign up for the waitlist?
Pay Per Use is Cloudflare's evolving marketplace that lets publishers charge AI companies when their content generates value for AI systems — not just when the page is fetched. Currently, the early partners are Ceramic.ai and You.com, and the model makes economic sense primarily for high-volume publishers. For most service businesses, the per-citation revenue from a local business blog would be negligible. Watch this space, but don't prioritize it over the more immediate task of auditing your crawler access settings before September 15.
I'm not on Cloudflare — does any of this affect me?
Not directly from Cloudflare's dashboard controls. However, the broader policy pressure Cloudflare is applying — pushing Google, Microsoft, and Apple to separate their mixed-use crawlers — could eventually change how those platforms crawl all sites, not just Cloudflare-protected ones. Additionally, if you're considering switching to Cloudflare for its CDN or security benefits, the new September 15 defaults are something to configure carefully during onboarding rather than accept passively.
---
Sources: