Cloudflare's September 15 AI Bot Defaults: The Switch That Could Hide You From Google

Published August 10, 2026 by AldoMedia

Every headline says Cloudflare is blocking AI bots. For a local business that part is close to a non-event. What is worth five minutes of your time is a reversal buried in Cloudflare's own documentation: a setting that today deliberately spares Googlebot stops sparing it on September 15, 2026.

On July 1, 2026, its second annual Content Independence Day, Cloudflare announced new AI crawler controls along with new defaults that take effect September 15, 2026. Cloudflare says it handles traffic for about 20 percent of the web, so when it changes a default, a lot of sites move at once.

Today is August 10, which leaves about five weeks. For most Buffalo and Western New York businesses the work here is a two-minute check, not a project.

What Actually Changes on September 15, 2026

Cloudflare now sorts automated traffic into three buckets and lets you set a policy for each. In its own wording:

• Search: "any behavior that collects or indexes your content, so it can answer questions about it later." The classic crawler, which Cloudflare argues should send traffic back.
• Agent: "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now." A chat assistant fetching your hours page because a customer just asked.
• Training: "a crawler taking your content to train or fine-tune a model."

Each has exactly three settings: Block (on all pages), Block on pages with ads, or Allow. These controls went live July 1, 2026 on every plan, including Free.

The new default arriving September 15 is narrower than the headlines suggest. Cloudflare's blog states it plainly: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."

Read that carefully. Training and Agent get blocked only on pages that display ads. Search stays allowed. If your site does not run advertising, the new default blocks nothing.

The Buried Reversal: Blocking Training Now Catches Googlebot

The sentence that should have led every article on this, from Cloudflare's own announcement:

"Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)."

The mechanism is simple. Googlebot does two jobs on a single crawl, and Cloudflare's developer documentation folds that dual role into Training by definition, describing it as "crawlers taking your content to train or fine-tune a model, including mixed-purpose crawlers that are used both for Training and for Search."

Most restrictive rule wins. If a crawler falls into two buckets and one is blocked, the crawler is blocked. Block Training, lose Search.

What makes this a trap rather than a policy debate is that it reverses current behavior. Cloudflare's documentation today, under the old Block AI bots setting, notes: "This option excludes mixed-purpose bots that are used both for Training and for Search." The legacy toggle currently goes out of its way to protect Googlebot. The same page says that after the change, "Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy Block AI bots option."

Same switch. Same position. Opposite meaning.

The legacy control is retired the same day. Its documentation heading now reads "Block AI bots [Deprecating on September 15, 2026]," and Cloudflare does not document what the toggle maps to when it disappears. Another reason to set the new controls yourself.

Who Is Exposed, and Why It Might Be You Without You Knowing

The obvious response is that you never turned on AI blocking, so none of this applies. That assumption is the problem.

Cloudflare changed its defaults once already. Its 2026 anniversary post restates what happened on July 1, 2025: "We changed the defaults. For all new domains on Cloudflare, AI training crawlers would be blocked by default unless domain owners chose otherwise."

So any domain added to Cloudflare in roughly the last thirteen months may already have training blocking switched on, without the owner deciding it. Plenty of those zones were set up by an agency, a host or a reseller as part of a launch checklist. Nobody wrote it down.

One caveat: Cloudflare's 2025 press release said new domain owners "will now be asked" whether to allow AI crawlers, which does not quite square with the 2026 wording. So do not assume your toggle is on. Assume it might be. And this is a per domain setting, so a portfolio of client sites means an audit pass, not one click.

Nobody Agrees on Who the New Defaults Cover

Almost every write-up skipped this, and it decides whether an existing site is affected at all.

The blog post scopes the change to new domains, and the developer documentation agrees: "On September 15, 2026, Cloudflare will set updated defaults for new domains... Before September 15, all customers can opt out of these new defaults."

Cloudflare's own press release is broader. It covers new customers and new sites for existing customers, then adds a sentence that appears in neither the blog post nor the docs: "On September 15, 2026 these changes will also be made for all existing free customers that have not changed their settings by September 15, 2026 in their dashboard." TechCrunch reported the same wider scope, attributed to the company.

Both versions are Cloudflare's own words and they do not agree. We are not picking a side. The disagreement is itself the argument for checking your settings manually instead of assuming you are outside the blast radius. And nobody addresses the case in between: an existing paid zone that has never touched the setting.

The Two-Minute Check, Step by Step

You do not need to follow the policy debate to do this correctly.

1. Open your domain's Security Settings

Log in to the Cloudflare dashboard and select the domain. Cloudflare's documentation gives the path: "To configure these policies, customers can go to Security Settings > Configure AI bot policies."

2. Look at Training first

If Training is set to Block (on all pages), that is what starts catching Googlebot, Bingbot and Applebot on September 15. Change it to Allow, or to Block on pages with ads. On an ad-free site the second option behaves like Allow while keeping the posture, a fair middle ground if you dislike the idea of your writing training a model.

3. Leave Search on Allow

Always. There is no version of this where a local business benefits from blocking search crawlers. If they cannot reach your pages, none of your Buffalo SEO work matters, because there is nothing to rank.

4. Decide on Agent deliberately

Agent is a judgment call, not an emergency. Blocking it keeps AI shopping and booking assistants from fetching your pages for a customer, so if you want your business reachable by an assistant acting for a real person, allow it. We covered that shift in what happens when AI agents start choosing businesses.

5. Check the legacy toggle, then repeat per domain

The old control sits under Security Settings and Block AI bots. If it is on, the same reversal catches it. Then repeat for every domain in the account. Cloudflare says all customers can opt out of the new defaults before September 15, confirming "that they want no changes on Training crawlers that also crawl for Search purposes." If you would rather not think about it again, that is the button.

What This Actually Means for a Buffalo Small Business

If your site runs no advertising and you have never turned on Cloudflare's AI blocking, September 15 will come and go with zero effect on you. The new default is scoped to pages that display ads, and a plumber, a bakery, a chiropractor or a law firm serves no ad units. The alarm is aimed at publishers who monetize with display advertising, not at a Western New York contractor with a twelve page brochure site.

How does Cloudflare decide a page displays ads? Automated detection, on all plans: scanning HTML for ad unit patterns and ad-server links, matching URLs against advertising filter lists, and reading CSP reports to catch dynamically loaded ad scripts. There are no published accuracy figures and no way to appeal, so whether an affiliate widget could pattern-match as an ad unit is an open question.

The one scenario that genuinely matters is the Training block. If your zone has AI training blocking set to Block on all pages, then on September 15 that same setting starts turning away Googlebot, Bingbot and Applebot. That is not an interpretation, it is Cloudflare's documentation. The consequence is search invisibility, which for a local business is the entire ballgame.

Losing AI training access costs you nothing you can measure. Losing Googlebot costs you the phone ringing. Low probability, high severity, two minute fix. Treat it as maintenance, the same way you would confirm your SSL certificate renewed.

One reported symptom, and why it is not proof

On August 4, 2026, Search Engine Journal covered a site owner's report that setting AI Training to Block was already causing Googlebot and Bingbot to receive HTTP 403 responses on sitemap fetches, ahead of the September date. Google's John Mueller asked for details. That is one user report, Cloudflare has not responded, and it could be an early rollout, a misconfiguration or an unrelated firewall rule. We relay it because it is relevant, not because it shows the change is live early.

The Honest Tradeoff: Protecting Content vs Being Found

The case for blocking is not irrational. Cloudflare's July 1, 2026 figures show 52 percent of crawler requests are now for AI training, up from 22 percent in spring 2025, and that over half of AI crawl traffic re-fetches pages that have not changed. If you write genuinely original content, watching it get ingested with nothing coming back feels bad because it is bad.

The case against blocking, for a small business, is stronger. Being cited in an AI answer is a discovery channel now. If an assistant cannot read your service page, it will recommend the competitor whose page it could read. That is the argument we made in how to block ChatGPT in robots.txt and why you probably should not, and the Cloudflare change strengthens it, because the blocking now costs you Google too.

The two goals are also less separate than people assume. The work that makes you crawlable, fast and clearly structured is the same work that makes you quotable by an AI system, the case we laid out in why SEO is the foundation of AI visibility and in our comparison of GEO and SEO for AI search. Crawl access is the ground floor of both, and Google's crawl budget documentation update shows how fast a blocked crawler turns into missing pages.

For a typical local business: Search allowed, Training set to Allow or Block on pages with ads, Agent allowed unless you have a specific reason not to. Then get back to work.

What You Can Safely Ignore

Pay per crawl, Pay Per Use and the Monetization Gateway. Pay per crawl launched in July 2025, in private beta, reviving the HTTP 402 Payment Required status code so publishers could set a flat per request price. One chronology error is circulating widely: it launched in 2025, not 2026. July 2026 is when Cloudflare evolved it into Pay Per Use, so publishers get paid when their content creates value, such as appearing in an answer, rather than merely when it is fetched. The Monetization Gateway, announced the same day, runs on the x402 protocol and is waitlist-only.

All of it targets publishers with libraries large enough that an AI company wants to license them. A twelve page local business site will not earn a meaningful amount, and payout figures have not been disclosed. Advice telling a small business to monetize content this way is noise.

The Bigger Picture, Without Overclaiming

Cloudflare is open about what it is doing. It frames these defaults as pressure on the mixed-purpose crawlers themselves, saying it hopes the changes "encourage mixed use crawlers to separate out search from agent use and training." Translated: if being classified as a training crawler costs Google search access to a slice of the web, Google has an incentive to run a separate search-only crawler.

That may work. As of August 10, 2026, though, none of Google, Microsoft or Apple has announced such a split, and until one does, the cost of the standoff sits with the site owner in the middle. That is you.

Two caveats. Cloudflare describes these defaults as proposed and still being finalized through feedback and testing "with a deadline of September 15, 2026," so specifics could move. And grouping Applebot as mixed-purpose is Cloudflare's own classification, notable given that Apple documents a separate Applebot-Extended token for AI training opt-out. Everything above was verified against Cloudflare's published sources on August 10, 2026, with the date unchanged. We will re-check the week of September 8.

Your Checklist Before September 15

• Open Security Settings in Cloudflare for each domain you own or manage.
• In Configure AI bot policies, confirm Training is not set to Block (on all pages), and that Search is Allow.
• Check the legacy Block AI bots switch in the same area. The same rule catches it.
• Repeat for every client or side domain. This is per zone, not per account.
• Ask your host or developer who set up the zone and when. If it was after July 1, 2025, check rather than assume.
• After September 15, run a live URL Inspection in Google Search Console and watch Crawl Stats for a sudden drop.

If your site is not behind Cloudflare, none of this applies today. It is still worth knowing who controls the layer between your server and the crawlers, one reason we are particular about where our Buffalo clients are hosted.

The Bottom Line

Cloudflare's new AI crawler defaults are narrower than the headlines suggest and will not touch most local business websites. The change that deserves your attention is the reversal underneath: a switch that today protects Googlebot from your AI training block stops protecting it on September 15, 2026.

Log in. Look at Training. Make sure Search says Allow. Do it for every domain, then forget about it until the week of September 8.

Our work on AI search optimization for Buffalo businesses covers what gets a local company cited in AI answers once the crawlers can reach you.

Sources

• Cloudflare Blog: Your site, your rules, new AI traffic options for all customers (July 1, 2026)
• Cloudflare Docs: Block AI Bots, including the September 15, 2026 defaults notice and dashboard path
• Cloudflare Press Release: Cloudflare Allows the Agentic Internet to Flourish (July 1, 2026)
• Cloudflare Docs: Bot concepts, the Search, Agent and Training classification
• Cloudflare Changelog: New options to manage AI traffic (July 1, 2026)
• Cloudflare Blog: Content Independence Day, one year on, crawler statistics (July 1, 2026)
• Cloudflare Docs: AI Crawl Control, manage AI crawlers
• Cloudflare Blog: Introducing pay per crawl (July 1, 2025)
• Cloudflare Blog: Control content use for AI training (July 1, 2025)
• Cloudflare Blog: Announcing the Monetization Gateway (July 1, 2026)
• Cloudflare Press Release: Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large (July 1, 2025)
• TechCrunch: Cloudflare's new policy pushes AI companies to pay for publishers' content (July 1, 2026)
• Help Net Security: Cloudflare changes AI crawler access rules (July 2, 2026)
• Search Engine Journal: Report That Cloudflare AI Bot Blocking Prevents Googlebot From Indexing Sites (August 4, 2026)

Frequently Asked Questions

Does the Cloudflare September 15 change affect my small business website?

For most local business sites, barely at all. The new default only blocks Training and Agent crawlers on pages that display ads, and a typical contractor, restaurant, clinic or law firm site runs no advertising. The one thing that does matter is whether AI training blocking is already switched on for your domain, because that setting starts catching Googlebot on September 15, 2026.

Is Cloudflare going to block Google?

Not on its own initiative. What changes is that Cloudflare will stop exempting Googlebot from your AI training block. Cloudflare classifies Googlebot, Bingbot and Applebot as mixed purpose crawlers and enforces the most restrictive applicable rule, so a site that blocks Training ends up blocking those search crawlers too.

Where do I check the AI crawler setting in Cloudflare?

Log in to the Cloudflare dashboard, select the domain, then open Security Settings and Configure AI bot policies. The older control is in the same Security Settings area under Block AI bots. Check both, and repeat it for every domain in the account, because this is a per domain setting rather than an account wide one.

Which sites are actually covered by the new defaults?

Cloudflare's own sources do not agree. The Cloudflare blog post and the developer documentation say the new defaults apply to new domains only. Cloudflare's press release, and TechCrunch quoting the company, say they also apply to new sites created by existing customers and to all existing free plan customers who have not changed their settings. Check your own settings regardless of which version is right.

Should a small business block AI crawlers at all?

Usually not. Search should stay allowed permanently, because that is how you get found and cited. Blocking Training gains a small business very little it can measure and now carries the risk of catching search crawlers. Agent is a genuine judgment call, since blocking it keeps AI assistants from fetching your pages on a customer's behalf.

What should I do after September 15 to confirm nothing broke?

Open Google Search Console and run the URL Inspection tool on your homepage and one or two important pages, using the live test. If Google can still fetch the page, crawling is fine. Also watch for a sudden drop in crawl activity in the Crawl Stats report, which would be the first sign that something is turning Googlebot away.