Cloudflare's Content Independance Day

A quick note on the recent news about Cloudflare’s “Content Independence Day”. It’s pretty massive imo.

https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/

The TLDR is that on July 1st, 2025, Cloudflare flipped the switch and started blocking AI crawlers by default. Crawlers can no longer freely train their models on Cloudflare’s hosted sites.

Cloudflare sits in front of about 20% of the web. This is definitely enough of an arm twist to the crawlers to force a behaviour change.

They cite they are building a marketplace to enable crawling the traffic, a sort of pay-per-crawl microtransaction undercurrent to the web.

In theory, this would enable people that write original content that _furthers knowledge_ to be paid when that content is going to be leveraged by the “front door of the web” (be it Google, ChatGPT, you talking to siri on your phone, etc.).

The previous mode of pay per impression had bad incentives that led to to seas of AI slop, and now tools to “quickly create content”. I am excited to see this come to an end.

One nuance is Google’s AI summaries are driven from their main Googlebot, not their Google-Extended used for Gemini training. So the above changes actually favour Google in their current incarnation in that they’ll still have access to Cloudflare’s hosted sites. I’m sure this will change over time.

The article sites “major publishers” and sure enough I found reference to 9 articles publicly applauding the move (link below). Interestingly enough, I found them via Perplexity, and had no luck finding anything via Google!

It is all cyclical though, I’m sure this will morph and be gamed, and evolve, but the slop creators and their aggregators have had it coming to them for years.

My moral here — do something useful. If your game is grift, it is short. If you’re building something that will try to game an algorithm or a bot, or a person, maybe think again. If you don’t think your mom would be proud of you for it, it’s probably a short game with a really short time horizon.

Attribution: I derived a lot of my above thoughts from listening to Ben Thompson’s thoughts on this: https://stratechery.com/2025/cloudflares-content-independence-day-googles-advantage-monetizing-ai/ (paywall)
Reference: Perplexity’s list of supporting publishers: https://www.perplexity.ai/search/cloudflare-recently-announced-BuCv02IhRp2._R9_ZL2Dlg#0
Reference: Cloudflare’s 20% of the web: https://www.cloudflare.com/about-overview/