Patreon shifts from requesting AI bots to blocking them
8 reported
Patreon announced on Thursday that it is working with Cloudflare to actively block AI bots from scraping creator content for training purposes, moving beyond its previous approach of simply requesting compliance through robots.txt files. The company stated that the strengthened measures were necessary because AI scraping has become more sophisticated since it first implemented deterrence measures in 2023. Patreon noted that its paywall has historically protected much content, but new discovery tools like a redesigned Home Feed and Quips could expose more material to crawlers. The company is using Cloudflare’s AI Crawl Control technology to update its policies and enforcement tools. A Patreon blog post explained the shift by stating, “Consent shouldn’t depend on whether a scraper chooses to behave.” During testing, individual AI training crawlers’ weekly attempts to access Patreon dropped from “thousands of attempts to zero,” indicating that scrapers had been ignoring the robots.txt file. Patreon said it will continue to allow bots that index pages and direct users back to the platform.
What’s reported
Patreon is working with Cloudflare to directly block AI bots from scraping content for training.
The company previously used robots.txt files to request that AI crawlers not scrape content.
Patreon says AI scraping has become more sophisticated since it first put measures in place in 2023.
New discovery tools like a redesigned Home Feed and Quips could expose more content to crawlers.
Cloudflare offers tools including a marketplace called Pay Per Crawl that lets websites charge AI bots for scraping.
Cloudflare changed its policies so that “mixed-use” crawlers are blocked by default on pages hosting ads.
During testing, individual AI training crawlers’ weekly attempts to access Patreon went from “thousands of attempts to zero.”
Patreon will allow bots that index pages and organize information to send users back to the platform.
Key figures
Drew Rowny, product chief at Patreon
Sources: TechCrunch
