One of the biggest debates on the internet, of you could call it that, during the advent of LLM AIs was scraping of websites for its training. Sites that don’t allow it may end up disappointed that their content got scraped anyway, while larger corporations sign deals with AI companies for the privilege. Internet infrastructure company Cloudflare has since introduced the latter, but on a smaller scale, to publishers without large financial backing.
Cloudflare calls the feature Par per Crawl which, as the name suggests, is the AI scraper equivalent of a Pay per View. Companies behind AI crawlers pay on an individual basis to scrape participating websites at a set rate. Website owners can even see if crawlers are scraping for AI training data, for AI search responses, or other purposes.
According to TechCrunch, new websites setting up with Cloudflare will also block AI crawlers by default. Website owners can then grant permission to individual crawlers. But the ability to charge them for it, that’s still in private beta.
“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve” says Matthew Prince, co-founder and CEO of Cloudflare. Though it’s unclear if AI companies see these smaller websites as having enough content to be worth scraping. It’s also unclear which model AI companies see as better, between big licensing deals and paying per crawl.
(Source: Cloudflare [1], [2], TechCrunch)
