Creators Are Here to Stay. We Support Pay Per Crawl in Shaping the Web’s Future.

By now, we’re all aware that generative AI is dramatically changing the web, leaving content creators — whose work is used to train powerful AI models — under duress.

But just because AI is here to stay doesn’t mean we have no agency in shaping the future. 

Content creators power the open web as we know it, and Mediavine’s mission is to ensure that their unique and diverse voices are heard — and fairly compensated.

While Mediavine has been collaborating with others in pursuit of this mission — like the IAB Tech Lab on its LLM Content Ingest API, as we discussed in our recent webinar — we’re particularly enthusiastic about Cloudflare’s new Pay Per Crawl feature, which was announced on July 1st.

We believe Pay Per Crawl is a significant step toward achieving this goal. It’s a straightforward concept in that it allows content creators to charge AI crawlers for access to their content.

Publishers who opt in will have AI crawlers blocked by Cloudflare; in response, AI crawlers will receive a notification stating the cost to access the content. 

If the AI crawler agrees to the terms and pays Cloudflare for access, Cloudflare will provide it, and subsequently distribute the payment to the publishers.

Cloudflare’s extensive reach — powering nearly 20% of the internet — suggests that this feature is likely to be adopted as a standard by numerous AI companies almost immediately.

Moreover, Cloudflare plays a pivotal role in Mediavine’s partnership with BigScoots. All of the Mediavine publisher-exclusive BigScoots hosting plans have a Cloudflare Enterprise CDN baked in.

This unique integration makes it exceptionally easy for Mediavine to offer this feature to our more than 18,000 publishers at scale. 

We are actively working with BigScoots and Cloudflare as this beta rolls out, and we will work to make this feature available to our publishers on Mediavine hosting plans as soon as possible.

Mediavine is committed to empowering content creators as the driving force behind the future of the web, and this initiative is one of many steps we are taking to that end.

Every day, as the debate around AI, copyright, and data usage unfolds, we are working on behalf of and joining other industry leaders in support of you, our publishers.

We’re in this together, and we’ve got this.