Pay-per-Crawl: What It Is (and What It Isn't)
Understanding the emerging pay-per-crawl model for AI data access. What it means for publishers, AI companies, and the open web.
"Pay-per-crawl" is becoming a buzzword in discussions about AI and publishing. But what does it actually mean? And more importantly, what doesn't it mean?
What pay-per-crawl is
At its core, pay-per-crawl is a monetization model where publishers charge AI companies for accessing their content. Instead of giving away data for free (or blocking crawlers entirely), publishers set prices and AI companies pay for what they use.
The model typically includes:
- Metered access: Track how many requests each crawler makes
- Pricing tiers: Different rates for different use cases
- Settlement: Regular billing based on usage
Think of it like cloud computing pricing. AWS doesn't charge you a flat fee for "using the cloud." They meter your compute, storage, and bandwidth, then bill accordingly.
What pay-per-crawl isn't
It's not a paywall
Paywalls block human readers until they pay. Pay-per-crawl specifically targets machine access. Human visitors can still read your content for free (or via your existing subscription model). The crawler pays.
It's not DRM
Digital rights management tries to prevent copying. Pay-per-crawl doesn't stop crawlers from copying—it charges them for the privilege. The content itself isn't locked down.
It's not anti-AI
Some publishers are blocking AI crawlers entirely. That's a scorched-earth approach. Pay-per-crawl is a commercial relationship—AI companies get data access, publishers get compensated.
It's not automatic revenue
Just because you set a price doesn't mean crawlers will pay it. You need:
- A way to verify crawler identity
- A way to enforce access policies
- AI companies willing to participate
This is where infrastructure like OpenBotAuth comes in.
How it works in practice
- Publisher creates a program: Define your pricing tiers, rate limits, and terms
- Crawler registers: AI companies sign up and agree to your terms
- Access granted: Registered crawlers can access your content
- Usage metered: Every request is tracked by crawler identity
- Settlement: Periodic billing based on metered usage
The identity prerequisite
Pay-per-crawl only works if you can reliably identify crawlers. Without identity:
- You can't meter usage by crawler
- You can't enforce different policies for paying vs. non-paying crawlers
- You can't prove who accessed what
This is why OpenBotAuth focuses on identity first. Cryptographic verification creates the foundation for commercial relationships.
Current state of the market
As of early 2025, pay-per-crawl is still emerging:
- Some publishers have announced deals with individual AI companies
- No standard infrastructure exists for cross-publisher programs
- Most AI companies are in wait-and-see mode
OpenBotAuth is building the infrastructure layer—identity, verification, metering—that enables pay-per-crawl at scale.
The open questions
Several things remain unclear:
- What's fair pricing? The market hasn't settled on rates
- Who administers programs? Publishers directly? Aggregators?
- How do disputes get resolved? What if usage reports don't match?
We're working through these questions with early access partners.
Why this matters
The current situation—AI companies scraping freely, publishers blocking or suing—isn't sustainable. Pay-per-crawl offers a path to:
- Publishers getting compensated for their content
- AI companies getting legitimate access to training data
- A functional market instead of a legal battleground
It's not guaranteed to work. But it's worth building.
Interested in pay-per-crawl for your content? Request early access to learn more about our program tooling.