Take control of AI crawler access
Verify who's crawling your site, enforce access policies, and create pay-per-crawl programs—all at your origin, without CDN vendor lock-in.
The challenge
AI crawlers are opaque
User-agent strings can be spoofed. IP ranges change. You can't verify who's actually crawling your content.
robots.txt is insufficient
It's advisory, not enforced. No way to meter usage, tier access, or monetize appropriately.
CDN solutions create lock-in
Bot management at the CDN layer ties you to a vendor. Switch CDNs? Lose your bot policies and data.
Origin verification changes everything
With OpenBotAuth, crawlers prove their identity cryptographically. You verify at your server using open standards.
Cryptographic Identity
Crawlers sign requests with their private keys. You verify against their published JWKS. No spoofing possible.
Works at Your Origin
Use any CDN—or none. OpenBotAuth verification happens at your server, using standard HTTP. No middleware lock-in.
Granular Policy Control
Allow some crawlers, block others. Set rate limits per identity. Tier access based on program participation.
Full Visibility
See exactly who's crawling, how often, and what they're accessing. Export logs. Own your data.
The pay-per-crawl playbook
Create monetization programs that let AI companies pay for access to your content.
Step 1: Define your program
Create access tiers with different pricing and permissions. Free tier for evaluation, paid tiers for production training.
tiers:
- name: Evaluation
rate_limit: 100/hour
price: free
- name: Standard
rate_limit: 10000/hour
price: $0.001/request
- name: Enterprise
rate_limit: unlimited
price: customStep 2: Crawlers enroll
Crawler operators discover your program and request access. You approve (or auto-approve) based on your criteria. Their identity is linked to their tier.
Step 3: Automatic metering
Every verified request is logged with the crawler identity. Usage accumulates against their tier limits. You get real-time dashboards and exportable reports.
Step 4: Settlement
Generate invoices based on metered usage. Export data for your billing system. Handle disputes with tamper-evident logs.
Integration options
Add OpenBotAuth verification to your stack in minutes.
Ready to take control?
Join the early access program and start verifying AI crawler traffic today.