Build identity. Get access.
Prove who you are to publishers using cryptographic identity. Get fewer blocks, better access tiers, and a path to pay-per-crawl programs.
The access problem
Blocked by default
Publishers are increasingly blocking AI crawlers. User-agent claims aren't trusted. You get lumped with bad actors.
No differentiation
Responsible crawlers can't distinguish themselves from scrapers. robots.txt doesn't give you a way to prove you respect it.
Compliance is invisible
Even when you do the right thing, publishers can't tell. There's no audit trail showing you followed their rules.
Why register with OpenBotAuth
Cryptographic identity that publishers can verify. A path from blocked to premium access.
Verifiable Identity
Publishers know exactly who you are. No spoofing, no ambiguity. Your signature proves your identity.
Fewer Blocks
Publishers using OpenBotAuth allow verified crawlers while blocking unknowns. Identification = access.
Access Tiers
Enroll in publisher programs for higher rate limits and premium content access. Pay-per-crawl when needed.
How it works
Generate keys
Create a cryptographic key pair. Your private key stays with you. The public key goes in your JWKS.
# Generate a key pair openbotauth keys generate --alg ES256 # Output: public.jwk, private.jwk
Register your crawler
Create an identity in the OpenBotAuth registry. Include your organization info, crawler purpose, and JWKS endpoint.
{
"crawler_id": "acme-search-bot",
"organization": "Acme AI Inc.",
"purpose": "Search index",
"jwks_uri": "https://acme.ai/.well-known/jwks.json"
}Sign your requests
Use HTTP Message Signatures (RFC 9421) to sign each request with your private key. The signature header proves your identity.
GET /article/123 HTTP/1.1
Host: publisher.com
Signature-Input: sig=("@method" "@path" "@authority")
Signature: sig=:BASE64_SIGNATURE:Access granted
Publishers verify your signature against your JWKS. Your identity is confirmed. Access is granted based on their policy.
Key management & compliance
Key Rotation
Rotate keys without downtime. Publish new keys to your JWKS before deprecating old ones. Publishers cache keys intelligently.
Key Revocation
Compromised key? Remove it from your JWKS immediately. The registry propagates revocations within minutes.
Transparency Logs
All key changes are logged. Publishers can audit your identity history. Demonstrates good-faith operation.
Compliance Posture
Verified identity creates an audit trail. When regulators ask how you handle crawling, you have receipts.
SDKs for every stack
Add signature generation to your crawler with a few lines of code.
Python
SDK available
Node.js
SDK available
Go
SDK available
Rust
SDK available
Register your crawler today
Get verifiable identity and join the ecosystem of trusted crawlers.