HTTP Message Signatures (RFC 9421): A Practical Guide for Bot Authentication

How do you know a crawler is who it claims to be? User-Agent strings can be spoofed in seconds. IP allowlists break when crawlers change infrastructure. You need cryptographic proof.

Enter RFC 9421: HTTP Message Signatures.

The problem with current approaches

Most bot identification today relies on weak signals:

User-Agent strings: Any crawler can claim to be "Googlebot" or "GPTBot". There's no verification. It's the equivalent of trusting someone's name tag at a conference.

IP allowlists: You can publish your crawler IPs, but they change. Maintaining allowlists is a constant game of whack-a-mole. And sophisticated bad actors can route through legitimate-looking infrastructure.

Reverse DNS: Better than User-Agent, but still not cryptographic proof. DNS can be manipulated, and not all legitimate crawlers have consistent reverse DNS.

None of these approaches provide cryptographic certainty about crawler identity.

What is RFC 9421?

RFC 9421 defines a standard way to sign HTTP messages using cryptographic keys. It evolved from the earlier draft-cavage-http-signatures and became an official IETF standard.

The core concept is simple:

1. The sender (crawler) has a private key
2. They sign specific parts of their HTTP request
3. The receiver (publisher) verifies the signature using the sender's public key
4. If the signature is valid, the request is authentically from that sender

This is the same principle behind HTTPS certificates, SSH keys, and code signing—applied to HTTP request authentication.

How it works

The signature components

When a crawler signs a request, it includes two headers:

Signature-Input: Describes what was signed and how

Signature-Input: sig1=("@method" "@authority" "@path" "content-digest");
  keyid="crawler-key-123";alg="ed25519";created=1704067200

Signature: The actual cryptographic signature

Signature: sig1=:BASE64_ENCODED_SIGNATURE:

What gets signed

The crawler chooses which request components to include in the signature:

Component	Description	Example
@method	HTTP method	GET, POST
@authority	Host header	example.com
@path	Request path	/articles/123
@query	Query string	?page=2
content-digest	Hash of request body	For POST/PUT requests

By signing these components, the crawler proves:

• They made this specific request
• To this specific host
• At this specific time
• The request wasn't tampered with in transit

A complete example

Here's what a signed request looks like:

GET /api/articles HTTP/1.1
Host: publisher.example.com
Date: Thu, 16 Jan 2026 12:00:00 GMT
Signature-Input: sig1=("@method" "@authority" "@path" "date");
  keyid="https://crawler.example.com/.well-known/jwks.json#key-1";
  alg="ed25519";created=1737028800
Signature: sig1=:dGhpcyBpcyBhIHNpZ25hdHVyZSBleGFtcGxl...:

The verification flow

When a publisher receives a signed request, verification happens in four steps:

Step 1: Extract the signature

Parse the Signature-Input and Signature headers to get:

• Which components were signed
• The key identifier (where to find the public key)
• The algorithm used
• The signature bytes

Step 2: Fetch the public key

The keyid parameter tells you where to find the public key. Typically this is a JWKS (JSON Web Key Set) endpoint:

https://crawler.example.com/.well-known/jwks.json

The JWKS contains the crawler's public keys:

{
  "keys": [
    {
      "kty": "OKP",
      "crv": "Ed25519",
      "kid": "key-1",
      "x": "BASE64_PUBLIC_KEY"
    }
  ]
}

Step 3: Reconstruct the signature base

Using the Signature-Input parameters, reconstruct exactly what was signed:

"@method": GET
"@authority": publisher.example.com
"@path": /api/articles
"date": Thu, 16 Jan 2026 12:00:00 GMT
"@signature-params": ("@method" "@authority" "@path" "date");
  keyid="...";alg="ed25519";created=1737028800

Step 4: Verify the signature

Use the public key to verify that the signature matches the reconstructed signature base. If it matches, the request is authentic.

Implementation with OpenBotAuth

You don't have to implement RFC 9421 from scratch. OpenBotAuth provides SDKs that handle the complexity.

Express.js middleware

import express from 'express';
import { createVerifier } from '@openbotauth/verifier-client/express';

const app = express();

const verifier = createVerifier({
  verifierUrl: 'https://verifier.openbotauth.org',
  // Cache public keys for performance
  cacheMaxAge: 3600
});

// Verify all incoming requests
app.use(verifier.middleware());

app.get('/api/articles', (req, res) => {
  if (req.oba.signed && req.oba.result?.verified) {
    // Request is from a verified crawler
    console.log('Agent:', req.oba.result.agent);

    // Serve content
    res.json({ articles: [...] });
  } else {
    // Unknown or unverified crawler
    res.status(403).json({ error: 'Verification required' });
  }
});

app.listen(3000);

Python FastAPI

from fastapi import FastAPI, Request, HTTPException
from openbotauth import OpenBotAuthVerifier

app = FastAPI()
verifier = OpenBotAuthVerifier(
    verifier_url="https://verifier.openbotauth.org"
)

@app.middleware("http")
async def verify_bot(request: Request, call_next):
    result = await verifier.verify(request)
    request.state.oba = result
    return await call_next(request)

@app.get("/api/articles")
async def get_articles(request: Request):
    if request.state.oba.signed and request.state.oba.result.verified:
        return {"articles": [...]}
    else:
        raise HTTPException(403, "Verification required")

WordPress plugin

For WordPress sites, install the OpenBotAuth plugin from GitHub (WordPress.org review pending):

# Clone and copy to plugins directory
git clone https://github.com/OpenBotAuth/openbotauth.git
cp -r openbotauth/plugins/wordpress-openbotauth /path/to/wp-content/plugins/

# Activate via WP-CLI
wp plugin activate wordpress-openbotauth

Then configure in Settings → OpenBotAuth.

Why this matters for publishers

HTTP Message Signatures solve the identity problem that has plagued bot management for years:

No more spoofing

A crawler can't fake a signature without the private key. Unlike User-Agent strings, signatures are cryptographically verifiable.

No CDN dependency

Verification happens at your origin server. You're not dependent on your CDN's bot database or their detection algorithms. Switch CDNs anytime without losing bot management capabilities.

Foundation for monetization

Once you can reliably identify crawlers, you can:

• Meter usage by crawler
• Apply different rate limits
• Charge for access
• Generate accurate usage reports

Identity is the prerequisite for everything else.

Audit trail

Every verified request gives you a cryptographic record of who accessed what and when. Useful for billing disputes and compliance.

The crawler side

For this system to work, crawlers need to sign their requests. The major AI companies are moving in this direction:

• OpenAI, Anthropic, and others are adopting HTTP Message Signatures
• The Web Bot Auth draft at IETF is standardizing the approach
• OpenBotAuth provides signing SDKs for crawler operators

If you're building a crawler, signing your requests benefits you too:

• Publishers can identify you as legitimate
• You get better access than anonymous crawlers
• You can participate in commercial programs

Getting started

For publishers

1. Deploy verification: Install the OpenBotAuth SDK or WordPress plugin
2. Monitor first: Run in logging mode to see who's crawling
3. Enforce gradually: Start blocking unverified crawlers on specific paths
4. Build policies: Create rules based on crawler identity

For crawler operators

1. Register: Create an account and register your crawler
2. Generate keys: Create your signing keys and publish your JWKS
3. Integrate signing: Add signature generation to your crawler
4. Join programs: Discover and join publisher programs

Further reading

• RFC 9421 Specification - The official standard
• OpenBotAuth Documentation - Implementation guides
• IETF Web Bot Auth Draft - Ongoing standardization work

Start integrating

Pick your stack:

• WordPress Plugin — 5-minute install
• Zero-code Proxy — npx @openbotauth/proxy
• Node.js SDK — Express / Next.js middleware
• Python SDK — FastAPI / Flask middleware

---

Ready to implement cryptographic bot verification? Check out our developer documentation or request access to OpenBotAuth Cloud.