| Internet-Draft | HTTP Message Signatures for Bots | April 2025 |
| Meunier | Expires 17 October 2025 | [Page] |
This document describes an architecture for identifying automated traffic using [HTTP-MESSAGE-SIGNATURES]. The goal is to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence.¶
This note is to be removed before publishing as an RFC.¶
The latest revision of this draft can be found at https://thibmeu.github.io/http-message-signatures-directory/draft-meunier-web-bot-auth-architecture.html. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/.¶
Discussion of this document takes place on the WG Web Bot Auth mailing list (mailto:web-bot-auth@ietf.org), which is archived at https://mailarchive.ietf.org/arch/browse/web-bot-auth/. Subscribe at https://www.ietf.org/mailman/listinfo/web-bot-auth/.¶
Source for this draft and an issue tracker can be found at https://github.com/thibmeu/http-message-signatures-directory.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 17 October 2025.¶
Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Agents are increasingly used in business and user workflows, including AI assistants, search indexing, content aggregation, and automated testing. These agents need to reliably identify themselves to origins for several reasons:¶
Regulatory compliance requiring transparency of automated systems¶
Origin resource management and access control¶
Protection against impersonation and reputation management¶
Service level differentiation between human and automated traffic¶
Current identification methods such as IP allowlisting, User-Agent strings, or shared API keys have significant limitations in security, scalability, and manageability. This document defines an architecture enabling agents to cryptographically identify themselves using [HTTP-MESSAGE-SIGNATURES]. It proposes that every request from bots to be signed by a private key owned by its provider. This way, every origin can validate the service identity.¶
There is an increase in agent traffic on the Internet. Many agents choose to identify their traffic today via IP Address lists and/or unique User-Agents. This is often done to demonstrate trust and safety claims, support allowlisting/denylisting the traffic in a granular manor, and enable sites to monitor and rate limit per agent operator. However, these mechanisms have drawbacks:¶
User-Agent, when used alone, can be spoofed meaning anyone may attempt to act as that agent. It is also overloaded - an agent may be using Chromium and wish to present itself as such to ensure rendering works, yet it still wants to differentiate its traffic to the site.¶
IP blocks alone can present a confusing story. IPs on cloud plaforms have layers of ownership - the platform owns the IP and registers it in their published IP blocks, only to be re-published by the agent with little to bind the publication to the actual service provider that may be renting infra. Purchasing dedicated IP blocks is expensive, time consuming, and requires significant specialist knowledge to set up. These IP blocks may have prior reputation history that needs to be carefully inspected and managed before purchase and use.¶
An agent may go to every website on the Internet and share a secret with them like a Bearer from [OAUTH-BEARER]. This is impractical to scale for any agent beyond select partnerships, and insecure, as key rotation is challenging and becomes less secure as the consumers scale.¶
Using well-established cryptography, we can instead define a simple and secure mechanism that empowers small and large agents to share their identity.¶
This architectures operates solely at the HTTP layer. It allows signatures to be generated and verified without modifying the transport layer or TLS stack. It enables flexible deployment across proxies, gateways, and origin servers, and aligns with existing tooling and infrastructure that already inspect and manipulate HTTP headers.¶
Because the signature is embedded in the request itself, it travels with the message through intermediaries, preserving end-to-end verifiability even when requests are forwarded or transformed within the HTTP layer.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The following terms are used throughout this document:¶
An entity initiating requests through an agent. May be a human operator or another system.¶
An orchestrated user agent (e.g. Chromium, CURL). It implements the HTTP protocol and constructs valid HTTP requests with [HTTP-MESSAGE-SIGNATURES] signatures.¶
An HTTP server receiving signed requests that implements the HTTP protocol and verifies [HTTP-MESSAGE-SIGNATURES] signatures. It acts as a verifier of the signature as defined by [HTTP-MESSAGE-SIGNATURES].¶
A User initiates an action requiring the Agent to perform an HTTP request.
The Agent constructs the request, generates a signature using its signing key,
and includes it in the request as defined in Section 3.1 of [HTTP-MESSAGE-SIGNATURES]
along with Signature-Agent header for discovery for its
verification key.
Upon receiving the request, the Origin ensures it has the verification key for the Agent,
validates the signature, and processes the request if the signature is valid.¶
Signature verification can be performed either directly by origins or delegated to a fronting proxy. Direct verification by origins provides simplicity and control. Proxy verification offloads processing and enables shared caching across multiple origins. The choice depends on traffic volume and operational requirements.¶
[HTTP-MESSAGE-SIGNATURES] defines components to be signed.¶
Agents MUST include the following component:¶
@authorityas defined in Section 2.2.1 of [HTTP-MESSAGE-SIGNATURES]¶
Agents MUST include the following @signature-params as defined in Section 2.3 of [HTTP-MESSAGE-SIGNATURES]¶
createdas defined in Section 2.3 of [HTTP-MESSAGE-SIGNATURES]¶
expiresas defined in Section 2.3 of [HTTP-MESSAGE-SIGNATURES]¶
keyidMUST be a base64url JWK SHA-256 Thumbprint as defined in Section 3.2 of [JWK-THUMBPRINT] for RSA and EC, and in Appendix A.3 of [JWK-OKP] foe ed25519.¶
tagMUST be web-bot-auth¶
The signing key is available to the agent at request time. Algorithms should be registered with IANA as part of HTTP Message Signatures Algorithm registry.¶
The creation of the signature is defined in Section 3.1 of [HTTP-MESSAGE-SIGNATURES].¶
It is RECOMMEND the expiry to be no more than 24 hours.¶
Origins MAY want to prevent signatures from being spoofed or used multiple times by bad actors and thus require a nonce to be added to the @signature-params.¶
Agents SHOULD extend @signature-parameters defined in Section 4.2 as follow¶
nonceBase10 encoded random uint32¶
This nonce MUST be unique for the validity window of the signature, as defined by created and expires attributes.
Because the nonce is controlled by the client, the origin needs to maintain a list of all nonces that it has seen that are still in the validity window of the signature.¶
An Agent SHOULD send a request with the signature generated above. Updating the architecture diagram, the flow look as follow.¶
The Agent SHOULD send requests with two headers¶
Signature defined in Section 4.2¶
Signature-Input defined in Section 4.2¶
Mentioned in a section below, the Agent MAY send request with an additional header
3. Signature-Agent defined in Section 4.5.3¶
Section 5 of [HTTP-MESSAGE-SIGNATURES] defines the Accept-Signature field which can be used to request a Message Signature from a client by an origin. Origin MAY choose to request signatures from clients that did not initially provide them. If requesting, Origins MUST use the same parameters as those defined by the Section 4.2.¶
Upon receiving an HTTP request, the origin has to verify the signature. The algorithm is provided in Section 3.2 of [HTTP-MESSAGE-SIGNATURES]. Similar to regular User-Agent check, this happens at the HTTP layer, once headers are received.¶
Additional requirements are placed on this validation:¶
During step 4, the Origin MAY discard signatures for which the tag is not set to web-bot-auth.¶
During step 5, the Origin MAY discard signatures for which they do not know the keyid.¶
During step 5, if the keyid is unknown to the origin, they MAY fetch the provider directory as indicated by Signature-Agent header defined in Section 4 of [DIRECTORY].¶
This section describes the discovery mechanism for the agent directory.¶
The reference for discovery is a FQDN. It SHOULD provide a directory hosted on the well known registered in Section 4 of [DIRECTORY].¶
We add one field to the directory defined in the other draft: "purpose": Ideally matches some IANA registry for preferences¶
Example¶
{
"keys": {
"kty": "OKP",
"crv": "Ed25519",
"kid": "NFcWBst6DXG-N35nHdzMrioWntdzNZghQSkjHNMMSjw",
"x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs",
"use": "sig",
"nbf": 1712793600,
"exp": 1715385600
},
"signature_agent": "https://directory.test",
"purpose": "rag"
}
¶
A service submitting their key to an origin, or the origin manually adding a service to their trusted list¶
Could be a GitHub repository like the public suffix list. The issue is the gating of such repositories, and therefore its governance.¶
This is defined in the sibling draft. This allows for backward compatibility with existing header agent filtering, and an upgrade to cryptographically secured protocol.¶
Origins must account for the overhead of signature verification in their operations. A local cache of public keys reduces network requests and verification latency. The choice of signing algorithm impacts CPU requirements. Origins should monitor verification latency and set appropriate timeouts to maintain service levels under load.¶
An agent signing key might get compromised.¶
If that happens, the agent SHOULD cease using the compromised key as soon as possible, notify affected origins if possible, and generate a new key pair.¶
To minimise the impact of a key compromise, the origin should support rapid key rotation and monitor for suspicious signature patterns.¶
Implementations MUST NOT reuse a signing key for different purposes. For example, if an agent implementor has two agents they want to differentiate, these should use distinct signing keys and signing key directories.¶
This architecture assumes that automated clients identify themselves explicitly using digital signatures. The identity associated with a signing key is expected to be publicly discoverable for verification purposes. This reduces anonymity and allows receivers to associate requests with specific agents. If an agent wishes not to identify itself, this is not the right choice of protocol for it.¶
The key used for signing MUST NOT be tied to a specific human individual. Keys SHOULD represent a role, company, or automation identity (e.g., "news-aggregator- bot", "example-crawler-v1"). This avoids accidental exposure of personally identifiable information and prevents the misuse of keys for user tracking or profiling.¶
To limit tracking risks, implementations SHOULD avoid long-lived, globally unique key identifiers unless strictly necessary. Key rotation SHOULD be supported, and clients SHOULD take care to avoid signing information that could be used to correlate activity across contexts, especially where sensitive user data is involved.¶
This document has no IANA actions.¶
This draft has a couple of implementations¶
Clients:¶
Servers:¶
A demontstration server has been deployed to https://http-message-signatures-example.research.cloudflare.com/.¶
It uses ed25519 example signing and verifying keys defined in Appendix B.1.4 of [HTTP-MESSAGE-SIGNATURES].¶
TODO acknowledge.¶