]> Cloudflare

maxime.guerreiro@gmail.com

Amazon

ulaskira@amazon.com

Cloudflare

ot-ietf@thibault.uk

Web and Internet Transport Web Bot Auth not-yet This document describes a JSON based format for clients using to advertise information about themselves. This document describes a JSON-based "Signature Agent Card" format for signature agent using to advertise metadata about themselve. This includes identity, purpose, rate expectations, and cryptographic keys. It also establishes an IANA registry for Signature Agent Card parameters, enabling extensible and interoperable discovery of agent information. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Web Bot Auth Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction Signature Agents are entities that originate or forward signed HTTP requests on behalf of users or services. They include bots developers, platforms providers, and other intermediaries using . These agents often need to identify themselves, and establish trust with origin servers. Today, the mechanisms for doing so are inconsistent: some rely on User-Agent strings (e.g. MyCompanyBot/1.0), others on IP address lists hosted on file servers (e.g. https://curated-bots.com/ips.json), and still others on out-of-band definitions (e.g. documentation on docs.example.com/mybot). This diversity makes it difficult for operators and origin servers to reliably discover and share a Signature Agent’s purpose, contact information, or rate expectations. Existing discovery mechanisms, such as , do not have the necessary granularity, and pursue different goals. This document introduces a JSON-based "Signature Agent Card" format for Signature Agents, to be published in registries and discovered by servers. It also creates a new IANA registry of "Signature Agent Card Parameters" to ensure extensibility and consistency of future attributes.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Signature Agent Card Signature-Agent header is defined in . This section describes Signature Agent Card, a JSON object containing parameters describing the Signature Agent. Unless otherwise specified, all parameters in this document are OPTIONAL. There MUST be at least one parameter set. Parameters for which the value is unknown MUST be ignored. All string values are UTF-8.

Client Name The client_name parameter provides a friendly identifier for the Signature Agent. Example

• ExampleBot
• My remote browser company

Client URI The client_uri parameter provides inline content or a web page describing the bot: e.g. what does it do, how it handles data it fetches. Only http, https or data:text/plain are allowed. Example

• https://example.com/bot/about.html
• data:text/plain,The Example bot is about providing an example.

Contacts The contacts parameter provides reliable communication channels in URI forms. Typically, this is an email address. Example

• ["mailto:bot-support@example.com"]
• ["https://example.com/contact"]

Logo URI The logo_uri parameter provides an image reference for visual identification. TODO: Recommendation for size and format, if there is a clear consensus or reference we can point to. Example

• data:image/svg+xml;base64,deadbeef
• https://example.com/logo.png

Expected user agent The expected-user-agent parameter specifies one or more User-Agent strings as defined in or prefix matches. Prefixes MAY use * as a wildcard. Example

• Mozilla/5.0 ExampleBot

robots.txt product token The rfc9309-product-token parameter specifies the product token used for robots.txt directives per . Example

• ExampleBot

robots.txt compliance The rfc9309-compliance parameter lists directives from robots.txt that the agent implements. Example

• ["User-Agent", "Disallow"]
• ["User-Agent", "Disallow", "CrawlDelay"]

Trigger The trigger parameter indicates the operational mode of the agent. Valid values:

1. fetcher - request initiated by the user
2. crawler - autonomous scanning

Purpose The purpose parameter describes the intended use of collected data. Values SHOULD be drawn from a controlled vocabulary, such as . Example

• search
• tdm

Targeted content The targeted-content parameter specifies the type of data the agent seeks. Its format is arbitrary UTF-8 encoded string. Example

• SEO analysis
• Vulnerability scanning
• Ads verification

Rate control The rate-control parameter indicates how origins can influence the agent’s request rate. TODO: specify a format Example

• CrawlDelay in robots.txt (non-standard)
• Custom tool
• 429 +

Rate expectation The rate-expectation parameter specifies anticipated request volume or burstiness. TODO: consider a format such as avg=10rps;max=100rps Example

• 500 rps
• Spikes during reindexing

Known URLs The known-urls parameter lists predictable endpoints accessed by the agent. These URLs may be absolute URLs like https://example.com/index.html. They could be relative path like /ads.txt. Or they can use * as wildcard such as *.png. Example

• ["/"]
• ["/ads.txt"]
• ["/favicon.ico"]
• ["/index.html"]

JWKS URI The jwks_uri parameter provides the URL of the signature agent's JWKS keys as defined in . This covers HTTP Message Signatures Directory as defined in . When present, this parameter separates key material discovery from metadata discovery. Clients that need key material SHOULD fetch the directory at the given URL rather than relying on the keys parameter. This separation allows registry operators to host metadata and key material on different endpoints, supporting deployment scenarios where the registry endpoint itself contains signature agent card metadata but the key directory is hosted elsewhere. If both jwks_uri and keys are present, the jwks_uri takes precedence for key discovery. The URI scheme MUST be https. Example

• https://example.com/.well-known/http-message-signatures-directory

IP Address List URI The ips_uri parameter provides the URL of the signature agent's IP address list as defined in . The URI scheme MUST be https. Example

• https://example.com/ips.json

Keys The keys parameter contains a JWKS as defined in . If keys is present, it is RECOMMENDED that the card is signed using . Content-Digest header MUST be included in the covered components. TODO: describe signature, CWS keys. Example

• https://example.com/.well-known/http-message-signatures-directory
• JWKS-directory

Discovery A registry is a list of URLs, each refering to a signature agent card. The URI scheme MUST be one of:

• https (RECOMMENDED): Points to an HTTPS resource serving a signature agent card
• http: Points to an HTTP resource serving a signature agent card
• data: Contains an inline signature agent card

Example

Formal Syntax Below is an Augmented Backus-Naur Form (ABNF) description, as described in . The below definition imports http-URI and https-URI from , and dataurl from .

Out-of-band communication between client and origin A signature agent MAY submit their signature agent card to an origin, or the origin MAY manually add them to their local registry.

Registry Endpoint A registry MAY be provided via a GitHub repository, a public file server, or a dedicated endpoint. The registry SHOULD be served over HTTPS. A client application SHOULD validate the registry format and reject malformed entries.

Authentication A registry endpoint MAY require authentication to restrict access to authorized clients. This allows a registry operator to expose registries without revealing their contents publicly. No specific authentication mechanism is mandated. Implementations MAY use pre-shared keys as mentioned in , bearer tokens as defined in , or HTTP Message Signatures as defined in . HTTP Message Signatures are RECOMMENDED when key rotation without out-of-band coordination is desired.

Efficient Polling with Conditional Requests Registry servers SHOULD include ETag and Last-Modified response header fields as defined in . Clients SHOULD send If-None-Match or If-Modified-Since precondition header fields as defined in on subsequent requests. A server SHOULD respond with 304 (Not Modified) when the registry has not changed, avoiding redundant data transfer.

Caching Registry servers SHOULD include a Cache-Control response header field as defined in to communicate the intended freshness lifetime of the registry content. Clients SHOULD respect these cache directives and SHOULD NOT poll more frequently than indicated.

Signature-Agent header Signature Agent Card format defined in extends the format of Signature-Agent header as defined in . When used for HTTP Message Signatures, and hosted on a well-known URL, Signature Agent Card MAY be discovered via a Signature-Agent header.

Change Notification Pull-based consumption with conditional requests is sufficient for most deployments. When lower notification latency is required (e.g., to promptly act on entry removal), a registry operator MAY implement a push-based change notification mechanism.

Advertising the Notification Endpoint A registry operator that supports change notifications SHOULD advertise its notification endpoint in the registry HTTP response using a Link header field as defined in : ; rel="registry-changes" ]]> The registry-changes link relation identifies an endpoint to which clients may register callbacks out of band. TODO: Register the registry-changes link relation with IANA, or replace it with an extension relation URI.

Callback Registration Registration of a client callback URL with the registry operator is performed out of band. No specific registration protocol is defined by this document. An unsubscribe mechanism SHOULD be considered, and MAY also be out of band.

Notification Requests When an entry is added to or removed from the registry, the registry operator MUST send an HTTP request to each registered callback URL. The format in is as follows: TODO: should we use application/json, a new media-type, permit binary encoding? TODO: should signature-agent be an array? action denotes the operation performed on the registry: 1. put when a new entry has been added, 2. delete when an entry has been removed. The request MUST use POST. It MUST be signed using with a key present in the registry operator's existing signature agent card. This is the signature agent card associated with the registry operator during out-of-band callback registration. The Content-Type SHOULD be application/json. Example notification for an added entry:

Notification Processing Upon receiving a notification, the client MUST verify the HTTP Message Signature against the registry operator's key, discovered via the registry operator's signature-agent card. Notifications with missing or invalid signatures MUST be rejected. A notification is advisory only. The registry endpoint remains the authoritative source of truth. After verifying a put notification, clients SHOULD re-fetch the affected signature agent card to obtain current metadata. For delete notifications, clients SHOULD confirm the removal by re-fetching the full registry. This confirmation does not need to happen synchronously with notification processing. Registry operators SHOULD retry delivery of failed notifications with exponential backoff. Clients that miss notifications will recover on their next conditional pull from the registry endpoint.

Security Considerations Malicious actors may put properties which are not theirs in the registry. If signatures are present, clients MUST verify them. Clients SHOULD reject cards with invalid signatures.

Registry Integrity When a registry is served over HTTPS, TLS provides channel integrity between the server and the client. To additionally bind the registry contents to the registry operator's cryptographic identity, registry servers SHOULD sign their HTTP responses using with a key present in the registry operator's own signature agent card. Clients SHOULD verify such signatures when present.

Binding Delegated Keys to the Directory Authority When a signature agent card delegates key discovery using jwks_uri, clients SHOULD validate the referenced directory response to ensure the authenticity and integrity of the delegated key material. When a directory server provides key material over HTTP or HTTPS, it is RECOMMENDED that it include one HTTP Message Signature per key in the response, with each key used to provide one signature. The signature SHOULD cover @authority with the req flag set, and SHOULD include the created, expires, keyid, and tag signature parameters. The tag value MUST be http-message-signatures-directory. Clients SHOULD validate these signatures using the keys provided by the directory. Clients SHOULD ignore keys from a directory response that do not have a corresponding valid signature binding the key material to the directory authority.

Private Registries Registry endpoints that require authentication as described in limit exposure of registry entries to authorized clients only. Registry operators SHOULD use authenticated endpoints when the enumeration of their registry entries is sensitive.

Privacy Considerations TODO

Access Patterns Registry servers SHOULD avoid logging personally identifiable information from client requests. Clients fetching a registry reveal their interest in its entries; registry servers SHOULD treat access logs as sensitive.

IANA Considerations

Registration template New registrations need to list the following attributes:

Parameter Name:

The name requested (e.g. "useragent"). This name is case sensitive. Names may not match other registered names in a case-insensitive manner unless the Designated Experts state that there is a compelling reason to allow an exception

Parameter Description:

Brief description of the Header Parameter

Change Controller:

For Standards Track RFCs, list the "IESG". For others, give the name of the responsible party. Other details (e.g., postal address, email address, home page URI) may also be included.

Reference:

Where this parameter is defined

Notes:

Any notes associated with the entry

New entries in this registry are subject to the Specification Required registration policy (). Designated experts need to ensure that the token type is defined to be used for both token issuance and redemption. Additionally, the experts can reject registrations on the basis that they do not meet the security and privacy requirements defined in TODO.

Initial Registry content This section registers the Signature Agent Card Parameter names defined in in this registry.

Client Name Parameter

Parameter Name:

client_name

Parameter Description:

A friendly name for your signature agent.

Change Controller:

IETF

Reference:

Notes:

N/A

Client URI Parameter

Parameter Name:

client_uri

Parameter Description:

Describes what the bot does inline with data URI or with an HTTP link to an external resource

Change Controller:

IETF

Reference:

Notes:

N/A

Logo URI Parameter

Parameter Name:

logo_uri

Parameter Description:

Image for a quick visual identification

Change Controller:

IETF

Reference:

Notes:

N/A

Contacts Parameter

Parameter Name:

contacts

Parameter Description:

An array of URI with a reliable communication channel; typically email addresses

Change Controller:

IETF

Reference:

Notes:

N/A

Expected User Agent Parameter

Parameter Name:

expected-user-agent

Parameter Description:

String or fragment patterns

Change Controller:

IETF

Reference:

Notes:

N/A

RFC9309 Product Token Parameter

Parameter Name:

rfc9309-product-token

Parameter Description:

Robots.txt product token your signature-agent satisfies.

Change Controller:

IETF

Reference:

Notes:

N/A

RFC9309 Compliance Parameter

Parameter Name:

rfc9309-compliance

Parameter Description:

Does your signature-agent respect robots.txt.

Change Controller:

IETF

Reference:

Notes:

N/A

Trigger Parameter

Parameter Name:

trigger

Parameter Description:

Fetcher/Crawler

Change Controller:

IETF

Reference:

Notes:

N/A

Purpose Parameter

Parameter Name:

purpose

Parameter Description:

Intended use for the collected data

Change Controller:

IETF

Reference:

Notes:

N/A

Targeted Content Parameter

Parameter Name:

targeted-content

Parameter Description:

Type of data your agent seeks

Change Controller:

IETF

Reference:

Notes:

N/A

Rate control Parameter

Parameter Name:

rate-control

Parameter Description:

How can an origin control your crawl rate

Change Controller:

IETF

Reference:

Notes:

N/A

Rate expectation Parameter

Parameter Name:

rate-expectation

Parameter Description:

Expected traffic and intensity

Change Controller:

IETF

Reference:

Notes:

N/A

Known URLs Parameter

Parameter Name:

known-urls

Parameter Description:

Predictable endpoint accessed

Change Controller:

IETF

Reference:

Notes:

N/A

JWKS URI Parameter

Parameter Name:

jwks_uri

Parameter Description:

URL of the signature agent's key set. This can be an HTTP Message Signatures Directory for key discovery

Change Controller:

IETF

Reference:

Notes:

N/A

IP Address List URI Parameter

Parameter Name:

ips_uri

Parameter Description:

URL of the signature agent's IP address list in format.

Change Controller:

IETF

Reference:

Notes:

N/A

Keys Parameter

Parameter Name:

keys

Parameter Description:

JWKS Endpoint

Change Controller:

IETF

Reference:

Notes:

N/A

References Normative References Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] Open Future Mozilla This document defines a vocabulary for expressing preferences regarding how digital assets are used by automated processing systems. This vocabulary allows for the declaration of restrictions or permissions for use of digital assets by such systems. This document proposes a notational convention to express Concise Binary Object Representation (CBOR) data structures (RFC 7049). Its main goal is to provide an easy and unambiguous way to express structures for protocol messages and data formats that use CBOR or JSON. Cloudflare Google This document describes a method for clients using [HTTP-MESSAGE-SIGNATURES] to advertise their signing keys. It defines a key directory format based on JWKS as defined in Section 5 of [JWK], as well as new HTTP Method Context to allow for in-band key discovery. This document describes a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message. This mechanism supports use cases where the full HTTP message may not be known to the signer and where the message may be transformed (e.g., by intermediaries) before reaching the verifier. This document also describes a means for requesting that a signature be applied to a subsequent HTTP message in an ongoing HTTP exchange. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document defines HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages. This document obsoletes RFC 7234. This document specifies additional HyperText Transfer Protocol (HTTP) status codes for a variety of common situations. [STANDARDS-TRACK] Independent This document defines a standardized JSON format for operators of automated HTTP clients (e.g., web crawlers, AI bots) to publicly disclose their IP address ranges. A consistent, machine-readable format for IP range publication simplifies the task of identifying and verifying legitimate automated traffic, thereby decreasing maintenance load on website operators while reducing the risk of inadvertently blocking beneficial clients. This specification codifies and extends common existing practices to provide a simple yet extensible format that accommodates a variety of use cases. A JSON Web Key (JWK) is a JavaScript Object Notation (JSON) data structure that represents a cryptographic key. This specification also defines a JWK Set JSON data structure that represents a set of JWKs. Cryptographic algorithms and identifiers for use with this specification are described in the separate JSON Web Algorithms (JWA) specification and IANA registries established by that specification. This document defines how to use the Diffie-Hellman algorithms "X25519" and "X448" as well as the signature algorithms "Ed25519" and "Ed448" from the IRTF CFRG elliptic curves work in JSON Object Signing and Encryption (JOSE). This specification defines a method for computing a hash value over a JSON Web Key (JWK). It defines which fields in a JWK are used in the hash computation, the method of creating a canonical form for those fields, and how to convert the resulting Unicode string into a byte sequence to be hashed. The resulting hash value can be used for identifying or selecting the key represented by the JWK that is the subject of the thumbprint. This specification defines a model for the relationships between resources on the Web ("links") and the type of those relationships ("link relation types"). It also defines the serialisation of such links in HTTP headers with the Link header field. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. Informative References Independent Ericsson Blind Five Year Old This document describes best practices for web crawlers. Discussion Venues This note is to be removed before publishing as an RFC. Source for this draft and an issue tracker can be found at https://github.com/garyillyes/cbcp. A new URL scheme, "data", is defined. It allows inclusion of small data items as "immediate" data, as if it had been included externally. [STANDARDS-TRACK] This specification describes how to use bearer tokens in HTTP requests to access OAuth 2.0 protected resources. Any party in possession of a bearer token (a "bearer") can use it to get access to the associated resources (without demonstrating possession of a cryptographic key). To prevent misuse, bearer tokens need to be protected from disclosure in storage and in transport. [STANDARDS-TRACK] n.d. This document provides usage guidance for external Pre-Shared Keys (PSKs) in Transport Layer Security (TLS) 1.3 as defined in RFC 8446. It lists TLS security properties provided by PSKs under certain assumptions, then it demonstrates how violations of these assumptions lead to attacks. Advice for applications to help meet these assumptions is provided. This document also discusses PSK use cases and provisioning processes. Finally, it lists the privacy and security properties that are not provided by TLS 1.3 when external PSKs are used. Team Digitale, Italian Government Red Hat Microsoft This document defines the RateLimit-Policy and RateLimit HTTP header fields for servers to advertise their quota policies and the current service limits, thereby allowing clients to avoid being throttled. This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations. This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279.

Test Vectors TODO

Implementations TODO

Acknowledgments TODO The editor would also like to thank the following individuals (listed in alphabetical order) for feedback, insight, and implementation of this document -

Changelog v02

• Add ips_uri parameter so client can expose IP addresses
• Add optional jwks_uri parameter to separate key material from metadata
• Fix inline data URL example in registry to use valid signature agent card
• Rename "Public list" to "Registry Endpoint" for clarity
• Add authentication guidance for private registry endpoints
• Add conditional GET (ETag/If-Modified-Since) guidance for efficient polling
• Add caching guidance referencing HTTP-CACHE
• Add change notification section: PUT/DELETE webhook signed with HTTP Message Signatures, OOB callback registration, pull as source of truth
• Expand Security Considerations: registry integrity via HTTP Message Signatures, private registry guidance
• Expand Privacy Considerations: customer list exposure, access patterns
• Add normative reference to RFC 8288 (Web Linking)

v01

• Add contributors
• Aligning registry draft with oauth dynamic client registration iana registry
• Add an about-url field
• Add ABNF for discovery of signature-agent card (registry)
• Add precisions about known URLs
• Add placeholder for image size

v00

• Initial draft