Network Working Group E. Arsentev
Internet-Draft Independent
Intended status: Informational 11 September 2026
Expires: 15 March 2027
Discovery and Retrieval of Publisher-Curated Context Files for Large
Language Model Consumers
draft-arsentev-llm-context-discovery-00
Abstract
Publishers have begun to serve a curated, plain-text summary of a web
origin intended for consumption by large language models and by the
crawlers that feed them, most visibly under the de facto file name
"llms.txt". The practice has no specification, no media type, and —
of direct operational consequence — no discovery mechanism: a
consumer that does not already guess the path cannot learn that such
a file exists.
This document specifies discovery and retrieval for publisher-curated
context files. It defines the well-known URI "llm-context", the link
relation type "llm-context", and an extension record for the robots
exclusion protocol, so that a publisher may advertise a context file
by three independent paths and a consumer may find it without
guessing. It specifies a two-tier arrangement of an index resource
and optional detail resources, states conditional-request and size
requirements that keep retrieval affordable for both parties, and
describes the relationship of this mechanism to the robots exclusion
protocol, to sitemaps, and to work in progress on expressing AI usage
preferences.
This document also reports measurements from an operational
deployment in which twenty crawlers operated by search and language-
model providers issued 44,005 requests to a host over fifteen days
without once retrieving the context file the host was serving, while
the same crawlers retrieved that host's robots.txt 577 times in the
three days after the context file was deployed. The absence of a
discovery mechanism, rather than the absence of interest, is the
hypothesis this document acts upon.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Arsentev Expires 15 March 2027 [Page 1]
Internet-Draft LLM Context File Discovery September 2026
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 15 March 2027.
Copyright Notice
Copyright (c) 2026 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Scope and Non-Goals . . . . . . . . . . . . . . . . . . . 4
1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5
2. Observed Behaviour of an Undiscoverable Convention . . . . . 5
2.1. Method . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2. Results . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3. Limitations . . . . . . . . . . . . . . . . . . . . . . . 7
3. The Context File . . . . . . . . . . . . . . . . . . . . . . 8
3.1. Format . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2. Index and Detail Resources . . . . . . . . . . . . . . . 8
3.3. Language and Variants . . . . . . . . . . . . . . . . . . 9
4. Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.1. The "llm-context" Well-Known URI . . . . . . . . . . . . 9
4.2. The "llm-context" Link Relation . . . . . . . . . . . . . 10
4.3. The "LLM-Context" Record in robots.txt . . . . . . . . . 11
4.4. Conflict Resolution and Precedence . . . . . . . . . . . 12
5. Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.1. Conditional Requests and Caching . . . . . . . . . . . . 12
5.2. Compression . . . . . . . . . . . . . . . . . . . . . . . 13
5.3. On-Demand Consumers . . . . . . . . . . . . . . . . . . . 13
Arsentev Expires 15 March 2027 [Page 2]
Internet-Draft LLM Context File Discovery September 2026
6. Relationship to Other Mechanisms . . . . . . . . . . . . . . 13
6.1. Robots Exclusion Protocol . . . . . . . . . . . . . . . . 13
6.2. Sitemaps . . . . . . . . . . . . . . . . . . . . . . . . 14
6.3. Expression of AI Usage Preferences . . . . . . . . . . . 14
6.4. Structured Data in HTML . . . . . . . . . . . . . . . . . 14
7. Security Considerations . . . . . . . . . . . . . . . . . . . 14
7.1. Content Retrieved Is Untrusted Input . . . . . . . . . . 14
7.2. Cross-Origin References . . . . . . . . . . . . . . . . . 15
7.3. Differential Serving . . . . . . . . . . . . . . . . . . 15
7.4. Resource Consumption . . . . . . . . . . . . . . . . . . 16
7.5. Privacy Considerations . . . . . . . . . . . . . . . . . 16
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16
8.1. Well-Known URI Registration . . . . . . . . . . . . . . . 16
8.2. Link Relation Type Registration . . . . . . . . . . . . . 16
8.3. Robots Exclusion Protocol Record . . . . . . . . . . . . 17
8.4. Media Type . . . . . . . . . . . . . . . . . . . . . . . 17
9. Implementation Status . . . . . . . . . . . . . . . . . . . . 17
10. Open Questions . . . . . . . . . . . . . . . . . . . . . . . 18
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 18
11.1. Normative References . . . . . . . . . . . . . . . . . . 18
11.2. Informative References . . . . . . . . . . . . . . . . . 19
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 20
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 20
1. Introduction
A web origin that wishes to be represented accurately to a large
language model (LLM) faces a problem that ordinary search engine
optimisation does not solve. The origin's HTML is written for a
browser: it carries navigation, boilerplate, advertising, and markup
whose only purpose is presentation. A consumer that ingests that
HTML in order to build a representation of the origin must strip it,
and the stripping is lossy and unsupervised. The publisher, who
knows which pages matter and how the origin should be described, has
no channel through which to say so.
Beginning in 2024 a convention emerged in which publishers serve a
plain-text, Markdown-formatted summary of the origin at the fixed
path "/llms.txt", frequently accompanied by a larger concatenation at
"/llms-full.txt". The convention is widely deployed and widely
discussed. It is also entirely unspecified: there is no media type,
no statement of what a conforming consumer is required to do, no size
discipline, and no way for a consumer to learn that the file exists
other than to request a path it has guessed.
Guessing is the weak point. The web already has two standardised
mechanisms by which an origin tells an automated consumer what it may
and should retrieve — the robots exclusion protocol [RFC9309] and the
Arsentev Expires 15 March 2027 [Page 3]
Internet-Draft LLM Context File Discovery September 2026
Sitemap protocol — and automated consumers use both routinely. A
third convention that is not reachable through either of them is
invisible to any consumer that has not been specifically programmed
for it.
Section 2 reports what that invisibility looks like in an operational
deployment. In summary: on a host serving a context file, crawlers
operated by search and language-model providers issued 13,917
requests over the three days following deployment, of which 577 were
for robots.txt and 249 for sitemap.xml, and none at all were for the
context file.
This document does not attempt to standardise the content of a
context file, which is a matter of editorial judgement and is likely
to remain so. It specifies the three things that must be common for
the mechanism to work at all:
* how a publisher advertises that a context file exists (Section 4);
* how a consumer retrieves it without imposing unbounded cost on
either party (Section 5);
* how the mechanism relates to the robots exclusion protocol, to
sitemaps, and to the expression of AI usage preferences
(Section 6).
The registrations required to make the mechanism usable are requested
in Section 8.
1.1. Scope and Non-Goals
This document is Informational. It describes a mechanism and
requests the IANA registrations that the mechanism needs. It does
not claim consensus of the IETF.
The following are explicitly out of scope:
* *Editorial content.* What a publisher should say about its origin
is not specified. This document constrains the container, not the
prose.
* *Permission.* A context file is an offer of material, not a grant
of rights over it. Whether a consumer may use retrieved content
for training, for retrieval-augmented generation, or for any other
purpose is governed by mechanisms outside this document; see
Section 6.3.
Arsentev Expires 15 March 2027 [Page 4]
Internet-Draft LLM Context File Discovery September 2026
* *Model behaviour.* Nothing here constrains what a model does with
retrieved text, and Section 7 explains why a consumer must not
assume the text is trustworthy.
* *Ranking.* Serving a context file confers no entitlement to be
retrieved, cited, or ranked.
1.2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
context file A resource, served by an origin, whose content is a
publisher-curated description of that origin intended for
consumption by a large language model or by software acting on
behalf of one.
index resource The context file that a discovery mechanism points
to. It is expected to be small and to link onward. See
Section 3.2.
detail resource A larger resource linked from the index resource,
containing full text rather than description.
consumer Software that retrieves a context file. This includes
crawlers that build a corpus offline and agents that fetch on
demand in response to a user's request; Section 5.3 distinguishes
them where the distinction matters.
publisher The party that controls what an origin serves.
origin As defined in [RFC6454].
2. Observed Behaviour of an Undiscoverable Convention
The design in this document is a response to a measurement rather
than to an intuition. This section reports the measurement, its
method, and its limitations. It is presented here, before the
specification, because the specification is difficult to justify
without it.
Arsentev Expires 15 March 2027 [Page 5]
Internet-Draft LLM Context File Discovery September 2026
2.1. Method
A single publisher operates two hosts, "arsentev.ai" and
"ru.arsentev.ai", serving the same content in two languages. Both
hosts began serving a context file at "/llms.txt" and a detail
resource at "/llms-full.txt" on 2026-09-09. Both files are generated
from the same source as the HTML pages and are rebuilt on deployment.
The access logs of the origin server were parsed for the period
2026-08-28 to 2026-09-11 inclusive, a window of fifteen days
containing 331,758 logged requests. Requests were attributed to a
crawler when the request's User-Agent header contained one of twenty-
two tokens published by search and language-model operators as
identifying their crawlers. Attribution by User-Agent is self-
asserted and unverified; Section 2.3 returns to this.
The sizes of the files as served, in octets, were:
+================+===========+================+
| Host | /llms.txt | /llms-full.txt |
+================+===========+================+
| arsentev.ai | 29,146 | 367,587 |
+----------------+-----------+----------------+
| ru.arsentev.ai | 38,982 | 614,939 |
+----------------+-----------+----------------+
Table 1: Context file sizes as served,
2026-09-11
2.2. Results
Over the full fifteen-day window, twenty distinct crawler tokens were
observed, accounting for 44,005 requests. The number of those
requests that retrieved "/llms.txt" or "/llms-full.txt" was zero.
Restricting attention to the three days after the context files were
deployed, 2026-09-09 to 2026-09-11, so that the crawlers had
something to retrieve:
Arsentev Expires 15 March 2027 [Page 6]
Internet-Draft LLM Context File Discovery September 2026
+===========================================+========+
| Quantity | Value |
+===========================================+========+
| Requests attributed to crawlers | 13,917 |
+-------------------------------------------+--------+
| Distinct crawler tokens | 18 |
+-------------------------------------------+--------+
| Retrievals of /robots.txt | 577 |
+-------------------------------------------+--------+
| Distinct crawlers retrieving /robots.txt | 12 |
+-------------------------------------------+--------+
| Retrievals of /sitemap.xml | 249 |
+-------------------------------------------+--------+
| Retrievals of /llms.txt or /llms-full.txt | 0 |
+-------------------------------------------+--------+
Table 2: Crawler requests, 2026-09-09 to 2026-09-11
The comparison that matters is internal to the table. The same
crawlers, in the same window, on the same host, retrieved the two
resources that a standard tells them to retrieve — robots.txt,
defined by [RFC9309], and sitemap.xml — several hundred times, and
retrieved the resource that no standard mentions zero times. The
crawlers were present, were active, and were willing to fetch
metadata about the origin. What they had no way to learn was that a
third piece of metadata was on offer.
2.3. Limitations
The following bound what may be concluded. They are stated in full
because the measurement is a single observation and the temptation to
over-read it is real.
* *Three days.* The post-deployment window is short. A crawler that
probes unfamiliar paths on a long cycle would not appear in it.
The fifteen-day figure of zero retrievals is consistent with the
three-day figure but covers a period during which, for most of its
length, there was nothing at the path to retrieve.
* *One publisher.* Two hosts under one operator, with one content
generator, in two languages. The hosts are modest in traffic and
authority, and crawler behaviour is known to vary with both.
* *Self-asserted identity.* Crawlers were identified by User-Agent
token and not by reverse DNS or by published address range. A
consumer that retrieved the context file while presenting a
browser User-Agent would not have been counted as a crawler. In
the full window, 40 of the 56 requests to the context files came
Arsentev Expires 15 March 2027 [Page 7]
Internet-Draft LLM Context File Discovery September 2026
from a command-line HTTP client operated by the publisher during
verification, and the remainder from ordinary browsers; none
presented a crawler token.
* *Absence of evidence.* Zero retrievals is consistent with the
hypothesis of this document — that consumers cannot find what they
are not told about — and also with other hypotheses, including
deliberate disregard of an unspecified convention. The
measurement distinguishes these poorly. It establishes that the
convention is not working on this deployment, not why.
The design response to this uncertainty is to make the mechanism
cheap for a consumer to ignore: every discovery path defined in
Section 4 is additive, none changes the meaning of an existing
record, and a consumer that implements none of them sees the origin
exactly as it does today.
3. The Context File
3.1. Format
A context file MUST be a UTF-8 encoded text document. It SHOULD be
formatted as Markdown, and a consumer MUST NOT fail to process a
context file that is not valid Markdown; the format is a convenience
for human maintainers and a hint for consumers, not a validation
gate.
A context file MUST NOT be served with a "Content-Type" of "text/
html". Publishers that generate the file through an HTML template
routinely misconfigure this, and a consumer that receives HTML cannot
distinguish a context file from an error page.
A context file SHOULD be served as "text/markdown;charset=UTF-8"
[RFC7763] or as "text/plain;charset=UTF-8". The registration of a
distinct media type is discussed in Section 8.4 and is deliberately
not requested by this document.
3.2. Index and Detail Resources
Deployed practice separates a short summary from a long concatenation
of full text, and the separation is load-bearing. Table 1 shows the
two differing by a factor of twelve on one host and sixteen on the
other; the detail resource on one host exceeds 600 kilobytes, and it
grows with the site without bound.
Accordingly:
Arsentev Expires 15 March 2027 [Page 8]
Internet-Draft LLM Context File Discovery September 2026
* A discovery mechanism defined in Section 4 MUST point at an index
resource, never directly at a detail resource.
* An index resource SHOULD NOT exceed 100,000 octets. A publisher
whose description does not fit SHOULD move material into detail
resources rather than grow the index.
* An index resource MAY link to any number of detail resources,
using ordinary Markdown or plain-text URI references. Relative
references are resolved against the URI of the index resource as
specified in [RFC3986].
* A detail resource SHOULD be divided when it exceeds
1,000,000 octets, so that a consumer may retrieve part of the
material without retrieving all of it.
The size guidance is a floor for interoperability, not a
recommendation to fill it. A consumer MAY impose its own limits and
MUST fail safely — by treating the origin as having no context file —
rather than truncate mid-document in a way that changes meaning.
3.3. Language and Variants
A publisher serving an origin in several languages SHOULD serve a
separate context file per origin where languages occupy separate
origins, as in the deployment of Section 2, and MAY use proactive
content negotiation on a single origin. Where negotiation is used,
the response MUST carry an appropriate "Content-Language" and MUST
include "Accept-Language" in "Vary".
4. Discovery
Three mechanisms are defined. They are independent: a publisher MAY
deploy any subset, and a consumer MAY implement any subset. A
publisher SHOULD deploy at least the well-known URI of Section 4.1,
because it is the only one of the three that a consumer can use
without first retrieving another resource.
Where more than one mechanism is present and they disagree, a
consumer MUST resolve the conflict as specified in Section 4.4.
4.1. The "llm-context" Well-Known URI
This document registers the well-known URI suffix "llm-context"
(Section 8.1), following [RFC8615].
Arsentev Expires 15 March 2027 [Page 9]
Internet-Draft LLM Context File Discovery September 2026
A publisher advertising a context file through this mechanism MUST
arrange that a GET request for "/.well-known/llm-context" on the
origin returns either:
* the index resource itself, with a 2xx status and a "Content-Type"
consistent with Section 3.1; or
* a redirect (301, 302, 307, or 308) to the index resource.
A consumer SHOULD follow at most five redirects and MUST treat a
redirect chain that leaves the origin as requiring the same scrutiny
as any other cross-origin reference; see Section 7.2.
A 404 response MUST be interpreted as "this origin does not advertise
a context file by this mechanism" and MUST NOT be interpreted as an
error to be retried aggressively. A consumer SHOULD NOT re-probe a
404-returning origin more often than once per 24 hours.
The well-known path is preferred over the deployed convention of a
root-level "/llms.txt" for the reason [RFC8615] gives generally: the
root namespace belongs to the publisher, and a specification that
claims a root path claims something it does not own. A publisher
that already serves "/llms.txt" can satisfy this document with a
single redirect and need not move the file.
4.2. The "llm-context" Link Relation
This document registers the link relation type "llm-context"
(Section 8.2).
A publisher MAY advertise a context file by including a "Link" header
field [RFC8288] in responses from the origin:
Link: ; rel="llm-context"; type="text/markdown"
or, in an HTML document, by a link element in the document head:
A publisher using the header form SHOULD emit it on responses for the
origin's home document and SHOULD NOT emit it on every response,
since the field costs bytes on each response and carries no per-
response information.
The "type" parameter, when present, is a hint. A consumer MUST rely
on the "Content-Type" of the retrieved representation and not on the
hint.
Arsentev Expires 15 March 2027 [Page 10]
Internet-Draft LLM Context File Discovery September 2026
This mechanism exists because it is the only one of the three that
works for a consumer which has already retrieved a page and has no
reason to make a second request speculatively. An agent fetching a
single document on a user's behalf (Section 5.3) is in exactly that
position.
4.3. The "LLM-Context" Record in robots.txt
A publisher MAY advertise a context file by including a record in the
origin's robots.txt:
User-Agent: *
Allow: /
LLM-Context: https://example.com/llms.txt
The record's name is "LLM-Context", compared case-insensitively as
[RFC9309] requires of record names. Its value MUST be an absolute
URI. A relative reference MUST be ignored rather than resolved, so
that a parser error cannot cause a consumer to request an unintended
path.
Section 2.2.4 of [RFC9309] permits a crawler to interpret records
that are not part of the robots exclusion protocol, giving "Sitemaps"
as its example, and requires that the parsing of such records "MUST
NOT interfere with the parsing of explicitly defined records". This
record is therefore safe to deploy: a crawler that has never heard of
it parses the rest of the file exactly as before. That property is
the reason for defining it despite the well-known URI already
existing — robots.txt is retrieved by consumers today, as Table 2
shows, and a publisher can therefore reach them through it without
waiting for new code to be deployed on the consumer side. The record
is nevertheless the weakest of the three mechanisms, because it
depends on parsers being as forgiving in practice as the
specification requires.
A record appearing outside any group applies to the origin as a
whole. A publisher SHOULD place it outside all groups. A consumer
encountering the record inside a group MUST treat it as applying to
the origin as a whole, matching the handling Section 2.2.4 of
[RFC9309] gives to the "Sitemaps" record, which "MUST NOT terminate a
group".
Arsentev Expires 15 March 2027 [Page 11]
Internet-Draft LLM Context File Discovery September 2026
A publisher MUST NOT use this record to advertise a context file
whose retrieval the same robots.txt disallows. A consumer
encountering that contradiction MUST honour the exclusion and MUST
NOT retrieve the file: a permission mechanism and a discovery
mechanism disagreeing is resolved in favour of the permission
mechanism, always.
4.4. Conflict Resolution and Precedence
Where several mechanisms advertise different URIs, a consumer MUST
apply the following precedence, highest first:
1. the "Link" relation from a response the consumer has already
received, because it is the most specific to what the consumer is
currently doing;
2. the well-known URI;
3. the robots.txt record.
A consumer MUST NOT retrieve more than one index resource per origin
per retrieval cycle merely because several mechanisms are present.
All three mechanisms are subordinate to the robots exclusion protocol
as a permission mechanism: a consumer MUST evaluate the exclusion
rules of [RFC9309] against the index resource's URI before retrieving
it, regardless of which mechanism advertised it.
5. Retrieval
5.1. Conditional Requests and Caching
A publisher SHOULD serve context files with a strong validator — an
"ETag", or a "Last-Modified" where an entity tag is not available —
as specified in [RFC9110].
A consumer that has previously retrieved a context file MUST issue a
conditional request on subsequent retrievals, using "If-None-Match"
where it holds an entity tag and "If-Modified-Since" otherwise.
This requirement is not a nicety. A detail resource of the size
reported in Table 1, retrieved unconditionally by eighteen crawlers
on the cadence at which those crawlers retrieved robots.txt in
Table 2, would transfer more data per day than the rest of the origin
combined. A mechanism that is expensive to support will not be
supported.
Arsentev Expires 15 March 2027 [Page 12]
Internet-Draft LLM Context File Discovery September 2026
A publisher SHOULD set an explicit "Cache-Control" "max-age" on
context files. A consumer SHOULD NOT retrieve an index resource more
often than once per hour, or a detail resource more often than once
per day, in the absence of an explicit shorter freshness lifetime.
5.2. Compression
A consumer SHOULD offer content coding in "Accept-Encoding", and a
publisher SHOULD serve context files compressed where the consumer
offers it. Context files are highly compressible prose and the
saving is substantial at the sizes involved.
5.3. On-Demand Consumers
Two kinds of consumer are in view and they have opposite cost
profiles. A crawler retrieves on its own schedule, in bulk, and can
afford a detail resource. An agent acting on a user's immediate
request retrieves one document, once, under a latency budget measured
in seconds, and cannot.
The asymmetry is not merely one of politeness. In an agent that
retransmits its accumulated context to a model on every step,
material admitted once is paid for on every subsequent step of the
run, so the cost of retrieving a 600-kilobyte detail resource is not
the transfer but its persistence in the context window;
[I-D.arsentev-agent-run-metrics] specifies an interchange format for
accounting of exactly that effect.
Accordingly an on-demand consumer SHOULD retrieve only the index
resource, and SHOULD NOT retrieve a detail resource unless the user's
request cannot be served from the index. A publisher SHOULD
therefore write the index resource so that it is useful alone, rather
than as a table of contents that is meaningless without its targets.
6. Relationship to Other Mechanisms
6.1. Robots Exclusion Protocol
[RFC9309] answers the question "may this consumer retrieve this URI".
This document answers the question "which URI should this consumer
retrieve". The questions are disjoint and this document does not
alter the answer to the first. A context file is subject to
exclusion like any other resource, and Section 4.3 and Section 4.4
both resolve contradictions in favour of exclusion.
Arsentev Expires 15 March 2027 [Page 13]
Internet-Draft LLM Context File Discovery September 2026
6.2. Sitemaps
A sitemap enumerates URIs and carries no judgement about them: it is
complete by intention. A context file is curated and selective by
intention. A publisher SHOULD serve both, and SHOULD NOT generate
one from the other mechanically, as a context file that is a
transcription of a sitemap communicates nothing that the sitemap did
not.
6.3. Expression of AI Usage Preferences
Work is in progress in the IETF on vocabularies for expressing
preferences about the use of content by AI systems, and on attaching
those preferences to content [I-D.ietf-aipref-vocab]
[I-D.ietf-aipref-attach]. That work and this document are orthogonal
and complementary: the former concerns what a consumer is permitted
to do with material it has obtained, the latter concerns how a
publisher offers material in the first place.
The two interact in one direction only, and the direction matters.
Serving a context file MUST NOT be construed as granting any usage
preference, relaxing any preference expressed by other means, or
waiving any right. A consumer MUST evaluate usage preferences
independently of discovery. A publisher that both restricts usage
and serves a context file is making a coherent statement — "if you
are permitted to use this origin at all, use this description of it"
— and a consumer MUST NOT treat the second half as withdrawing the
first.
6.4. Structured Data in HTML
Structured data embedded in pages describes entities in machine
terms. A context file describes an origin in prose to a consumer
whose native input is prose. They are not substitutes, and a
publisher SHOULD continue to serve structured data unchanged.
7. Security Considerations
7.1. Content Retrieved Is Untrusted Input
This is the consideration that matters most, and it is a consequence
of the mechanism's purpose rather than of any detail of its design.
A context file is written by a publisher for the express purpose of
being placed in the input of a language model. The publisher chooses
every word. A file may therefore contain text crafted to be read by
the model as instruction rather than as content — directions to
disregard prior instructions, to represent the origin in terms the
Arsentev Expires 15 March 2027 [Page 14]
Internet-Draft LLM Context File Discovery September 2026
publisher prefers, to disparage a competitor, or to emit a particular
URI in an answer. Nothing in this document, and nothing that could
be added to it, prevents this: the file is by definition attacker-
chosen text from the point of view of any consumer that does not
trust the publisher.
Therefore a consumer MUST treat the content of a context file as
untrusted data and MUST NOT treat any part of it as instruction. A
consumer that incorporates a context file into a model's input MUST
mark its provenance in a way that survives into the model's input, so
that content originating from an origin is distinguishable from the
consumer's own instructions. Consumers SHOULD apply whatever
isolation they apply to other retrieved web content; a context file
deserves no more trust than an arbitrary HTML page from the same
origin, and its curated presentation may make it appear to deserve
more.
A consumer MUST NOT follow URIs found in a context file automatically
without applying the same policy it applies to other discovered URIs,
including exclusion rules and cross-origin scrutiny.
7.2. Cross-Origin References
A discovery mechanism that names a URI on another origin allows one
party to make assertions about a resource it does not control, and
allows an origin to direct consumers elsewhere.
A consumer MUST NOT attribute the content of a cross-origin index
resource to the advertising origin. A consumer SHOULD decline to
retrieve a cross-origin index resource at all unless it has a
specific reason to accept the indirection, and MUST evaluate the
target origin's own exclusion rules before retrieving it.
7.3. Differential Serving
A publisher can serve a context file that does not correspond to what
a browser is shown, which is the long-standing practice known as
cloaking, applied to a new surface. This document cannot prevent it.
A consumer that relies on a context file SHOULD sample the origin's
ordinary representations and SHOULD reduce its reliance on a context
file that systematically disagrees with them.
Arsentev Expires 15 March 2027 [Page 15]
Internet-Draft LLM Context File Discovery September 2026
7.4. Resource Consumption
The size limits of Section 3.2 and the conditional request
requirements of Section 5.1 exist partly for safety. A publisher can
otherwise advertise an arbitrarily large resource to every consumer
that implements this mechanism, and a consumer that retrieves
unconditionally can otherwise impose unbounded cost on a publisher.
A consumer MUST impose its own ceiling on the size of any retrieved
context file and abandon a retrieval that exceeds it.
7.5. Privacy Considerations
A request for a well-known URI is as identifying as any other request
from the same client and carries no additional information about a
user. A publisher can, however, use a distinct path per consumer to
determine which consumers implement this mechanism; this is the
ordinary observability of an HTTP request and is noted here only
because the well-known path is otherwise uniform.
A publisher MUST NOT place in a context file personal data that it
would not place in a public HTML page. The file is world-readable,
and its purpose is to encourage copying of its contents into corpora
from which withdrawal is not practical.
8. IANA Considerations
8.1. Well-Known URI Registration
IANA is requested to register the following entry in the "Well-Known
URIs" registry established by [RFC8615]:
URI suffix: llm-context
Change controller: IETF
Reference: This document
Status: permanent
Related information: Returns, or redirects to, a publisher-curated
context file describing the origin for consumption by large
language models.
8.2. Link Relation Type Registration
IANA is requested to register the following entry in the "Link
Relation Types" registry established by [RFC8288]:
Arsentev Expires 15 March 2027 [Page 16]
Internet-Draft LLM Context File Discovery September 2026
Relation Name: llm-context
Description: Refers to a publisher-curated context file describing
the origin of the link context, intended for consumption by large
language models.
Reference: This document
8.3. Robots Exclusion Protocol Record
[RFC9309] does not establish an IANA registry of record names, so no
registration is requested. The record name "LLM-Context" is
documented in Section 4.3 of this document. Should a registry be
established, this document requests that "LLM-Context" be entered in
it with this document as reference.
8.4. Media Type
No media type registration is requested. A context file is Markdown
or plain text and the registrations of [RFC7763] and [RFC2046] are
adequate. A distinct media type would allow a consumer to recognise
a context file retrieved without discovery, but at the cost of a
registration whose only distinguishing property is the intent of its
author. The trade-off is noted here so that it can be revisited if
operational experience favours the other side.
9. Implementation Status
This section records the status of known implementation in accordance
with [RFC7942]. It is to be removed before publication as an RFC,
should that occur.
*Publisher-side, arsentev.ai and ru.arsentev.ai.* Two production
hosts serve an index resource at "/llms.txt" and a detail resource at
"/llms-full.txt", generated from the same source as the HTML pages,
since 2026-09-09. Sizes are given in Table 1. Coverage of this
document at the time of writing is partial: the two-tier arrangement
of Section 3.2 is implemented and the index resource is within the
size guidance on both hosts; the detail resource on ru.arsentev.ai is
within the division threshold but approaching it; the discovery
mechanisms of Section 4 are not yet deployed, since this document
defines them. The operator is the author, and this is therefore a
report of one deployment rather than of independent interoperation.
*Consumer-side.* No consumer implementation of the discovery
mechanisms defined here is known. Section 2 reports that eighteen
crawlers observed over the measurement window implemented none of
them, which is expected, since they did not exist.
Arsentev Expires 15 March 2027 [Page 17]
Internet-Draft LLM Context File Discovery September 2026
*Interoperation.* None to report. The mechanism has not been
exercised between independent implementations, and the author invites
reports from any publisher or consumer willing to attempt it.
10. Open Questions
The following are known to be unresolved and are listed to invite
comment rather than to promise resolution.
* *Authenticity.* Nothing binds a context file to the publisher
beyond the origin's own transport security. Whether a signature
mechanism is warranted, and what it would defend against given
Section 7.1, is open.
* *Freshness signalling.* Deployed context files are generated at
deployment time and can silently diverge from the pages they
describe. Whether the format should carry a generation timestamp,
and whether a consumer should act on staleness, is open.
* *Per-consumer variants.* Whether a publisher should be able to
offer different context files to different classes of consumer,
and whether that is useful curation or an invitation to the
cloaking of Section 7.3, is open.
* *Measurement.* The observation of Section 2 is one publisher over
three days. A multi-publisher measurement, ideally one in which
discovery is deployed on some origins and withheld on others,
would test this document's central hypothesis properly. The
author intends to conduct one and will report it in a revision.
11. References
11.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997,
.
[RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
Resource Identifier (URI): Generic Syntax", STD 66,
RFC 3986, January 2005,
.
[RFC6454] Barth, A., "The Web Origin Concept", RFC 6454, December
2011, .
Arsentev Expires 15 March 2027 [Page 18]
Internet-Draft LLM Context File Discovery September 2026
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, May 2017,
.
[RFC8288] Nottingham, M., "Web Linking", RFC 8288, October 2017,
.
[RFC8615] Nottingham, M., "Well-Known Uniform Resource Identifiers
(URIs)", RFC 8615, May 2019,
.
[RFC9110] Fielding, R., Nottingham, M., and J. Reschke, "HTTP
Semantics", STD 97, RFC 9110, June 2022,
.
[RFC9309] Koster, M., Illyes, G., Zeller, H., and L. Sassman,
"Robots Exclusion Protocol", RFC 9309, September 2022,
.
11.2. Informative References
[I-D.arsentev-agent-run-metrics]
Arsentev, E., "Agent Run Metrics: A JSON Interchange
Format for Resource Accounting of Language-Model Agent
Runs", Work in Progress, Internet-Draft, draft-arsentev-
agent-run-metrics-00, September 2026,
.
[I-D.ietf-aipref-attach]
IETF AI Preferences Working Group, "Associating AI Usage
Preferences with Content in HTTP", Work in Progress,
Internet-Draft, draft-ietf-aipref-attach, 2026,
.
[I-D.ietf-aipref-vocab]
IETF AI Preferences Working Group, "A Vocabulary For
Expressing AI Usage Preferences", Work in Progress,
Internet-Draft, draft-ietf-aipref-vocab, 2026,
.
[RFC2046] Freed, N. and N. Borenstein, "Multipurpose Internet Mail
Extensions (MIME) Part Two: Media Types", RFC 2046,
November 1996, .
Arsentev Expires 15 March 2027 [Page 19]
Internet-Draft LLM Context File Discovery September 2026
[RFC7763] Leonard, S., "The text/markdown Media Type", RFC 7763,
March 2016, .
[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running
Code: The Implementation Status Section", BCP 205,
RFC 7942, July 2016,
.
Acknowledgements
The convention that this document attempts to make discoverable was
proposed and popularised by others, and this document takes no credit
for it. The measurement of Section 2 was made on the author's own
hosts, with the limitations that implies and which Section 2.3
states.
Author's Address
Evgenii Arsentev
Independent
Email: hello@arsentev.ai
URI: https://arsentev.ai/
Arsentev Expires 15 March 2027 [Page 20]