Internet-Draft Agent Run Metrics November 2026
Arsentev Expires 5 May 2027 [Page]
Workgroup:
Network Working Group
Internet-Draft:
draft-arsentev-agent-run-metrics-02
Published:
Intended Status:
Standards Track
Expires:
Author:
E. Arsentev
Independent

Agent Run Metrics: A JSON Interchange Format for Resource Accounting of Language-Model Agent Runs

Abstract

Autonomous software agents driven by large language models execute multi-step runs in which the same conversational context is retransmitted to a model on every step. The resulting resource consumption is dominated by repeated input rather than by generated output, and it is reported today in mutually incompatible, vendor-specific shapes. This document defines Agent Run Metrics, a JSON interchange format that describes the resource consumption of a single agent run and of its constituent steps, together with a small set of derived quantities whose computation is specified exactly. It states normatively that one reported step corresponds to one completed model invocation, so that the several records a runtime may write while a single response is produced are not mistaken for several invocations, and it defines how the consumption of delegated sub-runs is attributed without being counted twice. The format carries counters, timing and cost attribution only; it deliberately excludes prompt and completion content. This document defines the initial values of two extensible enumerations and the rule by which they are extended; it requests no IANA action.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 5 May 2027.

▲

Table of Contents

1. Introduction

A growing class of software systems operates as an "agent": a program that pursues a goal over a sequence of steps, at each step submitting an accumulated context to a large language model, receiving a response, and optionally invoking external tools whose results are appended to that context. The unit of work in such a system is not a single request but a run: a bounded sequence of steps sharing one accumulating context.

Runs have a cost structure that differs qualitatively from that of the request/response interactions the surrounding infrastructure was designed to meter. Because the context accumulates, the input submitted at step n generally contains the input submitted at step n-1. Over a run of many steps, total input volume therefore grows super-linearly in the number of steps even when each individual step adds little. Content admitted into the context early is re-submitted on every subsequent step, so the position at which content enters a run affects total consumption as much as its size does. Practitioners consequently need per-run and per-step accounting, not merely per-request accounting, and they need it in a form that can be compared across implementations.

Such accounting exists today, but only in vendor-specific shapes. Different implementations disagree on questions that are not matters of taste: whether tokens served from a provider-side cache are included in the reported input count or reported beside it; whether internal reasoning tokens that are never returned to the caller are part of the output count; whether the several records a runtime writes while one response is produced describe one invocation or several; whether the tokens of a delegated sub-run belong to that sub-run, to the run that started it, or to both; whether a retried step is one step or two; whether the totals for a run are the sum of its steps. Two reports that use the same field names can therefore describe incompatible quantities, and a consumer cannot detect the difference from the data alone.

These are not questions of presentation. An accounting rule chosen wrongly changes a reported figure by a factor rather than by a percentage: a run whose journal records the progress of a response, rather than only its completion, will be reported with several times its true input if those records are summed, and a delegation tree whose parent and children both claim the children's tokens will be reported with roughly twice its true consumption. Errors of this class are silent, because the resulting report is internally consistent and no field states which rule was applied. Section 12 records where such errors were observed.

This document defines a JSON format, Agent Run Metrics, in which those questions are answered normatively. The format is small, content-free, and intended to be produced by an agent runtime and consumed by cost-management, capacity-planning, research and audit tooling.

1.1. Scope and Non-Goals

This document specifies:

  • a data model for a run, its steps, and their resource usage (Section 3);
  • a JSON serialization of that data model (Section 5);
  • exact definitions for a small set of derived quantities (Section 4);
  • an incremental reporting profile for runs observed while in progress (Section 6);
  • an OPTIONAL binding to HTTP for delivering reports to a collector (Section 7);
  • the initial values of two extensible enumerations and the rule for extending them (Section 10).

This document does not specify:

  • any protocol between an agent and a language model, or between an agent and its tools;
  • any pricing, billing or settlement mechanism; cost fields in this format are attributions supplied by the reporter, not invoices;
  • the semantic content of a run: prompts, completions, tool arguments and tool results are out of scope and, by Section 8.1, MUST NOT appear in a report;
  • a distributed tracing system. Where a tracing system is in use, Section 5.2 defines how to correlate with it.

1.2. Relationship to Existing Work

Distributed tracing conventions such as [TRACECONTEXT] and the semantic conventions maintained by the OpenTelemetry project [OTEL-GENAI] describe how to represent spans of execution, including spans that invoke a language model. Those conventions address the shape of a trace and the propagation of trace identity; they do not define an interchange document for a completed run, and they do not normatively resolve the counting questions listed in Section 1. The format defined here is complementary: it is a self-contained report about one run, it can be produced without a tracing pipeline, and Section 5.2 specifies how a report references trace identifiers when a tracing pipeline is present.

This format is not a metrics protocol in the sense of a time-series exposition format. A report describes one identified run; aggregation across runs is a function of the consumer.

2. Conventions and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Run:
A bounded sequence of Steps executed by one Agent in pursuit of one goal, sharing a single accumulating Context. A Run is the unit described by a Report.
Step:
One indivisible unit of work within a Run: a single Model Invocation, a single Tool Call, or another unit whose kind is defined in Section 10.2 or by an extension.
Model Invocation:
A single request to a language model and the corresponding response, taken as one unit from the submission of the request to the completion of the response. A response delivered incrementally, or composed of several content blocks, is one Model Invocation and not several; see Section 3.3.
Context:
The sequence of tokens submitted as input to a Model Invocation.
Token:
The unit in which a model counts input and output. Token counts are model-specific; a Report does not assert that counts from different models are comparable.
Delegated Run:
A Run started by an Agent in the course of another Run, in pursuit of a subordinate goal and with its own Context. Such Runs are also called sub-agent Runs.
Delegation Tree:
A Run together with every Run reachable from it by delegation.
Cached Prefix:
A portion of a Context that a model provider serves from provider-side state established by an earlier Model Invocation, rather than processing afresh.
Cache Lifetime:
The period for which a provider undertakes to retain the state underlying a Cached Prefix.
Reporter:
The party that produces a Report. Typically the agent runtime.
Collector:
The party that receives and processes a Report.
Report:
One JSON document conforming to this specification.

3. Data Model

A Report is a Run object. A Run object contains a Totals object, zero or more Step objects, and OPTIONAL descriptive and cost members. A Step object contains a Usage object when the Step consumed model resources.

  Run
   +- run_id, start, end, status, termination_reason
   +- agent { name, version }
   +- models [ { id, provider, context_window } ]
   +- totals   -> Usage
   +- cost     -> Cost
   +- derived  -> Derived
   +- steps [ Step ]
                +- index, kind, start, end
                +- usage -> Usage        (model_invocation)
                +- tool  -> Tool         (tool_call)
                +- error -> Error
Figure 1: Structure of a Report

3.1. The Run Object

A Run object has the members defined in Table 1.

Table 1: Members of the Run object
Member Type Presence Description
version string REQUIRED Format version; "1" for this specification.
run_id string REQUIRED Identifier of the Run, unique within the Reporter's scope.
parent_run_id string OPTIONAL run_id of the Run that delegated this Run; see Section 3.6.
root_run_id string OPTIONAL run_id of the Run at the root of the Delegation Tree; see Section 3.6.
start timestamp REQUIRED Instant at which the Run began.
end timestamp see text Instant at which the Run reached a terminal status.
status string REQUIRED One of "running", "completed", "failed", "aborted".
termination_reason string OPTIONAL A Termination Reason; see Section 10.1.
agent object OPTIONAL Descriptor of the producing agent; see Section 3.10.
models array OPTIONAL Descriptors of the models used; see Section 3.11.
step_count integer REQUIRED Number of Steps in the Run, including Steps omitted from "steps".
steps array OPTIONAL Step objects; see Section 3.2.
totals object REQUIRED Usage object aggregating this Run alone.
subtree_totals object OPTIONAL Usage object aggregating this Run and its Delegated Runs; see Section 3.6.
cost object OPTIONAL Cost object for the whole Run; see Section 3.7.
derived object OPTIONAL Derived quantities; see Section 4.
trace_id string OPTIONAL Correlation identifier; see Section 5.2.
labels object OPTIONAL Reporter-defined string-to-string labels; see Section 3.12.
evidence_refs array OPTIONAL URIs of records outside the Report that bear on its values; see Section 9.1.

The "end" member MUST be present when "status" is "completed", "failed" or "aborted", and MUST NOT be present when "status" is "running". When present, "end" MUST NOT be earlier than "start".

The "status" member is a closed enumeration; a Reporter MUST NOT emit a value other than the four listed, and a Collector that encounters an unrecognised value MUST treat the Report as malformed. Distinctions finer than these four values are expressed through "termination_reason", which is extensible.

The "step_count" member MUST equal the total number of Steps executed in the Run. When the "steps" array is present, "step_count" MUST be greater than or equal to the length of that array; a Reporter MAY omit individual Steps from the array, for example to bound Report size, while still accounting for them in "step_count" and in "totals".

3.2. The Step Object

Table 2: Members of the Step object
Member Type Presence Description
index integer REQUIRED Zero-based position of the Step within the Run.
kind string REQUIRED A Step kind; see Section 10.2.
start timestamp REQUIRED Instant at which the Step began.
end timestamp OPTIONAL Instant at which the Step ended.
model string see text Identifier of the model invoked.
invocation_id string see text Identifier of the Model Invocation; see Section 3.3.
usage object see text Usage object for this Step.
cost object OPTIONAL Cost object for this Step.
tool object see text Tool object; see Section 3.8.
error object OPTIONAL Error object; see Section 3.9.
child_run_id string see text run_id of the Run executed by a Step of kind "delegation"; see Section 3.6.
retry_of integer OPTIONAL index of the Step this Step retries.
span_id string OPTIONAL Correlation identifier; see Section 5.2.

Within one Run, "index" values MUST be unique and MUST be assigned in the order in which Steps began, with no requirement that the values present in a Report be contiguous. Steps that overlap in time are permitted; "index" then reflects start order only.

When "kind" is "model_invocation", the "usage" and "model" members MUST be present. When "kind" is "tool_call", the "tool" member MUST be present and the "usage" member MUST NOT be present; resources consumed by placing the tool result into the Context are attributed to the subsequent Model Invocation, where they are actually submitted.

A retried attempt MUST be reported as a distinct Step with its own "index" and its own "usage", and SHOULD carry "retry_of" identifying the first attempt. Reporting a set of attempts as one Step with summed counters loses the distinction between work that advanced the Run and work that did not, and MUST NOT be done.

3.3. One Step per Completed Model Invocation

A Step of kind "model_invocation" MUST describe exactly one completed Model Invocation, and a Report MUST NOT contain more than one Step describing the same Model Invocation.

This requirement is stated separately, and at length, because the journals from which Reporters are commonly built do not satisfy it. A model response is typically composed of several content blocks: internal reasoning, text addressed to the caller, and requests to invoke tools. A runtime frequently appends one journal record per block, or one record per streaming update, and each such record carries a usage structure of its own. Those records are not independent measurements of independent work:

  • the input counters ("input_tokens", "cache_read_tokens", "cache_write_tokens") are identical in every record of the group, because the input was submitted once;
  • the output counter is a cumulative snapshot of one response as it is produced, so the records of a group form a non-decreasing sequence that ends at the final count.

Two naive treatments of such a journal therefore fail in opposite directions. Summing the records multiplies the input of the invocation by the number of records in its group and counts most of its output more than once. Retaining only the first record of each group understates the output by the ratio of the earliest snapshot to the final count, which may be an order of magnitude. Both errors are silent: the Report that results satisfies every other invariant in this document, and a Collector cannot detect either from the data. In the corpus described in Section 12, the first of these errors overstated total consumption by a factor of about 1.90 and the second halved the reported output.

Accordingly:

  1. A partial or intermediate usage structure, observed while a response is still being produced, is not a metric. A Reporter MUST NOT emit a Step for it and MUST NOT include its counters in "totals".
  2. The "output_tokens" of a Step of kind "model_invocation" MUST be the count for the whole completed response. It MUST NOT be a snapshot taken before the response ended, and MUST NOT be the sum of the snapshots observed while it was produced.
  3. A response composed of several content blocks MUST be reported as one Step whose counters cover all of its blocks. Reasoning blocks are accounted for within that Step by "reasoning_tokens", and requests to invoke tools are part of its "output_tokens"; the Tool Calls to which those requests give rise are separate Steps of kind "tool_call".
  4. A Reporter deriving Reports from a source that emits several records per Model Invocation MUST group those records by an identifier of the invocation and emit one Step per group. It MUST NOT group by timestamp, by model identifier or by adjacency in the source, because concurrent invocations interleave and such grouping silently merges or splits them.

A Step of kind "model_invocation" SHOULD carry "invocation_id", a string identifying the Model Invocation and unique within the Run. Where the source of a Report distinguishes invocations by such an identifier, the Reporter MUST use it both as the grouping key required above and as the value of this member, so that a Collector can verify the grouping instead of trusting it. A Reporter MUST NOT emit two Steps of one Run with the same "invocation_id". A Collector that receives a Report containing two such Steps MUST treat the Report as malformed rather than silently discard one of them, since it cannot know which carries the final counters.

An invocation that never completed, because it failed, was cancelled or was abandoned, MAY be reported as a Step carrying an Error object together with whatever counters the provider metered for it. Such a Step is a completed unit of accounting even though it is not a completed response. The prohibition above concerns snapshots of an invocation that is still producing output, not invocations that ended early.

3.4. The Usage Object

The Usage object is the core of this format, and the definitions in this section are the ones on which interoperability depends.

Table 3: Members of the Usage object
Member Type Presence Description
input_tokens integer REQUIRED Total tokens submitted as input.
output_tokens integer REQUIRED Total tokens produced by the model.
cache_read_tokens integer OPTIONAL Input tokens served from a Cached Prefix.
cache_write_tokens integer OPTIONAL Input tokens stored into provider-side cache state.
cache_writes array OPTIONAL Breakdown of "cache_write_tokens" by Cache Lifetime; see Section 3.5.
reasoning_tokens integer OPTIONAL Output tokens not returned to the caller.
billable_input_tokens integer OPTIONAL Input tokens the provider metered.
billable_output_tokens integer OPTIONAL Output tokens the provider metered.

All members of a Usage object MUST be non-negative integers.

The following invariants are normative. A Reporter MUST NOT emit a Usage object that violates them, and a Collector MUST treat a Report containing such an object as malformed:

  1. "input_tokens" is the count of all tokens submitted as input, whether or not they were served from a Cached Prefix. "cache_read_tokens" is a subset of "input_tokens" and therefore MUST be less than or equal to it. A Reporter MUST NOT report cached tokens beside "input_tokens" as a disjoint quantity.
  2. "output_tokens" is the count of all tokens produced, including tokens consumed internally by the model and never returned to the caller. "reasoning_tokens" is a subset of "output_tokens" and therefore MUST be less than or equal to it.
  3. "cache_write_tokens" counts tokens that were submitted as input and, in the course of the same Model Invocation, stored into provider-side cache state for reuse by later invocations; it is likewise a subset of "input_tokens". A token that was served from a Cached Prefix was not processed and stored afresh in the same invocation, so "cache_read_tokens" and "cache_write_tokens" denote disjoint subsets of "input_tokens" and their sum MUST be less than or equal to "input_tokens". A Reporter MUST NOT report tokens stored into cache state beside "input_tokens" as a disjoint quantity, and MUST NOT report them a second time under "cache_read_tokens" merely because a later invocation read them.
  4. The "billable_" members carry the counts a provider used for metering, which may differ from the physical counts because of discounts applied to cached tokens, rounding, or minimum charges. They are unconstrained relative to the physical counts. A Reporter that has no metering information MUST omit them rather than copy the physical counts, so that a Collector can distinguish "metered and equal" from "not known".

The distinction between the three ways in which an input token can be handled, namely processed afresh, served from a Cached Prefix, or processed and stored into cache state, is not a refinement of interest only to implementers. Providers price the three differently, commonly charging a premium for tokens stored and a discount for tokens served from store, so two Runs with identical "input_tokens" can differ in cost by a large factor. In the corpus described in Section 12, tokens served from a Cached Prefix accounted for about 56 percent of attributed cost and tokens stored into cache state for about 27 percent, leaving less than a fifth for input processed afresh and for all output together; storing into cache was thus the second largest component of cost rather than a marginal one. A Reporter that omits "cache_read_tokens" and "cache_write_tokens" leaves the greater part of the cost of a Run unattributable, and SHOULD emit both whenever the provider reports them.

When a Run's "steps" array contains every Step of the Run, the "totals" object MUST be, member by member, the sum of the corresponding members of every Step's Usage object, with the exception of "cache_writes", whose aggregation is defined in Section 3.5, and a member MUST be omitted from "totals" if it is absent from every Step. When the "steps" array is incomplete, "totals" MUST still account for the whole Run, including omitted Steps.

3.5. Cache Lifetime

Provider-side cache state is retained for a bounded period, and a provider may offer several such periods at different prices. Two Model Invocations that stored the same number of tokens may therefore have been charged differently, and a single "cache_write_tokens" figure cannot distinguish them. A Usage object MAY for that reason carry "cache_writes", an array of objects each having the REQUIRED members "lifetime" and "tokens".

The "lifetime" member MUST be a string conforming to the "duration" rule of Appendix A of [RFC3339], giving the nominal retention period that the Reporter requested or that the provider applied, for example "PT5M" or "PT1H". The value states a nominal period and not an observed one; whether the state in fact survived that long is not reported. The "tokens" member MUST be a non-negative integer. One "lifetime" value MUST NOT appear in more than one element of the same array.

The sum of the "tokens" members of "cache_writes" MUST equal the "cache_write_tokens" member of the same Usage object. This does not always hold in the sources from which Reporters are built: in the corpus described in Section 12, a small number of records carried per-lifetime counters whose sum differed from the aggregate counter in the same record. A Reporter whose source does not reconcile MUST omit "cache_writes" for the affected Step. It MUST NOT adjust either figure so that they agree, and MUST NOT substitute the sum of the per-lifetime counters for "cache_write_tokens", because a discrepancy is evidence that at least one of the two figures is wrong, and either repair would assert an agreement that was never observed.

A Collector MUST treat as malformed a Report in which "cache_writes" is present and does not sum to "cache_write_tokens". Where "cache_writes" is absent, a Collector MUST NOT assume any particular lifetime.

Where the Usage objects of Steps carry "cache_writes", a "totals" or "subtree_totals" object that carries it MUST contain one element for each distinct "lifetime" that appears, whose "tokens" is the sum of the "tokens" reported for that lifetime; this is the aggregation rule referred to in Section 3.4.

There is deliberately no corresponding breakdown of "cache_read_tokens". A read is served from whatever state exists at the moment of the read, and attributing it to the lifetime with which that state was created would require knowledge the reader does not have.

3.6. Delegated Runs and Aggregation

An Agent commonly pursues part of its goal by starting a subordinate Run with its own Context, and that Run may itself delegate further. A Delegated Run is a Run in the sense of this document: it has its own "run_id", its own Steps and its own "totals", and it is described by its own Report. In the corpus described in Section 12, slightly more than half of all Model Invocations occurred in Delegated Runs rather than in the Runs that started them, so a treatment that ignored delegation would describe a minority of the work.

The relationship is expressed in three places, and a Reporter that observes both sides SHOULD populate all three:

  • the child Report's "parent_run_id", naming the Run that delegated it;
  • the child Report's "root_run_id", naming the Run at the root of the Delegation Tree. A Run that emits "root_run_id" and has no parent MUST set it equal to its own "run_id"; a Run that emits it and has a parent MUST set it to the "root_run_id" of that parent. The member exists so that a Collector can select a whole Tree without walking it edge by edge;
  • in the parent, a Step of kind "delegation" spanning the interval during which the child ran, carrying "child_run_id" where the parent knows the child's identifier.

These three members are accounting references.