IOAM network awareness for Low Latency, Low Loss, and Scalable Throughput (L4S)

Abstract

This document specifies a framework that uses operational and telemetry information collected by In Situ Operations, Administration, and Maintenance (IOAM) to support the monitoring and parameter adjustment of an L4S Dual-Queue Coupled AQM within a single limited domain. IOAM Direct Export reports path and node information, such as hop-by-hop delay and queue depth, to an IOAM Control Center (IOAM-C). The IOAM-C correlates this information and may adjust operator-configurable AQM target or threshold parameters within configured bounds. This document does not modify the L4S ECN protocol, the IOAM data formats, or the packet forwarding behavior defined by the referenced specifications.¶

1. Introduction

The L4S architecture [RFC9330] uses Explicit Congestion Notification (ECN) to provide early feedback about incipient congestion. The L4S ECN protocol is defined in [RFC9331], and the Dual-Queue Coupled AQM framework is defined in [RFC9332]. ECN marking provides end-to-end congestion indication, but it does not by itself identify the node that marked a packet or expose per-hop queue state. In Situ Operations, Administration, and Maintenance (IOAM) records and collects operational and telemetry information while a packet traverses a path between two points in a network. IOAM Direct Export is defined in [RFC9326].¶

The goal of this framework is to make IOAM path and node information available to an L4S deployment within the same limited domain. An IOAM Control Center (IOAM-C) stores, correlates, and analyzes the exported information and provides bounded parameter updates to operator-managed Dual-Queue Coupled AQM instances.¶

This specification defines how data collected by IOAM can provide path visibility and bounded parameter input to a Low Latency, Low Loss, and Scalable Throughput (L4S) deployment. The terms "encapsulation" and "decapsulation" are used in this document in the same way as [RFC9197]. An IOAM encapsulating node incorporates one or more IOAM Option-Types into packets that IOAM is enabled for.¶

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

L4S: Low Latency, Low Loss, and Scalable Throughput (L4S) as defined in [RFC9330].¶

L4S Dual-Queue Coupled AQM: A framework for coupling the Active Queue Management (AQM) algorithms in two queues intended for flows with different responses to congestion.¶

IOAM: In situ Operations, Administration, and Maintenance as defined in [RFC9197].¶

OAM: Operations, Administration, and Maintenance.¶

IOAM Transit Node (IOAM-T): An IOAM transit node updates one or more of the IOAM-Data-Fields. If both the Pre-allocated and the Incremental Trace Option-Types are present in the packet, each IOAM transit node will update at most one of these Option-Types.¶

IOAM Encapsulating Node (IOAM-E): An IOAM encapsulating node incorporates one or more IOAM Option-Types into packets that IOAM is enabled for. If IOAM is enabled for a selected subset of the traffic, the IOAM encapsulating node is responsible for applying the IOAM functionality to the selected subset.¶

IOAM Decapsulating Node (IOAM-DE): An IOAM decapsulating node removes any IOAM Option-Types from packets and processes and/or exports the associated data.¶

IOAM Node Identifier: The combination of IOAM node_id and IOAM Namespace-ID identifies an IOAM node within the operational context of an IOAM domain.¶

Direct Export: Direct Export is an IOAM mode of operation within which IOAM data are to be directly exported to a collector rather than be collected within the data packets. The IOAM Direct Export Option-Type consists of a fixed-size "IOAM direct export option header". Direct Export for IOAM is defined in [RFC9326].¶

3. IOAM network awareness in L4S framework

The following are the system components of the L4S-IOAM framework.¶

                                 +--------------+       +-----------+
                                +-------------+ |       |    L4S    |
                                | Monitoring/ | |======>|  DualQ AQM |
                                | Analytics   | |       |           |
                                | IOAM-C      |-+       +-----------+
                                +-------------+
                                       ^
                                       |  Processed/interpreted/
                                       |  aggregated IOAM data
                                       |
                                 +--------------+
                                +-------------+ |
                                | IOAM data   | |
                                | processing  | |
                                | system(s)   |-+
                                +-------------+
                                       ^
                                       |  Raw export of
                                       |  IOAM data
                                       |
                +--------------+-------+------+--------------+
                |              |              |              |
                |              |              |              |
   User     +---+----+     +---+----+     +---+----+     +---+-----+
   packets  | IOAM-E |     | IOAM-T |     | IOAM-T |     | IOAM-DE |
   -------->| Node   |====>| Node   |====>| Node   |====>| Node    |-->
            |        |     | A      |     | B      |     |         |
            +--------+     +--------+     +--------+     +---------+

IOAM Control Center (IOAM-C): The IOAM-C stores and manages exported network information and makes decisions based on an analysis of this information. Its inputs include hop-by-hop delay, queue depth, and node and interface identifiers. Its outputs are limited to operator-configurable target and threshold parameters of the Coupled AQM at IOAM-capable nodes, and each output MUST remain within operator-configured bounds. The IOAM-C MUST NOT change L4S ECN codepoint processing. If telemetry is unavailable, stale, or inconsistent, the IOAM-C MUST NOT issue a parameter update based on that telemetry.¶

IOAM Encapsulating Node (IOAM-E): The IOAM-E selects the traffic to which IOAM is applied and incorporates the selected IOAM Option-Types into packets, as specified in [RFC9197] and [RFC9326].¶

IOAM-E, IOAM-T, and IOAM-DE nodes that support Direct Export send selected IOAM data to a data processing system or to the IOAM-C using an operator-selected export mechanism. Before the information is used by the IOAM-C, records from individual nodes are correlated using available Namespace-ID, Flow ID, and Sequence Number information.¶

Figure 2 summarizes the L4S forwarding and queueing functions monitored by this framework.¶

                (3)                  (2)                    (4)
              .-------^------..------------^------------------.
                                                           _________
 ,-(1)-----.                                 _____        | IOAM    |
; ________  :            L4S    -------.    |     | /_ _ _| Monitor |
:|Scalable| :               _ __\     ||__\_|mark | \   | |_________|
:| sender | :  __________  /    /     ||  / |_____| \   |  _________
:|________|\; |          |/     -------'        ^    \  | |condit'nl|
 `---------'\_|  IP-ECN  |          Coupling    :     \_|_|priority |_\
  ________  / |Classifier|                      :     / | |scheduler| /
 |Classic |/  |__________|\       -------.    __:__  /  | |_________|
 | sender |                \_ __\ || | ||__\_|mark/|/   |
 |________|                     / || | ||  / |drop |/_ _|
                     Classic      -------'   |_____|\

(1) Scalable sending host
(2) Isolation in separate network queues
(3) Packet identification protocol
(4) Monitor in network queues

4. Information Details

IOAM for L4S is used to enhance diagnostics of L4S networks. It complements the Explicit Congestion Notification (ECN) protocol for L4S described in [RFC9331].¶

Figure 3 shows examples of information used to provide path and node awareness for an L4S deployment.¶

+-------------+----------------------+
| Awareness   | Network              |
| information | information          |
+-------------+----------------------+
|             | IOAM node location;  |
|             | IOAM node type;      |
| Capability  | IOAM node ID;        |
| parameters  | Topology information.|
|             |                      |
|             |                      |
|             |                      |
|             |                      |
|             |                      |
+-------------+----------------------+
|             | Service request      |
| Status      | information;         |
| parameters  | Traffic features;    |
|             | Communication        |
|             | information.         |
+-------------+----------------------+

"IOAM tracing data" is expected to be collected at every IOAM transit node that a packet traverses to ensure visibility into the entire path that a packet takes within an IOAM-Domain. In other words, in a typical deployment, all nodes in an IOAM-Domain would participate in IOAM and, thus, be IOAM transit nodes, IOAM encapsulating nodes, or IOAM decapsulating nodes. If not all nodes within a domain are IOAM capable, IOAM tracing information (i.e., node data, see below) will only be collected on those nodes that are IOAM capable.¶

IOAM tracing can, for example, collect the following types of information:¶

• Identification of the IOAM node. An IOAM node identifier can match to a device identifier or a particular control point or subsystem within a device.¶
• Identification of the interface that a packet was received on, i.e., ingress interface.¶
• Identification of the interface that a packet was sent out on, i.e., egress interface.¶
• Time of day when the packet was processed by the node as well as the transit delay. Different definitions of processing time are feasible and expected, though it is important that all devices of an IOAM-Domain follow the same definition.¶
• Generic data, which is format-free information, where the syntax and semantics of the information are defined by the operator in a specific deployment. For a specific IOAM-Namespace, all IOAM nodes should interpret the generic data the same way. Examples for generic IOAM data include geolocation information (location of the node at the time the packet was processed), buffer queue fill level or cache fill level at the time the packet was processed, or even a battery charge level.¶
• Information to detect whether IOAM trace data was added at every hop or whether certain hops in the domain weren't IOAM transit nodes.¶
• Data that relates to how the packet traversed a node (transit delay, buffer occupancy in case the packet was buffered, and queue depth in case the packet was queued).¶

         Export of      Export of      Export of     Export of
         IOAM data      IOAM data      IOAM data     IOAM data
             ^              ^              ^              ^
             |              |              |              |
             |              |              |              |
User     +---+----+     +---+----+     +---+----+     +---+-----+
packets  | IOAM-E |     | IOAM-T |     | IOAM-T |     | IOAM-DE |
-------->| Node   |====>| Node   |====>| Node   |====>| Node    |-->
         |        |     | A      |     | B      |     |         |
         +--------+     +--------+     +--------+     +---------+

This framework uses Direct Export mode for L4S-IOAM information gathering. OAM information about each IOAM node a packet traverses is collected and immediately exported to a collector. Direct Export could be used in situations where per-hop tracing information is desired, but gathering the information within the packet -- as with per-hop tracing -- is not feasible. Rather than automatically correlating the per-hop tracing information, as done with per-hop tracing, Direct Export requires a collector to correlate the information from the individual nodes. In addition, all nodes enabled for Direct Export need to be capable of exporting the IOAM information to the collector.¶

As required by [RFC9326], a node that performs Direct Export MUST support export rate limiting. When records from multiple nodes need to be correlated, the Flow ID and Sequence Number fields defined by [RFC9326] SHOULD be used.¶

This information can help the IOAM-C identify the location and severity of congestion. This document does not define new per-flow balancing, scheduling, or ECN codepoint behavior.¶

5. Use Cases

This section identifies representative environments in which IOAM path visibility can complement an L4S deployment. In each case, IOAM is used within a limited domain to locate and characterize congestion; the L4S service continues to operate according to [RFC9330], [RFC9331], and [RFC9332].¶

Where a use case spans multiple administrations, each IOAM deployment and each AQM parameter update remain confined to the limited domain controlled by one administration.¶

The following use cases for L4S are being considered by various interested parties:¶

• where the bottleneck is one of various types of access network, e.g., DSL, Passive Optical Networks (PONs), DOCSIS cable, mobile, satellite; or where it's a Wi-Fi link¶

• private networks of heterogeneous data centres, where there is no single administrator that can arrange for all the simultaneous changes to senders, receivers, and networks needed to deploy DCTCP:¶

  • a set of private data centres interconnected over a wide area with separate administrations but within the same company¶
  • a set of data centres operated by separate companies interconnected by a community of interest network (e.g., for the finance sector)¶
  • multi-tenant (cloud) data centres where tenants choose their operating system stack (Infrastructure as a Service (IaaS))¶

• different types of transport (or application) congestion control:¶

  • elastic (TCP/SCTP);¶
  • real-time (RTP, RMCAT); and¶
  • query-response (DNS/LDAP).¶

• where low delay QoS is required but without inspecting or intervening above the IP layer :¶

  • Mobile and other networks have tended to inspect higher layers in order to guess application QoS requirements. However, with growing demand for support of privacy and encryption, L4S offers an alternative. There is no need to select which traffic to favour for queuing when L4S can give favourable queuing to all traffic.¶
• If queuing delay is minimized, applications with a fixed delay budget can communicate over longer distances or via more circuitous paths, e.g., longer chains of service functions [RFC7665] or of onion routers.¶
• If delay jitter is minimized, it is possible to reduce the dejitter buffers on the receiving end of video streaming, which should improve the interactive experience.¶

9. References

Acknowledgments

The authors wish to thank Xue Zhang, Ziheng Xu and Xuetong Hu, for their invaluable comments.¶