Network Delivery Time Control

General Principle A key requirement for real-time video is ensuring each frame arrives on time to be presented. Consequently, NDTC focuses on timely frame delivery rather than attempting to maximize throughput. NDTC aims to ensure each frame reception duration is shorter than a frame period, providing sufficient headroom to ensure high probability of each frame arriving in time. In practice, the control loop of the algorithm stabilizes the frame reception duration by dynamically setting the target frame size to a fraction of the estimated path capacity. NDTC introduces and relies on Frame Dithering Available Capacity Estimation (FDACE), a heuristic that paces packetized frames over a dithered send duration and estimates available path capacity from the relationship between send durations and observed receive durations. Importantly, capacity estimation is performed without injecting additional traffic (beyond the video stream we wish to deliver) or filling the path bottleneck to capacity. We then ensure that the flow uses only a fraction of the measured capacity to prevent risks of self-induced congestion. This process is combined with an AIMD (Additive Increase Multiplicative Decrease) congestion control process for compatibility with both packet loss congestion signals and ECN (Explicit Congestion Notification) for the L4S (Low Latency, Low Loss, and Scalable Throughput) Internet Service . For the rest of this document we describe NDTC based on the following mental model:

The server is encoding video on-the-fly at a frame rate 1/TFRAME (where TFRAME is the frame period), while capping each frame to a video frame size target TARGET.
The server forms a path capacity probe by pacing the frame's constituent packets out over target send duration TSEND.
The path's estimated available capacity AVAILABLE is continuously updated using each frame's actual send and receive durations (SEND and RECV respectively).
TARGET is recalculated whenever AVAILABLE changes to ensure the next RECV is stabilized around target reception duration TRECV.
Additionally, TARGET is capped by a combined AIMD process reacting to congestion signals like packet loss and ECN.

To properly probe the path, TSEND MUST be a fraction of TRECV, and to minimize on-path congestion, TRECV MUST be a fraction of TFRAME. We RECOMMEND that TRECV = 0.6 * TFRAME and TSEND = 0.5 * TRECV. (For example, sending at 30fps would result in TFRAME = 33ms, TRECV = 20ms and TSEND = 10ms.) Note: NDTC focuses on uncovering enough available capacity to meet the application's needs, it is not trying to instantly discover all the available capacity on a path. Specifically, choosing TSEND = 0.5 * TRECV limits FDACE to probing up to 2.0 * AVAILABLE in any single iteration of the FDACE heuristic. NDTC initializes TARGET to INIT_TARGET, and whenever TARGET is recalculated the result is constrained to MIN_TARGET <= TARGET <= MAX_TARGET. Actual values for the INIT_TARGET, MIN_TARGET and MAX_TARGET parameters are application-specific.

Algorithm Overview This section details implementation of the whole NDTC feedback loop using variable and parameter names summarized in section (rather than the more mathematically-oriented names used in NDTC's conference paper ).

Illustrating using RTP/RTCP Although NDTC can be implemented for various real-time video transport protocols, this document illustrates the use of NDTC to support applications using RTP/RTCP . In particular, NDTC operates on a single video stream and it is frame-oriented. References to a "frame" in the rest of this document mean a sequence of consecutive RTP packets having the same SSRC and RTP timestamp and ending with a packet whose RTP marker bit is set. Defining a frame when NDTC is used for non-RTP flows is currently outside the scope of this document. NDTC can be generalized to transports with multiple video streams by running an instance of the algorithm for each video stream. This is currently outside the scope of this document.

Architecture Considerations NDTC can be implemented either sender-side or receiver-side depending on the application's needs. In the following discussion, the "agent" refers to the entity running the NDTC algorithm. A core requirement to execute FDACE is that, for every frame, the agent learns the precise durations over which a given frame was sent (SEND) and received (RECV). Recommendations for transmitting this timing information between senders and receivers are discussed in subsequent sections. If the agent is the sender, the algorithm runs on every frame feedback. If the agent is the receiver, the algorithm runs as soon as a frame is received and sends the updated target frame size (TARGET) to the server.

Sender-side Agent Sender-side NDTC can be instantiated by having receivers pass RECV back to the sender inside existing RTCP-based Feedback for Congestion Control or Transport-wide Congestion Control (TWCC) reports. Alternatively, an implementor might introduce application-specific RTCP feedback to make messages smaller and achieve better precision. The receiver is also required to report which packets have not been received (as described in ) and SHOULD also reflect Explicit Congestion Notification (ECN) markings of received packets (if support for L4S is desired). In this scenario, the agent has local control of the underlying packet pacer and video encoder.

Receiver-side Agent Receiver-side NDTC requires the receiver to regularly pass back updates to TARGET (to properly control the video encoder) and SLOPE (section ) to properly control the packet pacer. This may be achieved with e.g. RTCP messages like Receiver Estimated Maximum Bitrate (REMB) . Note: if the receiver agent cannot pass back SLOPE information, the sender's pacing heuristic SHOULD assume SLOPE = 1.

Timing and Timestamping The precision with which SEND and RECV are measured and reported MUST be 1ms or better, because the durations are typically around a few milliseconds only, and a lower precision would not be sufficient to obtain a consistent estimate, even by averaging multiple samples. We RECOMMEND a precision of 0.1ms. If the agent is implemented sender-side, we assume the agent derives SEND from local knowledge (e.g. timestamps generated low in the networking stack), and learns RECV from reports passed back by the receiver. If the agent is implemented receiver-side, we assume the agent derives RECV from local knowledge (e.g. kernel timestamps from the underlying network interface) and learns SEND via in-band messages from the sender. For instance, the sender can write the effective send time on each packet just before it goes out using the RTP extension.

Frame Dithering Available Capacity Estimation (FDACE) FDACE calculates an estimated available capacity AVAILABLE and a slope SLOPE given send and receive durations for a frame. Frame size LENGTH is calculated by summing up packet payload sizes for the frame according to . Therefore AVAILABLE will actually refer to application-layer capacity expressed in terms of video bitrate. If the frame consists of a single packet, if LENGTH is strictly less than MIN_LENGTH, or if the frame suffered packet loss in the sense of , the agent does not run FDACE. Instead, it retains the TARGET and SLOPE values from the most recent frame for which FDACE ran and immediately jumps to the AIMD congestion control described in . If the frame was packetized as 2 packets or more and suffered no packet loss, the agent first calculates the normalized send duration NSEND and the normalized reception duration NRECV for the frame by respectively dividing frame send duration SEND and frame receive duration RECV by frame size LENGTH. Send duration SEND corresponds to the elapsed time between sending the first packet and sending the last packet of the frame (i.e. the one with the RTP marker). SEND MUST be the actual duration over which the pacer sent the frame, not the calculated target duration from . Packet send times SHOULD be measured as close to socket send calls as possible. Receive duration RECV corresponds to the elapsed time between reception of the first packet and reception of the last packet of the frame (i.e. the one with the RTP marker). Note there is no need for sender and receiver clock synchronization. In practice, packet reception times SHOULD be measured using kernel timestamps for enhanced precision, as the application might take significant time to receive packets in userspace. The agent SHOULD cap RECV to a small multiple of the frame period in order to mitigate outliers caused by transient adverse network events while still allowing the value to grow large enough to capture bursts of cross-traffic and reflect sudden path capacity reductions. The RECOMMENDED maximum value for RECV is 3*TFRAME. A dual-variable EWMA process is updated with the new sample (NSEND, NRECV). The process calculates the averages AVG_NSEND, AVG_NRECV, the variances VAR_NSEND and VAR_NRECV, and the covariance COVAR. The RECOMMENDED weight is LAMBDA = 0.04. A suitable implementation of the dual-variable EWMA process is detailed in . The agent now derives the linear regression parameters from the output of the EWMA process. The slope SLOPE and y-intercept INTERCEPT of the linear equation NRECV = SLOPE * NSEND + INTERCEPT are calculated as follows: 0.0 && COVAR > 0.0) { SLOPE = min(COVAR / VAR_NSEND, 1.0) } else { SLOPE = 0.0 } INTERCEPT = max(AVG_NRECV - SLOPE * AVG_NSEND, 0.0) ]]> SLOPE reflects how pacing frames changes their reception as they go through the path bottleneck and intercept more or less cross-traffic. The relationship is locally linear because we observe normalized times and not rates, and network path components behave linearly in terms of processing time per byte to a good approximation. If the path is not constraining, for instance during ramp-up, receive duration is always equal to send duration, and SLOPE = 1, which can be interpreted as the path capacity looking infinite. If there is no visible cross-traffic and the flow is alone at bottleneck, the slope SLOPE is 0, the send rate does not matter as we always receive at bottleneck rate. If cross-traffic takes most of the bottleneck capacity, SLOPE is close to 1. More generally, we can calculate that in a single FIFO bottleneck scenario with constant-rate cross traffic, SLOPE is the fraction of the bottleneck capacity occupied by cross-traffic. However, this statement does not hold true in all scenarios. FDACE does not require such a simple scenario and performs correctly in more complex network setups, for instance with more advanced AQM (Active Queue Management) disciplines (see ). The y-intercept INTERCEPT reflects the normalized receive duration that we would measure if the frame was sent as an unpaced burst, reflecting total path capacity. The agent extrapolates ESTIMATE, the normalized reception duration that we would measure if the frame was sent at receive rate, reflecting available capacity on the path. It corresponds to the ordinate of intersection with the identity line NRECV = NSEND. In order to be robust when SLOPE is close or equal to 1, ESTIMATE is approximated with successive iterations from AVG_NRECV and converging to the intersection. The RECOMMENDED number of iterations is ITERATIONS = 3. If ITERATIONS is too low, NDTC will overestimate the available capacity in the presence of cross-traffic. If ITERATIONS is higher, NDTC yields more easily against elastic cross-traffic. This means that when the flow is not constrained and SLOPE = 1, ESTIMATE = AVG_NRECV, which is a conservative estimate for the capacity. For increased simplicity and performance in CPU-constrained environments, the loop MAY be unrolled as noted in . The agent SHOULD add a conservative margin MARGIN to ESTIMATE, increasing with the linear regression error. The idea is that when the model fits poorly, the agent should add a higher margin as a safety buffer. The RECOMMENDED formula to calculate MARGIN is to multiply a constant KMARGIN by the standard deviation of NRECV and 1-R2 where R2 is the coefficient of determination representing the proportion of variation in NRECV explained by the linear regression. The RECOMMENDED value for KMARGIN is 0.25. A value too close to 0 makes the algorithm react slower to abrupt changes. A value too high for KMARGIN makes the algorithm significantly underestimate the capacity in the presence of bursty cross traffic. 0.0 && VAR_NRECV > 0.0) { R2 = COVAR^2 / (VAR_NSEND * VAR_NRECV) MARGIN = KMARGIN * sqrt(VAR_NRECV) * (1-R2) } else { MARGIN = 0.0 } ]]> Available capacity AVAILABLE is calculated by taking the reciprocal of the sum ESTIMATE + MARGIN:

Target Frame Size from Available Capacity Before FDACE and the combined congestion control run for the first time, TARGET is initialized to INIT_TARGET and SLOPE is initialized to 1.0. Each time FDACE runs and updates the available capacity, the agent now derives the new target frame size TARGET from TRECV and AVAILABLE and caps it to MAX_TARGET. The agent MUST cap TARGET to MAX_TARGET even if the encoder has its own internal maximum frame size to prevent a runaway feedback loop in case the path capacity is very large. Without capping, NDTC would continue increasing the target frame size until it matches the available path capacity even if the encoded video can't follow it, pacing over shorter and shorter durations in the process. Frames would be sent as very short bursts, increasing the risk of packet loss. MAX_TARGET SHOULD be a sensible frame size that the video content can realistically reach.

Combined AIMD Congestion Control Even if FDACE can successfully run on its own most of the time, it doesn't react to conventional indications of network congestion, in particular packet loss, whereas reacting to packet loss is essential in the presence of a policer on the path. Therefore, NDTC combines an AIMD (Additive Increase Multiplicative Decrease) process with FDACE to also adjust the frame size and send duration in response to conventional congestion feedback. It allows the algorithm to seamlessly transition between capacity estimation and congestion signals, following the path evolution. The AIMD process computes a congestion target frame size CTARGET and a pseudo-slope CSLOPE to respectively cap TARGET and SLOPE, in order to limit the effective pacer rate. See for details about how the pacer adapts the sending duration according to the slope.

When CTARGET < TARGET, the flow is congestion-limited, therefore TARGET is set to CTARGET and CSLOPE is set to 0.0 so the pacer sends at rate CTARGET / TRECV.
When CTARGET >= TARGET, the flow is not congestion limited, therefore TARGET is unchanged and CSLOPE is set so that the pacer send duration results in a send rate around CTARGET / TRECV.
When CTARGET reaches TARGET * TRECV / TSEND, CSLOPE reaches 1.0 and the pacer reaches its minimum send duration and maximum send rate around TARGET/TRECV.

Therefore, the maximum congestion frame size for the AIMD logic is defined to correspond to the maximum send rate: For each transmitted frame, the agent identifies whether packets of the frame were lost as described in , and runs the AIMD logic to update a congestion frame size CSIZE. CSIZE SHOULD be initialized to MAX_TARGET as in most cases FDACE should be self-sufficient. If one or more packets were lost, the agent performs a decrease by capping CSIZE to CMAX and multiplying it by BETA. If no packets were lost and CSIZE is less than CMAX, the agent increases CSIZE by adding ALPHA, then caps the result to CMAX. The RECOMMENDED values are ALPHA=40 bytes and BETA=0.7 to make increase and decrease relatively conservative and minimize QoE reduction. The AIMD logic SHOULD NOT feature a slow start phase and both decrease and increase SHOULD be suppressed for one round-trip after a loss event. In order to support L4S , the agent SHOULD also react similarly to Prague congestion control , which requires supporting ECN , and if the agent is the sender, it also requires compatible feedback like . If the last ECN decrease is more recent than the last packet loss event, the increase SHOULD be performed by adding a larger EALPHA instead of ALPHA for the flow to compete more fairly with other flows. The RECOMMENDED value is EALPHA=400 bytes. The RECOMMENDED algorithm is described in . If the agent is the sender, it SHOULD also perform the decrease in case it does not get any feedback for a significant duration after sending a frame. In any case, the sender SHOULD implement a circuit breaker to stop sending after it stops receiving feedback for a significant duration, following the recommendations in . The congestion frame size CTARGET is calculated as the minimum of CSIZE and CMAX: Then, CSLOPE is calculated such that the average pacing rate is CTARGET/TRECV. This is obtained by solving for CSLOPE in the pacer equation in such that when LENGTH = TARGET, the send duration results in an effective send rate of CTARGET/TRECV:

Encoder Target Frame Size The agent caps the target frame size TARGET and the slope SLOPE with CTARGET and CSLOPE respectively to take into account the most conservative estimate: The agent <