Fully Adaptive Routing Ethernet in Multi-Plane Scale-Out Networks

Large AI training clusters (beyond 100,000 GPUs) increasingly use multi&nbhy;plane scale&nbhy;out network topologies (see below) to reduce the total number of switches and links. In such a topology, a high&nbhy;speed RNIC is split into multiple lower&nbhy;speed lanes, each connected to an independent CLOS fabric (a “plane”). Because there are no links between planes, the RNIC itself must decide which plane to use for each packet or flow. In other words, the RNIC must know the reachability of each plane and then perform global load balancing across planes.

(For simplicity, the diagram above omits the connections between RNICs and leaf switches. In practice, each RNIC is multi&nbhy;homed to one leaf switch in every plane.) FARE&nbhy;in&nbhy;SUN describes how to extend the FARE&nbhy;BGP protocol from switches to GPUs for scale&nbhy;up networks. Because scale&nbhy;up shares the same multi&nbhy;plane architectural pattern as multi-plane scale-out networks, the adaptive routing approach defined in FARE&nbhy;in&nbhy;SUN can be applied directly to multi&nbhy;plane scale&nbhy;out networks. The solution described in this document is almost identical to FARE&nbhy;in&nbhy;SUN, with the following two essential differences. First, FARE&nbhy;BGP is extended from switches to RNICs rather than to GPUs. Second, in a scale&nbhy;up network, the number of route entries is small (typically a few hundred) and can be installed directly on GPUs. In an isolated multi&nbhy;plane scale&nbhy;out network with 100,000 GPUs and four planes, each plane may propagate up to 100,000 host routes – a total of 400,000 routes. Storing all these routes on an RNIC is impractical. Therefore, the RNIC must suppress the routing table using the techniques described in Section 4. This document describes how to extend the Fully Adaptive Routing Ethernet (FARE) using BGP (FARE-BGP in short) as described in , which was originally designed for scale-out netowrks, to scale-up networks.

In an isolated multi&nbhy;plane scale&nbhy;out network, an RNIC connects to each plane and is configured as a stub BGP speaker per plane. It establishes separate BGP sessions with the attached leaf switches of each plane. The BGP neighbor discovery can be used to simplify configuration. Through these sessions, the RNIC learns routes to remote GPUs together with the path&nbhy;bandwidth extended community. Because the RNIC participates in BGP with each plane independently, it aggregates per&nbhy;plane path&nbhy;bandwidth information and performs weighted load balancing across planes. The RNIC thus performs the same Weighted Equal&nbhy;Cost Multi&nbhy;Path (WECMP) functions as a FARE&nbhy;capable switch, distributing traffic in proportion to the path bandwidth of each ECMP route. Two modes of WECMP are supported: Per&nbhy;flow WECMP (for RNICs that cannot handle disordered packet delivery): The RNIC establishes at least one QP per plane. The number of QPs allocated to a plane is proportional to the plane’s weight. All packets of a given flow go through the same plane, preserving order. Per&nbhy;packet WECMP (for RNICs that support out&nbhy;of&nbhy;order packet delivery): A single QP per (source, destination) RNIC pair suffices. The RNIC sprays each packet of that QP across all available planes according to the weights. In an isolated multi&nbhy;plane scale&nbhy;out network with 100,000 GPUs and four planes, each plane may propagate up to 100,000 host routes – a total of 400,000 routes. Storing all these routes on an RNIC is impractical. Two complementary approaches can reduce the number of routes the RNIC must store.

It's straightfoward to resort to route aggregation mechanism, i.e., aggregating host routes when advertising them from leaf to spine. However, naive aggregation can cause route blackholes: if a specific host within an aggregate becomes unreachable, the aggregated route still points to that plane. Consequently, traffic destined for that host will still be forwarded according to the aggregated route and then dropped. To address this issue, the switches MUST explicitly advertises unreachable host routes for a given RNIC to the other RNICs. When a RNIC becomes unreachable via a particular plane, the leaf switch advertises this unreachability to the RNIC using one of two methods: Path bandwidth value of 0: The leaf switch advertises the host route (NLRI) with the BGP path&nbhy;bandwidth extended community set to 0. The RNIC interprets this as “unreachable” and excludes that plane from the next&nbhy;hop set for that destination. Specific BGP unreachability advertisement: The leaf switch sends a dedicated BGP unreachability message. This is distinct from a standard BGP route withdrawal. It explicitly marks the host as unreachable via that plane while keeping the aggregated route intact. Upon receiving such an advertisement, the RNIC updates its forwarding table as follows: It locates the longest&nbhy;matching aggregated route that covers the unreachable host (e.g., a default route or a supernet prefix). From that aggregated route’s set of next&nbhy;hops (which originally included multiple planes), it removes the next&nbhy;hop corresponding to the plane where the host is unreachable. It then installs a host&nbhy;specific route for the unreachable destination, with the remaining next&nbhy;hops from the aggregated route. Example: Suppose an RNIC has a default route (0.0.0.0/0) with next&nbhy;hops pointing to planes A, B, C, and D. Host X (a specific /32) becomes unreachable via plane A. The RNIC learns an unreachable advertisement for X. It then creates a host route for X with next&nbhy;hops set to {B, C, D} – i.e., the original aggregated next&nbhy;hops minus the next&nbhy;hop associated with plane A. Traffic to X will never be sent to plane A, avoiding blackholes. This technique dramatically reduces BGP table size on the RNIC: the RNIC only needs to store aggregated routes (e.g., a handful of default routes per plane) plus explicit unreachable host routes for the small number of hosts that are actually unreachable. The majority of reachable hosts are covered by aggregates and require no per&nbhy;host state. The approach is especially effective when unreachability is rare, which is typical in well&nbhy;managed clusters. Switches within each plane does not need to install the unreachable host route into their FIB tables.

Since a given RNIC communicates only with a limited subset of GPUs (due to AI training parallelism patterns), it’s possible for the RNIC to filter routes to retain only those it actually needs. The RNIC sends Address Prefix ORF entries to its BGP peer (leaf switch) per plane. These entries indicate the host routes for remote RNICs the local RNIC is interested in. The peer filters outbound route updates accordingly, sending only the requested routes. In this way, the RNIC stores only a limited number of routes. For switches, there is no need install host routes for remote RNICs. Therefore, the FIB-suppression mechanism as described in Virtual Aggregation Auto-configuration could be reused.