Kademlia-directed ID-based Routing Architecture (KIRA)

Overview of KIRA KIRA's main objective is to provide self-organized robust control plane (CP) connectivity on top of a link-layer topology. The CP is typically used to configure, monitor, manage, and control networked resources (switches, routers, end-systems). The goal is to never lose control over the resources as long as there exist paths leading to them. KIRA is structured into a two-tier architecture consisting of a Routing Tier and a Forwarding Tier (see ). KIRA runs the zero-touch, distributed, highly scalable, ID-based routing protocol R²/Kad in the Routing Tier to find viable paths to destinations. The core concept of R²/Kad is that it discovers paths in the underlying topology by using an ID-based overlay routing scheme (based on Kademlia) combined with source routing between overlay hops. KIRA nodes employ this information to construct fast path forwarding tables in the Forwarding Tier for CP data traffic (e.g., packets from control plane applications). While source routing is robust (since it does not require converged routes) it can cause significant overhead, especially for small payloads. The Forwarding Tier avoids this overhead for CP data traffic by using PathIDs as forwarding state instead of the source routes and a scheme that is similar to label switching. However, existing forwarding support for IPv6 in hardware can be used, so KIRA does not require specialized new forwarding mechanisms. Add-on modules can make use of KIRA's routing information and available mechanisms, i.e., they are tightly coupled but offer optional support. Examples are a fully-distributed name resolution service (based on KIRA's built-in partial DHT functionality) and an efficient topology discovery service (KeLLy, ). R²/Kad employs a flat ID-based addressing scheme to easily support zero-touch operation, self-organization as well as mobility and multi-homing. ID-based routing (sometimes also denoted as name-independent routing or routing with flat identifiers) has the advantage of providing stable addresses (called NodeIDs) to upper layers. Thus, in case (virtual) resources are moved within the topology, any control connection to them stays alive. In contrast to other ID-based addressing approaches, KIRA is a genuine ID-based approach, because it does not use topological addresses at all and thus does not require any additional identifier-locator mapping (increased risk of inconsistency) and associated protocols (additional overhead and convergence time). As just motivated, R²/Kad uses topologically independent NodeIDs, generated by the KIRA nodes themselves, so address assignment is performed in a distributed manner by each node autonomously. Typically, NodeIDs are taken from a 112-bit address space, but depending on the network size, smaller NodeIDs are possible. KIRA uses IPv6 packets for its messages and CP data packets, because NodeIDs can be easily embedded into an IPv6 address (e.g., 16 bit prefix + 112 bit NodeID) and existing hardware-based and software-based forwarding mechanisms can be leveraged. Mainly very basic IPv6 features like link-local addresses, the packet format, fragmentation, and neighbor discovery are used, e.g., it does not require any address configuration features or router discovery. The Forwarding Tier is able to forward IPv6 packets, so that (control) applications can use IPv6 and all corresponding transport protocols above. R²/Kad messages use source routing based on NodeIDs whereas traffic in the Forwarding Tier uses NodeIDs and PathIDs for its forwarding decision. A PathID is conceptually a hash of a sequence of NodeIDs that form a path segment. PathIDs are unique (with high probability) for a path segment. To carry PathIDs in addition to the final destination NodeID, the original IPv6 packet becomes encapsulated, e.g., using a segment routing header that contains the PathID for the path segment that the packet should traverse next. Intermediate nodes simply exchange the PathID at each hop with the PathID of the remaining path segment (similar to label switching), or they strip the outer header when the next node is the end of a path segment. PathIDs are precomputed in a 2-hop vicinity of a KIRA node and are installed by R²/Kad signaling in some intermediate nodes on demand for paths longer than five hops. To forward CP packets KIRA nodes only need to perform lookups in their NodeID forwarding table and/or PathID forwarding table, perform encapsulation or decapsulation and rewrite PathIDs.

KIRA-Architecture Add-On Modules ┌─────────────────┐ ┌─────────────────┐ │ DHT-based │ │ KeLLy Topology │ │ Name Resolution │ │ Discovery │ Routing Tier └─────────────────┘ └─────────────────┘ ┌──────────────────────────────────────────────────┐ ┌─────────────┐ │R²/Kad │ │ Control │ │ ┌───────────────────┐ ┌───────────┐ ┌──────────┐ │ │ Plane │ │ │- Path Discovery │ │ Routing │ │Path │ │ │ Applications│ │ │- Routing │ │ Table │ │Management│ │ │ │ │ │- Failure Recovery │ │ │ │ │ │ │ │ │ └───────────────────┘ └───────────┘ └────┬─────┘ │ │ │ │ │ │ │ │ └──────▲───────────────────────────────────┼───────┘ └────▲────────┘ │ │ │Trans- │ ┌────────────┬───────┘ │port R²/Kad │ Forwarding │ │ │over Msgs │ Tier │ │ │IPv6 │ ┌─┬─────────▼─┬──┬───────▼─────┬─┬─────────┬─┬──▼───────┬─┐ │ │ │NodeID │ │PathID │ │Encaps./ │ │Node Local│ │ │ │ │Forwarding │ │Forwarding │ │Decaps. │ │Forwarding│ │ │ │ │Table │ │Table │ │ │ │ │ │ │ │ └───────────┘ └─────────────┘ └─────────┘ └─▲────────┘ │ │ │ │ │ │ │ ┌─────────────────────────────────┴─┐ │ │ │ ┌─►│ Fast Forwarding ├──┐ │ │ └─────────┼──┴───────────────────────────────────┴──┼─────┘ ▼ IPv6 ──┘ └─►IPv6 UDP+IPv6 CP applications (e.g., SDN controllers, Kubernetes Cluster Controllers, Virtual Infrastructure Manager, traditional OAM applications and so on) simply use NodeIDs as addresses for the resources/devices they want to control or for other controllers they want to exchange state with. Therefore, CP applications can transparently use the connectivity established by KIRA. Since NodeIDs are randomly generated, KIRA provides a simple built-in key/value store (Distributed Hash Table – DHT) that can be used as name service. KIRA nodes and services can dynamically register their NodeID under a certain well-known name and other KIRA nodes can lookup their corresponding NodeIDs. The DHT functionality will be specified as a separate KIRA module and corresponding application interfaces are out-of-scope for this specification. KIRA nodes possess relatively small routing tables, that grow with O(log(n)), where n is the number of KIRA nodes in the network (see for evaluation results). The advantage of small routing tables is scalability, but comes at the cost of path stretch. That is, packets to destinations that are not kept in the routing table of a node take a longer path than the shortest possible path, because they are using the ID-based overlay routing strategy. However, KIRA nodes will learn the shortest paths to all 'contacts' in their routing tables and it is a node local decision how large the routing table can be. For example, a controller node may add all KIRA nodes that it controls as contacts to its routing table. Because KIRA uses source routing in R²/Kad and PathID-based forwarding in its forwarding tier, it can easily support multi-path routing and keep backup paths for fast fail-over reactions. KIRA uses a mixture of reactive and periodic mechanisms to cope with link and node failures. Error messages that indicate failed links usually trigger routing updates and a path rediscovery procedure. However, routing updates are not flooded to all KIRA nodes, so some nodes may still have obsolete path information. These inconsistencies will be detected either when using the obsolete path to a contact (triggering an error message from the node before the broken link) or by a maintenance procedure that is carried out periodically. These periodic maintenance procedures test the validity of the currently known paths and may also trigger a rediscovery procedure to find alternative paths. Moreover, KIRA also possesses a specific end-system mode, where KIRA nodes are part of the KIRA network, but they are not exchanging routing information and are not forwarding packets for other KIRA nodes (see ). Finally, KIRA also supports a domain concept. A KIRA node may be member of one or multiple domains. Unless configured otherwise, a KIRA node is member of the global domain with DomainID=0 by default. KIRA nodes keep their NodeID for all domains, the only difference is that routes are guaranteed to run inside a domain D in case source and destination nodes are both members of this domain D. This allows for using domains for administrative purposes (e.g., all KIRA nodes inside the same Autonomous System could be part of the same domain) or to use domains to build clusters of KIRA nodes that are grouped by closeness in the underlying network topology.

Protocol Operation This section gives on overview of the main concepts with respect to the R²/Kad protocol operation. First, KIRA's ID-wise addressing concept is introduced, then the routing table structure is presented. After that, several procedures are described, beginning with node startup, vicinity discovery, and the join procedure to populate the routing table, followed by path discovery, overhearing, and rediscovery mechanisms.

Addressing, NodeIDs and the XOR Metric Every KIRA instance uses a single NodeID as its address. The NodeID is taken from a larger unstructured address space [0..2^B-1] (typically B=112). KIRA uses the XOR (logical exclusive or) metric in this address space to define the distance between two NodeIDs X and Y (see also ). The distance function d(X,Y)= X XOR Y is interpreted as integer and fulfills all properties of a metric (d(X,Y)>=0, and d(X,Y)=0 <=> X=Y; d(X,Y)=d(Y,X); as well as the triangle inequality d(X,Y)<=d(X,Z)+d(Z,Y)). This distance function largely corresponds to a prefix bit distance metric d_p(X,Y) = B-lcp(X,Y), where lcp(X,Y) denotes the length of the longest common prefix in bits. The XOR metric is finer than the d_p(X,Y) metric though, because when there are X,Y, and Z with d_p(X,Y)=d_p(Y,Z)=d_p(X,Z), the XOR metric can uniquely determine whether Y or Z are closer to X. More generally speaking, for a given distance d(X,Y)=d there exists exactly one Y so that d(X,Y)=d. This property is important since it allows a node to unambiguously determine which NodeID is ID-wise closer to a given other NodeID and it provides the basis for KIRA's loop-freedom. Note that the ID space with this metric is not cyclic (i.e., a node with a very small NodeID is not close to a node with a very large NodeID). The XOR metric defines an overlay structure across all KIRA nodes in the ID space: KIRA nodes establish logical connections with their ID-wise closest overlay neighbors (which are typically different from the underlay neighbors) with respect to the XOR metric as distance metric, i.e., the ID-wise closer neighbors have a smaller distance according to XOR. KIRA uses this metric to determine the next KIRA node that a KIRA message is forwarded to in order to reach a certain destination NodeID. R²/Kad messages are forwarded by using source routing between overlay hops. In addition to NodeIDs, KIRA can use any ID from the ID space as destination address. Typically, names of objects can be hashed to result in a key value, which is called Resource ID. In this case, the ID-wise closest KIRA node will be found as responsible node for storing a value (or a referral) for this key. This makes it possible to provide an integrated DHT for name-to-NodeID registration and lookup.

NodeID Creation NodeIDs are randomly generated and are taken from a 112 bit address space. Reserved NodeIDs listed in MUST NOT be used as NodeID of a KIRA node. Future versions of this specification will detail an algorithm to create self-certifying NodeIDs as using certain hash functions from a public key. NodeIDs are unique with high probability, e.g., for 10¹² nodes the probability for uniqueness is at least 1-9.63∙10⁻¹¹. However, in case two nodes possess the same NodeID, protocol mechanisms can be used to detect this situation (will be specified in a later version of this document) and the conflict can be resolved by letting both sides generate new NodeIDs. Depending on a KIRA node's capabilities, NodeIDs (together with other protocol parameters) may be stored in non-volatile memory so that nodes keep their NodeID even after restart. Other KIRA nodes may choose to generate a new NodeID on every restart.

Routing Table The entries in the routing table (RT) are called 'contacts'. Contact data contains the NodeID of the contact as well as a set of discovered paths that lead to this contact (besides other node state and attributes). A path to a contact is stored as path vector that contains a complete sequence of NodeIDs, which can be traversed to reach the contact. Except for contacts in its routing table, a KIRA node does not know paths to other destinations, but they can be discovered by using a recursive overlay routing strategy: a KIRA node source routes a packet to the contact (using the known path) that is the ID-wise closest to the destination ID according to its routing table. The next overlay hop performs the same action until the destination node is reached. After the initial discovery phase, only Underlay Neighbors (ULNs) and some contacts from the 3-hop underlay neighborhood are stored in the routing table. Underlay Neighbors are nodes attached to the links of the KIRA node (i.e., neighbors in the sense of that are directly reachable via link layer and the Internet-layer or higher-layer tunnels). R²/Kad's efficiency and flexibility are closely related to its routing table. It is structured as tree of k-buckets as in . A k-bucket in the routing table contains a list of (at most) k contacts in distance between 2ⁱ and 2ⁱ⁺¹ (i.e., the bucket's range, where 0<=i<112) from this node. Usually, k>=20 is constant and the same for all buckets and nodes, but it can also be varied per node (k=40 is RECOMMENDED as default for R²/Kad). Buckets at deeper levels share more prefix bits with the node's own ID; however, buckets for small values of i are generally empty as no appropriate nodes exist in this address space. Thus, the highest bucket (depth 1) contains contacts from half of the ID space whose highest NodeID bit differs from the node's ID, whereas the deepest buckets contain all nodes that are ID-wise closest to the node (i.e., the ID-wise closest overlay neighbors). If KIRA node X learns a new contact Y, it puts it into the corresponding k-bucket b_l in case it still has capacity left. The bucket index l is determined by calculating the common prefix length between X and Y (number of high-order zero bits of d(X,Y)). If the bucket contains k entries already, it is split into two new buckets (and the contained entries moved to them accordingly) in case X falls into the bucket's range. Otherwise, a selection algorithm determines whether the new contact should replace an existing entry in this bucket. KIRA uses Proximity Neighbor Selection (PNS) in all but the deepest two buckets so that contacts with shorter path lengths are preferred. If path lengths are equal, nodes with a higher degree (see NodeDegree in ) SHOULD be preferred as this results in shorter paths (following a powerlaw topology strategy). However, other contact selection strategies are possible even on a per-node basis, but the desired effect may not be visible unless all nodes follow the same strategy. Setting and changing strategies even during runtime inside a Domain is left for future work. An additional mechanism for path selection improves path diversity and prevents route flapping: in case an alternative path of equal length has been discovered for an already known contact, this path replaces the previous path only if the hash sum of the path elements is closer to the node's own NodeID. Due to the uniqueness property of the XOR metric, the path selection will always unambiguously converge to a unique path. Underlay neighbors are kept in special buckets that have no capacity limit, i.e., they will never be preempted. In general, routing also works without this ULN buckets extension of , but the resulting stretch will be slightly higher. X identifies its closest known contact in its RT by locating the k-bucket that corresponds to the longest matching prefix of destination Z with its own NodeID X by using d(X,Z). It then selects a contact with the shortest path vector from within the bucket; this is called Proximity Routing (PR). The XOR metric is used to uniquely select the ID-wise closest contact if all paths have equal length or if destination Z falls into the node's deepest bucket. Each contact typically contains the following information:

NodeID of the contact.
State: the state that the contact is considered to be in. It is one of undefined, valid, rediscovering, invalid, dead.
A set of path vectors, i.e., a complete sequence of NodeIDs that lead to the contact when traversed. There is an active path (used as default), a proposed path and possibly several backup or alternative paths. Paths also carry timestamps for points in time when they were last validated and when they were refreshed (using PathSetupRsp or ProbeRsp, see also ).
Sequence Number: a sequence number that is determined by the contact and that changes when the underlay connectivity state changes at the contact, i.e., with newly detected or failed underlay neighbors.
LastSeen: time of last direct contact
Sync ULN Sequence Number: a sequence number used for state synchronization with ULNs.
UnderlayNeighbor: a boolean value indicating that this contact is an underlay neighbor
NodeDegree: the contact's number of underlay neighbors.
Rediscovery State: temporary state keeping track of the rediscovery process.

Node Startup and Vicinity Discovery KIRA nodes generate their NodeID first (see ). After that they start an initial discovery phase to explore their underlay vicinity. After that, the join procedure and continuous discovery are periodically repeated to keep the node connected to its overlay and improve the quality of discovered routes. In its initial discovery phase, a KIRA node discovers its underlay adjacencies, i.e., its underlay neighbors (ULNs) that can be directly reached via link-local communication on one of their network interfaces. KIRA nodes periodically send ULNHello messages to a well-known link-local multicast address ALL-KIRA-NODES, and receiving nodes may reply with a ULNDiscoveryReq to set up an adjacency. This ULNDiscoveryReq MUST be answered by a ULNDiscoveryRsp to establish an adjacency. The protocol exchange ensures that bidirectional communication is possible between directly adjacent nodes. ULNs may be put into the ULNTable either after receiving a ULNDiscoveryReq as answer to a transmitted ULNHello or after receiving a ULNDiscoveryRsp as answer to a transmitted ULNDiscoveryReq. The ULNTable contains a mapping from NodeID to the link-local unicast address of the ULN. This address is either taken from the source address of the ULNDiscoveryReq or the ULNDiscoveryRsp respectively. KIRA nodes also discover all nodes in their 3-hop underlay neighborhood to populate their routing table, but the 2-hop vicinity is fully stored in a local graph structure ("vicinity graph"). The latter is used to precompute PathIDs for the 2-hop vicinity. ULNDiscoveryReq and ULNDiscoveryRsp messages contain a list of underlay neighbors so that all ULNs of ULNs will be learned, i.e., the 2-hop vicinity. However, possibly not all nodes in the 2-hop vicinity will be stored in the RT, therefore the vicinity graph structure is used to keep track of the 2-hop vicinity (new nodes and links). Especially, the state sequence numbers of all the nodes should be stored in the vicinity graph as well. All nodes in the 2-hop vicinity are queried for their ULNs by using a QueryRouteReq. QueryRouteReq/QueryRouteRsp messages are used to get ULNs or routing table information from nodes in the vicinity. Depending on their NodeID and other criteria (e.g., NodeDegree), nodes from the 3-hop vicinity will be stored as contacts into the routing table. The vicinity graph structure is also used to determine whether a new QueryRouteReq must be initiated according to the state-seq-num of the nodes in the vicinity. The vicinity graph can also be used to quickly calculate alternative paths, e.g., when direct links to ULNs break. A KIRA node continues to populate its routing table by sending FindNodeReq messages to certain nodes (especially those in the own ID-wise neighborhood) and also to randomly chosen destinations. The "Random Probing" procedure uses a randomly chosen ID from the NodeID space as destination for a FindNodeReq, however, a KIRA node does not need to exist for the chosen ID. The FindNodeReq will simply end at the KIRA node that is ID-wise closest to the destination ID. FindNodeRsp messages return a set of contacts that are ID-wise closest to the destination NodeID from the viewpoint of the responding KIRA node. The returned information is analyzed whether it can improve the local routing table, e.g., new and 'better' contacts or 'better' paths to already known contacts.

Join Procedure As a result of the vicinity discovery, all ULNs of X and some nodes within its 3-hop radius will populate X’s RT. However, in order to get network connectivity and to contribute to connectivity, the node needs to find its ID-wise closest overlay neighbors and make itself known to them. Thus, to join the network KIRA node X simply "searches" for the k closest nodes to its own NodeID: X sends a FindNodeReq for its own NodeID X and the closest neighbor replies with FindNodeRsp. Join requests can be detected by seeing that the source NodeID (src-node-id) and destination ID (dest-id) are identical. In this case the closest neighbors may know X, but they will exclude X from the result list and also answer the FindNodeReq instead of forwarding it to X. This is repeated with a limited exponential backoff in order to detect or heal any network partitioning. If node X finds itself in situations where it needs to respond with "Dead End" Error messages to FindNodeReqs, it SHOULD reset the backoff timer, because it may be a hint for network partitioning or other inconsistencies. In order to let a joining node X quickly learn all existing ID-wise closest overlay neighbors, X SHOULD send a QueryRouteReq to every newly learned contact that enters X’s currently deepest k-bucket. The queried contact replies with a QueryRouteRsp returning its RT entries for the k closest contacts to node X. The contacts returned in this manner will very likely also fall into X’s deepest bucket, possibly leading to a further split of its deepest bucket. Therefore, X will quickly populate its set of ID-wise closest overlay neighbors, which are needed for consistent overlay connectivity.

Path Discovery Consider the example topology in . Assume node X needs to send a message to node Z. If Z is a known contact of X, a path vector is stored already in the routing table that can be used for strict source routing in order to reach Z. Otherwise, a path to Z must be discovered using ID-based overlay routing. R²/Kad uses a recursive version of Kademlia (hence its name Routing with Recursive Kademlia – R²/Kad). The Path Discovery procedure uses a request/response message pair, FindNodeReq/FindNodeRsp. In this example, assume that X identifies its contact Y (learned from ULN A) as next (ID-wise closest) overlay hop toward Z. In order to discover a path to Z, X creates a FindNodeReq message that contains destination NodeID Z and source route r=⟨X, A, Y⟩ using path vector ⟨A⟩ of contact Y. The FindNodeReq is forwarded along r (strict source route). Message forwarding between overlay neighbors requires source routing, because the path in the underlay may lead via nodes that are (ID-wise) further away from the destination (e.g., Y routes via ⟨A, Q, M⟩ to Z in ): using the ID-based overlay routing scheme on a hop-by-hop basis (i.e., between directly adjacent nodes in the underlay) would inevitably lead to forwarding loops in most cases.

An example topology. Letters represent NodeIDs. Letters closer in the alphabet have smaller distance in ID space. Y / X--A--Q--M--Z \ / B----/ When the FindNodeReq arrives at Y, the same procedure is repeated (since it is a recursive variant of Kademlia). Node Y tries to find a contact closer to Z than Y itself. If this contact exists, source route r is appended by the corresponding path vector and the FindNodeReq is forwarded to this contact. In the given example of , we assume that Y knows Z as its contact with path vector ⟨A, Q, M⟩. It extends source route r of the FindNodeReq by ⟨A, Q, M, Z⟩ and forwards it to A as next hop in the source route. If routing information has converged, this ID-based routing scheme guarantees progress in the ID space during forwarding and eventually finds node Z. The destination node Z responds with a FindNodeRsp message along the reversed source path with any cycles removed (⟨Z, M, Q, A, X⟩). Due to XOR’s symmetry, the responding node Z also learns the new contact X as neighbor. The FindNodeRsp returned to X not only provides a path to Z, but also a list of k closest contacts to Z together with their path vectors. This list is used to improve X’s routing table. If the FindNodeReq arrives at Y and it cannot find a contact closer to Z, the FindNodeReq is terminated at Y and a response is sent back depending on the "ExactFlag" in the FindNodeReq. If the ExactFlag was not set, Y sends a FindNodeRsp back to originator X that contains an RT excerpt of Y’s (at most) k closest contacts to Z in a so called RTable object. This enables finding the responsible node for a destination ID Z if used as object key. This enables so called key-based routing that is used to realize DHTs. If exact was set, X assumed that a node with ID Z must exist, but the current node is the ID-wise closest node to Z and does not know Z as contact. Consequently, the node cannot forward the FindNodeReq closer to Z and returns a "Dead End" Error message (which may happen occasionally during convergence). If source route r contains a broken link or unreachable node, a "Segment Failure" Error message will be sent back to X along the reversed source route (with any cycles removed).

Overhearing of R²/Kad Messages KIRA nodes use overhearing mechanisms for R²/Kad messages. This is an important mechanism for KIRA to learn new contacts and better or improved paths. Nodes that forward R²/Kad messages SHOULD use the contained source route to improve their own routing information: they may learn new contacts or shortcut routes to known contacts. However, only the so far traversed path is considered as it can be assumed that all traversed links worked recently. Additionally, NotViaList information (see ) is used to invalidate contacts that have an active path vector containing a link from the NotViaList. The source routing path of incoming requests and responses is also considered for improving the RT. Some messages like FindNodeRsp, QueryRouteRsp or UpdateRouteReq contain RTable objects that are evaluated likewise. However, in contrast to the source route information that can be considered as being most recent information, RTable information may be less recent (and also less trustworthy) and needs to be validated first before it can be used. Therefore, in case a path from the RTable is considered being of interest (improvement over an existing path), it needs to be checked by sending a ProbeReq message and successfully receiving the corresponding ProbeRsp. See for more details. Bypassing QueryRouteRsp messages contain RTable objects (as requested by QueryRouteReq) and SHOULD be inspected for interesting contacts and paths.

Periodic Path Probing Periodic Path Probing aims at reliably detecting any RT inconsistencies (e.g., seemingly valid contacts with paths that contain recently failed links). Each node periodically checks the path validity for all of its contacts by sending a ProbeReq message to them. ID-wise closest neighbors are probed more often than other contacts and those recently contacted (≤ 2s) are not probed. If a path has a link or node failure, the ProbeReq will elicit a "Segment Failure" Error message from an intermediate node along the broken path, notifying about the failed link. The contact’s state will be set to invalid and a rediscovery process is scheduled (see ).

Dynamics: Recovery from Failures In order to improve R²/Kad’s robustness against link or node failures KIRA specifies a recovery procedure that notifies about failures and actively tries to find alternative paths that route around the failure. This procedure is highly robust and achieves a fast convergence. R²/Kad nodes detect link and node failures of ULNs by link layer notifications, missing ULNHello or ULNDiscoveryRsp messages as well as "Segment Failure" errors anytime during forwarding along source routes. To recover from such failures, R²/Kad’s recovery procedure uses the following mechanisms:

Notify own nearest overlay neighbors about failed links or unreachable nodes ("bad news") by sending UpdateRouteReqs via a non-impacted underlay link.
Rediscover a feasible alternative route to the affected node using FindNodeReqs. These carry NotViaList information about failed links that MUST NOT be considered for routing the FindNodeReq. Rediscovery is not performed for nodes that lost their only link, which can be deduced by the node’s degree information that is conveyed in R²/Kad messages.
Per contact state sequence numbers avoid using obsolete information for path rediscovery. Additionally, an aging mechanism is used to avoid dissemination of obsolete routing information. It uses time periods to assess the currentness of the related path.
Overhearing of NotViaList information and UpdateRouteReqs about failed links during forwarding R²/Kad messages informs nodes about failed links, which initiates a path rediscovery. Overhearing is also used to update obsolete path information.
When an alternative path has been found for a prior affected contact or a link comes back up again, an UpdateRouteReq is sent to own ID-wise closest overlay neighbors for disseminating the "good news".

The ID-based overlay routing scheme is used for rediscovery of a route, because NodeIDs are randomly distributed all over the underlying topology. Therefore, a rediscovery uses different paths that are likely not affected by the failure. However, if overlay nodes still have obsolete routing information, i.e., they would normally route via the failed link, they can detect the need to update their routes as well by seeing the more current NotViaList information.

Path Rediscovery A node X that detects its ULN B (cf. ) or the corresponding link ⟨X, B⟩ has failed, reacts as follows (unless isolated by that failure):

Set the state of the corresponding contact to invalid (in contact B). Invalid contacts will temporarily not be considered for routing.
Set the state of all contacts whose paths contain the failed link ⟨X, B⟩ to invalid (in contact Z with ⟨B, M, Z⟩ becomes invalid).
Send UpdateRouteReq messages indicating the failure to four of its ID-wise nearest neighbors (e.g., Y and Z in ) via non-affected contacts. The UpdateRouteReq will also carry a NotViaList that contains ⟨X, B⟩.
Trigger a rediscovery process (described below) for B (sets state to rediscovery) and for other invalid contacts.
If the rediscovery process is successful for a contact, its state is set to valid and UpdateRouteReq messages are sent to notify ID-wise closest neighbors about the change.

Since UpdateRouteReqs have notification character only, they do not create any responses (even no error messages if dropped). The rediscovery process simply sends a FindNodeReq for all invalid contacts (all invalid contacts will be ignored in finding the next hop). This FindNodeReq for rediscovery (also denoted as rediscovery message) contains a set ExactFlag and the failed link ⟨X, B⟩ as additional NotViaList information. It is sent to X’s currently known ID-wise closest neighbors of the invalid contact (e.g., A in the example), which will then try to forward the FindNodeReq further toward the failed contact. The NotViaList information prevents nodes from using obsolete routing information when forwarding the rediscovery message, i.e., paths that contain the failed link will not be used for forwarding. Node A may not have heard yet about the broken link and thus will invalidate contact B if its prior preferred path is via ⟨X, B⟩. In order to ensure that only current NotViaList information is considered, every link contained in the NotViaList is also accompanied by a related age value ∆T, specifying in milliseconds how long ago the sender heard about the failed link. When a FindNodeRsp is returned by B, a valid path has been discovered and the contact’s state is set to valid (triggering subsequent UpdateRouteReqs with the new path as mentioned before). Nodes receiving UpdateRouteReqs or FindNodeReqs containing the failed link also set their corresponding affected contacts to invalid and trigger a rediscovery process of the routes (like Z in the example). The actual rediscovery