Skip to content

Segment Routing for AI Fabrics · Part 6 of 8

Act 6 — Troubleshooting SR Fabrics

SR adds a new class of operational problems that require new diagnostic tools and mental models.

The three categories of SR failure

Category 1: SID not reachable — The headend cannot resolve the SID to a forwarding entry. This happens when the locator prefix is not being distributed by IS-IS or BGP.

# Symptom: traffic is black-holed at the headend
# Diagnosis:
ip -6 route show 2001:db8:0:spine02::/64
# If no route exists: IS-IS is not distributing the locator

vtysh -c "show isis database detail leaf01" | grep SRv6
# Check if SRv6 Capabilities TLV is present

vtysh -c "show isis srv6 node"
# Shows all SRv6 nodes discovered by IS-IS

# Fix: verify IS-IS adjacency and SRv6 is enabled in IS-IS config
vtysh -c "show isis neighbor"

Category 2: SRH MTU violation — The SRH adds bytes to every packet, reducing the effective MTU for the payload. If the fabric is configured for 9000-byte jumbo frames and an SRH with 3 SIDs adds 56 bytes, packets > 8944 bytes will be dropped silently.

# Calculate the effective MTU reduction:
# SRH overhead = 8 (fixed) + (N × 16) bytes
# For N=3 SIDs: 8 + 48 = 56 bytes
# Effective payload MTU = 9000 - 56 = 8944 bytes

# Verify MTU on all links in the SR domain:
nv show interface swp1 | grep mtu
ip link show | grep mtu

# Check for MTU-related drops (ICMP too big messages):
tcpdump -i any icmp6 and 'icmp6[0] == 2'  # ICMPv6 Packet Too Big

# Fix: either reduce application MTU or increase fabric MTU to 9100+
# NVUE: set larger MTU
nv set interface swp1 link mtu 9216
nv config apply

Category 3: SID function mismatch — The headend inserts an End.X SID for an interface that no longer exists, or uses an End.DT4 SID with the wrong VRF mapping.

# Symptom: traffic arrives at wrong node or gets dropped at egress
# Diagnosis: trace the path

# Step 1: Capture the SRH on the sender
tcpdump -i eth0 -v | grep -A10 "Routing: SRH"
# Shows the SID list in the outbound packet

# Step 2: Traceroute with SRH visible
traceroute6 -n 2001:db8:0:leaf08::1
# Each hop shows which SID is active

# Step 3: Verify the endpoint SID is correctly programmed on the remote leaf
ssh leaf-08 vtysh -c "show segment-routing srv6 sid" | grep "End.DT4"
# Confirm the SID exists and maps to the correct VRF

# Step 4: Check SRv6 forwarding table:
ip -6 route show table local | grep "seg6"
# Shows locally-active SID endpoints

WJH (What Just Happened) for SRv6 drops

On Spectrum-4 switches, WJH (Chapter 29) captures hardware-level drop events including SRv6-specific drops. Key WJH SRv6 drop reasons:

# Run WJH and filter for SRv6 events:
# (wjhd must be running — typically via DOCA or Docker)
docker exec wjhd wjh-util show --type forwarding | grep -i srv6

# Common SRv6 drop reasons:
# SRV6_SID_NOT_LOCAL   = Packet arrived with SID that doesn't match any local SID
# SRV6_SRH_INVALID     = SRH is malformed (bad segment list length)
# SRV6_NEXT_HEADER_ERR = After SRH processing, next header is unexpected

srinfo utility

The srinfo tool (available on Cumulus Linux) provides a summary view of the SR state:

srinfo
# Output:
# SRv6 Node:     leaf-01
# Locator:       2001:db8:0:leaf01::/64
# Active SIDs:   4
# IS-IS peers:   6
# BGP neighbors: 8
# SR-TE policies:3 active, 0 down
# Binding SIDs:  2

# Per-policy detail:
srinfo policy CHECKPOINT-STORAGE
# Shows: SID list, candidate path preference, active/backup, byte/packet counters