From the physical layer to a routed AI fabric

Physical layer orientation. What an HCA is, why NICs became SuperNICs, how a GPU node is wired, and the separate networks that surround it — told through DGX H100 and mapped onto HGX B300.

What runs on every device. How you access it after power-on. The management philosophy. CLI vs orchestrated. First power-on sequence.

The AllReduce barrier, why TCP fails, tail latency math, and the mental model shift from enterprise to AI networking.

The commands and discipline for reading HPC fabric state. Which commands run where, how to read their output, and the investigation workflow from physical layer to configuration.

The InfiniBand operations layer: ONYX CLI, error counter interpretation, Subnet Manager management, ibdiagnet fabric sweep, and UFM event correlation.

How losslessness actually works: PFC mechanics at the wire level, pause storm formation, ECN CE bit marking, DCQCN rate control algorithm, and the complete RoCEv2 port configuration checklist.

Why AI traffic is structurally low-entropy and how that breaks ECMP. The four load balancing modes (SLB, DLB, GLB, sDLB). Per-packet spraying and RSHP. In-cast congestion patterns and how to diagnose them from spine utilisation counters.

Fat-tree design, oversubscription arithmetic, rail-optimised versus rail-unified, switch selection by radix, and how the same tree is sized for 800 Gb/s HGX B300 endpoints.

How NCCL translates AllReduce into RDMA operations. Ring vs Tree vs Double-Binary Tree algorithms. The environment variables that determine whether NCCL finds RDMA or falls back to TCP. Reading nccl-tests output. Correlating busbw degradation to fabric diagnostics.

The physical layer beneath the fabric: 400G/800G optics, DSPs, fiber types, form factors, cable selection, and why signal integrity and power density now shape AI cluster design.

The separate network that feeds and protects training: storage isolation, GDS data paths, NVMe-oF transports, parallel file systems, checkpoint economics, and storage topology choices.

Know about problems before the ML engineer's Slack message arrives. UFM REST API, DCGM GPU metrics, Prometheus alert design, threshold calibration, and cross-layer correlation across four monitoring streams.

Network-relevant implications of GPU generations: the NVLink and NVSwitch generation table, SXM vs PCIe form factors, GH200, MIG, and Blackwell Ultra (HGX B300) as the platform this course builds against.

The scale-up fabric: NVLink Switch hardware, bandwidth arithmetic, NVLink Network addressing, Fabric Manager, diagnostics, and how rack-scale GB300 NVL72 inverts the relationship between scale-up and scale-out.

Torus, folded torus, dragonfly, and TPU Pod design choices - where they came from, what workloads they suit, and why fat-tree remains dominant for AI training clusters.

How modern AI fabrics use routed Ethernet: BGP unnumbered, ASN design, BGP DPF, RIFT comparisons, Flex Algo, SRv6 path steering, and multi-tenant EVPN-VXLAN design.

A packet-level walkthrough from NCCL work queue entries to remote DMA completion: DGX interfaces, Queue Pair mechanics, ConnectX-7 processing, switch forwarding, and end-to-end packet decode.

A packet-level walkthrough of a checkpoint write from GPU HBM to storage appliance: NVMe-oF capsules, DMA paths, storage-fabric behavior, frame anatomy, and diagnostics.

The out-of-band management fabric: BMC architecture, IPMI and Redfish internals, OOB topology, switch management isolation, UFM communication paths, BlueField-3 management architecture, and hardening guidance.

A complete addressing reference for DGX BasePOD and SuperPOD deployments: address families, RFC 1918 partitioning, loopback design, P2P links, /32 server routes, management-plane planning, VXLAN VNI allocation, and scaling pitfalls.

Why RoCEv2 has friction at scale, and how UEC addresses it: UET packet format, SACK-based reliability without lossless fabric, 1-RTT congestion feedback, native multipath spraying, switch requirements, and honest deployment readiness as of March 2026.

A rigorous, algorithm-level treatment of every congestion control scheme used in production AI fabrics — DCQCN, Swift, HPCC, TIMELY, and UEC CC — with practical guidance on parameter tuning and algorithm selection.

A practitioner's guide to deploying SRv6, SR-TE, EVPN+SRv6, and IS-IS Flex-Algo in production AI data centre fabrics.

The complete security layer for AI fabrics: RDMA threat model, RoCEv2 RKEY protection, Spectrum-X GBP microsegmentation, InfiniBand PKey isolation, BlueField-3 as a security enforcement point, and UFM Cyber-AI anomaly detection.

A deep dive into Spectrum-X: Spectrum-4 switch ASICs, BlueField-3 and ConnectX-8 SuperNICs, DOCA and NetQ, assembled into a coherent AI factory architecture with HGX B300 at the endpoints.

Configure RoCEv2 end to end on Spectrum-X: prerequisites, NVUE Day-0 workflow, QoS architecture, ECN/PFC tuning, and production verification.

Why flow-based ECMP fails AI collectives, how the Spectrum-4 adaptive routing engine rebalances in real time, and why per-packet spraying needs a SuperNIC reorder buffer — BlueField-3 on B200, ConnectX-8 on HGX B300.

How Spectrum-X enforces tenant isolation using BGP-EVPN, VXLAN, and GBP microsegmentation - from VNI planning to route-target configuration and operational troubleshooting.

Why air ran out above 40 kW, how the CDU and secondary loop are built, the flow and delta-T arithmetic, and the four ways cooling lands on the network team.

From delivered hardware to a schedulable machine: seven gates, out-of-band first, a frozen firmware manifest, rail audits, link soak, node health and scheduler plumbing.

Link qualification, per-unit and cluster-wide collectives, cross-unit consistency, interference and endurance, then training benchmarks against published baselines.

Four separate fabrics, rail-optimised sizing derived from switch radix, frontend and border port budgets, the security edge, out-of-band capacity, addressing and every packet path traced.

A no-incident calibration lab. Learn how UFM, DGX OS, Linux netdev names, and Cumulus NVUE each describe the same two-rail fabric before you start troubleshooting failures.

A healthy-link counter drill. Run a short RDMA write probe, then correlate switch-side PFC and RoCE counters with DGX ethtool counters so ECN marks, PFC pauses, and zero-drop behavior become familiar before incident labs.

A GPU rail has gone dark. Use the topology map and CLI tools to identify which rail, confirm the RDMA state, and isolate whether the fault is on the NIC or switch side.

A RoCEv2 workload is experiencing retransmissions. PFC is misconfigured. Diagnose and fix it.

Throughput has dropped 40%. Investigate using interface counters and ECN configuration commands.

AllReduce throughput has dropped with no drops visible. Diagnose why spine links are uneven and fix load balancing to restore full training throughput.

Two vendors propose different switch configurations for a 64-node DGX H100 cluster. Calculate oversubscription ratios, identify which proposal meets requirements, and submit a recommendation before the purchase order is signed.

A 16-node cluster shows 3 GB/s busbw instead of expected performance. All hardware is healthy. Diagnose why NCCL is using socket transport and fix the environment variable misconfiguration.

Training is 12% slower - but no hard errors anywhere. Use UFM port counters, DCGM GPU metrics, and switch counters to correlate a rising pre-FEC BER across three monitoring layers and identify the marginal physical connector before it becomes a full link failure.

A switch looks healthy at a glance, but the NIC reveals a severe pause storm. Check both ends, identify missing ECN, and restore rate control before continuous PFC pauses collapse throughput.

PFC is enabled, but on the wrong traffic class. Cross-check NIC drops, PFC priority output, and RoCE DSCP-to-priority mapping, then move PFC protection back to priority 3.

A rail has gone err-disabled after a physical fault. Recognise the NIC-side active-state trap, confirm the switch port failure, replace the optic, clear err-disable, and verify full recovery.

A reduced-capacity spine is still receiving equal ECMP traffic. Identify the missing BGP Link Bandwidth community and restore weighted ECMP before PFC storms spread.

A link failure triggers a bad 3-hop path because the spines use different ASNs. Trace the suboptimal route, unify the spine ASN design, and verify clean failover behavior.

Steer checkpoint-to-storage traffic onto a dedicated SRv6-TE path so collective traffic keeps the uncongested spine pool.

Reproduce an RDMA RKEY exposure attack from a multi-tenant cluster, identify the two misconfigurations that enabled it, and apply GID filtering and RKEY rotation to harden the environment.

Audit a newly racked Spectrum-X scalable unit before RoCE configuration begins by verifying version, ports, ASIC identity, MTU baseline, and BGP cleanliness.

Apply the Day-0 Spectrum-X RoCE QoS profile, verify DSCP trust and PFC/ECN state, and validate the fabric with ib_write_bw and ib_write_lat.

Diagnose a broken ECN profile, compare thresholds to the carved lossless buffer, then restore DCQCN marking under multi-flow load.

Fix a Spectrum-X fabric where per-packet AR silently fell back to a broken per-flowlet setup because BF3 reorder buffers were never enabled on the DGX B200 hosts.

Diagnose and fix a BGP-EVPN route-target misconfiguration that leaks TenantA's storage routes into TenantB's VRF.