Connector types at 400G
All current 400G AI fabric links use one of two connector families:
QSFP-DD (Quad Small Form-factor Double Density): 8 lanes x 50G PAM4 = 400G. The dominant connector for 400G Ethernet. Compatible with the QSFP28 form factor with an adapter. Used on most Spectrum-X switches.
OSFP (Octal Small Form-factor Pluggable): Also 8 lanes x 50G PAM4 = 400G. Physically larger than QSFP-DD, with better thermal headroom. Used on some high-port-density switches and on the ConnectX-7 NIC in DGX H100/H200.
Both connectors use the same optical transceivers at 400G. They are incompatible without an adapter, so it is important to match connector types when selecting switches and cables.
Real hardware: OSFP transceivers installed in a DGX H100 node's ConnectX-7 NICs. 8 NICs visible, each with one OSFP port. The short patch cables visible are going to the top-of-rack switch directly above.
![DGX H100 rear panel showing 8 ConnectX-7 NICs with OSFP connectors and patch cables] Source: NVIDIA DGX H100 hardware documentation and data center deployment guides Image placeholder: Search "DGX H100 rear panel ConnectX-7 OSFP" for official NVIDIA hardware images showing the NIC configuration.
Cable options by reach
| Cable type | Max reach | Use case | Cost relative |
|---|---|---|---|
| DAC (Direct Attach Copper) | 3-5m | Within same rack or adjacent rack | Lowest |
| AOC (Active Optical Cable) | 10-100m | Between racks, patch panel reach | Medium |
| Passive optical + transceiver | Up to 100m | Structured cabling plants | Medium-high |
| Active optical + amplifier | 500m+ | Campus or multi-building | High |
For a DGX BasePOD where compute and switch racks are adjacent, DAC cables are common for the DGX-to-leaf connections (typically 3m or less between racks). Leaf-to-spine connections often require AOC cables (10-30m if switch racks are separated).
For a SuperPOD with structured cabling to a patch panel, passive optical with MPO trunks and OSFP transceivers is standard. The patch panel approach allows re-patching without running new cable through the data center -- critical when you cannot recable without a maintenance window.
Why one wrong cable ruins a rail
The rail-optimised design's physical constraint is that every cable path must be precisely correct. Consider what happens when a DGX node is installed and the technician connects GPU 3's NIC to Leaf Switch 4 instead of Leaf Switch 3:
- GPU 3 on this node is now on Rail 4, not Rail 3
- When Rail 3's AllReduce runs, GPU 3 on this node is unreachable via the rail path
- When Rail 4's AllReduce runs, this GPU 3 appears but creates asymmetry (one node has 2 GPUs on Rail 4)
- The training framework (NCCL) will detect the topology inconsistency and either fail or run at degraded performance
This is the specific failure ibstat would show as Rail 3 having one fewer node than expected, and Rail 4 having one extra. It is a cabling error, not a hardware failure, and it is not obvious from any counter alone -- you need UFM topology view or show topology to see the misconnection.
Cabling documentation -- labelling every cable with source port, destination port, and rail assignment -- is not optional in a production AI cluster. NVIDIA's reference deployments include a specific labelling standard for exactly this reason.
Real deployment: A structured cabling patch panel for a DGX SuperPOD installation. MPO fibre trunks fanned out to individual LC or OSFP connections. Each connection labelled with rail number, node index, and switch port.
![AI cluster structured cabling patch panel with labelled MPO fibre connections] Source: AI data center deployment documentation from major integrators Image placeholder: Search "DGX SuperPOD cabling guide" or "AI cluster structured cabling patch panel" for relevant deployment images.