You have seen the compute frame anatomy in Ch16. Now trace the storage frame at the same level. The outer framing is identical -- Ethernet/IPv4/UDP/BTH -- but the payload above BTH is an RDMA Send carrying an NVMe-oF capsule instead of an AllReduce message.

For the SQE capsule SEND (step 1 of the exchange):

Ethernet II header         14 bytes
  Destination MAC          storage switch port MAC
  Source MAC               CX7 storage NIC MAC (enp170s0f0 or enp41s0f0)
  EtherType                0x0800 (IPv4)

IPv4 header                20 bytes
  Source IP                DGX node storage IP (e.g. 10.20.1.10)
  Destination IP           Storage appliance target portal IP (e.g. 10.20.2.5)
  Protocol                 UDP (17)
  DSCP                     26 or 46 -> priority 3 (site-configurable, verify with mlnx_qos)
  TTL                      64

UDP header                 8 bytes
  Source port              ephemeral (entropy for ECMP, typically 0xC000-0xFFFF)
  Destination port         4791 (RoCEv2 well-known port)

BTH (Base Transport Header) 12 bytes
  OpCode                   0x04 = SEND_ONLY (for SQE capsule delivery)
  Partition Key            0xFFFF (default)
  Queue Pair Number        NVMe-oF QP allocated at connection setup
  PSN                      per-QP packet sequence number
  Flags                    solicited event set (notifies target CQ)

SGL1 (inside the SQE) provides the virtual address, RKEY, and DMA length used by the target's
subsequent RDMA Read request against the initiator MR.

NVMe-oF SQE capsule        64 bytes
  Opcode                   0x01 (Write) or 0x02 (Read)
  NSID                     target namespace identifier
  LBA                      starting logical block address
  NLB                      number of logical blocks minus 1
  SGL1                     scatter/gather list descriptor
    type                   SGL_KEYED_DATA_BLOCK (0x44)
    length                 data transfer length in bytes
    address                virtual address of initiator data buffer
    RKEY                   memory region key (registered at connection setup)

ICRC (Invariant CRC)       4 bytes

Total header overhead for the SQE capsule frame: ~134 bytes. The SQE capsule itself is 64 bytes. That 64-byte command triggers the data transfer, which is carried in subsequent RDMA Read Response frames.

The data transfer frames

The data payload arrives in RDMA Read Response frames, not Send frames. The target issues an RDMA Read request against the initiator MR, and the initiator CX7 returns data in one or more Read Response packets. For a 4 MiB checkpoint block, at 4096-byte MTU you get roughly 1024 data frames. The BTH OpCode sequence is:

• First packet: RDMA_READ_RESPONSE_FIRST (0x0D)
• Middle packets: RDMA_READ_RESPONSE_MIDDLE (0x0E) -- data only
• Last packet: RDMA_READ_RESPONSE_LAST (0x0F) or RDMA_READ_RESPONSE_ONLY (0x10) if single-frame

The ICRC covers the invariant fields of all these frames. The switch never rewrites the ICRC -- the BTH and RETH are not modified in transit.

DSCP and priority on the storage fabric

The storage CX7 NIC stamps DSCP on outbound storage frames. The value is configured via the NVMe-oF subsystem or via explicit QoS policy, not by NCCL or cuFile. Verify with:

mlnx_qos -i enp170s0f0 --dscp

The switch must have a matching DSCP-to-priority map. On a storage switch running Cumulus Linux:

nv show qos dscp-map

If the NIC stamps DSCP 26 but the switch maps DSCP 26 to priority 0 (best-effort), the storage traffic competes with management traffic in the same output queue. This is a classic misconfiguration.