Acts 1 through 5 chose the fabric and the file system. This Act covers the step in between that gets skipped: how the client is mounted, which decides how many flows it offers the network -- and therefore whether any of the load balancing in Chapter 6 has anything to work with.
One mount option decides your storage entropy
A default NFS mount opens one TCP connection to the server. One connection is one 5-tuple. One 5-tuple hashes to exactly one path through the fabric, forever.
That is the elephant flow from Chapter 6, arriving from a direction nobody watches. The compute fabric gets careful attention -- adaptive routing, per-packet spraying, hash tuning. The storage fabric gets a mount command somebody typed once, and it pins every byte a node reads to a single spine.
nconnect fixes it by opening multiple connections to the same server:
sudo mount -t nfs \
-o proto=tcp,vers=3,nconnect=16 \
172.16.200.21:/exports/training /mnt/training
Sixteen connections, sixteen 5-tuples, sixteen hash results. Now ECMP has something to spread.
Verify it rather than assuming -- the option is silently ignored by older clients:
mount | grep training # confirm nconnect appears in the active options
ss -tn dst 172.16.200.21 # count the established connections
RDMA and multipath: fanning out across NICs and targets
The higher-performance form drops TCP entirely and adds two more axes:
sudo mount -t nfs \
-o vers=3,proto=rdma,port=20049,nconnect=16,\
localports=0.0.0.0-0.0.0.3,remoteports=172.16.200.21-172.16.200.36 \
172.16.200.21:/exports/training /mnt/training
Reading that option string left to right:
proto=rdma-- NFS over RDMA rather than TCP.port=20049-- not 2049. NFS/RDMA listens on its own port; using the TCP port here fails in a way that reads like a firewall problem.nconnect=16-- sixteen connections, as above.localports=-- a range of local addresses, one per client NIC. The client now sources traffic from four interfaces instead of one.remoteports=-- the range of target addresses, here all sixteen storage endpoints.
The result is a client fanning out across 4 local ports and 16 targets. That is a lot of entropy from one mount line, and it is why a storage fabric that looked hopelessly imbalanced can come good without a single switch change.
It also means the mount options are load-bearing configuration. A
localports range that lists three NICs on a four-NIC client silently gives
up a quarter of the client's bandwidth, with no error anywhere. After
replacing a storage NIC, re-read the mount options -- Chapter 9's
mst status -v will tell you what the interfaces are actually called now.
When the client is slow and every switch counter is clean
Two host-side causes that look exactly like a fabric problem, and are not:
Interrupt affinity. If one CPU core is servicing every interrupt from the storage NIC, that core saturates and the NIC never reaches line rate. Every switch counter is clean because the switch is fine. Spread the interrupts across cores.
The PCIe slot. A x16 card seated in a physically-x16 slot that is electrically x8 -- or in a Gen3 slot on a Gen4 card -- halves the available bandwidth with no indication except the throughput number. Confirm both width and generation after any physical work, not just that the card is present.
The isolation test is the one from Chapter 25, pointed at the storage NIC:
ib_write_bw -d mlx5_4 --iters 5000 --size 65536 <target>
- Result at line rate -- the hardware, driver, and physical path are fine. The problem is above them: mount options, file system, or application.
- Result well below line rate -- stop looking at NFS. The problem is the host, the slot, or the link.
That single split saves the most common wasted afternoon in storage troubleshooting, which is tuning a file system that was never the bottleneck.