An alert fires at 2 AM. GPU training throughput on Cluster A has dropped to 62% of baseline. The monitoring dashboard shows elevated AllReduce latency and one GPU contributing nothing to the training job. Your job is to find out why.
You open your laptop and SSH into the cluster. You have a terminal. You have a set of commands. What you type next -- and in what order -- determines whether you find the problem in ten minutes or spend three hours going in circles.
This chapter is about the discipline of reading a fabric from the command line. Not a reference manual. A way of thinking. Every command in this chapter exists because it answers a specific question in a specific order. By the end, you will know not just what to type, but why you type it when you do.
The three diagnostic questions — in order
The investigation always starts with the same three questions:
Question 1: Is the physical link up? Before you look at counters or configuration, you need to know whether the link exists at all. A link that is physically down answers all other questions immediately.
Question 2: What is the traffic doing? Counters tell you what is happening right now -- drops, pauses, errors. They are the symptoms.
Question 3: What is the configuration? Configuration tells you why the symptoms are happening. But you read it last, not first -- because a perfectly valid configuration can still produce bad behaviour if the traffic pattern is wrong.
This ordering matters. Jumping to configuration before checking counters is the most common mistake network engineers make when entering HPC environments. In enterprise networking, configuration problems dominate. In HPC, traffic behaviour problems dominate -- a valid PFC configuration can still produce a pause storm.