Training a large AI model is a different class of problem. To understand why, you need to understand what the computation actually looks like.

A neural network is a mathematical function with billions of parameters -- numbers that get tuned during training. The training process works like this:

1. Feed the model a batch of data
2. The model makes predictions
3. Measure how wrong the predictions are (the "loss")
4. Calculate how each parameter contributed to the error (gradients)
5. Nudge every parameter slightly in the direction that reduces the error
6. Repeat millions of times

The problem is step 4. Calculating gradients for a model with 70 billion parameters requires enormous memory and compute. A single GPU -- even an H100 with 80 GB of memory -- cannot hold the entire model. It must be split.

When you split training across many GPUs, those GPUs must communicate constantly. After every training step, every GPU's gradient updates must be combined and shared back -- this operation is called AllReduce. It is a synchronisation barrier: no GPU can start the next step until all GPUs have finished exchanging gradients.

The communication requirement scales with the number of GPUs. More GPUs means more parallelism, but also more data that must cross the network on every single step. This is the root of everything that follows.

This synchronisation pattern creates a distinctive traffic shape called burstiness. Between synchronisation points, traffic is moderate -- GPUs are computing, doing forward and backward passes. At the synchronisation point, every GPU sends its gradients simultaneously -- traffic spikes sharply, then returns to baseline when synchronisation completes. The fabric must absorb these spikes without dropping a single packet.

Buffer depth, ECN threshold placement, and PFC headroom are all sized around this burst pattern.

Four types of parallelism -- four different network traffic patterns

Training is accelerated by splitting the work across GPUs. There are four distinct approaches, and each one produces a different pattern of inter-GPU communication. As a network engineer, understanding which type dominates a given training run tells you what traffic pattern to expect on the fabric.

Data parallelism -- All GPUs run the same complete model but on different batches of data. After each training step, every GPU must share its gradient updates with every other GPU and receive all their updates in return. This is the AllReduce operation. The communication pattern is all-to-all: every GPU talks to every other GPU simultaneously. This is the highest-demand pattern for the external fabric -- it generates a traffic matrix where every node is both a sender and receiver to all other nodes at the same time. Rail-optimised topology and non-blocking switches exist specifically to handle this pattern efficiently.

Tensor parallelism -- A single model layer (for example, a large matrix multiplication) is split across multiple GPUs within a single node. Each GPU handles a shard of the computation and they exchange partial results continuously. Because this communication is so frequent and the data volumes are large, it runs over NVLink -- the 900 GB/s intra-node interconnect. Tensor parallelism almost never crosses the external fabric. If it does, the fabric cannot keep up. This is why tensor parallelism is deliberately kept intra-node.

Pipeline parallelism -- A model is split by layers. GPU 1 handles layers 1-10, GPU 2 handles layers 11-20, and so on. Data flows sequentially through the pipeline -- GPU 1 sends its output to GPU 2, which processes it and sends its output to GPU 3. The external network traffic is sequential point-to-point rather than all-to-all. Traffic volumes are lower than data parallelism but the latency of each hop in the chain matters -- a slow link between GPU 2 and GPU 3 stalls the entire pipeline.

Model parallelism (general) -- A term sometimes used to mean pipeline parallelism, or a combination of pipeline and tensor parallelism. Large-scale training jobs like GPT-4 or Llama 3 typically combine all four approaches simultaneously: tensor parallelism within a node (NVLink), data parallelism across nodes (external fabric, AllReduce), and pipeline parallelism across groups of nodes (external fabric, point-to-point). The fabric must handle all these traffic patterns simultaneously.

What this means for the external network: The external compute fabric primarily carries data parallelism AllReduce traffic and pipeline parallelism point-to-point traffic. Tensor parallelism stays on NVLink. A congestion event on the external fabric during AllReduce is more damaging than during pipeline communication -- AllReduce is a synchronisation barrier, so any stall holds every GPU in the job. A pipeline stall holds the pipeline stage but downstream GPUs can continue processing their current micro-batch.