Skip to content

Liquid Cooling and Heat Rejection · Part 4 of 7

Act 4 — What This Costs the Network Team

Building the model…

The same cluster with all four networks drawn: compute rails, frontend, storage and out-of-band, each with its own switches and its own failure domain.

It is tempting to file cooling under "facilities" and move on. Four consequences land squarely on the people who own the fabric, and all four show up during bring-up rather than in the design review.

Optics live in the residual air stream. An 800G OSFP transceiver dissipates on the order of 15–20 W and specifies a case temperature limit — commonly 70 °C for commercial-temperature parts, higher for extended-range SKUs. In a liquid-cooled rack the surrounding air is warmer and moving more slowly than it would be in an air-cooled one, because most of the fan energy went away with the heat. Switch selection therefore acquires a thermal dimension: check the switch's airflow direction against the rack's, confirm the transceiver SKU's temperature class, and read module temperature during burn-in rather than discovering it during a two-hour all-to-all. A marginal optic that passes at idle and errors under load is a cooling finding presented as a link-quality finding.

Manifolds and hoses compete with cabling for the same rear space. The vertical manifold occupies one side of the rack's rear aperture and the tray hoses cross the other. Fibre bend radius, cassette placement, and the service loop you were planning at the rear of the switch all have to be drawn against the hydraulic layout, not after it. Retrofitting a cable path around an installed manifold is the kind of mistake that costs a week and a lot of re-labelling.

Thermal events masquerade as fabric events. When a coolant filter loads up and flow to a rack falls, GPUs in that rack clock down. Collectives synchronise, so the whole job slows to match. The first evidence a fabric engineer sees is a straggler node in an NCCL profile, or a job whose step time drifted 8 % overnight with no configuration change. The habit worth building is to pull DCGM thermal and power fields alongside fabric counters in the same query, so "is this the network?" gets answered in one step:

# Correlate the straggler with its thermal state before blaming the fabric
$ dcgmi dmon -e 150,155,203,1002 -c 5 -i 0,1,2,3,4,5,6,7
# 150 GPU temp (C) | 155 power (W) | 203 SM clock | 1002 SM occupancy

$ nvidia-smi --query-gpu=index,clocks_throttle_reasons.active,temperature.gpu \
      --format=csv
index, clocks_throttle_reasons.active, temperature.gpu
0, 0x0000000000000000, 61
5, 0x0000000000000008, 88      # SW_THERMAL_SLOWDOWN -- this is a cooling fault

# And the loop's own view for the same rack
cdu-a1# show rack r14 flow
Manifold flow  118.7 L/min  (design 170)   <-- restricted
Filter dP       0.47 bar     (replace >0.45)

Maintenance windows change shape. A node that has to come out of a liquid-cooled rack has to be drained and disconnected, and the UQD pair has to be capped. The work is routine but it is not the thirty-second sled pull that air-cooled operations trained everyone to expect. Fabric maintenance that assumes a node can be swapped inside a job's checkpoint interval needs re-costing.


Read aloudPage 4