A direct-to-chip installation is two hydraulic circuits joined by a heat exchanger. Understanding which circuit a component belongs to is most of what an operator needs, because the two are engineered to different standards and fail in different ways.
The facility water system (FWS) — sometimes called the primary loop — is the building's own chilled or process water. It runs between the plant (chillers, dry coolers, cooling towers, an economiser if the climate allows) and the room. Its water is not clean enough, and its pressure not controlled tightly enough, to be trusted anywhere near a GPU. Facility water is the building's problem; nothing in it should ever touch a cold plate.
The technology cooling system (TCS) — the secondary loop — is the circuit that actually feeds the racks. It is a closed, filtered, chemically treated volume with its own pumps, its own pressure regime, and its own temperature control. Water quality here is a specification, not an aspiration: particulate filtration, corrosion inhibitor, biocide, and an inhibited propylene-glycol mix where freeze protection is needed. Mixing metals carelessly in this loop is how a cluster acquires a galvanic corrosion problem that surfaces eighteen months later as a blocked microchannel.
The coolant distribution unit (CDU) is the machine that separates them. It holds the heat exchanger, the secondary pumps, the filtration, the expansion provision, and the control logic that regulates secondary supply temperature by modulating primary flow. CDUs come in three placements, and the choice shapes the room:
| Placement | Typical capacity | What it implies |
|---|---|---|
| In-rack | 50 – 100 kW | One rack, one CDU. Simplest blast radius, worst floor-space efficiency, and the CDU eats U-space you wanted for compute. |
| In-row / sidecar | 300 kW – 1.5 MW | A cabinet beside the racks it serves. The common choice for NVL72 deployments; redundancy is achieved by pairing units across rows. |
| Perimeter / central | 2 MW+ | One plant room serving a hall. Best efficiency and easiest maintenance, but the secondary loop becomes a shared failure domain and the piping run is a construction project. |
From the CDU, insulated supply and return headers run to each rack, where a rack manifold (a vertical pair of pipes with a quick-disconnect port per tray) splits the flow. Each node or tray attaches with universal quick disconnects (UQDs) — dripless couplings that seal on both halves when parted. Inside the tray, hoses feed cold plates clamped to the GPUs, the CPUs, and on rack-scale systems the NVSwitch ASICs and the voltage regulators as well.
What is still air-cooled, and why it matters. Even in a "liquid-cooled" rack, a fraction of the heat leaves as air. DIMMs, low-power components, drives, and in most designs the optical transceivers are not on cold plates. Reference designs generally assume that 10–25 % of rack load still has to be handled by air, so the room does not get to abandon its CRAH capacity — it gets to shrink it. Sizing the residual air load to zero is a classic first-deployment error, and the symptom is transceiver temperature alarms on a rack whose GPUs are perfectly happy.
# Secondary-loop state as the CDU reports it (vendor CLI/Redfish varies;
# the fields do not)
cdu-a1# show thermal
Secondary supply 32.1 C setpoint 32.0 C
Secondary return 44.6 C delta-T 12.5 K
Secondary flow 168.4 L/min setpoint band 150-190
Secondary pressure 2.8 bar supply / 1.9 bar return
Filter differential 0.21 bar (replace above 0.45)
Primary supply 18.2 C valve position 61%
Pump A / Pump B RUNNING / STANDBY runtime 4128 h / 3902 h
Leak sensors 12 present, 12 dry