GPU clusters are not like general-purpose data centres. The networking stack was designed from the ground up to eliminate latency — and in doing so, it eliminated several security properties that engineers in traditional environments take for granted.
The most consequential of these is this: RDMA completely bypasses the Linux kernel network stack. Every iptables rule, every nftables chain, every netfilter hook — none of them see RDMA traffic. A correctly established RDMA Queue Pair sends data directly from a user-space memory buffer, through the ConnectX-7 NIC ASIC, onto the wire, and into the remote node's memory — without a single kernel system call on the data path. This is intentional. It is why RDMA achieves sub-microsecond latency. It is also why your firewall does nothing to protect it.
The RKEY exposure threat
Every RDMA Memory Region (MR) has an RKEY — a 32-bit token that the remote sender must present to read from or write to that region. If a remote node knows a valid RKEY for a Memory Region on another node, it can perform RDMA reads and writes to GPU memory without the target node's OS being involved at all. The OS never sees the traffic. No process on the target ever gets a syscall. The data moves directly from NIC to memory.
In a multi-tenant GPU cluster, this is a category-1 security failure. Tenant A's training gradient tensors live in GPU memory backed by RDMA MRs. If Tenant B obtains a valid RKEY — by guessing (more on entropy below), by a misconfigured permission, or by a fabric isolation failure — they can read those tensors. The model weights, the training checkpoint, the proprietary dataset: all readable at network speeds without any exploit of the OS.
The multi-tenant incast threat
A second threat vector is operational rather than data-exfiltration: a misbehaving or malicious tenant can trigger a denial-of-service against a co-located tenant through incast. If Tenant B's nodes simultaneously send large RDMA writes to a single destination, the switch buffer fills. PFC pause frames propagate. If Tenant A is on the same lossless priority class (TC3, DSCP 26), Tenant A's NCCL All-Reduce stalls. Training throughput collapses. The attack requires no exploitation — just misconfigured QoS isolation or a deliberate traffic burst.
The three enforcement points
Given that iptables is irrelevant, the actual security enforcement in an RDMA fabric comes from three distinct mechanisms, each operating at a different layer:
| Layer | Mechanism | What it prevents |
|---|---|---|
| NIC ASIC | GID filtering (ROCE_ADDR_FILTER) | Unauthorized QP establishment from remote GIDs |
| Fabric / SM | PKey partitioning | Cross-partition RDMA (IB fabrics) |
| Application | MR access flags + RKEY entropy | Unauthorized reads/writes even if QP is up |
All three must be deployed simultaneously. Any single mechanism alone is insufficient. A NIC with GID filtering enabled can still be vulnerable if RKEY entropy is low. PKey partitioning in InfiniBand is irrelevant if GID filtering on RoCEv2 is disabled.