PKey (Partition Key) is InfiniBand's native multi-tenancy mechanism. Unlike Spectrum-X's software-defined VRFs and GBP, PKey enforcement is built directly into every IB HCA's ASIC. It cannot be bypassed by the host OS, by the RDMA application, or by any software on the end node. Enforcement is at the hardware level, period.
PKey anatomy
A PKey is a 16-bit value. The MSB (bit 15) encodes membership type:
- MSB = 1 (Full membership): This node can communicate with all other Full and Limited members in this partition. Example: 0x8001 (partition 1, full membership).
- MSB = 0 (Limited membership): This node can communicate with Full members in this partition but not with other Limited members. Example: 0x0001 (partition 1, limited membership).
Two nodes can communicate via RDMA only if they share at least one PKey partition in which both hold at least one Full membership between them (one can be Limited if the other is Full, but both cannot be Limited).
PKey 0x8001 = partition 1, FULL membership (bit 15 set)
PKey 0x0001 = partition 1, LIMITED membership (bit 15 clear)
PKey 0xFFFF = default management partition, FULL (every port member by default)
PKey 0x7FFF = default management partition, LIMITED
Why the management PKey must be segregated
Every IB port is, by default, a member of the management partition (PKey 0xFFFF). This partition exists so that the Subnet Manager can communicate with every port for fabric management. If tenant compute nodes remain in the management partition, they can send SMPs (Subnet Management Packets) to the SM — potentially querying or even modifying routing tables and PKey assignments. This is a critical security boundary violation.
The correct practice is to assign tenant nodes exclusively to custom partitions (0x8001, 0x8002, etc.) and remove them from the management partition. The SM itself and UFM servers remain in 0xFFFF.
# OpenSM partitions.conf — define tenant partitions:
# Format: partition_name : rate=<rate>, mtu=<mtu> ;
# full = full membership
# limited = limited membership
PartitionConfig {
mgmt = ipoib, rate=NDR, mtu=4096, scope=0x00 ;
sm-01 = full ;
ufm-server = full ;
TenantA = ipoib, rate=NDR, mtu=4096, scope=0x00 ;
0x506b4b0300a1b200 = full ; # tenantA-node-01 GUID
0x506b4b0300a1b201 = full ; # tenantA-node-02 GUID
0x506b4b0300a1b210 = limited ;# storage-node GUID (limited = can serve but not initiate)
TenantB = ipoib, rate=NDR, mtu=4096, scope=0x00 ;
0x506b4b0300a1b202 = full ; # tenantB-node-01 GUID
0x506b4b0300a1b203 = full ; # tenantB-node-02 GUID
0x506b4b0300a1b210 = limited ;# storage-node GUID (shared storage)
}
SM PKey propagation
After the OpenSM reads this configuration, it pushes PKey tables to every HCA port via SMP (Subnet Management Packets). The HCA stores the PKey table in its hardware registers. From that point, any RDMA operation that does not present a matching PKey is rejected by the remote HCA before the packet is even delivered to the application.
# Verify PKey table on a node (read from hardware via SM):
smpquery pkeys <lid> <port>
# Example: smpquery pkeys 5 1
# Output:
# PKeys table for LID 5 port 1:
# Pkey[0] 0xffff <- management partition (full)
# Pkey[1] 0x8001 <- TenantA partition (full)
# Pkey[2] 0x0000 <- empty slot
# ...
# Verify connectivity across PKey boundary (should fail):
ibv_rc_pingpong -d mlx5_0 -g 1 # on tenantA-node, listen
ibv_rc_pingpong -d mlx5_0 -g 1 <tenantA-LID> # on tenantB-node, connect
# Expected: IBV_WC_REM_ACCESS_ERR or connection timeout
# Reason: tenantB has PKey 0x8002, tenantA has 0x8001 — no shared partition
# Verify connectivity within same partition (should succeed):
ibv_rc_pingpong -d mlx5_0 -g 1 <tenantA-node-02-LID> # from tenantA-node-01
# Expected: 1000 iterations, ~0.9µs latency
| SM Node | UFM Server | TenantA-01 | TenantA-02 | TenantB-01 | TenantB-02 | Storage Node | |
| SM Node | — | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| UFM Server | ✓ | — | ✗ | ✗ | ✗ | ✗ | ✗ |
| TenantA | ✗ | ✗ | — | ✓ | ✗ | ✗ | ✓ |
| TenantA | ✗ | ✗ | ✓ | — | ✗ | ✗ | ✓ |
| TenantB | ✗ | ✗ | ✗ | ✗ | — | ✓ | ✓ |
| TenantB | ✗ | ✗ | ✗ | ✗ | ✓ | — | ✓ |
| Storage Node | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | — |
PKeys table for TenantA-01 (GUID 0x506b4b0300a1b200): Pkey[0] 0x8001 ← TenantA partition, full membership Pkey[1] 0x0000 ← empty slot Pkey[2] 0x0000 ← empty slot Pkey[3] 0x0000 ← empty slot
The Limited membership storage pattern
A storage node that serves multiple tenants presents a challenge: it must be accessible to both TenantA and TenantB, but TenantA and TenantB must not be reachable to each other through the storage node's partition membership. Limited membership solves this exactly. The storage node holds limited membership in both TenantA and TenantB partitions. A Full member (a tenant GPU node) can initiate RDMA to a Limited member (the storage node). But two Limited members cannot communicate — so TenantA's storage connection and TenantB's storage connection cannot be used as a bridge between the tenants.