There is no universal NCCL multi-rail preset. The right settings depend on the fabric’s rail and switch layout, the HCAs and ports visible to each job rank, Kubernetes’ RDMA resource mapping, GPU-to-NIC topology, and the installed NCCL version. First expose and verify the intended network devices in the job; then constrain NCCL only where its automatic choices do not match the topology.
Separate the bootstrap network from the RDMA data path
NCCL_SOCKET_IFNAME filters IP interfaces used for NCCL’s socket communication and bootstrap-related connections. NCCL_IB_HCA filters the InfiniBand Verbs HCAs and ports available for RDMA. They address different selection problems: choosing an IP interface does not select an RDMA HCA, and selecting an HCA does not provide the process launcher’s connectivity.
As an Amazon Associate I earn from qualifying purchases.
NCCL relies on the application’s process-management and CPU-side communication systems for rank launch and bootstrap coordination; it is not itself a job launcher. Keep the launcher’s connectivity, NCCL socket interface, and high-speed RDMA path distinct when diagnosing a job.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Make the intended devices visible to every rank
Before setting NCCL filters, inventory the network devices on each node and inside the actual job containers. Check which HCAs and ports are present, their active state and link layer, and whether each rank sees the intended set. Kubernetes resource names advertised by a device plugin are not automatically the HCA names that NCCL expects.
#1 Best Overall
- 2.5 Gbps PCIe Network Card: With the 2.5G Base-T Technology, TX201 delivers high-speeds of up to 2.5 Gbps, which is 2.5x faster than typical Gigabit adapters. Performance varies by conditions, distance to devices, and obstacles such as walls
- Versatile Compatibility – The Ethernet Network Adapter is backwards compatible with multiple data rates(2.5 Gbps, 1 Gbps, 100 Mbps Base-T connectivity). The 2.5G Ethernet port automatically negotiates between higher and lower speed connection.
- QoS: Quality of Service technology delivers prioritized performance for gamers and ensures to avoid network congestion for PC gaming
- Wake on LAN – Remotely power on or off your computer with WOL, helps to manage your devices more easily
- Low-Profile and Full-Height Brackets: In addition to the standard bracket, a low-profile bracket is provided for mini tower computer cases
NVIDIA’s Network Operator can manage networking drivers, Kubernetes device plugins, and secondary network components. Its shared-device plugin can map one or more named host interfaces to a Kubernetes resource; separate resources can represent distinct interface groups. Configure the resource mapping for the cluster’s actual interface names and intended sharing or isolation model rather than copying example names. Keep the operator release’s component compatibility in view when changing tested versions.
The exact Kubernetes attachment mechanism—such as a shared RDMA resource, SR-IOV virtual function, host device, or secondary network—depends on the access and isolation model. Regardless of the mechanism, confirm the allocated resource gives the container access to the device and port set that NCCL is meant to use.
Set NCCL_SOCKET_IFNAME only when automatic selection is wrong
NCCL normally selects an IP interface automatically, excluding loopback and Docker interfaces when alternatives exist and favoring interface names beginning with ib. Manually setting NCCL_SOCKET_IFNAME bypasses that automatic selection algorithm, so use it only to select an interface that can communicate with the participating nodes for the required socket path.
Recommended Free Tools
Rank #2
- 10 Gbps PCIe Network Card: With the latest 10GBase-T Technology, TX401 delivers extreme speeds of up to 10 Gbps, which is 10× faster than typical Gigabit adapters, guaranteeing smooth data transmissions for both internet access and local data transmissions[1]
- Versatile Compatibility: With extreme speed and ultra-low latency, 10GBase-T is backwards compatible with multiple data rates (10 Gbps, 5 Gbps, 2.5 Gbps, 1 Gbps, 100 Mbps), automatically negotiating between higher and lower speed connections
- QoS: Quality of Service technology delivers prioritized performance for gamers and ensures to avoid network congestion for PC gaming
- Free CAT6A Ethernet Cable: To maximize TX401's performance, a 1.5 m CAT6A Ethernet Cable is included—rated for up to 10 Gbps while a regular cable is only rated for 1 Gbps
- Low-Profile and Full-Height Brackets: In addition to the standard bracket, a low-profile bracket is provided for mini tower computer cases
The value is a comma-separated list of interface-name prefixes. Use ^ to exclude matching names and = to require an exact name. For example, NCCL_SOCKET_IFNAME=eth0 matches names beginning with eth0, while NCCL_SOCKET_IFNAME==eth0 uses the exact-match marker to select only eth0.
An interface showing UP is not proof that it can reach peers. Validate connectivity on the selected IP path from the participating nodes or pods before forcing NCCL to use it.
Filter RDMA devices with NCCL_IB_HCA
NCCL_IB_HCA accepts comma-separated selectors for HCA, port, rail, and plane. Device names are prefix-matched by default; use the exact-match marker when names must match exactly, since an unmarked mlx5_1 can also match a similarly named device such as mlx5_10.
Rank #3
- The network adapter comes with low-profile bracket and full height bracket.8 cm low-profile bracket suitable for 2U chassis,the 12 cm full height bracket suitable for 3U common chassis
- PCl Express PCle v1.1(2.5GT/s)X1,easily compatible with slot PCI-E X1,X2,X4,X8,X16 ,pay attention:isn't compatible with PCI slot.
- I/O virtualization (IOV) support for VMware NetQueue and Microsoft VMQ
- Automatic Detection and Correction of Pair Swaps, Pair Skew and Pair Polarity
- Network Operating Systems (NOS) Software Support: Windows* 2000; Windows* Server 2003; Windows* Server 2008; Windows Professional XP* SP3; Windows Vista* SP1; Windows 7; Linux* RHEL 4.6; Linux* Kernel version 2.6.24; Linux* Kernel version 2.4.36.2; RHEL* 5.1; SLES* 9 SP4; SLES* 10 SP1; FreeBSD* 7.0; DOS*; DOSODI*; SCO OpenServer 6/Unixware* 7.1.x; Novell Netware* 6.5; Xen*; FreeBSD* 5.x or later; ESX* 3.x* support (for VMware).
For example, NCCL_IB_HCA==mlx5_0:1,=mlx5_1:1 selects port 1 on exactly those two HCA names. To assign rail and plane identities as well, NVIDIA documents the form NCCL_IB_HCA==mlx5_0:1:0:0,=mlx5_1:1:0:1: both selectors specify port 1 and rail 0, with plane IDs 0 and 1 respectively. Keep an empty port field if specifying rail or plane without constraining the port; omitted rail or plane fields remain unassigned.
These are syntax examples, not portable cluster presets. HCA names and port layouts can differ by node, and host names, container visibility, and Kubernetes resource names can differ from the values NCCL sees. Verify the device names from the job’s own environment before applying a filter.
Choose cross-NIC behavior from the physical fabric
NCCL_CROSS_NIC controls whether the same ring or tree should use the same NIC across nodes. NVIDIA documents these policy values in its NCCL 2.32.3 environment guide, accessed in 2026:
Rank #4
- PCI-Express 3.0 16x Riser Card: Install a full-sized PCI Express card in a 1U server case, eliminating the expense of purchasing small form factor PCI-e cards.
- PCI-Express 4.0 16x Riser Card: Install a full-sized PCI Express card in a 1U or 2U server case, eliminating the expense of purchasing small form factor PCIe cards.
- It is the right angle riser for the PCI Express X16 buses. The connector is soldered on the component side (B side) of the board.
- When an I/O board is inserted, the component side of the I/O board will face down, towards the motherboard.
- Golden finger protection cover and dustproof design. The PCI-Express 16X Riser Card makes the PCI-Express Card away from motherboard.
| Value | Policy | Topology guidance |
|---|---|---|
0 |
Keep a ring or tree on the same NIC across nodes. | Use when NICs connect to separate per-NIC rails or switches and inter-rail communication is slow. |
1 |
Allow different NICs across nodes. | Suitable when the nodes’ NICs connect to a common switch. |
2 |
Documented default: prefer the same NIC but allow another if NCCL considers it better. | A compromise when neither strict same-NIC nor unrestricted crossing is clearly appropriate. |
The setting has no effect on a one-NIC system. A communicator with non-identical GPU sets on each node may still require cross-NIC communication. The policy descriptions do not guarantee a speedup: compare job behavior and application throughput on the real workload and fabric.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Use automatic rail assignment only on a supported topology
NCCL_IB_RAIL_POLICY is documented as available since NCCL 2.30.5. NVIDIA describes CX9 for aarch64 systems with CX9 HCAs under a reference architecture, with CX9:FLIP, CX9:ALT, and CX9:BLOCK representing particular per-socket layouts. NONE disables automatic rail and plane assignment.
Automatic assignment does not overwrite rail or plane values explicitly set through NCCL_IB_HCA. The documentation also describes combining the policy with NCCL_NET_MERGE_POLICY=RAIL to merge ports on the same automatically detected rail. Check the installed NCCL release, CPU architecture, HCA model, and actual topology before using these options; the CX9 policy is not a general multi-rail setting for arbitrary clusters.
Verify the GPU-to-NIC path before expecting GPUDirect RDMA
GPUDirect RDMA depends on the deployed GPU, driver, kernel, CUDA, and network-driver combination. NVIDIA GPU Operator guidance recommends DMA-BUF over legacy nvidia-peermem and lists DMA-BUF requirements including an open GPU kernel module, CUDA 11.7 or later, Linux kernel 5.12 or later, and a Turing-class or newer GPU in the documented categories. These are DMA-BUF prerequisites, not universal NCCL requirements; requirements for network drivers differ between DMA-BUF and the legacy module.
Do not force NCCL_NET_GDR_LEVEL, NCCL_NET_GDR_READ, or another GPUDirect setting without validating the platform’s GPU-to-NIC topology and software support. NCCL documents GDR level as the maximum GPU-to-NIC distance and can choose a value based on the architecture and environment when it is not set. There is no safe universal value for unspecified hardware.
Quick Recap
Diagnose the network before changing NCCL tuning
- Confirm device allocation. Check that the expected host NICs and HCAs exist, the relevant ports are active, and the Kubernetes resource is advertised and allocated to the job.
- Test peer reachability. Validate communication over the chosen IP interface from each participating node or pod. An interface being UP alone does not establish peer reachability.
- Inspect RDMA link state. Use
ibstatusoribstatto check active state, physical link, InfiniBand versus Ethernet/RoCE link layer, and expected rate. - Isolate basic bandwidth. Run
ib_write_bwbetween two nodes to determine whether the basic RDMA path works apart from NCCL collective behavior. - Check TCP and firewall paths. Review NCCL’s TCP connection path and local firewall policy. If policy requires restricting Linux ephemeral TCP ports, choose an allowed range that fits local operations rather than copying an example range without validation.
- Use diagnostics temporarily. Enable NCCL diagnostic settings to investigate, then remove debug and workaround settings after resolving the cause. NVIDIA warns that keeping debugging variables set can cause suboptimal behavior, crashes, or hangs.
What a sound configuration decision depends on
- Fabric: whether rails are separated by slow inter-rail links or NICs attach to a common switch.
- Resource visibility: whether every rank sees the intended HCA and port set, rather than a partial or mismatched set.
- Kubernetes network mode: which device access and isolation model the workload uses.
- GPU/NIC support: whether the hardware and driver stack support DMA-BUF or the legacy peer-memory path.
- Validation: whether peer reachability and basic RDMA bandwidth work before NCCL behavior and application throughput are assessed.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →
