Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Reduce multi-GPU training stalls by measuring the communication path in layers: first GPU-to-GPU links inside each server, then node-to-node bandwidth and latency, then NCCL collectives under the real job. Fix the layer that fails before changing NCCL variables. This prevents a PCIe, NIC, routing, or congestion fault from being misdiagnosed as a software-tuning problem.
What a network bottleneck looks like in distributed training
In data-parallel training, GPUs repeatedly exchange gradients or activations. A slow rank can hold an entire collective, so average GPU utilization may fall even when the model’s arithmetic is efficient. The apparent “network problem” can originate in several places:
As an Amazon Associate I earn from qualifying purchases.
- The local GPU path (PCIe, NVLink, IOMMU, or ACS).
- The GPU-to-NIC path and GPUDirect RDMA support.
- The inter-node fabric, including link rate, routing, errors, and congestion.
- CPU, NUMA, container, or virtual-machine placement.
- NCCL topology selection, transport settings, or collective/message-size behavior.
Do not infer a root cause from one peak-bandwidth number. Record the GPU model and topology, NIC or HCA model and negotiated rate, fabric type, driver and NCCL versions, number of nodes and ranks, rank placement, collective type, and message sizes with every result. Compare identical conditions: host-memory and GPU-memory tests, the same number of processes, and the same message sizes.
A layered diagnostic workflow
| Layer | What to measure | Failure clues | Next action |
|---|---|---|---|
| Intra-node GPU path | Topology, peer access, GPU-pair bandwidth | Unexpectedly low pair throughput or missing peer access | Inspect PCIe/NVLink wiring, IOMMU, ACS, and GPU placement |
| GPU-to-NIC path | Host-buffer versus GPU-buffer RDMA throughput | GPU-buffer result is lower, or GPUDirect is unavailable | Validate adapter, driver, kernel module, DMA-BUF or nvidia-peermem, and PCIe locality |
| Inter-node fabric | Link state/rate, bandwidth, latency, counters | Downshifted rate, retries, drops, pause/PFC, ECN/CNP, or latency tails | Investigate cabling, switch ports, routing, and congestion control |
| NCCL and workload | Collective completion time, NCCL logs, message-size scaling | Lower-level tests meet expectations but training collectives remain slow | Check topology discovery, rank/NUMA affinity, transport choice, and one NCCL setting at a time |
NVIDIA’s performance-tuning guidance follows this order: establish that the lower-level GPU and fabric paths meet the installed system’s design expectations before attributing a problem to NCCL configuration.
#1 Best Overall
- The 25Gb dual-port SFP+ network card is based on the Mellanox ConnectX-5 Ex controller, which provide the highest performing and most flexible interconnect solution.
- Technical Support:PXE、 RDMA、UEFI、SR-IOV、1588 PTP、Jumbo Frames（9.5KB）
- Windows 10/11、Windows Server 2016/2019/2022、Deepin 15.11/20/20.6/20.9、VMware ESXi 6.5/6.7、Ubuntu 18.04.5/20.04.1、Ubuntu 22.04.2/22.04.3、RHEL/CentOS 7.6/7.9/8.2/8.3、ZTE New Fulcrum 3.2.2/5.0.5、SUSE 12.5/15.4、FreeBSD 13.2、NeoKylin 7.6、OpenKylin 0.7.5、Mikrotik、iKuai route、Galaxy Kylin v10、Zhongke Fangde desktop OS、Zhongke Fangde server OS、Tongxin UOS 20、Emind OS
- install the operating system with its driver CD, or download it from the official website. Includes low-profile and full-height stands to support standard and ultra-thin computers/servers.
- Enjoy 24/7 customer service, 30-day free returns, 1-year free warranty, and lifetime technical support for your peace of mind.
1. Establish the local GPU path
Map GPUs, NICs, CPUs, and PCIe devices
Run the topology command on every node, not just the head node:
nvidia-smi topo -mThe output shows how each GPU connects to CPUs, PCIe root complexes, and network adapters. A rank may perform poorly when its GPU is attached to a different socket or root complex from the NIC it uses, even though the adapter itself has ample bandwidth. Compare the output across otherwise identical servers; an outlier often identifies a cabling, slot, BIOS, or placement difference.
Check peer-to-peer capability
nvidia-smi topo -p2p p
nvidia-smi topo -p2p nThese checks report peer-to-peer capability for PCIe and NVLink paths. They are capability indicators, not a performance measurement. Use nvbandwidth to measure actual GPU-pair throughput for the pairs your job uses. Test both directions where the tool supports it and retain the command output with the topology.
Free tools Windows power users keep installed
One-click scans. No signup required.
Investigate IOMMU and ACS only when evidence points there
NVIDIA documents bare-metal IOMMU and PCI Access Control Services (ACS) as possible causes of failed or degraded PCIe peer communication. For affected Linux bare-metal PCIe configurations, translated IOMMU mode is unsupported for CUDA/NVIDIA-driver PCIe peer-to-peer memory transfer. That does not mean every system should disable IOMMU or ACS. These are platform-wide controls that affect isolation and host behavior; consult the server, kernel, and virtualization design and validate the change on a maintenance node before applying it to a cluster.
If nvbandwidth is below the expected level for the installed GPU and topology, fix this local path before testing network variables. A fast network cannot compensate for a GPU pair that is already communicating slowly inside the server.
2. Measure inter-node bandwidth and latency separately
Verify InfiniBand port state and negotiated rate
For an InfiniBand fabric, inspect every HCA port involved in the test:
ibstatus
ibstatConfirm that ports are active, the expected link layer is in use, and the negotiated rate matches the design. A downshifted link can reduce throughput without producing an obvious application error. Differences between nodes usually point to a port, cable, transceiver, switch lane, firmware, or configuration issue rather than an NCCL algorithm.
Recommended Free Tools
Run point-to-point bandwidth and latency tests
ib_write_bw
ib_write_latUse the server/client procedure and options appropriate to the installed perftest build. Measure node-to-node throughput with host memory, then test GPU memory when GPUDirect RDMA is configured. GPU-memory flags and syntax vary by build and platform, so check the local help output rather than copying an option blindly:
Rank #2
- Host Interface: PCI Express 5.0 x16
- Total Number of Ports: 1
- Expansion Slot Type: OSFP
- Media Type Supported: Optical Fiber
- Maximum Data Transfer Rate: 400 Gbit/s
ib_write_bw --helpRun latency as well as bandwidth. A link can reach its expected bulk throughput while suffering long-tail latency that periodically delays a collective. Repeat tests during the same traffic conditions in which training stalls; an idle-fabric result does not rule out congestion under production load.
Check InfiniBand fabric health
A Subnet Manager must be running for an InfiniBand fabric. Inspect port counters for symbol errors, link recovery, and link-down events. When access permits, use ibdiagnet for broader topology, routing, and fabric-error checks. Rising retry or error counters, repeated link recovery, or an unexpected route are signals to investigate the physical and transport path before touching NCCL.
Check RoCE counters and congestion signals
RoCE uses Ethernet, so inspect adapter statistics and the switch telemetry available in your design. On each relevant NIC, start with:
ethtool -S <interface>Look for drops, pause frames, Priority Flow Control (PFC) events, Explicit Congestion Notification (ECN) marks, Congestion Notification Packets (CNP), and queue drops. Rising counters alongside unstable throughput or latency tails indicate congestion or lossless-fabric problems. Verify queue priorities, buffer allocation, ECN thresholds, PFC scope, routing, and switch configuration against the actual design. Do not assume that enabling PFC alone makes a fabric lossless or correctly tuned.
These counters are diagnostic clues, not proof of one root cause. Confirm cabling, switch ports, routing, congestion behavior, platform support, and driver configuration as a set. A retry increase can reflect congestion, a faulty link, or a configuration mismatch.
3. Validate the GPU-to-NIC path and process placement
Confirm that GPUDirect RDMA is really in use
GPUDirect RDMA can transfer data between supported GPU memory and a network adapter without the ordinary host-memory staging path. It depends on compatible GPUs, adapters, drivers, kernel support, and a correctly configured platform. NVIDIA documents nvidia-peermem as a module that may need to be loaded. On kernels and drivers that support it, DMA-BUF can provide an alternative mechanism.
Determine which method your installed kernel, driver, adapter, and perftest build support. Then compare host-buffer and GPU-buffer results. If host memory reaches the expected rate but GPU memory does not, inspect the peer-memory path, module or DMA-BUF setup, IOMMU/ACS behavior, and PCIe locality. Do not label a low GPU-buffer result as a network-capacity limit until the data path is verified.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Expose the real topology to NCCL
Containers and virtual machines can present a virtual PCI topology that differs from the physical machine. NVIDIA warns that this can cause NCCL to select suboptimal communication paths. Ensure the runtime exposes the devices and topology information NCCL needs, then review NCCL initialization and topology logs.
Align ranks with GPUs, NICs, and NUMA nodes
On a multi-socket host, bind each process and its host memory near the GPU and NIC it serves. Inspect the host CPU/NUMA map and compare it with nvidia-smi topo -m. An otherwise healthy adapter can appear slow when a rank repeatedly crosses sockets or uses a remote memory controller. Make rank placement deterministic and document it with the benchmark result; changing placement changes the experiment.
4. Use NCCL logs to distinguish configuration from hardware
Enable NCCL debug logging selectively when lower-level tests are healthy but collectives remain slow. Logs can reveal the topology NCCL selected, environment settings, transport decisions, and CPU affinity. Preserve the exact NCCL, CUDA, driver, kernel, GPU, NIC, and job-size context with each log.
A useful decision rule is:
- If GPU-pair bandwidth is low, fix the local GPU path.
- If GPU-pair bandwidth is healthy but
ib_write_bworib_write_latis abnormal, fix the fabric or GPUDirect path. - If both lower-level tests meet expectations but NCCL is slow, examine topology discovery, rank placement, collective type, message sizes, and NCCL settings.
Forcing a particular NIC, transport, or interface can be a valuable diagnostic experiment. Treat it as temporary unless measurements show that the forced choice matches the intended production topology.
5. Tune NCCL only after the baseline is known
Change one variable at a time
NCCL uses topology-aware defaults, and a setting that helps one benchmark can hurt another job. Keep a baseline, change one parameter, rerun the same representative workload, and record both performance and resource effects. A microbenchmark alone is insufficient: the final test should use the model, number of nodes, rank placement, collective mix, and message sizes that matter in production.
Investigate queue pairs on RoCE
On RoCE, multiple queue pairs per connection may be necessary to saturate a fabric. NCCL exposes NCCL_IB_QPS_PER_CONNECTION for investigation. Increasing it can improve utilization in some topologies, but it also changes connection and switch behavior. Test several values against the baseline while watching throughput, latency tails, packet drops, PFC, ECN, and CNP counters.
Account for buffer and CTA trade-offs
NCCL buffer sizes and communication CTA counts alter how communication uses GPU memory and compute resources. More buffering can help overlap or absorb traffic variation, while additional communication CTAs can compete with model kernels. Evaluate step time, communication time, GPU memory headroom, and compute utilization together. Never promote a setting because it raises a communication microbenchmark if end-to-end training slows or memory pressure increases.
6. Match the test to the collective and workload
Data-parallel AllReduce is not the only communication pattern. AllGather, ReduceScatter, and AlltoAll exercise topology, routing, and message sizes differently. A fabric that looks satisfactory for one pattern can expose a bottleneck under another. Capture the collective type and message-size distribution when profiling a job.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Scale tests in the dimensions that change behavior:
Rank #4
- DUAL-PROTOCOL 100G: ConnectX-4 VPI (MCX456A-ECAT) runs EDR InfiniBand 100Gb/s or 100GbE per QSFP28 port with 100G/50G/40G/25G/10G auto-negotiation — one card serves IB and Ethernet fabrics.
- PCIe 3.0 x16, FULL BANDWIDTH: Dual ports sustain line-rate 100Gb/s each for HPC, AI training nodes and high-throughput storage fabrics.
- RDMA WITHOUT CPU COPIES: Native InfiniBand RDMA plus RoCE accelerate MPI, NVMe-oF and distributed storage; hardware offloads cut latency and free CPU cycles.
- HEAVY VIRTUALIZATION: SR-IOV with up to 127 VFs per port (254 per card) plus VXLAN/GENEVE/NVGRE overlay offload for multi-tenant clouds and dense VM hosts.
- DATA CENTER FEATURES: PXE/UEFI boot, NC-SI management, DCB, jumbo frames; Linux (MLNX_OFED), Windows (WinOF) and VMware ESXi support; brackets for any chassis.
- Number of GPUs per node and number of nodes.
- Per-rank message size and gradient-bucket configuration.
- Collective operation and frequency.
- Concurrent jobs sharing links, switches, or queue priorities.
- Rank and NIC placement.
Compare collective completion time and tail behavior, not just aggregate bandwidth. The slowest rank, a congested path, or a synchronization delay can dominate an iteration even when most transfers are fast.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.InfiniBand versus RoCE: how to choose on evidence
Both InfiniBand and RoCE are viable RDMA approaches. The cited NVIDIA guidance provides diagnostic methods for each, while a Meta paper published at ACM SIGCOMM 2024 describes production RoCE clusters supporting thousands of GPUs. That paper demonstrates that RoCE can operate at large scale with deliberate co-design of topology, routing, transport, operations, and collective-library behavior; it is not a controlled universal speed comparison against InfiniBand.
| Decision factor | Questions to answer in your cluster |
|---|---|
| Measured performance | Which fabric delivers the required bandwidth and latency under the actual collective mix and job scale? |
| Congestion behavior | Can the team observe and control drops, pause/PFC, ECN, CNP, queue occupancy, and latency tails? |
| Topology and routing | Does the design provide predictable paths and sufficient bisection bandwidth for the intended placement? |
| Operations | Are staff, monitoring, firmware, switch tooling, and incident procedures available for the chosen fabric? |
| Hardware support | Do the GPUs, adapters, switches, cables, drivers, and kernels support the required GPUDirect and RDMA path? |
| Workload fit | How do AllReduce, AllGather, ReduceScatter, or AlltoAll behave at your message sizes and scale? |
Do not select a fabric from a headline bandwidth figure. Select it from repeatable measurements and the operational ability to keep congestion, errors, and topology under control.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteCommon symptoms and targeted fixes
Training is slow on every node
- Run
nvidia-smi topo -mandnvbandwidthto rule out a shared local-path problem. - Run node-to-node bandwidth and latency tests with the same adapters and message sizes used by the job.
- Check whether every link negotiated at the designed rate.
Only some ranks are slow
- Compare rank-to-GPU-to-NIC mapping and NUMA affinity.
- Compare the affected ports’ error, retry, drop, PFC, ECN, and CNP counters with healthy ports.
- Inspect whether a container or virtual machine exposes a misleading topology.
Average throughput is high but iterations occasionally stall
- Measure latency and its tail, not only bandwidth.
- Sample congestion and queue counters during a stall.
- Check link recovery, retries, pause/PFC storms, ECN/CNP activity, and shared-fabric traffic.
Host-memory RDMA is fast but GPU-memory RDMA is slow
- Verify supported adapter, driver, kernel, and perftest combinations.
- Check
nvidia-peermemor the supported DMA-BUF path. - Revisit PCIe topology, IOMMU, ACS, and GPU/NIC locality.
A tuning change helped a benchmark but hurt training
- Restore the baseline and rerun the representative model.
- Check whether the change altered GPU memory use, CTA competition, or congestion.
- Retest at the production node count and collective/message-size mix before adopting it.
A repeatable change-and-validation checklist
- Record hardware, software versions, topology, link rates, rank placement, collective types, and message sizes.
- Run
nvidia-smi topo -m, both peer-to-peer capability checks, andnvbandwidth. - Verify InfiniBand or Ethernet port state and negotiated rates.
- Measure node-to-node bandwidth and latency with host buffers.
- Measure GPU-buffer RDMA where the platform and perftest build support it.
- Inspect InfiniBand or RoCE counters during both idle and loaded tests.
- Confirm GPUDirect method, module or DMA-BUF support, container visibility, and NUMA placement.
- Run the target NCCL collective mix and preserve selective debug logs.
- Change one NCCL or placement setting, then repeat the same tests.
- Accept a change only when end-to-end training improves without unacceptable memory use, latency tails, errors, or operational risk.
What large-scale evidence does—and does not—show
The Meta SIGCOMM 2024 paper analyzed approximately 30,000 randomly selected training jobs from Meta Platforms’ 2023 Q4 workload dataset. It describes production RoCE clusters supporting thousands of GPUs and discusses designs supporting up to 32,000 GPUs. Those figures characterize Meta’s fleet, workloads, and cited design scale; they are not a performance guarantee for another organization’s hardware or model.
The practical lesson is narrower and more useful: large clusters require co-design of network topology, routing, transport behavior, operations, and collective-library behavior. Apply that principle to your measured system rather than treating the paper’s scale as an expected result.
When hardware replacement is justified
Consider replacing an HCA, switch port, cable, or transceiver only after measurements identify a physical or compatibility fault and the replacement matches the complete design. An adapter upgrade alone will not fix a mismatched switch, unsupported GPUDirect path, poor PCIe slot, wrong cable, or congested topology. Validate the replacement with port state, negotiated rate, error counters, host-memory bandwidth, GPU-memory bandwidth, latency, and the representative NCCL workload.
Do not use a generic consumer network product for a multi-node GPU fabric. The relevant equipment is compatible cluster networking hardware, selected together with its switch ports, cables, drivers, PCIe topology, and fabric configuration.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesFrequently Asked Questions
Why is NCCL slow across nodes when bandwidth tests look normal?
Check latency tails, rank and NUMA placement, the topology visible inside containers or virtual machines, collective type, message sizes, and NCCL’s selected transport. A bulk bandwidth result can be healthy while synchronization is delayed by congestion or an unfavorable path.
How can I tell whether RoCE congestion is slowing training?
Sample NIC and switch telemetry during a slowdown. Rising drops, pause or PFC frames, ECN marks, CNPs, queue drops, and unstable latency or throughput indicate congestion or lossless-fabric issues that should be investigated before NCCL tuning.
Should I increase NCCL_IB_QPS_PER_CONNECTION immediately?
No. First establish healthy GPU, GPUDirect, and fabric measurements. Then change the setting one value at a time on RoCE, watching end-to-end training, latency tails, GPU memory, and congestion counters rather than relying on a microbenchmark.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
