Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteA slow GPU all-reduce is a symptom, not a diagnosis. The delay may come from GPU-to-GPU connectivity, PCIe or NVLink topology, GPU-to-NIC transfers, the inter-node fabric, collective synchronization, or host and application scheduling.
The reliable approach is to measure each layer separately, compare it with a known-good result on the same hardware and topology, inspect the path NCCL actually selected, and then trace the application timeline. Change NCCL settings only after those checks show that the underlying paths and software are healthy.
As an Amazon Associate I earn from qualifying purchases.
Why is my GPU all-reduce slow?
All-reduce combines values from every rank and returns the result to every rank. Its time therefore depends on more than a link’s advertised bandwidth. GPU placement, message size, collective algorithm, rank synchronization, concurrent computation, CPU launch work, and the number and placement of network rails all affect the elapsed time.
A useful diagnosis identifies where the time is spent and which path was exercised. A pairwise GPU test does not validate an eight-GPU collective; an InfiniBand bandwidth test does not prove that NCCL is using the intended interface; and a completed diagnostic does not establish that performance is acceptable. NVIDIA’s DCGM documentation states: “Completion without a reported workload error is not, by itself, a performance qualification.”
#1 Best Overall
- The 25Gb dual-port SFP+ network card is based on the Mellanox ConnectX-5 Ex controller, which provide the highest performing and most flexible interconnect solution.
- Technical Support:PXE、 RDMA、UEFI、SR-IOV、1588 PTP、Jumbo Frames（9.5KB）
- Windows 10/11、Windows Server 2016/2019/2022、Deepin 15.11/20/20.6/20.9、VMware ESXi 6.5/6.7、Ubuntu 18.04.5/20.04.1、Ubuntu 22.04.2/22.04.3、RHEL/CentOS 7.6/7.9/8.2/8.3、ZTE New Fulcrum 3.2.2/5.0.5、SUSE 12.5/15.4、FreeBSD 13.2、NeoKylin 7.6、OpenKylin 0.7.5、Mikrotik、iKuai route、Galaxy Kylin v10、Zhongke Fangde desktop OS、Zhongke Fangde server OS、Tongxin UOS 20、Emind OS
- install the operating system with its driver CD, or download it from the official website. Includes low-profile and full-height stands to support standard and ultra-thin computers/servers.
- Enjoy 24/7 customer service, 30-day free returns, 1-year free warranty, and lifetime technical support for your peace of mind.
Do not compare an application run with an unrelated benchmark or treat a theoretical link rate as guaranteed all-reduce throughput. Build a baseline for the same GPUs, topology, rank mapping, message sizes and launch configuration.
Start with a reproducible baseline
Before changing a setting, capture enough information to reproduce the slowdown. Store the output with the benchmark or job record.
- NCCL, CUDA, GPU driver, framework and network-software versions.
- GPU model and count, NIC model and count, node count, and the physical hosts involved.
- Rank-to-GPU and rank-to-node mapping, including which ranks share each node.
- Message-size range, number of iterations, warm-up behavior, and whether communication overlaps computation.
- Launcher, environment variables, CPU affinity and any container or scheduler configuration.
- The same test on a known-good node pair or an earlier run from the same system.
Repeat the slow case without changing hardware or software. Record both the median behavior and obvious outliers such as one node pair or one rail that is consistently slower. NVIDIA recommends testing network performance before production and retaining a baseline because achievable performance varies with topology.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteMeasure the collective before blaming a layer
Run a matching NCCL collective test
Use the NCCL test suite’s all_reduce_perf with the relevant number of GPUs, ranks and nodes. Match the application’s message-size range and launch geometry as closely as possible. Run several message sizes rather than a single point:
- Small messages are usually more sensitive to latency, kernel-launch overhead and synchronization.
- Large messages expose sustained bandwidth limits and contention more clearly.
- A curve that changes slope can indicate a protocol or path transition rather than one fixed hardware limit.
Keep the benchmark’s node list, rank placement and environment identical between the slow and known-good runs. If only a particular node pair is slow, repeat that pair against different peers. A problem that follows one node points toward that node; a problem that follows a pair or rail points toward the interconnect path.
Decide whether the application or the collective is at fault
If an equivalent NCCL benchmark is healthy while the application remains slow, do not keep tuning transport variables. The application may be waiting for a late rank, exposing communication that was previously hidden behind computation, or spending time in CPU, storage or scheduling work before the collective starts. Capture a full application timeline and compare rank arrival times at the collective.
Inspect intra-node GPU paths
Read the topology NCCL can see
On each node, run:
nvidia-smi topo -m
nvidia-smi topo -p2p p
nvidia-smi topo -p2p ntopo -m shows the relative placement of GPUs, CPUs and NICs. The P2P queries report PCIe and NVLink peer-to-peer status. Use these outputs to verify that the rank mapping matches the intended fast paths and that a supposedly connected pair is not crossing an unexpected PCIe root complex or socket.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
A positive P2P display is useful evidence, but it is not a complete health check. NVIDIA documents Linux bare-metal IOMMU translation and PCIe Access Control Services as possible causes of degraded or unavailable P2P and GPU Direct behavior. Inspect the platform’s PCIe layout and operating-system configuration before changing firmware or kernel settings; those changes can affect isolation, device assignment and other production requirements.
Rank #2
- Host Interface: PCI Express 5.0 x16
- Total Number of Ports: 1
- Expansion Slot Type: OSFP
- Media Type Supported: Optical Fiber
- Maximum Data Transfer Rate: 400 Gbit/s
Measure actual GPU-to-GPU bandwidth
Use nvbandwidth to measure pairwise GPU transfers. Compare its bandwidth matrix with the topology output and with a known-good node. Look for asymmetry: one direction or one pair can be slow even when other pairs are normal.
NCCL’s active diagnostics can test the P2P links it expects to use. A failed check identifies affected GPU pairs and paths. A passing check makes a local P2P path less likely as the primary cause, so move to GPU-to-NIC, inter-node or scheduling checks rather than repeatedly rerunning the same test.
Check GPU-to-NIC and inter-node communication
Verify the GPU Direct RDMA path
For GPU Direct RDMA, confirm that the NIC, driver and kernel support the path used by your deployment. NVIDIA documents nvidia-peermem as one supported module path; current Linux configurations may instead use DMA-BUF. Which mechanism is valid depends on the driver, kernel and distribution, so verify the combination installed on every node.
Recommended Free Tools
Do not assume NCCL chose the NIC or transport you intended. Enable NCCL diagnostics and inspect the log for the selected network interface, transport and physical device. Compare logs across ranks and nodes; an inconsistent interface selection can produce a collective that works but is limited by one slow path.
Test the fabric without NCCL
Measure the network independently with the verbs benchmarks:
ib_write_bw
ib_write_latRun them on the participating hosts with the tool installed and hostnames resolving correctly. Test the same node pairs used by the collective, then test those nodes against different peers. Compare bandwidth, latency and each available rail. A low-level fabric result that is already abnormal points to the NIC, link, switch, routing, driver or configuration; a healthy result shifts attention to GPU Direct integration, rank placement or NCCL’s selected path.
For GPU-aware traffic, use a multi-node nvbandwidth test or a matching nvloom suite when appropriate. Multi-node NVLink deployments have additional domain and service checks in NCCL’s troubleshooting guidance; apply those checks when the hardware actually uses that architecture rather than treating every cluster as an InfiniBand problem.
Use logs and traces to locate the missing time
Read NCCL diagnostics as path evidence
NCCL logs can show which network interface, transport and topology path each rank selected. Capture logs from all ranks for the same iteration range. Differences in rank initialization, interface selection or topology discovery can explain an apparently random slowdown. A log proves what NCCL attempted; it does not prove that the link delivered its expected bandwidth, so pair it with the corresponding low-level measurement.
Trace the application with Nsight Systems
Nsight Systems can place NCCL activity in the context of CUDA kernels, CPU threads and other application work. Capture the complete multi-rank execution when possible. In the timeline, distinguish these patterns:
- Communication itself is long on every rank: investigate the selected path and measured bandwidth or latency.
- One rank enters late: inspect the preceding GPU kernels, CPU work, data loading and synchronization on that rank.
- NCCL waits while another rank computes: the apparent communication time may be a scheduling or load-balance problem.
- Communication is hidden in one configuration but exposed in another: compare overlap, stream dependencies and kernel ordering before changing transport settings.
NVIDIA recommends profiling to identify CPU, storage and GPU bottlenecks. An incomplete trace from only one rank can hide the relationship between a late rank and the collective, so preserve the multi-rank context.
Separate bandwidth problems from synchronization failures
A timeout, hang or large rank-to-rank timing difference is not automatically a bandwidth issue. Verify that all ranks call compatible collectives in the same order and that each rank is attached to the intended device. A wrong rank-to-device mapping can create both poor topology use and apparent desynchronization.
For PyTorch distributed jobs, set these variables when investigating detected collective desynchronization:
TORCH_DISTRIBUTED_DEBUG=DETAIL
TORCH_SHOW_CPP_STACKTRACES=1Use the resulting diagnostics with the NCCL logs and profiler timeline. They can reveal mismatched collective order or the rank that failed to participate, issues that a bandwidth benchmark cannot identify.
Choose a diagnostic that matches the failure pattern
| Observed question | Diagnostic | What it measures | What it cannot prove |
|---|---|---|---|
| Does computation plus communication reproduce the slowdown? | mnubergemm |
A combined compute-and-communication workload | It does not isolate a particular transfer path |
| Is one GPU pair or direction abnormal? | mnnvbandwidth |
Directed GPU reads and writes, producing a bandwidth matrix | It may not reproduce contention from the full application |
| Does a particular many-GPU traffic pattern fail? | A matching nvloom suite |
The selected many-GPU traffic shape | A passing run does not establish an acceptable threshold |
| Is the inter-node fabric itself slow? | ib_write_bw and ib_write_lat |
Host-to-host fabric bandwidth and latency | They do not validate NCCL’s GPU path or rank scheduling |
| Is the application spending time outside the collective? | Nsight Systems trace | NCCL, CUDA, CPU and surrounding application timeline | A partial rank capture can hide collective relationships |
DCGM’s guidance emphasizes comparing measurements with a known-good system or a site requirement. The tools do not supply a universal bandwidth threshold, and successful completion alone is not performance qualification.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Compare results on the same axes
When two runs differ, record the dimensions that make the comparison valid:
Free tools Windows power users keep installed
One-click scans. No signup required.
| Axis | Questions to answer |
|---|---|
| Path | Was traffic using NVLink, PCIe P2P, GPU-to-NIC, InfiniBand or RoCE, or multi-node NVLink? |
| Scope | Was the measurement pairwise, a node pair, one rail, the full collective or the full application? |
| Traffic shape | Were direction, message size, concurrency and compute overlap the same? |
| Evidence | Do topology, measured bandwidth and latency, NCCL’s selected path and the application trace agree? |
| Repeatability | Does the result follow one node, one pair, one rail, one rank mapping or the workload itself? |
No universal all-reduce number is meaningful without the hardware, topology, message size, algorithm and workload context. A peak link specification is not an expected collective result.
Rank #4
- DUAL-PROTOCOL 100G: ConnectX-4 VPI (MCX456A-ECAT) runs EDR InfiniBand 100Gb/s or 100GbE per QSFP28 port with 100G/50G/40G/25G/10G auto-negotiation — one card serves IB and Ethernet fabrics.
- PCIe 3.0 x16, FULL BANDWIDTH: Dual ports sustain line-rate 100Gb/s each for HPC, AI training nodes and high-throughput storage fabrics.
- RDMA WITHOUT CPU COPIES: Native InfiniBand RDMA plus RoCE accelerate MPI, NVMe-oF and distributed storage; hardware offloads cut latency and free CPU cycles.
- HEAVY VIRTUALIZATION: SR-IOV with up to 127 VFs per port (254 per card) plus VXLAN/GENEVE/NVGRE overlay offload for multi-tenant clouds and dense VM hosts.
- DATA CENTER FEATURES: PXE/UEFI boot, NC-SI management, DCB, jumbo frames; Linux (MLNX_OFED), Windows (WinOF) and VMware ESXi support; brackets for any chassis.
Common failure patterns and the next check
Only one node pair is slow
Retest that pair against different peers and test the same peers against a healthy node. Inspect NIC and switch counters, rail selection, cabling and routing, then compare ib_write_bw and ib_write_lat. Keep the node software and firmware inventory beside the results.
All pairs on one node are slow
Inspect nvidia-smi topo -m, P2P status, PCIe placement, IOMMU and ACS configuration, GPU drivers and the local NIC path. Use nvbandwidth to determine whether the issue is confined to GPU pairs or appears only when traffic leaves the node.
Large messages plateau below the baseline
Check measured GPU-to-GPU and fabric bandwidth, NCCL’s selected interface and rail count, and GPU Direct RDMA support. If the low-level tests are healthy, compare rank placement, topology discovery and software configuration across all ranks.
Small messages are disproportionately slow
Focus on latency, launch and synchronization overhead. Use ib_write_lat, inspect CPU scheduling and rank arrival times in Nsight Systems, and verify that the benchmark and application use the same number of ranks and the same message-size range.
The benchmark is healthy but the application is slow
Profile the application rather than changing NCCL variables. Look for late ranks, host-side preparation, storage waits, stream dependencies and lost compute/communication overlap.
A diagnostic passes but production performance is poor
Confirm that the diagnostic exercised the same path, direction, concurrency and message size as production. A pairwise test can pass while a many-GPU traffic pattern contends for a shared bridge, NIC or rail.
Tune NCCL only after the path is healthy
NCCL exposes tuning variables such as CTA counts and buffer sizes. These settings can trade GPU memory for throughput and can behave differently on PCIe and other topologies. NCCL is tuned across many systems, so a change that improves one microbenchmark can regress another workload.
- Save the baseline environment, benchmark output and application trace.
- Change one variable at a time, leaving topology, rank mapping and message sizes unchanged.
- Repeat the same NCCL collective test and the actual application workload.
- Check both throughput and latency, plus GPU memory use and rank balance.
- Keep a change only if the improvement is repeatable on the production traffic pattern and does not create a regression elsewhere.
Do not use tuning to compensate for a missing P2P path, a misselected NIC, a slow rail or a desynchronized application. Fix the underlying layer first.
A compact investigation checklist
- Record software versions, hardware inventory, topology, rank mapping and message sizes.
- Reproduce the slowdown and save a known-good comparison.
- Run
all_reduce_perfwith matching ranks, nodes and message sizes. - Inspect
nvidia-smi topo -mand both P2P queries on every node. - Measure GPU pairs with
nvbandwidthand use NCCL’s active P2P diagnostics. - Verify GPU Direct RDMA support, then inspect NCCL’s selected interface and transport.
- Run
ib_write_bwandib_write_latacross the affected node pairs and rails. - Use multi-node
nvbandwidthor a matchingnvloomtest for the actual traffic shape. - Capture a complete Nsight Systems timeline if the application and collective results disagree.
- Enable PyTorch distributed debug variables for suspected collective desynchronization.
- Only after these checks, test one NCCL tuning variable at a time.
What a defensible diagnosis looks like
A strong incident report ties the symptom to evidence: the exact workload and message size, the affected nodes or ranks, the topology and NCCL-selected path, independent GPU and fabric measurements, and the timeline showing where time accumulated. It should also state what was ruled out. For example, healthy pairwise P2P and fabric tests with a late rank in the application trace support a scheduling or load-balance diagnosis, not a claim that the network is slow.
Keep the baseline with the cluster’s version and hardware records. Driver, kernel, NCCL and hardware compatibility can change, so recheck commands and supported mechanisms against the versions installed in the environment before applying a remediation.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →
