Free tools Windows power users keep installed
One-click scans. No signup required.
Start by reducing the failure to the smallest reproducible path: one GPU, one node, then multiple nodes. Record the exact GPU models, driver and CUDA versions, NCCL and framework versions, container image, kernel, NICs, node placement, GPU/NIC topology, and relevant environment variables before changing anything. A nonzero CUDA or NCCL return code is evidence at the API boundary, not a reason to add broad retries. Classify the symptom, run the diagnostic that matches that class, change one variable at a time, and preserve a known-good configuration.
Build a reliable incident record before changing configuration
Distributed GPU failures are often made harder by configuration drift. Capture the state of every participating rank and host while the job is failing. Store the command line, scheduler allocation, container digest or image tag, and complete stdout and stderr logs.
As an Amazon Associate I earn from qualifying purchases.
- Hardware: GPU model and count, PCI bus addresses, node names, NIC model, link speed, and whether the NIC is intended for InfiniBand, RoCE, or ordinary Ethernet.
- Software: NVIDIA driver, CUDA runtime and toolkit, NCCL, deep-learning framework, operating-system kernel, firmware, OFED or RDMA stack, and container runtime versions.
- Topology: GPU-to-GPU links, PCIe root complexes, NUMA nodes, GPU-to-NIC affinity, rack or switch placement, and whether all ranks see the same topology.
- Runtime settings: NCCL and CUDA environment variables, visible-device masks, network-interface selections, shared-memory settings, and scheduler-provided variables.
- Scope: failing rank, host, peer host, GPU pair, interface, transport, and the first timestamp at which the error appears.
Run a baseline inventory on each host and save the output:
nvidia-smi
nvidia-smi topo -m
nvcc --version
uname -a
ip -br link
Use the same commands inside the workload container. A host may expose a GPU, NIC, or topology that the container cannot actually access.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Classify the symptom before selecting a fix
| Symptom class | Typical evidence | First isolation target |
|---|---|---|
| CUDA runtime or kernel fault | Illegal access, launch failure, misaligned address, device-side assert, or an error reported after a later API call | API return checks, synchronization, minimal kernel reproducer, CUDA-GDB, and Compute Sanitizer |
| NCCL initialization or collective failure | Rank startup error, communicator creation failure, collective timeout, unhandled CUDA error, or a hang with no progress | Version and environment parity, NCCL diagnostics, interface and transport selection, shared memory, and a failing rank pair |
| GPU-to-GPU or GPU-to-NIC path problem | Peer access unavailable, sharply asymmetric bandwidth, failures only for certain GPU pairs, or GPUDirect RDMA errors | PCIe and NUMA topology, P2P capability, IOMMU/ACS, NIC affinity, and peer-memory path |
| Fabric or runtime problem | Link-up interfaces with drops, congestion, pause frames, queue errors, intermittent reachability, or container/VM-only failures | Independent network tests, switch and port counters, RDMA validation, and visibility of devices and modules |
Do not treat a timeout as proof of a network outage. A CUDA fault on one rank can prevent that rank from entering a collective, making every other rank appear to have a communication problem.
Reproduce in a controlled sequence
- One GPU: run the smallest model or kernel with one visible device. This removes peer access and network variables.
- Multiple GPUs on one node: use the same process count and device mapping as the job. A failure here points to CUDA, P2P, PCIe, shared memory, or local NCCL transport.
- Two nodes and two or more ranks: keep the GPU count small and record the exact host pair. This adds interface selection, routing, RDMA, and firewall variables.
- Full allocation: expand only after the reduced cases pass. Preserve logs from every stage so a regression is visible.
For each stage, change one setting only. Keep a known-good command and environment file so an experiment can be reversed immediately.
Debug CUDA kernel and runtime faults
Check every API boundary
CUDA errors can be asynchronous: a kernel launch may return successfully while the fault is reported by a later synchronization or memory operation. Check the return value of every runtime and driver API call, including launch and synchronization calls. Record the first error and the call that returned it; do not replace it with a blanket retry loop.
kernel<<<grid, block, 0, stream>>>(args);
auto launch_status = cudaGetLastError();
if (launch_status != cudaSuccess) { /* log and stop */ }
auto sync_status = cudaDeviceSynchronize();
if (sync_status != cudaSuccess) { /* log and stop */ }
Use synchronization around the smallest suspect region. This converts a delayed report into a location that can be investigated. Remove unnecessary synchronization after debugging because it changes timing and can reduce throughput.
Minimize the reproducer
- Reduce the batch, tensor, or input until the error still occurs.
- Run one stream and one process before reintroducing concurrency.
- Keep the same GPU architecture and driver when comparing a passing and failing host.
- Separate initialization, data transfer, kernel launch, and teardown so the failing phase is explicit.
A minimal reproducer distinguishes an invalid memory access from an allocator, stream-ordering, or framework integration issue.
Use CUDA-GDB and Compute Sanitizer for the right failure
CUDA-GDB is appropriate when you need breakpoints, device-code inspection, launch debugging, or a precise stop in a failing kernel. Compute Sanitizer targets memory errors, race conditions, synchronization errors, initialization errors, and related launch problems. Run the smallest reproducer under one tool at a time; instrumentation can alter timing and make a marginal race appear or disappear.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Interpret a sanitizer report together with the source-level allocation and launch parameters. An out-of-bounds address in a later kernel may be caused by an earlier write, so inspect the first reported invalid operation rather than only the final crash.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesDiagnose NCCL initialization, collectives, and hangs
Make NCCL produce controlled diagnostics
Enable NCCL’s built-in diagnostic logging for a reduced run and retain the output from every rank. Compare the first divergent line, not just the final timeout. Diagnostics can expose a configuration mismatch before any packet or DMA transfer occurs and can identify the GPU pair or communication path that stops progressing.
Check that every rank agrees on GPU visibility, CUDA driver, CUDA runtime, NCCL, framework, container, and relevant environment variables. A different device mask or interface variable on one host can create a communicator that never completes.
Separate initialization from collective progress
- Run communicator initialization with the smallest rank count.
- Run a simple all-reduce or broadcast with a small tensor.
- Increase tensor size and then test the application’s real collective pattern.
- Compare the first collective that hangs with a standalone NCCL diagnostic run.
If initialization fails, focus on rank discovery, shared memory, interface selection, and transport availability. If initialization passes but a particular collective hangs, inspect the failing GPU pair, stream ordering, tensor sizes, and whether one rank reported a CUDA error before entering the operation.
Check shared memory and process limits
Containerized jobs can have a smaller /dev/shm or stricter IPC and locked-memory limits than the host. Confirm that all ranks can create the shared-memory resources NCCL expects and that the container has the required IPC and device permissions. A local multi-GPU failure that disappears with a host-level run is strong evidence of a runtime or namespace restriction rather than a fabric fault.
Recommended Free Tools
Use interface and transport controls as experiments
Temporarily constrain NCCL to the intended network interface or transport to answer a question, not as a permanent cure. For example, test the dedicated data interface separately from management Ethernet, then compare socket transport with the available RDMA path. Record the selected interface in the log and revert the variable after the experiment. If only one interface works, investigate routing, port state, firewall rules, VLANs, or RDMA configuration instead of leaving a fragile override in production.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Verify GPU peer-to-peer and PCIe paths
Confirm that the topology is visible where the job runs
Use the topology view on the host and inside the container or VM. Compare GPU-to-GPU and GPU-to-NIC affinity, PCIe root complexes, NUMA placement, and link types. Incomplete topology exposure can cause NCCL to select a slower route or fail to establish a direct path.
Test peer access for the exact GPU pairs used by the process mapping. “GPU visible” does not mean that peer memory access is available; firmware, PCIe hierarchy, virtualization, and policy can prevent it.
Investigate IOMMU and ACS deliberately
IOMMU and PCIe Access Control Services (ACS) settings can redirect or restrict peer transactions. Depending on platform policy, an ACS configuration that forces traffic through a root complex can degrade or hang a GPU-direct path. Compare the failing node with a known-good node and involve the platform owner before changing firmware or kernel settings. Treat any IOMMU change as a controlled maintenance action because it affects isolation and device access beyond NCCL.
Validate GPUDirect RDMA prerequisites
GPUDirect RDMA requires a compatible NIC and driver combination plus a functioning peer-memory path. Confirm that the nvidia-peermem module is loaded when that path is used, or that the platform’s supported DMA-BUF path is configured. Check module and driver versions on every node, not only the node that reports the error. A missing module can leave ordinary host-memory networking functional while GPU-buffer RDMA fails.
Test networking independently of NCCL
Start with interface selection and reachability
- Verify that the intended interface is up on every node and has the expected address and MTU.
- Test both directions between the exact host pair allocated to the job.
- Check routing, name resolution, firewall policy, and ephemeral-port restrictions.
- Confirm that the container sees the same interface and route as the host, or document the deliberate difference.
These checks should pass before interpreting an NCCL timeout. If ordinary reachability fails, NCCL is only the first application to expose the network problem.
InfiniBand and RDMA checks
On InfiniBand, inspect port state, link rate, error counters, and subnet-manager status with the platform’s RDMA utilities. Run a low-level bandwidth and latency test such as the ib_write_bw family between the same two nodes, using the same interfaces and MTU intended for the workload. Compare measured behavior across ports and GPU pairs. A low-level test that fails without NCCL isolates the fabric or RDMA stack; one that passes shifts attention to GPU memory registration, NCCL selection, or application ordering.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
RoCE checks
RoCE can remain link-up while congestion or loss prevents useful progress. Inspect NIC port counters with ethtool, including queue drops and pause-related statistics. Check that Priority Flow Control (PFC), Explicit Congestion Notification (ECN), and Congestion Notification Packets (CNP) policies are consistent across hosts and switches. Use rping when appropriate to validate RDMA connection management independently of NCCL. Correlate counter changes with the exact test window; a static snapshot can hide transient congestion.
Unexpected pause frames, rising queue drops, or CNP activity during a hang point to fabric policy, buffer pressure, or traffic-class mismatches. Correct those conditions with the network team rather than masking them with a different NCCL timeout.
Compare the main isolation choices
| Comparison | Failure isolation | Observability | Performance impact | Reversibility |
|---|---|---|---|---|
| One node vs. multiple nodes | One node removes routing and fabric variables; multiple nodes exposes rank and transport interactions | High in one node; distributed logs are required across nodes | Single-node tests may use a different transport and cannot prove fabric performance | Easy; change allocation size |
| Sockets vs. InfiniBand/RDMA | Socket runs test application ordering and basic reachability without RDMA registration | Socket errors are usually easier to observe; RDMA adds NIC and memory-registration state | Socket transport is generally slower for large distributed workloads | High when used as a temporary diagnostic |
| Bare metal vs. VM/container | Bare metal removes namespace, device-pass-through, and virtual-switch variables | Host tools show more complete topology and counters | Virtualization and container boundaries can change latency and available paths | High if the same image and command are run on a test host |
| Passive checks vs. active path tests | Passive checks find mismatches; active tests prove a path can transfer data | Active tests provide counters, bandwidth, and failing endpoints | Active tests consume GPU, NIC, and fabric capacity while running | Run in a maintenance window or isolated partition, then stop |
Failure patterns and targeted remedies
“Unhandled CUDA error” appears during an NCCL call
Stop treating the NCCL call as the origin automatically. Check earlier kernels and memory copies on the same rank, add synchronization around the suspect launch, and run the minimal single-GPU case under Compute Sanitizer. If the single-GPU case fails, fix CUDA or application memory use before changing networking.
Only one GPU pair fails
Record the pair and its PCI bus path, then compare peer access, topology, NUMA affinity, link state, and NIC proximity with a passing pair. A pair-specific result is more consistent with PCIe, ACS/IOMMU, firmware, or device health than with a cluster-wide NCCL setting.
All ranks hang with no useful error
Identify the last log line from every rank and find the first rank that stops advancing. Verify that each rank entered the same collective in the same order and that no earlier CUDA error prevented one rank from reaching it. Then run a small standalone collective and inspect shared memory and interface selection.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Socket tests pass but RDMA tests fail
Check the RDMA device, driver and firmware parity, memory-registration permissions, GPUDirect peer-memory module or DMA-BUF path, and the selected NIC. Validate with a low-level RDMA test before returning to NCCL.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
RoCE links stay up while throughput collapses
Examine PFC pause, ECN/CNP, queue-drop, and error counters during the failure. Confirm that the traffic class and MTU are consistent from host NIC through every switch hop. A link-state check alone cannot rule out congestion.
The host works but the container or VM fails
Compare device nodes, capabilities, IPC and shared-memory limits, network namespaces, mounted topology information, RDMA devices, and loaded peer-memory modules. Re-run the same reduced test on the host and in the workload environment; the difference identifies the boundary that needs correction.
Escalate with evidence that can be acted on
When the problem requires a platform, network, or framework owner, send a compact incident bundle:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Exact command, scheduler allocation, container image, and environment variables.
- GPU, driver, CUDA, NCCL, framework, kernel, NIC, firmware, and RDMA versions for every host.
- Topology output, GPU-to-NIC mapping, and the failing rank and host pair.
- Complete NCCL and application logs with timestamps, plus the first CUDA return code.
- Results from one-GPU, one-node, multi-GPU, and two-node reproductions.
- InfiniBand or RoCE counters and low-level test results captured during the failure.
- The last known-good configuration and the single change made before the regression.
This record lets the recipient distinguish a reproducible software fault from a path-specific hardware or fabric condition without asking you to repeat the same experiments.
A compact runbook
- Freeze logs and inventory versions, topology, interfaces, and environment variables.
- Classify the symptom as CUDA, NCCL, GPU path, or fabric/runtime.
- Reproduce with one GPU, one node, then multiple nodes.
- For CUDA, check every API result, synchronize strategically, minimize, and use CUDA-GDB or Compute Sanitizer.
- For NCCL, compare rank configuration, enable diagnostics, inspect shared memory, and test interface and transport choices.
- For GPU paths, verify P2P, PCIe topology, IOMMU/ACS behavior, NIC compatibility, and
nvidia-peermemor DMA-BUF support. - For networking, test reachability independently; use InfiniBand bandwidth tools or RoCE counters and
rpingas appropriate. - Change one variable, retain the known-good state, and document the exact result.
Frequently Asked Questions
Why can a CUDA error appear to originate in NCCL?
CUDA work is asynchronous. A kernel may fail earlier while the error is reported when a later NCCL operation or synchronization checks the device. Check and log every API return value, then synchronize around the suspect launch.
What is the fastest way to tell whether a hang is a network problem?
Run the same reduced rank and host pair with an independent reachability test and, for the fabric in use, a low-level RDMA test. If those fail, investigate the network or RDMA stack before NCCL; if they pass, inspect rank ordering, CUDA errors, shared memory, and NCCL selection.
Does a visible GPU guarantee peer-to-peer access?
No. Peer access depends on PCIe hierarchy, firmware, virtualization, IOMMU/ACS policy, and the specific GPU pair. Check topology and peer capability in the environment where the job actually runs.
Why is checking RoCE link state insufficient?
RoCE can suffer congestion, queue drops, or pause-frame problems while links remain up. Capture NIC counters during the failure and verify PFC, ECN/CNP, traffic class, and MTU consistency across the path.
Should I leave a socket-only NCCL setting enabled after it makes a test pass?
Usually not. A forced transport or interface is useful as an isolation experiment. Once it identifies the failing RDMA or interface path, correct that underlying configuration and remove the temporary override unless the deployment intentionally requires it.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
