Use a layered, evidence-first workflow: capture the failing NCCL run, verify which transport and network plugin it selected, test GPU-to-GPU and node-to-node paths separately, then correlate latency and throughput with NIC and switch counters. Follow the InfiniBand or RoCE branch that matches your fabric. Change NCCL settings only after the hardware path and fabric have been checked, and test one change at a time.
Start by locating the failing layer
“Packet loss” in an AI cluster can describe several different failures. A GPU peer path inside one host, a host NIC, an InfiniBand or RoCE link, a switch queue, and NCCL’s transport selection can all produce a slow or hanging training job. Testing only the application hides those distinctions.
As an Amazon Associate I earn from qualifying purchases.
| Layer | Typical symptom | Evidence to collect |
|---|---|---|
| GPU-to-GPU inside one host | Peer-to-peer initialization errors or unusually low local collective performance | NCCL local diagnostics, PCIe/NVLink topology, device configuration |
| Node-to-node endpoint | High or variable latency, low bandwidth, or a failing point-to-point test | ib_write_bw, ib_write_lat, ibping or rping, NIC state |
| Multi-hop fabric | Performance changes by destination, rail, path, or time; retries, drops, or link events | NIC and switch counters, routing and topology checks, link telemetry |
| NCCL software path | Low-level tests are healthy but the distributed job is slow or uses an unexpected transport | NCCL logs, plugin and driver versions, environment-variable handling, affinity |
A nonzero counter is not, by itself, a root-cause diagnosis. The useful signal is a counter that increases while the problem reproduces and aligns with latency tails, unstable throughput, link-state changes, or endpoint failures.
Free tools Windows power users keep installed
One-click scans. No signup required.
1. Capture the failing NCCL run before changing anything
Reproduce the slowdown or hang with debug logging enabled and save logs from every participating host and rank. Console output from one process is rarely enough to reveal a path-selection or rank-specific problem.
#1 Best Overall
- 【Fiber Optic Cleaning Kit】 The fiber optic cleaning swab effectively cleans multiple surfaces. This swab functions as a cleaner for SC, FC, ST, APC, or LC fiber optics, as well as optical connectors. Capable of over 500 reusable cleanings, its ergonomic design ensures comfortable handling. Each swab provides a single-use cleaning solution, removing even stubborn contaminants in one pass.
- 【Stable Fiber Cleaning 800+】Our fiber optic cleaning pen delivers consistent anti-static cleaning performance, supporting over 800 cleaning cycles for lower per-use costs. Our LC cleaning pen thoroughly removes contaminants from fiber surfaces, helping you get the job done.
- 【Premium Exterior Design】This fiber optic cleaning pen seamlessly combines convenience with efficiency. Designed with the familiar shape of a standard pen, it offers comfortable grip and effortless handling during cleaning.This fiber optic cleaning pen is easy to operate. Its unique “click” confirmation sound signals completion of the cleaning process, ensuring thorough and efficient cleaning every time.
- 【Stable Cleaning】This cleaning kit effectively removes dust, grease, and other debris from fiber optic port surfaces without scratching the surface.
- 【Wide Application】This LC fiber optic cleaning set is the ideal choice for fiber optic cleaning. This highly effective cleaning pen is suitable for LC/SC/FC/ST/MU/MTP/MPO/D4/DIN connectors
- Set
NCCL_DEBUG=INFOfor the reproducing run. - Set
NCCL_DEBUG_SUBSYS=INIT,ENV,GRAPH,NET,COLL,TUNINGto include initialization, environment, topology, network, collective, and tuning messages. - Set
NCCL_DEBUG_FILEto a different file for each VM and rank. For example, use a naming scheme such as/var/log/nccl/node-a_rank-0.log; create the directory first and ensure the job can write to it. - Record the NCCL version, network-plugin version, GPU driver, NIC driver or firmware, switch software, and the fabric type. Record the exact job command, number of nodes, GPUs per node, and the time window of the failure.
- Read the initialization messages to confirm the selected interface, transport, plugin, and any rejected or ignored environment values.
NCCL can fall back to a built-in Socket or InfiniBand transport when a separately expected plugin is absent. Google Cloud’s gIB guidance documents this type of path-selection issue for its AI Hypercomputer environment; equivalent checks are needed on other platforms, using the instructions for the deployed plugin and fabric.
2. Establish a local GPU baseline
Separate an intra-node problem from a network problem before examining switches. NCCL’s diagnostics include passive readiness checks and active communication checks. Active checks run during communicator initialization and report findings; they are informational and do not abort initialization.
Run the peer-path check
Use NCCL’s documented P2P diagnostic where it is supported by your build and platform. A failed peer path points toward GPU topology, PCIe or NVLink connectivity, or device configuration rather than an inter-node packet-loss event. Save the diagnostic output with the NCCL logs.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Interpret a passing local test
If GPU peer checks pass while the distributed job is slow, move to the host-to-host baseline. A passing local check does not prove that the NIC, switch path, or NCCL network plugin is healthy; it only makes the local GPU path less likely to be the primary fault.
3. Measure node-to-node bandwidth and latency
First verify the interface and port state, link layer, and expected rate. Then measure both throughput and latency between the same nodes and rails used by the job. Bandwidth can look normal while tail latency is damaging collectives, so do not omit the latency test.
- On InfiniBand-capable paths, run
ib_write_bwbetween a server and a client using the intended device and port. - Run
ib_write_latover the same path and repeat the test during the period when the training job shows instability. - Compare results across node pairs and rails. A single bad pair or rail is more informative than a cluster-wide average.
- Keep the command options, message sizes, process placement, and test duration with the output. Example timings in vendor documentation are illustrative, not universal performance guarantees.
NCCL’s built-in network bandwidth check also invokes ib_write_bw over selected InfiniBand devices when its prerequisites are met. Use it as a corroborating check, not as a replacement for a fabric-specific investigation.
Rank #2
- 【Multi Testers Tool Sets】This fiber optic termination tool kit includes OPM, visual fault locator, fiber cleaning pen, fiber cleaner cassette, fiber cleaver, fiber stripper, Kevlar scissors, FC to LC adapter, cleaning core, liquid dispenser, fiber cleaning swabs, fiber cleaner wipes, fiber optic tool bag. This fiber optic tool kit is used for fiber stripping, cleaving, cleaning and testing. It provides the fiber optic tools you need to assemble mechanical splices and delivers greater work efficiency.
- 【Fiber Cleaning Kit】1.25mm & 2.5mm fiber optic cleaner pen & fiber optic cleaning box are used to clean LC/MU/SC/FC/ST/MTP/MPO/D4/DIN ferrule end face. The fiber cleaner can effectively clean stains and prevent dust from adhering to the ferrule again after cleaning.
- 【Visual Fault Locator Fiber Optic】Our fiber tester can intuitively and accurately detect and locate fiber breaks, poor connections, bends or cracks. It is designed for SC, FC, LC, ST interfaces both in circle and square shape of different fiber optic cables. The fiber optic tester has strong output power.
- 【Fiber Optic Meter with FC to LC Adapter】The fiber optic cable tester can measure 10 standdard wavelengths 850/980/1270/1300/1310/1490/1550/1577/1625/1650nm with a test range of -70dBm~+10dBm. The measurement error is less than ±0.2dB. Support simultaneous display of linear mW and nonlinear index dBm.
- 【Fiber Optic Cleaver】3-in-1 high precision fiber cleaver is used to cut bare fiber (0.25mm), pigtail (0.9mm) and leather cable. It has 16 cutting points, the cutting angle is about 0.5 degree, and it can cut about 36000 times.
4. Follow the InfiniBand diagnostic branch
InfiniBand exposes link, retry, port, and subnet-manager evidence that differs from RoCE. Run these checks on the affected hosts and, where permitted, on the relevant fabric devices.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Confirm active state and rate
- Use
ibstatusoribstatto confirm that the expected port is active, identify its link layer, and verify the negotiated rate. - Check that every rank is using the intended HCA and port. An active but unintended port can produce a valid connection with the wrong performance or topology.
Watch RDMA retry and timeout counters
Use rdma statistic and record a baseline, reproduce the issue, then take a second reading. Pay attention to counters such as rnr_nak_retry_err, packet_seq_err, implied_nak_seq_err, and local_ack_timeout_err when they are exposed by the deployed driver. Growth during the failing workload is stronger evidence than a historical nonzero value.
Inspect port errors
Where supported, run perfquery -x <lid> for the affected port. Examples of useful fields include SymbolErrorCounter, LinkErrorRecoveryCounter, and LinkDownedCounter. Correlate increases with link-state events and endpoint-test results; do not infer a single cause from one field.
Check the Subnet Manager and endpoint reachability
- Run
sminfoto verify that a Subnet Manager is present and responding. - Use
ibpingto test endpoint connectivity when the issue appears between particular hosts. - Use
ibdiagnet, when available and authorized, for broader topology, routing, and fabric-error checks. A fabric-wide tool is appropriate when failures vary by route or destination rather than by one server.
Check physical link health
NVIDIA documents mlxlink for physical-link health on supported hardware. Use the vendor’s version-specific syntax and compare optical or electrical health, negotiated speed, and link events with the time of the workload failure.
5. Follow the RoCE diagnostic branch
RoCE congestion is normally diagnosed by combining host NIC statistics with switch telemetry. A healthy endpoint ping does not rule out queue buildup or loss under load.
Recommended Free Tools
Collect host NIC counters
Run ethtool -S <nic_name> on every relevant NIC and save counters before and during reproduction. Names vary by driver, but look for transmit or receive errors, drops, pause frames, priority-flow-control events, ECN marks, CNP-related counters, and queue drops. Use the exact counter definitions supplied for your NIC and driver.
Rank #3
- 【Comprehensive 13 in 1 cleaning kit】This set includes 13 professional cleaning tools that can meet the cleaning needs of various fiber optic connectors. Not only for daily maintenance but also for troubleshooting, this set of tools can provide solutions to ensure that your work is twice as efficient.
- 【400x high-definition microscope for precise detection】Built in high-precision 400x magnification fiber optic microscope allows you to clearly observe small scratches, dust, and contamination on the fiber end face. Visually inspect the cleaning effect to ensure the quality of the connection and reduce signal loss.
- 【Widely compatible with multiple interface types】Compatible with fiber optic interfaces such as SC, FC, ST, LC, etc. Not only 2.5mm (SC/FC/ST) and 1.25mm (LC) ceramic plugs, but even precision MPO/MTP multi-core connectors, this set has corresponding tools for efficient cleaning.
- 【Efficient "one click" cleaning pen, convenient and fast】Equipped with a one click quick cleaning pen, just press lightly to complete an efficient cleaning. The operation is simple and fast, which is very suitable for frequent use during installation, testing, and maintenance, greatly improving work efficiency.
- 【Professional cleaning tools to protect precision equipment】Equipped with professional tools such as fiber optic wiping clips and spare wiping cloths, it can effectively remove stubborn stains and oil stains, while avoiding secondary pollution or physical damage to precision ceramic plugs.
Collect switch evidence
From the switch telemetry system, examine the same time window for interface errors, queue drops, pause or PFC activity, ECN marking, CNP traffic, and queue occupancy. Rising queue drops or pause events on one port or priority can identify a congested path even when aggregate interface utilization appears acceptable.
Correlate congestion with application symptoms
Latency tails that grow while ECN, CNP, PFC, or queue-drop indicators rise are evidence to investigate congestion-control and lossless-fabric configuration before changing NCCL. Stable counters with a reproducible NCCL-only slowdown point back toward software path selection, affinity, versions, or workload behavior.
Test endpoint reachability
Use rping when a RoCE endpoint check is needed. Run it between the same interfaces and hosts used by the training job and preserve the interface selection, address family, and results.
Check GID and traffic-class behavior
NCCL’s guidance says dynamic GID selection is used in NCCL 2.21 and later and warns against setting NCCL_IB_GID_INDEX on those versions. Older versions may require manual GID selection. Follow the instructions for the deployed NCCL and vendor stack rather than copying a value from another cluster. Likewise, use the vendor’s version-specific guidance for NCCL_IB_TC; a traffic-class value that is correct for one switch configuration may map to a different priority in another.
6. Investigate the physical path only when evidence points there
Physical defects can create symbol errors, link recovery, downed links, or intermittent endpoint failures. They are not a general explanation for RoCE queue congestion or an NCCL plugin mismatch.
- Escalate to the cabling and optics path when link counters rise, negotiated rate changes, the link repeatedly flaps, or one cable or transceiver follows the fault when swapped according to your change policy.
- Check connector seating, cable type, optic compatibility, bend radius, and vendor-reported module health.
- A compatible fiber-optic cleaning kit is relevant only after this evidence-led branch identifies a connector or cabling issue. Verify connector type, cleaning procedure, and maintenance authorization before using it; cleaning will not correct PFC, ECN, queue drops, or NCCL transport selection.
7. Interpret the combined evidence
| Observed result | Next investigation |
|---|---|
| Local GPU P2P diagnostic fails | GPU peer path, PCIe/NVLink topology, device configuration, and host hardware |
| Local checks pass; node-to-node test fails | NIC state, physical link, endpoint configuration, switch port, and fabric path |
| Endpoint test passes but latency or bandwidth varies by route | Switch queues, routing, rail selection, congestion control, and topology |
| RoCE congestion counters rise with latency tails or unstable throughput | PFC, ECN, CNP, queue configuration, traffic class, and lossless-fabric design |
| InfiniBand retry, timeout, or port-error counters rise during reproduction | Link health, HCA port, switch path, Subnet Manager, routing, and fabric errors |
| Hardware-appropriate tests are healthy but NCCL is slow | NCCL transport/plugin selection, versions, CPU and memory affinity, and workload-specific configuration |
Capture before-and-after readings rather than relying on a single snapshot. Align timestamps for NCCL logs, endpoint tests, NIC statistics, switch telemetry, and link events. This correlation is what distinguishes an active fault from a stale counter.
Rank #4
- optical fiber cleaning box：CLEANS MULTIPLE SURFACES - This box works as an SC, FC, ST, APC or LC Fiber Cleaner, as well as being an Optical Connector Cleaner.
- Optical Fiber Cleaner Pen ：FC/SC/ST Cleaner Pen, universal 2.5mm; LC MU Cleaner Pen1.25mm Comfortable design with single action cleaning Low cost per clean, up to 800+ Cleans
- Precise mechanical action delivers consistent cleaning results Automatically advance ensures each clean is performed with fresh cleaning tape
- Storage bag-Our optical fiber cleaning tools with waterproof empty bags can store fiber cleansing tools, and protecting products will not be damaged
- Industry professionals use this Optical Fiber Cleaner daily on mission critical projects. A hybrid fabric leaves no lint, and cleans dirty fiber with a swipe.
8. Tune NCCL only after the path is understood
NCCL variables can change interface choice, transport behavior, traffic class, or topology handling, but tuning can mask a fabric fault or improve one benchmark while harming another workload. Preserve the original environment and logs, then make one controlled change at a time.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Write down the baseline environment and exact workload command.
- Change one variable or plugin version, not several together.
- Repeat the same node pairs, message sizes, process placement, and workload duration.
- Compare throughput distribution and latency tails, not just the best run.
- Recheck NIC and switch counters to ensure the change did not move the problem to another queue or rail.
- Revert immediately if the result is worse or if error counters begin rising.
NVIDIA cautions that a gain in one benchmark can make other workloads suboptimal. Treat tuning as workload-specific validation, not a universal recipe.
Common failure modes and recovery steps
Logs show an unexpected transport or plugin
Confirm the installed plugin and its version on every node, verify library search paths and permissions, and check which environment values NCCL accepted. Do not assume that an interface being “up” means it is the intended data path.
Only one rail or node pair is slow
Run endpoint bandwidth and latency tests for that rail and pair, then compare link state and counters with a healthy pair. Inspect the specific switch ports and route instead of averaging across the cluster.
Throughput is acceptable but training has long stalls
Examine latency tails, pause or PFC activity, ECN and CNP behavior, queue drops, and retry or timeout counters during the stall. A bulk bandwidth result can conceal transient queueing that delays collectives.
Counters are nonzero but never increase
Take a fresh baseline, reproduce the issue, and measure the delta. A historical count may reflect an earlier event, a maintenance action, or normal device operation.
Best Value
- Complete 14-in-1 Fiber Optic Kit: Includes fiber optic magnifying glass, cleaning box with replacement core, SC/LC/MPO cleaning pens, 1.25mm and 2.5mm cleaning rods, cleaning paper, cleaning bottle, air blower, brush, and convenient tool bag for organized storage
- Multi-Interface Cleaning Box: Suitable for SC/FC/LC/ST interfaces with replaceable cleaning core, providing up to 500 cleaning cycles for maintaining optimal fiber optic connections
- Dual Cleaning Pen System: SC cleaning pen works with SC/FC/ST 2.5mm interfaces while LC cleaning pen handles LC 1.25mm connections, each offering up to 800 cleaning cycles for long-lasting performance
- MPO Cleaning Pen: Designed for MPO MTP interface male and female connectors, featuring automatic cleaning thread replacement and delivering up to 600 cleaning cycles
- 400X Fiber Optic Microscope: Handheld fiber optic microscope equipped with 2.5mm and 1.25mm adapters for versatile inspection capabilities, enabling detailed examination and verification of fiber optic connections across various devices
A configuration copied from another cluster makes results worse
Restore the saved baseline and verify NCCL, driver, NIC, switch, and fabric versions. Re-evaluate GID and traffic-class settings against the current vendor documentation; version-specific defaults matter.
What packet loss means for training time
Packet loss can significantly impact training completion times, as described in Juniper’s Introduction to Congestion Control in Juniper AI/ML Networks. That statement is qualitative: there is no single percentage that applies to every model, collective pattern, fabric, or loss-recovery mechanism. Measure the effect on your workload with synchronized logs and repeated runs rather than applying a universal estimate.
Frequently Asked Questions
Is there a universal counter threshold that proves RoCE congestion?
No. Counter names and semantics vary by NIC, driver, switch, and fabric design. Use a baseline and look for counters that increase during reproduction while latency tails or throughput instability change.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Should I set NCCL_IB_GID_INDEX on every RoCE cluster?
No. NCCL guidance says dynamic GID selection is used in NCCL 2.21 and later and warns against setting this variable there. Older versions may require manual selection, so follow the documentation for the deployed stack.
Can a fiber-cleaning kit fix packet loss?
Only when link evidence points to a compatible connector or cabling problem. It cannot correct congestion control, PFC or ECN behavior, queue drops, or an incorrect NCCL transport.
Why are NCCL diagnostics not stopping a bad run?
The active diagnostics are informational and report findings during communicator initialization; they do not abort initialization. Use their output to choose the next hardware, fabric, or configuration test.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
