Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Diagnose an inter-datacenter link by correlating training-step or collective-operation time with point-to-point latency, throughput, RDMA error counters, switch congestion telemetry, routing, and physical-link health. Bandwidth alone is not sufficient: a link can report a healthy rate while tail latency, retransmissions, pauses, or intermittent errors stall distributed training.
Start with a controlled reproduction, compare loaded and unloaded measurements, then follow the evidence through transport, fabric, routing, and hardware layers. Use site-specific baselines rather than a universal “bad link” threshold.
As an Amazon Associate I earn from qualifying purchases.
What a slow link looks like in distributed training
AI training usually exposes network defects through collective operations such as all-reduce, all-gather, and reduce-scatter. Affected workers wait for the slowest participant, so one impaired path can lengthen every rank’s step even when aggregate fabric utilization looks moderate. The first task is to prove that the slowdown tracks communication rather than data loading, GPU starvation, storage, or an application change.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallSymptoms that justify a network investigation
- Training-step or collective time rises in bursts rather than remaining steadily high.
- Only jobs using a particular rail, host group, rank pair, or inter-datacenter path are affected.
- RDMA throughput fluctuates at the same times that collective duration increases.
- Latency averages appear normal but P95, P99, or higher percentiles become erratic.
- Retries, out-of-sequence packets, link flaps, pause events, queue drops, or local acknowledgment timeouts increase during the incident.
Record the job identifier, participating hosts and ranks, datacenter pair, fabric rail, collective phase, start and end times, and whether the behavior is continuous or intermittent. A healthy run of the same workload is your most useful comparison.
#1 Best Overall
- 【Fiber Optic Cleaning Kit】 The fiber optic cleaning swab effectively cleans multiple surfaces. This swab functions as a cleaner for SC, FC, ST, APC, or LC fiber optics, as well as optical connectors. Capable of over 500 reusable cleanings, its ergonomic design ensures comfortable handling. Each swab provides a single-use cleaning solution, removing even stubborn contaminants in one pass.
- 【Stable Fiber Cleaning 800+】Our fiber optic cleaning pen delivers consistent anti-static cleaning performance, supporting over 800 cleaning cycles for lower per-use costs. Our LC cleaning pen thoroughly removes contaminants from fiber surfaces, helping you get the job done.
- 【Premium Exterior Design】This fiber optic cleaning pen seamlessly combines convenience with efficiency. Designed with the familiar shape of a standard pen, it offers comfortable grip and effortless handling during cleaning.This fiber optic cleaning pen is easy to operate. Its unique “click” confirmation sound signals completion of the cleaning process, ensuring thorough and efficient cleaning every time.
- 【Stable Cleaning】This cleaning kit effectively removes dust, grease, and other debris from fiber optic port surfaces without scratching the surface.
- 【Wide Application】This LC fiber optic cleaning set is the ideal choice for fiber optic cleaning. This highly effective cleaning pen is suitable for LC/SC/FC/ST/MU/MTP/MPO/D4/DIN connectors
Build a reproducible incident record
Do not begin by changing NCCL or other collective-library settings. First make the symptom measurable and repeatable.
Capture the workload context
- Training framework and collective library version.
- GPU model, host, NIC, PCIe-switch, and switch identifiers.
- Rank-to-host mapping and the rail or interface used by each rank.
- Message sizes, queue-pair (QP) counts, and any traffic classes or virtual lanes.
- Exact training step, collective name, and duration when the stall occurs.
- Concurrent jobs or maintenance, routing, image, and configuration changes in the same window.
Define a comparison run
Repeat the same workload with the same endpoints, message sizes, QP settings, and traffic concurrency. Save a healthy baseline and an incident capture. If the issue occurs on only one rail or a subset of paths, run the same test on an unaffected rail; that difference is strong localization evidence.
Measure throughput and latency together
Run an RDMA performance test with the message sizes and QP counts used by the workload, while also collecting point-to-point latency. NVIDIA’s networking troubleshooting guidance warns that “Bandwidth can look healthy while tail latency is still poor.” A high or unstable latency distribution, particularly when bandwidth also varies, can indicate congestion, retransmissions, or a degraded link.
Free tools Windows power users keep installed
One-click scans. No signup required.
Use unloaded and loaded tests
- Measure the endpoint pair with no competing traffic.
- Repeat while the training job or a controlled full-utilization load is active.
- Keep message size, QP count, duration, and endpoint selection constant between runs.
- Compare the latency distribution and throughput, not just their averages.
The February 2026 IETF Benchmarking Methodology Working Group draft for AI fabrics proposes reporting latency as minimum, mean, P50, P95, P99, P99.9, and maximum under zero-load and loaded conditions. It also proposes measuring sustainable throughput, burst absorption and loss, and fabric health. The document expired on August 30, 2026 and is a work-in-progress methodology, not an adopted standard or a universal production target.
Preserve tails and time alignment
Export timestamps for every test interval and align them with training-step and collective-duration logs. A short-lived P99.9 spike that coincides with a collective stall is more informative than a daily average. The draft proposes repeating latency characterization at least 20 times and testing each burst length at least 50 times; these are useful experimental procedures, not alert thresholds.
Interpret an apparent throughput problem
Low throughput with stable latency often points to a rate, MTU, QP, or application-setting mismatch. Throughput that oscillates while tail latency and retry counters rise is more consistent with congestion or loss. High throughput with severe tail latency means the link is not healthy for synchronization-sensitive training, even if a capacity dashboard is green.
Read RDMA counters while the slowdown happens
Collect counter deltas over a controlled interval rather than relying on lifetime totals. NVIDIA recommends running rdma statistic during a test and watching transport errors such as:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #2
- 【Multi Testers Tool Sets】This fiber optic termination tool kit includes OPM, visual fault locator, fiber cleaning pen, fiber cleaner cassette, fiber cleaver, fiber stripper, Kevlar scissors, FC to LC adapter, cleaning core, liquid dispenser, fiber cleaning swabs, fiber cleaner wipes, fiber optic tool bag. This fiber optic tool kit is used for fiber stripping, cleaving, cleaning and testing. It provides the fiber optic tools you need to assemble mechanical splices and delivers greater work efficiency.
- 【Fiber Cleaning Kit】1.25mm & 2.5mm fiber optic cleaner pen & fiber optic cleaning box are used to clean LC/MU/SC/FC/ST/MTP/MPO/D4/DIN ferrule end face. The fiber cleaner can effectively clean stains and prevent dust from adhering to the ferrule again after cleaning.
- 【Visual Fault Locator Fiber Optic】Our fiber tester can intuitively and accurately detect and locate fiber breaks, poor connections, bends or cracks. It is designed for SC, FC, LC, ST interfaces both in circle and square shape of different fiber optic cables. The fiber optic tester has strong output power.
- 【Fiber Optic Meter with FC to LC Adapter】The fiber optic cable tester can measure 10 standdard wavelengths 850/980/1270/1300/1310/1490/1550/1577/1625/1650nm with a test range of -70dBm~+10dBm. The measurement error is less than ±0.2dB. Support simultaneous display of linear mW and nonlinear index dBm.
- 【Fiber Optic Cleaver】3-in-1 high precision fiber cleaver is used to cut bare fiber (0.25mm), pigtail (0.9mm) and leather cable. It has 16 cutting points, the cutting angle is about 0.5 degree, and it can cut about 36000 times.
rnr_nak_retry_err— receiver-not-ready retry events.packet_seq_err— packet sequence errors.implied_nak_seq_err— sequence-related negative acknowledgments inferred by the transport.local_ack_timeout_err— local acknowledgment timeouts.
An increase during the affected collective is evidence of transport trouble, but no single counter identifies the root cause. Compare the NIC counters with switch-port drops, queue telemetry, link events, and the workload timeline.
Use deltas, not unexplained totals
- Clear or snapshot the current counters according to your platform’s supported procedure.
- Run a fixed-duration baseline test and record the starting and ending values.
- Repeat during a reproduced stall.
- Calculate the change per second or per test, then compare endpoint and switch-port results.
Meta’s ACM SIGCOMM 2024 experience collecting RDMA hardware counters across switches, NICs, PCIe switches, and GPUs highlights out-of-sequence packets, link flaps, and local acknowledgment timeouts as practical signals. A counter that rises only on one host, PCIe path, or port narrows the investigation more effectively than a fabric-wide aggregate.
Follow the correct diagnostic branch for your transport
RoCE over Ethernet
Inspect NIC statistics with:
ethtool -S <nic_name>Counter names differ by NIC and driver. Look for errors, drops, pause frames, priority-flow-control (PFC) events, explicit-congestion-notification (ECN) marks, congestion-notification packets (CNP), queue drops, and buffer occupancy. Collect corresponding switch-port counters for both ends of the path.
- Rising PFC pause time or pause frames can indicate a congested lossless class or a downstream queue that is not draining.
- ECN and CNP activity can show that congestion control is reacting even when packets are not being dropped.
- Queue drops, NIC receive errors, or retransmission-related counters point toward loss or an incorrectly provisioned lossless path.
- Errors isolated to one interface suggest a port, optic, cable, or host-side problem rather than a global traffic surge.
If counters remain normal but connectivity is suspect, use rping between the affected endpoints. Keep the test’s traffic class, MTU, and interface selection equivalent to the training path. Do not assume that a command or counter exists on every Ethernet adapter.
InfiniBand
Use perfquery to inspect port status and errors. NVIDIA identifies symbol errors, link-recovery events, and link-down counters as useful physical and fabric clues. Use ibping for node reachability and ibdiagnet for a broader view of topology, routing, and fabric errors.
A link-recovery or link-down increase during a collective stall is a different problem from a clean port carrying too much traffic. Preserve the port GUID, switch port, link width, and timestamp so the event can be matched to cabling and switch logs.
Distinguish congestion, routing, and packet loss
Cisco’s AI/ML networking guidance identifies congestion and bit errors as the two common causes of packet loss. The counters and test pattern help separate them.
Rank #3
- 【Comprehensive 13 in 1 cleaning kit】This set includes 13 professional cleaning tools that can meet the cleaning needs of various fiber optic connectors. Not only for daily maintenance but also for troubleshooting, this set of tools can provide solutions to ensure that your work is twice as efficient.
- 【400x high-definition microscope for precise detection】Built in high-precision 400x magnification fiber optic microscope allows you to clearly observe small scratches, dust, and contamination on the fiber end face. Visually inspect the cleaning effect to ensure the quality of the connection and reduce signal loss.
- 【Widely compatible with multiple interface types】Compatible with fiber optic interfaces such as SC, FC, ST, LC, etc. Not only 2.5mm (SC/FC/ST) and 1.25mm (LC) ceramic plugs, but even precision MPO/MTP multi-core connectors, this set has corresponding tools for efficient cleaning.
- 【Efficient "one click" cleaning pen, convenient and fast】Equipped with a one click quick cleaning pen, just press lightly to complete an efficient cleaning. The operation is simple and fast, which is very suitable for frequent use during installation, testing, and maintenance, greatly improving work efficiency.
- 【Professional cleaning tools to protect precision equipment】Equipped with professional tools such as fiber optic wiping clips and spare wiping cloths, it can effectively remove stubborn stains and oil stains, while avoiding secondary pollution or physical damage to precision ceramic plugs.
Signs of congestion
- High or rising utilization on one egress or fabric rail while alternate paths are less busy.
- PFC pauses, ECN marks, CNPs, queue drops, or buffer occupancy spikes that align with the slowdown.
- Latency tails worsen under load but largely disappear in an unloaded test.
- Several flows sharing one queue or route degrade together.
Check whether traffic is unevenly distributed across rails, whether a class is mapped to the intended priority, and whether a large burst is overrunning available buffers. A clean average-utilization graph can hide a brief queue saturation event.
Signs of loss or retransmission
- Packet-sequence, implied-NACK, or acknowledgment-timeout counters increase.
- Throughput oscillates as the transport retries.
- Switch output drops or NIC receive errors appear on a specific path.
- Latency spikes occur even when average utilization is not high.
Check both directions. A return-path issue can delay acknowledgments and create retries even when the forward-direction bandwidth appears normal.
Signs of routing or software placement problems
Inspect route and path utilization, ECMP or static-route assignments, and recent switch image or configuration changes. Meta’s deployment report describes static-routing traffic collisions that produced inconsistent performance and a regression associated with a switch software-image scheduling change. These examples show why a congestion dashboard alone cannot prove that routing, software, or latency is healthy.
Compare an affected rank pair with an unaffected pair that uses a different route. If only one path is slow, map every hop and verify that the intended rail and traffic class are actually selected.
Check physical link health
When transport counters or port logs implicate a link, inspect the physical layer before retuning software.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Validate negotiated state
Where supported, use mlxlink to check expected link speed and width, port state, and health. A link operating below its expected width or speed can reduce collective performance without producing obvious packet loss. Record the state on both ends and compare it with the NIC and switch configuration.
Inspect cables, optics, and connectors
Bit errors can come from faulty or dirty cable, transceiver, or connector interfaces. Cisco specifically lists dust at cable ends, transceivers, and patch panels as possible contributors. Follow your site’s change-control and safety procedures; a fiber-optic connector cleaning kit is appropriate only when inspection identifies contamination and the procedure authorizes cleaning. Verify link state and error counters after any intervention.
Rank #4
- optical fiber cleaning box：CLEANS MULTIPLE SURFACES - This box works as an SC, FC, ST, APC or LC Fiber Cleaner, as well as being an Optical Connector Cleaner.
- Optical Fiber Cleaner Pen ：FC/SC/ST Cleaner Pen, universal 2.5mm; LC MU Cleaner Pen1.25mm Comfortable design with single action cleaning Low cost per clean, up to 800+ Cleans
- Precise mechanical action delivers consistent cleaning results Automatically advance ensures each clean is performed with fresh cleaning tape
- Storage bag-Our optical fiber cleaning tools with waterproof empty bags can store fiber cleansing tools, and protecting products will not be damaged
- Industry professionals use this Optical Fiber Cleaner daily on mission critical projects. A hybrid fabric leaves no lint, and cleans dirty fiber with a swipe.
Use event timing
A link flap that matches the first bad training step is stronger evidence than an old lifetime error count. Correlate flap, recovery, symbol-error, temperature, and optic-diagnostic events with the collective timeline. If errors continue after moving the endpoint to a known-good cable or port, the fault may follow the optic, NIC, switch port, or host PCIe path; change one component at a time so the result remains interpretable.
Account for host and PCIe paths
Inter-datacenter symptoms can originate inside a host. Compare NIC counters with PCIe-switch and GPU telemetry, as Meta’s multi-layer collection model recommends. Check for a single GPU or NIC that repeatedly lags, PCIe link-width or retraining events, corrected-error bursts, and NUMA placement that differs from healthy hosts.
Recommended Free Tools
If one host remains slow while its peer on the same fabric path is healthy, isolate the host by moving the workload or swapping the suspect NIC, optic, or cable under controlled change procedures. If many hosts sharing one switch port degrade together, prioritize fabric analysis instead.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Use a controlled comparison matrix
For every before-and-after or path comparison, hold the workload and test settings constant. Record the following axes:
| Measurement | What to compare | What it helps reveal |
|---|---|---|
| Throughput | Goodput by message size and QP count | Rate limits, imbalance, retransmissions, or configuration mismatch |
| Latency | Unloaded and loaded min, mean, P50, P95, P99, P99.9, and max | Queueing and tail-latency sensitivity |
| RDMA transport | Counter deltas per test interval | Retries, sequence errors, and acknowledgment timeouts |
| Port telemetry | Utilization, queue drops, PFC, ECN, CNP, and buffer signals | Congestion and lossless-class behavior |
| Physical state | Speed, width, errors, and flap events | Optic, cable, connector, port, or link degradation |
| Application impact | Collective and training-step time with isolated and concurrent traffic | Whether the network symptom explains the job slowdown |
The IETF draft suggests measuring each throughput rate for at least 60 seconds. Treat that duration as a repeatable benchmark choice, not a universal requirement.
Interpret deployment-specific thresholds correctly
Published examples are useful for designing telemetry but should not be copied as universal alert limits. Meta reported a watchdog for PFC pauses longer than 200 milliseconds on its RTSW and CTSW devices and alarms above 80% buffer utilization on RTSW devices; the authors said the buffer threshold had not been breached in practice. Those values describe one deployment, hardware set, and operating policy.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsBuild your own limits from healthy distributions for each fabric, rail, message class, and workload phase. Alert on a sustained deviation from that baseline and on combinations such as rising P99.9 latency plus transport errors, rather than on a single counter in isolation. NVIDIA, Meta, and the IETF draft provide diagnostic signals and test methods, but no universal “bad link” latency or counter threshold applies to every topology and transport.
Best Value
- Complete 14-in-1 Fiber Optic Kit: Includes fiber optic magnifying glass, cleaning box with replacement core, SC/LC/MPO cleaning pens, 1.25mm and 2.5mm cleaning rods, cleaning paper, cleaning bottle, air blower, brush, and convenient tool bag for organized storage
- Multi-Interface Cleaning Box: Suitable for SC/FC/LC/ST interfaces with replaceable cleaning core, providing up to 500 cleaning cycles for maintaining optimal fiber optic connections
- Dual Cleaning Pen System: SC cleaning pen works with SC/FC/ST 2.5mm interfaces while LC cleaning pen handles LC 1.25mm connections, each offering up to 800 cleaning cycles for long-lasting performance
- MPO Cleaning Pen: Designed for MPO MTP interface male and female connectors, featuring automatic cleaning thread replacement and delivering up to 600 cleaning cycles
- 400X Fiber Optic Microscope: Handheld fiber optic microscope equipped with 2.5mm and 1.25mm adapters for versatile inspection capabilities, enabling detailed examination and verification of fiber optic connections across various devices
Common mistakes and safer alternatives
Mistake: treating bandwidth as a health check
Better: collect point-to-point latency and its tail percentiles during the actual stall.
Mistake: reading lifetime counters
Better: snapshot counters before and after a fixed test and align their deltas with training timestamps.
Mistake: changing collective settings first
Better: establish whether the issue is transport, congestion, routing, or hardware before modifying NCCL or traffic-class configuration.
Mistake: assuming a fabric-wide average represents every path
Better: inspect per-rail, per-port, per-rank-pair, and per-queue telemetry.
Mistake: using a vendor example as a universal threshold
Better: use deployment-specific baselines and document the hardware, traffic class, and policy behind every alert.
A practical escalation checklist
- Identify the affected job, collective, ranks, rail, datacenter path, and incident window.
- Compare the same workload with a known-good run.
- Measure RDMA throughput and point-to-point latency with no load and with reproduced load.
- Save latency percentiles, including P99 and P99.9, rather than only averages.
- Collect RDMA counter deltas while the test runs.
- Use the RoCE or InfiniBand command set appropriate to the transport.
- Match NIC counters to switch-port utilization, queue, PFC, ECN, CNP, and drop telemetry.
- Check routes, rail balance, static-route collisions, and recent switch image or configuration changes.
- Inspect negotiated speed and width, link events, optics, cables, connectors, and host PCIe paths.
- Repeat the comparison after one controlled change and document the result.
What a defensible diagnosis looks like
A strong incident report connects the job symptom to a reproducible path-level observation: for example, collective P99 latency rose only under load on one rail, that rail showed queue and PFC growth, and the affected NIC recorded sequence or acknowledgment errors. A different report might show stable transport counters but repeated link-down events and reduced negotiated width, supporting a physical or port fault. If measurements do not correlate, state that the network has not yet been shown to explain the slowdown and continue checking host, storage, and application layers rather than forcing a network conclusion.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
