Free tools Windows power users keep installed
One-click scans. No signup required.
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Troubleshooting packet loss and congestion in an AI data-center network starts by separating physical-layer errors from queue pressure, endpoint backpressure, and path imbalance. The reliable approach is to align GPU, NIC, host, switch, and workload timestamps; localize the first abnormal port or queue; classify the evidence; make one targeted change; and verify that both the symptom and its diagnostic signal improve. In RoCEv2 fabrics, rising CRC or symbol errors point first to the link, while ECN, CNP, queue, and PFC evidence describes congestion and its response. A fabric-wide average can still look healthy when one equal-cost path is overloaded, so per-port and per-path counters are important.
How do I tell packet loss from congestion?
Packet loss is an observed delivery failure; congestion is one possible cause. Finite switch buffers drop traffic when ingress exceeds egress, but corrupted frames caused by a physical fault can also be discarded. Slow or stuck NICs, oversubscribed links, queue-mapping errors, and uneven ECMP distribution can create similar bandwidth or latency symptoms.
As an Amazon Associate I earn from qualifying purchases.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsStart with correlation rather than a single graph. A bandwidth dip accompanied by rising CRC, FEC-uncorrectable, or symbol-error counters on one port is a physical-layer lead. A rise in ECN marks, CNPs, queue occupancy, PFC pauses, and latency without localized link errors is a congestion-control or bottleneck lead. If neither group changes, investigate endpoint health, workload phase, telemetry coverage, and timestamp alignment before changing QoS.
Cisco’s AI/ML networking paper describes congestion and bit errors as common reasons for loss; its examples are explanatory scenarios, not universal rates or thresholds.
Popular on Amazon
- 【90 Degree Full Function USB C Adapter】Supports the latest USB4 standard for up to 40Gbps of bandwidth, 8K video support, and up to 240W Fast charging; Support USB PD 3.1 Extended Power Range (EPR) including 140W, 180W and 240W. Your productivity is easily doubled. *Warm Tip: For the best fit, please make sure the USB-C port is fully exposed; thick or bulky cases may affect compatibility.
- 【Reliable Quality】Made from high-quality, scratch-resistant aluminum alloy, delicate metal frame. get excellent wear resistance and heat dissipation. Non-Slip Design for easier insertion and removal.
- 【Small and portable】Ultra thin size, ultra compact design, a petite physique that matches the minimalist design of your modern laptops. Coming with a slim figure that won't block the port next to the one it's plugged into.
- 【Compatibility】Widely work with most usb c, thunderbolt 4/3 devices. Compatibilit for ROG Ally, Steam Deck, Switch, MacBook Pro, Quest 3 VR headset, Surface Book 2, iPhone 18/17/16/15, Samsung Galaxy S23/S22/S21, Note 20/10, Google Pixel, ChromeBook, etc.
- 【Excellence Customer Support】If you experience any issues or cable failure, simply reach out to our dedicated customer service team. We provide a prompt response within 18 hours, providing you with the necessary technical support to resolve any concerns.
What should I capture before changing anything?
Build a time-aligned incident record first. Without a before-and-after baseline, a configuration change can hide the original cause or move congestion to another queue.
- Define the symptom and time window. Record when effective bandwidth, collective latency, job-step time, retransmissions, or application errors changed. Note whether the impact affects one job, one host, one rack, or several paths.
- Map the traffic path. Identify the GPU and NIC, host switch port, leaf and spine ports, link speeds, traffic class, and equal-cost paths used by the affected flows. Include the software, firmware, and driver versions that determine queue and congestion behavior.
- Align counters to the same clock. Export endpoint, switch, and workload metrics with timestamps that cover the incident. Compare counter rates during the event, not only lifetime totals.
- Save a baseline. Capture per-port utilization, queue occupancy and drops, CRC and symbol errors, FEC correction status, ECN marks, CNPs, PFC frames, retransmissions, and latency. Preserve the original data before clearing counters or moving a cable.
- Write down the topology and configuration. Record link speed, transceiver type, fiber or cable path, ECMP membership, QoS classes, PFC-enabled priorities, ECN settings, and MTU. A mismatch between nominally equivalent paths is itself a useful lead.
One link-utilization chart cannot establish root cause. In a reported NVIDIA telemetry example, workload bandwidth was interpreted alongside NIC retransmissions and switch-port symbol errors; the value was the cross-layer correlation, not any single metric.
What do CRC, FEC and symbol errors mean?
CRC failures, symbol errors, and uncorrectable FEC events should move the physical link to the top of the investigation. FEC can repair some corruption; errors beyond its recovery capability can fail CRC and be dropped. The important question is whether the counter rate increases on a specific port during the incident.
Physical-layer investigation steps
- Identify the first changing counter. Compare both ends of the link and adjacent ports. Distinguish corrected FEC events from uncorrectable events, and separate rising rates from historical totals.
- Check the optical or electrical path. Inspect the transceiver, cable, connector, and patch-panel positions associated with the localized port. Dust, looseness, damage, or an incompatible component can produce intermittent corruption.
- Check transceiver health and compatibility. Confirm that the module, wavelength, fiber mode, lane configuration, and speed match the installation. Do not order a generic replacement before identifying the connector and link requirements.
- Clean and reseat only under change control. If contamination is plausible, use the site’s approved connector-cleaning procedure and document what was touched. Reseat or replace one component at a time so the evidence remains attributable.
- Confirm the result. After the physical correction, verify that the relevant error rate stops increasing during a comparable workload window and that retransmissions or the original performance symptom also recede.
NVIDIA’s published Spectrum-X telemetry case links RoCE retransmissions to symbol errors on one spine port and reports recovery after that faulty port was disabled. Treat this as a platform-specific vendor case that illustrates localization, not proof that every retransmission is a cable fault. The Cisco guidance on bit errors likewise points operators toward the affected cable, transceiver, connector, and patch panel instead of indiscriminate replacement.
Also popular on Amazon
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
How does RoCEv2 congestion feedback work?
RoCEv2 congestion management is a feedback loop: a switch marks congestion with ECN, the receiver can generate a Congestion Notification Packet (CNP), and the sender reduces its rate according to the deployed congestion-control implementation. This is different from a physical error, where a frame is corrupted or discarded.
NVIDIA’s RoCEv2 explanation describes the ECN-to-CNP control loop. When diagnosing it, inspect every link in that loop:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →- Marking: Are ECN-capable packets marked when the congested queue reaches the configured condition?
- Notification: Are CNPs generated by the receiver, carried in the expected traffic class, and delivered back to the sender?
- Reaction: Does the sender reduce its rate, and does it later recover according to the configured algorithm?
- Outcome: Do queue occupancy, latency, retransmissions, and application bandwidth change in the expected direction?
Compare the switch ASIC, NIC, driver, firmware, queue mapping, and traffic-class configuration. A setting that is valid for one implementation or release is not automatically valid for another. The sources reviewed do not establish universal ECN thresholds, so use the documentation for the actual equipment and workload.
What do ECN and PFC counters mean?
ECN is normally the early rate-reduction signal; Priority Flow Control (PFC) is a pause mechanism intended to prevent a priority class from overrunning its buffer. PFC can avert drops, but it can also delay traffic and propagate backpressure to otherwise healthy flows.
Also popular on Amazon
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
| Signal | What it suggests | What to check next |
|---|---|---|
| ECN-marked packets rising | A queue is experiencing congestion and is asking sources to slow. | Marking condition, queue mapping, traffic class, CNP generation, and sender response. |
| CNPs rising without lower queues or latency | Feedback is present but may be delayed, misprioritized, or ineffective. | CNP path and priority, receiver behavior, sender rate response, and NIC/driver settings. |
| PFC pauses sent or received | A priority class is applying backpressure to avoid buffer overflow. | First congested queue, pause scope, slow endpoint, oversubscribed egress, and propagation across hops. |
| Queue drops with little ECN or PFC activity | Marking or pause protection may be absent, late, or mapped to the wrong class. | ECN capability, thresholds documented for the platform, queue assignment, and configuration parity. |
Check the ordering: ECN should act before PFC reaches its pause threshold. Juniper’s validated design and Microsoft’s May 22, 2026 guidance describe ECN as the first line of defense and PFC as a last resort. That is vendor guidance for implemented systems, not a standards-mandated numeric value. Excessive or premature PFC can interrupt flows and contribute to oscillation or pause storms.
How can I find a slow endpoint or pause storm?
PFC activity at several hops shows that backpressure is propagating, but it does not identify the original bottleneck. Find the first queue or endpoint that became constrained, then follow the pause direction.
- Locate the first congested queue. Compare queue occupancy, drops, ECN marks, and PFC events hop by hop. The first sustained rise is more informative than the largest downstream counter.
- Check endpoint health. Inspect NIC errors, link negotiation, receive and transmit rates, driver or firmware alerts, and GPU-to-NIC behavior. A slow or stuck NIC can make upstream switches pause an entire priority.
- Check for speed and capacity mismatches. A lower-speed egress, oversubscribed uplink, or unexpected traffic-class mapping can create a bottleneck even when the source links are lightly loaded.
- Trace pause propagation. Record which device sends and receives PFC frames and which priorities are affected. Determine whether unrelated flows share the paused class or path.
- Isolate before retuning. If one endpoint or egress is responsible, correct or temporarily isolate it under change control. Broadly raising thresholds or disabling protection can convert backpressure into drops.
Cisco’s discussion of slow or stuck NICs and pause propagation supports this hop-by-hop approach. PFC is a safety mechanism, not evidence that the underlying bottleneck has been fixed.
How do I detect ECMP or path imbalance?
Compare equivalent links individually, not just the fabric average. A small number of hot links can dominate collective latency while the average utilization of all leaves and spines appears normal.
Also popular on Amazon
- Portable and powerful USB-C HUB: BENFEI USB Type-C HUB, with super-soft and knot-free silicone woven design cable, meets most mobile office needs. Compact, lightweight, stylish, and powerful portable USB C Hub equipped with 1 x HDMI port, 1 x 100W charging, and 3 x USB ports. 18-month warranty, 24-hour response, to ensure you feel at ease when using our product.
- Design centered on comfort and reliability: Thanks to BENFEI's end-to-end in-house cable production capability, in-house PCBA and assembly capability, using the industry's most advanced silicone woven design and process, 20cm cable in length, no knots, super-soft, the HUB is easy to use in all scenarios: laptop, tablet, stand etc. Super-soft, 25000+ life cycles, to meet your daily carrying and office needs.
- 100W Charging: Support up to 90W USB C pass-through charging via Type-C port to keep your laptop powered. 10W is reserved for other interface operations. No data and video function on the Type-C port.
- 4K HDMI Display: The HDMI port supports media display at resolutions up to 4K 30Hz, keeping every incredible moment detailed and ultra vivid. Please note that the C port of the Host device needs to support video output.
- Transfer Files in Seconds: Transfer files and from your laptop at speeds up to 10 Gbps with USB A 3.2 port. Extra 2 USB A 2.0 ports are perfectly for your keyboards and mouse.
- Group equal-cost paths. For each affected flow or job, list the leaf-spine and spine-leaf links that can carry it, including negotiated speed.
- Compare utilization and queueing. Look for one or more hot links, rising queue occupancy, and latency that coincide with the workload symptom while peer links remain cooler.
- Inspect flow distribution. Five-tuple ECMP hashing can place unequal numbers or sizes of flows on different paths. Check whether the workload’s flow pattern naturally creates a skew.
- Check configuration parity. Compare ECMP membership, link speed, QoS maps, MTU, routing state, and port settings across the supposedly equivalent paths. A mismatch may be more important than the hashing algorithm itself.
- Recheck after a localized change. If a path is drained, restored, or reconfigured, verify that traffic redistributes and that the original hot-link and latency signals change together.
The NVIDIA telemetry example treats unbalanced distribution as a possible sign of leaf configuration differences. That is a useful diagnostic hypothesis, not a universal rule; validate it against the actual counters and configuration.
What should I do when the symptoms do not match one cause?
Use a symptom-to-lead map, then collect confirming evidence before changing settings.
| Observed pattern | First lead | Next evidence |
|---|---|---|
| RoCE retransmissions rise with symbol or CRC errors on one port | Physical link, transceiver, connector, or switch port | Per-port CRC, FEC, and symbol rates; transceiver health; localized path inspection. |
| ECN marks and CNPs rise while queues or latency continue growing | Congestion feedback is ineffective, delayed, or misconfigured | Marking condition, CNP path and priority, sender response, and QoS mapping. |
| PFC pauses rise across successive devices or priorities | Backpressure propagation; possible endpoint or bottleneck | First congested queue, slow or stuck NIC, oversubscribed egress, and pause scope. |
| One equal-cost link is hot while peers are less loaded | Traffic imbalance or configuration/speed mismatch | Per-path flow distribution, ECMP state, configuration parity, and link rates. |
| Bandwidth dips without localized error or congestion evidence | Cross-layer issue not yet localized | Workload phase, endpoint health, telemetry gaps, and timestamp alignment. |
Do not infer causality from a counter that merely moves at the same time. For example, PFC can rise because of a downstream bottleneck, while the original cause is a slow endpoint several hops away. Likewise, retransmissions can accompany congestion or physical corruption; the adjacent error and queue evidence determines which branch deserves priority.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How should I validate a fix?
Validate one targeted change against the same workload or a comparable observation window. The expected diagnostic signal should change along with the user-visible symptom.
Also popular on Amazon
- 【Read before You Buy】 Note: This adapter is USB 2.0 version and cannot transmit HDM, VGA, or other video signals. Nevertheless, they are fully capable of charging devices,duplicating files, and can be seamlessly integrated with USB-C earphones, wireless network adapters, and SD card adapters.
- 【Easy to Use and Safe】Thanks to modern manufacturing techniques, we've made these adapters sufficiently compact so that they can remain inserted in your device without noticeable protrusion or inconvenience. This means you don't have to constantly plug and unplug them, reducing the likelihood of wear and misalignment, and making them more convenient to carry around.
- 【Reliable Quality】Engineered for enhanced durability, durable aluminum alloy shell material,get excellent wear resistance and heat dissipation,the male USB C and female USB connectors & the male USB and female USB C connectors are designed to be plugged and unplugged up to 20000 times. Provides data transfer, device charging, memory space expansion, and additional computer peripherals.
- 【Think more for you】Do you need to connect a USB-C-enabled device to an odd device that only has USB-A ports? Then these USB-A male to USB-C female adapters are made for you.Simply insert the adapter into the USB-A port on the device, and you can instantly connect your USB-C device to the device that originally didn't have the correct connection port.
- 【Trusted xiwxi Quality and Service】As a responsible brand seller, you will receive xiwxi's 12-month Quality assurance and lifetime friendly customer service, you're not just buying a product,you're investing in peace of mind with all-round quality and exceptional customer service.
- State the hypothesis. Examples: “This port’s rising symbol errors are causing retransmissions,” or “This egress queue is driving PFC across two hops.”
- Make one reversible change. Drain a faulty path, correct a physical connection, repair an endpoint, or fix a documented queue or path mismatch. Avoid simultaneous threshold, routing, and hardware changes.
- Repeat the observation. Use the same job phase where possible, or clearly label differences in workload and traffic volume.
- Check both sides of the prediction. A physical repair should stop the localized error rate and reduce the associated retransmissions. A congestion correction should reduce queue pressure, pause propagation, or path skew while improving latency or bandwidth.
- Keep the evidence trail. Save pre-change and post-change counters, timestamps, topology, and configuration diffs. If the symptom returns, this record makes regression comparison possible.
This validation sequence is operational guidance derived from the cited diagnostic methods; it is not a claim that the site performed a benchmark or hands-on test.
Which troubleshooting mistakes make AI-fabric incidents worse?
- Clearing counters before exporting them. Lifetime totals can hide when the incident started and which port changed first.
- Relying on fabric-wide averages. Averages conceal a hot ECMP link, a single congested queue, or one damaged lane.
- Replacing hardware without localization. Swapping cables or transceivers at random destroys the ability to connect a correction with a counter change and can introduce compatibility problems.
- Treating every retransmission as a cable fault. Retransmissions must be correlated with physical-error and congestion evidence.
- Reading PFC as proof of a healthy network. Pauses can prevent drops while increasing latency and spreading backpressure.
- Changing ECN or PFC thresholds by rule of thumb. Thresholds depend on switch implementation, NIC behavior, queue sizes, traffic class, and workload. The reviewed sources provide no universal values.
- Changing several variables at once. You may improve the symptom temporarily without knowing whether the root cause was fixed.
- Ignoring configuration parity. A single link speed, QoS map, ECMP member, or firmware difference can make otherwise identical paths behave differently.
Can I use this as an incident checklist?
Yes. Keep the following sequence visible during an incident:
- Record the symptom, affected jobs and hosts, start time, and workload phase.
- Align GPU, NIC, host, switch, and application timestamps.
- Check per-port CRC, FEC, symbol, and transceiver-health counters for rates that rise during the event.
- If errors localize, inspect the cable, transceiver, connector, and patch panel before changing QoS.
- Check queue occupancy, queue drops, ECN marks, CNPs, PFC sent and received, retransmissions, and latency.
- Verify that ECN marking, CNP delivery, and sender rate reduction occur in the expected order.
- Trace PFC propagation to find the first congested queue, slow endpoint, or oversubscribed egress.
- Compare utilization and queueing on every equal-cost path, including link speed and configuration parity.
- Make one reversible, localized correction.
- Repeat the workload or observation window and confirm that both the original symptom and its diagnostic signal improve.
The central rule is simple: localize first, classify second, change third. Physical errors, congestion feedback, pause propagation, and ECMP imbalance can overlap, but they leave different combinations of counters. Use those combinations to narrow the fault instead of applying a fabric-wide setting change.
Frequently Asked Questions
Can I disable PFC to stop packet loss?
Disabling PFC globally can turn backpressure into buffer drops and is not a safe default. First identify the affected priority and the queue or endpoint creating pressure; any PFC change should follow the equipment documentation and a reversible maintenance plan.
Are RoCE retransmissions proof that a cable is bad?
No. Retransmissions can accompany physical corruption, congestion, or an endpoint problem. A cable or transceiver becomes the stronger lead when retransmissions rise with localized CRC, symbol, or uncorrectable-FEC errors on the same path.
Do ECN marks mean the switch is already dropping packets?
No. ECN marks are intended to signal congestion before loss, allowing the receiver and sender to reduce the sending rate. Drops, rising latency, or continuing queue growth despite marks indicate that the response or capacity is not keeping up.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Why can a network-wide utilization chart look normal during an outage?
Averages hide concentration. ECMP hashing, a link-speed mismatch, or a single congested queue can overload one path while peer links keep the overall average moderate; per-link and per-queue views are needed to expose that skew.

