The reliable way to find an AI-cluster bottleneck is to correlate a workload symptom with time-aligned telemetry and an isolated subsystem test. GPU utilization alone cannot tell you whether a job is compute-bound, waiting for data, stalled in collective communication, or being constrained by power or temperature.
Start with the job’s step time or throughput, identify exactly when and where it degrades, then test one path at a time: GPU compute and health, GPU-to-GPU links, GPU-to-NIC and inter-node fabric, storage delivery, and facility power or cooling. Compare affected devices with equivalent unaffected peers under controlled conditions before changing software settings or replacing hardware.
As an Amazon Associate I earn from qualifying purchases.
Start with the workload symptom, not a dashboard metric
Write down the slowdown in terms the application exposes: training steps per second, tokens per second, samples per second, inference latency, checkpoint duration, or time spent in a defined pipeline stage. Record the start and end of the affected interval, participating ranks and nodes, data set or input shard, software versions, placement, and whether the behavior repeats.
Free tools Windows power users keep installed
One-click scans. No signup required.
Compare like with like. A step-time increase after changing batch size, sequence length, model parallelism, rank placement, input data, driver, CUDA stack, or orchestration policy is not directly comparable with an earlier run. Preserve those variables in the incident record. If only one rank or node is slow, the investigation should initially favor a localized path or device hypothesis; if every rank slows together, a shared fabric, storage service, power, cooling, or application phase is more plausible.
#1 Best Overall
- [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
- [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
- [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
- [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
- [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.
- Outcome: throughput, latency, or step-time series from the job.
- Scope: affected rank, GPU, node, rack, fabric path, or whole cluster.
- Repeatability: whether the same pattern appears on a rerun with controlled inputs and placement.
- Change log: software, topology, scheduling, data, and configuration changes near the first occurrence.
Align telemetry to the same time window
Export or view telemetry at a resolution that can be aligned with the job’s slow steps. NVIDIA’s observability guidance treats compute, networking, storage, orchestration, and application layers as one diagnostic problem; no single dashboard panel is a complete explanation.
At minimum, align these signals:
- Per-GPU utilization, memory use, clocks, power draw, temperature, health events, and error counters.
- GPU-to-GPU communication activity, peer-access state, and link or copy-path errors.
- GPU-to-NIC and inter-node fabric throughput, latency, retransmissions, link state, and port errors.
- Node CPU utilization, memory pressure, PCIe events, process scheduling, and rank or CPU affinity.
- Storage request latency, queue depth, throughput, cache state, and server-side errors for the actual file or object system.
- Application wait states, input stalls, collective-operation duration, checkpoint time, and dataloader queues.
- GPU power limits, thermal events, rack or room electrical readings, and cooling alarms where available.
Keep timestamps synchronized across nodes and monitoring systems. A temperature or link error recorded minutes outside the slow interval is a clue, not proof of causation.
Turn the symptom into one testable hypothesis
Separate the possible wait locations before running tests. A GPU may be waiting on its own arithmetic or memory path, another GPU, a NIC, an inter-node collective, data delivery, or a facility constraint. For each candidate, write the observation that would distinguish it from alternatives.
| Candidate location | Workload evidence | Discriminating measurement |
|---|---|---|
| GPU compute or memory | Long compute phases, inconsistent throughput on one device, clock or error anomalies | Equivalent-GPU telemetry and an isolated compute or memory diagnostic |
| GPU-to-GPU path | Collective or peer-copy time rises while local compute is normal | Topology, peer access, and GPU communication bandwidth or latency test |
| GPU-to-NIC or inter-node fabric | Inter-node collectives are slower than intra-node ones; affected ranks map to one path | Fabric bandwidth and latency test plus GPU-to-NIC path verification |
| Storage or data delivery | Dataloader queues empty, input wait overlaps storage latency or queue growth | Workload-shaped storage test with server and client telemetry |
| Power or cooling | Clocks or throughput fall when temperature, power, or throttle events change | Time-correlated GPU and site telemetry, followed by a controlled diagnostic |
Run one hypothesis at a time. If a test changes several variables simultaneously, a pass or failure cannot identify the cause.
Establish component baselines before tuning NCCL
Collect isolated measurements for the installed hardware and topology. NVIDIA recommends nvbandwidth or nvloom for GPU memory and GPU communication measurements, and ib_write_bw and ib_write_lat for fabric bandwidth and latency. These tools answer different questions from an application collective benchmark.
Rank #2
- 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
- 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
- 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
- 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
- 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
There is no universal pass number that applies to every GPU, NIC, PCIe generation, switch layout, or link configuration. Define the expected baseline from the actual system design, vendor documentation, and a known-good node or path. Record message sizes, concurrency, process placement, NUMA and CPU affinity, GPU and NIC pairs, and whether the test is intra-node or inter-node.
Interpret the comparison in two stages:
- Measure the physical or near-physical path in isolation and compare it with an equivalent healthy path on the same hardware generation and topology.
- Measure the application’s collective performance under the same rank placement. If isolated paths meet their expected baseline but NCCL remains slow, investigate NCCL configuration, rank-to-CPU/GPU/NIC affinity, process launch, and runtime behavior rather than assuming the fabric is defective.
NCCL’s troubleshooting documentation organizes diagnosis into GPU, networking, runtime or MPI, performance and tuning, logging, and reliability, availability, and serviceability areas. Use that order as a fault-domain map, not as a reason to change tuning variables first.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsCheck GPU compute and health outliers
Use passive DCGM health monitoring to watch field values such as device state, memory and data-path errors, clocks, throughput consistency, power, and thermal events. Compare equivalent GPUs in the same job and compare the affected device with an unaffected peer under the same load.
Passive monitoring and active diagnostics are different evidence. A clean passive view means no recorded field event was observed; it does not prove that a device sustains application performance under the particular workload. DCGM diagnostic suites can actively exercise compute and framebuffer correctness, PCIe and peer bandwidth, memory bandwidth, targeted stress and power, and NVLink or PCIe copy paths. They can expose a device or path outlier, but they do not prove that the application is configured correctly.
Run active tests only on suitable, isolated GPUs or nodes. NVIDIA DCGM documentation cautions that diagnostics can consume GPU compute, memory, PCIe, NVLink, CPU, power, and cooling capacity. Schedule them outside production jobs or drain the target resources through the cluster’s normal procedure.
Rank #3
- [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
- [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
- [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
- [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
- [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by
What a GPU outlier looks like
- One GPU has materially different clocks or throughput from equivalent devices during the same phase.
- Correctable or uncorrectable memory, PCIe, or data-path events coincide with the slowdown.
- A compute, memory, or copy diagnostic fails or is substantially different on the affected device.
- The workload slows when the device enters a thermal or power-brake state.
Do not turn one warning into an immediate replacement decision. Repeat the measurement, compare another workload or node, and follow the platform team’s process for isolating or draining a node when evidence indicates a health problem.
Recommended Free Tools
Determine whether GPU communication is intra-node or inter-node
A collective slowdown can originate inside a server or across the fabric. Inspect the topology exposed to the process and verify which GPUs, PCIe root complexes, and NICs are being used. Confirm that the intended peer-to-peer and GPU Direct paths are available.
NVIDIA’s GPU troubleshooting guidance identifies several path dependencies:
- Peer-to-peer access and topology discovery must reflect the physical arrangement.
- GPU Direct RDMA requires the appropriate support path, such as
nvidia-peermemor DMA-BUF, according to the platform and deployment mode. - IOMMU behavior and PCIe Access Control Services can affect peer or direct-memory paths.
- Virtualized environments may expose a topology that differs from the physical machine.
Check these conditions with the cluster or platform administrator. Changing IOMMU, ACS, kernel modules, or virtualization settings without understanding the deployment can create a new failure or violate platform requirements.
Use an unaffected path as the control
Run the same GPU communication test on a healthy GPU pair in the same node, then across an equivalent node pair. If only one pair is slow, map that pair to its PCIe, NVLink, NIC, or switch path. If all intra-node pairs are healthy but inter-node tests degrade, focus on NIC selection, fabric ports, routing, congestion, or rank placement. If isolated GPU communication is normal while NCCL collectives are slow, retain the communication path as a control and investigate process affinity and NCCL runtime configuration.
Rank #4
- Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
- Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
- All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
- Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
- Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
Separate fabric performance from NCCL performance
Use ib_write_bw to assess fabric bandwidth and ib_write_lat to assess latency under a defined message and connection setup. Capture the exact ports, devices, nodes, message sizes, and concurrency. Compare both directions where the test permits and repeat on an unaffected path.
Then compare those results with NCCL collective timings for the same nodes, rank count, and placement. A healthy point-to-point fabric test does not guarantee a healthy collective: collectives add rank synchronization, GPU-to-NIC transfers, CPU and NUMA affinity, process launch behavior, and algorithm selection. Conversely, a slow fabric result is evidence to fix the path before tuning collective algorithms.
Questions to answer before changing NCCL variables
- Did the process select the intended NIC and GPU Direct path?
- Are ranks evenly placed across GPUs, CPUs, NUMA domains, and network interfaces?
- Is the slowdown limited to inter-node collectives or also present for intra-node operations?
- Do point-to-point bandwidth and latency match the system-specific baseline?
- Are there link errors, retransmissions, congestion, or a downshifted link near the event?
Change one configuration variable at a time, preserve the previous setting, and rerun the same benchmark. Tuning is a confirmation step after topology and baseline connectivity checks, not a substitute for them.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Test storage and data delivery in workload context
Treat an input wait as a storage hypothesis only when application timing and storage telemetry overlap. An empty dataloader queue, rising request latency, or queue depth during the same training phase is stronger evidence than a low average storage bandwidth number collected at another time.
No workload-independent storage threshold applies to every AI cluster. The useful test depends on the file or object system, data layout, cache state, request size, concurrency, metadata behavior, and access pattern. A single sequential-bandwidth result cannot prove that storage is or is not limiting a training workload.
Define the storage experiment
- Describe the actual system: file or object protocol, clients, servers, cache layers, dataset layout, and local spill or staging behavior.
- Record the workload pattern: read or write mix, request size, concurrency, number of ranks, shard distribution, and whether access is sequential, random, or metadata-heavy.
- Measure client-side wait time and queue depth while collecting server-side latency, throughput, errors, and saturation indicators.
- Repeat with a known-good data set or local cache where possible. Keep GPU count, rank placement, and input order fixed.
- Compare the measured phase with the job’s step-time trace. A storage result is relevant only if it explains the observed wait and reproduces under the same pattern.
Use the storage system’s own design limits and alarms for interpretation. Do not import a benchmark threshold from a different architecture.
Diagnose power and cooling constraints
Correlate GPU temperature, clock frequency, board power, configured power limit, and throttle or health events with the exact slow steps. A power or thermal hypothesis is credible when the workload slowdown begins or worsens as a constraint appears and recovers when the constraint clears, preferably on one affected device while peers remain stable.
DCGM’s targeted-power test sustains a compute load and checks whether a GPU reaches a configured board-power threshold. Interpret it carefully: a competing workload can interfere with the test, and an enforced power limit below the target can prevent a meaningful result. Treat that outcome as a configuration constraint to investigate, not as proof that the hardware is defective.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Distinguish GPU limits from facility limits
A GPU-level power or thermal event says something about that device under that load. It cannot establish that a rack, room, or facility lacks electrical or cooling headroom. For facility conclusions, correlate rack and room electrical readings, inlet and exhaust temperatures, cooling-unit alarms, and design limits with the same workload interval. The applicable thresholds are site- and architecture-specific; use the facility’s engineering documentation and alarm policy.
When a node is suspected of overheating or hitting a power constraint, compare it with neighboring nodes at the same utilization and ambient conditions. Check airflow, fan state, firmware and power policy, and placement before escalating. Keep production traffic off the test node when an active stress or power diagnostic could affect shared capacity.
Run a controlled diagnostic sequence
- Freeze the symptom. Save the job identifier, throughput or step-time trace, rank and node list, software versions, placement, input set, and exact incident window.
- Build the timeline. Overlay application waits, GPU clocks and utilization, communication timings, fabric counters, storage latency, power, temperature, and health events.
- Choose one fault domain. State what measurement would support or reject a GPU, communication, storage, power, or cooling hypothesis.
- Compare peers. Select an unaffected GPU, node, path, or rack with equivalent hardware and placement. Use the same workload phase and test parameters.
- Run isolated baselines. Use
nvbandwidthornvloomfor GPU communication, andib_write_bworib_write_latfor the fabric, with documented topology and affinity. Use workload-shaped storage measurements rather than a generic figure. - Schedule active DCGM diagnostics. Drain or isolate resources, select the relevant suite, and record the test’s power, thermal, compute, memory, PCIe, and NVLink impact.
- Repeat under controlled load. Keep rank placement, message sizes, concurrency, and data access pattern fixed. A one-off result is a lead, not a diagnosis.
- Confirm the boundary. If physical baselines pass but the application remains slow, move to runtime, affinity, orchestration, or NCCL configuration. If a path or device fails repeatedly, involve the platform owner.
- Remediate through operations. Drain or isolate a node when evidence supports a health issue, apply one configuration change, and rerun the original workload to verify recovery.
Interpret common evidence patterns
| Observed pattern | Most useful next check |
|---|---|
| All GPUs show low utilization while input queues are empty and storage latency rises | Correlate dataloader waits with client and server storage telemetry; reproduce the access pattern. |
| One GPU has lower clocks, a thermal or power event, and longer step time than peers | Repeat with an equivalent workload, inspect cooling and power policy, then run an isolated DCGM test on the drained device. |
| Intra-node communication is normal but inter-node collectives slow down | Check NIC choice, GPU Direct RDMA path, fabric port counters, congestion, and rank-to-NIC affinity. |
| Point-to-point fabric tests are below the healthy-node baseline | Investigate link state, port errors, routing, congestion, and switch or NIC path before changing NCCL tuning. |
| GPU and fabric baselines pass but NCCL remains slow | Inspect rank, CPU, NUMA, GPU, and NIC affinity; then test NCCL configuration one variable at a time. |
| All nodes slow together while rack power or cooling alarms change | Use site electrical and thermal telemetry to determine whether a shared facility constraint overlaps the workload event. |
Common diagnostic mistakes
- Calling low GPU utilization the cause. Utilization is an outcome that can fall when a GPU waits on communication, storage, synchronization, or throttling.
- Tuning NCCL first. Without topology and fabric baselines, a setting change can hide the original path problem or make comparisons impossible.
- Using a universal storage number. Storage behavior depends on layout, cache, request pattern, concurrency, and protocol.
- Confusing passive health with proof of performance. Field monitoring and active diagnostics answer different questions.
- Running stress tests inside production. Active diagnostics consume the same compute, memory, PCIe, interconnect, power, and cooling resources that the job needs.
- Changing platform settings casually. IOMMU, ACS, kernel modules, virtualization, and power policies require deployment-specific review.
- Ignoring scope. A single slow GPU, one node, one fabric path, a rack, and the whole cluster imply different owners and different tests.
- Replacing hardware on one warning. Require repeatability, peer comparison, and a controlled confirmation before taking a node out of service.
The operational decision rule
Call a bottleneck only when three observations agree: the workload outcome changes in a defined interval, aligned telemetry shows a corresponding subsystem signal, and an isolated or peer comparison reproduces the difference. If any one of those is missing, keep the result as a hypothesis and test the nearest alternative.
This method narrows the fault domain without assuming that a particular GPU, switch, storage device, or facility alarm limit is universal. It also gives the next operator a reproducible record: what slowed, where it slowed, which path was measured, what baseline was expected for that installation, and what changed after remediation.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
