What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
To measure GPU utilization reliably, combine interval-based device telemetry with application throughput, then use timeline and kernel profilers to explain the gaps. NVIDIA Data Center GPU Manager (DCGM) shows whether engines, streaming multiprocessors (SMs), tensor pipes, memory, PCIe, and NVLink are active. DCGM Exporter makes selected fields available to Prometheus for fleet dashboards. Nsight Systems then shows when GPU work is absent or delayed, while Nsight Compute can attribute a problem to a kernel, instruction, or source line. No single percentage proves that a workload is compute-bound, memory-bound, or healthy.
What GPU utilization actually measures
“GPU utilization” is an umbrella term, not one universal statistic. Different tools measure different things: an interval-average hardware counter, the amount of time represented by API-trace operations, or the occupancy of a particular execution pipeline. Those measurements answer different operational questions.
As an Amazon Associate I earn from qualifying purchases.
DCGM profiling values are averaged over a measurement interval. They are useful for comparing steady-state phases, replicas, devices, and GPU instances, but they are not kernel traces. Nsight Systems uses API-trace timelines for its utilization rules. A short memcpy and a large kernel can each count as GPU-in-use for the time they occupy the trace, even though they consume very different amounts of GPU capacity. Concurrent operations can make a time-utilization segment exceed 100 percent.
NVIDIA states this distinction directly in its Nsight Systems Post-Collection Analysis Guide: “The utilization refers to the ‘time’ utilization and not the ‘resource’ utilization.” Treat every percentage as a defined measurement, not as a generic score.
#1 Best Overall
- Valued Carpenter Pencil Set: You will get 2 pcs solid carpenter pencils with 26 piece 2.8 mm refills, 1 replaceable sharpener, 1 plastic storage box.The complete carpenter pencils combination allows you to finish your work faster and more easily
- Deep Hole Marker Pencil: The deep-hole construction pencils adopts 45mm elongated tip design, which is more convenient to mark in the small hole or in other tight areas that other carpenter markers cannot reach
- Carpenter Pencils with Sharpener: The sharpener is screwed into the top of the work pencil, which won't get lost either. Built-in pencil sharpener that keep the lead with pointed and smooth to Improves line of sight in fine work
- Stronger Solid Lead: This work pencil is matched with a 2.8 mm thick lead , which is much thicker and stronger during the drawing process of construction work, it will not break or damage easily
- Marks on Various Surfaces: 3 colors solid construction pencil can marks on various surfaces,such as metal, plastic, wood, paper etc. Ideals for woodworkers, contractors, craftsmen, builders, merchants and masons
Signals to collect together
| Signal | What it represents | What it cannot prove alone |
|---|---|---|
| SM activity | The fraction of time at least one warp was active on an SM, averaged across multiprocessors. | It does not prove that warps performed useful arithmetic. A warp waiting on a memory request still counts as active. |
| Tensor activity | The fraction of cycles during which the tensor pipe was active over the interval. | Low or high tensor-pipe activity does not identify why a model phase is slow. |
| DRAM activity | The fraction of cycles with device-memory traffic. | Traffic alone does not establish that the workload is memory-bound. |
| SM occupancy | Resident warps as a fraction of the maximum supported resident warps. | Higher occupancy does not guarantee higher effective throughput. |
| Graphics or compute engine activity | Broad activity of the device engine during the interval. | It lacks the kernel and source-level attribution needed to explain a phase. |
| PCIe and NVLink traffic | Data movement over the host/device and GPU-to-GPU interconnects. | Traffic volume by itself does not establish that communication is the limiting stage. |
The DCGM Profiling documentation warns: “A high value in one field does not alone prove that the application is compute- or memory-bound; compare several related metrics over a representative interval and correlate them with application throughput.” That rule should govern dashboards, incident reviews, and capacity decisions.
Step 1: Define a representative measurement window
Start with the workload, not the monitoring product. Select a phase that represents the question you need to answer: a steady-state training window after warm-up, a fixed inference batch, a communication-heavy synchronization step, or a known production request mix. Record application throughput during the same interval, such as samples per second, tokens per second, requests per second, step time, or job completion rate.
Keep comparisons like-for-like. Compare the same phase across replicas, the same batch and sequence lengths across configurations, and the same model stage across devices. Warm-up, checkpointing, validation, autoscaling, and failure recovery can have very different behavior from steady state. Label those phases instead of averaging them into one fleet-wide percentage.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →- Write down the workload version, model, batch or request shape, precision mode, number of replicas, and parallelism arrangement.
- Mark the beginning and end of the phase in the telemetry system and, where possible, with NVTX annotations in the application.
- Capture throughput and latency with the GPU signals so a high activity reading can be tested against useful output.
- Repeat the window long enough to include normal variation, but avoid combining unrelated phases in one average.
Step 2: Establish fleet visibility with DCGM
DCGM provides tools and libraries for observing and managing NVIDIA data-center accelerators. At fleet scale, DCGM Exporter selects configured DCGM fields and serves them in Prometheus format at a /metrics endpoint. NVIDIA documents package-managed service, container, and Kubernetes DaemonSet deployment paths, allowing an organization to match collection to its host and cluster operating model.
Choose a deployment path
- Package-managed service: Install and run the exporter with the host’s service manager when GPU nodes are managed as conventional servers.
- Container: Run the exporter beside the workloads when the platform standardizes node agents as containers.
- Kubernetes DaemonSet: Deploy one exporter instance per eligible node so scheduling and node lifecycle are handled by the cluster.
The exporter configuration is a request for fields, not proof that every row will be emitted. A field can be absent because the GPU, DCGM release, driver, entity type, permissions, or deployment does not support it. Verify the actual endpoint output on each hardware and software combination, then check the field definition before building an alert around it.
Rank #2
- Ergonomically Designed: Work in tight areas with a compact design that gets into tough spots
- Both tools are designed to fit into difficult to reach spaces
- Complete a wide range of applications with a high-performance motor that delivers up to 300 UWO, and an impact driver delivering up to 1,400 in-lbs of torque
- Both the DEWALT impact driver and drill driver feature integrated LED work lights with a convenient 20-second delay, ensuring enhanced visibility in dimly lit or challenging work areas
- One-Handed Loading - Keep one hand free with a 1/4 in. hex chuck that accepts 1 in. bit tips
Make the endpoint and labels operationally trustworthy
- Confirm that the exporter process can access the GPU devices and the DCGM host engine under the permissions used in production.
- Check that the running DCGM version is the one supported by the exporter release and driver combination.
- Inspect labels for physical GPUs, GPU instances, processes, and other entities before aggregating. A query that silently mixes entities can turn a real imbalance into an apparently normal average.
- Test the endpoint after node replacement, driver upgrades, MIG reconfiguration, and exporter restarts; support can change with any of these events.
- Record which fields are actually present rather than assuming that a configured field is available everywhere.
Step 3: Build a multi-signal dashboard
A useful dashboard places application throughput beside the related DCGM series for the same phase. At minimum, show SM activity, tensor activity, DRAM activity, SM occupancy, engine activity, and PCIe or NVLink traffic when those fields are supported. Keep the time range and sampling interval visible so an interval average is not mistaken for a point-in-time event.
Read the patterns, not a single headline number
- Sustained low activity: The device may be waiting for work, but the chart does not identify whether launch overhead, input preparation, synchronization, or another stage is responsible.
- Long idle gaps: Treat the gaps as candidates for timeline investigation. Their duration and placement relative to CPU and communication events matter more than a single average.
- High SM activity with weak throughput: Active warps may be stalled on memory. Compare tensor activity, DRAM activity, occupancy, and application throughput before labeling the workload compute-bound.
- High DRAM activity: Memory traffic is present. It becomes a bottleneck hypothesis only when throughput, timing, and other counters support that interpretation.
- Low tensor activity in an AI phase: The measured phase may not use tensor pipes heavily, or another stage may dominate. Check the phase definition and profile the operations before changing the model or hardware.
- Uneven devices or replicas: Work may be imbalanced, entities may not be comparable, or a field may be missing on only part of the fleet. Inspect labels, MIG visibility, field support, permissions, and runtime output.
Use rates and averages consistently. A brief burst can disappear in a long interval average, while a short synchronization pause can dominate a latency-sensitive request. Preserve raw time series for the interval that triggered an investigation.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Step 4: Use time-series evidence to form a bottleneck hypothesis
Telemetry should narrow the next question, not end the investigation. The following matrix maps common observations to a defensible next check.
| Observed pattern | What it can indicate | Next check |
|---|---|---|
| GPU activity has long idle gaps | Work may not be reaching the GPU continuously; the metric does not identify why. | Use an Nsight Systems trace with CPU sampling, blocked-state backtraces, synchronization-related OS runtime APIs, and NVTX ranges. |
| SM activity is high but throughput is disappointing | Warps are active, but some may be waiting on memory or otherwise making little useful progress. | Compare tensor activity, memory activity, occupancy, and throughput, then profile the application. |
| Tensor activity is low during an AI workload | Tensor-pipe use may be limited in that phase; the reading alone does not explain the cause. | Check the workload phase and related SM, memory, and throughput measurements; attribute operations with a profiler. |
| Device-memory activity is high | Memory traffic is present, but one field does not prove a memory-bound workload. | Compare multiple metrics over the same representative interval and correlate them with throughput. |
| GPU instances or devices look inconsistent | Workloads or measured entities may differ, or selected fields may not be uniformly available. | Check entity selection, MIG visibility, field support, labels, permissions, and exporter runtime output. |
| Profiling fields go blank during an Nsight session | DCGM profiling may have been paused or may conflict with another client’s counter access. | Inspect DCGM profiling watches and coordinate pause and resume across host-engine clients. |
Step 5: Find idle gaps with Nsight Systems
When the fleet view shows low utilization or unexplained pauses, capture an application timeline with Nsight Systems. Its low-utilization findings are based on GPU operations in API traces. They answer when traced GPU work occupies the timeline, not how many GPU resources those operations consume.
A single running memcpy can therefore count as GPU-in-use for its duration just as a large kernel can. Concurrent operations can cause a time-utilization segment to exceed 100 percent. Do not compare that value directly with an SM, tensor, or DRAM activity percentage; they have different definitions.
Rank #3
- 【Great Compatibility】This Katerk 1/4 inch hex shank bit holder is specifically designed for 1/4 inch hex shank drill bits. It's compatible with most 1/4 fast hex handles, hex sockets, various electric screwdrivers, and handheld screwdrivers. The bit holder makes it a valuable addition for any handyman.
- 【Secure and Safe】Built with a secure backup nut design, each drill bit holder securely locks onto your bits, ensuring they stay firmly in place. Additionally, our bit holder incorporates a high-quality steel ball rolling design that holds up to several kilograms of weight, ensuring your various drill bits don't fall off.
- 【Easy One-Handed Operation】The bit holder for impact driver allows you to change bits single-handedly, simplifying your workflow. Its multi-color design further allows for quick identification of the drill bit you need.
- 【Compact and Convenient】Thanks to its compact size, this 1/4 inch bit holder is easy to carry around. The bit holder allows for easy attachment to various tools, making this a convenient addition to your construction accessories. The Katerk bit holder is cast from high-quality alloy material, promising a long product lifespan. Despite its rugged strength, the bit holder remains lightweight, making it portable.
- 【Cool Christmas Gift For Men Stocking Stuffers】 This screwdriver bit holder, driver bit holder, impact bit holder, can be given as a gift to your loved one, especially for anyone involved in construction or electrical work. It's a must-have for stocking stuffers for men and women, tools gifts for dad, tech gadgets for men, gifts for dad, gifts for him, gifts for husband, gifts for boyfriend, cool gadgets for men, and cool gifts for dad.
Investigate starvation intervals
Use the trace to locate gaps between kernels, copies, and synchronization calls. NVIDIA’s documented GPU Starvation rule marks idle periods longer than 500 milliseconds by default, and that threshold is adjustable. It is an analysis-rule setting, not an industry-wide definition of an underutilized GPU.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallFor each suspicious gap, inspect:
- CPU samples around the launch or copy that should have filled the gap.
- Operating-system runtime blocked-state backtraces showing threads waiting on locks, I/O, scheduling, or other conditions.
- Synchronization-related OS runtime APIs that may serialize otherwise independent work.
- NVTX ranges that identify data loading, preprocessing, batching, communication, and model stages.
- The relationship between the gap and application throughput or tail latency.
This process distinguishes a GPU that is genuinely waiting for work from one that is busy with a low-resource operation. It also exposes host-side launch delays and synchronization that fleet telemetry cannot attribute to a source function.
Step 6: Escalate to kernel-level profiling
Use Nsight Compute when the question has become kernel-specific: which kernel consumes the interval, which instruction or pipeline is limiting it, or which source line needs attention. DCGM identifies device-level patterns; Nsight Systems maps time and API activity; Nsight Compute supplies detailed kernel attribution. Escalate only after the representative phase and affected device or replica are known, so the profile answers a bounded question instead of producing an unmanageable capture.
Record the exact model phase, input shape, replica, device or GPU instance, software versions, and start and end times with the profile. Those details let you match a kernel trace to the DCGM series and avoid attributing a profile from warm-up or validation to a production phase.
Step 7: Prevent counter conflicts before profiling
DCGM profiling metrics and NVIDIA developer profilers can contend for the same hardware counters. Before starting Nsight Systems or Nsight Compute, check whether DCGM profiling watches are active. NVIDIA documents pausing DCGM profiling collection during the developer-profiler session and resuming it afterward.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #4
- Long Nib and Deep Hole Marker: Our mechanical carpenter pencil with 45mm nib is designed for easy marking of deep holes or narrow areas. These construction pencils are the great choice for woodworking tools, construction tools, carpenter tools, contractor tools, wood carpentry tools and architect tools
- Extra Refills in 2 Colors for Versatile Marking: The construction mechanical pencil comes with 12 extra 2.8mm refills, including 6 red and 6 black refills. The black refill is suitable for light surfaces, while the red wax is perfect for dark surfaces. Our carpenter mechanical pencil makes sure that you'll have an ample supply for extended use
- Built-in Sharpener: Our construction pencil comes with a built-in sharpener to ensure the mechanical pencil tip is always sharp and ready for use. Never buy an extra pencil sharpener again. A great tool for any woodworker pencil, contractor pencils. The refill can easily be extended or retracted with a simple click of the pencils mechanical, allowing you to work more efficiently and accurately
- Portable Clip Design: Our deep hole construction pencil features a portable clip design, easy to carry and attach to your pocket or tool box, so that you can keep the carpenter pencils mechanical close at hand, making it a convenient tool to have on the go. Great gifts choice for carpenters
- Stronger Pencil Lead: The black refills are made of lead, sturdy and smooth. The red refills are made of wax, clear and light. These marking pencils are much thicker and stronger than normal pencils during the marking process of construction work, suitable for various surfaces, such as glasses, metal, boards, floors, walls, furniture, etc. The written marks can be easily wiped with a wet paper towel when needed
The pause applies to the entire host engine, not merely one exporter process. Profiling fields return blank values while collection is paused. Coordinate with every monitoring user on that host, record the pause interval, and expect a deliberate gap in fleet profiling data. Resume collection after the capture and verify that the expected fields have returned.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Enterprise interpretation: scope, entities, and versions
Physical GPUs versus GPU instances
In systems using partitioned GPUs, a physical device and a GPU instance are different measurement entities. Confirm which entity a field supports and which label the exporter emits before summing or averaging values. Comparing a full device with one instance can create a false imbalance even when scheduling is correct.
Hardware and driver support
DCGM field availability depends on the accelerator, driver, DCGM version, entity, and permissions. Keep a compatibility record for each node pool. After an upgrade, compare the actual endpoint output with the previous release and review dashboards for missing series rather than treating missing data as zero utilization.
Aggregation and alerting
Aggregate only comparable series. A fleet average can hide one saturated replica and several idle replicas; a maximum can overreact to a short burst. Pair alerts with a minimum observation window, throughput or latency, and the entity labels needed for an operator to locate the affected workload.
A practical operating runbook
- State the incident in workload terms. Record the affected model or service, phase, throughput or latency symptom, replica set, and time window.
- Check exporter health. Verify that the Prometheus endpoint is reachable and that expected DCGM fields are present for the affected entities.
- Compare related signals. Review SM, tensor, DRAM, occupancy, engine, and interconnect activity together with application throughput.
- Compare peers. Check other replicas, GPU instances, nodes, and workload phases using the same entity definitions.
- Classify the symptom. Decide whether the evidence is primarily idle time, memory or pipeline pressure, interconnect activity, or an observability gap.
- Capture a timeline. Use Nsight Systems for the affected phase and inspect CPU samples, blocked-state stacks, synchronization APIs, and NVTX ranges around gaps.
- Profile a kernel if needed. Use Nsight Compute when the timeline identifies a kernel-level question.
- Coordinate counters. Pause DCGM profiling before the developer-profiler capture, notify other host-engine clients, and resume collection afterward.
- Document the conclusion. Separate what was measured, what was inferred, and what remains unverified. Include field definitions, versions, entities, capture times, and throughput.
Common mistakes and their fixes
Calling 90 percent utilization proof of good performance
Fix the interpretation by checking which signal produced 90 percent, its interval, and the corresponding throughput. High SM activity can include warps stalled on memory, while high time utilization can come from a small copy.
Best Value
- Milwaukee Ink all Fine Point Marker, Black, 4 Per Pack
- 4 per pack Features Clog Resistant Marker Tip Writes through Dusty, Wet and Oily Surfaces Durable Marker Tip for Writing on Concrete, OSB and Rough Surfaces
- Clog resistant tip writes on dusty, wet and oily surfaces and is optimized for rough surfaces such as OSB, cinderblock and concrete
- Hard hat clip- attaches for easy access
- Quick dry time with reduced smearing and marking
Using one exporter field as a bottleneck detector
Fix the dashboard by pairing the field with related counters and an application metric over a representative phase. A single field is a clue, not a diagnosis.
Treating absent metrics as zero
Fix the data pipeline by distinguishing missing, unsupported, and zero values. Inspect hardware, driver, DCGM version, entity, permission, configuration, and endpoint output.
Comparing unlike entities
Fix the aggregation by separating physical GPUs, GPU instances, processes, replicas, and phases. Verify labels before calculating fleet statistics.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteProfiling while DCGM owns the counters
Fix the procedure by coordinating a host-engine-wide pause, recording the resulting blank interval, running the developer profiler, and verifying DCGM recovery afterward.
Assuming a 500-millisecond gap is universally bad
Fix the rule interpretation by treating 500 milliseconds as Nsight Systems’ adjustable GPU Starvation threshold. Whether a gap is harmful depends on the workload’s latency and throughput objectives.
Choosing the right tool for the question
| Question | Best starting point | Measurement style | Operational note |
|---|---|---|---|
| Which nodes, devices, or instances are underused or imbalanced? | DCGM with DCGM Exporter and Prometheus | Continuous or sampled interval-average hardware fields | Confirm field, entity, hardware, driver, permission, and version support. |
| When does GPU work disappear or stall? | Nsight Systems | API-trace timeline and CPU or synchronization attribution | Its utilization rules measure time, not consumed GPU resources. |
| Which kernel, instruction, or source line limits execution? | Nsight Compute | Application and kernel-level profiling | Coordinate hardware-counter access with DCGM before capture. |
| Did a change improve useful work? | All three, correlated with application metrics | Fleet trend plus timeline and kernel evidence | Compare the same workload phase, configuration, and entity. |
What a defensible bottleneck finding looks like
A strong finding names the workload phase and entity, gives the observed telemetry, and connects it to throughput or latency. For example: a specific replica showed repeated idle intervals during a steady-state inference window; an Nsight Systems trace placed those intervals after a CPU-side synchronization call; and application throughput dropped during the same windows. That is stronger than saying the GPU was “only 40 percent utilized.”
Likewise, a memory-bound conclusion should identify the measured signals and phase: elevated DRAM activity, the related SM or tensor behavior, a throughput impact, and kernel-level evidence if the remediation depends on a particular operation. If the counters disagree or fields are unavailable, report the uncertainty instead of forcing a category.
Summary
Measure GPU utilization as a set of defined signals tied to application output. Use DCGM and DCGM Exporter to establish fleet visibility, validate the fields and entities that are actually available, and compare SM, tensor, memory, occupancy, engine, and interconnect activity over representative phases. Use Nsight Systems to explain idle time and CPU or synchronization gaps, and Nsight Compute to attribute a confirmed kernel-level problem. Coordinate counter ownership so profiling does not silently remove DCGM data. This layered method turns a vague utilization percentage into an evidence-backed escalation path.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
