Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteSome links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
To benchmark AI inference throughput on NVIDIA DGX and RTX systems credibly, hold the model checkpoint, precision, tokenizer, prompt and output distributions, software stack, GPU count, parallelism, clocks, and workload policy constant. Warm up the server, measure a stable window repeatedly, and report tokens per second together with time to first token (TTFT), inter-token latency (ITL), requests per second (RPS), and tail latencies. Use trtllm-bench throughput for maximum-load offline capacity, trtllm-bench latency for low-latency tests, GenAI-Perf for an OpenAI-compatible online endpoint, and MLPerf Inference when you need a prescribed, externally comparable result. A DGX result and an RTX result are comparable only when GPU count, topology, parallelism, precision, software versions, and the definition of throughput are explicitly matched.
What does an AI inference throughput benchmark need to prove?
A useful benchmark answers a specific capacity or user-experience question, rather than producing one impressive number. You may want to know the maximum output-token rate under saturation, how many requests a service can complete per second, whether a model meets a first-token target, or how many GPUs are needed for a stated service-level objective (SLO). Those are different tests and can produce different winners.
As an Amazon Associate I earn from qualifying purchases.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallBefore running anything, write down the comparison you intend to make. For example: “Compare total output tokens per second at P95 TTFT below 500 ms,” or “Find the highest sustainable RPS while P99 end-to-end latency stays below two seconds.” A maximum-load result is not a latency guarantee, and a low-latency result is not a capacity ceiling.
Freeze the variables that change performance
Record the exact model name and checkpoint revision, tokenizer, precision or quantization, runtime and driver versions, GPU model and count, interconnect and topology, tensor and pipeline parallelism, batch or concurrency policy, request rate, KV-cache settings, and power and clock configuration. Include whether the reported throughput is per GPU or for the complete system.
#1 Best Overall
- [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
- [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
- [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
- [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
- [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.
NVIDIA’s TensorRT-LLM guidance recommends persistence mode and standardized clock, power-limit, and boost settings for rigorous comparisons. Without those controls, a clock change, thermal state, or power cap can look like a software or hardware improvement.
Use a deterministic workload
Every request in the benchmark should be reproducible. TensorRT-LLM’s throughput schema requires a task ID, either a prompt or input IDs, and an output-token count. Keep input and output lengths representative of the application. Synthetic 128-input/128-output requests are convenient for repeatability, but they do not represent every workload: chat, retrieval-augmented generation, summarization, and code completion have different length distributions.
Free tools Windows power users keep installed
One-click scans. No signup required.
Store the generated request file with the result. If you change the tokenizer, prompt template, stop conditions, or output-length distribution, treat it as a new workload rather than a continuation of the old series.
How should you prepare a DGX or RTX system before measuring?
- Identify the hardware. Record the precise GPU variant, memory capacity, count, interconnect (for example, NVLink or PCIe), NUMA placement, and whether the result is per GPU or aggregate.
- Pin the software. Save the operating-system image, NVIDIA driver, CUDA version, TensorRT-LLM or PyTorch version, serving layer, tokenizer package, and model commit or checksum.
- Set power and clocks. Enable persistence mode and apply the same documented power limit and clock or boost policy to every comparison system. Record ambient and thermal conditions when they can affect sustained clocks.
- Configure parallelism. Write down tensor-parallel and pipeline-parallel degrees, rank placement, communication settings, and the GPU-to-process mapping. A multi-GPU DGX run and a single- or dual-GPU RTX run are different configurations even when they use the same model.
- Configure memory behavior. Record maximum sequence length, KV-cache data type and capacity, paged-attention settings, and any admission or eviction policy. Out-of-memory failures and cache evictions must be reported, not silently excluded.
- Check isolation. Stop unrelated GPU jobs, confirm that clocks are stable, and capture utilization, memory use, temperature, and power telemetry during the run.
Build or load the engine only after these settings are fixed. If the engine is autotuned, retain the build log and tuning choices; rebuilding with different tactic selections can change the result.
Which NVIDIA tool should you use for inference benchmarking?
| Question | Recommended mode | What it measures | When to use it |
|---|---|---|---|
| How fast can an offline engine run at saturation? | trtllm-bench throughput |
Maximum-load throughput and summary latency statistics | Capacity upper bounds, engine and kernel tuning |
| Can the system meet a low-latency target? | trtllm-bench latency |
Latency-oriented behavior with controlled load | Interactive or latency-sensitive inference |
| How will an online API behave? | GenAI-Perf | Client-observed TTFT, ITL, end-to-end latency, TPS, and RPS | OpenAI-compatible production or staging endpoints |
| Do I need a prescribed public comparison? | MLPerf Inference | Results under a fixed, published benchmark specification | Standardized hardware and software comparisons |
trtllm-bench can build tuned engines, run Python-based workflows, and emit throughput and latency summaries. For an online test, expose an OpenAI-compatible service and drive it with GenAI-Perf or TensorRT-LLM’s serving benchmark. Save the tool’s CSV or JSON output instead of copying only a headline number.
Rank #2
- 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
- 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
- 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
- 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
- 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
Offline versus online measurement
Offline tests submit work to an engine without modeling a user’s arrival pattern. They are useful for a maximum-load upper bound and for isolating runtime changes. Online tests include client scheduling, queueing, network and serialization overhead, admission control, and streaming behavior. They are the appropriate evidence for a deployed endpoint.
Do not present an offline TPS number as the capacity of an API unless the online path has been measured. Conversely, do not dismiss online overhead when selecting hardware for a service: it is part of the user’s observed latency and sustainable RPS.
How do you run a repeatable throughput benchmark?
- Create the request dataset. For each task, include the required ID, prompt or input IDs, and target output-token count. Use a fixed seed where sampling is enabled, and publish the prompt and output-length distributions.
- Build or select the engine. Keep precision, quantization, maximum sequence length, and parallelism identical across systems. Capture build-time warnings and tactic-selection logs.
- Start the service or benchmark process. Verify that every intended GPU is visible and that the process-to-GPU mapping matches the recorded topology.
- Warm up. Send enough requests to populate caches and reach steady clocks. Do not include compilation, engine loading, CUDA graph capture, or first-request initialization in the representative interval.
- Choose the load sweep. For offline capacity, use maximum load. For online behavior, sweep concurrency or request rate from lightly loaded to saturated. Keep each level long enough to observe stable queueing and tail latency.
- Measure a stable window. GenAI-Perf uses a sliding window so warm-up and cool-down requests are excluded from representative statistics. Use an equivalent exclusion rule when your tool does not provide one.
- Repeat each point. Run multiple trials, preserve every raw output file, and report the median or distribution across trials. Investigate outliers instead of deleting them.
- Validate accounting. Confirm whether token counts include only generated tokens, whether request preparation is inside the timed interval, and whether aborted or failed requests are counted.
- Archive the recipe. Keep the dataset, command line, environment capture, engine artifacts, logs, telemetry, and result files together. A number without its recipe cannot be reproduced.
When a run is interrupted by an out-of-memory error, timeout, server restart, or queue overflow, mark that load point as a failure. Do not replace it with a lower-load result and label the system “stable.”
Which metrics should you report, and how are they calculated?
Throughput and latency describe different parts of the same request timeline. Report both aggregate and tail values so a high average does not hide an unacceptable user experience.
| Metric | Definition | What it tells you |
|---|---|---|
| TTFT | Time from request submission until the first output token | Perceived responsiveness and queueing or prefill cost |
| ITL (also called TPOT) | Average delay between consecutive output tokens; GenAI-Perf excludes the first token from this interval | Streaming token cadence after generation begins |
| End-to-end latency | TTFT + generation time |
Total wait until the response is complete |
| TPS | Total generated output tokens divided by the stated measurement interval | Token capacity; define exactly what the interval includes |
| RPS | Completed requests divided by the stated measurement interval | Request-serving capacity |
| P50, P90, P95, P99 | Latency percentiles across requests | Typical and tail behavior under the tested load |
Always state whether TPS is aggregate or per GPU, whether input tokens are excluded, and whether the interval includes request preparation, queueing, network transfer, and post-processing. Tools differ in these boundaries, so two values labeled “tokens per second” can measure different things.
Rank #3
- [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
- [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
- [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
- [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
- [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by
For streaming systems, publish TTFT and ITL alongside TPS. A system can generate many tokens per second in aggregate while making users wait too long for the first token or producing an uneven token cadence. Report at least median, P90, P95, and P99 latency where the tooling supports it.
How should you compare DGX and RTX results?
Compare systems at equivalent operating points, not by placing one headline number beside another. DGX configurations often combine several GPUs and a high-bandwidth interconnect; an RTX result may use one GPU or a different topology. State GPU count, interconnect, tensor and pipeline parallelism, precision, model, software versions, dataset, and whether throughput is total or per GPU.
| Comparison axis | Correct method | Why it matters |
|---|---|---|
| Throughput at a latency budget | Find the highest TPS or RPS while TTFT and P95/P99 stay below the same limits | Matches an interactive service SLO |
| Latency at a throughput target | Hold TPS or RPS constant and compare TTFT, ITL, and end-to-end percentiles | Shows user experience at required capacity |
| Per-GPU versus total | Publish both when possible and label the denominator | Prevents a multi-GPU total from being mistaken for single-GPU performance |
| Scaling efficiency | Measure the same workload at one, two, four, and more GPUs; compare observed speedup with ideal linear speedup | Reveals communication and scheduling overhead |
| Power or cost context | Include only measured power, energy, or cost data with its measurement method | Prevents unsupported efficiency claims |
When sweeping concurrency, plot throughput against TTFT, P95/P99 latency, or your stated SLO. The useful operating point is often near the knee of this curve: adding concurrency initially raises throughput, then produces rapidly rising latency with little additional throughput. Choose the highest point that still meets the service target, not necessarily the absolute TPS peak.
Interpreting published NVIDIA reference figures
NVIDIA’s TensorRT-LLM overview publishes output throughput per GPU for systems including H100 SXM in DGX H100, H200 SXM in DGX H200, B200 in DGX B200, GB200 NVL72, and RTX 6000 Pro Blackwell Server Edition. NVIDIA describes these as reference measurements, not guaranteed peak performance, and notes that hardware variants can differ in TDP, memory bandwidth, core count, and features. Use them as directional context, then reproduce the exact workload on your own configuration.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
A TensorRT-LLM documentation example reports 20,685.5510 tokens/sec and 161.6059 requests/sec, with 18,563.6825 ms total latency, for a documented Llama 3.1 8B PyTorch-backend run using 128 input and 128 output tokens. Those figures apply only when the model, backend, dataset, software, and configuration match; they are not universal DGX or RTX limits.
Rank #4
- Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
- Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
- All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
- Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
- Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
When should you use MLPerf instead of a custom benchmark?
Use MLPerf Inference when the goal is a prescribed, externally comparable result. Its evaluations are conducted under specified conditions across hardware, software, and services, which makes the rules, scenarios, and result-compliance requirements part of the measurement.
Use trtllm-bench or GenAI-Perf for application-specific capacity planning, model and runtime tuning, or an OpenAI-compatible production service. A custom test can use your real prompt lengths, arrival pattern, sampling settings, and SLO; that realism is valuable even though the result is not directly interchangeable with an MLPerf submission.
Do not mix a compliant standardized result with an internally tuned application result in one ranking. Label the benchmark class, workload, and constraints so readers know whether they are seeing a standard score or a deployment decision.
What commonly invalidates a DGX-versus-RTX benchmark?
- Different model or checkpoint: Even a minor revision can alter kernels, memory use, or generation behavior.
- Different precision or quantization: FP16, BF16, FP8, and weight-only quantization trade quality, memory, and speed differently.
- Unmatched prompt lengths: Prefill work changes with input length, while decode work changes with output length.
- Unreported concurrency: A saturated batch and a single-request latency test answer different questions.
- Cold-start contamination: Engine loading, compilation, or cache population inflates early latency.
- Unstable clocks or thermal throttling: A later run may be slower for reasons unrelated to the software change.
- Topology blindness: Cross-GPU communication can dominate when tensor or pipeline parallelism spans links with different bandwidth.
- Token-accounting mismatch: One tool may include only generated tokens while another includes preparation or counts failed requests differently.
- Average-only reporting: Mean latency can conceal a severe P99 tail under queueing.
- Cherry-picked load points: Reporting only the best concurrency hides overload behavior and failure thresholds.
Fix these problems by rerunning both systems from the same dataset and software recipe, documenting the difference that remains, and publishing the complete sweep rather than a selected point.
What should a publishable benchmark report contain?
Include a compact configuration block before the charts or tables:
- GPU model, count, memory, interconnect, topology, and per-GPU versus total scope.
- Model checkpoint, tokenizer, prompt template, precision or quantization, and maximum sequence length.
- Runtime, serving stack, driver, CUDA, operating-system image, and engine-build settings.
- Tensor parallelism, pipeline parallelism, batching, concurrency or request-rate sweep, and KV-cache policy.
- Dataset schema, input and output length distributions, sampling parameters, and random seed.
- Persistence, clock, boost, and power-limit settings, plus relevant temperature and utilization observations.
- Warm-up policy, measurement-window definition, number of repetitions, and treatment of failures.
- TPS, RPS, TTFT, ITL, end-to-end latency, and P50/P90/P95/P99 values at every reported load point.
Provide the command line or configuration file and preserve raw CSV or JSON output. If a value is not measured, say “not measured” rather than infer it from another system or from a vendor reference table.
How do you turn benchmark data into a capacity decision?
- Define the service target. Set an arrival rate or RPS, a TTFT limit, an end-to-end latency limit, and acceptable P95 or P99 tails.
- Locate the feasible operating region. From the concurrency or request-rate sweep, discard points that violate any SLO or produce errors.
- Select the knee, not the peak. Among feasible points, choose the one with headroom before latency rises sharply. Leave capacity for burstiness and variance.
- Check scaling. Repeat the workload with additional GPUs and calculate measured speedup and scaling efficiency. Explain any loss through communication, memory, or scheduling observations rather than assuming linear scaling.
- Validate with application traffic. Re-run the chosen configuration using production-like prompt and output distributions and the actual endpoint path.
This process produces a defensible answer to “how many GPUs do we need?” without pretending that one synthetic maximum-load number predicts every deployment.
Frequently Asked Questions
How many warm-up requests should an LLM benchmark use?
There is no universal count because engine compilation, CUDA graph capture, cache population, and thermal stabilization vary by runtime and model. Increase warm-up until clocks, memory use, and latency statistics stop trending, then document the rule and excluded requests rather than publishing a fixed count as a standard.
Should input tokens be included in tokens-per-second?
For generation capacity, most reports use generated output tokens, but the convention must be stated. Input processing still consumes compute and affects TTFT, so publish input length distributions and TTFT even when TPS excludes input tokens.
Why can a faster GPU show worse time to first token?
TTFT includes queueing and prefill, not just decode speed. Different batching, parallelism, tokenizer overhead, network path, or a saturated scheduler can make a faster accelerator deliver a slower first token at the tested load.
How do I compare quantized and full-precision engines fairly?
Treat precision or quantization as a separate experimental variable and keep every other setting fixed. Report quality or accuracy checks separately, because a throughput gain is not an equivalent result if the generated model behavior is materially different.
What should I do when P99 latency is extremely noisy?
Lengthen the steady-state window, repeat the point, and inspect request arrivals, GPU clocks, host scheduling, and network or service logs. Keep the variance visible; replacing noisy tails with an average or a convenient percentile hides the operating risk.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
