Benchmark AI inference hardware by measuring the complete serving stack—not the accelerator in isolation—under a workload that matches your application. Hold the model, token lengths, software, and serving settings constant; sweep concurrency from light load to saturation; record time to first token (TTFT), end-to-end latency, inter-token speed, aggregate throughput, failures, and power; then compare cost at the same latency or service-level target.
A peak tokens-per-second figure is not a buying answer by itself. It can represent a heavily batched system that feels slow to an individual user, while a lower-throughput configuration may satisfy an interactive latency target at a better cost. The procedure below produces results you can reproduce and compare.
As an Amazon Associate I earn from qualifying purchases.
Start with the decision you need to make
Write down the deployment question before running a test. An interactive assistant, an offline summarization queue, and a real-time vision API need different operating points.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Interactive chat
Set a ceiling for a high-percentile TTFT, a maximum acceptable inter-token interval (ITL) or time per output token (TPOT), and a completion-latency target. Also define the number of simultaneous users or the request rate you must support. A median-only result can hide users who wait much longer.
#1 Best Overall
- Various Monitoring Parameters: The power meter plug can monitor the power (W), energy (kWh), volts, amps, hertz, power factor, cost, minimum and maximum power (W), cumulative days and time of your appliances. By switching 7 display modes, you can easily know the various parameters while the appliance is working. The home energy monitor can also calculate and display how much power your appliance uses and how much electricity bill it cost in cumulative time
- Upgraded LCD Display: With large screen size 2.36 inch x 1.85 inch, clearer monitor backlit, our electrical usage monitor can display the data clearer and more visible no matter day or night. 180°full wide viewing angles is great for reading and recording the data in any angles. No need to stand on the front of the display and bend over to read the numbers
- Adjustable Backlight Time: Our upgraded watt meter has 5 options of backlight time. The default backlight time duration is 10 minutes(bL-0). If you want to change the backlight time, you can press and hold "UP" and "DOWN" button at the same time to enter backlight time setting, then press "UP" and "DOWN" to select the backlight time (bL-0 =10 minutes, bL-1=1 hour, bL-2=4 hours, bL-3=8 hours, bL-4=always on), finally press the "COST" to save the backlight time settings
- Overload protection: When the power of the appliance exceeds the overload power, the LCD will display “OVERLOAD” to warn the user. All the buttons will quit working and can only be workable when you lower or remove the load power. The default overload power is 3680W and is adjustable from 0 to 3680W. In general, you need to set the overload power to 1800W before using. Just press the "function" button for more than 3 seconds to enter the setting
- Data Memory Function: The wattage meter will record your power consumption data when you remove it from socket, or remove appliances from the electricity monitor. You can directly see the last data when you use it next time. This function can also automatically save the data when there is a sudden power failure
Batch or offline inference
Prioritize completed tokens or requests per second, total job time, and cost per useful output. You can usually tolerate higher queueing latency, but document the batch size, scheduling policy, and maximum job duration.
Mixed or API traffic
Define both a capacity target and a user-facing latency limit. Record the traffic mix, burst pattern, and whether requests are allowed to queue. MLPerf Endpoints presents throughput, interactivity, P95 TTFT, and concurrency as related dimensions rather than a single score: MLPerf Endpoints.
Choose the pass/fail rule in advance—for example, P95 TTFT below a stated limit while sustaining a required output-token rate. Stop increasing load when that rule is breached or when additional concurrency no longer raises completed throughput.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Lock the workload before comparing hardware
Every workload variable can change the result. Record these items in a run manifest and use the same values on every system.
- Model name, exact revision or checkpoint, tokenizer, and context window.
- Precision or quantization format, including any weight-only or activation quantization.
- Input and output token-length distributions, averages, and percentiles.
- Dataset or prompt source, with a representative sample retained for repeatability.
- Decoding settings such as temperature, top-p, beam settings, repetition penalties, and stop conditions.
- Whether outputs are fixed length or stop at an end-of-sequence token.
- Request arrival pattern, concurrency or offered rate, and whether retries are enabled.
A synthetic prompt set is useful for isolating system behavior, but disclose that it is synthetic and make its length distribution resemble production before using it for procurement. Never compare one accelerator with short prompts and another with long context or different output lengths. Token counts must accompany request-per-second figures because two requests can represent radically different work.
Record the complete system configuration
The tested system is the accelerator plus its host, interconnect, software, and serving policy. Capture the following for every run:
Rank #2
- Multi-function power monitor: Our electric usage monitor can monitor the power (W), electricity(kWh), voltage(V), frequency(Hz), current(A), power factor(PF), unit price($/kWh), total cost($) of your appliances. By switching 8 display modes, you can easily know the various parameters while the appliance is working. The “electricity” mode can calculate and display how much power your appliance uses. And the “total cost” mode will show how much electricity bill it cost in cumulative time
- Overload protection: When the loading power of the appliance is over the default overload threshold 1800W, the whole display with backlight and the word “OVERLOAD” will keep flashing to warn the users. Please turn off the appliance for safety concern
- Premium Material: The whole body of our energy meter is made of high-quality ABS material. It makes our home electricity usage monitor more long lasting, fireproof and anti-drop. The standard US socket and plug is suitable for all US standard appliances
- Backlight Display: With white backlight and black words, the LCD display of our home power monitor can display the data clearer and help you to read the data easier no matter day or night. The backlight will only lights up when the device is connected to AC power. If no button is pressed, the backlight turns off automatically after 10 minutes. You can also press the "UP" button to turn off the backlight manually, and press any button to turn on backlight again
- Easy to reset: No reset tool needed, our appliance power usage meter is easy to reset. You can press the “M” button for 5 seconds directly to reset the device. After reset, all cumulative data (electricity quantity, cost) will be cleared, and all settings will be restored to factory settings
- Accelerator model, memory capacity, accelerator count, and topology.
- CPU model and count, host memory, storage, network adapter, and interconnect.
- Power limits, clock or performance settings, and thermal conditions.
- Operating-system version, accelerator driver, runtime, and inference-engine versions.
- Tensor, pipeline, data, or expert parallelism settings.
- Batching, continuous-batching, queue, and scheduler limits.
- Kernel fusions, compilation options, KV-cache settings, and quantization implementation.
Vendor documentation can explain how a published test was run, but that number is a measurement of the vendor’s stated configuration, not hardware-only performance. TensorRT-LLM’s benchmark workflow starts with a prepared dataset and separates throughput and low-latency modes; its documentation also identifies the benchmarking suite as a work in progress: TensorRT-LLM benchmarking documentation.
Free tools Windows power users keep installed
One-click scans. No signup required.
Use latency metrics that describe the user experience
For streaming language-model serving, report all three timing views. NVIDIA’s definitions are documented in NIM benchmarking metrics.
| Metric | What it measures | Why it matters | How to report it |
|---|---|---|---|
| TTFT | Query submission until the first output token is received, when the response contains a token. | Prompt processing, queueing, batching, and the wait before visible output. | Median and high percentiles such as P95; state the client and network boundary. |
| End-to-end latency | Submission until the complete response is received. | Total user wait, including queueing, generation, and documented network time. | Report distribution, output length, and whether failed or timed-out requests are excluded. |
| ITL / TPOT | Average interval between generated tokens; AIPerf generally excludes TTFT. | Per-user streaming pace after the first token. | State whether TTFT is excluded, how requests are weighted, and the exact tool version. |
| System output TPS | Aggregate generated output tokens per second across concurrent requests. | Total serving capacity at a selected load. | Pair with concurrency, token lengths, latency percentiles, and service target. |
| Requests per second | Successfully completed requests during the measurement interval. | API capacity when request sizes are comparable. | Include input/output token distributions; request rate alone can mislead. |
Metric names do not guarantee identical calculations. NVIDIA notes that GenAI-Perf and LLMPerf differ in ITL treatment and benchmark-duration boundaries. The NIM documentation states: “Tool implementations vary, so compare results only when definitions align.” Use one tool and version across systems, or publish the formulas and time windows beside the results.
Warm up, then sweep load to saturation
A single run at one concurrency is not a benchmark. Follow this sequence for each system.
- Validate correctness. Run a small sample and verify tokenization, output quality, stop behavior, and numerical or task-level acceptance criteria. A faster system producing invalid output is not equivalent.
- Warm the stack. Separate model loading, compilation, memory allocation, and cache warm-up from the measured interval. State the warm-up duration or condition and ensure every system reaches the same readiness state.
- Measure low load. Start with one request or the lowest realistic concurrency. This reveals responsiveness without queueing and gives a baseline for TTFT, ITL/TPOT, and completion latency.
- Increase offered load in controlled steps. Raise concurrency or request rate gradually. Use the same sequence and dwell time on every system. Keep a steady-state measurement window at each step.
- Collect the same fields. At every point record successful requests, output-token throughput, TTFT, ITL/TPOT, end-to-end latency, concurrency, queue time when available, and failed, timed-out, or rejected requests.
- Continue until the target fails. Stop when a percentile limit is exceeded, errors appear, or completed throughput plateaus or falls. Record the first violating point and the last passing point.
- Repeat enough to expose variability. Keep the workload and configuration fixed across repetitions. Report the aggregation method and any run discarded for a documented operational reason.
Label throughput as offered, completed, or measured only during a steady-state window. A client that sends 100 requests per second is not necessarily served at 100 requests per second if requests queue or time out.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsRead the load curve instead of chasing a peak
Plot concurrency or offered rate on the horizontal axis and completed throughput, TTFT, ITL/TPOT, and end-to-end latency on separate vertical axes or charts. Throughput normally rises with concurrency until available compute saturates; beyond that point it can flatten or decline while latency keeps increasing. This behavior is described in the NIM metrics documentation.
Rank #3
- Various Monitoring Parameters: The power energy meter can monitor the power (W), energy (kWh), volts, amps, hertz, power factor, cost, minimum and maximum power (W), cumulative days and time of your appliances. By switching 7 display modes, you can easily know the various parameters while the appliance is working. The home energy monitor can also calculate and display how much power your appliance uses and how much electricity bill it cost in cumulative time
- Upgraded LCD Display: With large screen size 2.36 inch x 1.85 inch, clearer monitor backlit, our electrical usage monitor can display the data clearer and more visible no matter day or night. 180°full wide viewing angles is great for reading and recording the data in any angles. No need to stand on the front of the display and bend over to read the numbers
- Adjustable Backlight Time: Our upgraded watt meter has 5 options of backlight time. The default backlight time duration is 10 minutes(bL-0). If you want to change the backlight time, you can press and hold "UP" and "DOWN" button at the same time to enter backlight time setting, then press "UP" and "DOWN" to select the backlight time (bL-0 =10 minutes, bL-1=1 hour, bL-2=4 hours, bL-3=8 hours, bL-4=always on), finally press the "COST" to save the backlight time settings
- Overload Protection: When the power of the appliance exceeds the overload power, the LCD will display “OVERLOAD” to warn the user. All the buttons will quit working and can only be workable when you lower or remove the load power. The default overload power is 3680W and is adjustable from 0 to 3680W. In general, you need to set the overload power to 1800W before using. Just press the "function" button for more than 3 seconds to enter the setting
- Data Memory Function: The wattage meter will record your power consumption data when you remove it from socket, or remove appliances from the electricity monitor. You can directly see the last data when you use it next time. This function can also automatically save the data when there is a sudden power failure
Choose a shared operating point
Select the highest load at which each system meets the same service rule, such as P95 TTFT below a ceiling and P95 completion latency below another ceiling. Compare completed output TPS, successful requests, power, and cost at that point. If systems cannot share one point, show the full curves and identify the target each one can satisfy.
Separate system capacity from per-user speed
Aggregate output TPS can increase as more users are added even while each user’s token stream slows. Report per-request ITL/TPOT or tokens per second alongside aggregate TPS. This answers both “How many users can it serve?” and “How fast does one user see tokens?”
Include tail behavior
Median latency describes a typical request; P95 or another declared percentile exposes queueing and contention. State the percentile, sample count, and whether errors and timeouts are included in the denominator. Do not silently remove slow requests.
Compare hardware at matched quality and software conditions
Keep model output quality and acceptance criteria constant. The comparison should include:
| Axis | Required disclosure |
|---|---|
| Model and quality | Checkpoint, tokenizer, precision or quantization, decoding settings, and task-quality threshold. |
| Scale | Accelerator model and count, host system, memory, and interconnect. |
| Software | Inference engine, runtime, driver, kernel or compiler optimizations, parallelism, batching, and scheduler. |
| Responsiveness | TTFT and end-to-end latency percentiles at stated concurrency. |
| Streaming speed | ITL/TPOT or per-user output-token rate, with its formula. |
| Capacity | Completed request and output-token throughput at the target latency. |
| Power | Measurement boundary, sampling method, and whether readings cover the accelerator or whole system. |
| Economics | Cost boundary, utilization assumption, accounting period, and cost per successful useful output. |
| Availability | Whether the result is a standard benchmark submission, cloud instance, owned server, or another category. |
Calculate inference cost without hiding assumptions
First state what “cost” includes. A cloud comparison can use the actual rental or service price for a named region and pricing date. An owned-server comparison should state purchase price, amortization period, expected utilization, and included operating costs. Hardware purchase price alone is not a universal cost-per-token figure.
Cost per output token
For a defined measurement period:
cost per output token = total cost assigned to the period ÷ successfully generated output tokens in that period
Rank #4
- ✓ Backlit LCD Display: The watt meter features an upgraded backlit LCD display with a 160° wide viewing angle, providing clear and distinct data readings from any angle, day or night. (Note: To conserve energy, the backlight will automatically turn off after 15 seconds of continuous use.)
- ✓ 9 Types of Data Display: This energy meter plug accurately monitors a wide range of electrical parameters, including power (W), apparent power (VA), energy consumption (kWh), usage time (hours), power frequency (Hz), power factor (PF), voltage (V), current (A), and electricity cost. The user-friendly interface allows you to easily read and navigate through 9 types of data with a single button.
- ✓ Save Money and Energy: By toggling through the 9 display modes, you can gain insights into various operational parameters of your devices. The home energy monitor also calculates and displays the electricity consumption and cost over time, helping you save on your electricity bills by monitoring and managing energy usage.
- ✓ Data Storage: The plug-in electricity meter stores all non-real-time data, ensuring no information is lost in the event of a sudden power outage. If data exceeds the screen’s maximum display range, the screen will flash as a reminder. Pressing and holding the reset button for 3 seconds will reset the cumulative data to zero.
- ✓ Safe to Use: Rated voltage 125VAC, 60Hz, 15A maximum 1875W resistor and tungsten wire, 1/2HP. Clean and tidy interface, very simple to use.
Use the same operating point that produced the latency and throughput numbers. Count only successful, accepted output unless you explicitly present a separate failed-output metric. State whether the numerator includes rental, amortized capital, electricity, cooling, facility overhead, software fees, support, and networking.
Cloud accounting
Record provider, instance or service type, region, pricing date, billing granularity, minimum commitments, and idle time. If the benchmark ran below full utilization, do not extrapolate a fully utilized cost without showing the utilization assumption.
Owned hardware accounting
Document purchase cost, useful life or amortization period, expected duty cycle, maintenance, and facility expenses. A server that is fast but idle most of the time can have a higher cost per useful token than a slower system with sustained utilization.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Measure power at a declared boundary
Power results are meaningful only when the boundary is clear. Say whether the reading covers one accelerator, the complete server, or the facility. Also state whether it is instantaneous, average, or integrated energy over the benchmark window.
MLPerf Datacenter describes average AC power for the entire system measured at the wall during performance measurement. Its result pages are available at MLPerf Datacenter results. A plug-in AC power meter can work for a compatible workstation, but verify electrical ratings, measurement accuracy, and facility requirements; a consumer meter is not automatically suitable for a multi-GPU server or rack circuit.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallUse standardized and application benchmarks together
MLPerf Inference is designed for architecture-neutral, representative, reproducible comparison, with published rules, reference software, and submission requirements: MLCommons MLPerf Inference. Standardized results are useful for comparing submitted systems under a defined workload and scenario. They do not replace a deployment test using your model, traffic shape, latency target, and serving stack.
Best Value
- Various Monitoring Parameters: The power meter plug can monitor the power (W), energy (kWh), volts, amps, hertz, power factor, cost, minimum and maximum power (W), cumulative days and time of your appliances. By switching 7 display modes, you can easily know the various parameters while the appliance is working. The home energy monitor can also calculate and display how much power your appliance uses and how much electricity bill it cost in cumulative time
- Upgraded LCD display: With large screen size 2.36 inch x 1.85 inch, clearer monitor backlit, our electrical usage monitor can display the data clearer and more visible no matter day or night. 180°full wide viewing angles is great for reading and recording the data in any angles. No need to stand on the front of the display and bend over to read the numbers
- Adjustable Backlight Time: Our upgraded watt meter has 5 options of backlight time. The default backlight time duration is 10 minutes(bL-0). If you want to change the backlight time, you can press and hold "UP" and "DOWN" button at the same time to enter backlight time setting, then press "UP" and "DOWN" to select the backlight time (bL-0 =10 minutes, bL-1=1 hour, bL-2=4 hours, bL-3=8 hours, bL-4=always on), finally press the "COST" to save the backlight time settings
- Overload protection: When the power of the appliance exceeds the overload power, the LCD will display “OVERLOAD” to warn the user. All the buttons will quit working and can only be workable when you lower or remove the load power. The default overload power is 3680W and is adjustable from 0 to 3680W. In general, you need to set the overload power to 1800W before using. Just press the "function" button for more than 3 seconds to enter the setting
- Data Memory Function: The wattage meter will record your power consumption data when you remove it from socket, or remove appliances from the electricity monitor. You can directly see the last data when you use it next time. This function can also automatically save the data when there is a sudden power failure
Benchmark rounds and available systems change. MLCommons announced v6.1 on September 16, 2026, including new tests for emerging inference patterns. The announcement reports a 2.99× improvement for the best per-accelerator server-scenario result in its VLM benchmark over v6.0; that is a result for those rounds, workloads, and configurations—not a universal hardware gain: MLPerf Inference v6.1 announcement. Verify the applicable round, quality target, scenario rules, result artifact, and system availability category before quoting a leaderboard.
Why your result differs from a vendor benchmark
Differences are usually methodological rather than evidence that one party is wrong. Check these causes in order:
- Different model revision, precision, quantization, tokenizer, or kernel implementation.
- Different prompt and output lengths, stop behavior, or decoding settings.
- Different concurrency, request-arrival pattern, batch limits, or scheduler.
- One result reports offered throughput while the other reports completed throughput.
- Warm-up, compilation, cache state, or benchmark-duration windows differ.
- TTFT, ITL/TPOT, or end-to-end latency formulas differ.
- One test measures a complete server and network path while the other measures an accelerator-local client.
- Power limits, clocks, thermal conditions, host CPU, interconnect, or driver versions differ.
- Failed, timed-out, or queued requests were excluded in one result.
Compare run manifests and formulas before comparing headline numbers. The NVIDIA overview also distinguishes benchmarking from load testing: “Performance benchmarking and load testing answer different questions:” NIM benchmarking overview. A benchmark characterizes a controlled configuration; a load test examines behavior under realistic traffic, failure, and recovery conditions.
Recommended Free Tools
Publish a result others can reproduce
A useful report lets a reader reconstruct the operating point and accounting boundary. Include:
- A system and software manifest with versions and configuration files.
- The prompt or dataset description, token-length distributions, and decoding settings.
- Warm-up procedure, measured-window duration, repetition count, and client location.
- Concurrency or offered-rate sweep, with successful, failed, timed-out, and queued request counts.
- TTFT, ITL/TPOT, end-to-end latency, requests per second, and output TPS at each load.
- The selected service target and the exact point used for hardware and cost comparison.
- Power boundary and measurement method.
- Cloud or ownership cost assumptions, utilization, pricing region and date, and included overhead.
- Tool names, versions, metric formulas, and any deviations from standard benchmark rules.
Present a load-curve chart or table, not only the maximum number. If a metric is unavailable, write “not stated” and explain why; do not substitute a value from a different run.
The Bottom Line
The fair comparison is the system that meets your stated latency and quality target at the required load with the lowest transparent cost—not the system with the largest isolated peak-throughput number.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
