Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteSome links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
A useful NVIDIA B200 benchmark is a reproducible experiment, not a single tokens-per-second number. Hold the model checkpoint, tokenizer, quantization, sequence lengths, concurrency, software stack, topology and power settings constant; then report output throughput together with time to first token (TTFT), inter-token latency, end-to-end latency, tail latency, memory use and cost per token. B200’s 180GB of HBM3e and high-bandwidth NVLink make GPU count, parallelism and KV-cache placement especially important. For multi-agent systems, use trace-backed conversations with tool calls, retries, parallel sub-agents and long contexts, and test both aggregated and disaggregated serving. The figures below are recipe-specific baselines, not a universal B200 rating.
What makes a B200 benchmark genuinely comparable?
Two results can be labeled “B200 inference” while measuring different problems. A 4,096-token prompt and 256-token completion at concurrency 8 exercises a different part of the system than a 128,000-token prompt and 2,048-token completion at concurrency 512. Quantization, speculative decoding, prefix caching and tensor parallelism can change the answer as much as the GPU itself.
As an Amazon Associate I earn from qualifying purchases.
Recommended Free Tools
Freeze and publish every variable that can alter execution:
- Model: exact checkpoint, revision, tokenizer and serving template. State whether the model is dense or mixture-of-experts and which experts or parallelism modes are enabled.
- Numerical format: FP4, FP8, BF16 or another format, including weight, activation and KV-cache precision.
- Workload shape: input sequence length (ISL), output sequence length (OSL), batch size or request concurrency, arrival pattern and stop conditions.
- Software: vLLM, TensorRT-LLM, Triton or NIM version; driver and CUDA versions; compiler, kernel and container image identifiers.
- Hardware: B200 form factor, GPU count, SXM or other module, CPU and host memory, NVLink/NVSwitch topology, network fabric and storage.
- Runtime features: tensor or expert parallelism, prefix caching, KV-cache offload, speculative decoding, MTP/EAGLE3, continuous batching and routing policy.
- Power and clock policy: fixed power cap, default policy or an explicitly documented performance mode.
Record a target accuracy or quality threshold. A faster quantized or speculative run is not an apples-to-apples improvement if it produces materially different answers.
#1 Best Overall
- Discrete graphics card memory 40 GB
- Memory bandwidth (max) 1555 GB/s
- Graphics processor family NVIDIA
- Graphics processor A100
What B200 hardware details must the report include?
Memory capacity and GPU-to-GPU communication determine whether a model fits, how much KV cache remains for concurrent users and how much time is spent moving activations. NVIDIA’s 2026 HGX AI Factory reference architecture specifies the following baseline:
| Characteristic | Published figure | Why it matters in a benchmark |
|---|---|---|
| HBM3e per B200 GPU | 180GB | Sets room for weights, activations and KV cache; report peak and reserved memory. |
| GPU memory bandwidth | Up to 8TB/s per GPU | Influences decode and memory-bound kernels; actual rate depends on workload and clocks. |
| Aggregate bandwidth in an eight-GPU node | Up to 64TB/s | Useful only when the test uses all eight GPUs and the stated topology. |
| GPU-to-GPU NVLink bandwidth | 1,800GB/s | Affects tensor, pipeline and expert-parallel communication. |
| DGX B200 total GPU memory | 1,440GB across eight GPUs | Describe whether capacity is pooled through parallelism rather than treated as one ordinary memory space. |
| DGX B200 aggregate NVLink-switch bandwidth | 14.4TB/s | Report when comparing multi-GPU scaling on that eight-GPU system. |
These are architecture-level specifications, not guaranteed application measurements. A benchmark report should identify the exact node, topology and number of active GPUs, because an eight-way run, a two-way run and a single-GPU run have different communication paths and memory headroom.
Which metrics should a B200 inference benchmark report?
Throughput alone hides queueing and user experience. Report the following metrics for every configuration:
- Output tokens per second: generated tokens divided by measurement time. TensorRT-LLM’s published unit is output tokens per second per GPU, so preserve that denominator and also show node-wide throughput.
- Requests per second: completed requests divided by the measurement interval.
- TTFT: time from request admission until the first generated token. NIM documentation uses this term explicitly.
- Inter-token latency (ITL): time between successive generated tokens after the first token. NIM also calls this decode-side metric ITL.
- End-to-end latency: request completion time, including queueing, prefill and decode.
- Percentiles: p50, p95 and p99 for TTFT, ITL and end-to-end latency. Averages alone conceal overloaded users.
- Concurrency: active requests, offered load and the point at which queues begin growing.
- Per-user throughput: output tokens per second available to an individual user in a shared workload.
- Resource use: peak and steady-state GPU memory, utilization, power, host CPU, host memory, network traffic and KV-cache occupancy.
- Cost per token: total hourly infrastructure cost divided by output tokens produced in that hour, with the cost window and utilization assumption stated.
For agentic workloads add tool-call latency, number of turns, retries, parallel branches, routing overhead and the fraction of time spent in prefill versus decode. Keep quality metrics beside speed metrics when speculative decoding or aggressive quantization is enabled.
How do I set up a reproducible B200 benchmark?
Use a written test plan and keep the configuration under version control. The following sequence produces a result another team can repeat.
- Choose the question. Decide whether you are measuring lowest single-user latency, maximum throughput, a service-level objective, cost efficiency or multi-agent responsiveness. One run cannot optimize all of them.
- Pin the model and tokenizer. Record the exact checkpoint, revision, prompt template and stop-token behavior. Save a checksum for downloaded artifacts.
- Fix numerical settings. Declare FP4, FP8, BF16 or another format for weights, activations and KV cache. If comparing formats, use the same accuracy acceptance test.
- Declare sequence lengths and load. Set ISL and OSL explicitly, then define batch size, concurrency and request-arrival distribution. For variable-length tests, publish the distribution rather than only its mean.
- Freeze the software stack. Capture serving-engine versions, CUDA and driver versions, container digest, kernel options and enabled optimizations. A minor release can change scheduling or kernels.
- Describe topology and power. List GPU count, GPU identifiers, tensor or expert-parallel degree, NVLink/NVSwitch layout, network fabric and power policy.
- Warm up before measuring. Run a documented warm-up phase so model loading, kernel compilation, memory allocation and cache population are not mixed with steady-state timing. The duration is part of the recipe and must be reported.
- Measure a fixed window. State the measurement duration, number of requests, sampling method, random seed and whether failed or cancelled requests are excluded.
- Collect synchronized telemetry. Align request logs with GPU memory, utilization, power, host and network counters. Save raw per-request timestamps so percentiles can be recomputed.
- Repeat and publish variance. Repeat each configuration with the same seed and load pattern. Report run-to-run spread or confidence intervals rather than presenting a single lucky run.
How should I test latency, throughput and saturation?
Use two complementary passes: an isolated-latency pass and a load-sweep pass.
Isolated and low-concurrency latency
Start at concurrency one, then use a small fixed concurrency that reflects your service objective. Measure TTFT, ITL and end-to-end latency for short, medium and long ISL/OSL pairs. This reveals the cost of prefill and decode without a large queue masking the GPU behavior.
Rank #2
- 3328 optimized CUDA Cores, 7.99 TFLOPS
- 104 third generation Tensor Cores, 63.9 TFLOPS
- 26 third generation RT Cores, 15.6 TFLOPS
- Dual-slot width, low-profile form factor
- 70W maximum power consumption
Throughput and saturation sweep
Increase concurrency in controlled steps until output throughput stops increasing or the latency objective is violated. At each step capture requests per second, output tokens per second, per-GPU throughput, queue time, p50/p95/p99 latency and memory. Mark the highest load that satisfies your service-level target; the absolute peak after that point is a stress result, not a production recommendation.
Keep arrival rate and concurrency distinct. A closed-loop test starts a new request only when one finishes, while an open-loop test injects requests at a chosen rate. Label which method you use because closed-loop traffic can hide queue growth.
How do vLLM and TensorRT-LLM results compare on B200?
Compare engines only after matching model, quantization, sequence lengths, parallelism, feature flags and quality target. TensorRT-LLM’s B200 tables cover DeepSeek R1, GPT-OSS 120B, Llama 3.3 70B FP4 and Llama 3.1 405B FP4 across multiple ISL/OSL pairs. Each row is a reproducible configuration, not a universal score for every B200 deployment.
Free tools Windows power users keep installed
One-click scans. No signup required.
Preserve the published unit: TensorRT-LLM reports output tokens per second per GPU. Convert it to node throughput only by multiplying by the stated active-GPU count, and do not compare that number with requests per second or per-user throughput without showing the conversion.
NIM’s B200-SXM-180GB measurements report TTFT and ITL at tensor-parallel degrees TP2, TP4 and TP8. Include those metrics when changing tensor parallelism: a larger degree can reduce per-request latency or make a model fit, while adding communication and potentially changing throughput. The only defensible comparison is a table with identical load and a clear TP value for every row.
How do I construct a realistic multi-agent benchmark?
A multi-agent benchmark should replay traces, not a stream of identical prompts. Build a workload that preserves the control-flow features responsible for latency and cache pressure.
Trace ingredients
- Multiple turns per user, with each turn linked to the preceding context.
- Variable prompt and completion lengths, including long-context requests.
- Tool calls with realistic arguments, tool-result payloads and return delays.
- Retries, failures and backoff, rather than only successful single-pass conversations.
- Parallel sub-agents that compete for GPU capacity and may return at different times.
- Shared prefixes and repeated instructions when the production system would use prefix caching.
- A defined user-concurrency distribution and an arrival process that includes bursts.
For every trace, record turn count, tool count, branch fan-out, ISL, OSL, queue time, TTFT, ITL, completion time and total tokens. Report both aggregate output tokens per second and per-user tokens per second. A high aggregate number can coexist with poor individual responsiveness if a small number of users consume the available decode slots.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallRank #3
- NVIDIA Ampere Streaming Multiprocessors: Building blocks for the world's fastest, most efficient GPUs, the all-new Ampere SM brings twice the FP32 throughput and improved energy efficiency
- 2nd Generation RT Cores - Experience 2x the 1st Generation RT Cores throughput, plus competitive RT and shading for a whole new level of ray-tracing performance
- 【3rd Generation Tensor Cores】Get up to 2X the throughput with structural sparsity and advanced AI algorithms such as DLSS
- Core Clock: 1837MHz
- WINDFORCE 3X Cooler
Quality and speculation controls
If the run enables MTP or EAGLE3 speculative decoding, record the draft model, acceptance behavior and any fallback path. NVIDIA’s Dynamo recipe describes its GPT-OSS-120B agentic result as a synthetic-acceptance EAGLE3 throughput proxy; generated text is intentionally not a quality result. Reproduce the same recipe before making a speed comparison, and run a separate quality evaluation if the output will be used to judge product behavior.
What do aggregated and disaggregated serving results tell me?
Aggregated serving performs prefill and decode on the same serving pool. Disaggregated serving assigns those phases to different resources and moves KV cache or related state between them. The split can improve per-user responsiveness under the right traffic mix while reducing per-GPU aggregate throughput because of routing, transfer and synchronization overhead.
| Configuration | Trace and load | Throughput | Per-user rate | Average TTFT | Interpretation |
|---|---|---|---|---|---|
| Aggregated vLLM | Eight B200 GPUs; GPT-OSS-120B agentic 15% trace; concurrency 512 | 2,896 output tok/s/GPU | 58 tok/s/user | 4.4 seconds | Higher aggregate and system-wide efficiency in this recipe, with more users sharing the decode pool. |
| Disaggregated vLLM | Eight B200 GPUs; same trace; concurrency 256 | 2,069 output tok/s/GPU | 99 tok/s/user | 5.6 seconds | Fewer concurrent users and lower aggregate rate, but higher measured per-user decode rate. |
NVIDIA Dynamo published these figures in 2026 as a synthetic-acceptance EAGLE3 throughput proxy. They are not interchangeable service-level guarantees: concurrency differs, the trace is specific, and average TTFT alone does not show p95 or p99 behavior. When evaluating a split architecture, measure KV-cache movement, routing delay, prefill and decode utilization, dropped or retried requests and tail latency at the same user-level objective.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How do I compare scaling across one, two, four and eight B200 GPUs?
Run the same model and workload at each supported GPU count, then calculate scaling efficiency instead of looking only at raw throughput. A simple measure is:
scaling efficiency = (throughput at N GPUs / (N × throughput at 1 GPU)) × 100
Publish whether throughput means node-wide output tokens per second or per-GPU output tokens per second. Also show latency at a fixed concurrency and at a fixed throughput target. Tensor parallelism can make a larger model fit or reduce per-request latency while communication overhead lowers efficiency; an eight-GPU result should therefore include the NVLink topology and parallel degree.
For multi-agent systems, add concurrency at saturation, per-user throughput, KV-cache utilization and routing overhead. If a model cannot run on one GPU because of memory, state that the scaling baseline starts at the smallest valid configuration rather than treating the missing one-GPU point as zero.
Rank #4
- 900-1G136-2505-000
How should I calculate cost per token?
Use the same time window and utilization assumptions for every configuration:
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →cost per output token = total serving cost during the window / output tokens produced during the window
Total serving cost can include GPU rental or depreciation, host systems, networking and power. State whether the figure is an on-demand hourly rate, an internal amortization rate or another accounting basis, and include idle capacity if the service must reserve it. NVIDIA’s benchmarking FAQ identifies cost per token as the most important metric for AI-inference cost-effectiveness; a throughput leader can still be more expensive when it needs additional GPUs or leaves capacity idle.
What mistakes make B200 benchmark results misleading?
- Publishing one average: Add p50, p95 and p99 TTFT, ITL and end-to-end latency.
- Changing several variables at once: Compare one engine, quantization or parallelism change at a time, or label the run as a new recipe.
- Ignoring topology: A GPU count without NVLink/NVSwitch and parallelism details cannot be reproduced.
- Mixing denominators: Distinguish output tok/s per GPU, node tok/s, requests/s and per-user tok/s.
- Measuring cold start: Separate model loading and compilation from warmed steady-state service.
- Using fixed prompts for agents: Replay variable turns, tools, retries and branch fan-out.
- Reporting only averages for a split architecture: Capture KV-cache transfer and routing tails, not just mean TTFT.
- Claiming quality from a proxy: Synthetic-acceptance EAGLE3 throughput is not a text-quality evaluation.
- Omitting power and host limits: CPU, network or power throttling can make an apparent GPU comparison invalid.
- Presenting vendor tables as guarantees: Treat each published row as a recipe-specific baseline and reproduce it before extrapolating.
What should a publication-ready B200 benchmark report contain?
Make the report auditable by putting the recipe beside the result, not in a separate appendix that readers cannot connect to a number.
- System identification: B200 model, memory, GPU count, node, topology, firmware and power policy.
- Model identification: checkpoint, tokenizer, template, parameter count, quantization and KV-cache format.
- Software identification: serving engine, version, CUDA, driver, container and enabled optimizations.
- Workload definition: ISL/OSL, concurrency or arrival rate, batch policy, trace composition, tool behavior and quality target.
- Measurement method: warm-up, window, repeats, seeds, excluded failures and telemetry sources.
- Results table: output tok/s per GPU, node tok/s, requests/s, TTFT, ITL, end-to-end p50/p95/p99, per-user tok/s, memory, power and cost per token.
- Scaling and architecture notes: tensor or expert parallelism, aggregated versus disaggregated serving, KV-cache movement and routing overhead.
- Reproduction package: configuration files, trace schema, launch arguments and enough metadata to identify every software artifact.
This format lets a reader answer the practical question behind the benchmark: which configuration meets a stated latency and quality target at the lowest sustainable cost, rather than which bar in a chart is tallest.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Frequently Asked Questions
Can I compare a B200 result with a different GPU using only tokens per second?
Only as a rough capacity signal. A valid cross-GPU comparison keeps the model, quantization, ISL, OSL, concurrency, software features, quality target and power policy identical, then reports latency and cost alongside throughput. Differences in memory capacity and interconnect can otherwise make one system solve a different problem.
How long should the warm-up phase be?
There is no universal duration because compilation, model loading and cache behavior vary by engine and model. Continue until those one-time effects have settled, record the elapsed warm-up and request count, and use the same rule for every configuration.
Why can per-user throughput rise while total throughput falls?
A serving design may reserve more capacity for each active user or separate prefill from decode, reducing contention and raising each user’s token rate. The trade-off can be lower GPU utilization or fewer concurrent users, so inspect both per-user and aggregate figures at the same service objective.
What files should I keep so another team can reproduce the test?
Keep the exact model and tokenizer identifiers, container digest, engine and driver versions, launch configuration, hardware topology, workload trace or generator seed, and raw per-request timestamps. Include the quality-check procedure and any scripts that calculate percentiles and cost so the published tables can be independently regenerated.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
