Tokens per second is determined by the model, the work being measured, the hardware and the serving setup—not by the model alone. Prompt processing (prefill), token generation (decode) and total server throughput are different measurements, so a speed figure is meaningful only when its workload and test conditions are clear.
First, identify what “tokens per second” measures
LLM inference has distinct phases. During prefill, the system processes the prompt and computes the key and value (KV) states used for generation. Because the prompt is already known, much of this work can be parallelized, making compute capacity and kernel efficiency important. During decode, the model generates output autoregressively: each new token depends on the preceding sequence. Moving model weights and the growing KV cache can therefore be a major per-token cost.
As an Amazon Associate I earn from qualifying purchases.
| Metric | What it measures | What it helps answer |
|---|---|---|
| Prefill throughput | Prompt tokens processed per second. | How quickly a system can process an input prompt. |
| Per-request decode rate | Generated tokens per second for one request or stream. | How quickly an individual user sees output arrive. |
| Aggregate throughput | Tokens produced per second across concurrent requests; the benchmark should say whether it counts output tokens only or combines prompt and output tokens. | How much total work a serving system handles under a specified load. |
Time to first token and the delay between subsequent tokens add useful latency context: a high aggregate rate does not necessarily mean that any one user receives a fast response. NVIDIA’s “Mastering LLM Techniques: Inference Optimization” (November 17, 2023) cautions that models with similar output tokens-per-second figures may still not be comparable if their tokenizers count the same text differently.
Recommended Free Tools
How the model and its context affect speed
Model size and precision
More parameters, or a higher-precision representation of them, generally increase the weight footprint and the amount of data the system must move. Quantization reduces that footprint and may allow a larger batch or faster execution, but the result depends on the model, runtime and hardware support; reduced precision is not a guaranteed speedup in every setup.
#1 Best Overall
As an illustrative estimate, NVIDIA’s 2023 technical article says that 7 billion parameters stored in 16-bit precision require roughly 14 GB for the weights. That estimate excludes other runtime memory needs. The same article estimates approximately 2 GB of KV cache for a Llama 2 7B example at 16-bit precision, batch size 1 and sequence length 4096. This is a configuration-specific illustration, not a general cache requirement for every 7B model.
Prompt length and retained context
A longer prompt means more input work during prefill. The context retained for generation also affects KV-cache size; during decode, the system reads the cache as it generates tokens. NVIDIA’s 2026 dense-attention analysis describes attention work in its analyzed setting as scaling approximately with the square of input sequence length for prefill, while decode KV traffic scales approximately linearly with cache length. These describe attention behavior in that analysis, not guaranteed wall-clock scaling for every model or serving stack; fixed setup costs and other overheads can make observed scaling less steep, especially at short lengths.
KV-cache capacity can limit how many sequences, or how much context per sequence, remain active at once—even when compute resources are available. Cache management, compression, sparse or sliding-window attention, and prefix reuse can change memory use or work, but their effects depend on the model and workload.
Attention architecture
Models differ in how they represent and use attention state. The number of query heads sharing each KV head, head dimension, sequence length and tensor-parallel layout all affect cache use and kernel efficiency. NVIDIA’s 2026 attention analysis explains that greater query-head sharing can improve decode arithmetic intensity in its described dense-attention setting; hardware alignment and parallelization choices matter as well. Those findings are specific to the analyzed GPU and kernel context, not a guarantee across accelerators.
How hardware and serving choices change the result
Compute, memory bandwidth and capacity
- Compute throughput matters particularly during parallel prompt prefill.
- Memory bandwidth is central to decode because each step repeatedly moves model weights and KV state. NVIDIA’s 2023 inference article describes decode in this context as a memory-bound operation.
- Memory capacity must accommodate weights, runtime allocations and KV cache. The memory left after loading a model constrains cache capacity and the number of active requests.
- Multi-GPU communication can offset the benefit of adding accelerators. More GPUs may make a model fit or provide cache room, but communication overhead and the parallelization layout affect whether throughput improves.
NVIDIA Dynamo’s version 0.8.1 tuning guide describes a workload-dependent continuum: too few GPUs can leave inadequate cache room; an intermediate count may trade throughput per GPU against user latency; and beyond a communication-scalability limit, overhead can dominate. These examples are tied to particular models and hardware, so they do not establish a universal GPU count.
Batching, concurrency and scheduling
Serving multiple requests together can spread model-weight movement across more generated tokens, improving aggregate throughput. Each active sequence also consumes KV-cache memory, which limits how far batch size can grow. A static batch may wait for its longest generation to finish; continuous or in-flight batching can admit new requests as others complete, subject to the runtime’s implementation and available cache.
Rank #4
Throughput and latency are separate objectives. Under a fixed hardware budget, Google Cloud’s March 28, 2026 article, “Five techniques to reach the efficient frontier of LLM inference,” frames them as a tradeoff. NVIDIA’s Dynamo guidance likewise discusses tuning against service-level objectives. A setup tuned for total tokens served may not minimize time to first token or the delay between tokens for an individual request.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhat inference optimizations can—and cannot—do
Quantization and optimized attention
Quantization can reduce weight and sometimes cache-related memory demands, potentially freeing capacity for concurrent sequences. Optimized attention kernels and cache management can improve utilization or reduce wasted memory. Neither yields a fixed speed gain independent of model, hardware, runtime and request pattern; compare measured results under a matched workload.
Best Value
- Language fundamentals grade 1
- Language skills
- Grammar practice
Speculative decoding
Speculative decoding uses a draft model to propose several tokens, which a target model then verifies together. When proposals are accepted efficiently, the target may need fewer sequential generation steps. The benefit depends on draft-model cost, how many proposed tokens are accepted, batch size and whether the workload is limited by compute or memory. NVIDIA’s September 2, 2026 guidance treats draft length and mechanism as tuning choices, not universal settings.
Prefix reuse and separating prefill from decode
Prefix caching can avoid repeating work for shared prompt content when the serving system supports it and the workload actually reuses prefixes. Chunked prefill and other scheduling choices can also alter how prompt processing competes with ongoing generation.
Disaggregated serving assigns prompt prefill and token decode to separate resources. The KV state must then be transferred between them, adding a system and data-transfer consideration. vLLM documentation describes a prefill instance, a decode instance and a connector for KV-cache transfer. NVIDIA Dynamo’s version 0.8.1 guidance emphasizes that the best tuning depends on load and workload. Separation can help under some serving conditions, but it is not an automatic improvement.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →How to compare tokens-per-second claims fairly
Before comparing two figures, check whether they describe the same task and count tokens in the same way. A useful benchmark report includes:
- Model name and exact configuration, including precision or quantization and attention architecture where known.
- Tokenizer and token-counting convention.
- Accelerator model and count, memory capacity, and relevant interconnect or deployment arrangement.
- Runtime or serving engine, its version and major inference options.
- Prompt length, output length, batch size or concurrency, and whether requests share a prefix.
- The metric: prefill throughput, single-request decode rate, aggregate throughput or an end-to-end average. For latency-sensitive use, include time to first token and inter-token latency.
If a report combines prompt and generated tokens, it should also state the workload mix and concurrency. Without these conditions, a tokens-per-second number cannot establish a universal model-speed ranking. Choose measurements that match the model, system and request pattern you plan to use.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
