Recommended Free Tools
Language model inference is the stage where a trained model is run on new input to compute output. When a chatbot answers a question, the model is not learning from that exchange. It is calculating each output token from the prompt and from the tokens it has already produced. The term covers that computation, and the surrounding system that delivers it to users is called serving.
What the term means
In machine learning, inference is the execution of a model after training is finished. The model’s parameters are fixed, and new inputs go in while predictions or generated outputs come out. Training is different: it adjusts parameters using large datasets and gradient updates. Inference does not change the weights.
As an Amazon Associate I earn from qualifying purchases.
For a language model, the input text is first converted into tokens, which are small units such as whole words, word fragments, or punctuation marks. The model processes those tokens and then emits output tokens according to its generation method. The description below follows the widely used autoregressive, decoder-only path that most current text-generation LLMs use. Not every language model or generation architecture follows exactly this sequence, so treat it as the common case rather than a universal rule.
Inference and serving are different layers
People often use “inference” to mean everything that happens when a user sends a request. The clearer split is between the model computation and the system around it.
#1 Best Overall
| Term | What it refers to | Typical examples |
|---|---|---|
| Inference | Running the trained model to compute outputs for new inputs | Prefill over the prompt, then token-by-token decoding |
| Serving | The request-handling system around that computation | Queueing, batching, streaming partial output, metrics, returning responses |
Serving engines and benchmarking tools measure the serving layer, which is why a measured response time reflects both the model’s computation and the system’s scheduling.
The autoregressive path, step by step
A decoder-only LLM answering a prompt usually moves through the following stages.
1. Tokenization and request setup
The prompt is converted into tokens using the model’s tokenizer. The tokenizer matters when comparing speed numbers. NVIDIA’s documentation cautions that one token in one tokenizer may correspond to a different amount of text in another, so a tokens-per-second figure from two different models is not a like-for-like comparison unless the tokenization is accounted for.
2. Prefill
During prefill, the model processes the whole input context in a forward pass and computes attention state for the prompt tokens. This work prepares everything needed to produce the first output token. NVIDIA describes this phase as context processing, and AWS Prescriptive Guidance describes it as a forward pass across the tokenized prompt.
3. Decode
Decoding produces output one token at a time. Each newly generated token becomes part of the context for the next step, so the model cannot produce the whole answer in one pass. The attention information for earlier tokens is reused through the KV cache, described below.
4. Stopping and returning output
Generation continues until a stopping condition is met. Common conditions include a model-specific end-of-sequence token or a configured maximum output length. These rules are set by the model and the deployment, so one stopping setting should not be presented as universal. A serving system can stream tokens to the user as they arrive instead of waiting for the full response.
Why the KV cache matters
The KV cache stores the attention keys and values computed for earlier tokens. Without it, the model would recompute that information at every decode step. The cache is what makes token-by-token generation practical at reasonable speed.
Free tools Windows power users keep installed
One-click scans. No signup required.
The cache also consumes memory. Its size depends on the model architecture, numeric precision, sequence length, and the number of requests active at once. Long contexts and high concurrency can create memory pressure, which can limit how many requests a single accelerator can serve.
A plain way to state it: the cache saves repeated work by keeping attention state for earlier tokens, and that state takes memory. The cache does not remove all repeated computation, and it does not improve performance in every setting regardless of memory limits.
Serving and optimization tradeoffs
Batching
Batching processes several requests together so the hardware stays busy. This can raise aggregate throughput. With static batching, requests may wait for a batch to fill, and shorter requests can be held up by longer ones. Continuous, or in-flight, batching lets the serving engine add and remove requests as work progresses. The throughput and latency results depend on arrival patterns, prompt and output lengths, model size, hardware, and latency targets.
Colocated and disaggregated serving
In colocated serving, prefill and decode run on the same GPU resources. NVIDIA’s TensorRT-LLM documentation notes that prefill work can interfere with token generation and affect token-to-token latency. Disaggregated serving places the two phases on separate GPU pools so each can be tuned separately, but the KV-cache blocks must then be transferred between pools. The same documentation identifies long input sequences with moderate output lengths as a workload where separation can help. That is a workload-specific observation, not a general recommendation.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Quantization and model parallelism
Quantization stores weights or performs computation at lower numeric precision. It can reduce memory use and serving cost, but it can also change output quality, and its effects vary by model and hardware. Evaluate it on the model and task you actually run. Model parallelism splits a model across several accelerators when it does not fit on one. The tradeoff is added communication and operational complexity.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to read inference performance claims
A headline speed figure is only meaningful when the metric is defined. The four metrics below are the ones most often confused.
| Metric | What it measures | What to check |
|---|---|---|
| Time to first token (TTFT) | Time from query submission to the first output token | Usually includes queueing, prefill, and network latency. Longer prompts tend to raise it because the full context must be processed first. |
| End-to-end request latency | Time from submission until the full response arrives | Includes queueing, batching, and network latency, not only model computation. |
| Inter-token latency (ITL), also called time per output token (TPOT) | Average time between successive output tokens | Tools differ on whether the average includes TTFT. NVIDIA’s AIPerf definition excludes it. |
| Tokens per second (TPS) | Output throughput, either aggregate across the system or per request | Aggregate TPS can rise with concurrency until resources saturate, while per-user speed often falls as latency grows. Confirm which definition a report uses. |
NVIDIA’s benchmarking documentation warns that measurement tools differ, so two reported numbers may not share a boundary even when they share a name.
What a fair comparison records
- The model name and version, and the tokenizer used
- Prompt and output token distributions, not only averages
- Arrival rate and concurrency level
- Decoding settings, including maximum output length
- Hardware, serving software, and its version
- The exact metric formula and where timing starts and stops
Which axes to compare
| Axis | What to compare |
|---|---|
| Time to first token | User wait before output begins, measured from the same starting point |
| Inter-token latency | Pace of output once it starts, with one consistent definition |
| End-to-end latency | Time to the complete answer, including serving overhead |
| Aggregate throughput | Total output tokens over a stated period and concurrency |
| Memory headroom | Weights plus KV-cache demand at the target context length and concurrency |
| Workload fit | Whether the job is prefill-heavy (long prompts, short answers) or decode-heavy (short prompts, long answers) |
| Output quality and compatibility | Changes introduced by quantization or other optimizations on the actual model and hardware |
Only compare real setups when their model, workload, hardware, software, and measurement conditions are stated. Otherwise, the axes explain what to look for without implying a winner.
What the evidence does not establish
NVIDIA’s technical material includes memory calculations, but they use assumed model configurations to illustrate the arithmetic. They are not benchmark results and should not be read as typical performance. No single inference speed figure applies across models, hardware, or workloads. Vendor documentation and serving software change between releases, so check the version you are reading against before relying on a specific number.
Where to start
If you are new to the term, start with the distinction between inference and serving, then follow one request through prefill and decoding. Once that sequence is clear, the metric definitions and tradeoffs above make it possible to judge any speed claim you encounter.
Quick Recap
The Bottom Line
“”
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
