When an LLM generates a token, it uses the current context to score possible next tokens, a decoding method selects one, and the model adds that token to the sequence. It repeats this process until a stopping condition is met. A token is not necessarily a whole word: depending on the model’s tokenizer, it can be a word, part of a word, punctuation, or a special token.
What does “one token” mean?
Readers often ask, “What happens when an LLM generates a token?” or “How does an LLM predict the next word?” The short answer is that the model predicts a token, not necessarily a word. Tokenizers divide text into model-specific units, so a familiar word might be one token or several, and a token may also represent a word fragment, punctuation, or a special marker.
As an Amazon Associate I earn from qualifying purchases.
The process below describes the common autoregressive transformer workflow documented for Hugging Face Transformers. It is a useful model of token-by-token generation, but implementations, tokenizers, serving systems, and stopping rules can differ. It does not mean every service performs precisely one model operation for each visible text token.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How one token is generated
- The model receives context. The prompt is formatted for the particular model and converted into token IDs. A chat application may include template structure or other application context beyond the text a user sees. Hugging Face’s generation tutorial shows tokenizer-produced
input_idsbeing passed to a model. - The model scores possible next tokens. A forward pass produces logits for vocabulary choices at the next position. These are scores, not a selected word or a finished response. In the documented example, the selection uses logits from the final sequence position.
- A decoding method selects a token. The method determines how the scores become a choice. Greedy decoding takes the highest-scoring token; sampling draws from a probability distribution, with settings such as temperature affecting selection when sampling is enabled. Beam search keeps multiple candidate sequences and compares their overall probability.
- The selected token is appended. The token ID joins the generated sequence, becoming part of the context for the next step.
- The loop continues or stops. The model repeats the process until it generates an end-of-sequence token or reaches a configured limit or custom stopping condition. One token step is an iteration in the loop, not a whole answer.
In short, a typical next-token step is: context in, candidate scores, one token selected, token appended. Hugging Face’s Transformers v4.38.1 tutorial describes generation as iteratively sampling a next token, appending it, and continuing until a finishing token is produced.
#1 Best Overall
How decoding methods differ
Decoding is the selection procedure, not a different meaning of “token generation.” The choice of method changes which continuation the model produces and how much variation is possible.
| Method | Selection rule | Variation across runs | Typical use described in the documentation |
|---|---|---|---|
| Greedy decoding | Selects the highest-scoring next token at each step. | Does not introduce sampling variation; other system details can still matter. | Directly choosing the locally most likely token. |
| Sampling | Draws a token from a probability distribution over candidates; temperature can affect the selection behavior when sampling is enabled. | Can yield more varied continuations. | When variation in the generated continuation is desired. |
| Beam search | Maintains several candidate sequences and compares their overall probability. | Compares multiple sequences rather than making only a single greedy choice. | Hugging Face describes it as useful for input-grounded tasks. |
No method is universally best: the appropriate choice depends on the task and the behavior sought. See Hugging Face’s generation strategies guide for the documented distinctions.
What the KV cache does during generation
Transformer attention layers compute key and value representations for tokens. A key-value (KV) cache retains previously computed states so later decoding steps can reuse them instead of recalculating those states for the full prior sequence. In a cached loop, the prompt first populates the cache; later steps can process the newly selected token while prior attention states remain available. The cache grows as tokens are added.
This reuse reduces redundant computation and can speed inference, but the retained states use memory, which increases with context length. The actual speed and memory requirements depend on the model and runtime. Hugging Face’s Transformers v4.44.0 cache guide explains the cache’s role in autoregressive generation.
| Approach | Recomputation | Memory | Output considerations |
|---|---|---|---|
| With KV cache | Reuses previously computed attention key/value states. | Retains those states; memory use rises with context length. | Hugging Face notes that kernel differences can cause slightly different output. |
| Without KV cache | Does not reuse retained prior key/value states in this way. | Does not hold that cache, though total memory use still depends on the model and runtime. | No general output-equivalence guarantee is established by the cited documentation. |
What one token does—and does not—tell you
- It is a model-specific text unit. It may be a whole word, a fragment, punctuation, or a special token; there is no universal word-to-token mapping.
- It is one selection in a larger loop. Ordinary autoregressive generation appends the selected token to context and runs another step until a stopping rule applies.
- It does not have a universal time or cost. The cited official documentation does not establish a generally applicable milliseconds-per-token, compute, or energy figure. Any such measurement would need to identify at least the model, hardware, context length, batch size, software version, and measurement source.
These explanations describe the common documented workflow, not every generation technique. Model architectures and runtime implementations can vary, including methods that generate or verify multiple tokens through specialized procedures.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
