Yes, your output can differ between runs even when speculative decoding preserves the target model’s sampling distribution. The guarantee is about the probabilities assigned to possible outputs—not that every run produces identical text. Random sampling can yield different answers from the same distribution, and finite-precision arithmetic or implementation behavior can add further variation.
What speculative decoding guarantees
Speculative decoding uses a faster draft model to propose tokens and a target model to verify them. In the ideal sampling algorithm, rejection sampling corrects proposals so the resulting output distribution matches the target model’s distribution. The draft can help generate tokens more quickly without changing that probability law under the algorithm’s assumptions.
As an Amazon Associate I earn from qualifying purchases.
In simplified terms, the system keeps draft tokens that pass the target’s verification. If a proposed token is rejected, a correction draw accounts for probability mass the target assigns beyond the draft proposal. This is how the algorithm preserves the target distribution rather than simply accepting whatever the draft suggests. See the [2022 speculative decoding paper by Yaniv Leviathan, Matan Kalman, and Yossi Matias] and [Cai and colleagues’ 2023 speculative sampling paper].
“Same distribution” means that repeated outputs follow the same probability law. It does not mean that two individual samples must contain the same tokens. A coin can have the same 50/50 distribution on every toss and still land differently; sampled text can vary for the same basic reason.
#1 Best Overall
Why two runs can produce different text
Random sampling can vary on its own
If the target model samples among multiple possible next tokens, separate runs can draw different tokens even when the distribution is unchanged. Different sampled sequences are not, by themselves, evidence that speculative decoding changed the distribution.
Numerical precision can affect real implementations
The mathematical guarantee assumes ideal calculations. Actual inference uses finite-precision arithmetic, and small numerical differences can affect token probabilities. The [vLLM v0.21.0 speculative decoding documentation] describes sampling as theoretically lossless up to hardware numerical precision and warns that floating-point differences can slightly change distributions.
Batching and implementation behavior can also matter
vLLM says batch size can affect log probabilities and output probabilities through non-deterministic batched operations or numerical instability. It also states that it does not currently guarantee stable token log probabilities across runs. These implementation-level effects can lead to different outputs; they are distinct from the ideal algorithm’s distribution-preserving correction.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →What kind of sameness are you checking?
- Same probability distribution: the ideal sampling guarantee. Possible outputs retain the target model’s probability law.
- Same sampled text: not guaranteed when generation is stochastic. Two draws from the same distribution can differ.
- Same result under greedy decoding: a separate question. Greedy decoding selects the highest-probability token rather than drawing a random sample. vLLM documents greedy-sampling equality separately from rejection-sampler convergence; neither should be confused with a general promise of repeatability across real implementations.
- Same token log probabilities across runs: not guaranteed by vLLM’s current documentation.
To investigate a changed answer, first distinguish ordinary sampling variation from a change in the underlying probabilities. Then consider numerical precision, batch size, and the implementation’s repeatability guarantees. A changed output alone does not reveal which of these caused it.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Does preserving the distribution make decoding faster?
It can, but the speedup depends on the model and workload rather than following from the correctness guarantee. In their 2022 paper, Leviathan, Kalman, and Matias reported 2–3× acceleration on T5-XXL compared with the standard T5X implementation. Cai and colleagues reported a 2–2.5× decoding speedup in a distributed Chinchilla 70-billion-parameter benchmark in 2023. These are results for the named experimental setups, not universal expectations.
A 2026 vLLM report on AMD GPUs found that output-token throughput varied with drafting method, proposal length, model family, draft checkpoint, workload, and acceptance behavior. For a deployment, measure latency or output-token throughput on the intended workload, and account for batch size, draft-target compatibility, and how often proposed tokens are accepted. No one method or hardware configuration is established as best for every case.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
