Prompt caching usually produces the clearest bill reduction for hosted API users because providers can charge a lower rate for input tokens whose prefix was reused. OpenAI documents cached-input discounts of up to 95% on supported models. KV-cache reuse is the underlying serving technique; for a self-hosted model it reduces repeated prefill computation, GPU time, time to first token (TTFT), and sometimes the capacity you need to serve a workload. There is no universal winner: the result depends on repeated-prefix length, hit rate, cache lifetime, pricing, GPU memory, concurrency, and eviction.
The practical choice is therefore economic rather than semantic. API customers should calculate cached and uncached token charges. Teams operating their own inference stack should measure avoided prefill work and capacity against the memory and scheduling cost of retaining KV blocks.
As an Amazon Associate I earn from qualifying purchases.
What prompt caching and KV-cache reuse actually mean
Both approaches reuse attention state that a model already computed for an earlier request. During inference, each processed token produces key and value tensors used by later attention layers. Keeping those tensors means a repeated prefix does not have to be processed from the beginning again.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Prompt caching is usually the provider-facing feature
A hosted API may identify an unchanged prompt prefix, retain its computed state, and apply a separate cached-input price when that state is reused. OpenAI describes the implementation plainly: “The prompt cache stores key-value (KV) tensors, not the tokens themselves.” The service still processes every new suffix token, and a cache hit is conditional on the prefix matching the provider’s rules.
Prompt caching can reduce two different things at once: the provider’s work before generation and the amount you pay for eligible input tokens. The discount is not a discount on output tokens, and it does not make a changed prefix free. Provider-specific details such as minimum prefix length, cache lifetime, write charges, model support, and regional behavior determine whether a request qualifies.
KV-cache reuse is the runtime operation
Inference engines expose KV reuse as an internal optimization. vLLM’s automatic prefix caching hashes blocks using the block’s tokens and the preceding-prefix context, then reuses matching blocks. TensorRT-LLM similarly shares KV pages among requests. Allocation, eviction, isolation, and scheduling are runtime responsibilities rather than API billing features.
A self-hosted operator normally receives no automatic per-token price cut. The financial benefit appears indirectly: fewer GPU cycles spent on prefill, lower TTFT, more requests completed per hour, or fewer GPUs required for a target service level. You convert those operational changes into dollars using your actual GPU price and utilization.
Where the two approaches overlap—and where they do not
| Axis | Prompt caching | KV-cache reuse |
|---|---|---|
| Primary layer | Hosted API or provider feature | Inference runtime or serving engine |
| Direct cost effect | Lower cached-input token price when a prefix qualifies | Avoided GPU prefill work and potentially lower capacity cost |
| Main latency effect | Less pre-response processing | Lower TTFT and potentially higher throughput |
| Best fit | Repeated prefixes sent to an API | Self-hosted, high-volume workloads sharing prefixes |
| Typical failure modes | Cache miss, prefix mismatch, provider TTL or eligibility rule | Eviction, memory pressure, scheduling limits, incomplete producer request |
| What to measure | Cached versus uncached input tokens, hit rate, and invoice impact | KV usage, reuse gap, eviction, TTFT, throughput, and GPU cost per request |
Calling prompt caching and KV-cache reuse “the same thing” is incomplete. They generally reuse the same kind of computed KV state, but the terms describe different layers. Prompt caching is a product and pricing capability exposed to an API customer; KV reuse is the mechanism a serving engine uses to avoid recomputation.
How prompt caching changes a hosted API bill
For a request with a reusable prefix, separate the input into three parts:
- Uncached prefix tokens: tokens that did not qualify for reuse.
- Cached prefix tokens: tokens whose prior KV state was accepted and billed at the cached-input rate.
- Suffix tokens: new instructions, user data, or other tokens after the shared prefix.
A useful approximation is:
total input cost = uncached-prefix tokens × uncached rate + cached-prefix tokens × cached rate + suffix tokens × applicable rate
Output-token charges remain separate. If your application sends a 20,000-token policy and tool schema followed by a 200-token user question, the economic opportunity is in keeping that 20,000-token prefix identical. Moving a changing value—such as a timestamp, request ID, or conversation transcript—into the beginning of the prompt can destroy the match even when most of the later text is unchanged.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesWhy “up to 95%” is not a guaranteed saving
OpenAI documents cached-input discounts of up to 95% on supported models. “Up to” matters: the percentage depends on the model and the provider’s current pricing schedule, while your effective saving depends on how many tokens actually hit the cache. A 95% discount on a small prefix, or on a prefix that rarely repeats, can be worth less than a smaller discount applied consistently to a large prefix.
Providers may also impose a minimum prefix length, a finite retention window, model- or region-specific support, and rules for when a cache entry is written. A first request can therefore pay the uncached rate, while later requests qualify only if they arrive before expiry and preserve the exact required prefix. Treat provider documentation and your usage records—not a headline percentage—as the source of truth for a forecast.
Hosted-API break-even calculation
Estimate the monthly difference with measured values:
- Record total input tokens, shared-prefix tokens, suffix tokens, and output tokens for a representative period.
- Measure the fraction of requests in which the prefix was accepted as cached.
- Apply the provider’s uncached and cached rates to those token counts, including any cache-write or minimum-length rules.
- Compare the resulting bill with the all-uncached case.
If P is the shared prefix length, h is the hit rate, R is the uncached input rate, and C is the cached rate, the approximate saving on the prefix is P × h × (R − C) per request, before provider-specific adjustments. The formula is intentionally simple; it does not assume a universal TTL or discount.
Recommended Free Tools
How KV-cache reuse lowers self-hosted cost
Self-hosted inference spends substantial GPU work on prefill: processing the input sequence before the first generated token. Reusing a long prefix skips that repeated computation. NVIDIA’s TensorRT-LLM documentation notes that reuse can greatly lower first-token latency; when the shared prefix is long relative to the entire prompt, an identical back-to-back request can approach the latency of subsequent-token generation for its first token.
The financial model is different from an API discount:
net saving = avoided prefill GPU cost and capacity cost − KV memory cost − cache-management and movement overhead − utilization lost to eviction or fragmentation
To put a value on the result, multiply the GPU-hours avoided (or the additional requests served by the same GPUs) by your actual hourly infrastructure cost. A workload that is latency-sensitive but lightly utilized may value TTFT more than raw GPU-hour savings. A heavily utilized batch service may benefit more from throughput and deferred hardware expansion.
Memory is the main trade-off
KV blocks occupy GPU memory while they are retained. Longer prefixes, larger models, larger batches, and longer context windows all increase the amount of state that can be held. Reserving memory for a cache leaves less room for active sequences and model execution. If the cache is too large, it can cause allocation failures or force eviction; if it is too small, entries disappear before they are reused and the prefill work returns.
The right question is not “How much memory does KV reuse need?” in isolation. Measure the bytes per cached token for your model and precision, multiply by the number of retained tokens, and include runtime metadata, fragmentation, and active-request headroom. Those values vary by model architecture and engine configuration, so a figure from another deployment is not a reliable budget.
When prefix caching pays off
Long, stable prefixes
The strongest candidates are large system instructions, policy text, tool definitions, retrieval templates, or document headers that remain byte-for-byte—or token-for-token—identical across many requests. The more shared tokens relative to the full prompt, the more prefill work is avoided and the larger the potential API discount base.
High reuse frequency
A prefix used once cannot amortize cache creation or memory occupancy. Reuse becomes attractive when many requests arrive within the cache’s retention period. Burst patterns are especially favorable because entries are still warm and concurrent requests can share them.
Free tools Windows power users keep installed
One-click scans. No signup required.
Predictable ordering
Put stable content first and variable content later. A changing user ID, date, random seed, or tool result at the front makes the remainder part of a different prefix. Keep serialization deterministic: the same fields should appear in the same order with the same whitespace and encoding.
Workloads with meaningful TTFT requirements
Interactive assistants, streaming applications, and retrieval pipelines often care about the delay before the first token. Even if the total generation time changes little, avoiding a long prefill can make the service feel substantially faster.
When reuse disappoints
- Prefix mismatch: one changed token, reordered field, different tokenizer behavior, or altered tool schema prevents a hit.
- Short prompts: cache lookup and management overhead can outweigh the small amount of prefill saved.
- Low hit rate: a cache that is constantly populated and rarely reused consumes memory without reducing work.
- Eviction and fragmentation: active traffic can displace useful blocks or leave memory in unusable fragments.
- Concurrency and scheduling: a reusable state may not be available to a waiting request when the scheduler chooses a different execution path.
- Producer request still running: TensorRT-LLM documentation identifies cases where reuse fails because the request that produced the state has not terminated.
- Isolation requirements: shared state may need salting or tenant partitioning so one customer’s prefix cannot be reused in another customer’s context.
- Offload and data movement: moving KV blocks between GPU and CPU or across devices can consume enough bandwidth to reduce the gain.
Measuring the real cost and benefit
Do not infer savings from latency alone. Build a baseline with caching disabled or bypassed, then run the same traffic with reuse enabled. Keep model, tokenizer, hardware, batching policy, sampling settings, and request mix constant.
Metrics for hosted APIs
- Input tokens billed at the uncached rate.
- Input tokens billed at the cached rate.
- Prefix-cache hit rate by endpoint, model, and tenant.
- Average and percentile shared-prefix length.
- Cache age at reuse and the distribution of misses after idle periods.
- TTFT and end-to-end latency for hit and miss groups.
- Actual cost per request and cost per successful task.
Metrics for self-hosted engines
- KV-cache occupancy and allocation failures.
- Reuse-gap statistics: how much time or request distance separates creation from reuse.
- Eviction count, eviction reason, and fragmentation.
- Prefill versus decode GPU utilization.
- TTFT, tokens per second, queue time, and throughput at each concurrency level.
- GPU memory reserved for KV blocks, active sequences, and model weights.
- Cost per request at the measured GPU-hour price.
vLLM exposes KV-cache usage and reuse-gap metrics. Its benchmarking guidance warns that reused prompts can inflate measured throughput, so a benchmark that mixes warm-cache and cold-cache runs can make an engine appear faster than it will be for your real traffic. Report cold-start, warm-hit, and mixed-workload results separately.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Also test concurrency rather than a single sequential loop. PrefixBench-H100 reports that benefits depend on scheduling, cache granularity, concurrency, and memory pressure; differences between runtimes can arise from scheduling decisions above the cache layer, not just from the cache implementation itself.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A practical evaluation procedure
- Describe the traffic. Group requests by model, tenant, endpoint, prompt template, prefix length, suffix length, arrival burst, and concurrency.
- Canonicalize prompts. Make serialization deterministic and move volatile fields after stable instructions. Record a hash of the intended shared prefix for diagnostics.
- Establish a cold baseline. Measure cost, TTFT, throughput, GPU utilization, and memory with reuse disabled or with unique prefixes.
- Enable the smallest useful cache. Configure a conservative KV budget, retention policy, and isolation strategy. Avoid consuming memory needed for active requests.
- Run warm and mixed tests. Include repeated prefixes, one-off prefixes, bursts, idle gaps, and concurrent producers and consumers.
- Separate hits from misses. Compare token counts, latency, GPU work, and failures for each class instead of averaging them together.
- Calculate money. For an API, use cached and uncached token rates. For self-hosting, use measured GPU-hours, utilization, and any additional hardware deferred.
- Stress the failure boundary. Increase context length and concurrency until eviction, allocation failures, queueing, or scheduling effects appear.
- Set a policy. Keep caching enabled only for workloads whose measured savings exceed memory, complexity, and operational risk.
Choosing between the two for common architectures
| Architecture | Most relevant lever | Reason | First measurement |
|---|---|---|---|
| Application calling a hosted model API | Provider prompt caching | The provider controls KV storage and can expose a cached-input rate | Cached versus uncached input tokens and invoice cost |
| Dedicated self-hosted model server | Runtime KV-cache reuse | You control memory, scheduling, and hardware utilization | TTFT, prefill utilization, KV occupancy, and GPU cost per request |
| Hybrid gateway with multiple tenants | Both, with strict isolation | API caching may cut token charges while local reuse can reduce gateway or fallback compute | Hit rate by tenant, isolation overhead, and duplicate caching layers |
| Mostly unique, short prompts | Usually neither is a priority | There is little shared work to amortize | Prefix-length distribution and cold-cache latency |
| Long repeated prompts at high concurrency | KV reuse plus provider caching where available | Both billing and prefill pressure are material | Warm-hit throughput, eviction rate, and total cost under peak load |
Security, correctness, and operations
Cache keys must include every context element that can change the model’s interpretation. A prefix that is textually identical but associated with a different tenant, authorization scope, tool permission, or policy version must not be shared accidentally. vLLM supports optional salting for cache isolation; use an equivalent tenant or policy boundary in any engine that serves multiple security domains.
Invalidate or version prefixes when system instructions, tool schemas, safety policies, tokenizer settings, model weights, or retrieval templates change. Otherwise, an apparently successful hit can reuse state produced under an obsolete configuration. Track cache-version identifiers in logs without recording sensitive prompt content.
Plan for cold starts and failure recovery. A restarted process, GPU eviction, deployment, or provider expiration can remove all entries. The application must continue correctly on a miss, and capacity planning must assume a cold-cache surge rather than treating warm-cache latency as guaranteed.
Answers to the cost question
For a hosted API, prompt caching is usually cheaper in the most visible sense: it can lower the input-token line item, with documented discounts of up to 95% on supported OpenAI models. Your actual reduction is the eligible cached-token volume multiplied by the difference between the two rates, adjusted for hit rate and provider rules.
For self-hosted inference, KV-cache reuse is usually the relevant optimization. It can lower TTFT, prefill GPU work, and the number of GPUs needed for a target throughput, but it also consumes memory and can create eviction or scheduling costs. Its value is the measured infrastructure saving, not a universal token discount.
If both options are available, evaluate them independently and then together. A provider-side cache may reduce your API bill while a local gateway cache reduces work before requests reach the provider; duplicated layers can also add complexity without improving hit rate. The deciding evidence is your workload’s shared-prefix tokens, cache hits, TTFT, GPU memory pressure, eviction behavior, concurrency, and cost per request.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →
