Azure AI Foundry prompt caching can lower input-token charges and latency when requests repeat a long, identical prefix. To benefit, make the first 1,024 tokens byte-for-byte stable, put changing content at the end, and use generation-appropriate cache controls. Standard deployments discount cache reads; Provisioned deployments can receive up to a 100% input-token discount. GPT-5.6 and later add explicit breakpoints and cache keys, but may charge for cache writes, so a high hit rate—not simply enabling caching—is the optimization target.
What Azure prompt caching reuses
A normal request runs the model’s attention calculations over every input token. Azure AI Foundry prompt caching retains temporary computations for a repeated beginning of the prompt. In transformer terminology, those retained key/value tensors are the KV cache. A later request with the same prefix can reuse the tensors instead of recomputing them, reducing time to first token and the amount of input processing billed.
As an Amazon Associate I earn from qualifying purchases.
Microsoft describes the feature as reducing latency and cost for longer prompts with identical content at the beginning, without changing the generated output. Caching affects computation and billing, not the model’s response logic: with the same model, parameters and complete prompt, a cache hit should produce the same result as a miss.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →The optimization is useful for applications that repeatedly send a large system instruction, tool catalog, output schema, policy document or reference file. It is much less useful when every request has a different prefix or is shorter than the eligibility threshold.
#1 Best Overall
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
Eligibility: the first 1,024 tokens decide the hit
A request must contain at least 1,024 input tokens. The first 1,024 tokens must be identical to a previously cached prefix. Matching is exact: Microsoft notes that a single-character difference in that opening segment causes a miss. Differences in whitespace, punctuation, JSON key order, escaping, date strings or an injected identifier can therefore invalidate reuse if they occur early enough.
After the first 1,024 tokens, matching behavior depends on the model generation:
| Model generation | How later cached tokens are counted | Practical implication |
|---|---|---|
| GPT-5.5 and earlier | Cache hits after the first 1,024 tokens are rounded in 128-token increments. | Keep reusable blocks comfortably larger than each 128-token boundary; small suffix changes may not produce a proportional cached count. |
| GPT-5.6 and later | The documented behavior does not use that 128-token rounding. | Long stable blocks can be measured without the earlier rounding rule, while write charges make hit-rate monitoring more important. |
Think of the prefix as an immutable header. Put dynamic data—such as the user’s question, current time, account ID, retrieved snippets and tool results—after that header. If a conversation is edited near its beginning, every subsequent request may lose the reusable prefix; an append-only history preserves more of it.
Rank #2
- Delivers 25 TFLOPS of AI inference performance using four MemryX MX3 accelerators in a compact M.2 2280 M-key module.
- Connects through PCI Express Gen 3 and above for straightforward integration into embedded and edge systems with an M.2 M-key slot.
- Includes 42 MB of distributed on-chip weight storage to support low-latency, batch-1 pipelined inference.
- Designed for low-power deployment with a 1 W–12 W operating range and host-managed airflow requirements.
- Supports ONNX, TensorFlow, Keras, and TFLite models through the MemryX software stack on Linux and Windows hosts.
Controls by model generation
GPT-5.6 and later: keys, breakpoints and modes
For GPT-5.6 and later families, send a stable prompt_cache_key for requests that share a prefix. The key is an affinity label, not a substitute for identical content: requests using the same key still need the same eligible prefix to hit. Microsoft warns that more than approximately 15 requests per minute for one prefix/key combination can cause misses. At higher volume, distribute traffic across several stable keys while keeping each key mapped consistently to the same prefix shape.
Standard pay-as-you-go deployments also support explicit prompt_cache_breakpoint markers and prompt_cache_options.mode. The default implicit mode lets the service choose eligible cache points. explicit mode uses only the breakpoints you declare, which is useful when you need tight control over what is written or reused. The only supported prompt_cache_options.ttl value is 30m; it establishes a minimum cache lifetime rather than a promise that the entry will remain available indefinitely.
Each request can create up to four new cache writes. Cache reads consider up to the latest 50 breakpoints. These limits matter when a request contains many separately marked stable sections: prioritize the blocks that recur most often instead of marking every possible boundary.
Rank #3
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
Models before GPT-5.6: automatic caching
Earlier models do not accept prompt_cache_options or prompt_cache_breakpoint. They use automatic prompt caching. Where the model supports it, prompt_cache_retention selects in_memory or 24h extended retention. Do not send the newer breakpoint fields to these models; use the controls documented for the deployed model version.
Retention, expiry and what KV-cache reuse means
In-memory retention
In-memory entries are typically cleared after 5–10 minutes of inactivity and always within one hour of their last use. “Typically” is important for capacity planning: a request that arrives after an idle period can miss even when its text is unchanged. A hit keeps the entry active, but there is still a one-hour maximum for this retention mode.
Extended prompt caching
Extended retention can keep supported prefixes for up to 24 hours. When GPU memory is full, Azure can offload the key/value tensors to GPU-local storage and restore them later. The documented supported list includes gpt-5.5, gpt-5.4, gpt-5.3-codex, gpt-5.2, gpt-5.1 variants, gpt-5, gpt-5-codex and gpt-4.1. Availability is model-specific; verify that the exact deployment supports the 24h option before relying on overnight or cross-shift reuse.
KV-cache reuse is therefore an implementation detail behind prompt caching, not a separate model capability that changes tokenization. The service stores attention state for an eligible prefix and reuses it while the retention policy and capacity allow. It cannot rescue a prefix that changed, a request below 1,024 tokens or a request routed in a way that misses the relevant cache entry.
How deployment type changes the economics
| Deployment | Cache behavior and controls | Cost point to verify |
|---|---|---|
| Standard pay-as-you-go | Discounted cache reads; GPT-5.6+ supports explicit breakpoints, implicit/explicit mode and a 30m minimum TTL setting. |
GPT-5.6+ can charge for cache writes as well as discounted reads. Compare write tokens with later read tokens. |
| Provisioned (including PTU-M considerations) | Provisioned deployments can receive up to a 100% input-token discount on cache reads. PTU-M does not support breakpoints. | Do not assume Standard breakpoint settings work on PTU-M. Confirm the model and deployment documentation for the controls available to your capacity. |
Before GPT-5.6, cache writes have no extra charge according to Microsoft’s guidance. GPT-5.6 and later can bill writes in addition to discounted reads. A workload that constantly changes its supposedly stable header may pay to create entries that are rarely read. Caching is profitable when the cost avoided by repeated reads exceeds the cost of writing and retaining the prefix.
Recommended Free Tools
Build a cache-friendly prompt
- Freeze the opening. Place stable system and developer instructions first. Keep wording, order, whitespace and serialization consistent across requests.
- Follow with reusable assets. Put tool definitions, structured-output schemas and reference files that recur across calls immediately after the instructions.
- Move volatility to the end. Append the user question, timestamps, request IDs, retrieved passages, account-specific facts and fresh tool results after the stable section.
- Keep history append-only. Add each turn to the end of a conversation. If you must correct an early instruction, expect the old prefix to stop matching and consider starting a new cache lineage.
- Choose a stable key on GPT-5.6+. Derive
prompt_cache_keyfrom the prefix template or tenant class, not from a per-request ID. Keep the mapping deterministic so identical prefixes return to the same key. - Shard busy keys. If one prefix/key pair approaches 15 requests per minute, spread requests across multiple stable keys. Do this by design—for example, hash a tenant or conversation group into a small key pool—rather than randomly changing keys per request.
- Add deliberate breakpoints when appropriate. On Standard GPT-5.6+ deployments, place
prompt_cache_breakpointafter large stable blocks. Selectexplicitmode when only declared blocks should be eligible; retainimplicitwhen service-selected caching is acceptable. - Pick retention to match traffic. Use in-memory caching for frequent bursts. Use
24honly on supported models when requests recur across longer gaps and the extra retention is operationally valuable.
Measure hits, writes and actual spend
Do not infer success from lower latency alone. Standard responses expose cached_tokens; newer model families also expose cache_write_tokens. Record both fields with the model, deployment, key and request timestamp.
Best Value
- Memory Size: 16 GB GDDR6 ECC.
- Memory Bus Width: 128-bit.
- Memory Bandwidth: 200 GB/s.
- CUDA Cores: 1280.
- Peak Single Precision floating point performance: 18 Tflops (GPU Boost Clocks).
- Hit volume: Sum
cached_tokensand divide by total input tokens to see how much input was served from cache. - Write volume: Track
cache_write_tokenson GPT-5.6+ and identify prefixes that are written repeatedly but seldom read. - Key pressure: Count requests per prefix/key combination per minute. Spikes near the approximately 15-request threshold can explain intermittent misses.
- Prefix drift: Log a non-sensitive template version or hash of the stable section. A sudden increase in misses often follows a deployment that changed a system prompt, schema or tool definition.
- Effective input cost: Apply the deployment’s current read discount and any write charge to the measured token totals. Provisioned and Standard economics are not interchangeable.
Monitor these metrics by model generation. A dashboard that combines GPT-5.5 and GPT-5.6 can hide the rounding change and the introduction of write charges, making a healthy workload appear inconsistent.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshoot a low cache-hit rate
Every request misses
- Check that the request has at least 1,024 input tokens.
- Compare the first 1,024 tokens exactly, including invisible whitespace, punctuation and serialized object order.
- Verify that dynamic fields were not inserted into the system prompt or tool definitions.
- Confirm that the deployment is using the intended model generation and that unsupported breakpoint fields are not being sent to an older model.
Only some requests miss
- Inspect request rate for one prefix/key pair; traffic above approximately 15 requests per minute can produce misses.
- Check whether a load balancer assigns equivalent prefixes to different keys inconsistently. Either preserve affinity or deliberately shard across a fixed key pool.
- Look for conversation edits, prompt-template rollouts or retrieval content accidentally placed before the stable block.
- Compare request spacing with retention. A five-to-10-minute idle period can clear in-memory entries, and all in-memory entries expire within one hour of last use.
Costs rise after enabling caching
- On GPT-5.6+, compare cache writes with reads. Repeatedly changing the reusable prefix can create chargeable writes without enough discounted reads.
- Reduce unnecessary breakpoints and keep the highest-value stable blocks within the four-write-per-request limit.
- Validate that traffic is actually reaching the deployment where the cache metrics are being collected; moving between deployments can remove reuse.
Example architecture for a support assistant
Suppose a support application sends a 3,000-token policy and tool bundle on every call. Construct the prompt in this order:
- Stable system and developer rules, including safety and response-format instructions.
- Versioned tool definitions and the JSON output schema.
- The company’s reusable policy and product reference text.
- The append-only conversation history.
- The current customer question, account ID, current timestamp and newly retrieved passages.
On GPT-5.6+ Standard, assign a stable key for the policy-and-tools template, add breakpoints after the large policy blocks, and use explicit mode if only those blocks should be written. Keep customer-specific data after the breakpoints. If the service receives more than approximately 15 calls per minute for that exact key, route the same template through a small deterministic set of additional keys. Observe cached_tokens and cache_write_tokens before deciding whether more retention or additional sharding is worthwhile.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWhen prompt caching is—and is not—the right optimization
| Workload shape | Expected value | Recommended approach |
|---|---|---|
| Long, repeated system prompt with many calls in minutes | High; many tokens can be read from cache while in-memory retention is warm. | Stabilize the prefix, use a consistent key on GPT-5.6+, and monitor hit and write tokens. |
| Nightly or shift-based repetition | Potentially high if a supported model can retain entries for up to 24 hours. | Evaluate extended retention and confirm model support for 24h. |
| Mostly unique prompts or short requests | Low; requests may fail the 1,024-token minimum or never reuse a prefix. | Optimize prompt size and routing first rather than adding cache controls. |
| Very high rate on one prefix/key | Unpredictable; some requests may miss despite identical text. | Use deterministic key sharding and watch per-key request rates. |
| PTU-M deployment needing breakpoints | Limited by control availability. | PTU-M does not support breakpoints; use the controls documented for that deployment. |
Implementation checklist
- Confirm the model generation and deployment type before selecting fields.
- Ensure every cache candidate is at least 1,024 tokens long.
- Make the first 1,024 tokens byte-for-byte identical.
- Place stable instructions, tools, schemas and reference files before dynamic content.
- Keep conversation history append-only.
- Use a deterministic
prompt_cache_keyon GPT-5.6 and later, with planned sharding for busy prefixes. - Use Standard breakpoints only where supported; remember PTU-M has no breakpoint support.
- Choose
in_memoryor supported24hretention based on inter-request gaps. - Track
cached_tokens,cache_write_tokens, key-level request rate and effective input-token spend. - Recheck economics after prompt-template, model or deployment changes.
Bottom line for Azure cost planning
Prompt caching is a prefix-design problem before it is a switch-setting problem. A stable 1,024-token opening enables reuse; careful placement of dynamic data determines whether that reuse survives real application traffic. Standard deployments provide discounted reads, while Provisioned deployments can reach up to a 100% input-token discount, but GPT-5.6+ introduces write charges that reward repeated reads and penalize churn. Treat KV-cache reuse as a measurable workload behavior: stabilize the prompt, control key pressure, select retention deliberately and verify the result in token-level metrics.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
