Recommended Free Tools
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Quantization, FlashAttention, and KV-cache settings speed up local LLM inference in different ways: quantization shrinks the model weights, FlashAttention reduces attention memory traffic, and KV-cache quantization reduces the cache that grows with every context token. A practical default for a CUDA machine is a Q4_K_M GGUF that fits entirely on the GPU, -fa on, a context length you actually use, and the KV cache left at f16 unless it causes an out-of-memory error. Use matched q8_0 K/V cache types when the cache, rather than the weights, is your capacity limit.
What does each setting change?
These controls target different objects and bottlenecks. Confusing them leads to the wrong optimization: a smaller weight file does not reduce the KV cache, and a smaller KV cache does not automatically increase single-stream tokens per second.
As an Amazon Associate I earn from qualifying purchases.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall| Setting | What is compressed or rearranged | Main benefit | Typical cost |
|---|---|---|---|
| Weight quantization (for example, GGUF Q4_K_M) | Model parameters | Smaller file, lower VRAM use, usually faster decode | Some quality loss at very low bit widths; unpacking overhead below about 4 bits |
| FlashAttention | Attention intermediate memory traffic | Lower long-context memory use and faster prefill | Requires a supported kernel; benefits vary by GPU, sequence length, and runtime |
KV-cache type (-ctk/-ctv) |
Per-token key and value cache | More context or concurrent slots in the same VRAM | Stock llama.cpp can decode substantially slower at long context while dequantizing the cache |
The settings stack. A quantized model can use FlashAttention and an independently quantized KV cache, but each choice should be justified by whether weights, attention memory, or cache capacity is limiting your workload.
How does weight quantization make decoding faster?
Weight quantization stores parameters with fewer bits. FP16 uses two bytes per parameter, while an INT4-style representation uses roughly half a byte before block metadata. Local runtimes commonly distribute this data as a memory-mappable GGUF file containing tensors and metadata such as architecture, tokenizer, context length, and chat template.
Popular on Amazon
- 【90 Degree Full Function USB C Adapter】Supports the latest USB4 standard for up to 40Gbps of bandwidth, 8K video support, and up to 240W Fast charging; Support USB PD 3.1 Extended Power Range (EPR) including 140W, 180W and 240W. Your productivity is easily doubled. *Warm Tip: For the best fit, please make sure the USB-C port is fully exposed; thick or bulky cases may affect compatibility.
- 【Reliable Quality】Made from high-quality, scratch-resistant aluminum alloy, delicate metal frame. get excellent wear resistance and heat dissipation. Non-Slip Design for easier insertion and removal.
- 【Small and portable】Ultra thin size, ultra compact design, a petite physique that matches the minimalist design of your modern laptops. Coming with a slim figure that won't block the port next to the one it's plugged into.
- 【Compatibility】Widely work with most usb c, thunderbolt 4/3 devices. Compatibilit for ROG Ally, Steam Deck, Switch, MacBook Pro, Quest 3 VR headset, Surface Book 2, iPhone 18/17/16/15, Samsung Galaxy S23/S22/S21, Note 20/10, Google Pixel, ChromeBook, etc.
- 【Excellence Customer Support】If you experience any issues or cable failure, simply reach out to our dedicated customer service team. We provide a prompt response within 18 hours, providing you with the necessary technical support to resolve any concerns.
During autoregressive decode, each new token streams through most of the model for matrix-vector products. That operation is commonly limited by memory bandwidth, so reading fewer weight bytes usually raises tokens per second. Prompt processing (prefill) is more compute-bound and often changes much less between quantization levels.
A name such as Q4_K_M is a recipe, not a single four-bit codec. “Q4” describes the base width; “K” denotes k-quant super-blocks with shared scales; and “M” means selected sensitive tensors are promoted to a higher precision. In llama.cpp’s recipe, many tensors use Q4_K while output, attention-value, and selected feed-forward tensors receive higher-precision treatment. The resulting Llama 3.1 8B file is about 4.89 bits per weight, not exactly four.
The official llama.cpp quantization table, reproduced by Compile N Run, illustrates the direction. Its hardware is not identified, so treat the throughput figures as comparative guidance rather than a promise for your GPU:
| Format | Bits/weight | Model size | Prompt, 512 tokens | Generation, 128 tokens |
|---|---|---|---|---|
| F16 | 16.00 | 14.96 GiB | 923 tok/s | 29.2 tok/s |
| Q8_0 | 8.50 | 7.95 GiB | 865 tok/s | 50.9 tok/s |
| Q6_K | 6.56 | 6.14 GiB | 812 tok/s | 58.7 tok/s |
| Q5_K_M | 5.70 | 5.33 GiB | 759 tok/s | 67.2 tok/s |
| Q4_K_M | 4.89 | 4.58 GiB | 822 tok/s | 71.9 tok/s |
| Q4_K_S | 4.67 | 4.36 GiB | 819 tok/s | 76.7 tok/s |
| Q3_K_M | 4.00 | 3.74 GiB | 783 tok/s | 71.7 tok/s |
| Q2_K | 3.16 | 2.95 GiB | 784 tok/s | 79.9 tok/s |
The table’s generation column rises as the file shrinks until unpacking and arithmetic overhead begin to offset the byte savings. Q4_K_M is therefore a common local starting point: it is substantially smaller than F16 while avoiding the larger quality penalties seen in very low-bit formats.
Perplexity data summarized by QuestBrief show why the format choice is model-dependent. For Llama 3 8B, reported loss was 0.04% for Q8_0, 2.8% for Q4_K_M, 10.5% for Q3_K_M, and 56.5% for Q2_K. The same Q4_K_M setting raised perplexity about 1.4% on Llama 2 7B, so a quantization label cannot be assumed to have identical quality across model families. Importance-matrix (imatrix) quantization improved the cited Q4_K_M delta from 0.175 to 0.151.
Use Q5_K_M or Q6_K when your GPU has room and the task is sensitive to exact wording, code, or structured extraction. Move below Q4 only when fitting the model is otherwise impossible, and do not requantize an already quantized file; start from the original higher-precision weights.
What does FlashAttention change?
Standard attention forms score and probability matrices whose storage grows with the square of sequence length. FlashAttention tiles queries, keys, and values through fast on-chip SRAM, applies an online softmax, and avoids writing those large intermediates to high-bandwidth memory. The result is exact attention, not an approximation.
Also popular on Amazon
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Tri Dao’s FlashAttention-2 paper reports “2-4× wall-clock time speedup over optimized baselines, up to 10-20× memory saving, with no approximation.” The same paper describes FlashAttention-2 as about twice as fast as the first version, reaching 50–73% of theoretical A100 FLOPs/s. Those are algorithm-level results, not guaranteed llama.cpp numbers, but they explain why long prompts and prefill benefit most.
FlashAttention-2 improves parallelism and reduces non-matrix-multiplication work. FlashAttention-3 adds Hopper-focused asynchronous execution and FP8 attention; a secondary 2026 recap reports an additional 1.5–2× in suitable H100 workloads, but that figure should not be treated as a universal local-inference result.
In llama.cpp, enable the runtime’s own CUDA or HIP kernels with -fa on; -fa auto lets the runtime select the path. This is not the same as installing a separate Python flash-attn package. In Ollama, the corresponding switch is OLLAMA_FLASH_ATTENTION=1. Hugging Face Transformers exposes attn_implementation="flash_attention_2". Support and performance depend on the model architecture, GPU, build, and sequence length.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsFlashAttention is also a prerequisite for a quantized V cache in current llama.cpp. If you request one without it, the runtime reports: quantized V cache was requested, but this requires Flash Attention. Leaving -fa auto enabled normally avoids that startup failure.
Why is the KV cache a separate VRAM problem?
The KV cache stores each layer’s key and value vectors for tokens already in the conversation. Without it, every generated token would recompute attention for the entire prefix. The cache therefore grows linearly with context length and with the number of active conversation slots.
For a full-attention model, a useful estimate is:
KV bytes = 2 × cached layers × KV heads × head dimension × tokens × bytes per value
For Llama 3.1 8B in FP16, the calculation is 2 × 32 × 8 × 128 × 2 = 131,072 bytes per token, or 128 KiB. That is approximately 1 GiB at 8K tokens, 4 GiB at 32K, and 16 GiB at 128K. A Q4_K_M model that occupies about 4.6 GiB can therefore fit easily while its 128K FP16 cache exceeds the budget of a 24 GB card.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Also popular on Amazon
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Grouped-query attention reduces this burden before cache quantization: fewer KV heads serve more query heads. Sliding-window or hybrid architectures reduce it further by caching only full-attention layers. Always check the model architecture rather than multiplying parameter count by a generic rule.
| Example | FP16 cache | Q8_0 cache | Q4_0 cache |
|---|---|---|---|
| Llama 3.1 8B, 32K | 4.00 GB | 2.13 GB | 1.13 GB |
| Llama 3.1 70B, 128K, one request | 40.0 GB | 21.3 GB | 11.3 GB |
| Devstral Small 2 24B, 128K | 21.47 GB | 11.41 GB | 6.04 GB |
These values come from the cache-size calculations collected by ModelVRAM and the cited model configurations. They describe capacity, not a promise that every runtime will reserve exactly those numbers.
Should I use q8_0 or q4_0 for the KV cache?
Use FP16 when it fits and single-stream speed at long context is important. Use matched q8_0 K/V types when context length or concurrent slots would otherwise run out of VRAM; use q4_0 only when the additional capacity is essential and you accept a larger quality and speed trade-off.
In llama.cpp, q8_0 stores 34 bytes for 32 values (8.5 bits/value), about 53% of FP16 storage. q4_0 stores 18 bytes for 32 values (4.5 bits/value), about 28% of FP16. K and V are not equally sensitive: keys participate in the query-key ranking, while values are combined after that ranking. A public llama.cpp comparison reported only +0.03% perplexity for q8_0 K/V, +0.35% for q8_0 K with q4_0 V, and +3.31% for q4_0 K/V; the author concluded that K is more sensitive than V.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Quality is model-dependent. In a Qwen3-8B measurement, q8_0 stayed within roughly -0.0% to +0.1% perplexity of FP16 over 4K–32K, while q4_0 rose about 0.5–1.1%. Other cited measurements found considerably larger KL changes on some model families, so “q8 is lossless” is not a general law.
Speed is the less pleasant trade-off. SOTAAZ measured Qwen3-8B Q4_K_M on one A100 80GB with llama.cpp build b11390. At 32K cache depth, FP16 used 4.50 GiB, q8_0 2.39 GiB, and q4_0 1.27 GiB. At 64K decode depth, FP16 produced 79.6 tok/s, q8_0 44.3 tok/s (56%), and q4_0 42.4 tok/s (53%). Prefill at 4,096 tokens stayed within 3% across those types. At 8K depth q8_0 retained 83% of FP16 speed, showing why short chats can hide the penalty.
The reason is that stock kernels may dequantize the cache on every decode step. Newer builds and specialized kernels can change this balance, and a serving engine may trade cache bytes for higher batch throughput. Treat the cited A100 results as measured behavior for that model and build, not a universal benchmark.
Also popular on Amazon
- Portable and powerful USB-C HUB: BENFEI USB Type-C HUB, with super-soft and knot-free silicone woven design cable, meets most mobile office needs. Compact, lightweight, stylish, and powerful portable USB C Hub equipped with 1 x HDMI port, 1 x 100W charging, and 3 x USB ports. 18-month warranty, 24-hour response, to ensure you feel at ease when using our product.
- Design centered on comfort and reliability: Thanks to BENFEI's end-to-end in-house cable production capability, in-house PCBA and assembly capability, using the industry's most advanced silicone woven design and process, 20cm cable in length, no knots, super-soft, the HUB is easy to use in all scenarios: laptop, tablet, stand etc. Super-soft, 25000+ life cycles, to meet your daily carrying and office needs.
- 100W Charging: Support up to 90W USB C pass-through charging via Type-C port to keep your laptop powered. 10W is reserved for other interface operations. No data and video function on the Type-C port.
- 4K HDMI Display: The HDMI port supports media display at resolutions up to 4K 30Hz, keeping every incredible moment detailed and ultra vivid. Please note that the C port of the Host device needs to support video output.
- Transfer Files in Seconds: Transfer files and from your laptop at speeds up to 10 Gbps with USB A 3.2 port. Extra 2 USB A 2.0 ports are perfectly for your keyboards and mouse.
How do I configure llama.cpp?
Start with enough VRAM for the model, the intended context, and roughly 1 GiB of runtime headroom. Then use full GPU offload and FlashAttention before changing cache precision.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Choose the weight file. Prefer Q4_K_M when capacity is tight; choose Q5_K_M or Q6_K if quality headroom remains.
- Offload all layers. Use
-ngl 99(or-ngl allwhere supported). Partial CPU offload can make decode dramatically slower. - Enable attention tiling. Add
-fa on, or leave-fa autoand verify the startup log. - Set realistic context. For example,
-c 32768allocates for 32K rather than the model’s maximum. A large default context can consume VRAM before a request arrives. - Quantize the cache only if needed. Use matched types, for example:
llama-server -m model.gguf -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 32768
The short options are -ctk/--cache-type-k and -ctv/--cache-type-v; there is no --ctk spelling. Environment equivalents are LLAMA_ARG_CACHE_TYPE_K and LLAMA_ARG_CACHE_TYPE_V. Supported values include f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, and q5_1, subject to the build’s compiled kernels.
CUDA builds may support only selected quantized K/V pairs natively. If the log says it is “converting K and V to f16 instead,” the requested pair is falling back to a slower path. Matching K and V at q8_0 is the safest first choice; mixed q8_0/q4_0 can save more memory but measured only 42% of FP16 decode speed at 64K in the cited SOTAAZ build.
To compare your actual workload, benchmark at the context depth you use rather than an empty cache:
for t in f16 q8_0 q4_0; do
llama-bench -m model.gguf -ngl 99 -fa on
-ctk $t -ctv $t -p 4096 -n 128 -d 0 -d 32768
done
Builds change frequently, so record the llama.cpp commit and GPU with any result.
How do I configure Ollama, LM Studio, vLLM, or MLX?
Each runtime exposes the same underlying idea with different labels.
- Ollama: start the server with
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve. The cache setting is global to the server, and quantization is applied only when FlashAttention is enabled. - LM Studio: select K/V cache quantization in the llama.cpp engine load settings. Its SDK exposes
llamaKCacheQuantizationTypeandllamaVCacheQuantizationType; the feature was added in version 0.3.7 with llama.cpp runtime 1.9.0 or newer. - vLLM: use
--kv-cache-dtype fp8, for examplevllm serve <model> --kv-cache-dtype fp8. CUDA supportsfp8_e4m3andfp8_e5m2; ROCm supportsfp8_e4m3. Calibrated scales are preferable, and--kv-cache-dtype-skip-layers sliding_windowcan leave sensitive layers at their native type. - MLX:
mlx_lm.serverexposes--kv-bits 4or--kv-bits 8. The documented option disables batching, so it is aimed at a different workload from a multi-request serving engine.
Serving results do not map directly to single-user llama.cpp. In a 2024 H100 measurement, TensorRT-LLM’s 8-bit KV mode reached up to 1.45× throughput on decode-heavy traffic, while vLLM’s FP8 KV path did not improve throughput in that setup because its default FlashAttention-2 backend did not support quantized KV and the run fell back to other backends. Backend support has evolved, so check the version you deploy.
Also popular on Amazon
- 【Read before You Buy】 Note: This adapter is USB 2.0 version and cannot transmit HDM, VGA, or other video signals. Nevertheless, they are fully capable of charging devices,duplicating files, and can be seamlessly integrated with USB-C earphones, wireless network adapters, and SD card adapters.
- 【Easy to Use and Safe】Thanks to modern manufacturing techniques, we've made these adapters sufficiently compact so that they can remain inserted in your device without noticeable protrusion or inconvenience. This means you don't have to constantly plug and unplug them, reducing the likelihood of wear and misalignment, and making them more convenient to carry around.
- 【Reliable Quality】Engineered for enhanced durability, durable aluminum alloy shell material,get excellent wear resistance and heat dissipation,the male USB C and female USB connectors & the male USB and female USB C connectors are designed to be plugged and unplugged up to 20000 times. Provides data transfer, device charging, memory space expansion, and additional computer peripherals.
- 【Think more for you】Do you need to connect a USB-C-enabled device to an odd device that only has USB-A ports? Then these USB-A male to USB-C female adapters are made for you.Simply insert the adapter into the USB-A port on the device, and you can instantly connect your USB-C device to the device that originally didn't have the correct connection port.
- 【Trusted xiwxi Quality and Service】As a responsible brand seller, you will receive xiwxi's 12-month Quality assurance and lifetime friendly customer service, you're not just buying a product,you're investing in peace of mind with all-round quality and exceptional customer service.
What is the best order for tuning a local setup?
For a single-user desktop, optimize capacity first, then attention, then cache precision.
- Fit the largest useful model. A rule of thumb from ComputingForGeeks puts Q4_K_M near 0.57 GiB per billion parameters: about 4.6 GiB for 8B, 18.2 GiB for 32B, and 39.9 GiB for 70B. A 24 GB-class card can often target a 32B Q4_K_M model, but context and runtime overhead still require calculation.
- Reserve the cache. On a 24 GB card, a 32B Q4_K_M model plus overhead may leave only a few GiB. If FP16 cache reaches the limit around 19K tokens, q8_0 can extend the practical range to roughly 35K in the cited arithmetic example.
- Turn on FlashAttention. It is especially valuable for long prompts and is required for quantized V cache.
- Keep FP16 KV when it fits. This preserves long-context decode speed.
- Use q8_0 K/V when the cache is the blocker. Move to q4_0 only when the extra capacity determines whether the workload starts.
- For multi-user serving, consider a serving engine. Continuous batching, paged cache allocation, and prefix reuse can raise aggregate throughput; those gains should not be advertised as single-stream latency improvements.
Architecture can be a better lever than cache bits. Grouped-query attention, sliding windows, and hybrid layers reduce cache growth at native precision. Also keep concurrent slots under control: four slots at 32K can require roughly four times the cache of one slot.
What failures and misleading results should I watch for?
- Out-of-memory at startup: lower
-c, reduce the number of slots, use a smaller weight quantization, or switch matched K/V cache to q8_0. - Quantized V-cache error: enable
-fa onor-fa auto; V quantization cannot use the non-FlashAttention path. - Unexpectedly slow decode: inspect logs for K/V conversion to FP16, CPU-offloaded layers, or an unsupported kernel pair. Benchmark at real context depth.
- Quality degradation: restore FP16 KV, then q8_0, before changing the model weights. If weight quality remains insufficient, move from Q4_K_M to Q5_K_M or Q6_K.
- Comparing unlike measurements: note GPU, model, llama.cpp build, context depth, batch size, and whether the number is prefill or decode. A short prompt can conceal a long-context cache penalty.
The central distinction is capacity versus speed. Quantized KV can double the number of tokens or active requests that fit, but on stock llama.cpp it may reduce single-stream decode speed once the cache is deep. FlashAttention reduces attention I/O and memory growth without approximating the result. Weight quantization usually delivers the most obvious decode speedup because every generated token reads the model weights.
How we chose the practical defaults
The recommendations above follow four criteria in the cited research: which object is compressed, whether the goal is model fit, context capacity, or tokens per second, the measured quality impact, and whether the runtime has a native kernel for the selected type. They also separate single-stream local use from batched serving, because the same cache reduction can hurt one workload while helping another.
Frequently Asked Questions
How much VRAM does a 32K context window add to an 8B model?
For the cited Llama 3.1 8B architecture, the FP16 KV cache is about 4.00 GB at 32K tokens. Q8_0 lowers that to about 2.13 GB and Q4_0 to about 1.13 GB, before allocator and runtime overhead.
Does FlashAttention reduce the model file size?
No. FlashAttention changes how attention intermediates are tiled and moved through memory at runtime. The GGUF weights occupy the same space; weight quantization is the setting that shrinks that file.
Can I quantize only the K cache?
llama.cpp permits separate K and V types, and K-only quantization does not require FlashAttention. In practice, matched K/V types are easier to keep on a native CUDA kernel, while K is generally more sensitive to precision than V.
Why does a larger context setting consume VRAM before I send a long prompt?
Runtimes reserve cache capacity for the configured context and number of slots. Setting a model maximum that you rarely use can allocate tens of gigabytes unnecessarily; set -c close to your real workload and increase it when needed.

