Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesFor the Q4_K_M example at 32K context, a useful starting point is --n-cpu-moe 22 on a 12 GB GPU, 13 on a 16 GB GPU, and 0 on a 24 GB GPU. At 128K context, the same example uses 26 and 17 for 12 GB and 16 GB, respectively. These are configuration-specific starting values—not universal optima. Your exact GGUF, context, KV-cache type, runtime build, batch settings, and available memory determine what fits.
What `–n-cpu-moe` changes
In the behavior described for the configurations below, --n-cpu-moe N keeps expert feed-forward tensors for the first N layers in system RAM and runs those experts on the CPU. The remaining experts stay on the GPU; attention, shared weights, and the KV cache remain GPU-resident. Raising N can reduce GPU memory pressure, but it also moves routed expert work to the CPU.
As an Amazon Associate I earn from qualifying purchases.
This behavior description is not a guarantee for every llama.cpp version or fork. Check the help output and model-loading log for the build you are actually running. Donald Lee’s 2026 DEV Community article explains its calculation this way: “Guessing and restarting the server a few times works, but you can compute it from the GGUF file itself.” That is the article author’s method, not an official llama.cpp statement.
Free tools Windows power users keep installed
One-click scans. No signup required.
Starting values for 12, 16, and 24 GB GPUs
The following values and throughput ranges come from a 2026 DEV Community article’s configuration table. The Q4_K_M rows refer to its stated example; the figures are not a controlled comparison across GPU tiers.
#1 Best Overall
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- 0dB technology lets you enjoy light gaming in relative silence
- Dual BIOS switch lets you toggle between Quiet and Performance BIOS profiles
- Dual ball fan bearings last up to twice as long as sleeve bearing designs
| GPU configuration | Context | --n-cpu-moe |
Reported VRAM | Reported decode rate |
|---|---|---|---|---|
| 12 GB RTX 3060 | 32K | 22 | 11.8 GiB | 24–41 tok/s |
| 12 GB | 128K | 26 | 11.8 GiB | 16–27 tok/s |
| 16 GB RTX 5060 Ti | 32K | 13 | 15.8 GiB | 31–53 tok/s |
| 16 GB | 128K | 17 | 15.9 GiB | 21–35 tok/s |
| 24 GB RTX 4090 | 32K | 0 | 21.7 GiB | 81–142 tok/s |
The source lists memory bandwidths of 360 GB/s, 448 GB/s, and the 24 GB RTX 4090 configuration for these examples. GPU model, CPU, quantization, runtime build, context, and other settings differ, so the token rates should not be read as a benchmark ranking or a prediction for your machine.
Why context changes the value
Longer context requires more KV-cache memory, leaving less room for GPU-resident weights and buffers. In the named model configuration, the DEV Community article estimates FP16 KV cache at 20,480 bytes per token: its calculation counts K and V for 10 full-attention layers, 2 KV heads, dimension 256, and 2 bytes per value. This is a configuration-specific estimate, not a universal cost for every quant or runtime setup.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
The table’s 12 GB example moves from N=22 at 32K to N=26 at 128K; the 16 GB example moves from 13 to 17. A separate 2026 community guide illustrates the same context effect with a different setup: its preferred values were 16 at 32K, 17 at 64K, 20 at 128K, and 22 at 258K. That guide used an APEX/abliterated GGUF, Windows-native ik_llama.cpp b5095, an RTX 4070 SUPER, an i5-14600KF, and 32 GB DDR4, so its settings are corroborating examples rather than direct replacements for the table above.
KV-cache type can affect practical fit and prefill
In that community guide’s reported 45K-token input comparison, prefill changed from 397 seconds with q8 KV to 85.5 seconds with q4_0 KV. The author also reported a “KL 0.10 overall” quality assessment for that setup; it should not be generalized to other prompts, models, or configurations. The example shows why context length alone is not enough to predict memory use or prompt-processing behavior.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
How to tune the value for your own GGUF
- Identify the exact model file and target. Record the GGUF variant and quantization, desired context, KV-cache type, and whether you need multiple parallel request slots. A value measured for another file or context is only a rough reference.
- Start from the nearest applicable example. For the Q4_K_M 32K example above, begin with 22, 13, or 0 for 12, 16, or 24 GB. For its 128K examples, begin with 26 or 17 for 12 or 16 GB. Treat these as initial trials, not guaranteed working values.
- Account for memory beyond weights. KV cache, compute buffers, batch and ubatch sizes, and parallel slots all affect headroom. The llama.cpp tutorial cited by the community guide notes that context is divided among parallel slots and that larger batch sizes use more memory. Check the settings and actual allocation in your build.
- Load, inspect, and adjust in small steps. Check the model-loading log and GPU memory use, then increase N if the configuration does not fit or lacks safe headroom. A higher N shifts more expert tensors to CPU. The community guide observed a sharp performance cliff near its own fit boundary and recommended a small sweep; that boundary is specific to its setup, not a universal N value.
- Measure prompt processing and generation separately. Prefill and decode can respond differently to memory and KV-cache choices. Record both if long prompts matter to your use case, rather than relying on one tokens-per-second figure.
- Keep the environment attached to the result. Note the llama.cpp version or fork, backend/build, GPU, CPU, system RAM, GGUF and quantization, context, KV-cache type, batch sizes, and parallel slots. In particular, results from Windows-native
ik_llama.cppshould not be presented as if they came from another build.
What the GGUF calculation can—and cannot—tell you
For unsloth/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, the DEV Community article reports 22,123,538,944 tensor bytes. Its fit calculation separates non-expert weights from expert tensors, subtracts the input embedding it says remains CPU-resident, adds GPU-resident experts and KV cache for the chosen context, and reserves roughly 1 GiB for CUDA context and compute buffers.
For that file, the article reports about 486,539,264 bytes of expert tensors per layer on most of 40 layers, and 2,555,013,632 bytes for the other tensors. Expert-layer sizes vary slightly by quant, so those values are file-specific; summing actual layer tensor sizes is more reliable than treating a per-layer estimate as a constant. File arithmetic helps estimate fit, but it does not replace a load test with your runtime, GPU, and serving settings.
Rank #4
- Powered by Radeon RX 9070 XT
- WINDFORCE Cooling System
- Hawk Fan
- Server-grade Thermal Conductive Gel
- RGB Lighting
What a 24 GB result does—and does not—establish
The Q4_K_M table’s 24 GB RTX 4090 example runs at 32K with N=0. Separately, Patrick Gawron reported a working recipe for a 24 GB setup using IQ4_XS weights, f16 KV, a 262,144-token context, and no CPU expert offload. The article did not measure exact VRAM use or speed for that recipe. It demonstrates that N=0 can work in at least one reported configuration; it does not establish that every 24 GB card, quantization, batch size, or parallel configuration will fit.
What to record when comparing settings
- Available VRAM and exact GPU model
- GGUF filename, quantization, and weight-file size
- Context length and KV-cache type
--n-cpu-moevalue- Batch and ubatch sizes, plus parallel slots
- CPU and system RAM
- Runtime version or fork, backend, and build
- Prompt-processing and decode throughput, measured separately if both matter
The published examples come from different hardware and software environments, and none of the cited sources provides a controlled test varying only GPU capacity. Compare settings within the same setup rather than ranking GPUs from their reported token rates.
Quick Recap
Best Value
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- Phase-change GPU thermal pad helps ensure optimal heat transfer, lowering GPU temperatures for enhanced performance and reliability
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- Dual-ball fan bearings last up to twice as long as standard conventional sleeve bearings designs
- 0dB technology lets you enjoy light gaming in relative silence
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
