Estimate an LLM accelerator in four passes: store the model weights, size the key-value (KV) cache for your longest and busiest workload, reserve memory for runtime allocations, then test whether the device’s memory bandwidth can move that data quickly enough during decode. A model that fits its weights can still run out of memory when context length or concurrency increases.
The calculations below are planning estimates, not guarantees of tokens per second. Exact fit and performance depend on the model’s attention configuration, quantization implementation, serving runtime, sharding, context limits, and batch behavior.
As an Amazon Associate I earn from qualifying purchases.
What the estimate must include
For an inference deployment, “GPU memory needed” is more than checkpoint size. A useful capacity estimate has four components:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Weights: the model parameters stored in the selected format, divided among tensor-parallel shards.
- KV cache: keys and values retained for every active sequence and token in its context.
- Runtime allocations: activations, temporary tensors, communication buffers, CUDA graphs, adapters, multimodal reservations, and other framework state.
- Headroom: memory left for allocator fragmentation, request variation, and runtime behavior that is not captured by a simple arithmetic sum.
After capacity, evaluate bandwidth. Prefill processes prompt tokens in parallel, while decode generates one token at a time. Decode repeatedly moves weights, KV data, and activations, so a device with enough capacity can still deliver poor latency if its memory system cannot feed the computation.
#1 Best Overall
- 16,384 NVIDIA CUDA Cores
- Supports 4K 120Hz HDR, 8K 60Hz HDR and variable refresh rate as indicated in HDMI 2.1A
- New streaming multiprocessors: up to 2x power and power efficiency
- Fourth generation tensor cores: up to 2x AI power
- Third-generation RT cores: up to 2x ray tracing performance
Collect the model and workload inputs first
Write down the values that control the estimate before choosing a card or server. Use the model configuration and checkpoint metadata rather than a rounded model name whenever possible.
Model values
- Total parameter count.
- Weight format and actual bytes per parameter after quantization.
- Number of transformer layers.
- Hidden size.
- Number of attention heads, number of KV heads, and head dimension.
- KV-cache element format, which can differ from the weight format.
- Any mixture-of-experts, multimodal, adapter, or hybrid-model components that the runtime loads.
Serving values
- Maximum prompt length.
- Maximum generated-token allowance.
- Maximum simultaneous sequences or requests.
- Whether the runtime counts both prompt and generated tokens in the cache reservation.
- Tensor-parallel degree and any pipeline or expert parallelism.
- Runtime features such as CUDA graphs, LoRA adapters, prefix caching, speculative decoding, or CPU KV offload.
Capacity planning should use the target runtime’s definitions. A “128K context” setting, for example, may reserve space differently from another server even when the advertised context limit is identical.
Step 1: estimate model-weight memory
The first-order relationship is:
weight bytes per GPU ≈ total parameter count × bytes per parameter ÷ tensor-parallel degree
For rough planning, NVIDIA’s NIM memory guidance uses these values:
| Weight format | Planning value | What it represents |
|---|---|---|
| BF16 or FP16 | 2 bytes per parameter | Uncompressed 16-bit weights |
| FP8 | 1 byte per parameter | 8-bit floating-point weights |
| INT4 or NVFP4 | 0.5 byte per parameter | Four-bit weight storage before implementation overhead |
These are storage heuristics. Scale factors, metadata, padding, checkpoint layout, kernel requirements, and runtime-specific packing can make the allocated amount differ from the multiplication above.
Worked weight examples
| Model and format | Sharding | Arithmetic estimate | Weight memory per GPU |
|---|---|---|---|
| Llama 3.1 8B, BF16 | One GPU | 8 billion × 2 bytes | 16 GB |
| Llama 3.3 70B, BF16 | Four GPUs | 70 billion × 2 ÷ 4 | 35 GB per GPU |
| Llama 3.3 70B, FP8 | Two GPUs | 70 billion × 1 ÷ 2 | 35 GB per GPU |
The 8B example is the basis for NVIDIA’s illustration of a 24 GB GPU, including an RTX 4090, leaving some memory for KV cache and runtime overhead. It does not mean every 8B serving workload fits on every 24 GB card: long contexts, concurrent users, cache precision, and framework allocations consume the remainder.
For a sharded model, divide the weight estimate by the intended tensor-parallel degree only when the implementation distributes the relevant tensors that way. Replication, uneven layer placement, expert routing, and pipeline stages can produce a different per-device footprint.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteStep 2: calculate KV-cache memory
The KV cache stores attention keys and values for tokens that remain in a sequence’s context. Its size rises with the number of layers, context tokens, active sequences, and bytes used for each cached element.
For a conventional attention layout, NVIDIA gives this per-token relationship in its inference-optimization article:
KV bytes per token = 2 × number of layers × (number of attention heads × head dimension) × bytes per cache element
The factor of two is for keys and values. For a batch of sequences:
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #2
- TRI FROZR 3-Stay cool and quiet. MSI’s TRI FROZR 3 thermal design enhances heat dissipation all around the graphics card.
- TORX FAN 5.0-Fan blades linked by ring arcs and a fan cowl work together to stabilize and maintain high-pressure airflow.
- Copper Baseplate-Heat from the GPU and memory modules is captured by a copper baseplate and then rapidly transferred to Core Pipes.
- Core Pipe-Precision-machined heat pipes ensure max contact and spread heat along the full length of the heatsink.
- Airflow Control-Sections of different heatsink fins disrupt unwanted airflow harmonics and reduce noise.
total KV bytes ≈ batch size × sequence length × 2 × number of layers × hidden size × bytes per cache element
Use the second form only when number of attention heads × head dimension = hidden size. Multi-query attention (MQA), grouped-query attention (GQA), and other designs use fewer KV heads than query heads. In those cases, substitute KV heads × head dimension for hidden size. This adjustment can materially reduce cache memory.
Count the full cached length
Define sequence length the way the serving engine does. If a request has a 12,000-token prompt and may generate 2,000 tokens, a runtime that reserves the full conversation needs capacity for about 14,000 cached tokens, not merely 12,000. Multiply that length by the maximum simultaneous sequences for a worst-case reservation. If your service enforces separate prompt and output limits, use the resulting maximum total per sequence.
Reference calculations
For Llama 2 7B, NVIDIA’s example uses 32 layers, a hidden size of 4,096, FP16 cache elements, a 4,096-token sequence, and batch one. The resulting KV cache is approximately 2 GB. That figure is an illustration for those dimensions and cache format; it is not a universal 7B allowance.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →A separate NVIDIA KV-cache offload article describes Llama 3 70B in FP16 as approximately 140 GB of weights and approximately 40 GB of KV cache for a single user at 128K tokens. NVIDIA notes that the cache portion scales linearly with the number of users in that example. Model architecture, cache precision, and allocator policy can change the actual result.
Why concurrency is often the capacity limiter
Weights are loaded once per shard, but each active sequence needs its own KV entries. Doubling concurrent sequences approximately doubles the cache reservation at a fixed context length. The same is true when doubling the context limit at a fixed concurrency. A service designed for many short requests can therefore fit on a device that cannot support a few very long requests, even though the model weights are identical.
Step 3: reserve runtime memory and headroom
Do not declare a deployment viable after adding only weights and KV cache. NVIDIA’s NIM documentation lists activations, communication buffers, CUDA graphs, LoRA adapters, multimodal reservations, and hybrid-model state among other consumers. The TensorRT-LLM memory documentation describes weights, activation tensors, runtime buffers, and KV-cache pools as separate allocations.
There is no universal runtime-overhead percentage that applies across frameworks and versions. A graph-captured engine, a dynamic-shape engine, and an eager-mode server can reserve different amounts. Quantization metadata and adapter modules can also alter the footprint.
Build a per-device budget
- Start with the accelerator’s advertised physical memory.
- Subtract the runtime’s measured non-cache allocations, including weights, activations, communication buffers, graph storage, adapters, and reserved multimodal state.
- Use the remainder as the maximum KV-cache pool, then keep operational headroom for allocator fragmentation and request variability.
- Check whether tensor-parallel ranks have equal free memory. The least-free rank determines whether a launch succeeds.
Do not convert the remaining number into a fixed “safe” percentage without measuring the chosen runtime. Instead, launch the intended engine with representative settings and inspect startup or profiling logs for actual allocations and reported cache capacity.
Step 4: turn the arithmetic into a sizing workflow
The following procedure produces a defensible first pass before benchmarking.
- Record model metadata. Capture parameter count, layers, hidden size, query-head count, KV-head count, head dimension, weight precision, and KV precision from the model configuration or checkpoint.
- Define the service envelope. Set maximum prompt tokens, maximum generated tokens, simultaneous sequences, and any reserved system or tool-call tokens. Make clear whether the runtime’s sequence length includes both input and output.
- Choose sharding. Select tensor-parallel and, if applicable, pipeline or expert parallel degrees. Compute weight memory for every rank rather than assuming perfect balance.
- Compute KV demand. Use the actual KV-head dimensions and cache bytes per element. Multiply by the worst-case sequence count and total cached tokens.
- Add measured runtime allocations. Include buffers, activations, graphs, adapters, communication storage, and multimodal reservations reported by the engine.
- Leave headroom. Keep capacity for allocator behavior and traffic that reaches the configured maximum. If the service must never reject a maximum-size request, size for that request rather than an average.
- Validate with the production engine. Run a startup/profile pass, confirm the cache pool, and then load a representative mixture of short and long requests.
Memory bandwidth and decode speed
Capacity answers “can it fit?” Bandwidth helps answer “can it generate fast enough?” NVIDIA separates prefill, which processes input tokens in parallel, from decode, which generates output autoregressively. During decode, the engine repeatedly transfers weights, keys, values, and activations. NVIDIA summarizes the consequence as: In other words, this is a memory-bound operation.
The statement appears in NVIDIA’s Mastering LLM Techniques: Inference Optimization article.
Rank #3
- NVIDIA Ada Lovelace Streaming Multiprocessors: Up to 2x performance and energy efficiency
- Tensor Cores of the 4th Generation: up to 2x AI performance
- RT-cores of the 3rd Generation: up to 2x raytracing performance
- OC mode: Boost clock 2595 MHz (OC mode) / 2565 MHz (gaming mode)
- Axial Tech fans deliver up to 23% higher airflow
A first-order bandwidth ceiling
For an intuitive low-batch estimate, divide effective memory bandwidth by the bytes that must be moved for each generated token:
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →rough tokens per second ceiling ≈ effective memory bandwidth ÷ bytes moved per generated token
Model weights are a major recurring movement cost in this simplified picture. KV reads increase with active context, while batching, cache reuse, kernel fusion, arithmetic intensity, compression, and hardware utilization change the amount actually transferred. Consequently, use the division as an order-of-magnitude reasoning tool, not as a benchmark formula or a promise of achieved tokens per second. Peak bandwidth is not the same as sustained bandwidth available to one request.
Prefill and decode stress different resources
- Prefill: large prompt batches expose matrix-compute throughput and can process many tokens concurrently. Long prompts may produce high temporary activation use.
- Decode: one or a few new tokens per sequence expose memory movement and scheduling overhead. More active sequences can improve utilization, but they also enlarge the KV working set.
- Mixed traffic: a production server may prefill one request while decoding others. Measure this overlap rather than extrapolating from an isolated prompt or isolated generation test.
Read the exact accelerator specification
Compare the named product variant, not just a family name. NVIDIA’s H100 specifications list these on-device values:
| Variant | GPU memory | GPU memory bandwidth | Planning implication |
|---|---|---|---|
| H100 SXM | 80 GB | 3.35 TB/s | Higher bandwidth than many host or PCIe alternatives, with 80 GB of capacity per device |
| H100 NVL | 94 GB | 3.9 TB/s | More capacity and bandwidth than the SXM figure listed above |
The same product page lists GPU-to-GPU interconnect specifications separately. Do not treat interconnect bandwidth as on-device GPU memory bandwidth. For multi-GPU inference, both matter: memory bandwidth feeds each rank’s local work, while the interconnect carries tensor-parallel or pipeline traffic.
Recommended Free Tools
Worked planning scenarios
8B BF16 on one 24 GB GPU
The first-order weight estimate is 16 GB. That leaves roughly 8 GB before accounting for the runtime, which must cover KV cache, activations, graphs, buffers, and any adapters. A short-context, low-concurrency service may fit, while a long-context or highly concurrent service may not. NVIDIA uses a 24 GB GPU, including an RTX 4090 example, only to illustrate that the weights can fit with some space remaining; it is not a throughput guarantee.
70B BF16 across four devices
The weight arithmetic gives 35 GB per GPU. The deployment still needs KV capacity and runtime allocations on every rank. If a 70B implementation uses GQA, calculate its cache from the configured KV-head count instead of multiplying by the full query-head count. A four-way layout also requires an interconnect suitable for the synchronization pattern used by the engine.
70B FP8 across two devices
Using the one-byte planning value, weights again total about 35 GB per GPU. The apparent equality with the BF16 four-way example does not imply equal performance: the two configurations differ in device count, communication volume, available cache headroom, kernel support, and the quality or calibration of the FP8 implementation.
128K single-user context
The cited Llama 3 70B illustration assigns about 40 GB of FP16 KV cache to one user at 128K tokens, in addition to about 140 GB of weights. Raising concurrency to two users would approximately double that cache component under the same assumptions. Before selecting hardware, verify the model’s actual KV dimensions and whether the runtime supports the requested context without hidden reservations.
When the estimate does not fit
Reduce the cache requirement
- Lower maximum context or generated-token limits where product requirements allow.
- Cap simultaneous sequences or use admission control for long requests.
- Use a lower-precision KV cache only when the runtime and model quality target support it.
- Evaluate prefix or prompt caching for repeated prefixes, recognizing that cached content still consumes memory in the implementation.
- Consider CPU KV offload when latency and host-memory capacity make the trade-off acceptable; transfers can reduce GPU pressure but add movement overhead.
Change the weight footprint
- Use a supported lower-precision weight format after checking accuracy and kernel compatibility.
- Increase tensor-parallel degree or move to devices with more memory.
- Choose a smaller model or a model with an architecture whose KV dimensions better match the workload.
Diagnose an out-of-memory launch
- Confirm the actual checkpoint and quantization format loaded, rather than the format named in a model card.
- Compare each rank’s allocation; one imbalanced rank can fail first.
- Inspect runtime logs for activation, graph, communication, adapter, and multimodal reservations.
- Check whether the configured maximum sequence length includes output tokens and whether the engine preallocates the full cache.
- Lower concurrency or context temporarily to identify whether weights or KV capacity is the limiting component.
- Repeat the test with the production engine version and kernel settings, because defaults are framework- and version-specific.
Validate with measurements before deployment
Once the arithmetic shows a plausible configuration, run the exact model, runtime, precision, and sharding combination. Record:
- Per-device weight allocation after loading.
- Non-cache allocations and the KV-cache pool size reported at startup.
- Prompt-processing latency at several input lengths.
- Decode tokens per second at one sequence and at target concurrency.
- Latency distributions while mixing short and long contexts.
- Failure behavior when the cache pool is exhausted or a request exceeds its limit.
Use the same tokenizer, stopping rules, batch scheduler, and maximum lengths as production. A benchmark that tests only a short prompt at batch one will not reveal the cache pressure or scheduling behavior of a high-concurrency, long-context service.
What these calculations cannot establish
They do not establish a universal runtime-overhead percentage, a guaranteed tokens-per-second result from peak bandwidth, current accelerator retail pricing, or a cost per million tokens. Infrastructure economics additionally require the device or instance price, utilization, power, cooling, orchestration, storage, and the measured token rate for the target traffic. Treat the formulas as a filter for impossible configurations, then use runtime telemetry and workload benchmarks for the final decision.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
