Parameter count multiplied by bytes per parameter gives a useful first estimate for model weights, not a complete GPU-memory requirement. In inference, add the key-value (KV) cache, activations, runtime buffers and workload concurrency. In training, add gradients, optimizer states, master-weight copies, saved activations and temporary tensors. A reliable estimate therefore starts with arithmetic, states every workload assumption, and ends with a measurement on the exact model-serving or training stack.
What GPU memory must hold
GPU memory is consumed by several different categories, and each category responds to different inputs:
As an Amazon Associate I earn from qualifying purchases.
- Weights: the model parameters in the format used by the runtime.
- KV cache: attention keys and values retained for active inference requests.
- Activations: intermediate tensors produced while generating tokens or running training batches.
- Training state: gradients, optimizer moments, master copies and related model-state tensors.
- Runtime allocations: input/output tensors, CUDA workspaces, communication buffers, graph allocations, allocator fragmentation and other engine-specific memory.
NVIDIA describes weights and KV cache as the two main contributors to inference memory, but neither category can be considered in isolation. A model that fits by weight arithmetic can still fail when the requested context, concurrency or runtime workspace is added.
Inputs to record before calculating
Write down the exact configuration rather than using a model name alone. Two checkpoints with the same nominal parameter count can have different KV-cache requirements or runtime behavior.
#1 Best Overall
- 16,384 NVIDIA CUDA Cores
- Supports 4K 120Hz HDR, 8K 60Hz HDR and variable refresh rate as indicated in HDMI 2.1A
- New streaming multiprocessors: up to 2x power and power efficiency
- Fourth generation tensor cores: up to 2x AI power
- Third-generation RT cores: up to 2x ray tracing performance
- Parameter count and the exact checkpoint or quantized variant.
- Number of layers, hidden size, attention head dimension and number of key/value heads.
- Weight precision and KV-cache precision: for example, BF16, FP16, FP8 or INT4.
- Maximum prompt length, maximum generated length and the resulting context length.
- Inference batch size or maximum concurrent requests.
- For training: optimizer, mixed-precision policy, microbatch size, global batch size, sequence length, activation checkpointing and parallelism.
- Tensor-parallel and pipeline-parallel degrees, plus any parameter or optimizer sharding.
- GPU memory available to the process after the operating system, driver and other resident processes have taken their share.
NVIDIA notes that parameter counts are commonly available in a model card or safetensors index metadata. Confirm the value from the actual files; do not infer it from a rounded marketing name such as “7B” or “8B.”
Inference: a repeatable memory estimate
1. Calculate weight storage
Use this planning equation:
weight_memory = parameter_count × bytes_per_parameter ÷ tensor_parallel_degree
NVIDIA’s current NIM guidance uses 2 bytes per parameter for BF16 or FP16, 1 byte for FP8, and 0.5 byte for INT4 or NVFP4. These are arithmetic values for the parameter data itself. Quantized formats may also require scales, metadata, dequantization buffers or other non-weight allocations.
| Format | Planning bytes per parameter | Example for 8 billion parameters before sharding |
|---|---|---|
| BF16 or FP16 | 2 | About 16 GB |
| FP8 | 1 | About 8 GB |
| INT4 or NVFP4 | 0.5 | About 4 GB |
For tensor parallelism of four, divide the weight figure by four as a first approximation. The division describes storage distribution; it does not remove communication buffers or guarantee equal peak usage on every GPU.
2. Estimate the KV cache
Every active request retains key and value tensors for its context. The cache grows with the number of tokens and with the number of simultaneous sequences, so a short single-user test can understate production memory.
For a conventional full-head transformer, NVIDIA gives this simplified form:
KV_cache = batch × sequence_length × 2 × layers × hidden_size × bytes_per_element
The factor of two represents keys and values. Grouped-query attention (GQA) and multi-query attention (MQA) use fewer KV heads than query heads, so the simplified full-head expression can overestimate or misrepresent the actual requirement. Use the architecture-specific form instead:
KV_cache = 2 × kv_dtype_bytes × layers × kv_heads × head_dim × context_length × batch_size
A planned context must include both prompt tokens and generated tokens. If the service accepts a 12,000-token prompt and can generate 4,000 tokens, size the cache for 16,000 tokens unless the engine enforces a smaller limit. Prefix caching, paged-cache implementations and different cache precisions can change allocation behavior, but they do not make the token, layer or KV-head inputs irrelevant.
3. Add activations, I/O and runtime memory
Inference engines allocate internal activations and input/output tensors in addition to weights and KV cache. TensorRT-LLM documentation treats weights, internal activations and I/O tensors as major categories and provides configurable KV-cache allocation. CUDA workspaces, graph capture, tokenizer or multimodal staging buffers, adapters and safety or reranking components may also reserve memory.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Because these allocations depend on the engine and launch settings, do not hide them inside a universal percentage. Leave an explicit overhead line in your worksheet and replace it with a runtime-reported value during validation.
4. Compare with usable, not advertised, capacity
Compare the estimate with memory available to the model process, not the number printed on the GPU box. If the estimate is close to the limit, allocator fragmentation or a temporary workspace can cause an out-of-memory error even when the arithmetic total appears slightly smaller.
Rank #2
- TRI FROZR 3-Stay cool and quiet. MSI’s TRI FROZR 3 thermal design enhances heat dissipation all around the graphics card.
- TORX FAN 5.0-Fan blades linked by ring arcs and a fan cowl work together to stabilize and maintain high-pressure airflow.
- Copper Baseplate-Heat from the GPU and memory modules is captured by a copper baseplate and then rapidly transferred to Core Pipes.
- Core Pipe-Precision-machined heat pipes ensure max contact and spread heat along the full length of the heatsink.
- Airflow Control-Sections of different heatsink fins disrupt unwanted airflow harmonics and reduce noise.
If it does not fit, change one workload variable at a time:
- Use a smaller checkpoint or a supported lower-bit weight format.
- Reduce maximum context or maximum concurrent sequences.
- Use a lower-precision KV cache if the model and engine support it and quality remains acceptable.
- Enable an engine’s paged or budgeted KV-cache policy.
- Shard the model with tensor or pipeline parallelism.
- Move noncritical components off the GPU where the runtime supports it.
Inference examples and what they do (and do not) prove
| Published example | Calculation or condition | Correct interpretation |
|---|---|---|
| 8B model in BF16 | About 16 GB for weights | NVIDIA uses this as a weight-load example. A 24 GB GPU such as an RTX 4090 or A10G may have remaining capacity for KV cache and overhead in the cited scenario, but context, concurrency, architecture and runtime determine the actual fit. |
| 70B model in BF16 with tensor parallelism 4 | About 35 GB of weights per GPU | The remaining memory for KV cache and runtime allocations depends on each GPU’s capacity and launch configuration. |
| 7B model in FP16 or BF16 | About 14 GB for weights | NVIDIA Developer presents this as weights only, not a complete serving requirement. |
| 7B Llama 2, 4,096 tokens, batch 1, FP16 | About 2 GB KV cache | An architecture-specific illustration; it should not be applied to a different layer count, KV-head layout, context or cache format. |
| Mistral 7B, 16,000 tokens, one request | About 1.95 GB KV cache in BF16 | AWS’s GQA example uses 8 KV heads, 32 layers and head dimension 128. |
| Mistral 7B, 16,000 tokens, four concurrent requests | About 7.81 GB KV cache in the same BF16 example | The near-linear increase demonstrates why concurrency belongs in the estimate. |
| 70B model in FP8 | Approximately 70 GB for weights | AWS gives this as a weights-only example; one 48 GB L40S cannot hold that component alone. |
These figures are source-published technical examples, not universal capacity guarantees or dated market statistics.
Training: account for model state and activations
Training normally needs much more memory than inference because the GPU must retain information needed for backpropagation and parameter updates. The right calculation depends on whether you are full-fine-tuning the model or using a parameter-efficient method, which optimizer is selected, how precision is managed and how state is sharded.
1. Declare the training configuration
Before using a multiplier, record:
- Full-model training, supervised fine-tuning, or a parameter-efficient method.
- Optimizer type and whether it stores two moment tensors, as Adam does.
- Weight, gradient, optimizer-state and master-copy precisions.
- Microbatch size per GPU, global batch size and sequence length.
- Activation checkpointing or recomputation settings.
- Data, tensor, pipeline and optimizer-sharding strategy.
- Any CPU or NVMe offload and the resulting transfer constraints.
The available accounting examples describe full-model state. They do not establish one universal LoRA or QLoRA total; adapter memory, frozen quantized weights, optimizer choice and activation requirements still need to be measured for the selected framework.
2. Calculate model-state memory from the actual optimizer
Hugging Face’s mixed-precision example counts:
- 6 bytes per parameter for FP16 working weights plus FP32 weight copies.
- 8 bytes per parameter for two FP32 Adam states (the first and second moments).
- 4 bytes per parameter for FP32 gradients.
That arrangement totals 18 bytes per parameter before saved activations and temporary tensors. A 4-billion-parameter model would therefore require roughly 72 GB for those state categories under that accounting, before activation memory is added. Hugging Face also documents an illustrative 4B mixed-precision configuration with batch size 16 using roughly 85 GB of total GPU memory; treat it as a configuration-specific example, not a general multiplier.
Megatron Bridge describes a different but related planning convention: 18 bytes per parameter for model state when its distributed optimizer is disabled, or:
Recommended Free Tools
model_state_bytes_per_parameter = 6 + 12 ÷ shard_size
when its distributed optimizer is enabled. The shard size and the framework’s definitions determine the result. Do not combine this equation with the Hugging Face breakdown as though they were independent charges, and do not apply either number to another optimizer without checking its state layout.
3. Estimate saved activations
Activations are often the largest variable after model state. Their peak depends on microbatch size, sequence length, layer count and hidden size. Longer sequences and larger microbatches increase the number of intermediate values retained for backward computation. Activation checkpointing reduces what is stored by recomputing selected layers, trading memory for additional compute.
Temporary tensors created by attention, matrix multiplications, normalization and collective operations can make peak usage higher than a simple sum of persistent tensors. The peak can also occur during an optimizer step rather than during the forward pass, so inspect the entire iteration.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
4. Include sharding and exclusions
Sharding distributes model state across GPUs, but it changes communication and peak-memory behavior. Megatron Bridge’s theoretical estimate targets the most-loaded GPU shard and explicitly excludes allocator fragmentation, CUDA workspace, NCCL buffers, CUDA-graph static buffers, token-routing imbalance and CPU offloading. Those exclusions are reasons to treat the result as a planning signal rather than a launch guarantee.
How parallelism changes the decision
Tensor parallelism
Tensor parallelism splits layers or matrix operations across GPUs. It can reduce the weight and some state held by each GPU, but every participating GPU needs communication buffers and must exchange intermediate results. Increasing the degree can make a model eligible while reducing efficiency if interconnect bandwidth is insufficient.
Pipeline parallelism
Pipeline parallelism assigns groups of layers to different GPUs. It changes which activations are resident on each device and introduces pipeline scheduling effects. Microbatch count, bubbles and activation-transfer buffers therefore matter when estimating peak memory and throughput.
Rank #3
- NVIDIA Ada Lovelace Streaming Multiprocessors: Up to 2x performance and energy efficiency
- Tensor Cores of the 4th Generation: up to 2x AI performance
- RT-cores of the 3rd Generation: up to 2x raytracing performance
- OC mode: Boost clock 2595 MHz (OC mode) / 2565 MHz (gaming mode)
- Axial Tech fans deliver up to 23% higher airflow
Optimizer and parameter sharding
Distributed optimizers divide optimizer states or other model-state tensors across workers. The per-GPU reduction depends on shard count and implementation. Verify whether gradients, master weights and temporary update buffers are replicated or partitioned; a label such as “distributed optimizer” is not enough to determine the total.
Free tools Windows power users keep installed
One-click scans. No signup required.
For inference, sharding can make the weights fit but does not automatically make the service fast. AWS recommends evaluating latency and throughput after confirming that the accelerator memory is sufficient.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A practical worksheet
Use separate worksheets for inference and training so that a weight-only number is never mistaken for a total.
Inference worksheet
- Enter parameter count and the exact weight format.
- Convert the format to bytes per parameter: 2 for BF16/FP16, 1 for FP8 or 0.5 for INT4/NVFP4 as an initial planning value.
- Multiply parameters by bytes and divide by tensor-parallel degree.
- Enter layers, KV heads, head dimension, KV-cache bytes, maximum prompt tokens, maximum generated tokens and concurrent sequences.
- Calculate the general KV-cache expression using total context tokens.
- Add a separate line for activations, I/O tensors, runtime workspaces, graph memory, adapters and any multimodal reservations.
- Compare the sum with usable process memory, then test the exact engine at maximum planned context and concurrency.
Training worksheet
- Specify optimizer, precision policy, full fine-tuning or parameter-efficient mode, and all sharding degrees.
- Calculate model-state bytes from the framework’s documented tensor layout rather than borrowing a generic multiplier.
- Enter microbatch size, sequence length, layers and hidden size for activation planning.
- Mark whether activation checkpointing, recomputation, offload or memory-efficient attention is enabled.
- Add temporary optimizer, communication and workspace requirements if the framework reports them.
- Estimate the most-loaded GPU, not only the average across devices.
- Profile a representative iteration, including forward, backward and optimizer-step peaks.
Validation: turn an estimate into a safe launch
A theoretical sum is useful for rejecting obviously undersized GPUs, but a borderline result needs a measurement. Validate the same checkpoint, precision, engine version, parallelism, maximum sequence length and concurrency that production will use.
- Start with a small, controlled run and record peak allocated and reserved memory.
- Warm up the engine before measuring; graph capture and kernel autotuning can allocate memory after the first request.
- Exercise the longest prompt, longest generation and maximum concurrent requests separately and together.
- For training, include gradient accumulation, checkpointing, evaluation and the optimizer step in the profile.
- Check every GPU in a distributed job for the highest peak and for imbalance.
- Leave operational headroom for fragmentation, variable request lengths, retries and rolling model updates.
If measured peak usage is substantially above the worksheet, inspect runtime logs for workspace, communication, graph, cache or allocator reservations. If it is below the estimate, retain the conservative assumptions unless the same configuration has been tested across expected workloads.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteCommon estimation mistakes
Treating weights as the whole inference footprint
Parameter count times nominal bytes excludes KV cache and runtime allocations. It can identify a lower bound, not a safe serving capacity.
Using the full-head KV formula for GQA or MQA
Count the actual KV heads and head dimension. The number of query heads alone does not determine cache size.
Multiplying a training model size by one “standard” factor
Optimizer states, master weights, gradients, sharding and precision differ by framework. State the configuration behind every multiplier.
Ignoring token concurrency
KV cache scales with active sequences and context tokens. A batch-one benchmark does not represent a four-request service.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesEquating memory eligibility with performance
A multi-GPU arrangement can fit while failing latency or throughput objectives because of interconnect and synchronization costs. Measure both memory and performance.
Trusting advertised capacity at the boundary
Drivers, other processes, fragmentation and temporary workspaces reduce memory available to the model. A few hundred megabytes of theoretical headroom is not a production safety margin.
Choosing the next action from the estimate
| Result | Next action |
|---|---|
| Weights alone exceed usable memory | Change precision, use a smaller model, or increase weight sharding. No cache or runtime optimization can make the unsharded weight load disappear. |
| Weights fit but KV cache exceeds the budget | Reduce context or concurrency, use a supported cache format, enable cache paging or add GPUs. |
| Persistent tensors fit but peak allocation fails | Inspect activations, workspaces, graph capture, communication buffers and fragmentation; then profile with the target engine. |
| Training state fits but activations do not | Reduce microbatch or sequence length, enable checkpointing, use memory-efficient kernels or increase activation sharding. |
| Memory fits but latency is unacceptable | Evaluate tensor/pipeline degree, interconnect, batch policy and kernel support; capacity alone is not a performance recommendation. |
The dependable workflow is therefore: record the architecture and workload, calculate weights, calculate KV cache or training state with the correct formula, add runtime categories, account for sharding, and profile the exact target configuration. That sequence produces an estimate that can guide hardware selection without pretending that nominal parameter bytes are an on-device guarantee.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →
