How to Size GPU Memory and Interconnects for Private AI Inference

Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.

Short answer: size a private inference server from the workload, not the model’s parameter count alone. Budget GPU memory for weights, activation and runtime buffers, and the KV cache; then reserve headroom for fragmentation and non-model processes. A 70B model in FP16 illustrates the problem: its weights alone are about 140 GB, while one 128k-token Llama 3 70B context can consume roughly 40 GB of KV-cache memory. You may therefore need several GPUs, quantization, a larger-memory accelerator, or a CPU-memory tier. Choose NVLink for tightly synchronized tensor-parallel work when the topology is balanced; use PCIe, pipeline parallelism, or multiple nodes when communication is less coupled or NVLink is unavailable. The final design must be validated in the target runtime with representative prompts, concurrency and latency tests.

What must you size for private inference?

Start by writing a workload envelope before choosing a GPU. Record the exact model revision, parameter count, weight format, maximum prompt-plus-output length, number of simultaneous sequences, target decode throughput, first-token latency and sustained latency objective. “Runs the model” is not a sufficient requirement: a configuration that loads one request may fail when ten users stream long answers.

#	Preview	Product	Price
1		NVIDIA Tesla A100 Ampere 40 GB Graphics Processor Accelerator - PCIe 4.0 x16 - Dual Slot	$5,669.99	Buy on Amazon
2		QTHREE GeForce GT 730 4GB Graphics Card,2X HDMI, DP,VGA,DDR3,64 Bit,Low Profile Video Card for...	$89.99	Buy on Amazon
3		NVIDIA GeForce RTX 4060Ti Founders Edition	$899.00	Buy on Amazon
4		RGEEK PCIe 5.0 Riser Cable X16, 200mm Right Angle for Vertical GPU Mount | Compatible PCIe 4.0, EMI...	$37.99	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

• Model: parameter count, architecture, context limit and any mixture-of-experts behavior.
• Precision: FP16, BF16, FP8, INT8 or another quantized format, including any scales or metadata stored with the weights.
• Request shape: maximum input plus generated tokens, typical length distribution and whether requests arrive in bursts.
• Concurrency: peak active sequences, queueing limit and batch or continuous-batching policy.
• Service objectives: time to first token, decode tokens per second, tail latency and availability target.
• Deployment boundary: whether CPU RAM, NVMe or another memory tier may hold model pages or KV-cache data.

These inputs determine both capacity and communication. A model may fit in aggregate memory yet miss its latency target because GPUs exchange tensors too slowly, or fit comfortably but deliver poor concurrency because the KV cache consumes the remaining HBM.

How is GPU memory divided at inference time?

Inference memory has three principal categories: model weights, internal activation tensors and I/O tensors. The KV cache is normally the largest I/O allocation after the weights for long-context, multi-user serving. TensorRT-LLM describes these same three contributors when sizing a TensorRT engine.

#1 Best Overall

NVIDIA Tesla A100 Ampere 40 GB Graphics Processor Accelerator - PCIe 4.0 x16 - Dual Slot

• Discrete graphics card memory 40 GB
• Memory bandwidth (max) 1555 GB/s
• Graphics processor family NVIDIA
• Graphics processor A100

$5,669.99

Buy on Amazon

Model weights

A first estimate is parameter count multiplied by the bytes used by the chosen weight format. As a simple arithmetic guide, 70 billion parameters at two bytes each is about 140 GB before scales, metadata, padding and runtime allocations. NVIDIA cites approximately 140 GB for Llama 3 70B in FP16. INT8 or FP8 can substantially reduce this category, but the real footprint depends on the quantizer, per-channel or per-group scales, alignment and whether some layers remain at higher precision. Treat the arithmetic as a lower-bound estimate, not a capacity guarantee.

Activations and framework buffers

Temporary activations are created during prompt processing and decoding. Their peak depends on batch shape, hidden dimensions, kernels and the runtime’s workspace choices. CUDA graphs, attention workspaces, communication buffers and allocator fragmentation also consume memory. Reserve this space before assigning the remainder to the KV cache; a model that uses every apparent free megabyte will be fragile under load.

KV cache and other I/O tensors

For each active sequence, the runtime stores key and value tensors for tokens already processed. Demand rises with context length and the number of active requests. A 128k-token single-user KV cache for Llama 3 70B is about 40 GB in NVIDIA’s example, so a long context can rival a large fraction of a GPU even when only one user is connected. Lower-precision KV storage can increase token capacity where the model and backend support it, but validate output quality and kernel support.

Operational headroom

Leave room for memory fragmentation, telemetry, tokenization services, health checks and other processes sharing the device. A practical allocation is the measured model and runtime footprint plus a deliberate safety margin, not a plan to consume 100% of HBM. If the server will be reconfigured frequently, leave more margin than a fixed appliance.

How do I calculate a first-pass memory budget?

Use a worksheet that separates known quantities from measurements. It will not replace a runtime test, but it quickly rejects undersized designs.

1. Estimate weights: multiply parameters by bytes per stored weight, then add quantization scales, metadata and alignment overhead.
2. Reserve activations and runtime buffers: use a measurement from the selected backend if available; otherwise hold back a conservative portion of each GPU and refine it during a dry run.
3. Estimate KV demand: evaluate the maximum prompt-plus-output tokens multiplied by peak active sequences. The per-token cost is model- and precision-specific, so obtain it from the runtime’s profiler or startup report rather than assuming a universal number.
4. Apply parallelism: divide weight and, where supported, KV allocations across tensor-parallel ranks. Include communication buffers on every rank.
5. Check headroom: subtract the reserved margin and ensure the result remains positive during the longest expected request.
6. Test the envelope: run representative prompts at the planned concurrency and record out-of-memory events, KV capacity, first-token latency and decode throughput.

For an initial spreadsheet, use separate rows for weights, activations, KV cache, communication buffers, framework overhead and headroom. Do not hide all overhead in one optimistic percentage: identifying which row grows when context or concurrency changes makes capacity decisions explainable.

How do context length and concurrency change capacity?

Context and concurrency are multiplicative pressure on the KV cache. Doubling the maximum tokens per request roughly doubles the cache reserved for each active sequence; doubling active sequences roughly doubles total cache demand. The exact slope depends on the architecture and KV dtype, but the direction is consistent.

Set limits from observed traffic

A model card may advertise a very large context window that your users rarely need. If the scheduler reserves for that maximum, it can reduce the number of concurrent requests even when typical prompts are short. Set the runtime’s maximum model length and sequence limit from measured traffic, then provide a separate policy for exceptional long-context jobs.

Use a capacity gate in the runtime

vLLM exposes --max-model-len, --max-num-seqs, --gpu-memory-utilization, --kv-cache-memory-bytes and --kv-cache-dtype. Its documented default for --gpu-memory-utilization is 0.92. At startup, vLLM reports total GPU KV-cache tokens and an estimated maximum concurrency for a selected request length. Compare those figures with the production envelope before accepting the configuration. If the report is below your required concurrency, reduce the context limit, use a smaller KV dtype where supported, add memory, or lower the concurrency target.

Rank #2

QTHREE GeForce GT 730 4GB Graphics Card,2X HDMI, DP,VGA,DDR3,64 Bit,Low Profile Video Card for PC,Computer GPU,PCI Express X8,SFF,DirectX 12,Support Winows 11

• NVIDIA GT 730 graphics cards offer basic display capabilities for office work and light multimedia,which with 1000 MHz Memory Clock 4GB DDR3 on Kepler architecture, support multiple monitors and HD video playback,easily upgrading for convenient usage to save your budget for your old pc
• The low-profile design of the PC graphics card saves installation space, easy to install,plug &play,making it easy to build a compact computer system, even compatible with ITX chassis.
• The 4x outputs enables multi-monitor productivity on up to 4 monitors simultaneously,including 2x HDMI,VGA,DP.Designed for full-size chassis and small case installations.
• PCI Express based PC is required with one X8 lane graphics slot available on the motherboard. 300 Watt or greater power supply. This video card can automatically install new drivers and support Win11,DirectX 12.
• 30W low power,no external power supply and the all-solid-state capacitor keeps low power consumption and high performance.If you have any problems about this card,please contact us via amazon messages.

$89.99

Buy on Amazon

Understand scheduler trade-offs

Continuous batching improves aggregate utilization but allows more active sequences to compete for KV space. A strict maximum-sequence setting protects latency and memory predictability; an unbounded queue merely moves the failure to admission control. Define what happens when the cache is full: reject, queue, truncate, or route the request to another worker.

How many GPUs are required for a 70B model?

There is no single answer because precision, context, concurrency and GPU capacity differ. NVIDIA’s reference point is 140 GB for Llama 3 70B weights in FP16, before activations and KV cache. A 70B deployment therefore needs more than two 80 GB cards if it must also hold substantial KV cache and runtime overhead, while a quantized build may fit on fewer or smaller cards if quality and tooling are acceptable.

For example, four 80 GB GPUs provide 320 GB of aggregate HBM, but the usable amount is less after runtime reservations. Whether that is comfortable for FP16 70B depends on the KV-cache target and implementation. Conversely, two 94 GB GPUs provide 188 GB, which may load the weights only with little room for long-context serving. Count memory and communication together rather than selecting a GPU count from weight arithmetic alone.

When should I use quantization?

Quantization is the most direct way to reduce weight memory, and sometimes KV-cache memory, without changing the model’s parameter count. INT8 and FP8 are explicitly identified by TensorRT-LLM as lower-precision choices that reduce weight memory. The decision is a quality-and-operations trade-off, not merely a capacity trick.

• Use it when: the FP16 or BF16 model exceeds available HBM, the workload is memory-bandwidth bound, or you need more concurrent sequences.
• Check accuracy: evaluate the prompts and tools that matter to your application, including structured output, code and retrieval-augmented answers.
• Check kernel support: ensure the chosen runtime, GPU generation and quantization format support efficient execution rather than silently dequantizing.
• Check mixed precision: embedding, normalization or output layers may remain at higher precision, so the reduction will not equal a simple parameter-times-byte calculation.
• Check KV dtype separately: weight quantization does not automatically mean the KV cache is quantized.

Keep an unquantized fallback for comparison during acceptance testing. A smaller model that meets quality and latency goals can be preferable to an aggressively quantized larger model that requires expensive troubleshooting.

What is the difference between tensor, pipeline and data parallelism?

Parallelism determines both how the model fits and how much traffic crosses GPU links. Choose it after measuring topology, not just the number of visible GPUs.

Tensor parallelism

Tensor parallelism splits layer operations or matrix dimensions across GPUs. Each rank holds a fraction of the weights, but ranks synchronize frequently during prompt processing and decoding. It benefits most from high-bandwidth, low-latency peer links and a balanced topology. vLLM recommends setting tensor_parallel_size to the number of GPUs used per node.

Pipeline parallelism

Pipeline parallelism assigns groups of layers to stages. Communication occurs between stages rather than through an all-reduce for every tensor operation, so it can be a better fit when GPUs lack NVLink or are distributed across nodes. vLLM recommends setting pipeline_parallel_size to the number of nodes in a multi-node deployment. Pipeline stages can introduce bubbles and scheduling complexity, but they also support uneven GPU splits when a model fits in a node and the GPU count does not divide evenly.

Data parallelism

Data-parallel replicas keep a complete model copy per worker and route requests among them. This increases aggregate throughput and isolation but does not make a model fit on a single GPU. It is useful after one replica meets memory and latency requirements.

Rank #3

NVIDIA GeForce RTX 4060Ti Founders Edition

• NVIDIA GeForce RTX 4060Ti Founders Edition
• VIDEO_CARD
• generisch

$899.00

Buy on Amazon

Combining schemes

A common multi-node layout uses tensor parallelism within each node and pipeline parallelism across nodes. Ensure that the product of the two sizes matches the intended GPU count and that every rank is placed where the runtime expects it. Validate startup logs rather than assuming automatic placement is optimal.

Do I need NVLink, or is PCIe enough?

NVLink is valuable when tensor-parallel ranks exchange large synchronized tensors frequently. PCIe can be sufficient for loosely coupled work, pipeline stages, data-parallel replicas or systems where NVLink is unavailable, but the answer depends on measured communication and topology.

Reference bandwidths

These are product reference points, not universal requirements. NVIDIA’s inference-sizing guidance prefers NVLink-enabled systems for models larger than 13B that require more than one GPU. Treat that as a design preference for communication-heavy inference, then confirm with your actual engine and topology.

Topology matters more than the label

Inspect the PCIe root complexes, NUMA domains, NVLink or NVSwitch fabric and NIC path. Two cards advertised with the same link speed can perform differently if traffic crosses a CPU socket or an oversubscribed switch. Pin worker processes and network interfaces to the matching NUMA node where the platform and runtime allow it. A nominal bandwidth number cannot compensate for poor placement.

When PCIe is a reasonable choice

Choose PCIe when the model is replicated, pipeline stages communicate relatively infrequently, GPUs are separated by nodes, or the workload’s latency target tolerates transfer time. Benchmark prompt ingestion and token decode separately: PCIe may look adequate for throughput while missing a strict first-token objective.

Can one model be split across several GPUs?

Yes. Tensor parallelism and pipeline parallelism are the usual methods. The split must leave every rank with enough memory for its weight shard, activations, communication buffers and assigned KV data. A model can therefore fail even when aggregate memory appears sufficient if one GPU has less capacity or a rank receives an uneven layer assignment.

1. Inventory GPU memory, compute capability, peer links and node placement.
2. Choose a tensor-parallel group that stays within a fast, balanced topology where possible.
3. If the group crosses slow links, test pipeline parallelism or a different node layout.
4. Configure the runtime’s tensor and pipeline sizes explicitly and confirm rank-to-GPU mapping in logs.
5. Exercise the longest prompt and peak sequence count; watch the per-GPU allocator, not just total memory.

Do not mix unlike GPUs in one tightly synchronized tensor-parallel group unless the runtime and scheduler support it predictably. The slowest rank can determine the step time, and the smallest HBM capacity can determine whether the model loads.

What does CPU or NVMe offload change?

Offload can make an oversized model start, but it changes latency and failure modes. NVIDIA describes a GH200 configuration with 96 GB of GPU memory and 480 GB of CPU memory connected by 900 GB/s NVLink-C2C as an example of memory-tiering approaches when a model exceeds local GPU capacity. That is not equivalent to having all data in HBM: transfers, page placement and access patterns still matter.

• CPU RAM: useful for weights or cache portions that are accessed predictably, with lower bandwidth and higher latency than HBM.
• NVMe: provides capacity for loading or cold pages, but paging during decode can make token latency highly variable.
• Tiering policy: keep hot weights and active KV blocks on the fastest tier; measure eviction and prefetch behavior under concurrency.
• Operational impact: budget CPU memory bandwidth, PCIe lanes, NUMA placement, storage endurance and recovery time.

Use offload when the latency objective allows it or when it avoids buying a much larger accelerator. For interactive chat with a tight first-token and decode SLO, an all-HBM design is usually easier to make predictable.

Rank #4

RGEEK PCIe 5.0 Riser Cable X16, 200mm Right Angle for Vertical GPU Mount | Compatible PCIe 4.0, EMI Shielded, Fits ATX MATX Case for Professional Graphics Cards RTX 5080 5090 RX 9070 A100 A800

• Broad GPU & PCIe Compatibility：Supports NVIDIA, AMD, Intel gaming and professional GPUs, including RTX 5090/5080, RX 9070, and A100/A800/H100/H800. Backward compatible with PCIe Gen5/4/3. Perfect for AI deep learning, 3D rendering, and design workstations.
• Tested 128GB/s PCIe 5.0 Performance：Validated by 3DMark PCI Express bandwidth tests, this riser supports full 128GB/s bidirectional speed. It maintains stable, near-zero-loss signal transmission even under heavy GPU loads.
• 90-Degree Design for Vertical GPU Mounting：This right-angle PCIe riser cable is ideal for vertical GPU installation in ATX and MATX cases where the motherboard and GPU are on the same side. Its flexible design allows for easy shaping and bending without compromising signal transmission or stability, helping you save space and build a clean, high-airflow PC.
• Superior PCIe Performance & Multi-Layer Shielding：Equipped with high-quality PCIe slots for good electrical performance, paired with a sturdy PC+ABS housing and high-strength metal armor for durability. It features a reliable SMT socket design to ensure strong retention force and stable connections. Built on a 6-layer M7-grade low-loss PCB with 5U gold-plated fingers for good conductivity. Integrated aluminum foil shielding helps reduce electromagnetic interference (EMI), supporting stable high-speed signal transmission under normal and heavy loads.
• Package Include: PCIe 5.0 Riser Cable (20cm cable length, 24cm total length including connectors)

$37.99

Buy on Amazon

How do I validate a proposed server before deployment?

Run a staged acceptance test that mirrors production rather than relying on a successful model load.

1. Topology check: record nvidia-smi topo -m output, GPU-to-GPU paths, NUMA locality and NIC placement. Confirm that the intended ranks use the expected links.
2. Single-request baseline: measure load time, first-token latency and decode rate with a short prompt and then the maximum prompt-plus-output length.
3. Concurrency ramp: increase active sequences until the required peak. Record KV-cache tokens, queue time, first-token latency, decode throughput and out-of-memory events.
4. Traffic mix: combine short and long prompts, streaming and non-streaming responses, tool calls and retrieval context if those occur in production.
5. Fault and recovery: restart a worker, fill the cache, reject an over-limit request and verify that health checks remove and restore capacity correctly.
6. Soak test: run long enough to expose fragmentation, thermal throttling, power limits and memory leaks.

For vLLM, treat the startup report of total KV-cache tokens and estimated maximum concurrency as an admission-control input, then confirm it with the load test. For TensorRT-LLM, verify the configured paged KV-cache pool and the memory left after weights and buffers. TensorRT-LLM documents a default paged KV-cache pool of 90% of the remaining free memory when no explicit limit is supplied; an explicit limit can make capacity more predictable.

Which hardware and platform checks are easy to miss?

• Chassis and power: H100 PCIe or NVL cards require a server with the correct slots, auxiliary power, airflow and sustained power budget; a desktop chassis is not an equivalent platform.
• Cooling: verify inlet temperature, fan curves and whether neighboring slots reduce airflow. Thermal throttling can look like an interconnect or kernel problem.
• HBM capacity versus bandwidth: enough memory does not guarantee decode speed; memory bandwidth and kernel efficiency often dominate token generation.
• CPU and PCIe layout: place GPUs and NICs under the intended root complex and NUMA node.
• Network fabric: multi-node pipeline or tensor groups need NIC bandwidth, switch capacity and a supported collective-communication path.
• Support and warranty: include firmware, driver, CUDA and runtime version compatibility, replacement time and vendor support in the design cost.
• Availability: decide whether a failed GPU takes down one replica, one pipeline or the entire service, then size spare capacity accordingly.

A configuration aid such as Lenovo’s LLM sizing tool can help model workload settings and options such as NVLink, retrieval-augmented generation and high availability, but its result still needs validation on the exact server, runtime and traffic pattern you will operate.

What is a practical decision process?

Use this sequence to turn a model request into an infrastructure choice:

1. Define the envelope: model, precision, maximum tokens, peak sequences, target tokens per second and latency SLOs.
2. Choose an accuracy floor: decide whether FP16 or BF16 is required, or whether INT8/FP8 quantization is acceptable after evaluation.
3. Calculate weights: include scales, metadata and unquantized layers.
4. Reserve non-KV memory: account for activations, workspaces, communication buffers, framework overhead and headroom.
5. Size KV capacity: use the runtime’s measured per-token allocation and set context and concurrency limits from traffic.
6. Select a topology: use NVLink for communication-heavy tensor parallelism when the fabric is balanced; otherwise test pipeline, PCIe or multi-node designs.
7. Check tiers: decide whether CPU or NVMe offload meets latency and reliability requirements.
8. Validate: run representative prompts, concurrency ramps and a soak test, then adjust limits before production.

Compare candidate designs on total and per-GPU memory, supported weight and KV-cache dtypes, usable context, concurrent sequences, first-token latency, decode throughput, GPU-to-GPU bandwidth, topology, offload behavior, power and cooling, and acquisition and support cost. A cheaper server that misses the latency SLO is not a lower-cost design once capacity and user experience are included.

Frequently Asked Questions

How much GPU memory does a 13B model need for private serving?

Parameter count alone cannot produce a safe number. Multiply the model’s stored precision by its parameters, then add activations, runtime buffers, KV cache for your maximum active tokens and operational headroom; a short-context single-user test can fit where a multi-user service cannot.

Can I increase context length after deployment without changing GPUs?

Only if the existing KV-cache pool has unused capacity at the desired concurrency. Raising the limit can lower admission capacity or trigger out-of-memory failures, so change the runtime limit together with load tests and an explicit request policy.

Why is first-token latency high when decode speed looks good?

Prompt processing may be saturating compute, memory bandwidth or interconnect collectives before generation begins. Measure prompt and decode phases separately, then inspect tensor-parallel communication, NUMA placement and long retrieval contexts.

Should I buy one large-memory GPU or several smaller GPUs?

One large-memory device simplifies placement and avoids inter-GPU synchronization, while several smaller GPUs can provide more aggregate throughput or redundancy. Compare per-rank fit, topology, power, cooling, support and the latency impact of the chosen parallelism rather than comparing aggregate HBM only.

Quick Recap

Bestseller No. 1

NVIDIA Tesla A100 Ampere 40 GB Graphics Processor Accelerator - PCIe 4.0 x16 - Dual Slot

Discrete graphics card memory 40 GB; Memory bandwidth (max) 1555 GB/s; Graphics processor family NVIDIA

$5,669.99

Bestseller No. 2

QTHREE GeForce GT 730 4GB Graphics Card,2X HDMI, DP,VGA,DDR3,64 Bit,Low Profile Video Card for PC,Computer GPU,PCI Express X8,SFF,DirectX 12,Support Winows 11

$89.99

Bestseller No. 3

NVIDIA GeForce RTX 4060Ti Founders Edition

NVIDIA GeForce RTX 4060Ti Founders Edition; VIDEO_CARD; generisch

$899.00

Bestseller No. 4

RGEEK PCIe 5.0 Riser Cable X16, 200mm Right Angle for Vertical GPU Mount | Compatible PCIe 4.0, EMI Shielded, Fits ATX MATX Case for Professional Graphics Cards RTX 5080 5090 RX 9070 A100 A800

$37.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

AI inferenceGPU memoryKV cacheLLM deploymentNVLinkPCIepipeline parallelismtensor parallelism