What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
There is no universal “AI server” specification. Size an inference system from the model, precision, prompt and output lengths, concurrency, request rate, latency target, serving engine, and cache policy. Then validate the proposed GPU, CPU, memory, and storage with representative traffic. Model parameter count alone is not enough.
Start with a workload specification, not a hardware list
Two deployments serving the same model can need very different hardware. An interactive assistant with short prompts and strict time-to-first-token (TTFT) limits behaves differently from an offline summarization queue that maximizes tokens per second. A multi-user long-context service can be limited by key-value (KV) cache capacity even when the model weights fit comfortably in GPU memory.
As an Amazon Associate I earn from qualifying purchases.
Write down the following before comparing servers. NVIDIA reference architecture guidance and AWS inference guidance both identify these variables as primary compute drivers.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →| Input | What to record | Why it changes sizing |
|---|---|---|
| Model | Name, architecture, parameter count, number of layers, and active versus total parameters for mixture-of-experts models | Determines weight memory, compute pattern, and communication needs |
| Precision | FP32, FP16/BF16, INT8, INT4, or the exact quantization format | Changes weight memory, arithmetic throughput, and sometimes output quality |
| Request shape | Prompt-token and output-token distributions, not just averages | Long prompts consume prefill compute and KV cache; long generations consume decode time and cache |
| Sequence limits | Maximum and typical context or sequence length | Sets the upper bound for cache allocation and affects latency |
| Traffic | Requests per second, burst pattern, concurrent requests, and peak period | Determines batching, queueing, and the amount of active cache |
| Service objective | TTFT, inter-token latency (ITL), end-to-end latency, throughput, or a stated combination | A configuration optimized for throughput may violate an interactive latency target |
| Mode | Interactive, batch, or mixed | Controls how aggressively you can batch and tolerate queueing |
| Operations | Model versions, cache policy, autoscaling trigger, network mode, storage retention, and co-located services | Influences capacity, startup time, and reproducibility of benchmarks |
Use percentile targets (for example, p95 or p99 latency) rather than an average alone. Averages can hide queueing during bursts, which is often when users notice an undersized system.
#1 Best Overall
- This product has been replaced by our latest generation. Please search for the SANDISK Optimus GX 7100 NVMe SSD
- HIGH-OCTANE GAMING. Experience speeds up to 7,250MB/s read and 6,900MB/s write (1-2TB models), with up to 35% faster performance than previous generation.
- PURPOSE-BUILT. Designed for serious on-the-go gamers, with a PCIe Gen4 interface and SANDISK’s next generation TLC 3D NAND.
- MORE TIME TO CLEAR THAT CHECKPOINT. Built with laptops and handheld gaming devices in mind, with up to 100% more power efficiency over the previous generation.
- DO MORE WITH DASHBOARD. Ensure your drive is optimized for prime performance with the downloadable WD_BLACK Dashboard (Windows only).
Size GPU memory and compute in two passes
Pass 1: account for weights, runtime overhead, and KV cache
GPU memory must hold the model weights and the serving runtime’s allocations. For autoregressive language models, active requests also need KV cache. Cache demand rises with the number of active sequences and their token lengths, and the exact amount depends on model architecture, precision, paged-cache implementation, and cache settings.
A useful first calculation for weights is:
weight memory ≈ parameter count × bytes per parameter
That is only a starting point. A 7-billion-parameter model represented in 16-bit arithmetic has roughly 14 GB of raw weights before metadata, workspace, allocator fragmentation, and cache. Quantization can reduce weight storage, but the runtime may retain some layers or temporary tensors at higher precision.
KV cache is workload-dependent. NVIDIA’s inference-optimization example reports approximately 2 GB of KV cache for Llama 2 7B in 16-bit precision at batch size 1 under that article’s stated sequence assumptions. Treat that as a conditional illustration, not a typical value for every prompt length or engine. Longer contexts and more concurrent sequences can consume many times more cache.
Use published capacity examples only as starting points
A 2026 NVIDIA Technical Blog article discusses around 24 GB of GPU memory for a smaller 7–8B model and 48 GB for a 13B model with room for KV cache at the prompt lengths and multi-user assumptions described there. Those figures are conditional recommendations, not universal minimums. A different context limit, quantization scheme, batching policy, or runtime can move the requirement substantially.
| Published example | What it represents | How to use it |
|---|---|---|
| Approximately 2 GB KV cache | NVIDIA Developer Blog, 2023; Llama 2 7B, 16-bit, batch size 1, and the article’s sequence assumptions | Illustrates that cache is a material allocation; do not generalize it to another workload |
| Around 24 GB GPU memory | NVIDIA Technical Blog, 2026; smaller 7–8B model under discussed prompt and multi-user conditions | Initial candidate capacity, subject to measurement |
| 48 GB GPU memory | NVIDIA Technical Blog, 2026; 13B model with cache room under discussed conditions | Initial candidate capacity, subject to measurement |
Pass 2: benchmark headroom at the target operating point
Leave measurable headroom instead of filling the card to its nominal capacity. Run the exact model artifact and serving engine with representative prompt and output distributions. Increase concurrency until the latency objective or error budget is breached, while recording allocated memory, cache usage, queue time, and tokens per second.
Rank #2
- Ideal for high speed, low power storage
- Gen 4x4 NVMe PCle performance
- Up to 6,000MB/s read, 4,000MB/s write
- Includes Acronis cloning software
- 5-year limited warranty
If weights fit but cache does not, possible remedies include a larger-memory GPU, a lower-memory precision format that meets your quality requirement, a lower maximum context, fewer simultaneous sequences, more conservative batching, or tensor parallelism. Tensor parallelism introduces inter-GPU communication, so verify that the interconnect and topology sustain the required latency.
Do not treat system RAM or disk as a transparent substitute for GPU memory. Offload can be a deliberate design choice, but it changes latency and throughput and must be measured with the selected runtime.
Size CPU for the complete serving path
The GPU performs model kernels, but the host still handles network I/O, authentication and request handling, tokenization, input preparation, scheduling, batching, response streaming, logging, and GPU-worker coordination. CPU starvation can therefore reduce GPU utilization and increase tail latency.
vLLM-specific process guidance
vLLM documents at least one API server, one engine core, and one GPU worker per GPU: a minimum of 2 + N processes for N GPUs in the described deployment. Its documentation warns that having fewer physical CPU cores than these processes can create contention and degrade throughput and latency. This is a vLLM deployment floor, not a universal cores-per-GPU rule for every serving engine.
Reserve additional cores for the operating system, network stack, telemetry, tokenization parallelism, and any sidecars or co-located services. Pinning processes and aligning them with NUMA nodes can help on multi-socket hosts, but the correct placement is workload- and engine-specific.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteMeasure CPU pressure instead of counting GPUs
During a load test, record per-core utilization, run queue length, context switches, tokenizer and scheduler time, and GPU utilization. A pattern of low GPU utilization with saturated host cores indicates a CPU or input pipeline bottleneck. High GPU utilization with growing request queues usually points to insufficient GPU capacity, an overly aggressive batch policy, or a latency target that the selected card cannot meet.
Rank #3
- PCIe 4.0 Performance: Delivers up to 7,100 MB/s read and 6,000 MB/s write speeds for quicker game load times, bootups, and smooth multitasking
- Spacious 1TB SSD: Provides space for AAA games, apps, and media with standard Gen4 NVMe performance for casual gamers and home users
- Broad Compatibility: Works seamlessly with laptops, desktops, and select gaming consoles including ROG Ally X, Lenovo Legion Go, and AYANEO Kun. Also backward compatible with PCIe Gen3 systems for flexible upgrades
- Better Productivity: Up to 2x faster than previous Gen3 generation. Improve performance for real world tasks like booting Windows, starting applications like Adobe Photoshop and Illustrator, and working in applications like Microsoft Excel and PowerPoint
- Trusted Micron Quality: Built with advanced G8 NAND and thermal control for reliable Gen4 performance trusted by gamers and home users
Plan host memory around the platform and workload
System memory holds the operating system, model-loading buffers, runtime state, CPU-side caches, tokenizer data, telemetry, and any offloaded tensors. Capacity and bandwidth matter; so do socket locality and the number of populated memory channels.
NVIDIA’s Certified Systems Configuration Guide specifies system memory of at least twice the total GPU memory for the inference-server configurations it covers, distributed across CPU sockets and memory channels. This is a platform configuration recommendation, not a law for every deployment. A host with CPU-side caching, multiple model replicas, offload, or substantial co-located services may require more, while a tightly dedicated appliance may have different needs.
Check memory during the worst transition, not only steady state. Loading a new model revision while the old one remains live can temporarily require memory for both versions plus staging buffers. Rolling upgrades, replica warm-up, and concurrent downloads should be included in capacity planning.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Size storage for artifacts, cache, and recovery time
Local storage contains model artifacts, tokenizer files, container layers, compilation caches, logs, and retained versions. It also affects how quickly a new replica becomes ready after a failure or autoscaling event.
For the covered configurations, NVIDIA’s guide states a 1 TB minimum local-storage capacity and recommends one local NVMe drive per CPU socket. Apply those figures only where the server configuration matches the guide. Verify the drive interface, form factor, endurance rating, thermal limits, RAID or redundancy design, and server compatibility before purchasing.
Capacity planning should include the largest model revision, at least one rollback version, container and driver layers, cache growth, logs, and free space for updates. Measure four timings:
Rank #4
- HUGE SPEED BOOST: Get random read/write speeds that are 40%/55% faster than 980 PRO; Experience up to 1400K/1550K IOPS, while sequential read/write speeds up to 7,450/6,900 MB/s reach near the max performance of PCIe 4.0*
- BREAKTHROUGH POWER EFFICIENCY: Use less power and get more performance; Enjoy up to 50% improved performance per watt over 980 PRO, plus optimal power efficiency with max PCIe 4.0 performance**
- SMART THERMAL CONTROL: Samsung's own nickel-coated controller delivers effective thermal control; With its slim size, 990 PRO is a perfect fit for desktops and laptops that meet the PCI-SIG D8 standard***
- THE CHAMPION MAKER: Up to 65% improvement in random performance enables faster loads for an ultimate gaming experience on PS5 and DirectStorage PC games****
- SAMSUNG MAGICIAN SOFTWARE: Get the most out of your SSD with Samsung Magician's advanced yet intuitive optimization tools; Monitor drive health, protect valuable data, and receive important updates for your 990 PRO
- Download or transfer time for a cold model artifact.
- Cache-hit time when the artifact is already local.
- Disk-to-GPU transfer time during model loading.
- Time from process start to readiness under a realistic restart or scale-out.
Fast storage cannot make weights or active KV cache fit in GPU memory. Its role is reducing load and recovery time and preventing artifact or cache I/O from delaying readiness.
Recommended Free Tools
Build a representative benchmark before buying
A specification is a hypothesis. Validate it with the same model revision, backend and version, hardware, network mode, cache state, request distribution, and concurrency on every candidate. NVIDIA notes that comparisons omitting details such as backend version, model revision, network mode, or cache state are insufficient for an architectural conclusion.
- Freeze the software. Record driver, CUDA or equivalent runtime, serving-engine version, container image, tokenizer, model revision, quantization settings, and operating-system image.
- Prepare traffic data. Use a prompt and output-token distribution that reflects production, including long-tail requests and burst behavior. Separate interactive, batch, and mixed scenarios.
- Warm and cold runs. Measure both a warmed cache and a restart or scale-out path. Note whether compilation, weight loading, and tokenizer caches are already complete.
- Sweep concurrency. Test from one request through the planned peak, including burst levels. For each level, allow the system to reach steady state before collecting results.
- Capture resource telemetry. Collect GPU memory and utilization, KV-cache occupancy, CPU per-core usage, host-memory bandwidth, disk throughput, network traffic, queue depth, and errors.
- Apply the service objective. Select the highest concurrency that meets the required latency percentiles and error rate, not the run with the highest isolated throughput.
- Repeat for failover and scaling. Measure model-load time, readiness time, and behavior while adding or removing replicas.
| Metric | What it tells you | Use in a decision |
|---|---|---|
| TTFT | Prefill, queueing, and admission delay before the first token | Critical for interactive responsiveness |
| ITL | Time between generated tokens | Shows decode smoothness and streaming quality |
| End-to-end latency | Total request completion time | Confirms the user-visible target |
| Output throughput | Generated tokens per second at a stated concurrency | Primary metric for batch economics and capacity |
| Maximum compliant concurrency | Highest load that remains within latency and error limits | More useful than peak throughput alone |
| Error and timeout rate | Failures under load, including out-of-memory events | Defines safe operating capacity |
| Load and readiness time | Cold-start and recovery duration | Determines autoscaling and failover impact |
Tune for interactive, batch, or mixed traffic
Interactive services
Prioritize TTFT, ITL, and tail latency. Keep queueing bounded, cap context lengths where product requirements allow, and select a concurrency point that leaves cache and CPU headroom. A larger batch can improve utilization but may delay individual requests.
Batch inference
Throughput and utilization can take precedence over per-request latency. Larger batches often improve GPU efficiency, but they increase memory pressure and can make a single oversized request delay the batch. Use a batch-size sweep and enforce a maximum sequence length.
Mixed workloads
Separate interactive and batch capacity when possible. If they share a pool, use admission control, priority queues, per-class concurrency limits, and separate latency objectives. Benchmark the worst mixture, not each class in isolation.
Compare candidate systems on useful capacity
When evaluating two or more systems, hold the workload and software constant and compare the capacity that meets your objective.
Best Value
- SPEED UP PROJECTS. Launch creator applications fast with uncompromising PCIe 4.0 read speeds up to 7,100MB/s,[2] (1TB and 2TB[1] models) and write speeds up to 6,700MB/s[2] (1TB[1]-4TB[1] models).
- CREATE AND STORE MORE. Make more room for your 4K videos and high-resolution images with capacities from 500GB[1] up to 4TB[1] on M.2 2280 built with our trusted 8th generation SANDISK BiCS QLC 3D CBA NAND.
- IT GOES WHERE YOU GO. With an all-new power efficient design, your drive delivers high performance with low power, giving you more time to be productive while on the go.
- UNCOMPROMISED RELIABILITY. With up to 1,200 TBW[3] (4TB[1] model) endurance rating, your drive is designed for creators.
- KEEP YOUR DRIVE UPDATED. Monitor your SSD’s performance and check for updates with the downloadable SANDISK Dashboard application.[5]
| Dimension | Questions to answer |
|---|---|
| GPU | Does measured memory cover weights, runtime allocations, and peak KV cache? What is the maximum compliant concurrency? |
| CPU | Are API, tokenizer, scheduler, and worker processes receiving dedicated physical cores without NUMA contention? |
| Host memory | Is there enough capacity and bandwidth for loading, runtime state, offload, and co-located services? |
| Storage | How long do cold load, cache-hit load, disk-to-GPU transfer, and readiness take? Is there room for rollback versions and logs? |
| Interconnect | For multi-GPU or multi-node serving, does the topology support the required tensor-parallel communication? |
| Service result | At the target latency percentile, what throughput, compliant concurrency, and error rate are achieved? |
| Economics | If reliable prices are available, what is the cost per useful output at the compliant operating point? |
Do not compare a peak tokens-per-second figure from one system with a latency-compliant figure from another. Keep cache state, prompt distribution, output distribution, and software versions visible in the benchmark record.
Troubleshoot common sizing failures
Out-of-memory errors during long prompts
Check KV-cache occupancy and the actual sequence-length distribution first. Reduce maximum context or concurrency, adjust cache allocation, use a suitable quantization format, or move to a GPU with more memory. Confirm that the remedy still meets quality and latency requirements.
GPU utilization is low while latency rises
Inspect CPU cores, tokenizer time, scheduler queues, network handling, and storage stalls. In vLLM, ensure the documented process floor is not competing for too few physical cores. A saturated host can starve the GPU.
Throughput improves but users time out
The batch or concurrency point is beyond the interactive latency objective. Reduce batch size or admission concurrency, reserve capacity for bursts, and evaluate TTFT and tail latency rather than throughput alone.
Cold starts are too slow
Measure artifact transfer, cache-hit behavior, disk-to-GPU transfer, compilation, and readiness separately. Keep required artifacts local, use storage with sufficient sustained performance and endurance, and size replica pools or autoscaling delays from measured readiness time.
Results cannot be reproduced
Record model revision, backend version, driver and runtime versions, network mode, cache state, prompt and output distributions, concurrency, and hardware topology. Without those controls, a numerical difference may reflect the test environment rather than the system architecture.
Quick Recap
A practical sizing decision sequence
- Define production request shapes, concurrency, rate, mode, and latency or throughput objectives.
- Estimate raw weight memory from parameter count and precision, then add runtime overhead and workload-dependent KV cache.
- Select candidate GPU memory capacities and multi-GPU topologies with measurable headroom.
- Allocate CPU cores for the serving path, using engine-specific documentation such as vLLM’s
2 + Nprocess floor as a constraint rather than a universal rule. - Plan host memory for loading, runtime state, caches, offload, and co-located services; treat the two-to-one NVIDIA configuration guidance as a platform-specific reference.
- Plan local storage for artifacts, versions, caches, logs, and recovery, using the covered NVIDIA configuration’s 1 TB minimum and one-NVMe-per-socket recommendation only when applicable.
- Benchmark cold and warm behavior under representative traffic, sweeping concurrency and batch settings.
- Choose the configuration that meets latency, throughput, error, readiness, and operational objectives simultaneously.
- Document the assumptions and repeat the test whenever the model, quantization, serving engine, context limit, or traffic mix changes.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitches
