Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
LLM token speed on an RTX 3090 depends on what you mean by speed: prompt processing (prefill), generated-token speed (decode), or total throughput across concurrent requests. There is no single batch size, quantization, engine, or clock setting that maximizes all three. For one interactive request, prioritize a model and KV cache that fit in the card’s 24 GB, then measure decode speed at batch 1. For a serving workload, tune concurrency and batching for aggregate throughput while recording latency separately.
Published RTX 3090 results illustrate why a workload-specific sweep is essential. One 7B Llama study reported 32.02 tokens/s for a 5-token prompt and 8,187 generated tokens at batch 1, but 225.31 tokens/s for a 512-token prompt and 512 generated tokens at batch 9. Another comparison measured 35.94 tokens/s with Hugging Face, 54.78 with vLLM, and 63.303 with FlashDecoding++ for Llama 2 7B at batch 1 with 128 input and 128 output tokens. Those figures belong to their exact software, model, and workload conditions—not to every RTX 3090 setup.
As an Amazon Associate I earn from qualifying purchases.
At a glance
| Name | Best for | Key spec | Platforms | Checked |
|---|---|---|---|---|
| Batch-1 decode profile | Fast response for one user | Measure generated tokens/s separately from prompt time; batch 1 | RTX 3090 (24 GB), your chosen engine and model | November 2026 |
| Concurrent-throughput profile | Maximum aggregate output | Sweep concurrency and batch; latency must be reported alongside throughput | RTX 3090 serving stack | November 2026 |
| KV-cache capacity profile | Long context or many simultaneous requests | Fit model weights, context, and KV cache in 24 GB; automatic or explicit cache sizing | vLLM on RTX 3090 | November 2026 |
| Prompt-heavy prefill profile | Large prompts and document ingestion | Benchmark prompt-processing rate independently of decode | RTX 3090 with a fixed prompt length and engine version | November 2026 |
What does “token speed” mean on an RTX 3090?
“Token speed” is three different measurements. Prefill is how quickly the engine processes the input prompt. It dominates applications that submit long documents before generating a short answer. Decode is the rate at which new output tokens are generated after the prompt has been processed; it is the figure most people feel as streaming response speed. Aggregate throughput is the total output-token rate from several requests running together.
A high aggregate number can coexist with slower time-to-first-token or slower per-user streaming. For example, increasing batch size may keep the GPU busy and raise total tokens per second while making each request wait longer. Record prompt length, output length, batch or concurrency, and both latency and throughput so a serving result is not mistaken for an interactive result. The vLLM throughput benchmarking guidance describes this kind of controlled workload comparison.
Popular on Amazon
- 【90 Degree Full Function USB C Adapter】Supports the latest USB4 standard for up to 40Gbps of bandwidth, 8K video support, and up to 240W Fast charging; Support USB PD 3.1 Extended Power Range (EPR) including 140W, 180W and 240W. Your productivity is easily doubled. *Warm Tip: For the best fit, please make sure the USB-C port is fully exposed; thick or bulky cases may affect compatibility.
- 【Reliable Quality】Made from high-quality, scratch-resistant aluminum alloy, delicate metal frame. get excellent wear resistance and heat dissipation. Non-Slip Design for easier insertion and removal.
- 【Small and portable】Ultra thin size, ultra compact design, a petite physique that matches the minimalist design of your modern laptops. Coming with a slim figure that won't block the port next to the one it's plugged into.
- 【Compatibility】Widely work with most usb c, thunderbolt 4/3 devices. Compatibilit for ROG Ally, Steam Deck, Switch, MacBook Pro, Quest 3 VR headset, Surface Book 2, iPhone 18/17/16/15, Samsung Galaxy S23/S22/S21, Note 20/10, Google Pixel, ChromeBook, etc.
- 【Excellence Customer Support】If you experience any issues or cable failure, simply reach out to our dedicated customer service team. We provide a prompt response within 18 hours, providing you with the necessary technical support to resolve any concerns.
Which settings have the largest effect?
The biggest gains usually come from matching memory, workload, and execution engine rather than from a single “turbo” flag. Tune the following variables in order, changing one category at a time.
1. Model size and quantization
Weights must fit alongside runtime allocations and the KV cache. A smaller or reduced-precision model can leave more memory for context and concurrent sequences, but quantization changes the arithmetic path and may affect output quality. The evidence does not identify one quantization format as fastest for every model. Treat model and quantization as part of the benchmark identity, not as interchangeable settings.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall2. KV-cache capacity and data type
The KV cache stores attention keys and values for active sequences. Longer contexts and more concurrent requests consume more of it. vLLM can size the cache automatically from a GPU-memory-utilization target, or you can provide an explicit cache-memory limit when other allocations need to be accounted for. An explicit capacity takes precedence over the utilization-based setting, according to the project’s optimization documentation.
Start with automatic sizing for a clean baseline. Use an explicit byte limit when the model, CUDA allocations, adapters, or another process make the available memory predictable. A reduced-precision KV-cache data type may increase capacity, but support depends on the model and hardware path; validate quality and speed for the intended workload before adopting it. If requests fail with out-of-memory errors, reduce maximum context or concurrency, select a smaller model, or allocate the cache explicitly with headroom.
Also popular on Amazon
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
3. Batch size and concurrency
Batching is a workload choice. For one user, begin at batch 1 and measure decode tokens per second and time to first token. For a server, sweep concurrent requests or batch limits and stop when latency violates the service objective, even if aggregate throughput continues to rise. Do not assume the largest batch is fastest for an individual response.
4. Engine and attention implementation
Different engines and attention kernels have different scheduling and memory behavior. Keep the engine, version, CUDA runtime, and supported attention path fixed while comparing model settings. When comparing engines, rerun the same prompt and output lengths, because a result from one implementation cannot establish a universal ranking.
Free tools Windows power users keep installed
One-click scans. No signup required.
5. CPU resources
Tokenization, request scheduling, and output processing run on the host. If GPU utilization is unexpectedly low, CPU contention may be the bottleneck, as vLLM notes in its optimization guidance. Reserve sufficient CPU cores, avoid competing jobs, and check whether tokenization or post-processing is consuming the interval in which the GPU is idle.
6. Stable clocks, temperature, and power
Use a repeatable GPU state when comparing settings. NVIDIA’s TensorRT-LLM benchmarking guide recommends persistence mode for rigorous measurements and cautions that locking clocks at maximum can trigger thermal throttling and reduce performance. Keep airflow and temperature stable, monitor clocks and power, and prefer the card’s dynamic clock behavior over assuming a maximum lock is faster.
Also popular on Amazon
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
How should you benchmark a workload-specific optimum?
A short, controlled sweep is more reliable than copying a command from an unrelated benchmark. Use the following sequence.
- Define the target. Choose interactive decode, prompt-heavy prefill, or aggregate serving throughput. Set an explicit latency limit if users are waiting on each response.
- Freeze the workload description. Record the exact model revision, quantization, engine and software versions, context limit, prompt length, output length, sampling settings, and concurrency. Keep these unchanged while testing one variable.
- Establish a baseline. Run repeated measurements with a warm GPU. Report prompt-processing time, time to first token, generated-token rate, end-to-end latency, and aggregate throughput as separate fields. NVIDIA’s benchmarking utility is intended for building and benchmarking tuned engines; its workload parameters should be matched when making comparisons.
- Verify memory before chasing speed. Confirm that weights, runtime allocations, requested context, and the KV cache fit in the RTX 3090’s available memory. Start with automatic vLLM cache sizing, then test an explicit cache limit if other consumers need reserved space.
- Sweep the variable that matches the goal. For serving throughput, test increasing concurrency or batch limits. For long-context workloads, test context and cache capacity. For execution speed, compare supported engines or attention paths with all other conditions fixed.
- Repeat at the intended operating point. Include enough requests to expose queueing and thermal changes, but discard warm-up effects consistently. Keep a run log so a later software update does not get confused with a setting change.
- Inspect bottlenecks. Correlate tokens per second with GPU utilization, CPU utilization, memory use, temperature, clocks, and power. Low GPU utilization with busy CPU cores points to host-side contention; high temperature with falling clocks points to thermal throttling.
What RTX 3090 measurements are actually published?
Published figures are useful reference points, not promises. The paper Layer-Condensed KV Cache for Efficient Inference of Large Language Models reports standard Llama generation throughput on a 24 GB RTX 3090 for 7B-model tests. Its reported values include:
| Prompt tokens | Generated tokens | Batch | Reported rate |
|---|---|---|---|
| 5 | 8,187 | 1 | 32.02 tokens/s |
| 2,048 | 2,048 | 2 | 56.98 tokens/s |
| 5 | 2,043 | 5 | 140.88 tokens/s |
| 512 | 512 | 9 | 225.31 tokens/s |
These results come from the paper’s specific implementation, software, sequence lengths, and batch choices; they should not be used as universal RTX 3090 expectations. The full methodology is in the Association for Computational Linguistics paper.
A separate comparison used Llama 2 7B, batch size 1, 128 input tokens, 128 output tokens, and CUDA 11.6 on an RTX 3090. It reported 35.94 tokens/s for Hugging Face, 54.78 tokens/s for vLLM, and 63.303 tokens/s for FlashDecoding++. The report is hosted by the National Institute of Computer and Network Security Engineering. Different software versions, prompt lengths, output lengths, or quantization can reverse the apparent ordering.
Also popular on Amazon
- Portable and powerful USB-C HUB: BENFEI USB Type-C HUB, with super-soft and knot-free silicone woven design cable, meets most mobile office needs. Compact, lightweight, stylish, and powerful portable USB C Hub equipped with 1 x HDMI port, 1 x 100W charging, and 3 x USB ports. 18-month warranty, 24-hour response, to ensure you feel at ease when using our product.
- Design centered on comfort and reliability: Thanks to BENFEI's end-to-end in-house cable production capability, in-house PCBA and assembly capability, using the industry's most advanced silicone woven design and process, 20cm cable in length, no knots, super-soft, the HUB is easy to use in all scenarios: laptop, tablet, stand etc. Super-soft, 25000+ life cycles, to meet your daily carrying and office needs.
- 100W Charging: Support up to 90W USB C pass-through charging via Type-C port to keep your laptop powered. 10W is reserved for other interface operations. No data and video function on the Type-C port.
- 4K HDMI Display: The HDMI port supports media display at resolutions up to 4K 30Hz, keeping every incredible moment detailed and ultra vivid. Please note that the C port of the Host device needs to support video output.
- Transfer Files in Seconds: Transfer files and from your laptop at speeds up to 10 Gbps with USB A 3.2 port. Extra 2 USB A 2.0 ports are perfectly for your keyboards and mouse.
What is the best starting profile for one interactive user?
Use batch 1, a fixed prompt and output length, and a model that leaves enough memory for the intended context. Measure time to first token and steady decode tokens per second separately. Automatic KV-cache sizing is a sensible first vLLM baseline; only increase context or concurrency after confirming that memory remains stable.
If decode is slow while the GPU is fully occupied, compare a supported engine or attention path and a model quantization that fits without swapping or cache eviction. If the GPU is mostly idle, inspect CPU tokenization, scheduling, and output handling before changing model flags. A larger batch is not an interactive optimization unless multiple requests are intentionally served together.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What is the best profile for maximum server throughput?
Set a throughput target, then sweep concurrency or batch until the marginal gain is smaller than the latency cost. Keep a per-request latency percentile in the same report as aggregate tokens per second. Provision KV-cache capacity for the maximum number of active sequences and context length; otherwise, a high nominal batch can collapse into memory errors or queueing.
Use a representative mix of prompt and output lengths. A benchmark with short prompts and short generations may favor a different setting from a document-question-answering service with long prefill and short decode. Re-run the sweep after changing model, quantization, engine version, or CUDA runtime.
Also popular on Amazon
- 【Read before You Buy】 Note: This adapter is USB 2.0 version and cannot transmit HDM, VGA, or other video signals. Nevertheless, they are fully capable of charging devices,duplicating files, and can be seamlessly integrated with USB-C earphones, wireless network adapters, and SD card adapters.
- 【Easy to Use and Safe】Thanks to modern manufacturing techniques, we've made these adapters sufficiently compact so that they can remain inserted in your device without noticeable protrusion or inconvenience. This means you don't have to constantly plug and unplug them, reducing the likelihood of wear and misalignment, and making them more convenient to carry around.
- 【Reliable Quality】Engineered for enhanced durability, durable aluminum alloy shell material,get excellent wear resistance and heat dissipation,the male USB C and female USB connectors & the male USB and female USB C connectors are designed to be plugged and unplugged up to 20000 times. Provides data transfer, device charging, memory space expansion, and additional computer peripherals.
- 【Think more for you】Do you need to connect a USB-C-enabled device to an odd device that only has USB-A ports? Then these USB-A male to USB-C female adapters are made for you.Simply insert the adapter into the USB-A port on the device, and you can instantly connect your USB-C device to the device that originally didn't have the correct connection port.
- 【Trusted xiwxi Quality and Service】As a responsible brand seller, you will receive xiwxi's 12-month Quality assurance and lifetime friendly customer service, you're not just buying a product,you're investing in peace of mind with all-round quality and exceptional customer service.
How do you diagnose an RTX 3090 speed problem?
| Symptom | Likely cause | Action |
|---|---|---|
| Out-of-memory at startup or during longer prompts | Weights, context, and KV cache exceed available memory | Lower context or concurrency, use a smaller/reduced-precision model, or set an explicit cache limit with headroom. |
| GPU utilization is low | CPU contention, tokenization, scheduling, or output processing | Check host core availability and CPU utilization before changing GPU settings. |
| Throughput rises but responses feel slower | Batching improved aggregate rate while increasing queueing or per-request latency | Report latency and throughput separately; reduce concurrency for interactive traffic. |
| Speed falls during a long run | Thermal throttling or changing clocks | Monitor temperature, clocks, and power; improve cooling and use stable persistence-mode conditions. |
| Two engines produce different rates | Different kernels, versions, scheduling, or benchmark methods | Match model, lengths, batch, CUDA, and measurement definitions before drawing a conclusion. |
How do you choose settings without overfitting to a benchmark?
Keep a benchmark record that names every condition, and select the setting that meets the real service objective rather than the largest isolated tokens-per-second figure. For interactive use, prioritize time to first token and steady decode at the expected concurrency. For batch jobs, prioritize aggregate throughput while enforcing a completion-time limit. For long context, prioritize reliable memory fit and acceptable quality before chasing a smaller cache data type.
The honest answer to “which inference settings maximize LLM token speed on an RTX 3090?” is therefore a procedure: fit the model and KV cache, choose the metric, sweep batch or concurrency for that metric, remove CPU and thermal bottlenecks, and validate with the exact model and software you will deploy. No cited source establishes a universal winning engine, quantization, batch size, or flag set.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsFrequently Asked Questions
Does increasing batch size always make an RTX 3090 generate tokens faster?
No. It often increases total tokens per second when several requests can be processed together, but it can increase queueing and per-request latency. The useful batch limit is the one that meets your latency target while improving aggregate throughput.
Should I lock my RTX 3090 to its maximum clock for LLM inference?
Not as a blanket rule. NVIDIA warns that maximum locked clocks can cause thermal throttling and lower performance; stable persistence-mode measurements with monitored temperatures are safer for comparisons.
How much KV-cache memory should I reserve in vLLM?
Reserve enough for the model’s maximum context and the number of simultaneous sequences, while leaving room for weights and other allocations. Begin with automatic sizing, then use an explicit cache limit when you need predictable headroom or have non-vLLM memory consumers.
Why is my GPU underused while generation is slow?
The host may be limiting the pipeline. Tokenization, scheduling, and output processing use CPU resources, so check CPU core availability and contention before assuming the GPU setting is wrong.
Recommended Free Tools

