Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →To serve NVIDIA’s Qwen3.8-2.4T-A95B-NVFP4 checkpoint with an NVFP4 KV cache, add --kv-cache-dtype nvfp4 to a vLLM or SGLang launch command. NVIDIA’s model card says this needs a recent runtime release with NVFP4 KV support and an NVIDIA Blackwell GPU. The flag sets the precision of the attention cache while serving. It does not change the checkpoint’s weights, it is not the FP8 KV cache used in the checkpoint’s own quantization recipe, and it is not TensorRT-LLM’s NVFP4 cold-page compression, which is a separate tiered-cache feature.
Three settings that look alike
Four different settings in this area involve quantization or KV-cache precision, and each acts at a different point in the model’s lifecycle.
As an Amazon Associate I earn from qualifying purchases.
| Setting | What it controls | Where it is set | What the sources say for this checkpoint or runtime |
|---|---|---|---|
| Model weight quantization | Storage precision of the model weights | Stored in the checkpoint | Routed experts use NVFP4; self-attention and gated-delta linear attention use FP8 W8A8; the MTP block stays BF16 (NVIDIA Model Optimizer Qwen3.8 recipe) |
| Recipe KV cache | Precision the checkpoint recipe assigns to the KV cache | Applied when the checkpoint was produced with NVIDIA Model Optimizer | FP8 cast (NVIDIA Model Optimizer Qwen3.8 recipe) |
| Serving KV-cache dtype | Precision of the attention KV cache used while the model serves requests | Launch flag --kv-cache-dtype |
The model card’s examples pass nvfp4 to vLLM and SGLang; omitting the flag uses the runtime’s default KV precision |
| Cold-page compression | NVFP4 storage for eligible attention KV held in host or disk cold tiers | TensorRT-LLM tiered-cache feature | Restored to normal runtime precision before attention; the active GPU cache stays in FP16, BF16, or FP8 |
In practice, the weight precision is fixed by the checkpoint you download, the serving KV dtype is a runtime choice, and cold-page compression is a different path that leaves the GPU’s active cache unchanged.
Requirements before you launch
The model card states the requirement in one sentence:
#1 Best Overall
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
“NVFP4 KV cache requires a recent vLLM or SGLang release with NVFP4 KV support and an NVIDIA Blackwell GPU.”
The card says “recent” and does not name a minimum version, so the release notes for your runtime decide whether support exists. The sources used for this article were checked in early October 2026. Runtime flags, hardware matrices, and supported releases change, so confirm them on the day you deploy.
- GPU: NVIDIA Blackwell. TensorRT-LLM’s hardware matrix lists NVFP4 KV cache for Blackwell sm100 and sm103, which correspond to compute capability 10.0 and 10.3. The same matrix does not list Hopper or Ada.
- Runtime: a vLLM or SGLang release that supports NVFP4 KV. Support in one runtime does not carry over to another. TensorRT-LLM’s matrix is a separate statement about a separate engine.
- Checkpoint:
nvidia/Qwen3.8-2.4T-A95B-NVFP4. The model card is pinned to a repository revision, so the commands describe that revision. - Parallelism and context length: the card’s examples use eight-way tensor parallelism and a 262,144-token maximum length. Match the parallel size to your GPU count, and confirm the model loads at the context length you choose.
The checkpoint and its quantization recipe
Architecture
The card identifies the checkpoint as nvidia/Qwen3.8-2.4T-A95B-NVFP4, based on Qwen3.8-2.4T-A95B. It describes a Transformer Mixture-of-Experts model with hybrid attention and fine-grained MoE blocks, with 2.4T total parameters and 95B activated. The listed release date is 2026-08-27. These figures describe this checkpoint only, not every model loosely called a hybrid Qwen.
According to NVIDIA Model Optimizer’s Qwen3.8 recipe, the hybrid attention interleaves gated-delta (linear-attention) layers with full-attention layers. TensorRT-LLM’s Qwen3.8-Flash-Next deployment guide sets the gated-delta recurrent-state dtype separately from the KV-cache dtype. That guide covers a different, specific Qwen3.8-Flash-Next configuration, so treat it as an illustration of how the settings are separated, not as a recipe for this checkpoint.
Rank #2
- ATTENTION BUYERS: EXPORTING this item outside the U.S. requires strict compliance with the Bureau of Industry and Security (BIS) U.S. Export Administration Regulations (EAR). Non-compliance carries fines up to $1 MILLION PER VIOLATION AND UP TO 20 YEARS IMPRISONMENT for both the buyer and seller. To uphold our regulatory obligations, we strictly follow BIS "Know Your Customer" guidance by screening each order and immediately aborting transactions showing unresolved regulatory red flags. ***
- [Double-Flow-Through Design] The RTX PRO 6000 Blackwell features a double-flow-through cooling design, optimizing efficiency and airflow to sustain peak performance under 600W power loads. | [5th Gen Tensor Cores] Deliver up to 3X the performance of the previous generation and support for FP4 precision for faster AI model processing times with reduced memory usage, enabling local fine-tuning of LLMs and generative AI | [4th Gen Ray Tracing Cores] Double the ray-triangle intersection rate of the previous generation to create photoreal, physically accurate scenes and immersive 3D designs with RTX Mega Geometry, which enables up to 100X more ray-traced triangles.
- [PCIe Gen 5] Support for PCIe Gen 5 provides double the bandwidth of PCIe Gen 4, improving data-transfer speeds from CPU memory and unlocking faster performance for data-intensive tasks like AI, data science, and 3D modeling. | [GDDR7 Memory] With 96 GB of GPU memory and 1.8 TB ps bandwidth, it can tackle massive 3D and AI projects, fine-tune AI models locally, explore large-scale VR environments, and drive larger multi-app workflows.
- [DisplayPort 2.1] Achieve unparalleled visual clarity and performance, driving high resolution displays at up to 8K at 240 Hz and 16K at 60 Hz. Increased bandwidth enables seamless multi-monitor setups while HDR and higher color depth support ensures superior color accuracy for precision work, such as video editing, 3D design, and live broadcasting.
- [NVIDIA Blackwell Streaming Multiprocessor] The new SM features increased processing throughput, and new neural shaders that integrate neural networks inside of programmable shaders | DLSS 4: Multi Frame Generation ensures ultra-smooth frame pacing for lifelike simulations. | [Universal MIG] Divide a single RTX PRO 6000 Blackwell into multiple isolated instances, each with dedicated resources, allowing for concurrent execution of multiple workloads, optimized GPU utilization, and secure isolation of different applications or users. [WARRANTY] 3 YR Manufacturer's Warranty. Bulk OEM Packaging. Retail Packaging is NOT included.
What the Model Optimizer recipe quantizes
NVIDIA Model Optimizer’s Qwen3.8 recipe assigns these precisions:
- Routed experts: NVFP4
- Self-attention and gated-delta linear attention: FP8 W8A8
- KV cache: FP8 cast
- Remaining components, such as the MTP block: BF16
This recipe describes how the stored checkpoint was produced. It is not evidence that NVFP4 KV is active by default at serving time. The card’s own usage example requests NVFP4 KV with an explicit flag, and the recipe’s FP8 KV setting does not decide what your server uses.
Serving commands
These are the model card’s example forms. Use them as templates for the flags, not as commands guaranteed to run on every release or GPU.
vLLM
vllm serve nvidia/Qwen3.8-2.4T-A95B-NVFP4 \
--port 8000 \
--tensor-parallel-size 8 \
--max-model-len 262144 \
--kv-cache-dtype nvfp4 \
--reasoning-parser qwen3SGLang
python -m sglang.launch_server \
--model-path nvidia/Qwen3.8-2.4T-A95B-NVFP4 \
--port 8000 \
--tp-size 8 \
--context-length 262144 \
--kv-cache-dtype nvfp4 \
--reasoning-parser qwen3The two commands map onto each other. --tensor-parallel-size and --tp-size set the parallel degree, and --max-model-len and --context-length set the maximum length. In both examples, only --kv-cache-dtype nvfp4 controls cache precision. --reasoning-parser qwen3 controls how reasoning output is parsed. If you omit --kv-cache-dtype, the runtime uses its default KV-cache precision.
Rank #3
- System Compatibility Note: 2-slot card, 271x112x39mm, single 8-pin power, 200W TDP. Verify chassis clearance and PSU capacity before purchase.
- Dedicated Support: Please contact us directly through Amazon for any product questions or assistance you may require.
- 24GB GDDR6 on 192-Bit Bus: Massive 24GB memory with 456 GB/s bandwidth – ideal for LLMs, AI inference, 3D rendering, and generative design.
- Intel Xe2-HPG Architecture: Built on Intel's next-gen architecture with 20 Xe cores and 160 XMX engines for AI acceleration (197 INT8 TOPS).
- PCIe 5.0 Support: PCI Express 5.0 x16 interface for maximum bandwidth with the latest workstation platforms.
Launch procedure
- Confirm the GPU architecture. Run
nvidia-smi --query-gpu=name,compute_cap --format=csv. You need compute capability 10.0 or 10.3. Any other result means the card’s Blackwell requirement is not met. - Confirm the runtime version. Run
vllm --versionfor vLLM orpip show sglangfor SGLang, then compare the version with the release notes that list NVFP4 KV support. - Launch with the flag. Use the command for your runtime from the section above, adjusted to your GPU count.
- Measure against a baseline. Run the same prompts through a launch without
--kv-cache-dtype, and compare the outputs and any metrics that matter for your workload.
When the setup fails
| Symptom | Likely cause | What to do |
|---|---|---|
| The flag is rejected or has no effect | The runtime release predates NVFP4 KV support | Upgrade to a release whose notes list NVFP4 KV support, then relaunch |
| The GPU is Hopper or Ada | The card requires Blackwell, and TensorRT-LLM’s matrix does not list NVFP4 KV for these GPUs | Omit the flag and serve with the runtime’s default KV-cache precision |
| Output quality drops on your workload | Accuracy loss after post-training quantization varies by model and method, according to the Hugging Face PTQ documentation | Change or disable KV-cache quantization, or use quantization-aware training, as that documentation suggests |
TensorRT-LLM: a separate NVFP4 KV path
Generating an NVFP4 KV checkpoint
TensorRT-LLM’s general quantization documentation says its NVFP4 KV checkpoint-generation flow currently requires FP8 weight and activation quantization. It lists Qwen-3 as supported for NVFP4 KV cache. The same page distinguishes NVFP4 KV from general NVFP4 weight support, so a setup that runs NVFP4 weights in TensorRT-LLM has not thereby enabled NVFP4 KV.
Cold-page compression
TensorRT-LLM’s cold-page compression stores eligible attention KV in NVFP4 in host or disk cold tiers. The active GPU cache keeps its ordinary runtime type, such as FP16, BF16, or FP8. When a cold page is needed, it is restored to normal runtime precision before attention runs. The documentation explicitly separates this from active GPU KV configured as NVFP4.
Enabling cold-page compression therefore does not give you an NVFP4 KV cache on the GPU. The vLLM and SGLang flag is a serving-cache setting, not the TensorRT-LLM cold-tier feature.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minutePublished benchmark results
NVIDIA’s model card reports scores for three configurations: BF16, NVFP4 weights, and NVFP4 weights with an NVFP4 KV cache. These are the publisher’s own results, dated 2026, not measurements made for this article. The card specifies temperature 1.0, top-p 0.95, and top-k 20. Maximum new tokens varied by benchmark, as shown in the last column.
| Benchmark | BF16 | NVFP4 | NVFP4 + NVFP4 KV | Max new tokens |
|---|---|---|---|---|
| GPQA Diamond | 92.55 | 92.58 | 92.33 | 65,536 |
| HLE | 41.43 | 40.55 | 40.64 | 131,072 |
| SciCode | 54.44 | 56.21 | 55.92 | 65,536 |
| AA-LCR | 71.50 | 71.63 | 71.25 | 65,536 |
| IFBench | 79.93 | 81.73 | 81.33 | 65,536 |
| Terminal Bench 2.1 | 76.03 | 76.40 | 77.25 | 262,144 |
Against the BF16 column, the NVFP4 KV row ranges from 0.79 points lower on HLE to 1.48 points higher on SciCode. These are single reported figures, and a gap of about a point can fall within normal run-to-run variation. Read them as no large loss on these six benchmarks under the card’s settings, not as a measured speedup or a guaranteed accuracy result for other workloads.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
