Free tools Windows power users keep installed
One-click scans. No signup required.
SM120’s shared-memory limit is a real kernel-design constraint, but the available evidence does not establish it as the cause of FP8 KV-cache crashes in vLLM. A cache-allocation out-of-memory error, a kernel dispatch failure, and a CUDA stream-capture error point to different problems. The GPU model, vLLM version, attention backend, model, and first traceback are needed to identify which one you have.
What the SM120 shared-memory “cliff” actually is
NVIDIA’s CUDA 13.0 Blackwell Tuning Guide lists lower shared-memory limits for compute capability 12.0 (SM120) than for compute capability 10.0 (SM100). The distinction matters at two levels: the total capacity available to a streaming multiprocessor (SM) and the maximum a single thread block can use.
As an Amazon Associate I earn from qualifying purchases.
| Compute capability | Shared memory capacity per SM | Maximum shared memory per thread block |
|---|---|---|
| 12.0 (SM120) | 128 KB | 99 KB |
| 10.0 (SM100) | 228 KB | 227 KB |
These figures come from NVIDIA’s Blackwell Tuning Guide for CUDA 13.0. A kernel that needs more shared memory per block than the device allows cannot run in that configuration; a kernel can also be constrained by how much of the SM’s shared memory is available for concurrent blocks.
That is a kernel resource limit, not a statement that the KV cache is stored in per-block shared memory. vLLM’s --kv-cache-dtype selects the representation used for cache storage. A kernel may use shared memory while operating on cache data, but the architecture table alone does not show that an FP8 cache exceeds the block limit or caused a particular crash.
#1 Best Overall
- [ Maximum AI Compute Power ] Dominate complex workloads with the ASUS ESC8000A-E13. This 4U rack server is a powerhouse engineered for mass-scale AI, machine learning, and deep training. Featuring support for dual AMD EPYC 9005/9004 processors and up to eight dual-slot GPUs, it delivers the raw computational muscle required to train LLMs and run complex simulations effortlessly. Accelerate your data science pipeline and transform raw data into actionable intelligence faster than ever.
- [ Advanced Thermal Efficiency ] High performance demands elite cooling. The ESC8000A-E13 features a cutting-edge aerodynamic design with independent CPU and GPU airflow tunnels. Equipped with redundant hot-swap fans and optimized for liquid cooling integrations, this 4U server ensures maximum uptime under heavy, sustained workloads. Keep your data center running cool, quiet, and highly efficient while preventing thermal throttling during mission-critical enterprise operations.
- [ Scale with Flexible Storage ] Future-proof your infrastructure with unmatched storage and expansion flexibility. This offers comprehensive front-panel drive bays supporting Gen5 NVMe, SAS, or SATA drives alongside multiple PCIe 5.0 slots. Designed as a high-density 4U server capable of housing eight dual-slot GPUs: NVD H200, RTX PRO 6000 Blackwell, RTX PRO 4500 Blackwell or AMD Instinct MI350P PCIe Card, each supporting up to 600 watts.
- [ Enterprise-Grade Reliability ] Minimize downtime and secure your ecosystem with server-grade redundancy. The ESC8000A-E13 is built for 24/7 continuous operation, boasting 2+2 redundant (3200W total) 80 PLUS Titanium power supplies and integrated ASUS ASMB11-iKVM for comprehensive out-of-band management. Ideal for cloud service providers, rendering farms, and large enterprise infrastructure, it combines robust physical hardware with smart remote monitoring to safeguard your digital assets.
- [Reliability Guaranteed] Shop with total peace of mind knowing that every new computer component we sell is backed by our EPC 3-year warranty. Whether you are investing in high-speed DDR5 RAM or a powerhouse GPU, we protect your build against defects and performance failures. We stand firmly behind the quality of our hardware, ensuring that your setup remains fast, stable, and secure for years to come.
Why FP8 KV cache does not identify the cause
“FP8” can refer to different parts of inference. FP8 KV cache means the cache’s storage dtype; FP8 model weights or activations use different execution paths. vLLM’s serve CLI documentation lists fp8, fp8_e4m3, and fp8_e5m2 as KV-cache dtype options, as well as controls including --gpu-memory-utilization and --kv-cache-memory-bytes. A listed option does not guarantee that every model, GPU, software version, and attention backend combination will work.
Keep the failure categories separate. A device-memory allocation failure is not the same symptom as a per-block shared-memory limit, and neither is interchangeable with a dispatch error, illegal memory access, or failure during CUDA graph capture. Changing the cache dtype may affect memory use or the selected kernel path, but it does not by itself reveal which category produced the error.
Rank #2
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
What reported SM120 and FP8 failures show
The reports below describe distinct situations. Issue reports establish that an error was encountered; they do not automatically establish an independently confirmed root cause.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →| Context | Reported problem | What the report supports |
|---|---|---|
| DGX Spark, FP8 workload | NVIDIA’s vLLM Release 26.02 notes describe CUDA stream-capture errors associated with illegal synchronization operations in FlashInfer kernels, and say a FlashInfer fix is available. | A documented issue in that platform and backend context—not proof that all SM120 workstation FP8 workloads fail. See NVIDIA’s vLLM Release 26.02 notes, last updated September 29, 2026. |
| Nemotron Super V3, FP8 attention | The same NVIDIA notes specify a Triton attention backend requirement for this model’s FP8 operation. | A model-specific backend requirement, not a general rule for every FP8 KV cache. |
| RTX PRO 6000 Blackwell under WSL2 | A vLLM issue reports a KV-cache allocation failure involving a large contiguous Mamba state allocation; its author says switching the KV cache to FP8 did not resolve that case. | The issue author proposed hidden WSL2 memory overhead as an explanation. That is the author’s hypothesis, not a project-confirmed root cause. See the Qwen3.6 hybrid Mamba KV allocation issue. |
| SM120, FP8 block-scaled DeepSeek model | A separate issue reports a Cutlass dispatch_scaled_mm failure during profiling. |
This concerns model-weight kernel dispatch, not evidence of an FP8 KV-cache failure. See the SM120 FP8 block-scaled Cutlass issue. |
| RTX PRO 4000 SM120, low-batch workload | A community author reported FP8 KV-cache dequantization overhead and recommended --kv-cache-dtype auto for that workload. |
This is a workload-specific field report, not a universal compatibility or performance rule. The author says the setup used vLLM 0.15.1, 0.17.1, and 0.18.1 with CUDA 13.0; the report is dated April 3, 2026. See the RTX PRO 4000 SM120 field report. |
How to narrow down the crash
Start with the first error, not the last line of a long log. Note whether vLLM fails during model loading, warmup or profiling, graph capture, or request execution. That stage can help distinguish a cache allocation problem from a kernel or capture problem.
Quick Recap
Rank #4
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
Rank #3
- AI-Optimized: Designed to support up to 4 GPUs, it is perfect for handling intensive AI and machine learning tasks, ensuring high performance and scalability for advanced computational needs.
- Intelligent Storage: Equipped with 8 hot-swappable 3.5" SATA/SAS drives (12Gbps), featuring SGPIO and temperature control, it ensures efficient data management and reliable storage performance.
- Robust Cooling: The system includes 3x 12038 hot-swap PWM fans and 2x 8038 rear fans, providing advanced thermal management to maintain optimal temperatures and ensure stable operation under heavy workloads.
- Rack-Ready: Comes with a pre-installed rail kit, allowing for quick and easy installation in standard 19-inch server racks, making it ideal for data center environments and enterprise setups.
- Versatile Connectivity: Offers USB 3.0 and the latest USB 3.2 Type-C ports, ensuring high-speed data transfer and compatibility with a wide range of peripherals and devices for enhanced connectivity options.
- Record the exact setup. Capture the GPU SKU and compute capability, vLLM version or image tag, CUDA toolkit/runtime, driver, PyTorch version, FlashInfer version, model name, and attention backend.
- Clarify what “FP8” means in the run. Record whether FP8 applies to KV cache, quantized weights or activations, or more than one of them. Include the full launch command, especially the attention-backend setting,
--kv-cache-dtype, and any memory-sizing flags. - Classify the first failure. Preserve the complete first error and traceback. An allocation OOM, shared-memory resource error, illegal memory access, dispatch failure, and stream-capture error are different observations; do not relabel one as another.
- Change one variable per reproduction. If testing a different KV dtype or attention backend, keep the other settings fixed and compare the first error and failure stage. This helps tell whether the issue follows the cache representation or a particular kernel/backend path.
- For a suspected memory-budget problem, test the budget separately. vLLM exposes
--gpu-memory-utilizationand--kv-cache-memory-bytesfor device-memory budgeting. NVIDIA’s release notes advise reducing utilization in shared/unified-memory OOM contexts. Such a test addresses memory budgeting; it does not demonstrate that a shared-memory ceiling caused the original failure.
What to include in a useful bug report
- Exact GPU SKU and compute capability, such as RTX PRO 6000 Blackwell (SM120).
- vLLM version or container image tag, CUDA toolkit/runtime, driver, PyTorch, and FlashInfer versions.
- Model name and a clear statement of whether FP8 refers to cache storage, model weights or activations, or both.
- Attention backend, full launch flags, KV dtype, and memory-utilization or cache-size settings.
- Startup logs and the full first traceback, plus the stage at which the failure occurs: loading, warmup/profile, graph capture, or request execution.
- A minimal reproduction and controlled comparisons that change only one setting at a time.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
