Neither INT8 nor FP8 is a universal winner for large language model (LLM) quantization. Accuracy and speed depend on the specific quantization recipe—especially how it handles unusually large activation values and how many values share each scale—as well as the model, hardware, and kernels. INT8 methods can isolate or transform outliers; FP8 uses a floating-point representation with its own range-and-precision trade-off.
Why do LLM activations have outliers?
An activation is an intermediate value produced as a model processes its input. Quantization represents these values with fewer bits than the original computation, mapping a range of values onto a smaller set of representable levels. The challenge is that activation values are not always spread evenly: a small number can be much larger than the rest.
As an Amazon Associate I earn from qualifying purchases.
In their analysis of the transformer models they studied, the authors of LLM.int8() found large values concentrated in a small number of feature dimensions, rather than appearing only as isolated random spikes. They reported magnitudes up to about 20 times those of other dimensions in that analysis. In their model series, affected layers increased with model scale; at around 6.7 billion parameters, they reported outlier features across all layers, concentrated in a small set of dimensions. Removing those dimensions substantially harmed the attention and perplexity measures they evaluated. These observations describe the paper’s models and experiments, not a universal threshold for every current architecture.
Recommended Free Tools
How can one scale make ordinary values harder to represent?
A quantizer’s scale determines how its limited set of output levels maps to the values being represented. In a simple symmetric INT8 example, a scale may be set using the largest absolute value in the group. If that group contains an extreme activation, the range must stretch to cover it. With one shared, coarse scale, many more common, smaller values can then occupy only a narrow portion of the available integer levels. Rounding those values to the nearest level may lose detail that matters to the model.
#1 Best Overall
- Graphics Card Interface: Pci E
This is an intuition, not a description of every quantizer: implementations can use different calibration procedures, symmetric or asymmetric mappings, and different scale shapes. The key question is which values share a scale. A tensor-wide scale is simple, but a local extreme can govern the range for the entire tensor. Row-, vector-, channel-, token-, or group-level scales can fit local ranges more closely. Their added scales and conversions also have implementation costs, and finer granularity is not automatically faster or better.
How do INT8 methods handle activation outliers?
LLM.int8(): route exceptional dimensions separately
LLM.int8() uses vector-wise quantization and a mixed-precision decomposition. It sends outlier feature dimensions through a 16-bit matrix-multiplication path while keeping the bulk of computation in INT8. The authors report that more than 99.9% of values are still multiplied in 8-bit. The point is not that INT8 alone makes outliers disappear: the method identifies dimensions that need special treatment and limits the higher-precision path to them.
Rank #2
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
SmoothQuant: redistribute the quantization difficulty
SmoothQuant takes another approach. Its offline, mathematically equivalent transformation scales down activation channels with outliers and compensates by scaling the corresponding weights. This shifts part of the quantization burden from activations to weights, which the authors found easier to quantize. Their 2023 PMLR paper presents this as training-free W8A8 INT8 quantization for LLM matrix multiplications.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →The SmoothQuant authors report up to 1.56× speedup and 2× memory reduction in their tested models and setups. Those are reported maxima, not guaranteed gains for a different model, kernel, or deployment.
Rank #3
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
How does FP8 compare with INT8?
INT8 uses integer values; FP8 refers to a family of 8-bit floating-point encodings. Different FP8 encodings allocate bits differently between exponent and significand, changing the range and precision trade-off. Neither label specifies the whole quantization method: the scale strategy, calibration or training procedure, which tensors are quantized, and the kernels used all affect the result.
| Approach | Representation and outlier strategy | What the cited work establishes |
|---|---|---|
| LLM.int8() | INT8 vector-wise quantization with an outlier-dimension path using 16-bit multiplication. | The authors report more than 99.9% of values multiplied in 8-bit in their method. Paper |
| SmoothQuant | W8A8 INT8 after an offline transformation that reduces activation extremes while compensating in weights. | The authors report up to 1.56× speedup and 2× memory reduction for their tested setups. Paper |
| ZeroQuant-FP study | FP8 activation quantization compared with an INT8 equivalent in the paper’s post-training LLM experiments. | The authors report FP8 activations outperforming the INT8 equivalent in their tested configurations, with a more noticeable difference for models above one billion parameters. This is a study-specific finding, not a format-wide guarantee. Preprint |
The ZeroQuant-FP result is evidence that FP8 can be advantageous in a particular method and benchmark setup, not that FP8 always wins. Its authors study post-training quantization and discuss FP8/FP4 in the context of NVIDIA H100 hardware. A separate comparison needs to hold the model, evaluation, hardware, and serving implementation as consistent as possible before treating one result as a deployment recommendation.
Rank #4
- Robust Design:Constructed to withstand high temperatures, the V100 16GB SXM2 card operates efficiently up to 105℃.
- Advanced Connectivity:Features a SXM2 connector for seamless integration with a wide range of systems, ensuring compatibility.
Training results should not be mistaken for inference results. A 2024 study of long-running FP8 training, describing models trained on datasets up to 2 trillion tokens, associates an instability with prolonged amplification of SwiGLU outliers and proposes Smooth-SwiGLU. That finding concerns training dynamics; it does not establish that FP8 inference is inherently unstable or inferior. Read the study.
What should you compare before choosing a quantization recipe?
There is no common benchmark in the cited work that tests current INT8 and FP8 implementations across the same models, hardware, kernels, and evaluation sets. Treat paper results as evidence about their particular methods and setups. For a real deployment, compare the candidate recipes on the target system:
- Model quality: Evaluate perplexity and the task-specific outputs that matter to your application, using the same prompts, data, and evaluation procedure.
- Performance: Measure prefill and decode latency and throughput separately. Include conversion, calibration, or mixed-precision overhead where applicable.
- Memory: Account for weights, activations, scale metadata, and any higher-precision side path—not just the nominal bits per value.
- Scaling and outlier handling: Identify which values share scales and whether the recipe leaves outliers in the low-precision path, routes them separately, or transforms the distribution.
- Hardware and kernels: Check accelerator support, library and kernel availability, and serving-stack compatibility. NVIDIA’s post-training quantization discussion emphasizes that sensitivity and target hardware matter.
Choose the recipe that meets your quality and performance requirements on the hardware and workload you will actually use. A format name by itself cannot predict the outcome.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
