Quantization stores a model’s weights in a lower-precision format so the model needs less memory, at the price of some approximation error. Going from float16 or bfloat16 to 4 bits can shrink the weights to roughly a quarter of their size. That does not mean the model computes in 4-bit arithmetic, that total GPU memory drops by 4x, or that inference gets faster. This article explains what changes, what stays the same, and which claims you can trust.
What 4-bit quantization means
Hugging Face’s Transformers documentation describes quantization as lowering “the memory requirements of loading and using a model by storing the weights in a lower precision while trying to preserve as much accuracy as possible” (Hugging Face Transformers, “Quantization overview”). Two ideas are in that sentence: the change is to storage, and accuracy is something the method tries to preserve, not something it guarantees.
A float16 weight uses 16 bits to encode a sign, an exponent and a significand, so it can take tens of thousands of distinct values. A 4-bit code has only 16 possible values. To make that work, a quantizer maps each original weight to the nearest of a small set of representable values. It usually stores extra metadata, such as a scale for each small group of weights, so the code can be turned back into an approximate weight. The exact encoding differs by method: some use integer-like grids, others use specialized 4-bit data types such as NF4 in bitsandbytes. “4-bit” on its own does not tell you which scheme was used.
Storage precision is not compute precision
In the bitsandbytes workflow documented by Hugging Face (“4-bit quantization with Transformers and bitsandbytes”), weights are kept compressed but the arithmetic is not done in 4 bits. The guide states that “the computation is not done in 4bit, the weights and activations are compressed to that format and the computation is still kept in the desired or native dtype.” You choose that compute dtype, and it can be float16 or bfloat16. Weights are dequantized as needed for the calculation.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →#1 Best Overall
So a 4-bit model is best read as “weights stored in 4-bit form, computed in a higher-precision format.” Other methods and runtimes handle this differently through their own kernels, which is one reason their speed varies.
How much memory does it save?
Hugging Face’s “Selecting a quantization method” page (Transformers v5.6.2 docs, accessed 2026-10-05) reports about 4x memory savings for its listed 4-bit methods versus bf16. That matches simple arithmetic: 16 bits down to 4 bits is a factor of four on the weights.
| Illustrative model size | Weights at 16-bit | Weights at 4-bit (before scale metadata) |
|---|---|---|
| 8 billion parameters | about 16 GB | about 4 GB |
| 70 billion parameters | about 140 GB | about 35 GB |
These are back-of-envelope figures for weights only. Real files are somewhat larger because of scales and any layers left unquantized. Total memory at run time also includes activations, temporary buffers, the context (KV) cache and framework overhead. A small checkpoint file is therefore not proof that a model fits in an equally small amount of GPU memory, and long contexts can consume a lot of what quantization saved.
Does quantization reduce accuracy?
It introduces approximation error, because the original values are squeezed onto fewer levels. Methods differ in how they limit the damage:
Rank #3
- GPTQ is a one-shot post-training method based on approximate second-order information. Its authors (Frantar et al., 2022) reported quantizing GPT models with 175 billion parameters in approximately four GPU hours.
- AWQ uses activation statistics to find the salient channels. Lin et al. (2023) found that protecting only about 1% of salient weights can greatly reduce quantization error, while keeping the quantization weight-only and hardware-friendly. That 1% is the paper’s finding, not a rule every quantizer follows.
- bitsandbytes 4-bit quantizes on the fly without a calibration dataset for inference, which makes it simple to apply.
Hugging Face’s guide calls the accuracy of its listed 4-bit methods relatively high, but the sources reviewed do not give one universal quality-loss percentage for “4-bit quantization,” and numbers from one paper should not be moved to another model. Safe wording is that quantization can preserve much of a model’s quality in tested settings. Whether it does for your model and task has to be measured on that task, such as your own prompts or evaluation set.
Does a 4-bit model run faster?
Not automatically. Speed depends on the method, the available kernels, the hardware and the workload. Hugging Face explicitly says speedup is not guaranteed with bitsandbytes inference. Reduced memory can help when a workload is limited by memory traffic or when it lets a model fit on a device at all, but dequantization has a cost.
The GPTQ paper reports around 3.25x end-to-end inference speedup on NVIDIA A100 GPUs and 4.5x on A6000 GPUs. Those are results for that paper’s experiments and setup; they are not a general promise for 4-bit models.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.GPTQ, AWQ, bitsandbytes and GGUF compared
| Approach | What the sources say | What to check |
|---|---|---|
| bitsandbytes 4-bit | On-the-fly quantization with no calibration dataset for inference; primarily optimized for NVIDIA/CUDA; speedup not guaranteed. | Device support and measured speed. |
| GPTQ | One-shot weight quantization using approximate second-order information; Hugging Face groups it with calibration-based methods. | Calibration effort, quality on your task, kernel support. |
| AWQ | Activation-aware, weight-only; calibration is needed if you quantize yourself; Hugging Face reports strong 4-bit accuracy in its guide. | Calibration data and time, optimized kernels for your runtime. |
| GGUF (llama.cpp ecosystem) and other formats | Hugging Face’s overview lists support by method across CPU and accelerator types; formats are not interchangeable. | Target hardware, loader compatibility, the exact file you download. |
No method wins everywhere. Hugging Face’s own comparison uses Llama 3.1 8B and 70B with stated GPU, batch size, generation length and precision conditions. Those conditions are part of the result, so verify on your own model and runtime. The support matrix in the overview page also changes as libraries evolve.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsDo you need a new GPU?
No. Understanding quantization, or using a smaller model representation, does not require buying hardware. What you need depends on the model, the library and the runtime. The bitsandbytes 4-bit workflow described by Hugging Face is built around GPU/CUDA, while the Transformers overview lists CPU and several accelerator types across different methods. The practical order is: work out the model’s real memory footprint including context, confirm that your runtime supports the chosen format on your hardware, and only then consider whether new hardware is justified. The sources here do not support recommending a specific card or VRAM size.
Quick Recap
A practical checklist
- Identify the exact format (bitsandbytes NF4, GPTQ, AWQ, GGUF variant) rather than just “4-bit.”
- Confirm your runtime and hardware support that format.
- Estimate weights at about a quarter of 16-bit size, then add room for context cache, activations and overhead.
- Test quality on your own task against the 16-bit model.
- Measure speed on your setup instead of assuming a gain.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
