The Core Principle: In full 16-bit precision, a 70-billion parameter model demands roughly 140 GB of VRAM—far beyond any consumer workstation. Quantization maps those continuous weights into lower-precision integers (like 4-bit or 8-bit), shrinking footprint by over 65% with minimal measurable loss in reasoning ability.
A 70-billion parameter language model requires approximately 140 GB of memory in its native 16-bit precision. Your workstation or high-end laptop likely has 16 GB to 32 GB.
Yet engineers and researchers run models of that scale on everyday hardware every day. The technological foundation enabling this is LLM quantization: storing a model’s numeric weights using fewer bits per parameter so the entire neural network occupies dramatically less VRAM.
This technical guide demystifies quantization in plain English, walks through the memory formulas, compares standard formats like GGUF, GPTQ, and AWQ, and explains the precise trade-offs between precision and inference speed when running LLMs locally on Windows or Linux.
Quantization in One Sentence
A neural language model consists of billions of numeric weights. In native FP16 or BF16 format, each weight consumes 16 bits of memory. Quantization compresses each weight into fewer bits—such as 8, 5, or 4 bits—combined with a shared scaling coefficient across small blocks of numbers.
An Everyday Analogy: Think of rounding grocery prices from $4.87 to $5.00. You lose fractional precision on individual items, but when totaling thousands of transactions, the overall accounting remains remarkably accurate while the ledger becomes drastically easier to process.
The Memory Math: Sizing Weights
A reliable heuristic for estimating raw model weight memory:
Memory in GB ≈ (Parameters in Billions × Effective Bits per Weight) ÷ 8
| Model Size | 16-Bit (FP16 / BF16) | 8-Bit (Q8_0 / FP8) | 4-Bit (Q4_K_M, ~4.9 effective bits) |
|---|---|---|---|
| 8B Parameters | ~16.0 GB | ~8.0 GB | ~4.9 GB |
| 14B Parameters | ~28.0 GB | ~14.0 GB | ~8.6 GB |
| 32B Parameters | ~64.0 GB | ~32.0 GB | ~19.6 GB |
| 70B Parameters | ~140.0 GB | ~70.0 GB | ~43.0 GB |
Keep in mind that these numbers represent static weight storage only. In real-world inference, additional memory must be reserved for the dynamic KV cache and runtime overhead, which can be modeled using an LLM VRAM calculator.
Why Smaller Quantization Also Accelerates Speed
For single-user local inference, token generation speed is fundamentally memory bandwidth-bound. The processor must load every model parameter from memory into compute cores for every single token generated.
- Bandwidth Reduction: Reading a 4.5 GB file from RAM takes roughly one-third the time of reading a 15 GB file, resulting in 2x to 3x higher tokens per second.
- Real Benchmark: In llama.cpp tests on consumer hardware, Llama 3.1 8B jumped from ~29 tokens/sec in 16-bit to ~72 tokens/sec in Q4_K_M.
- Multi-User Serving Nuance: In high-concurrency enterprise servers, compute saturation shifts the bottleneck. Under heavy batching, 8-bit formats (like FP8) frequently outperform 4-bit due to tensor core throughput, as highlighted in GPU vs CPU inference benchmarks.
What You Give Up: The Precision Trade-Off
Quantization is an engineering compromise between memory footprint and perceptual quality:
| Quantization Level | 8B File Size | Observed Quality Impact |
|---|---|---|
| Q8_0 (8-bit) | 7.96 GiB | Practically lossless compared to native 16-bit. |
| Q6_K (6-bit) | 6.14 GiB | Near-lossless; imperceptible difference on standard benchmarks. |
| Q5_K_M (5-bit) | 5.33 GiB | Extremely high retention; excellent sweet spot for reasoning tasks. |
| Q4_K_M (4-bit) | 4.58 GiB | The gold standard consumer balance between speed and perplexity. |
| Q3_K_M (3-bit) | 3.74 GiB | Measurable degradation; syntax errors emerge in complex code. |
| Q2_K (2-bit) | 2.96 GiB | Severe quality loss; frequent hallucinations and degraded grammar. |
Rule of Thumb: 8-bit is practically lossless. 4-bit to 6-bit represents the optimal trade-off. Below 4 bits, quality degrades steeply—especially on models smaller than 14B parameters.
The Major Quantization Formats Compared
| Format | Execution Runtime | Primary Deployment Target |
|---|---|---|
| GGUF | llama.cpp, Ollama and LM Studio | Consumer PCs, laptops, CPU+GPU offloading, Apple Silicon unified memory. |
| GPTQ & AWQ | vLLM, Hugging Face TGI, ExLlamaV2 | Dedicated Linux GPU servers serving low-concurrency API workloads. |
| FP8 | vLLM, TensorRT-LLM, SGLang | Modern enterprise datacenter GPUs (Nvidia Ada, Hopper, Blackwell). |
| NVFP4 / MXFP4 | Blackwell & next-gen architectures | Native microscopic 4-bit floating point hardware execution. |
| bitsandbytes | Hugging Face Transformers | QLoRA parameter-efficient fine-tuning on consumer GPUs. |
The Hidden Memory Consumer: The KV Cache
Many developers assume model weights represent their entire VRAM footprint. However, as conversation context deepens, the KV cache (which stores key and value projection tensors for all tokens in context) expands dynamically:
- Context Overhead: For a Qwen-32B model in 16-bit, caching 32,768 tokens consumes roughly 8 GiB of VRAM per concurrent session.
- Multi-Turn Growth: In long-context agentic sessions, the KV cache can easily exceed the total size of the quantized model weights.
- Cache Quantization: Modern runtimes like llama.cpp and vLLM now allow quantizing the KV cache itself to 8-bit or 4-bit, saving up to 50% of context memory with negligible degradation.
Which Quantization Level Should You Choose?
| Your Available Hardware | Recommended Configuration |
|---|---|
| 16 GB RAM Laptop / PC (8 GB VRAM) | 7B to 8B model in Q4_K_M or Q5_K_M. |
| 24 GB to 32 GB System Memory | 14B model in Q5_K_M, or 32B model in Q4_K_M. |
| Mac with 36 GB+ Unified Memory | 32B model in Q4_K_M (Apple Silicon metal acceleration). |
| Dedicated 24 GB GPU (RTX 3090 / 4090) | 32B model in 4-bit AWQ/GGUF, leaving comfortable room for 16K+ context. |
| Enterprise Cluster (A100 / H100) | FP8 for high-throughput multi-user production inference. |
Related Reading & Deeper Context
- Best LLMs for 8GB, 16GB, and 32GB RAM — exact model sizes calibrated for 8GB, 16GB, and 32GB consumer machines.
Frequently Asked Questions
What is LLM quantization?
Quantization is the technique of reducing the numeric precision of a model’s weights from 16-bit floating point down to 8-bit, 5-bit, or 4-bit integers to drastically cut memory consumption and accelerate inference.
What does Q4_K_M mean in GGUF filenames?
Q4 designates roughly 4 bits per weight; K indicates a k-quant block-based scaling method; and M denotes a medium mix, providing higher bit precision to critical model layers while compressing less sensitive layers.
Can a 70B model run on a single 24 GB GPU?
Not comfortably. A 4-bit 70B model requires roughly 43 GB for weights alone. On a 24 GB card, a 27B to 32B model in Q4_K_M fits perfectly with sufficient headroom for KV cache context.
Does quantization make models faster?
Yes. Because single-stream generation is constrained by memory bandwidth, reducing the physical byte size of weights means the GPU or CPU can load them significantly faster on every token generation step.
Is 4-bit better than 8-bit?
4-bit saves the most memory and is ideal for consumer workstations. 8-bit is practically lossless and offers superior tensor core efficiency on high-throughput enterprise servers handling many concurrent requests.