What Is LLM Quantization? How Smaller Models Run on Less Memory
Quantization stores an AI model's numbers in fewer bits, so a model that needs 140 GB can fit in about 43 GB. Here is how it works, what it costs in quality, which format to choose, and how to test it.