What is quantization in LLMs? INT8, 4-bit, and GGUF

Also called LLM quantization, model quantization, 4-bit quantization

Definition

Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.

Quantization is the process of reducing the number of bits used to store each number in a model. Many open-weight LLMs are released in 16-bit formats such as BF16; quantizing to 8 bits halves the memory the weights need, and quantizing to 4 bits cuts it to roughly a quarter. Smaller weights let a model fit on fewer or cheaper GPUs, and because generating each token requires reading the weights from memory, smaller weights can also speed up generation.

How LLM quantization works#

A quantizer maps a range of floating-point values onto a small set of integers plus a scale factor, usually computed per block or per channel of weights. The main methods differ in how they keep the rounding error small where it matters most:

  • LLM.int8() (Dettmers et al., 2022) keeps rare outlier feature dimensions in 16-bit and runs more than 99.9% of the computation in 8-bit, with no reported degradation up to 175B parameters.
  • GPTQ (Frantar et al., 2022) uses approximate second-order information to quantize weights to 3 or 4 bits in one pass, processing a 175B model in about four GPU hours.
  • AWQ (Lin et al., 2023) identifies the roughly 1% of salient weights from activation statistics and scales them to protect accuracy.
  • GGUF is the single-file format used by llama.cpp and Ollama; its Q4_K type stores about 4.5 bits per weight.

LLM.int8(), GPTQ, and AWQ are post-training quantization (PTQ) methods, applied to a finished model. Quantization-aware training (QAT) simulates low precision during training instead, at the cost of a training run. The same idea applies to the KV cache: vLLM, for example, can store it in FP8 to fit more tokens in GPU memory.

Example: how much memory quantization saves#

Weight memory is roughly the parameter count times the bytes per parameter, before counting the KV cache and activations:

Precision Bytes per weight 8B model 70B model
BF16 or FP16 2 ~16 GB ~140 GB
INT8 or FP8 1 ~8 GB ~70 GB
4-bit ~0.5 ~4 GB ~35 GB

That difference determines whether a model needs several GPUs, fits on one, or runs on a laptop. Quality loss varies by model, method, bit width, and task, so measure it on your own evaluation set.

Quantization pairs naturally with knowledge distillation, which shrinks the model itself. Serving engines differ in which formats they load; the vLLM vs Ollama comparison covers that split.

Cite this definition

The AI Engineering Editors. "What is quantization in LLMs? INT8, 4-bit, and GGUF." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/quantization

Title
What is quantization in LLMs? INT8, 4-bit, and GGUF
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM inference and serving hub · Updated