---
title: "What is quantization in LLMs? INT8, 4-bit, and GGUF"
description: "LLM quantization stores model weights at lower precision, such as 8-bit or 4-bit, to cut memory use and speed up inference. Methods, formats, and trade-offs."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/quantization"
markdown_url: "https://theaiengineering.com/glossary/quantization.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Quantization","LLM quantization","model quantization","4-bit quantization"]
citation: "The AI Engineering Editors. \"What is quantization in LLMs? INT8, 4-bit, and GGUF.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/quantization"
---

# What is quantization in LLMs? INT8, 4-bit, and GGUF

Also called LLM quantization, model quantization, 4-bit quantization.

> **Definition of Quantization:** Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.

Quantization is the process of reducing the number of bits used to store each number in a model. Many [open-weight](https://theaiengineering.com/glossary/open-weights-model) LLMs are released in 16-bit formats such as BF16; quantizing to 8 bits halves the memory the weights need, and quantizing to 4 bits cuts it to roughly a quarter. Smaller weights let a model fit on fewer or cheaper GPUs, and because generating each token requires reading the weights from memory, smaller weights can also speed up generation.

## How LLM quantization works

A quantizer maps a range of floating-point values onto a small set of integers plus a scale factor, usually computed per block or per channel of weights. The main methods differ in how they keep the rounding error small where it matters most:

- **LLM.int8()** ([Dettmers et al., 2022](https://arxiv.org/abs/2208.07339)) keeps rare outlier feature dimensions in 16-bit and runs more than 99.9% of the computation in 8-bit, with no reported degradation up to 175B parameters.
- **GPTQ** ([Frantar et al., 2022](https://arxiv.org/abs/2210.17323)) uses approximate second-order information to quantize weights to 3 or 4 bits in one pass, processing a 175B model in about four GPU hours.
- **AWQ** (Lin et al., 2023) identifies the roughly 1% of salient weights from activation statistics and scales them to protect accuracy.
- **[GGUF](https://theaiengineering.com/glossary/gguf)** is the single-file format used by llama.cpp and Ollama; its Q4_K type stores about 4.5 bits per weight.

LLM.int8(), GPTQ, and AWQ are post-training quantization (PTQ) methods, applied to a finished model. Quantization-aware training (QAT) simulates low precision during training instead, at the cost of a training run. The same idea applies to the [KV cache](https://theaiengineering.com/glossary/kv-cache): vLLM, for example, can store it in FP8 to fit more tokens in GPU memory.

## Example: how much memory quantization saves

Weight memory is roughly the parameter count times the bytes per parameter, before counting the KV cache and activations:

| Precision | Bytes per weight | 8B model | 70B model |
|---|---|---|---|
| BF16 or FP16 | 2 | ~16 GB | ~140 GB |
| INT8 or FP8 | 1 | ~8 GB | ~70 GB |
| 4-bit | ~0.5 | ~4 GB | ~35 GB |

That difference determines whether a model needs several GPUs, fits on one, or runs on a laptop. Quality loss varies by model, method, bit width, and task, so measure it on your own [evaluation](https://theaiengineering.com/glossary/llm-evaluation) set.

Quantization pairs naturally with [knowledge distillation](https://theaiengineering.com/glossary/knowledge-distillation), which shrinks the model itself. [Serving engines](https://theaiengineering.com/platforms/serving-engines) differ in which formats they load; the [vLLM vs Ollama](https://theaiengineering.com/articles/vllm-vs-ollama) comparison covers that split.

## Related terms

- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Small language model](https://theaiengineering.com/glossary/small-language-model): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
- [Knowledge distillation](https://theaiengineering.com/glossary/knowledge-distillation): Knowledge distillation is a model compression technique in which a smaller student model is trained to reproduce the outputs or output probabilities of a larger, more capable teacher model.
- [LoRA](https://theaiengineering.com/glossary/lora): LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters and memory use.
- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [GGUF](https://theaiengineering.com/glossary/gguf): GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.

## Guides that cover quantization

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is quantization in LLMs? INT8, 4-bit, and GGUF." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/quantization

- Title: What is quantization in LLMs? INT8, 4-bit, and GGUF
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/quantization
- Markdown: https://theaiengineering.com/glossary/quantization.md
