---
title: "What is GGUF? The llama.cpp model file format"
description: "GGUF is the single-file model format used by llama.cpp and Ollama, storing quantized weights with tokenizer and model metadata. How it works and vs safetensors."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/gguf"
markdown_url: "https://theaiengineering.com/glossary/gguf.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["GGUF","GGUF format","GGUF file","GGUF quantization"]
citation: "The AI Engineering Editors. \"What is GGUF? The llama.cpp model file format.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/gguf"
---

# What is GGUF? The llama.cpp model file format

Also called GGUF format, GGUF file, GGUF quantization.

> **Definition of GGUF:** GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file.

GGUF is the model file format of the ggml ecosystem. Tools such as llama.cpp, Ollama, and LM Studio run it, and the Hugging Face Hub has built-in support for it. The [GGUF specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) describes a binary format for models used with GGML and GGML-based executors, designed so that one file contains everything needed to load a model. GGUF files usually hold [quantized](https://theaiengineering.com/glossary/quantization) weights, which makes the format a common way to run 4-bit or 5-bit versions of [open-weights models](https://theaiengineering.com/glossary/open-weights-model) on laptops, Apple silicon, and CPUs.

## How a GGUF file is structured

A GGUF file has four parts in order: a header with the magic bytes `GGUF` and a version number, typed metadata key-value pairs, a table describing each tensor, and the aligned tensor data. The metadata records the architecture, hyperparameters, tokenizer, and chat template, so loading the model needs no separate config files.

The specification's current version is 3. The required `general.architecture` key names the model family, such as `llama`, and keys under `tokenizer.ggml` store the vocabulary, while `tokenizer.chat_template` can hold a Jinja chat template. Tensor data is padded to an alignment boundary (32 bytes unless `general.alignment` says otherwise) so engines can memory-map the file with `mmap` and load it quickly. GGUF replaced the earlier GGML, GGMF, and GGJT formats, mainly so that files record their architecture and can gain new metadata without breaking older readers. Large models can be split into numbered shards, such as `00001-of-00003`.

## GGUF quantization types

GGUF quantization type names describe how weights are packed into blocks. Q4_0, Q5_0, and Q8_0 are older round-to-nearest formats with 32-weight blocks. K-quants such as Q4_K and Q6_K arrange weights in small blocks, each with its own scale, grouped into super-blocks. IQ types use an importance matrix and go as low as 1.56 bits per weight for IQ1_S.

Selected types from the [Hugging Face Hub GGUF documentation](https://huggingface.co/docs/hub/gguf), which lists the bits per weight and block layout of each:

| Type | Bits per weight | Block layout |
|---|---|---|
| Q6_K | 6.5625 | Super-blocks of 16 blocks, 16 weights each |
| Q5_K | 5.5 | Super-blocks of 8 blocks, 32 weights each |
| Q4_K | 4.5 | Super-blocks of 8 blocks, 32 weights each |
| Q3_K | 3.4375 | Super-blocks of 16 blocks, 16 weights each |
| IQ4_XS | 4.25 | 256-weight super-blocks with an importance matrix |
| IQ2_XXS | 2.06 | 256-weight super-blocks with an importance matrix |

File names often add a variant suffix, as in Q4_K_M, a type that llama.cpp's `llama-quantize` tool produces. The usual workflow converts a Hugging Face checkpoint with llama.cpp's `convert_hf_to_gguf.py`, then quantizes the result. Ollama imports an existing GGUF file through a Modelfile `FROM` line and does not quantize it during import.

## GGUF vs safetensors

Safetensors stores only tensors, while GGUF stores tensors plus a standardized set of metadata in one file. [Safetensors](https://huggingface.co/docs/safetensors/index) was built as a safe alternative to pickle with zero-copy loading, and Hugging Face checkpoints keep the config and tokenizer in separate files beside it. GGUF is the native format of llama.cpp and the tools built on it.

| | GGUF | Safetensors |
|---|---|---|
| Contents | Tensors plus architecture, tokenizer, and chat template metadata | Tensors only |
| Files per model | One file, or numbered shards | Weight files plus separate config and tokenizer files |
| Typical runtimes | llama.cpp, Ollama, LM Studio | Hugging Face Transformers and GPU serving engines |
| Loading | Aligned for `mmap` | Zero-copy loading |

The formats overlap at the edges. Ollama imports both, and vLLM can load GGUF through a plugin, but its [GGUF documentation](https://docs.vllm.ai/en/latest/features/quantization/gguf/) calls that support highly experimental and recommends using the base model's tokenizer instead of the one converted from the GGUF file. In practice, the format follows from the choice of [inference engine](https://theaiengineering.com/glossary/inference-engine).

[vLLM vs Ollama](https://theaiengineering.com/articles/vllm-vs-ollama) compares the engines that run GGUF files with those that serve Hugging Face checkpoints.

## Related terms

- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.
- [Open-weights model](https://theaiengineering.com/glossary/open-weights-model): An open-weights model is an AI model whose trained parameters are published for download, so anyone can run, fine-tune, or quantize it on their own hardware under the terms of its license.
- [Small language model](https://theaiengineering.com/glossary/small-language-model): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.

## Guides that cover gguf

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is GGUF? The llama.cpp model file format." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/gguf

- Title: What is GGUF? The llama.cpp model file format
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/gguf
- Markdown: https://theaiengineering.com/glossary/gguf.md
