What is GGUF? The llama.cpp model file format

Also called GGUF format, GGUF file, GGUF quantization

Definition

GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file.

GGUF is the model file format of the ggml ecosystem. Tools such as llama.cpp, Ollama, and LM Studio run it, and the Hugging Face Hub has built-in support for it. The GGUF specification describes a binary format for models used with GGML and GGML-based executors, designed so that one file contains everything needed to load a model. GGUF files usually hold quantized weights, which makes the format a common way to run 4-bit or 5-bit versions of open-weights models on laptops, Apple silicon, and CPUs.

How a GGUF file is structured#

A GGUF file has four parts in order: a header with the magic bytes GGUF and a version number, typed metadata key-value pairs, a table describing each tensor, and the aligned tensor data. The metadata records the architecture, hyperparameters, tokenizer, and chat template, so loading the model needs no separate config files.

The specification's current version is 3. The required general.architecture key names the model family, such as llama, and keys under tokenizer.ggml store the vocabulary, while tokenizer.chat_template can hold a Jinja chat template. Tensor data is padded to an alignment boundary (32 bytes unless general.alignment says otherwise) so engines can memory-map the file with mmap and load it quickly. GGUF replaced the earlier GGML, GGMF, and GGJT formats, mainly so that files record their architecture and can gain new metadata without breaking older readers. Large models can be split into numbered shards, such as 00001-of-00003.

GGUF quantization types#

GGUF quantization type names describe how weights are packed into blocks. Q4_0, Q5_0, and Q8_0 are older round-to-nearest formats with 32-weight blocks. K-quants such as Q4_K and Q6_K arrange weights in small blocks, each with its own scale, grouped into super-blocks. IQ types use an importance matrix and go as low as 1.56 bits per weight for IQ1_S.

Selected types from the Hugging Face Hub GGUF documentation, which lists the bits per weight and block layout of each:

Type Bits per weight Block layout
Q6_K 6.5625 Super-blocks of 16 blocks, 16 weights each
Q5_K 5.5 Super-blocks of 8 blocks, 32 weights each
Q4_K 4.5 Super-blocks of 8 blocks, 32 weights each
Q3_K 3.4375 Super-blocks of 16 blocks, 16 weights each
IQ4_XS 4.25 256-weight super-blocks with an importance matrix
IQ2_XXS 2.06 256-weight super-blocks with an importance matrix

File names often add a variant suffix, as in Q4_K_M, a type that llama.cpp's llama-quantize tool produces. The usual workflow converts a Hugging Face checkpoint with llama.cpp's convert_hf_to_gguf.py, then quantizes the result. Ollama imports an existing GGUF file through a Modelfile FROM line and does not quantize it during import.

GGUF vs safetensors#

Safetensors stores only tensors, while GGUF stores tensors plus a standardized set of metadata in one file. Safetensors was built as a safe alternative to pickle with zero-copy loading, and Hugging Face checkpoints keep the config and tokenizer in separate files beside it. GGUF is the native format of llama.cpp and the tools built on it.

GGUF Safetensors
Contents Tensors plus architecture, tokenizer, and chat template metadata Tensors only
Files per model One file, or numbered shards Weight files plus separate config and tokenizer files
Typical runtimes llama.cpp, Ollama, LM Studio Hugging Face Transformers and GPU serving engines
Loading Aligned for mmap Zero-copy loading

The formats overlap at the edges. Ollama imports both, and vLLM can load GGUF through a plugin, but its GGUF documentation calls that support highly experimental and recommends using the base model's tokenizer instead of the one converted from the GGUF file. In practice, the format follows from the choice of inference engine.

vLLM vs Ollama compares the engines that run GGUF files with those that serve Hugging Face checkpoints.

Cite this definition

The AI Engineering Editors. "What is GGUF? The llama.cpp model file format." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/gguf

Title
What is GGUF? The llama.cpp model file format
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM inference and serving hub · Updated