---
title: "What is pretraining in LLMs?"
description: "Pretraining is the first stage of LLM training, where a model learns next-token prediction on a large corpus. How it works, and how it differs from fine-tuning."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/pretraining"
markdown_url: "https://theaiengineering.com/glossary/pretraining.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Pretraining","pre-training","LLM pretraining"]
citation: "The AI Engineering Editors. \"What is pretraining in LLMs?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/pretraining"
---

# What is pretraining in LLMs?

Also called pre-training, LLM pretraining.

> **Definition of Pretraining:** Pretraining is the first stage of training a language model, in which it learns to predict the next token across a large unlabeled corpus of text and code, producing a general-purpose base model.

Pretraining is the first stage of training a [large language model](https://theaiengineering.com/glossary/large-language-model): the model reads a large corpus of text and code and learns to predict the next [token](https://theaiengineering.com/glossary/tokens) at every position. No human labels are needed, because the text itself supplies the correct answer for each prediction. The result is a base model that has absorbed grammar, facts, and patterns from its data but does not yet follow instructions reliably. Application teams rarely pretrain; they pick a pretrained model and adapt it.

## How pretraining works

Pretraining works by showing the model a sequence of tokens, asking it to predict each next token, and adjusting its weights to reduce the prediction error, measured as cross-entropy loss. Repeated across a corpus collected largely from the web, this single objective teaches language, facts, and reasoning patterns without task-specific labels.

The [GPT-3 paper](https://arxiv.org/abs/2005.14165) (Brown et al., 2020) showed what this stage can produce at scale. Its 175-billion-parameter model, given no task-specific fine-tuning, handled translation, question answering, and 3-digit arithmetic from instructions and a few examples in the prompt, an ability now called [few-shot prompting](https://theaiengineering.com/glossary/few-shot-prompting).

Pretraining also fixes what a model knows. The model has no information about events after its training data was collected, its knowledge cutoff. Benchmark questions published on the web can end up in the corpus, a problem known as [benchmark contamination](https://theaiengineering.com/glossary/benchmark-contamination). [Gekhman et al. (2024)](https://arxiv.org/abs/2405.05904) found that models learn new facts slowly during fine-tuning and that such facts increase hallucination, which supports the view that LLMs acquire factual knowledge mostly through pretraining.

## Pretraining scaling laws

Scaling laws describe how pretraining loss falls as model size, training data, and compute grow. [Kaplan et al. (2020)](https://arxiv.org/abs/2001.08361) found that loss follows a power law in each of the three, with some trends spanning more than seven orders of magnitude, and that architectural details such as network width or depth matter little within a wide range.

[Hoffmann et al. (2022)](https://arxiv.org/abs/2203.15556) changed how labs spend a fixed compute budget. After training more than 400 models, they concluded that current large models were significantly undertrained, and that parameters and training tokens should grow in equal proportion: each doubling of model size calls for doubling the training tokens. Their 70-billion-parameter Chinchilla used the same compute as the 280-billion-parameter Gopher with four times more data, and outperformed Gopher and GPT-3 on a large range of evaluation tasks.

## Pretraining vs fine-tuning and post-training

Pretraining builds general capability from unlabeled text, while post-training and fine-tuning shape behavior with curated data. Post-training, run by the model developer, turns a base model into an assistant through supervised fine-tuning, preference tuning such as [RLHF](https://theaiengineering.com/glossary/rlhf), and reinforcement learning on checkable problems. Fine-tuning adapts the result to a narrower task.

| Stage | Training data | Usually run by | Result |
|---|---|---|---|
| Pretraining | Large unlabeled corpus of text and code | Model developers | A base model that continues text |
| Post-training | Instruction examples, preference rankings, problems with checkable answers | Model developers | An assistant or [reasoning model](https://theaiengineering.com/glossary/reasoning-model) |
| [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning) | A team's own task examples | Anyone with the weights or a tuning API | A model specialized for one task or format |

Pretraining decides what a model knows and how capable it can become. Post-training and fine-tuning decide how it behaves, and retrieval supplies facts that arrived after the knowledge cutoff.

[How reasoning models work](https://theaiengineering.com/articles/reasoning-models) shows how post-training on top of a pretrained base produces models that reason before they answer.

## Related terms

- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format.
- [RLHF](https://theaiengineering.com/glossary/rlhf): RLHF (reinforcement learning from human feedback) is a fine-tuning method that trains a reward model on human rankings of model outputs, then optimizes the language model with reinforcement learning to score highly.
- [Foundation model](https://theaiengineering.com/glossary/foundation-model): A foundation model is a large AI model trained on broad data, usually with self-supervised learning at scale, that can be adapted through prompting or fine-tuning to a wide range of downstream tasks.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Benchmark contamination](https://theaiengineering.com/glossary/benchmark-contamination): Benchmark contamination is the presence of a benchmark's test questions, answers, or solutions in a model's training data, so its score can reflect memorization instead of the capability the benchmark measures.

## Guides that cover pretraining

- [What is AI engineering? The discipline, the stack, and the work](https://theaiengineering.com/articles/what-is-ai-engineering): What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.
- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.
- [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is pretraining in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/pretraining

- Title: What is pretraining in LLMs?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/pretraining
- Markdown: https://theaiengineering.com/glossary/pretraining.md
