What is pretraining in LLMs?
Also called pre-training, LLM pretraining
Definition
Pretraining is the first stage of training a language model, in which it learns to predict the next token across a large unlabeled corpus of text and code, producing a general-purpose base model.
Pretraining is the first stage of training a large language model: the model reads a large corpus of text and code and learns to predict the next token at every position. No human labels are needed, because the text itself supplies the correct answer for each prediction. The result is a base model that has absorbed grammar, facts, and patterns from its data but does not yet follow instructions reliably. Application teams rarely pretrain; they pick a pretrained model and adapt it.
How pretraining works#
Pretraining works by showing the model a sequence of tokens, asking it to predict each next token, and adjusting its weights to reduce the prediction error, measured as cross-entropy loss. Repeated across a corpus collected largely from the web, this single objective teaches language, facts, and reasoning patterns without task-specific labels.
The GPT-3 paper (Brown et al., 2020) showed what this stage can produce at scale. Its 175-billion-parameter model, given no task-specific fine-tuning, handled translation, question answering, and 3-digit arithmetic from instructions and a few examples in the prompt, an ability now called few-shot prompting.
Pretraining also fixes what a model knows. The model has no information about events after its training data was collected, its knowledge cutoff. Benchmark questions published on the web can end up in the corpus, a problem known as benchmark contamination. Gekhman et al. (2024) found that models learn new facts slowly during fine-tuning and that such facts increase hallucination, which supports the view that LLMs acquire factual knowledge mostly through pretraining.
Pretraining scaling laws#
Scaling laws describe how pretraining loss falls as model size, training data, and compute grow. Kaplan et al. (2020) found that loss follows a power law in each of the three, with some trends spanning more than seven orders of magnitude, and that architectural details such as network width or depth matter little within a wide range.
Hoffmann et al. (2022) changed how labs spend a fixed compute budget. After training more than 400 models, they concluded that current large models were significantly undertrained, and that parameters and training tokens should grow in equal proportion: each doubling of model size calls for doubling the training tokens. Their 70-billion-parameter Chinchilla used the same compute as the 280-billion-parameter Gopher with four times more data, and outperformed Gopher and GPT-3 on a large range of evaluation tasks.
Pretraining vs fine-tuning and post-training#
Pretraining builds general capability from unlabeled text, while post-training and fine-tuning shape behavior with curated data. Post-training, run by the model developer, turns a base model into an assistant through supervised fine-tuning, preference tuning such as RLHF, and reinforcement learning on checkable problems. Fine-tuning adapts the result to a narrower task.
| Stage | Training data | Usually run by | Result |
|---|---|---|---|
| Pretraining | Large unlabeled corpus of text and code | Model developers | A base model that continues text |
| Post-training | Instruction examples, preference rankings, problems with checkable answers | Model developers | An assistant or reasoning model |
| Fine-tuning | A team's own task examples | Anyone with the weights or a tuning API | A model specialized for one task or format |
Pretraining decides what a model knows and how capable it can become. Post-training and fine-tuning decide how it behaves, and retrieval supplies facts that arrived after the knowledge cutoff.
How reasoning models work shows how post-training on top of a pretrained base produces models that reason before they answer.
Related terms
Guides that cover pretraining
Cite this definition
The AI Engineering Editors. "What is pretraining in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/pretraining
- Title
- What is pretraining in LLMs?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated