---
title: "What is LoRA (low-rank adaptation) in AI?"
description: "LoRA (low-rank adaptation) fine-tunes a model by training small low-rank matrices while its base weights stay frozen. How LoRA works, and LoRA vs QLoRA."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/lora"
markdown_url: "https://theaiengineering.com/glossary/lora.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["LoRA","low-rank adaptation","LoRA adapter","QLoRA"]
citation: "The AI Engineering Editors. \"What is LoRA (low-rank adaptation) in AI?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/lora"
---

# What is LoRA (low-rank adaptation) in AI?

Also called low-rank adaptation, LoRA adapter, QLoRA.

> **Definition of LoRA:** LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters and memory use.

LoRA, short for low-rank adaptation, is a technique for [fine-tuning](https://theaiengineering.com/glossary/fine-tuning) large models at a fraction of the usual memory and storage cost. Instead of updating a weight matrix directly, LoRA freezes it and learns the change as the product of two much smaller matrices. Edward Hu and colleagues at Microsoft introduced the method in the 2021 paper [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685). LoRA is also widely used to adapt image generation models; Hugging Face's diffusers library, for example, supports LoRA training for Stable Diffusion.

## How LoRA works

For a frozen pretrained weight matrix W₀ of size d × k, LoRA represents the update as ΔW = BA, where B is d × r, A is r × k, and the rank r is much smaller than d or k. Only A and B receive gradients during training. B is initialized to zero, so training starts from the unmodified model.

**Figure 1: LoRA adds a trainable low-rank path beside a frozen weight.**

Input x flows through the frozen weight W₀ and, in parallel, through trainable low-rank matrices A then B; both paths add to give the output h.

Components:

- **Input x**
- **Frozen W₀**: Pretrained weight, d × k
- **A**: r × k
- **B**: d × r, starts at zero
- **Output h**: h = W₀x + BAx

Groups:

- Trainable, rank r: A, B

Connections:

- Input x → Frozen W₀
- Frozen W₀ → Output h
- Input x → A
- A → B
- B → Output h

The paper's results explain the method's popularity. For GPT-3 175B, LoRA reduced trainable parameters by 10,000 times and GPU memory by 3 times compared with full fine-tuning using Adam. With rank 4 applied to the attention query and value projections of the [transformer](https://theaiengineering.com/glossary/transformer), the checkpoint shrank from about 350 GB to 35 MB. After training, BA can be merged into W₀, so LoRA adds no inference latency. Serving engines such as vLLM can also keep adapters separate and serve several of them on one base model.

## LoRA vs QLoRA

[QLoRA](https://arxiv.org/abs/2305.14314) (Dettmers et al., 2023) trains LoRA adapters on top of a base model [quantized](https://theaiengineering.com/glossary/quantization) to 4 bits. The QLoRA paper introduced the 4-bit NormalFloat data type, double quantization, and paged optimizers, and made it possible to fine-tune a 65B-parameter model on a single 48 GB GPU. Standard LoRA keeps the frozen base weights in 16-bit precision.

## LoRA vs full fine-tuning

Full fine-tuning updates every weight and stores a complete model copy per task. LoRA stores only the adapter, so one base model can host many task-specific adapters. The original paper reports quality on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3. Later work found limits: [LoRA Learns Less and Forgets Less](https://arxiv.org/abs/2405.09673) (Biderman et al., 2024) reports that LoRA substantially underperforms full fine-tuning on code and math in standard low-rank settings, while preserving more of the base model's performance outside the target domain. Compare both on your own evaluation set when quality matters, and see [RAG vs fine-tuning](https://theaiengineering.com/articles/rag-vs-fine-tuning) for when training is the right tool at all.

## Related terms

- [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format.
- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- [Transformer](https://theaiengineering.com/glossary/transformer): A transformer is a neural network architecture that processes sequences with self-attention, letting each token weigh information from other tokens in parallel; it is the basis of most modern large language models.
- [RLHF](https://theaiengineering.com/glossary/rlhf): RLHF (reinforcement learning from human feedback) is a fine-tuning method that trains a reward model on human rankings of model outputs, then optimizes the language model with reinforcement learning to score highly.

## Guides that cover lora

- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is LoRA (low-rank adaptation) in AI?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/lora

- Title: What is LoRA (low-rank adaptation) in AI?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/lora
- Markdown: https://theaiengineering.com/glossary/lora.md
