What is LoRA (low-rank adaptation) in AI?
Also called low-rank adaptation, LoRA adapter, QLoRA
Definition
LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters and memory use.
LoRA, short for low-rank adaptation, is a technique for fine-tuning large models at a fraction of the usual memory and storage cost. Instead of updating a weight matrix directly, LoRA freezes it and learns the change as the product of two much smaller matrices. Edward Hu and colleagues at Microsoft introduced the method in the 2021 paper LoRA: Low-Rank Adaptation of Large Language Models. LoRA is also widely used to adapt image generation models; Hugging Face's diffusers library, for example, supports LoRA training for Stable Diffusion.
How LoRA works#
For a frozen pretrained weight matrix W₀ of size d × k, LoRA represents the update as ΔW = BA, where B is d × r, A is r × k, and the rank r is much smaller than d or k. Only A and B receive gradients during training. B is initialized to zero, so training starts from the unmodified model.
The paper's results explain the method's popularity. For GPT-3 175B, LoRA reduced trainable parameters by 10,000 times and GPU memory by 3 times compared with full fine-tuning using Adam. With rank 4 applied to the attention query and value projections of the transformer, the checkpoint shrank from about 350 GB to 35 MB. After training, BA can be merged into W₀, so LoRA adds no inference latency. Serving engines such as vLLM can also keep adapters separate and serve several of them on one base model.
LoRA vs QLoRA#
QLoRA (Dettmers et al., 2023) trains LoRA adapters on top of a base model quantized to 4 bits. The QLoRA paper introduced the 4-bit NormalFloat data type, double quantization, and paged optimizers, and made it possible to fine-tune a 65B-parameter model on a single 48 GB GPU. Standard LoRA keeps the frozen base weights in 16-bit precision.
LoRA vs full fine-tuning#
Full fine-tuning updates every weight and stores a complete model copy per task. LoRA stores only the adapter, so one base model can host many task-specific adapters. The original paper reports quality on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3. Later work found limits: LoRA Learns Less and Forgets Less (Biderman et al., 2024) reports that LoRA substantially underperforms full fine-tuning on code and math in standard low-rank settings, while preserving more of the base model's performance outside the target domain. Compare both on your own evaluation set when quality matters, and see RAG vs fine-tuning for when training is the right tool at all.
Related terms
Guides that cover lora
Cite this definition
The AI Engineering Editors. "What is LoRA (low-rank adaptation) in AI?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/lora
- Title
- What is LoRA (low-rank adaptation) in AI?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated