---
title: "What is tensor parallelism in LLM inference?"
description: "Tensor parallelism splits each layer's weight matrices across GPUs that compute every token together. How it works and how it differs from pipeline parallelism."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/tensor-parallelism"
markdown_url: "https://theaiengineering.com/glossary/tensor-parallelism.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Tensor parallelism","TP","intra-layer model parallelism"]
citation: "The AI Engineering Editors. \"What is tensor parallelism in LLM inference?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/tensor-parallelism"
---

# What is tensor parallelism in LLM inference?

Also called TP, intra-layer model parallelism.

> **Definition of Tensor parallelism:** Tensor parallelism is a model parallelism technique that splits the weight matrices inside each transformer layer across several GPUs, which compute every token together and combine partial results through all-reduce operations.

Tensor parallelism is a way to run one model across several GPUs by splitting the weight matrices inside each layer, so every GPU holds a slice of every layer and works on every token. Teams use it for inference when the weights plus the [KV cache](https://theaiengineering.com/glossary/kv-cache) do not fit in one GPU's memory. Because each GPU stores and reads only its slice, the memory traffic of every decode step is spread across several devices. Its cost is communication. The GPUs exchange partial results inside every layer, so tensor parallelism needs a fast interconnect between them.

## How tensor parallelism works

Tensor parallelism partitions each layer's matrix multiplications so that every GPU computes part of the output, then combines the parts with an all-reduce. In the Megatron-LM layout, the attention heads and the first feed-forward matrix are split by columns, and the matrices that follow them are split by rows, so each block needs one all-reduce in the forward pass.

[Shoeybi et al.](https://arxiv.org/abs/1909.08053) introduced this layout in Megatron-LM to train [transformers](https://theaiengineering.com/glossary/transformer) with billions of parameters, implemented with a few communication operations in native PyTorch and no new compiler. In the feed-forward block, the first matrix multiplication is split by columns so that each GPU can apply the GeLU activation to its own partition independently. The second is split by rows, so it consumes those partitions directly. In self-attention, the query, key, and value projections are split by columns, so each GPU computes its own set of attention heads, and the output projection is split by rows.

Each transformer layer then needs two all-reduces in the forward pass and two in the backward pass. Inference runs only the forward pass, so every generated token triggers two all-reduces per layer, and a model with dozens of layers synchronizes its GPUs many times per token. That is why tensor parallelism works best among GPUs in one server joined by a high-bandwidth link such as NVLink.

## Tensor parallelism vs pipeline parallelism

Tensor parallelism splits every layer across GPUs, while pipeline parallelism assigns groups of consecutive layers to different GPUs. Tensor parallelism communicates inside every layer and suits GPUs in one node with a fast interconnect. Pipeline parallelism passes activations only between stages, so it tolerates slower links and suits multi-node deployments.

| | Tensor parallelism | Pipeline parallelism |
|---|---|---|
| What is split | The matrices inside each layer | The model's layers, in consecutive groups |
| Communication | All-reduce inside every layer | Activations passed between stages |
| Interconnect needs | High; best within one node | Lower; works across nodes |
| vLLM setting | `tensor_parallel_size` | `pipeline_parallel_size` |

[vLLM's parallelism guide](https://docs.vllm.ai/en/latest/serving/parallelism_scaling/) gives a simple decision path:

- If the model fits on one GPU, distributed inference is probably unnecessary.
- If it fits on one node, set `tensor_parallel_size` to the number of GPUs in the node.
- If it needs several nodes, set `tensor_parallel_size` to the GPUs per node and `pipeline_parallel_size` to the number of nodes.
- On GPUs without NVLink, such as L40S, prefer pipeline parallelism for higher throughput and lower communication overhead.

Pipeline parallelism also supports uneven splits, which helps when the layer count does not divide evenly across GPUs. Engines add other strategies on top: data parallelism runs independent replicas for throughput, and [mixture-of-experts](https://theaiengineering.com/glossary/mixture-of-experts) models use expert parallelism to place different experts on different GPUs.

The [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization) covers when to scale out with parallelism and which cheaper techniques to try first.

## Related terms

- [Mixture of experts](https://theaiengineering.com/glossary/mixture-of-experts): Mixture of experts (MoE) is a neural network architecture in which a learned router sends each token to a few specialized subnetworks, called experts, so only part of the model runs per token.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [Transformer](https://theaiengineering.com/glossary/transformer): A transformer is a neural network architecture that processes sequences with self-attention, letting each token weigh information from other tokens in parallel; it is the basis of most modern large language models.

## Guides that cover tensor parallelism

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is tensor parallelism in LLM inference?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/tensor-parallelism

- Title: What is tensor parallelism in LLM inference?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/tensor-parallelism
- Markdown: https://theaiengineering.com/glossary/tensor-parallelism.md
