What is tensor parallelism in LLM inference?
Also called TP, intra-layer model parallelism
Definition
Tensor parallelism is a model parallelism technique that splits the weight matrices inside each transformer layer across several GPUs, which compute every token together and combine partial results through all-reduce operations.
Tensor parallelism is a way to run one model across several GPUs by splitting the weight matrices inside each layer, so every GPU holds a slice of every layer and works on every token. Teams use it for inference when the weights plus the KV cache do not fit in one GPU's memory. Because each GPU stores and reads only its slice, the memory traffic of every decode step is spread across several devices. Its cost is communication. The GPUs exchange partial results inside every layer, so tensor parallelism needs a fast interconnect between them.
How tensor parallelism works#
Tensor parallelism partitions each layer's matrix multiplications so that every GPU computes part of the output, then combines the parts with an all-reduce. In the Megatron-LM layout, the attention heads and the first feed-forward matrix are split by columns, and the matrices that follow them are split by rows, so each block needs one all-reduce in the forward pass.
Shoeybi et al. introduced this layout in Megatron-LM to train transformers with billions of parameters, implemented with a few communication operations in native PyTorch and no new compiler. In the feed-forward block, the first matrix multiplication is split by columns so that each GPU can apply the GeLU activation to its own partition independently. The second is split by rows, so it consumes those partitions directly. In self-attention, the query, key, and value projections are split by columns, so each GPU computes its own set of attention heads, and the output projection is split by rows.
Each transformer layer then needs two all-reduces in the forward pass and two in the backward pass. Inference runs only the forward pass, so every generated token triggers two all-reduces per layer, and a model with dozens of layers synchronizes its GPUs many times per token. That is why tensor parallelism works best among GPUs in one server joined by a high-bandwidth link such as NVLink.
Tensor parallelism vs pipeline parallelism#
Tensor parallelism splits every layer across GPUs, while pipeline parallelism assigns groups of consecutive layers to different GPUs. Tensor parallelism communicates inside every layer and suits GPUs in one node with a fast interconnect. Pipeline parallelism passes activations only between stages, so it tolerates slower links and suits multi-node deployments.
| Tensor parallelism | Pipeline parallelism | |
|---|---|---|
| What is split | The matrices inside each layer | The model's layers, in consecutive groups |
| Communication | All-reduce inside every layer | Activations passed between stages |
| Interconnect needs | High; best within one node | Lower; works across nodes |
| vLLM setting | tensor_parallel_size |
pipeline_parallel_size |
vLLM's parallelism guide gives a simple decision path:
- If the model fits on one GPU, distributed inference is probably unnecessary.
- If it fits on one node, set
tensor_parallel_sizeto the number of GPUs in the node. - If it needs several nodes, set
tensor_parallel_sizeto the GPUs per node andpipeline_parallel_sizeto the number of nodes. - On GPUs without NVLink, such as L40S, prefer pipeline parallelism for higher throughput and lower communication overhead.
Pipeline parallelism also supports uneven splits, which helps when the layer count does not divide evenly across GPUs. Engines add other strategies on top: data parallelism runs independent replicas for throughput, and mixture-of-experts models use expert parallelism to place different experts on different GPUs.
The LLM inference optimization guide covers when to scale out with parallelism and which cheaper techniques to try first.
Related terms
Guides that cover tensor parallelism
Cite this definition
The AI Engineering Editors. "What is tensor parallelism in LLM inference?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/tensor-parallelism
- Title
- What is tensor parallelism in LLM inference?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated