---
title: "What is a multimodal model? Vision-language models"
description: "A multimodal model processes more than one type of data, such as text and images. How vision-language models work, what images cost in tokens, and their limits."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/multimodal-model"
markdown_url: "https://theaiengineering.com/glossary/multimodal-model.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Multimodal model","multimodal LLM","vision-language model","VLM","large multimodal model"]
citation: "The AI Engineering Editors. \"What is a multimodal model? Vision-language models.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/multimodal-model"
---

# What is a multimodal model? Vision-language models

Also called multimodal LLM, vision-language model, VLM, large multimodal model.

> **Definition of Multimodal model:** A multimodal model is an AI model that takes in or produces more than one type of data, such as text, images, audio, or video, and combines them within a single model.

A multimodal model is a model that accepts or generates more than one kind of data, such as text, images, audio, or video. The most common type in AI engineering is the vision-language model, which takes images and text as input and returns text. Applications use vision-language models to read scanned documents, charts, and screenshots, and [computer-use agents](https://theaiengineering.com/glossary/computer-use-agent) rely on them to see the screen they operate. Many current [large language models](https://theaiengineering.com/glossary/large-language-model) accept image input, so multimodal support is often a property of a model a team already uses.

## How multimodal models work

Most vision-language models pair a vision encoder with a language model. The encoder turns an image into a sequence of feature vectors, a projection layer maps those vectors into the language model's embedding space, and the language model attends over the image tokens and the text tokens together, generating text one token at a time.

Two papers show how the pattern developed. [CLIP](https://arxiv.org/abs/2103.00020) (Radford et al., 2021) trained an image encoder and a text encoder on 400 million image and text pairs from the internet, with the task of predicting which caption goes with which image. The resulting [embeddings](https://theaiengineering.com/glossary/embeddings) let natural language describe visual concepts, and the authors matched the accuracy of the original ResNet-50 on ImageNet zero-shot, without using any of its 1.28 million training examples.

[LLaVA](https://arxiv.org/abs/2304.08485) (Liu et al., 2023) connected CLIP's ViT-L/14 vision encoder to the Vicuna language model through a single linear projection. It trained in two stages: first only the projection, to align image features with the word embeddings of the frozen language model, then the projection and the language model together on instruction-following data generated with GPT-4.

## Image input: tokens, cost, and limits

An image in a multimodal request becomes a sequence of [tokens](https://theaiengineering.com/glossary/tokens) and is billed as input, so image size drives both cost and latency. Anthropic's [vision documentation](https://platform.claude.com/docs/en/build-with-claude/vision) counts one visual token per 28 by 28 pixel patch, so a 1000 by 1000 pixel image costs 1,296 tokens; larger images are downscaled to each model's limit.

OpenAI's [image input guide](https://developers.openai.com/api/docs/guides/images-vision) accepts images as URLs, base64 data URLs, or file IDs, and a `detail` setting controls how much resolution the model receives. Anthropic reports that its models work best when images come before the text that refers to them.

Both providers document similar weak spots:

- Object counts are approximate, especially for many small objects.
- Precise spatial localization is unreliable, so verify coordinates before acting on them.
- Specialized medical scans such as CT images are out of scope.

Resize and crop images before sending them, so the model spends tokens on the region that matters. The [release tracker](https://theaiengineering.com/releases) lists the input and output modalities of each model release.

[LLM benchmarks](https://theaiengineering.com/articles/llm-benchmarks) covers how models are compared, including benchmarks with image-based questions, and [vLLM vs Ollama](https://theaiengineering.com/articles/vllm-vs-ollama) covers running multimodal models on your own hardware.

## Related terms

- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Embeddings](https://theaiengineering.com/glossary/embeddings): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic.
- [Foundation model](https://theaiengineering.com/glossary/foundation-model): A foundation model is a large AI model trained on broad data, usually with self-supervised learning at scale, that can be adapted through prompting or fine-tuning to a wide range of downstream tasks.
- [Computer-use agent](https://theaiengineering.com/glossary/computer-use-agent): A computer-use agent is an AI agent that operates a graphical interface, such as a desktop or web browser, by reading screenshots and issuing mouse and keyboard actions instead of calling APIs.
- [Transformer](https://theaiengineering.com/glossary/transformer): A transformer is a neural network architecture that processes sequences with self-attention, letting each token weigh information from other tokens in parallel; it is the basis of most modern large language models.

## Guides that cover multimodal model

- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is a multimodal model? Vision-language models." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/multimodal-model

- Title: What is a multimodal model? Vision-language models
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/multimodal-model
- Markdown: https://theaiengineering.com/glossary/multimodal-model.md
