What is a multimodal model? Vision-language models

Also called multimodal LLM, vision-language model, VLM, large multimodal model

Definition

A multimodal model is an AI model that takes in or produces more than one type of data, such as text, images, audio, or video, and combines them within a single model.

A multimodal model is a model that accepts or generates more than one kind of data, such as text, images, audio, or video. The most common type in AI engineering is the vision-language model, which takes images and text as input and returns text. Applications use vision-language models to read scanned documents, charts, and screenshots, and computer-use agents rely on them to see the screen they operate. Many current large language models accept image input, so multimodal support is often a property of a model a team already uses.

How multimodal models work#

Most vision-language models pair a vision encoder with a language model. The encoder turns an image into a sequence of feature vectors, a projection layer maps those vectors into the language model's embedding space, and the language model attends over the image tokens and the text tokens together, generating text one token at a time.

Two papers show how the pattern developed. CLIP (Radford et al., 2021) trained an image encoder and a text encoder on 400 million image and text pairs from the internet, with the task of predicting which caption goes with which image. The resulting embeddings let natural language describe visual concepts, and the authors matched the accuracy of the original ResNet-50 on ImageNet zero-shot, without using any of its 1.28 million training examples.

LLaVA (Liu et al., 2023) connected CLIP's ViT-L/14 vision encoder to the Vicuna language model through a single linear projection. It trained in two stages: first only the projection, to align image features with the word embeddings of the frozen language model, then the projection and the language model together on instruction-following data generated with GPT-4.

Image input: tokens, cost, and limits#

An image in a multimodal request becomes a sequence of tokens and is billed as input, so image size drives both cost and latency. Anthropic's vision documentation counts one visual token per 28 by 28 pixel patch, so a 1000 by 1000 pixel image costs 1,296 tokens; larger images are downscaled to each model's limit.

OpenAI's image input guide accepts images as URLs, base64 data URLs, or file IDs, and a detail setting controls how much resolution the model receives. Anthropic reports that its models work best when images come before the text that refers to them.

Both providers document similar weak spots:

  • Object counts are approximate, especially for many small objects.
  • Precise spatial localization is unreliable, so verify coordinates before acting on them.
  • Specialized medical scans such as CT images are out of scope.

Resize and crop images before sending them, so the model spends tokens on the region that matters. The release tracker lists the input and output modalities of each model release.

LLM benchmarks covers how models are compared, including benchmarks with image-based questions, and vLLM vs Ollama covers running multimodal models on your own hardware.

Cite this definition

The AI Engineering Editors. "What is a multimodal model? Vision-language models." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/multimodal-model

Title
What is a multimodal model? Vision-language models
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM models and benchmarks hub · Updated