---
title: "AI engineering glossary: LLM terms explained"
description: "Plain-language definitions of AI engineering and LLM terms: tokens, context windows, embeddings, RAG, MCP, AI gateways, quantization, evals and more."
type: "index"
canonical_url: "https://theaiengineering.com/glossary"
markdown_url: "https://theaiengineering.com/glossary.md"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
citation: "The AI Engineering Editors. \"AI engineering glossary: LLM terms explained.\" The AI Engineering. https://theaiengineering.com/glossary"
---

# AI engineering glossary: LLM terms explained

## A

- [A2A protocol](https://theaiengineering.com/glossary/a2a-protocol): The A2A (Agent2Agent) protocol is an open standard, introduced by Google in April 2025, that lets AI agents built on different frameworks discover each other, exchange messages, and delegate tasks.
- [Agent Card](https://theaiengineering.com/glossary/agent-card): An Agent Card is a JSON document that an A2A server publishes, usually at /.well-known/agent-card.json, describing the agent's identity, endpoints, capabilities, skills, and authentication requirements.
- [Agent handoff](https://theaiengineering.com/glossary/agent-handoff): An agent handoff is a transfer of control in which one AI agent passes the conversation, and responsibility for the task, to another agent better suited to the next part of the work.
- [Agent memory](https://theaiengineering.com/glossary/agent-memory): Agent memory is the set of mechanisms that let an AI agent keep and retrieve information beyond a single model call, from the current context window to external stores that persist across sessions.
- [Agent orchestration](https://theaiengineering.com/glossary/agent-orchestration): Agent orchestration is the control logic that decides which agent, model call, or tool runs next in an AI system and what each receives, whether written in code or chosen by a model.
- [Agent sandbox](https://theaiengineering.com/glossary/agent-sandbox): An agent sandbox is an isolated execution environment in which an AI agent runs code, shell commands, or a browser, with enforced limits on the files, network hosts, and credentials it can reach.
- [Agentic AI](https://theaiengineering.com/glossary/agentic-ai): Agentic AI is the class of AI systems that pursue a goal through multiple steps, using a language model to plan, call tools, and act on the results with limited human input.
- [Agentic RAG](https://theaiengineering.com/glossary/agentic-rag): Agentic RAG is a retrieval-augmented generation design in which an AI agent treats retrieval as a tool, deciding when to search, what query to send, which source to use, and whether to search again.
- [AGENTS.md](https://theaiengineering.com/glossary/agents-md): AGENTS.md is an open format for a plain Markdown file, placed at a repository's root and optionally in subdirectories, that gives AI coding agents project instructions such as build commands, test commands, and conventions.
- [AI agent](https://theaiengineering.com/glossary/ai-agent): An AI agent is a software system in which a large language model runs in a loop, calling tools, reading their results, and repeating until the task is complete or a stop condition is reached.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Approximate nearest neighbor search](https://theaiengineering.com/glossary/approximate-nearest-neighbor): Approximate nearest neighbor (ANN) search is a family of indexing methods that find vectors close to a query vector without comparing it to every stored vector, accepting that some true nearest neighbors are missed.

## B

- [Benchmark contamination](https://theaiengineering.com/glossary/benchmark-contamination): Benchmark contamination is the presence of a benchmark's test questions, answers, or solutions in a model's training data, so its score can reflect memorization instead of the capability the benchmark measures.
- [BM25](https://theaiengineering.com/glossary/bm25): BM25 is a keyword ranking function that scores a document against a query using how often each query term appears, how rare the term is across the corpus, and how long the document is.

## C

- [Chain-of-thought prompting](https://theaiengineering.com/glossary/chain-of-thought): Chain-of-thought prompting is a technique that has a language model write intermediate reasoning steps before its final answer, which improves large models' accuracy on arithmetic, commonsense, and symbolic reasoning tasks.
- [Chunking](https://theaiengineering.com/glossary/chunking): Chunking is the step in a retrieval pipeline that splits documents into smaller passages, called chunks, so each one can be embedded, indexed, and retrieved on its own.
- [Computer-use agent](https://theaiengineering.com/glossary/computer-use-agent): A computer-use agent is an AI agent that operates a graphical interface, such as a desktop or web browser, by reading screenshots and issuing mouse and keyboard actions instead of calling APIs.
- [Context compaction](https://theaiengineering.com/glossary/context-compaction): Context compaction is a context management technique that replaces older conversation turns with a model-written summary as the context window nears its limit, so an agent can keep working.
- [Context engineering](https://theaiengineering.com/glossary/context-engineering): Context engineering is the practice of selecting, structuring, and pruning all the information placed in an LLM's context window at each step, including instructions, retrieved data, tool results, and memory.
- [Context rot](https://theaiengineering.com/glossary/context-rot): Context rot is the decline in a language model's accuracy and recall as the number of tokens in its input grows, even when the information a task needs is present in the context.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Contextual retrieval](https://theaiengineering.com/glossary/contextual-retrieval): Contextual retrieval is a RAG indexing technique in which a language model writes a short note situating each chunk within its source document, and the note is prepended before embedding and BM25 indexing.
- [Continuous batching](https://theaiengineering.com/glossary/continuous-batching): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.
- [Cosine similarity](https://theaiengineering.com/glossary/cosine-similarity): Cosine similarity is a measure of how closely two vectors point in the same direction, computed as their dot product divided by the product of their lengths, with values from -1 to 1.

## D

- [Direct preference optimization](https://theaiengineering.com/glossary/direct-preference-optimization): Direct preference optimization (DPO) is a fine-tuning method that trains a language model on pairs of preferred and rejected responses with a simple classification loss, without the reward model and reinforcement learning that RLHF needs.

## E

- [Embeddings](https://theaiengineering.com/glossary/embeddings): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic.

## F

- [Faithfulness](https://theaiengineering.com/glossary/faithfulness): Faithfulness is an evaluation metric that measures what share of the claims in a generated answer can be inferred from the context the model was given, usually scored from 0 to 1.
- [Few-shot prompting](https://theaiengineering.com/glossary/few-shot-prompting): Few-shot prompting is a technique that puts a few example inputs paired with desired outputs in a prompt, so a language model infers the task, format, and labels without changing its weights.
- [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format.
- [Foundation model](https://theaiengineering.com/glossary/foundation-model): A foundation model is a large AI model trained on broad data, usually with self-supervised learning at scale, that can be adapted through prompting or fine-tuning to a wide range of downstream tasks.

## G

- [GGUF](https://theaiengineering.com/glossary/gguf): GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.

## H

- [Hallucination](https://theaiengineering.com/glossary/hallucination): A hallucination is output from a language model that is fluent and plausible but factually false, unsupported by the provided sources, or invented outright, such as a fabricated citation or package name.
- [Human-in-the-loop](https://theaiengineering.com/glossary/human-in-the-loop): Human-in-the-loop (HITL) is an agent design in which the run pauses at defined checkpoints, such as before a sensitive tool call, so a person can approve, edit, or reject the action.
- [Hybrid search](https://theaiengineering.com/glossary/hybrid-search): Hybrid search is a retrieval method that runs keyword search, typically BM25, and vector similarity search on the same query and merges the two ranked lists into one result set.

## I

- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.

## J

- [JSON Schema](https://theaiengineering.com/glossary/json-schema): JSON Schema is a declarative vocabulary, itself written in JSON, for describing the structure of JSON data; LLM APIs and MCP use it to define tool arguments and constrain model output.
- [JSON-RPC](https://theaiengineering.com/glossary/json-rpc): JSON-RPC is a lightweight, transport-agnostic remote procedure call protocol that encodes method calls, results, and errors as JSON objects; version 2.0 is the message format of MCP and A2A.

## K

- [Knowledge distillation](https://theaiengineering.com/glossary/knowledge-distillation): Knowledge distillation is a model compression technique in which a smaller student model is trained to reproduce the outputs or output probabilities of a larger, more capable teacher model.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.

## L

- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Late chunking](https://theaiengineering.com/glossary/late-chunking): Late chunking is an embedding technique that runs a long-context embedding model over a whole document first, then splits the token embeddings at chunk boundaries and averages each span into one chunk vector.
- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.
- [LLM observability](https://theaiengineering.com/glossary/llm-observability): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.
- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [LLM streaming](https://theaiengineering.com/glossary/llm-streaming): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [LMArena](https://theaiengineering.com/glossary/lmarena): LMArena, originally Chatbot Arena and branded Arena since January 2026, is a public leaderboard that ranks language models from crowdsourced human votes between pairs of anonymous model responses.
- [LoRA](https://theaiengineering.com/glossary/lora): LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters and memory use.

## M

- [MCP gateway](https://theaiengineering.com/glossary/mcp-gateway): An MCP gateway is a proxy that sits between MCP hosts and many MCP servers, exposing them through one endpoint and centralizing authentication, tool access policy, and audit logging.
- [MCP server](https://theaiengineering.com/glossary/mcp-server): An MCP server is a program that exposes tools, resources, and prompts to AI applications through the Model Context Protocol, running locally over stdio or remotely over Streamable HTTP.
- [MCP transport](https://theaiengineering.com/glossary/mcp-transport): An MCP transport is the binding that carries Model Context Protocol messages between a client and a server; the specification defines stdio for local subprocesses and Streamable HTTP for remote servers.
- [Mixture of experts](https://theaiengineering.com/glossary/mixture-of-experts): Mixture of experts (MoE) is a neural network architecture in which a learned router sends each token to a few specialized subnetworks, called experts, so only part of the model runs per token.
- [MMLU](https://theaiengineering.com/glossary/mmlu): MMLU (Massive Multitask Language Understanding) is a benchmark of 15,908 four-option multiple-choice questions across 57 subjects, from elementary mathematics to law, that measures a language model's broad academic knowledge.
- [Model cascade](https://theaiengineering.com/glossary/model-cascade): A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails.
- [Model Context Protocol (MCP)](https://theaiengineering.com/glossary/model-context-protocol): Model Context Protocol (MCP) is an open protocol, introduced by Anthropic in November 2024, that standardizes how AI applications connect to external tools, data sources, and prompts through MCP servers.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [Multi-agent system](https://theaiengineering.com/glossary/multi-agent-system): A multi-agent system is an AI architecture in which several language-model agents, each with its own prompt, tools, and context, work on parts of a task and pass results to one another.
- [Multimodal model](https://theaiengineering.com/glossary/multimodal-model): A multimodal model is an AI model that takes in or produces more than one type of data, such as text, images, audio, or video, and combines them within a single model.

## O

- [Online evaluation](https://theaiengineering.com/glossary/online-evaluation): Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.
- [Open-weights model](https://theaiengineering.com/glossary/open-weights-model): An open-weights model is an AI model whose trained parameters are published for download, so anyone can run, fine-tune, or quantize it on their own hardware under the terms of its license.
- [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api): An OpenAI-compatible API is an HTTP API that accepts OpenAI's request and response formats, so code written for OpenAI's SDKs can call another server by changing the base URL, API key, and model name.
- [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry): OpenTelemetry is an open source, vendor-neutral observability framework of APIs, SDKs, a wire protocol, and naming conventions for traces, metrics, and logs, including conventions for LLM calls and agent runs.

## P

- [PagedAttention](https://theaiengineering.com/glossary/paged-attention): PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging.
- [pass@k](https://theaiengineering.com/glossary/pass-at-k): pass@k is an evaluation metric that estimates the probability that at least one of k sampled outputs for a task passes its tests, used mainly in code generation and agent benchmarks.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
- [Pretraining](https://theaiengineering.com/glossary/pretraining): Pretraining is the first stage of training a language model, in which it learns to predict the next token across a large unlabeled corpus of text and code, producing a general-purpose base model.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Prompt engineering](https://theaiengineering.com/glossary/prompt-engineering): Prompt engineering is the practice of designing the instructions, examples, and formatting given to a large language model so it produces accurate, consistent output for a task, without changing model weights.
- [Prompt injection](https://theaiengineering.com/glossary/prompt-injection): Prompt injection is an attack in which text supplied to a large language model, typed by a user or hidden in content the model reads, alters the application's intended behavior or output.

## Q

- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.

## R

- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [ReAct agent](https://theaiengineering.com/glossary/react-agent): A ReAct agent is an AI agent that alternates a written reasoning step, an action such as a tool call, and an observation of the result, repeating the cycle until it can answer.
- [Reasoning model](https://theaiengineering.com/glossary/reasoning-model): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems.
- [Recall@k](https://theaiengineering.com/glossary/recall-at-k): Recall@k is a retrieval metric that measures the fraction of a query's relevant documents that appear in the top k results, averaged over a set of test queries.
- [Reranking](https://theaiengineering.com/glossary/reranking): Reranking is a second retrieval stage that rescores an initial list of candidate documents with a more accurate but slower model, usually a cross-encoder, and keeps only the most relevant ones.
- [Retrieval-augmented generation (RAG)](https://theaiengineering.com/glossary/retrieval-augmented-generation): Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from an external knowledge source at query time and adds them to a language model's prompt so the answer is grounded in that data.
- [RLHF](https://theaiengineering.com/glossary/rlhf): RLHF (reinforcement learning from human feedback) is a fine-tuning method that trains a reward model on human rankings of model outputs, then optimizes the language model with reinforcement learning to score highly.

## S

- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Small language model](https://theaiengineering.com/glossary/small-language-model): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
- [Speculative decoding](https://theaiengineering.com/glossary/speculative-decoding): Speculative decoding is an inference technique where a cheap draft proposes a few tokens and the target model verifies them in one forward pass, emitting several tokens per step without changing the output distribution.
- [Structured outputs](https://theaiengineering.com/glossary/structured-outputs): Structured outputs is an LLM API feature that constrains generation so the model's response always conforms to a developer-supplied JSON Schema, producing output that code can parse without retries.
- [Subagent](https://theaiengineering.com/glossary/subagent): A subagent is an AI agent that another agent starts for one subtask, running in a separate context window with its own instructions and tools, and returning only a condensed result.
- [SWE-bench](https://theaiengineering.com/glossary/swe-bench): SWE-bench is a benchmark of 2,294 real GitHub issues from 12 Python repositories, in which a model or agent must write a patch that resolves the issue and passes the repository's tests.
- [System prompt](https://theaiengineering.com/glossary/system-prompt): A system prompt is the set of developer-written instructions sent to a language model ahead of the conversation, defining its role, rules, output format, and tool use for every turn that follows.

## T

- [Temperature](https://theaiengineering.com/glossary/temperature): Temperature is a sampling parameter that rescales a language model's next-token probabilities: low values concentrate probability on the likeliest tokens, and high values flatten the distribution for more varied output.
- [Tensor parallelism](https://theaiengineering.com/glossary/tensor-parallelism): Tensor parallelism is a model parallelism technique that splits the weight matrices inside each transformer layer across several GPUs, which compute every token together and combine partial results through all-reduce operations.
- [Test-time compute](https://theaiengineering.com/glossary/test-time-compute): Test-time compute is the computation a language model spends while answering a query, through longer reasoning, multiple samples, or search, as opposed to the train-time compute spent building the model.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Tool calling](https://theaiengineering.com/glossary/tool-calling): Tool calling is an LLM capability in which the model, given a list of tool definitions, outputs a structured request naming a tool and its arguments, which the application executes and returns.
- [Tool poisoning](https://theaiengineering.com/glossary/tool-poisoning): Tool poisoning is an attack in which a malicious or compromised MCP server hides instructions in a tool's description or schema, which the model reads as trusted context and may follow.
- [Transformer](https://theaiengineering.com/glossary/transformer): A transformer is a neural network architecture that processes sequences with self-attention, letting each token weigh information from other tokens in parallel; it is the basis of most modern large language models.

## V

- [Vector database](https://theaiengineering.com/glossary/vector-database): A vector database is a database built to store embedding vectors and quickly return the vectors most similar to a query vector, usually with approximate nearest neighbor indexes such as HNSW.
- [Virtual key](https://theaiengineering.com/glossary/virtual-key): A virtual key is an API key issued by an AI gateway that maps a caller to its allowed models, budget, and rate limits, while the gateway keeps the real provider credentials.

## Cite this page

The AI Engineering Editors. "AI engineering glossary: LLM terms explained." The AI Engineering. https://theaiengineering.com/glossary

- Title: AI engineering glossary: LLM terms explained
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- URL: https://theaiengineering.com/glossary
- Markdown: https://theaiengineering.com/glossary.md
