# The AI Engineering > Guides, explainers and a living reference for engineers building with LLMs: AI agents, MCP, RAG, evals, AI gateways, inference and model releases. Source-linked guides for engineers building with large language models. Every guide opens with a TL;DR and ends with an FAQ. Every page has a Markdown version at its URL plus .md, and the home page is at /index.md. Each Markdown file starts with YAML front matter holding the page's canonical URL, dates, and a ready-made citation, and ends with a section on how to cite it. When citing, link to the canonical URL rather than the .md file. ## Topic hubs - [AI engineering](https://theaiengineering.com/topics/ai-engineering.md): What AI engineers build, the stack they build it on, and how to learn it. - [AI agents](https://theaiengineering.com/topics/ai-agents.md): Models that plan, call tools and act, and the frameworks used to build them. - [Model Context Protocol](https://theaiengineering.com/topics/mcp.md): The open protocol that connects models to tools and data, and how it fits with A2A. - [RAG and vector search](https://theaiengineering.com/topics/rag.md): Grounding model output in your own data: pipelines, vector search and trade-offs. - [LLM evaluation and observability](https://theaiengineering.com/topics/llm-evaluation.md): Measuring whether LLM systems work, before and after they ship. - [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways.md): The control layer between applications and models: routing, keys, caching and policy. - [LLM inference and serving](https://theaiengineering.com/topics/llm-inference.md): Serving models fast and cheaply: engines, memory, batching and hardware. - [LLM models and benchmarks](https://theaiengineering.com/topics/models.md): What new models change, how to read benchmarks, and a dated release record. - [AI coding agents](https://theaiengineering.com/topics/ai-coding.md): Agents that read, write and test code, and how teams put them to work. - [Prompt and context engineering](https://theaiengineering.com/topics/prompt-engineering.md): Deciding what the model sees: prompts, context, tools and structured output. ## Guides - [Best LLM for coding: models compared on agentic coding benchmarks](https://theaiengineering.com/articles/best-llm-for-coding.md): The best LLM for coding depends on the job: 11 models compared on Terminal-Bench 4.0, DeepSWE, and SWE-bench Pro, plus context, open weights, and cost. - [Agentic workflows: five patterns and when to use each](https://theaiengineering.com/articles/agentic-workflows.md): Agentic workflows run LLM and tool calls through steps fixed in code. Learn five patterns, their cost and latency, failure modes, and when to use an agent. - [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm.md): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration. - [Claude Code vs Cursor: how the two coding agents differ](https://theaiengineering.com/articles/claude-code-vs-cursor.md): Claude Code vs Cursor compared: terminal agent vs AI editor, models, subagents, hooks, rules files, code review, CI use, pricing, and enterprise controls. - [Semantic caching for LLMs: how it works and when it pays off](https://theaiengineering.com/articles/semantic-caching.md): Semantic caching returns a stored LLM response for prompts with similar meaning. Learn how a semantic cache works, how to tune it, and what never to cache. - [How to build an MCP server: a Python and TypeScript tutorial](https://theaiengineering.com/articles/how-to-build-an-mcp-server.md): Learn how to build an MCP server with the official Python and TypeScript SDKs: tools, resources, transports, testing, client config, and production security. - [How to build an AI agent: a step-by-step guide in Python](https://theaiengineering.com/articles/how-to-build-an-ai-agent.md): Learn how to build an AI agent from scratch in Python: when you need one, tool schemas, the agent loop, stop conditions, guardrails, evals, and tracing. - [AI coding agents: how they work and how the major tools compare](https://theaiengineering.com/articles/ai-coding-agents.md): AI coding agents compared: how they differ from autocomplete, the four main categories, a comparison table of major tools, and how to evaluate and secure them. - [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama.md): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use. - [AI agent frameworks compared: LangGraph, CrewAI, ADK, and more](https://theaiengineering.com/articles/ai-agent-frameworks.md): An AI agent frameworks comparison of LangGraph, LangChain, CrewAI, the OpenAI, Claude, and Google SDKs, Pydantic AI, Mastra, and more, by language and license. - [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability.md): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private. - [AI agent evaluation: metrics, graders, and how to test agents](https://theaiengineering.com/articles/ai-agent-evaluation.md): AI agent evaluation guide: measure task success, tool use, safety, cost, and pass^k reliability with code graders, LLM judges, eval sets from traces, and CI. - [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization.md): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs. - [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering.md): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results. - [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing.md): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls. - [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway.md): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options. - [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models.md): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency. - [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks.md): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard. - [A2A protocol vs MCP: differences and when to use each](https://theaiengineering.com/articles/a2a-vs-mcp.md): A2A protocol vs MCP: A2A connects AI agents to other agents, and MCP connects an agent to tools and data. Compare agent cards, tasks, transports, and security. - [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation.md): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI. - [Chunking strategies for RAG: how to choose, tune, and evaluate them](https://theaiengineering.com/articles/chunking-strategies-for-rag.md): Chunking strategies for RAG compared: fixed-size, recursive, document-aware, semantic, parent-child, and late chunking, with tested Python and an eval method. - [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning.md): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids. - [What is tool calling? How LLM function calling works](https://theaiengineering.com/articles/tool-calling.md): Tool calling lets an LLM request actions from your code through structured calls. Learn the request loop, parallel calls, structured outputs, security, and MCP. - [What is the Model Context Protocol (MCP)? How it works](https://theaiengineering.com/articles/what-is-model-context-protocol.md): The Model Context Protocol (MCP) is an open standard for connecting AI apps to tools and data. Learn its architecture, transports, tool calls, and security. - [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap.md): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production. - [What are AI agents? How agents work and when to use one](https://theaiengineering.com/articles/what-are-ai-agents.md): AI agents are LLM systems that call tools in a loop until a task is done. Learn how the loop works, agents vs agentic AI, examples, failure modes, and limits. - [What is retrieval-augmented generation (RAG)? How RAG works](https://theaiengineering.com/articles/what-is-rag.md): Retrieval-augmented generation (RAG) grounds LLM answers in your documents. Learn the RAG pipeline, hybrid search, agentic RAG, evaluation, and failure modes. - [What is AI engineering? The discipline, the stack, and the work](https://theaiengineering.com/articles/what-is-ai-engineering.md): What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day. ## Reference - [AI model release tracker](https://theaiengineering.com/releases.md): dated record of significant model releases, linked to official announcements - [Release data as CSV](https://theaiengineering.com/releases.csv): every tracked release, free to reuse under CC BY 4.0 - [Release data as JSON](https://theaiengineering.com/releases.json): the same records with the citation and license - [Google model history](https://theaiengineering.com/releases/google.md): 106 releases - [Alibaba Qwen model history](https://theaiengineering.com/releases/alibaba-qwen.md): 84 releases - [OpenAI model history](https://theaiengineering.com/releases/openai.md): 79 releases - [Mistral AI model history](https://theaiengineering.com/releases/mistral.md): 59 releases - [Meta model history](https://theaiengineering.com/releases/meta.md): 44 releases - [Z.ai model history](https://theaiengineering.com/releases/z-ai.md): 43 releases - [Microsoft model history](https://theaiengineering.com/releases/microsoft.md): 42 releases - [Tencent model history](https://theaiengineering.com/releases/tencent.md): 42 releases - [ByteDance Seed model history](https://theaiengineering.com/releases/bytedance-seed.md): 40 releases - [DeepSeek model history](https://theaiengineering.com/releases/deepseek.md): 38 releases - [Liquid AI model history](https://theaiengineering.com/releases/liquid-ai.md): 34 releases - [Cohere model history](https://theaiengineering.com/releases/cohere.md): 31 releases - [IBM model history](https://theaiengineering.com/releases/ibm.md): 30 releases - [xAI model history](https://theaiengineering.com/releases/xai.md): 30 releases - [NVIDIA model history](https://theaiengineering.com/releases/nvidia.md): 29 releases - [MiniMax model history](https://theaiengineering.com/releases/minimax.md): 28 releases - [Stability AI model history](https://theaiengineering.com/releases/stability-ai.md): 28 releases - [Anthropic model history](https://theaiengineering.com/releases/anthropic.md): 26 releases - [Ai2 model history](https://theaiengineering.com/releases/ai2.md): 23 releases - [StepFun model history](https://theaiengineering.com/releases/stepfun.md): 22 releases - [Amazon model history](https://theaiengineering.com/releases/amazon.md): 21 releases - [Baidu model history](https://theaiengineering.com/releases/baidu.md): 21 releases - [ElevenLabs model history](https://theaiengineering.com/releases/elevenlabs.md): 21 releases - [Black Forest Labs model history](https://theaiengineering.com/releases/black-forest-labs.md): 18 releases - [Kuaishou model history](https://theaiengineering.com/releases/kuaishou.md): 16 releases - [Meituan model history](https://theaiengineering.com/releases/meituan.md): 16 releases - [Runway model history](https://theaiengineering.com/releases/runway.md): 16 releases - [Moonshot AI model history](https://theaiengineering.com/releases/moonshot-ai.md): 15 releases - [Midjourney model history](https://theaiengineering.com/releases/midjourney.md): 14 releases - [Salesforce model history](https://theaiengineering.com/releases/salesforce.md): 13 releases - [Xiaomi model history](https://theaiengineering.com/releases/xiaomi.md): 13 releases - [Nous Research model history](https://theaiengineering.com/releases/nous-research.md): 11 releases - [Perplexity model history](https://theaiengineering.com/releases/perplexity.md): 10 releases - [AI21 Labs model history](https://theaiengineering.com/releases/ai21.md): 8 releases - [Apple model history](https://theaiengineering.com/releases/apple.md): 8 releases - [Ideogram model history](https://theaiengineering.com/releases/ideogram.md): 8 releases - [01.AI model history](https://theaiengineering.com/releases/01-ai.md): 7 releases - [Hugging Face model history](https://theaiengineering.com/releases/hugging-face.md): 7 releases - [Lightricks model history](https://theaiengineering.com/releases/lightricks.md): 7 releases - [Luma AI model history](https://theaiengineering.com/releases/luma-ai.md): 7 releases - [Reka model history](https://theaiengineering.com/releases/reka.md): 7 releases - [Suno model history](https://theaiengineering.com/releases/suno.md): 7 releases - [Databricks model history](https://theaiengineering.com/releases/databricks.md): 5 releases - [Leonardo.Ai model history](https://theaiengineering.com/releases/leonardo-ai.md): 5 releases - [Pika model history](https://theaiengineering.com/releases/pika.md): 5 releases - [Recraft model history](https://theaiengineering.com/releases/recraft.md): 5 releases - [Inflection AI model history](https://theaiengineering.com/releases/inflection-ai.md): 3 releases - [MBZUAI IFM model history](https://theaiengineering.com/releases/mbzuai-ifm.md): 3 releases - [Snowflake model history](https://theaiengineering.com/releases/snowflake.md): 3 releases - [Thinking Machines Lab model history](https://theaiengineering.com/releases/thinking-machines-lab.md): 3 releases - [TypeSafe AI model history](https://theaiengineering.com/releases/typesafe-ai.md): 2 releases - [Cognition model history](https://theaiengineering.com/releases/cognition.md): 1 releases - [Genmo model history](https://theaiengineering.com/releases/genmo.md): 1 releases - [Reflection AI model history](https://theaiengineering.com/releases/reflection-ai.md): 1 releases - [Ricursive Intelligence model history](https://theaiengineering.com/releases/ricursive-intelligence.md): 1 releases - [AI models released in 2026](https://theaiengineering.com/releases/2026.md): 400 releases, with one page per month - [AI models released in 2025](https://theaiengineering.com/releases/2025.md): 422 releases, with one page per month - [AI models released in 2024](https://theaiengineering.com/releases/2024.md): 233 releases, with one page per month - [AI models released in 2023](https://theaiengineering.com/releases/2023.md): 112 releases, with one page per month - [Platform directory](https://theaiengineering.com/platforms.md): profiles of AI engineering platforms, linked to official sites - [Model providers](https://theaiengineering.com/platforms/model-providers.md) - [Inference providers](https://theaiengineering.com/platforms/inference-providers.md) - [GPU clouds](https://theaiengineering.com/platforms/gpu-clouds.md) - [Serving engines](https://theaiengineering.com/platforms/serving-engines.md) - [Vector databases](https://theaiengineering.com/platforms/vector-databases.md) - [Embedding and reranking models](https://theaiengineering.com/platforms/embedding-reranking.md) - [Document parsing and data ingestion](https://theaiengineering.com/platforms/data-ingestion.md) - [Fine-tuning and post-training](https://theaiengineering.com/platforms/fine-tuning.md) - [Agent frameworks](https://theaiengineering.com/platforms/agent-frameworks.md) - [Agent infrastructure](https://theaiengineering.com/platforms/agent-infrastructure.md) - [Web search APIs](https://theaiengineering.com/platforms/web-search-apis.md) - [MCP platforms](https://theaiengineering.com/platforms/mcp-platforms.md) - [Voice AI](https://theaiengineering.com/platforms/voice-ai.md) - [AI gateways](https://theaiengineering.com/platforms/ai-gateways.md) - [LLM observability and evaluation](https://theaiengineering.com/platforms/llm-observability.md) - [Guardrails and AI security](https://theaiengineering.com/platforms/guardrails-security.md) - [AI coding tools](https://theaiengineering.com/platforms/ai-coding-tools.md) - [Glossary](https://theaiengineering.com/glossary.md): 98 AI engineering terms - [Full text](https://theaiengineering.com/llms-full.txt): every guide, glossary term, platform category, and month of releases in one file ## Optional - [A2A protocol](https://theaiengineering.com/glossary/a2a-protocol.md): The A2A (Agent2Agent) protocol is an open standard, introduced by Google in April 2025, that lets AI agents built on different frameworks discover each other, exchange messages, and delegate tasks. - [Agent Card](https://theaiengineering.com/glossary/agent-card.md): An Agent Card is a JSON document that an A2A server publishes, usually at /.well-known/agent-card.json, describing the agent's identity, endpoints, capabilities, skills, and authentication requirements. - [Agent handoff](https://theaiengineering.com/glossary/agent-handoff.md): An agent handoff is a transfer of control in which one AI agent passes the conversation, and responsibility for the task, to another agent better suited to the next part of the work. - [Agent memory](https://theaiengineering.com/glossary/agent-memory.md): Agent memory is the set of mechanisms that let an AI agent keep and retrieve information beyond a single model call, from the current context window to external stores that persist across sessions. - [Agent orchestration](https://theaiengineering.com/glossary/agent-orchestration.md): Agent orchestration is the control logic that decides which agent, model call, or tool runs next in an AI system and what each receives, whether written in code or chosen by a model. - [Agent sandbox](https://theaiengineering.com/glossary/agent-sandbox.md): An agent sandbox is an isolated execution environment in which an AI agent runs code, shell commands, or a browser, with enforced limits on the files, network hosts, and credentials it can reach. - [Agentic AI](https://theaiengineering.com/glossary/agentic-ai.md): Agentic AI is the class of AI systems that pursue a goal through multiple steps, using a language model to plan, call tools, and act on the results with limited human input. - [Agentic RAG](https://theaiengineering.com/glossary/agentic-rag.md): Agentic RAG is a retrieval-augmented generation design in which an AI agent treats retrieval as a tool, deciding when to search, what query to send, which source to use, and whether to search again. - [AGENTS.md](https://theaiengineering.com/glossary/agents-md.md): AGENTS.md is an open format for a plain Markdown file, placed at a repository's root and optionally in subdirectories, that gives AI coding agents project instructions such as build commands, test commands, and conventions. - [AI agent](https://theaiengineering.com/glossary/ai-agent.md): An AI agent is a software system in which a large language model runs in a loop, calling tools, reading their results, and repeating until the task is complete or a stop condition is reached. - [AI gateway](https://theaiengineering.com/glossary/ai-gateway.md): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging. - [Approximate nearest neighbor search](https://theaiengineering.com/glossary/approximate-nearest-neighbor.md): Approximate nearest neighbor (ANN) search is a family of indexing methods that find vectors close to a query vector without comparing it to every stored vector, accepting that some true nearest neighbors are missed. - [Benchmark contamination](https://theaiengineering.com/glossary/benchmark-contamination.md): Benchmark contamination is the presence of a benchmark's test questions, answers, or solutions in a model's training data, so its score can reflect memorization instead of the capability the benchmark measures. - [BM25](https://theaiengineering.com/glossary/bm25.md): BM25 is a keyword ranking function that scores a document against a query using how often each query term appears, how rare the term is across the corpus, and how long the document is. - [Chain-of-thought prompting](https://theaiengineering.com/glossary/chain-of-thought.md): Chain-of-thought prompting is a technique that has a language model write intermediate reasoning steps before its final answer, which improves large models' accuracy on arithmetic, commonsense, and symbolic reasoning tasks. - [Chunking](https://theaiengineering.com/glossary/chunking.md): Chunking is the step in a retrieval pipeline that splits documents into smaller passages, called chunks, so each one can be embedded, indexed, and retrieved on its own. - [Computer-use agent](https://theaiengineering.com/glossary/computer-use-agent.md): A computer-use agent is an AI agent that operates a graphical interface, such as a desktop or web browser, by reading screenshots and issuing mouse and keyboard actions instead of calling APIs. - [Context compaction](https://theaiengineering.com/glossary/context-compaction.md): Context compaction is a context management technique that replaces older conversation turns with a model-written summary as the context window nears its limit, so an agent can keep working. - [Context engineering](https://theaiengineering.com/glossary/context-engineering.md): Context engineering is the practice of selecting, structuring, and pruning all the information placed in an LLM's context window at each step, including instructions, retrieved data, tool results, and memory. - [Context rot](https://theaiengineering.com/glossary/context-rot.md): Context rot is the decline in a language model's accuracy and recall as the number of tokens in its input grows, even when the information a task needs is present in the context. - [Context window](https://theaiengineering.com/glossary/context-window.md): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output. - [Contextual retrieval](https://theaiengineering.com/glossary/contextual-retrieval.md): Contextual retrieval is a RAG indexing technique in which a language model writes a short note situating each chunk within its source document, and the note is prepended before embedding and BM25 indexing. - [Continuous batching](https://theaiengineering.com/glossary/continuous-batching.md): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step. - [Cosine similarity](https://theaiengineering.com/glossary/cosine-similarity.md): Cosine similarity is a measure of how closely two vectors point in the same direction, computed as their dot product divided by the product of their lengths, with values from -1 to 1. - [Direct preference optimization](https://theaiengineering.com/glossary/direct-preference-optimization.md): Direct preference optimization (DPO) is a fine-tuning method that trains a language model on pairs of preferred and rejected responses with a simple classification loss, without the reward model and reinforcement learning that RLHF needs. - [Embeddings](https://theaiengineering.com/glossary/embeddings.md): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic. - [Faithfulness](https://theaiengineering.com/glossary/faithfulness.md): Faithfulness is an evaluation metric that measures what share of the claims in a generated answer can be inferred from the context the model was given, usually scored from 0 to 1. - [Few-shot prompting](https://theaiengineering.com/glossary/few-shot-prompting.md): Few-shot prompting is a technique that puts a few example inputs paired with desired outputs in a prompt, so a language model infers the task, format, and labels without changing its weights. - [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning.md): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format. - [Foundation model](https://theaiengineering.com/glossary/foundation-model.md): A foundation model is a large AI model trained on broad data, usually with self-supervised learning at scale, that can be adapted through prompting or fine-tuning to a wide range of downstream tasks. - [GGUF](https://theaiengineering.com/glossary/gguf.md): GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file. - [Golden dataset](https://theaiengineering.com/glossary/golden-dataset.md): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application. - [Guardrails](https://theaiengineering.com/glossary/guardrails.md): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits. - [Hallucination](https://theaiengineering.com/glossary/hallucination.md): A hallucination is output from a language model that is fluent and plausible but factually false, unsupported by the provided sources, or invented outright, such as a fabricated citation or package name. - [Human-in-the-loop](https://theaiengineering.com/glossary/human-in-the-loop.md): Human-in-the-loop (HITL) is an agent design in which the run pauses at defined checkpoints, such as before a sensitive tool call, so a person can approve, edit, or reject the action. - [Hybrid search](https://theaiengineering.com/glossary/hybrid-search.md): Hybrid search is a retrieval method that runs keyword search, typically BM25, and vector similarity search on the same query and merges the two ranked lists into one result set. - [Inference engine](https://theaiengineering.com/glossary/inference-engine.md): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API. - [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency.md): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts. - [JSON Schema](https://theaiengineering.com/glossary/json-schema.md): JSON Schema is a declarative vocabulary, itself written in JSON, for describing the structure of JSON data; LLM APIs and MCP use it to define tool arguments and constrain model output. - [JSON-RPC](https://theaiengineering.com/glossary/json-rpc.md): JSON-RPC is a lightweight, transport-agnostic remote procedure call protocol that encodes method calls, results, and errors as JSON objects; version 2.0 is the message format of MCP and A2A. - [Knowledge distillation](https://theaiengineering.com/glossary/knowledge-distillation.md): Knowledge distillation is a model compression technique in which a smaller student model is trained to reproduce the outputs or output probabilities of a larger, more capable teacher model. - [KV cache](https://theaiengineering.com/glossary/kv-cache.md): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step. - [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model.md): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions. - [Late chunking](https://theaiengineering.com/glossary/late-chunking.md): Late chunking is an embedding technique that runs a long-context embedding model over a whole document first, then splits the token embeddings at chunk boundaries and averages each span into one chunk vector. - [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation.md): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review. - [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing.md): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota. - [LLM observability](https://theaiengineering.com/glossary/llm-observability.md): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze. - [LLM routing](https://theaiengineering.com/glossary/llm-routing.md): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability. - [LLM streaming](https://theaiengineering.com/glossary/llm-streaming.md): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete. - [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge.md): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders. - [LMArena](https://theaiengineering.com/glossary/lmarena.md): LMArena, originally Chatbot Arena and branded Arena since January 2026, is a public leaderboard that ranks language models from crowdsourced human votes between pairs of anonymous model responses. - [LoRA](https://theaiengineering.com/glossary/lora.md): LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters and memory use. - [MCP gateway](https://theaiengineering.com/glossary/mcp-gateway.md): An MCP gateway is a proxy that sits between MCP hosts and many MCP servers, exposing them through one endpoint and centralizing authentication, tool access policy, and audit logging. - [MCP server](https://theaiengineering.com/glossary/mcp-server.md): An MCP server is a program that exposes tools, resources, and prompts to AI applications through the Model Context Protocol, running locally over stdio or remotely over Streamable HTTP. - [MCP transport](https://theaiengineering.com/glossary/mcp-transport.md): An MCP transport is the binding that carries Model Context Protocol messages between a client and a server; the specification defines stdio for local subprocesses and Streamable HTTP for remote servers. - [Mixture of experts](https://theaiengineering.com/glossary/mixture-of-experts.md): Mixture of experts (MoE) is a neural network architecture in which a learned router sends each token to a few specialized subnetworks, called experts, so only part of the model runs per token. - [MMLU](https://theaiengineering.com/glossary/mmlu.md): MMLU (Massive Multitask Language Understanding) is a benchmark of 15,908 four-option multiple-choice questions across 57 subjects, from elementary mathematics to law, that measures a language model's broad academic knowledge. - [Model cascade](https://theaiengineering.com/glossary/model-cascade.md): A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails. - [Model Context Protocol (MCP)](https://theaiengineering.com/glossary/model-context-protocol.md): Model Context Protocol (MCP) is an open protocol, introduced by Anthropic in November 2024, that standardizes how AI applications connect to external tools, data sources, and prompts through MCP servers. - [Model failover](https://theaiengineering.com/glossary/model-failover.md): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit. - [Multi-agent system](https://theaiengineering.com/glossary/multi-agent-system.md): A multi-agent system is an AI architecture in which several language-model agents, each with its own prompt, tools, and context, work on parts of a task and pass results to one another. - [Multimodal model](https://theaiengineering.com/glossary/multimodal-model.md): A multimodal model is an AI model that takes in or produces more than one type of data, such as text, images, audio, or video, and combines them within a single model. - [Online evaluation](https://theaiengineering.com/glossary/online-evaluation.md): Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release. - [Open-weights model](https://theaiengineering.com/glossary/open-weights-model.md): An open-weights model is an AI model whose trained parameters are published for download, so anyone can run, fine-tune, or quantize it on their own hardware under the terms of its license. - [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api.md): An OpenAI-compatible API is an HTTP API that accepts OpenAI's request and response formats, so code written for OpenAI's SDKs can call another server by changing the base URL, API key, and model name. - [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry.md): OpenTelemetry is an open source, vendor-neutral observability framework of APIs, SDKs, a wire protocol, and naming conventions for traces, metrics, and logs, including conventions for LLM calls and agent runs. - [PagedAttention](https://theaiengineering.com/glossary/paged-attention.md): PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging. - [pass@k](https://theaiengineering.com/glossary/pass-at-k.md): pass@k is an evaluation metric that estimates the probability that at least one of k sampled outputs for a task passes its tests, used mainly in code generation and agent benchmarks. - [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode.md): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass. - [Pretraining](https://theaiengineering.com/glossary/pretraining.md): Pretraining is the first stage of training a language model, in which it learns to predict the next token across a large unlabeled corpus of text and code, producing a general-purpose base model. - [Prompt caching](https://theaiengineering.com/glossary/prompt-caching.md): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it. - [Prompt engineering](https://theaiengineering.com/glossary/prompt-engineering.md): Prompt engineering is the practice of designing the instructions, examples, and formatting given to a large language model so it produces accurate, consistent output for a task, without changing model weights. - [Prompt injection](https://theaiengineering.com/glossary/prompt-injection.md): Prompt injection is an attack in which text supplied to a large language model, typed by a user or hidden in content the model reads, alters the application's intended behavior or output. - [Quantization](https://theaiengineering.com/glossary/quantization.md): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats. - [Rate limiting](https://theaiengineering.com/glossary/rate-limiting.md): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429. - [ReAct agent](https://theaiengineering.com/glossary/react-agent.md): A ReAct agent is an AI agent that alternates a written reasoning step, an action such as a tool call, and an observation of the result, repeating the cycle until it can answer. - [Reasoning model](https://theaiengineering.com/glossary/reasoning-model.md): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems. - [Recall@k](https://theaiengineering.com/glossary/recall-at-k.md): Recall@k is a retrieval metric that measures the fraction of a query's relevant documents that appear in the top k results, averaged over a set of test queries. - [Reranking](https://theaiengineering.com/glossary/reranking.md): Reranking is a second retrieval stage that rescores an initial list of candidate documents with a more accurate but slower model, usually a cross-encoder, and keeps only the most relevant ones. - [Retrieval-augmented generation (RAG)](https://theaiengineering.com/glossary/retrieval-augmented-generation.md): Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from an external knowledge source at query time and adds them to a language model's prompt so the answer is grounded in that data. - [RLHF](https://theaiengineering.com/glossary/rlhf.md): RLHF (reinforcement learning from human feedback) is a fine-tuning method that trains a reward model on human rankings of model outputs, then optimizes the language model with reinforcement learning to score highly. - [Semantic caching](https://theaiengineering.com/glossary/semantic-caching.md): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match. - [Small language model](https://theaiengineering.com/glossary/small-language-model.md): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters. - [Speculative decoding](https://theaiengineering.com/glossary/speculative-decoding.md): Speculative decoding is an inference technique where a cheap draft proposes a few tokens and the target model verifies them in one forward pass, emitting several tokens per step without changing the output distribution. - [Structured outputs](https://theaiengineering.com/glossary/structured-outputs.md): Structured outputs is an LLM API feature that constrains generation so the model's response always conforms to a developer-supplied JSON Schema, producing output that code can parse without retries. - [Subagent](https://theaiengineering.com/glossary/subagent.md): A subagent is an AI agent that another agent starts for one subtask, running in a separate context window with its own instructions and tools, and returning only a condensed result. - [SWE-bench](https://theaiengineering.com/glossary/swe-bench.md): SWE-bench is a benchmark of 2,294 real GitHub issues from 12 Python repositories, in which a model or agent must write a patch that resolves the issue and passes the repository's tests. - [System prompt](https://theaiengineering.com/glossary/system-prompt.md): A system prompt is the set of developer-written instructions sent to a language model ahead of the conversation, defining its role, rules, output format, and tool use for every turn that follows. - [Temperature](https://theaiengineering.com/glossary/temperature.md): Temperature is a sampling parameter that rescales a language model's next-token probabilities: low values concentrate probability on the likeliest tokens, and high values flatten the distribution for more varied output. - [Tensor parallelism](https://theaiengineering.com/glossary/tensor-parallelism.md): Tensor parallelism is a model parallelism technique that splits the weight matrices inside each transformer layer across several GPUs, which compute every token together and combine partial results through all-reduce operations. - [Test-time compute](https://theaiengineering.com/glossary/test-time-compute.md): Test-time compute is the computation a language model spends while answering a query, through longer reasoning, multiple samples, or search, as opposed to the train-time compute spent building the model. - [Time to first token](https://theaiengineering.com/glossary/time-to-first-token.md): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill. - [Tokens](https://theaiengineering.com/glossary/tokens.md): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary. - [Tool calling](https://theaiengineering.com/glossary/tool-calling.md): Tool calling is an LLM capability in which the model, given a list of tool definitions, outputs a structured request naming a tool and its arguments, which the application executes and returns. - [Tool poisoning](https://theaiengineering.com/glossary/tool-poisoning.md): Tool poisoning is an attack in which a malicious or compromised MCP server hides instructions in a tool's description or schema, which the model reads as trusted context and may follow. - [Transformer](https://theaiengineering.com/glossary/transformer.md): A transformer is a neural network architecture that processes sequences with self-attention, letting each token weigh information from other tokens in parallel; it is the basis of most modern large language models. - [Vector database](https://theaiengineering.com/glossary/vector-database.md): A vector database is a database built to store embedding vectors and quickly return the vectors most similar to a query vector, usually with approximate nearest neighbor indexes such as HNSW. - [Virtual key](https://theaiengineering.com/glossary/virtual-key.md): A virtual key is an API key issued by an AI gateway that maps a caller to its allowed models, budget, and rate limits, while the gateway keeps the real provider credentials.