What is AI engineering? The discipline, the stack, and the work
AI engineering is the discipline of building software products on top of pretrained foundation models such as large language models. An AI engineer takes a model that another team trained, surrounds that model with instructions, data, tools, and tests, and ships a system that behaves reliably for real users. This explainer covers what AI engineering is, how the role differs from ML engineering and software engineering, the stack layer by layer, and the daily work, as the entry point to our AI engineering fundamentals and careers hub.
What is AI engineering?#
AI engineering is the practice of building applications that use pretrained foundation models as core components, with the engineering effort focused on context, integration, evaluation, and operations.
Chip Huyen's book AI Engineering (O'Reilly, 2025) describes the field as "the process of building applications with readily available foundation models." An earlier Latent Space essay, The Rise of the AI Engineer (June 2023), argued for the AI engineer title and a new software engineering subdiscipline: engineers who build products on top of models through APIs, without needing a research background.
The shift behind the term is about where the model sits. In classic machine learning, a team trained a separate model for each task, such as fraud scoring or product recommendations, and the trained model was the main deliverable. With foundation models, one general model can handle many tasks through instructions. The model becomes a dependency, like a database, and the engineering work moves to specifying, constraining, and verifying what the model does inside a product.
That work is harder to test than ordinary code. The same input can produce different outputs, a provider's model update can change behavior without any code change, and failures such as hallucination do not throw exceptions. AI engineering is the set of practices that make such a component dependable anyway.
AI engineer vs ML engineer vs software engineer#
An ML engineer trains and deploys models from data; a software engineer builds deterministic systems from code; an AI engineer builds products around models that other teams trained. The three roles share tools and often share people, but they differ in their primary artifact, their typical failure modes, and how they check that the work is correct.
| ML engineer | Software engineer | AI engineer | |
|---|---|---|---|
| Primary artifact | A trained model and its data pipeline | Application code and services | An application built around a pretrained model |
| Usual starting point | Collect and label data, then train | Write requirements, then code | Prompt a capable model, then add data and tests |
| Core skills | Statistics, training, feature engineering, MLOps | Languages, system design, testing | Context design, retrieval, evals, cost and latency control |
| How correctness is checked | Held-out metrics such as accuracy or AUC | Unit, integration, and end-to-end tests | Eval sets, judge models, human review, online metrics |
| Typical failure modes | Data drift, poor generalization | Bugs, outages, regressions | Hallucinations, prompt regressions, tool misuse, cost spikes |
AI engineer vs ML engineer#
The Latent Space essay drew the line by ownership: ML engineers handle model training and general benchmarks, while AI engineers treat product-specific data and evals as their own job. The workflow also runs in the opposite direction. An ML project usually starts by collecting a dataset; an AI engineering project usually starts with a working prototype built from prompts, and the team collects data afterward to measure and improve that prototype. The roles meet at fine-tuning, where an AI engineer adapts an existing model with task data, which draws on ML skills without training from scratch.
AI engineering vs software engineering#
AI engineering is a branch of software engineering, not a separate profession. AI engineers use the same version control, CI pipelines, APIs, queues, and databases as any backend team. The difference is nondeterminism. A software test asserts an exact result; an AI eval measures a pass rate across many cases. A software spec is code; much of an AI system's behavior is specified in natural-language instructions that a model interprets. Treating prompts as untested configuration lets regressions ship unseen.
The AI engineering stack#
The AI engineering stack is the set of layers between a foundation model and a user-facing product. Working up from the bottom, the layers are the model, inference, gateways and observability, retrieval, prompts and context, and tools and agents, with evals measuring the output of every layer. Each layer has its own tools, failure modes, and design decisions.
Models#
The model layer is the choice of which foundation model, or models, the product calls. The options are proprietary models served through a provider's API and open-weights models that a team can host itself. The tradeoffs are capability on the specific task, cost per token, latency, context length, data-handling terms, and license. Many systems use several models: a capable model for hard requests and a smaller, cheaper one for routine requests. Our model providers directory lists the main options.
Prompts and context#
The prompt and context layer decides what the model sees on each call: system instructions, examples, conversation history, retrieved documents, and tool results. Context engineering treats this as a design problem, since the context window is finite and irrelevant material degrades answers. Structured output schemas belong here too, because they turn free text into data the rest of the system can validate.
Retrieval#
Retrieval supplies facts the model was never trained on, such as a company's documents or a customer's account history. Retrieval-augmented generation combines embeddings, vector or keyword search, and reranking to pick the passages that go into context. When a model "doesn't know our data," the fix is usually a retrieval fix, not a model change. Our guide to RAG covers the design choices.
Tools and agents#
Tool calling lets a model request a function call, such as a database query or an API request, which the application then executes. An agent runs that pattern in a loop until a task is done. The Model Context Protocol standardizes how applications expose tools and data to models; the MCP specification defines the message format.
Tool access is also where risk concentrates, because a tool call can change data in a real system. Our explainer on AI agents covers the agent loop, its failure modes, and the controls that limit damage.
Evals#
Evals are the test suite of an AI system: datasets of inputs with expected properties, scored by code checks, human reviewers, or an LLM-as-a-judge. Teams run offline evals before each change and track online metrics after release. Without evals, no one can say whether a prompt edit or model upgrade made the product better. Our LLM evaluation guide covers methods and tooling.
Gateways and observability#
An AI gateway sits between the application and model providers. Gateways such as Bifrost and LiteLLM handle routing, fallbacks when a provider fails, API key management, rate limits, and caching. Observability records a trace of each request: the prompt, retrieved context, tool calls, tokens, latency, and cost. OpenTelemetry maintains semantic conventions for generative AI so that model and agent spans share a common schema across tools.
Inference#
Inference is where tokens are actually generated. Teams that use hosted APIs mostly tune request patterns, such as batching, streaming, and prompt caching. Teams that self-host open-weights models run serving engines and work on quantization, KV cache management, and GPU utilization to hit latency and cost targets.
What does an AI engineer do day to day?#
An AI engineer's day mixes ordinary software work with a model-specific improvement loop: reading traces of failed requests, turning failures into eval cases, changing a prompt, retrieval step, or model, rerunning evals, and shipping when scores hold. Integration code, data plumbing, and cost reviews fill most of the remaining time.
Typical recurring tasks include:
- Reviewing production traces and user feedback to find failure patterns.
- Adding failing cases to the eval set so each fix stays fixed.
- Editing prompts, tool descriptions, or retrieval settings and measuring the effect.
- Comparing a new model version against the current one before switching.
- Writing integration code: APIs, authentication, queues, and data pipelines.
- Tracking cost and latency budgets, and adding caching, routing, or smaller models where they help.
- Reviewing security exposure, including prompt injection, data leakage, and tool permissions.
- Working with product owners and domain experts to define what a good answer looks like.
The loop is the main difference from a typical backend role. A backend engineer fixes a bug once and a test guards it. An AI engineer fixes a failure pattern, measures the fix across hundreds of cases, and checks that the change did not break cases that previously passed.
AI engineer skills#
The core AI engineer skills are strong general programming, an accurate model of how LLMs behave, context and retrieval design, evaluation methods, and production operations for cost, latency, and security. Machine learning theory and model training matter for fine-tuning and inference optimization work, but they sit on top of the software foundation rather than replacing it.
| Skill area | What it covers | Why it matters |
|---|---|---|
| Programming | Python or TypeScript, async I/O, HTTP APIs, testing | Much of AI engineering code is integration code |
| LLM fundamentals | Tokens, context windows, sampling, structured outputs | Predicts failure modes and cost before they happen |
| Context design | Instructions, examples, tool schemas, context assembly | The main lever on model behavior without retraining |
| Retrieval | Embeddings, chunking, hybrid search, reranking | Grounds answers in data the model never saw |
| Evaluation | Test set design, judge models, basic statistics | The only way to know whether a change helped |
| Production operations | Tracing, gateways, caching, rate limits, cost tracking | Keeps the system usable and affordable at scale |
| Security | Prompt injection, data exposure, least-privilege tools | Agents act on real systems with real permissions |
| Optional depth | Fine-tuning, LoRA, inference optimization | Needed when hosted APIs cannot meet requirements |
For a staged plan that builds these skills in order, with a project per stage, see the AI engineer roadmap.
An example: building a support assistant#
A customer-support assistant shows how the layers fit together in one product. The goal is an assistant that answers questions from a help center, looks up order status, and hands off to a human when needed. Each requirement maps to a specific layer of the stack and a specific engineering decision.
- Model and inference. The team compares two hosted models on 200 real support questions and picks the cheaper one that meets the accuracy bar, keeping the stronger model as a fallback.
- Retrieval. Help-center articles are chunked, embedded, and indexed with keyword search alongside vector search, because product names and error codes match poorly on embeddings alone.
- Context. The system prompt states scope, tone, and when to escalate. Each request includes the top retrieved passages and the last few conversation turns, not the full history.
- Tools. The assistant gets two tools: a read-only order lookup and an escalation tool. No tool can issue refunds, so a manipulated prompt cannot trigger one.
- Evals. The 200 questions become the first eval set, scored for factual accuracy against the retrieved source, correct escalation, and tone. Every production failure adds a case.
- Gateway and observability. Requests pass through a gateway with provider fallback and per-team budgets. Each conversation produces a trace, so a bad answer can be traced to the retrieval step, the prompt, or the model.
None of these steps involve training a model. Every step involves engineering judgment about data, interfaces, risk, and measurement, which is what the AI engineer title describes.
Where to go next#
The AI engineer roadmap turns the skills above into a staged learning plan with projects. For the layer that most separates AI engineering from ordinary backend work, read the LLM evaluation guide. To see how tool use becomes autonomous behavior, continue with what AI agents are and how they work.
Frequently asked questions
What is AI engineering in simple words?
- AI engineering is building useful software around an AI model that someone else trained. The AI engineer picks a model, decides what information and instructions it receives, connects it to data and tools, tests whether its answers are good enough, and keeps the whole system fast, affordable, and safe once real users depend on it. The model is one component; the product is the engineering around it.
Is AI engineering hard?
- AI engineering is approachable for working software engineers, because most of the code is ordinary integration, data, and service work. The hard part is different from classic programming: model outputs vary between runs, so you cannot prove correctness with a few unit tests. Learning to design evaluation sets, read traces, and reason about failure rates takes deliberate practice and is where most newcomers struggle.
Can a software engineer become an AI engineer?
- Yes. Software engineering is the most common path into AI engineering, because the role is a branch of software engineering. A software engineer already has the programming, testing, and systems skills. The additions are an accurate mental model of how language models behave, context and retrieval design, evaluation methods, and production concerns such as token cost, latency, and prompt injection.
Do AI engineers need a degree?
- A specific degree is not a technical requirement for AI engineering work, because the core skills are programming, system design, and evaluation, which can be learned and demonstrated through projects. Individual employers set their own hiring criteria, and some research-adjacent roles do ask for degrees in computer science or related fields. A portfolio of shipped, measured projects is the most direct evidence of ability.
What is the difference between AI engineering and data science?
- Data science focuses on analysis: asking questions of data, running experiments, and building predictive models to inform decisions. AI engineering focuses on production software: building and operating applications that call foundation models for end users. The two overlap in evaluation and statistics, since both need sound measurement, but an AI engineer's output is a running system rather than an analysis or report.
Terms in this guide
- Large language model (LLM)
- Context engineering
- Context window
- Retrieval-augmented generation (RAG)
- Tool calling
- Model Context Protocol (MCP)
- LLM-as-a-judge
- AI gateway
- Hallucination
- Fine-tuning
- Foundation model
- Pretraining
- Golden dataset
- Online evaluation
- Open-weights model
- System prompt
- Model failover
- Rate limiting
- OpenTelemetry
- LLM streaming
- Inference engine
Cite this guide
The AI Engineering Editors. "What is AI engineering? The discipline, the stack, and the work." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/what-is-ai-engineering
- Title
- What is AI engineering? The discipline, the stack, and the work
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Published
- Updated