---
title: "What is AI engineering? The discipline, the stack, and the work"
description: "What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day."
summary: "AI engineering is the discipline of building reliable software products on top of pretrained foundation models. This explainer separates it from ML and software engineering, walks through each layer of the stack, and describes the daily work and skills behind the AI engineer title."
type: "explainer"
canonical_url: "https://theaiengineering.com/articles/what-is-ai-engineering"
markdown_url: "https://theaiengineering.com/articles/what-is-ai-engineering.md"
topic: "AI engineering"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-07-01"
updated: "2026-11-11"
keywords: ["what is ai engineering","ai engineering","ai engineer","what does an ai engineer do","ai engineer vs ml engineer","ai engineering vs software engineering","ai engineering stack"]
citation: "The AI Engineering Editors. \"What is AI engineering? The discipline, the stack, and the work.\" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/what-is-ai-engineering"
---

# What is AI engineering? The discipline, the stack, and the work

## TL;DR

- AI engineering is the practice of building production applications on top of pretrained foundation models, instead of training a new model for each task.
- AI engineers differ from ML engineers mainly in where the effort goes: context design, retrieval, evaluation, and system operations rather than model training and feature pipelines.
- The AI engineering stack has seven working layers: models, prompts and context, retrieval, tools and agents, evals, gateways and observability, and inference.
- Evaluation is the central discipline of AI engineering, because model outputs are probabilistic and shift when prompts, models, or data change.
- Most of the job is still software engineering; the new skills are context design, retrieval, evaluation methods, and cost and latency control.

AI engineering is the discipline of building software products on top of pretrained foundation models such as [large language models](https://theaiengineering.com/glossary/large-language-model). An AI engineer takes a model that another team trained, surrounds that model with instructions, data, tools, and tests, and ships a system that behaves reliably for real users. This explainer covers what AI engineering is, how the role differs from ML engineering and software engineering, the stack layer by layer, and the daily work, as the entry point to our [AI engineering fundamentals and careers](https://theaiengineering.com/topics/ai-engineering) hub.

## What is AI engineering?

AI engineering is the practice of building applications that use pretrained [foundation models](https://theaiengineering.com/glossary/foundation-model) as core components, with the engineering effort focused on context, integration, evaluation, and operations.

Chip Huyen's book [AI Engineering (O'Reilly, 2025)](https://huyenchip.com/books/) describes the field as "the process of building applications with readily available foundation models." An earlier Latent Space essay, [The Rise of the AI Engineer](https://www.latent.space/p/ai-engineer) (June 2023), argued for the AI engineer title and a new software engineering subdiscipline: engineers who build products on top of models through APIs, without needing a research background.

The shift behind the term is about where the model sits. In classic machine learning, a team trained a separate model for each task, such as fraud scoring or product recommendations, and the trained model was the main deliverable. With foundation models, one general model can handle many tasks through instructions. The model becomes a dependency, like a database, and the engineering work moves to specifying, constraining, and verifying what the model does inside a product.

That work is harder to test than ordinary code. The same input can produce different outputs, a provider's model update can change behavior without any code change, and failures such as hallucination do not throw exceptions. AI engineering is the set of practices that make such a component dependable anyway.

## AI engineer vs ML engineer vs software engineer

An ML engineer trains and deploys models from data; a software engineer builds deterministic systems from code; an AI engineer builds products around models that other teams trained. The three roles share tools and often share people, but they differ in their primary artifact, their typical failure modes, and how they check that the work is correct.

| | ML engineer | Software engineer | AI engineer |
|---|---|---|---|
| Primary artifact | A trained model and its data pipeline | Application code and services | An application built around a pretrained model |
| Usual starting point | Collect and label data, then train | Write requirements, then code | Prompt a capable model, then add data and tests |
| Core skills | Statistics, training, feature engineering, MLOps | Languages, system design, testing | Context design, retrieval, evals, cost and latency control |
| How correctness is checked | Held-out metrics such as accuracy or AUC | Unit, integration, and end-to-end tests | Eval sets, judge models, human review, online metrics |
| Typical failure modes | Data drift, poor generalization | Bugs, outages, regressions | Hallucinations, prompt regressions, tool misuse, cost spikes |

### AI engineer vs ML engineer

The Latent Space essay drew the line by ownership: ML engineers handle model training and general benchmarks, while AI engineers treat product-specific data and evals as their own job. The workflow also runs in the opposite direction. An ML project usually starts by collecting a dataset; an AI engineering project usually starts with a working prototype built from prompts, and the team collects data afterward to measure and improve that prototype. The roles meet at [fine-tuning](https://theaiengineering.com/glossary/fine-tuning), where an AI engineer adapts an existing model with task data, which draws on ML skills without training from scratch.

### AI engineering vs software engineering

AI engineering is a branch of software engineering, not a separate profession. AI engineers use the same version control, CI pipelines, APIs, queues, and databases as any backend team. The difference is nondeterminism. A software test asserts an exact result; an AI eval measures a pass rate across many cases. A software spec is code; much of an AI system's behavior is specified in natural-language instructions that a model interprets. Treating prompts as untested configuration lets regressions ship unseen.

## The AI engineering stack

The AI engineering stack is the set of layers between a foundation model and a user-facing product. Working up from the bottom, the layers are the model, inference, gateways and observability, retrieval, prompts and context, and tools and agents, with evals measuring the output of every layer. Each layer has its own tools, failure modes, and design decisions.

**Figure 1: The AI engineering stack, from model to product, with evals as a cross-cutting layer.**

Seven layers from product at the top down to models at the bottom, with evals drawn as a bracket spanning every layer.

Layers, top to bottom:

1. **Product**: UI, API, or workflow
2. **Tools and agents**: Tool calls, MCP servers, agent loops
3. **Prompts and context**: Instructions, history, examples
4. **Retrieval**: Embeddings, keyword search, reranking
5. **Gateway and observability**: Routing, fallbacks, keys, caching, traces
6. **Inference**: Hosted APIs or self-served models
7. **Models**: Proprietary and open-weights models

Evals spans Product through Models.

### Models

The model layer is the choice of which foundation model, or models, the product calls. The options are proprietary models served through a provider's API and open-weights models that a team can host itself. The tradeoffs are capability on the specific task, cost per token, latency, context length, data-handling terms, and license. Many systems use several models: a capable model for hard requests and a smaller, cheaper one for routine requests. Our [model providers directory](https://theaiengineering.com/platforms/model-providers) lists the main options.

### Prompts and context

The prompt and context layer decides what the model sees on each call: system instructions, examples, conversation history, retrieved documents, and tool results. [Context engineering](https://theaiengineering.com/glossary/context-engineering) treats this as a design problem, since the [context window](https://theaiengineering.com/glossary/context-window) is finite and irrelevant material degrades answers. Structured output schemas belong here too, because they turn free text into data the rest of the system can validate.

### Retrieval

Retrieval supplies facts the model was never trained on, such as a company's documents or a customer's account history. [Retrieval-augmented generation](https://theaiengineering.com/glossary/retrieval-augmented-generation) combines embeddings, vector or keyword search, and reranking to pick the passages that go into context. When a model "doesn't know our data," the fix is usually a retrieval fix, not a model change. Our guide to [RAG](https://theaiengineering.com/articles/what-is-rag) covers the design choices.

### Tools and agents

[Tool calling](https://theaiengineering.com/glossary/tool-calling) lets a model request a function call, such as a database query or an API request, which the application then executes. An agent runs that pattern in a loop until a task is done. The [Model Context Protocol](https://theaiengineering.com/glossary/model-context-protocol) standardizes how applications expose tools and data to models; the [MCP specification](https://modelcontextprotocol.io/specification/latest) defines the message format.

Tool access is also where risk concentrates, because a tool call can change data in a real system. Our explainer on [AI agents](https://theaiengineering.com/articles/what-are-ai-agents) covers the agent loop, its failure modes, and the controls that limit damage.

### Evals

Evals are the test suite of an AI system: datasets of inputs with expected properties, scored by code checks, human reviewers, or an [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge). Teams run offline evals before each change and track online metrics after release. Without evals, no one can say whether a prompt edit or model upgrade made the product better. Our [LLM evaluation guide](https://theaiengineering.com/articles/llm-evaluation) covers methods and tooling.

### Gateways and observability

An [AI gateway](https://theaiengineering.com/glossary/ai-gateway) sits between the application and model providers. Gateways such as Bifrost and LiteLLM handle routing, fallbacks when a provider fails, API key management, rate limits, and caching. Observability records a trace of each request: the prompt, retrieved context, tool calls, tokens, latency, and cost. OpenTelemetry maintains [semantic conventions for generative AI](https://github.com/open-telemetry/semantic-conventions-genai) so that model and agent spans share a common schema across tools.

### Inference

Inference is where tokens are actually generated. Teams that use hosted APIs mostly tune request patterns, such as batching, streaming, and prompt caching. Teams that self-host open-weights models run [serving engines](https://theaiengineering.com/platforms/serving-engines) and work on quantization, KV cache management, and GPU utilization to hit latency and cost targets.

## What does an AI engineer do day to day?

An AI engineer's day mixes ordinary software work with a model-specific improvement loop: reading traces of failed requests, turning failures into eval cases, changing a prompt, retrieval step, or model, rerunning evals, and shipping when scores hold. Integration code, data plumbing, and cost reviews fill most of the remaining time.

Typical recurring tasks include:

- Reviewing production traces and user feedback to find failure patterns.
- Adding failing cases to the eval set so each fix stays fixed.
- Editing prompts, tool descriptions, or retrieval settings and measuring the effect.
- Comparing a new model version against the current one before switching.
- Writing integration code: APIs, authentication, queues, and data pipelines.
- Tracking cost and latency budgets, and adding caching, routing, or smaller models where they help.
- Reviewing security exposure, including prompt injection, data leakage, and tool permissions.
- Working with product owners and domain experts to define what a good answer looks like.

**Figure 2: The improvement loop at the center of day-to-day AI engineering work.**

A six-step cycle: production traces, failure review, new eval cases, a change to prompt, data, or model, an eval run, then shipping back to production.

Components:

- **Production traces**: With user feedback
- **Failure review**: Find failure patterns
- **New eval cases**: Add the failing cases
- **Make a change**: Prompt, data, or model
- **Run evals**: Compare, catch regressions
- **Ship the change**: Back to production

Connections:

- Production traces → Failure review
- Failure review → New eval cases
- New eval cases → Make a change
- Make a change → Run evals
- Run evals → Ship the change
- Ship the change → Production traces

The loop is the main difference from a typical backend role. A backend engineer fixes a bug once and a test guards it. An AI engineer fixes a failure pattern, measures the fix across hundreds of cases, and checks that the change did not break cases that previously passed.

## AI engineer skills

The core AI engineer skills are strong general programming, an accurate model of how LLMs behave, context and retrieval design, evaluation methods, and production operations for cost, latency, and security. Machine learning theory and model training matter for fine-tuning and inference optimization work, but they sit on top of the software foundation rather than replacing it.

| Skill area | What it covers | Why it matters |
|---|---|---|
| Programming | Python or TypeScript, async I/O, HTTP APIs, testing | Much of AI engineering code is integration code |
| LLM fundamentals | Tokens, context windows, sampling, structured outputs | Predicts failure modes and cost before they happen |
| Context design | Instructions, examples, tool schemas, context assembly | The main lever on model behavior without retraining |
| Retrieval | Embeddings, chunking, hybrid search, reranking | Grounds answers in data the model never saw |
| Evaluation | Test set design, judge models, basic statistics | The only way to know whether a change helped |
| Production operations | Tracing, gateways, caching, rate limits, cost tracking | Keeps the system usable and affordable at scale |
| Security | Prompt injection, data exposure, least-privilege tools | Agents act on real systems with real permissions |
| Optional depth | Fine-tuning, LoRA, inference optimization | Needed when hosted APIs cannot meet requirements |

For a staged plan that builds these skills in order, with a project per stage, see the [AI engineer roadmap](https://theaiengineering.com/articles/ai-engineer-roadmap).

## An example: building a support assistant

A customer-support assistant shows how the layers fit together in one product. The goal is an assistant that answers questions from a help center, looks up order status, and hands off to a human when needed. Each requirement maps to a specific layer of the stack and a specific engineering decision.

1. **Model and inference.** The team compares two hosted models on 200 real support questions and picks the cheaper one that meets the accuracy bar, keeping the stronger model as a fallback.
2. **Retrieval.** Help-center articles are chunked, embedded, and indexed with keyword search alongside vector search, because product names and error codes match poorly on embeddings alone.
3. **Context.** The system prompt states scope, tone, and when to escalate. Each request includes the top retrieved passages and the last few conversation turns, not the full history.
4. **Tools.** The assistant gets two tools: a read-only order lookup and an escalation tool. No tool can issue refunds, so a manipulated prompt cannot trigger one.
5. **Evals.** The 200 questions become the first eval set, scored for factual accuracy against the retrieved source, correct escalation, and tone. Every production failure adds a case.
6. **Gateway and observability.** Requests pass through a gateway with provider fallback and per-team budgets. Each conversation produces a trace, so a bad answer can be traced to the retrieval step, the prompt, or the model.

None of these steps involve training a model. Every step involves engineering judgment about data, interfaces, risk, and measurement, which is what the AI engineer title describes.

## Where to go next

The [AI engineer roadmap](https://theaiengineering.com/articles/ai-engineer-roadmap) turns the skills above into a staged learning plan with projects. For the layer that most separates AI engineering from ordinary backend work, read the [LLM evaluation guide](https://theaiengineering.com/articles/llm-evaluation). To see how tool use becomes autonomous behavior, continue with [what AI agents are and how they work](https://theaiengineering.com/articles/what-are-ai-agents).

## Frequently asked questions

### What is AI engineering in simple words?

AI engineering is building useful software around an AI model that someone else trained. The AI engineer picks a model, decides what information and instructions it receives, connects it to data and tools, tests whether its answers are good enough, and keeps the whole system fast, affordable, and safe once real users depend on it. The model is one component; the product is the engineering around it.

### Is AI engineering hard?

AI engineering is approachable for working software engineers, because most of the code is ordinary integration, data, and service work. The hard part is different from classic programming: model outputs vary between runs, so you cannot prove correctness with a few unit tests. Learning to design evaluation sets, read traces, and reason about failure rates takes deliberate practice and is where most newcomers struggle.

### Can a software engineer become an AI engineer?

Yes. Software engineering is the most common path into AI engineering, because the role is a branch of software engineering. A software engineer already has the programming, testing, and systems skills. The additions are an accurate mental model of how language models behave, context and retrieval design, evaluation methods, and production concerns such as token cost, latency, and prompt injection.

### Do AI engineers need a degree?

A specific degree is not a technical requirement for AI engineering work, because the core skills are programming, system design, and evaluation, which can be learned and demonstrated through projects. Individual employers set their own hiring criteria, and some research-adjacent roles do ask for degrees in computer science or related fields. A portfolio of shipped, measured projects is the most direct evidence of ability.

### What is the difference between AI engineering and data science?

Data science focuses on analysis: asking questions of data, running experiments, and building predictive models to inform decisions. AI engineering focuses on production software: building and operating applications that call foundation models for end users. The two overlap in evaluation and statistics, since both need sound measurement, but an AI engineer's output is a running system rather than an analysis or report.

## Terms in this guide

- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Context engineering](https://theaiengineering.com/glossary/context-engineering): Context engineering is the practice of selecting, structuring, and pruning all the information placed in an LLM's context window at each step, including instructions, retrieved data, tool results, and memory.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Retrieval-augmented generation (RAG)](https://theaiengineering.com/glossary/retrieval-augmented-generation): Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from an external knowledge source at query time and adds them to a language model's prompt so the answer is grounded in that data.
- [Tool calling](https://theaiengineering.com/glossary/tool-calling): Tool calling is an LLM capability in which the model, given a list of tool definitions, outputs a structured request naming a tool and its arguments, which the application executes and returns.
- [Model Context Protocol (MCP)](https://theaiengineering.com/glossary/model-context-protocol): Model Context Protocol (MCP) is an open protocol, introduced by Anthropic in November 2024, that standardizes how AI applications connect to external tools, data sources, and prompts through MCP servers.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Hallucination](https://theaiengineering.com/glossary/hallucination): A hallucination is output from a language model that is fluent and plausible but factually false, unsupported by the provided sources, or invented outright, such as a fabricated citation or package name.
- [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format.
- [Foundation model](https://theaiengineering.com/glossary/foundation-model): A foundation model is a large AI model trained on broad data, usually with self-supervised learning at scale, that can be adapted through prompting or fine-tuning to a wide range of downstream tasks.
- [Pretraining](https://theaiengineering.com/glossary/pretraining): Pretraining is the first stage of training a language model, in which it learns to predict the next token across a large unlabeled corpus of text and code, producing a general-purpose base model.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Online evaluation](https://theaiengineering.com/glossary/online-evaluation): Online evaluation is the practice of scoring an LLM application's live production traffic, usually a sample of traces, with reference-free graders, policy checks, and user feedback to detect quality problems after release.
- [Open-weights model](https://theaiengineering.com/glossary/open-weights-model): An open-weights model is an AI model whose trained parameters are published for download, so anyone can run, fine-tune, or quantize it on their own hardware under the terms of its license.
- [System prompt](https://theaiengineering.com/glossary/system-prompt): A system prompt is the set of developer-written instructions sent to a language model ahead of the conversation, defining its role, rules, output format, and tool use for every turn that follows.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry): OpenTelemetry is an open source, vendor-neutral observability framework of APIs, SDKs, a wire protocol, and naming conventions for traces, metrics, and logs, including conventions for LLM calls and agent runs.
- [LLM streaming](https://theaiengineering.com/glossary/llm-streaming): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.

## Related guides

- [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production.
- [What are AI agents? How agents work and when to use one](https://theaiengineering.com/articles/what-are-ai-agents): AI agents are LLM systems that call tools in a loop until a task is done. Learn how the loop works, agents vs agentic AI, examples, failure modes, and limits.
- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results.

Part of the [AI engineering](https://theaiengineering.com/topics/ai-engineering) topic hub.

## Cite this guide

The AI Engineering Editors. "What is AI engineering? The discipline, the stack, and the work." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/what-is-ai-engineering

- Title: What is AI engineering? The discipline, the stack, and the work
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-07-01
- Updated: 2026-11-11
- URL: https://theaiengineering.com/articles/what-is-ai-engineering
- Markdown: https://theaiengineering.com/articles/what-is-ai-engineering.md

To cite one section, link to its anchor:

- [What is AI engineering?](https://theaiengineering.com/articles/what-is-ai-engineering#what-is-ai-engineering)
- [AI engineer vs ML engineer vs software engineer](https://theaiengineering.com/articles/what-is-ai-engineering#ai-engineer-vs-ml-engineer-vs-software-engineer)
- [The AI engineering stack](https://theaiengineering.com/articles/what-is-ai-engineering#the-ai-engineering-stack)
- [What does an AI engineer do day to day?](https://theaiengineering.com/articles/what-is-ai-engineering#what-does-an-ai-engineer-do-day-to-day)
- [AI engineer skills](https://theaiengineering.com/articles/what-is-ai-engineering#ai-engineer-skills)
- [An example: building a support assistant](https://theaiengineering.com/articles/what-is-ai-engineering#an-example-building-a-support-assistant)
- [Where to go next](https://theaiengineering.com/articles/what-is-ai-engineering#where-to-go-next)
- [Frequently asked questions](https://theaiengineering.com/articles/what-is-ai-engineering#faq)
