What is agent memory in AI? Types and architecture
Also called AI agent memory, LLM memory, long-term memory for agents
Definition
Agent memory is the set of mechanisms that let an AI agent keep and retrieve information beyond a single model call, from the current context window to external stores that persist across sessions.
Agent memory is the set of mechanisms that let an AI agent keep information beyond a single model call. A language model holds no state between API requests, so anything an agent "remembers" must be stored by the application and placed back into the prompt. Designing agent memory means choosing what to store, where to store it, and when to retrieve it.
How agent memory works#
Agent memory works in two tiers. Short-term memory is whatever sits in the model's context window for the current task. Long-term memory lives in external storage that the agent reads and writes through tools or retrieval. The CoALA framework (Sumers et al., 2023) pairs working memory with three long-term types.
| Memory type | What it holds | Typical storage |
|---|---|---|
| Working (short-term) | Current goal, recent messages, tool results | The context window |
| Episodic | Records of past runs and interactions | Logs, databases, vector indexes |
| Semantic | Facts about the user, the domain, or the world | Documents, key-value stores, vector databases |
| Procedural | How to perform tasks | Model weights, prompts, and agent code |
Retrieval usually relies on embeddings for similarity search, combined with filters on time, user, or topic. Writing is harder than reading. The application needs rules for which facts to keep, how to resolve a new fact that contradicts a stored one, and when to expire stale entries. Without those rules, memory stores fill with duplicates and outdated claims that degrade later answers.
Agent memory vs context window#
Agent memory and the context window are different layers. The context window is the fixed-size input a model reads on each call; agent memory is the storage and retrieval logic that determines what enters that window. A longer context window reduces the need for external memory within one session, but it does not carry state across sessions.
Filling the window is not free either. Anthropic's context window documentation notes that accuracy and recall degrade as token count grows, which it calls context rot. MemGPT (Packer et al., 2023) framed the problem like an operating system, paging data between a small fast tier (the context) and larger slow tiers (external storage).
Memory is one part of context engineering. The context engineering guide covers how to budget tokens between memory, instructions, and tool results, and the AI agents hub links related material.
Related terms
Guides that cover agent memory
Cite this definition
The AI Engineering Editors. "What is agent memory in AI? Types and architecture." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/agent-memory
- Title
- What is agent memory in AI? Types and architecture
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated