Context vs. Memory Engineering in Agentic AI Systems
In this article, you will learn how context engineering and memory engineering solve different problems in agentic AI systems, and how the two disciplines meet at the point where retrieved memory enters the context window
Introduction
As AI agentsmove into longer workflows and multi-session use cases, a familiar pattern emerges. Constraints get dropped mid-task, retrieved information resurfaces when it shouldn’t, and context from an earlier step bleeds into the current one. The failures are hard to pinpoint because no single component is obviously at fault
Most of the time, the problem lies in two areas that get built together, conflated, or skipped:context engineering andmemory engineering. They are related but distinct, fail in different ways, and require different systems to get right
This article covers the core decisions behind each discipline and where they interact:
Understanding both, separately and together, is what determines whether an agent holds up across real workloads
An Overview of Context and Memory Engineering
Context engineeringcovers the design of a single inference call: what to include, what to compress, where to place things, and what to discard. Everything in scope is ephemeral. When the call ends, the window clears
Memory engineeringfocuses on what survives beyond a single interaction with a model. It encompasses the systems and policies responsible for writing, storing, retrieving, updating, and governing information so that future interactions can make use of it. When an agent recalls information from a previous session, coordinates with another agent, or applies a user preference learned days or weeks earlier, it is relying on memory engineering
While context engineering determines what information is available to the model during a specific request, memory engineering determines what information persists across requests and how that information is maintained, retrieved, and trusted over time. Here’s an overview:
| ASPECT | CONTEXT ENGINEERING | MEMORY ENGINEERING |
|----------------------|----------------------------------------------|---------------------------------------------|
| Scope | One inference call | Across calls, sessions, agents |
| Where data lives | Inside the model’s active window | External stores: vector DB, K/V, relational |
| Primary problem | What to include and how to arrange it | What to persist, retrieve, and trust |
| Fails when | Window fills, placement is wrong, noise overwhelms signal | Retrieval misses, staleness, poisoning, no write policy |
| Engineering surface | Prompt structure, compression, token budgeting | Storage schema, retrieval strategy, write and update policies |
| Lifespan of data | Duration of one LLM call | Depends on the memory type |Context Engineering: Assembling the Optimal Context Window
For an agent running a multi-step workflow, every inference call assembles a context window from multiple sources: system prompt, task description, conversation history, tool outputs, retrieved documents, subagent summaries. Context engineering is the set of decisions that determine what each component contributes, in what form, and in what position
Selective Inclusion
Not everything available should enter the context. A database query returning hundreds of rows, a web search returning five complete articles, a code executor logging verbose output ➜ all of these bloat the window and reduce reasoning quality before the token limit is reached. The decision about what gets included verbatim, what gets compressed to key facts, and what gets dropped is a design choice, not a default
Structural Placement
Where information sits in the window affects how reliably the model uses it. Models attend more strongly to content at the beginning and end of long contexts, with material in the middle receiving significantly less weight. This is known as the"lost in the middle" effect
Hard constraints and task-critical instructions belong at the top of the window. Retrieved information that is most relevant to the current task should be placed near the end of the context window
The current user query or task should typically follow the retrieved information, positioning both the relevant context and the immediate objective as close as possible to the generation point. This arrangement increases the likelihood that the model will effectively use the retrieved information when producing its response
Compression on Arrival
Tool outputs should be compressed after a call returns, not after the window fills. A raw API response carrying 3,000 tokens, of which the agent needs only 150, should be summarized before it enters context for the next step. Waiting until the window is full and then scrambling to truncate is reactive management of a problem that compression at the source prevents
Conversation History Management
Conversation history grows faster than any other context component. For long-running agents, carrying the full history into every call makes every subsequent inference more expensive and less reliable. A compression strategy ➜ rolling window, hierarchical summarization, or structured state extraction ➜ should be applied at defined intervals, not when the window overflows
Memory Engineering: Designing Persistent AI Memory Systems
Once an inference call completes, memory engineering determines what deserves to persist and under what conditions it gets used again. This covers four distinct concerns: what to write, where to store it, how to retrieve it, and how to keep it accurate over time
Write Policy Design
Write policy design is one of the most overlooked aspects of memory engineering, yet it has a disproportionate impact on memory quality over time. While retrieval systems often receive the most attention, retrieval quality is ultimately constrained by what enters the memory store in the first place
A well-defined write policy specifies:
Without explicit write policies, systems often default to storing too much information, assigning equal trust to all entries, and retaining data indefinitely. Over time, low-value and outdated memories accumulate, signal-to-noise ratios decline, and retrieval quality degrades. The result is a memory system that grows continuously while becoming progressively less useful
Storage Layer Selection
Different memory types serve different purposes and require different storage backends. The choice of backend also constrains which retrieval strategies are available
| MEMORY TYPE | WHAT IT STORES | STORAGE BACKEND | RETRIEVAL METHOD |
|-------------|-----------------------------------------|--------------------------------------|-------------------------------|
| Working | Active task state, intermediate results | In-memory or short-lived K/V (Redis) | Direct key lookup |
| Episodic | Past interactions, task runs, decisions | Vector store (Pinecone, Weaviate, Chroma) | Semantic similarity search |
| Semantic | Persistent facts, user preferences, domain knowledge | Vector store + K/V hybrid | Semantic search or exact key |
| Procedural | Learned workflows, successful action patterns | Structured store or prompt injection | Pattern match, direct retrieval |OpenAI’s context personalization cookbook makes a useful distinction between retrieval-based memory and state-based memory for use cases requiring continuity. Retrieval-based memory treats past interactions as loosely related documents and is brittle to phrasing variation and conflicting updates. Structured state extraction ➜ writing typed, validated facts rather than embedding raw conversation ➜ produces more consistent results for facts that need to be applied reliably across sessions
Retrieval Strategy
Reading from memory is not a single operation
A well-designed retrieval layer checks working memory first (fast, cheap, exact key lookup) ➜ falls back to semantic search in episodic or semantic memory when nothing relevant surfaces ➜ applies metadata filters for recency and trust level before returning results ➜ and injects only what the current step needs
Memory Maintenance
A store with no maintenance policy degrades over time. The entries accumulate, stale facts compete with current ones, and retrieval quality falls as signal-to-noise ratio drops. The following maintenance routines matter in practice: confidence decay on volatile facts, deduplication of semantically similar entries, TTL-based expiry on working memory and time-sensitive data, and periodic compression of old episodic records into session-level summaries
A MemoryEntry schema that encodes these concerns directly makes write and maintenance logic easier to reason about:
class MemoryEntry(BaseModel):
content: str
memory_type: str # working | episodic | semantic | procedural
importance: float # 0.0–1.0, gates long-term storage
confidence: float # decays over time for volatile facts
trust_level: float # 1.0 internal system, 0.5 user input, 0.0 external
created_at: datetime
expires_at: datetime | None
provenance: dict # agent_id, tool_name, session_id, input_hash
def should_write_to_long_term(entry: MemoryEntry) -> bool:
return (
entry.importance >= 0.6
and entry.confidence >= 0.7
and entry.trust_level >= 0.5
)The Retrieval Boundary: Connecting Memory and Context Engineering
Memory engineering and context engineering are often discussed as separate disciplines, but in practice they are deeply interconnected. Both exist to solve the same core problem: ensuring that a model has access to the right information at the right time
At a high level:
Memory systems produce candidate information. Context assembly then decides:
Managing this boundary well is what transforms a collection of memory components into a coherent agent system
Failure Mode #1: Retrieval Without a Context Budget
One of the most common failures occurs when retrieval is treated independently from context assembly
A memory search returns a set of relevant entries, and the context assembler injects all of them into the prompt. As more memories are added, the context window gradually fills with retrieved content, leaving less room for instructions, tool outputs, reasoning traces, and task-specific information
The resulting symptoms are often misleading:
In many cases, the memory system has done its job correctly. The failure occurs because context assembly lacks a budgeting mechanism
A better approach is retrieval-aware context assembly. Instead of retrieving first and budgeting later, the context layer allocates a token budget before retrieval begins. The retrieval layer then returns only the highest-value memories that fit within that budget
async def retrieve_for_step(
self,
step: AgentStep,
max_tokens: int
) -> str:
candidates = await self.memory.search(
query=step.retrieval_query,
max_results=10,
filters={
"trust_level": {"gte": 0.5},
"expires_at": {"gt": datetime.now()}
}
)
selected = []
used = 0
for entry in sorted(
candidates,
key=lambda e: e.relevance_score,
reverse=True
):
cost = self.token_count(entry.content)
if used + cost > max_tokens:
break
selected.append(entry.content)
used += cost
return "\n\n".join(selected)Retrieval must operate within context constraints, not assume unlimited space downstream
Failure Mode #2: Poor Placement of Retrieved Information
Retrieval quality alone is not sufficient. Even highly relevant memories can fail if they are placed incorrectly inside the context window
A common issue is treating retrieval purely as a search problem while ignoring placement. Retrieved memories are appended wherever they arrive, without considering their role in the current reasoning step
This becomes more impactful in long contexts. Attention is not uniformly distributed across the prompt. Information placed deep inside a long context can receive significantly less influence than information positioned near the beginning or end. This leads to a subtle failure mode:
The retrieval succeeded but the placement failed. Context assembly should therefore optimize both:
Retrieved information that must influence the current step should be positioned near the active reasoning region rather than appended arbitrarily
Summary
Context and memory engineering are two layers of a single system that controls what the model knows, when it knows it, and how that knowledge is used
Context engineering operates at inference time, shaping the active information window. Memory engineering operates across time, shaping what information persists and how it can be retrieved later
| DIMENSION | CONTEXT ENGINEERING | MEMORY ENGINEERING |
|-----------------|----------------------------------------------------------|-----------------------------------------------------------------|
| Core question | What should the model see right now, and how? | What should the system retain, and for how long? |
| Primary artifact| Assembled context window per inference call | Persisted memory entries across calls and sessions |
| Token management| Budget allocation per window component | Storage cost per entry type; retrieval cost per query |
| Compression | Tool outputs summarized before injection; history rolled or extracted | Old episodic records compressed; stale facts decayed or pruned |
| Freshness | Rolling history window; stale turns dropped | TTL on volatile facts; confidence decay over time |
| Trust | Source hierarchy governs assembly order | Provenance tracked per entry; low-trust content sanitized before write |
| Multi-agent | Each agent assembles its own window independently | Scoped namespaces per agent; shared namespace for cross-agent facts |
| Failure mode | Overflow, attention degradation, noisy assembly | Poisoning, staleness, retrieval miss, unbounded growth |
| Maintenance | Proactive compression at defined intervals | TTL expiry, deduplication, confidence decay, episodic archiving |
| Where they meet | Retrieved memory enters context: budget and placement govern how | Context assembly requests retrieval within a token budget constraint |To sum up, an agentic system only works when both layers are aligned: memory determines what is available, and context determines what becomes actionable
Save this so you don't lose it
Follow @beamnxw for more technical posts :)
