Production agents fail quietly. Not from bad reasoning or hallucinations, but from drowning in their own context. Conversation histories pile up, tool outputs balloon, and token budgets explode. The agent forgets what it learned three turns ago or pays quadratic token costs every time it thinks.
A new ArXiv paper (2607.21503v1) frames this as a lifecycle problem, not a prompt engineering problem. The authors argue that managing what an agent holds in mind is a discipline spanning ingestion, scoping, anticipation, and compaction. They call it Agentic Context Management (ACM) and propose five primitives that treat context as a managed resource with provenance, budget constraints, and decay policies.
This matters because most production agent failures trace back to context overflow, not model capability. The plumbing question is not “can the model reason?” but “what stays in context, what gets evicted, and how do you measure the impact on task success?”
The Context Bloat Problem
Multi-turn agents accumulate state faster than you expect:
- Conversation history: Every user message and agent response.
- Tool definitions: JSON schemas for every available function, often several kilobytes each.
- Tool outputs: API responses, database query results, file contents.
- System prompts: Instructions, examples, guardrails.
A customer support agent might start with 2,000 tokens of system prompt and tool definitions. After ten turns with a few tool calls per turn, context can hit 20,000 tokens. After fifty turns, you are paying for 100,000+ tokens per inference call, and the model starts missing details from early in the conversation.
The naive approach is to keep everything. Token cost grows quadratically with conversation length because every new turn re-processes the entire history. The crude fix is to summarize periodically, but summarization loses detail and introduces a fidelity cliff where the agent can no longer answer specific questions about earlier interactions.
The Five Primitives of Agentic Context Management
The paper decomposes context management into five operations:
1. Architecting
Decide what kind of memory the agent needs and where it lives. Options include:
- Stateless agents: Re-prompt everything from scratch each turn. Simple, but expensive and fragile.
- Stateful agents: Serialize intermediate state to disk, database, or vector store. More complex, but cheaper and more reliable for long-running tasks.
- Hybrid: Keep recent context in-prompt, older context in retrieval.
The architecture choice determines failure modes. Stateless agents lose context on every restart. Stateful agents need serialization boundaries, schema versioning, and migration paths.
2. Ingesting
Extract and structure what the agent should remember. This is not just appending raw text. It means:
- Parsing tool outputs into structured records.
- Tagging messages with metadata (timestamp, user ID, session ID).
- Deciding what is ephemeral (a weather API response) versus durable (a customer preference).
Ingestion is where you enforce schema. If you store unstructured blobs, you cannot query or compact them later.
3. Scoping
Decide what subset of memory is relevant right now. Scoping strategies include:
- Recency: Keep the last N turns.
- Relevance: Embed and retrieve based on semantic similarity to the current query.
- Hierarchy: Scope by user, team, organization, or tenant.
Scoping is the difference between an agent that remembers everything about everyone (privacy nightmare, token explosion) and one that remembers only what it needs for this task.
4. Anticipating
Predict what the agent will need next and prefetch it. This is speculative retrieval:
- If the agent is debugging a deployment, anticipate that it will need logs, config files, and recent commits.
- If the agent is drafting a proposal, anticipate that it will need past proposals, templates, and stakeholder feedback.
Anticipation reduces latency and avoids the “I need to look that up” loop that burns extra turns.
5. Compacting and Consolidation
Reduce context size without losing fidelity. Techniques include:
- Summarization: Condense conversation history into a shorter narrative.
- Deduplication: Remove redundant tool outputs or repeated instructions.
- Forgetting: Evict old, low-relevance data with a decay policy.
- Provenance tracking: Keep pointers to the original data so the agent can retrieve details if needed.
The paper argues that validated compaction (where you test that the compacted context still supports the same queries) is the only way to achieve linear token cost without an accuracy cliff.
Stateless vs. Stateful: The Core Trade-Off
| Dimension | Stateless Agents | Stateful Agents |
|---|---|---|
| Complexity | Low (no persistence layer) | High (serialization, schema, migrations) |
| Token Cost | High (re-prompt everything) | Low (only active context in-prompt) |
| Failure Mode | Lose all context on restart | State corruption, schema drift |
| Observability | Easy (everything in logs) | Hard (state spread across stores) |
| Latency | High (large prompts every turn) | Low (small prompts, retrieval on demand) |
| Best For | Short tasks, debugging, demos | Long-running workflows, multi-session |
Stateless agents are easier to reason about but do not scale past a few dozen turns. Stateful agents require infrastructure (a database, a vector store, a cache) and careful lifecycle management, but they are the only viable option for production workflows that span hours or days.
Reference Architecture: Maximem Synap
The paper describes a reference implementation called Maximem Synap that realizes the five primitives as a multi-tenant service. Key components:
- Ingestion pipeline: Parses tool outputs, extracts entities, tags with metadata.
- Scoping engine: Queries by user, session, or tenant and applies recency or relevance filters.
- Compaction service: Runs summarization and deduplication on a schedule, validates output against a test set.
- Provenance store: Keeps pointers to original data for retrieval if the agent needs details.
The system reports 92% accuracy on LongMemEval and 93.2% on LoCoMo, benchmarks that test long-context recall and multi-session coherence. The key insight is that compaction must be validated, not assumed. You cannot just summarize and hope it works.
Instrumentation and Observability
You cannot manage what you do not measure. Context management requires tracking:
- Context size per turn: How many tokens are in-prompt versus retrieved?
- Eviction rate: How often are you pruning old data?
- Retrieval precision: When the agent retrieves context, is it the right context?
- Task success rate: Does compaction hurt the agent’s ability to complete tasks?
A simple instrumentation pattern:
class ContextManager:
def __init__(self, max_tokens=8000):
self.max_tokens = max_tokens
self.context = []
self.evicted = []
self.metrics = {"turns": 0, "evictions": 0, "retrievals": 0}
def add_turn(self, user_msg, agent_msg, tool_outputs):
turn = {
"user": user_msg,
"agent": agent_msg,
"tools": tool_outputs,
"timestamp": time.time(),
}
self.context.append(turn)
self.metrics["turns"] += 1
self._compact_if_needed()
def _compact_if_needed(self):
current_size = sum(len(json.dumps(t)) for t in self.context)
if current_size > self.max_tokens:
# Evict oldest turn
evicted = self.context.pop(0)
self.evicted.append(evicted)
self.metrics["evictions"] += 1
def retrieve_relevant(self, query):
# Semantic search over evicted context
self.metrics["retrievals"] += 1
return [t for t in self.evicted if query.lower() in json.dumps(t).lower()]This is a toy example, but it shows the pattern: track what goes in, what gets evicted, and what gets retrieved. In production, you would use embeddings for retrieval and log metrics to a time-series database.
Failure Modes and Mitigation
| Failure Mode | Cause | Mitigation |
|---|---|---|
| Context rot | Evicting data the agent still needs | Validate compaction against task success metrics |
| Quadratic cost | Keeping everything in-prompt | Implement scoping and compaction |
| Retrieval miss | Evicted data not indexed correctly | Use embeddings, tag with metadata |
| State corruption | Serialization bugs, schema drift | Version schemas, test migrations |
| Privacy leak | Scoping by user but retrieving across tenants | Enforce tenant boundaries in queries |
The most common failure is evicting something the agent needs later. The only fix is to test compaction strategies against real tasks and measure the impact on success rate.
When to Use Stateless vs. Stateful
Use stateless agents when:
- Tasks complete in fewer than 10 turns.
- You need simple debugging and observability.
- Token cost is not a constraint.
- You can afford to re-prompt everything on restart.
Use stateful agents when:
- Tasks span dozens or hundreds of turns.
- You need to preserve context across sessions.
- Token cost matters (it always does at scale).
- You can invest in serialization, schema management, and retrieval infrastructure.
Most production agents start stateless and migrate to stateful as they scale. The migration is painful because it requires rethinking how you store, query, and compact context.
Technical Verdict
Use this approach when:
- You are building agents that run longer than a few dozen turns.
- Token cost is eating your budget.
- You need to preserve context across sessions or users.
- You can invest in a persistence layer (database, vector store, cache).
Avoid this approach when:
- Your agent completes tasks in under 10 turns.
- You need the simplicity of stateless re-prompting.
- You do not have the infrastructure to manage serialization and retrieval.
Context management is the unglamorous plumbing that determines whether your agent works at scale. Treat it as a lifecycle problem, not a prompt engineering problem. Instrument what gets evicted, validate compaction strategies, and test against real tasks. The agents that survive production are the ones that know what to forget.