The gap between an agent that works in a demo and one that survives production is wider than most teams expect. RAG systems have bounded failure modes: retrieve context, generate response, done. Agents have unbounded failure potential. They take actions, update records, trigger processes, and spawn other agents. When they fail, the blast radius extends beyond a single user interaction.
This is not a tutorial on building agents. This is a catalog of the specific things that break in production and the architectural decisions that prevent or mitigate those failures.
Why Agents Fail Differently
A retrieval error produces a wrong answer. An agent error produces a wrong action. Wrong actions in enterprise systems often have consequences that are difficult or impossible to reverse.
The difference:
- RAG failure: User gets incorrect information, loses trust in one answer
- Agent failure: System sends 500 emails, deletes production records, or triggers a cascade of downstream processes
This fundamental difference in failure surface requires a different set of architectural precautions. Most RAG-first teams are surprised by the problems they encounter when they add action capabilities.
The State Management Problem
The most common production failure is state loss. The agent loses track of where it is in a multi-step task and either starts over, gets stuck, or makes decisions based on an incorrect model of what has already happened.
State Checkpoint Architecture
You need explicit checkpoints between workflow steps. Not logs. Not traces. Durable state records that survive process restarts.
class WorkflowState:
def __init__(self, workflow_id: str, store: StateStore):
self.workflow_id = workflow_id
self.store = store
self.state = self.store.load(workflow_id) or {}
def checkpoint(self, step: str, data: dict):
self.state[step] = {
"completed_at": datetime.utcnow().isoformat(),
"data": data,
"status": "completed"
}
self.store.save(self.workflow_id, self.state)
def get_last_completed_step(self) -> str:
completed = [
step for step, meta in self.state.items()
if meta.get("status") == "completed"
]
return completed[-1] if completed else None
def resume_from_checkpoint(self) -> dict:
last_step = self.get_last_completed_step()
if last_step:
return self.state[last_step]["data"]
return {}The state store can be Redis, DynamoDB, or Postgres. The key requirement is atomic writes and the ability to query by workflow ID. You need to be able to answer “what was the last successfully completed step?” without replaying logs.
Idempotency Keys
Every action an agent takes needs an idempotency key. If the agent crashes after sending an API request but before recording the result, it will retry that request on restart. Without idempotency, you get duplicate actions.
def execute_action(action_id: str, action_fn, *args):
result = result_cache.get(action_id)
if result:
return result
result = action_fn(*args)
result_cache.set(action_id, result, ttl=86400)
return resultThe action ID should be deterministic based on workflow ID and step number. Not a random UUID. You need to be able to reconstruct the same action ID on retry.
Timeout and Retry Strategies
Agents call external APIs with unpredictable latency. LLM inference can take 30 seconds. Database queries can hang. Third-party APIs can go down.
Timeout Hierarchy
| Layer | Timeout | Retry Strategy | Failure Action |
|---|---|---|---|
| LLM inference | 60s | 3 attempts, exponential backoff | Degrade to simpler model or cached response |
| Tool API call | 30s | 2 attempts, linear backoff | Skip tool, mark step as degraded |
| Database query | 10s | 1 attempt | Fail fast, checkpoint before retry |
| Workflow step | 5 minutes | Resume from last checkpoint | Alert operator, enter manual review |
The key insight: timeouts cascade. If your workflow step timeout is 5 minutes but your LLM timeout is 60 seconds with 3 retries, you can hit 3 minutes of LLM time alone. Budget your timeouts from the bottom up.
Circuit Breaker Pattern
When a tool fails repeatedly, stop calling it. Track failure rates per tool and open a circuit breaker after a threshold.
class CircuitBreaker:
def __init__(self, failure_threshold: int = 5, timeout: int = 60):
self.failure_count = 0
self.failure_threshold = failure_threshold
self.timeout = timeout
self.opened_at = None
def call(self, fn, *args):
if self.is_open():
raise CircuitOpenError("Circuit breaker is open")
try:
result = fn(*args)
self.on_success()
return result
except Exception as e:
self.on_failure()
raise e
def is_open(self) -> bool:
if self.opened_at is None:
return False
elapsed = time.time() - self.opened_at
if elapsed > self.timeout:
self.reset()
return False
return True
def on_failure(self):
self.failure_count += 1
if self.failure_count >= self.failure_threshold:
self.opened_at = time.time()
def on_success(self):
self.failure_count = 0
def reset(self):
self.failure_count = 0
self.opened_at = NoneWhen the circuit is open, the agent needs a fallback path. This is where graceful degradation comes in.
Graceful Degradation
When a critical tool becomes unavailable, the agent has three options:
- Fail the workflow: Stop execution, alert operator
- Degrade functionality: Complete the workflow with reduced capability
- Queue for retry: Park the workflow and retry later
The choice depends on the business impact of partial completion versus delay.
Degradation Decision Matrix
- Critical path, no fallback: Fail immediately (example: payment processing)
- Critical path, fallback available: Degrade to fallback (example: use cached data instead of live API)
- Non-critical path: Skip step, mark as degraded (example: optional notification)
- Retriable: Queue for retry (example: data sync that can happen later)
Implement this as a tool registry with degradation policies:
tool_registry = {
"payment_api": {
"critical": True,
"fallback": None,
"degradation_policy": "fail"
},
"user_data_api": {
"critical": True,
"fallback": "cached_user_data",
"degradation_policy": "degrade"
},
"notification_service": {
"critical": False,
"fallback": None,
"degradation_policy": "skip"
}
}Observability Primitives
You cannot debug agent failures without structured observability. Logs are not enough. You need traces that show the full execution path, spans that measure timing, and structured events that capture state transitions.
Essential Observability Data
- Workflow trace: Every workflow execution gets a trace ID that follows it through all steps
- Step spans: Each step is a span with start time, end time, status, and metadata
- Tool call events: Every tool invocation is an event with input, output, latency, and error
- State transition events: Every checkpoint is an event with before/after state
- Degradation events: Every fallback or skip is an event with reason and impact
Use OpenTelemetry. Do not build your own tracing system.
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def execute_workflow(workflow_id: str):
with tracer.start_as_current_span(
"workflow.execute",
attributes={"workflow.id": workflow_id}
) as span:
state = WorkflowState(workflow_id, state_store)
for step in workflow_steps:
with tracer.start_as_current_span(
f"workflow.step.{step.name}",
attributes={"step.name": step.name}
) as step_span:
try:
result = step.execute(state)
state.checkpoint(step.name, result)
step_span.set_status(trace.Status(trace.StatusCode.OK))
except Exception as e:
step_span.set_status(
trace.Status(trace.StatusCode.ERROR, str(e))
)
step_span.record_exception(e)
raiseShip traces to Jaeger, Tempo, or Honeycomb. You need to be able to query by workflow ID and see the full execution timeline.
Deployment Shape
Agents need different infrastructure than stateless APIs.
Infrastructure Requirements
- State store: Redis or DynamoDB for workflow state
- Message queue: SQS or RabbitMQ for async task execution
- Worker pool: Horizontal scaling for parallel workflow execution
- Cron scheduler: For retry and cleanup jobs
- Trace collector: OpenTelemetry collector for observability
Do not run agents in serverless functions unless you have a robust external state store and can handle cold start latency. The state management overhead makes serverless a poor fit for most agentic workflows.
Scaling Considerations
Agents do not scale like web APIs. Each workflow execution is long-lived and stateful. You scale by adding workers, not by increasing request throughput.
Monitor:
- Active workflow count
- Average workflow duration
- Worker utilization
- State store latency
- Tool API error rates
Scale workers when utilization exceeds 70%. Do not wait for queues to back up.
Failure Modes You Will Encounter
- Partial execution: Agent completes 3 of 5 steps, crashes, restarts from step 1
- Duplicate actions: Agent sends the same email twice because it did not record the first send
- State corruption: Two workers update the same workflow state simultaneously
- Timeout cascade: LLM timeout triggers workflow timeout triggers orchestrator timeout
- Tool unavailability: Critical API goes down mid-workflow
- Rate limiting: Agent hits API rate limit, fails all subsequent requests
- Memory leak: Long-running workflow accumulates state in memory, crashes worker
All of these are preventable with the patterns described above. None of them are obvious during development.
Technical Verdict
Use this approach when:
- You are building multi-step workflows that take actions in external systems
- You need to survive process restarts without losing progress
- You need to debug failures that happened hours or days ago
- You are deploying to production and need operational confidence
Avoid this approach when:
- You are building a simple RAG system with no actions
- Your workflows are stateless and complete in under 10 seconds
- You are prototyping and do not need production reliability yet
The overhead of state management, checkpointing, and observability is not worth it for simple agents. But if you are building workflows that matter, you will encounter every failure mode described here. Build the guardrails early.