Multi-agent systems fail differently than microservices. Traditional APM tools capture latency, error rates, and throughput. They miss prompt drift, tool-selection errors, cross-agent state corruption, and emergent coordination bugs. AWS addresses this gap with a dual-layer monitoring pattern: AgentCore Evaluations for continuous quality scoring and DevOps Agent for autonomous infrastructure investigation.
The reference implementation is a four-agent airline reservation system. Each agent handles a distinct domain (flight search, booking, payment, customer service), and the system demonstrates how to instrument both quality degradation and infrastructure failures without conflating the two concerns.
The Monitoring Gap in Multi-Agent Systems
Single-agent systems fail when the model hallucinates, the prompt drifts, or a tool call returns malformed data. Multi-agent systems add coordination failures: agent A passes corrupted state to agent B, agent C makes a decision based on stale context from agent D, or the orchestrator routes a request to the wrong specialist.
Traditional observability stacks track:
- Request latency
- Error rates
- Resource utilization
- Trace spans
They do not track:
- Task completion quality
- Inter-agent handoff correctness
- Prompt effectiveness over time
- Tool selection accuracy
AWS splits these concerns into two layers. AgentCore Evaluations runs continuous quality checks. DevOps Agent investigates infrastructure and configuration issues autonomously.
AgentCore Evaluations: Continuous Quality Scoring
AgentCore Evaluations is a managed service that runs eval datasets against production agents on a schedule. You define test cases, expected outputs, and scoring functions. The service executes them without touching production traffic.
Eval Dataset Structure
Each eval case includes:
- Input prompt or task description
- Expected output or success criteria
- Scoring function (exact match, semantic similarity, custom logic)
- Agent identifier
For the airline system, evals cover:
- Flight search accuracy (correct routes, dates, prices)
- Booking completion (reservation created, payment processed, confirmation sent)
- Customer service response quality (relevant answers, correct policy citations)
- Cross-agent handoffs (state passed correctly between agents)
Instrumentation Pattern
AgentCore Evaluations runs outside the production request path. It does not add latency or error risk to live traffic. The service:
- Schedules eval runs (hourly, daily, or triggered by deployment)
- Invokes agents with test inputs
- Compares outputs to expected results
- Publishes scores to CloudWatch Metrics
- Triggers alarms when scores drop below thresholds
This separates quality monitoring from infrastructure monitoring. A drop in eval scores indicates prompt drift, model degradation, or tool misconfiguration. A spike in error rates indicates infrastructure failure.
Scoring Functions
AWS provides built-in scorers:
- Exact match
- Semantic similarity (embedding distance)
- JSON schema validation
- Custom Lambda functions
For multi-agent coordination, custom scorers check state handoff correctness:
def score_handoff(agent_output, expected_state):
"""
Validate that agent A passed correct state to agent B.
"""
if "booking_id" not in agent_output:
return 0.0
if agent_output["booking_id"] != expected_state["booking_id"]:
return 0.0
if agent_output["passenger_count"] != expected_state["passenger_count"]:
return 0.5 # Partial credit for correct ID but wrong count
return 1.0This catches coordination bugs that traditional tracing misses. A trace might show successful HTTP calls between agents, but the state passed was incorrect.
DevOps Agent: Autonomous Infrastructure Investigation
DevOps Agent is an autonomous agent that investigates infrastructure and configuration issues. It reads CloudWatch alarms, queries logs, inspects agent configurations, and proposes fixes. It does not auto-remediate by default (you can enable that), but it surfaces root causes faster than manual investigation.
Investigation Flow
When an alarm fires:
- DevOps Agent receives the alarm event
- Queries CloudWatch Logs for error patterns
- Inspects agent configuration (prompts, tool definitions, IAM roles)
- Checks recent deployments and configuration changes
- Correlates errors across agents
- Generates a root cause hypothesis
- Proposes remediation steps
For the airline system, a booking failure might trigger:
- Log query: “Show me all booking agent errors in the last hour”
- Configuration check: “Did the payment tool endpoint change?”
- Cross-agent correlation: “Are flight search agents also failing?”
- Hypothesis: “Payment service endpoint was updated but booking agent configuration was not”
Tool Access and Safety Boundaries
DevOps Agent has read access to:
- CloudWatch Logs and Metrics
- Agent configurations (Bedrock Agent definitions)
- IAM role policies
- Recent CloudFormation or CDK deployments
It has write access to:
- Incident reports (stored in S3)
- Remediation proposals (stored in DynamoDB)
- Optional auto-remediation (disabled by default)
The safety boundary is explicit. DevOps Agent can read production state and propose fixes, but it cannot modify production configurations unless you enable auto-remediation and define allowed actions (restart agent, roll back deployment, update configuration parameter).
Example Investigation
A customer service agent starts returning incorrect policy information. DevOps Agent:
- Queries logs: “customer_service_agent errors last 2 hours”
- Finds pattern: “Policy document retrieval returning 404”
- Checks configuration: “Policy document S3 bucket changed from
policies-prodtopolicies-prod-v2” - Checks recent deployments: “S3 bucket renamed 3 hours ago”
- Hypothesis: “Agent configuration still points to old bucket name”
- Proposal: “Update agent configuration to use
policies-prod-v2”
This investigation takes seconds instead of the 20 minutes a human would spend grepping logs and checking configurations.
Four-Agent Airline System Architecture
The reference implementation has four agents:
| Agent | Responsibility | Tools | Failure Modes |
|---|---|---|---|
| Flight Search | Query availability, prices, routes | Flight API, cache | Stale cache, API rate limits, incorrect date parsing |
| Booking | Create reservations, manage inventory | Booking API, inventory DB | Race conditions, double-booking, state corruption |
| Payment | Process transactions, handle refunds | Payment gateway, fraud check | Gateway timeouts, fraud false positives, currency conversion errors |
| Customer Service | Answer questions, handle complaints | Policy DB, ticket system | Outdated policy docs, incorrect routing, context loss |
Orchestration Layer
A coordinator agent routes requests to specialists. It maintains conversation context and handles multi-turn interactions. The coordinator:
- Parses user intent
- Selects the appropriate specialist agent
- Passes context and state
- Aggregates responses
- Handles errors and retries
State Management
Each agent writes state to DynamoDB. The coordinator reads and aggregates state. This creates coordination failure opportunities:
- Agent A writes state, agent B reads stale state
- Agent A writes malformed state, agent B fails to parse
- Coordinator loses context between turns
AgentCore Evaluations catches these with cross-agent test cases. DevOps Agent investigates when they happen in production.
Instrumentation Trade-offs
| Approach | Pros | Cons | Best For |
|---|---|---|---|
| Inline evals (production traffic) | Real user data, no synthetic gaps | Adds latency, risk of eval logic bugs affecting users | Low-traffic systems, non-critical paths |
| Scheduled evals (AgentCore) | No production impact, controlled test cases | Misses edge cases from real traffic | High-traffic systems, critical paths |
| Autonomous investigation (DevOps Agent) | Fast root cause, reduces MTTR | Requires careful permission boundaries | Complex multi-agent systems |
| Manual investigation | Full control, no automation risk | Slow, error-prone, does not scale | Simple systems, rare incidents |
The AWS pattern combines scheduled evals with autonomous investigation. This separates quality monitoring (did the agent do the right thing?) from infrastructure monitoring (did the system stay up?).
Deployment Shape
The monitoring stack runs alongside the agent system:
┌─────────────────────────────────────────┐
│ Production Agent System │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Flight │ │ Booking │ │ Payment │ │
│ │ Agent │ │ Agent │ │ Agent │ │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
│ │ │ │ │
│ └───────────┴───────────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ Coordinator │ │
│ └──────┬──────┘ │
└──────────────────┼──────────────────────┘
│
┌──────────┼──────────┐
│ │ │
┌────▼────┐ ┌──▼───────┐ ┌▼─────────┐
│CloudWatch│ │AgentCore │ │ DevOps │
│ Logs │ │ Evals │ │ Agent │
└─────────┘ └──────────┘ └──────────┘AgentCore Evaluations runs on a schedule. DevOps Agent listens to CloudWatch alarms. Both are decoupled from production request paths.
Likely Failure Modes
Eval dataset drift: Test cases become outdated as agent behavior evolves. Evals pass but production quality drops. Mitigation: version eval datasets with agent deployments, review eval coverage quarterly.
Investigation permission creep: DevOps Agent gains too many write permissions, auto-remediation causes cascading failures. Mitigation: start with read-only, enable auto-remediation for specific, safe actions only (restart, rollback).
Alarm fatigue: Too many low-severity alarms trigger DevOps Agent investigations. Mitigation: tune alarm thresholds, use composite alarms, rate-limit investigations.
Cross-agent eval gaps: Evals test individual agents but miss coordination bugs. Mitigation: write end-to-end test cases that exercise full workflows across multiple agents.
DevOps Agent hallucination: Agent proposes incorrect root cause or dangerous remediation. Mitigation: require human approval for all remediations, log all investigations for audit.
Technical Verdict
Use this pattern when you have multiple agents coordinating on complex workflows and traditional APM tools are not catching quality degradation or coordination bugs. The dual-layer approach (quality evals + autonomous investigation) makes sense when:
- You have more than two agents with inter-agent dependencies
- Agent behavior changes frequently (prompt updates, model swaps, tool changes)
- You need to detect quality drift before users complain
- You want to reduce mean time to resolution for infrastructure issues
Avoid this pattern when:
- You have a single agent with simple, deterministic tasks
- Your agent behavior is stable and rarely changes
- You have low traffic and can manually investigate all incidents
- You do not have the operational maturity to manage autonomous investigation safely
The AWS implementation is opinionated: it assumes you are using Bedrock Agents, CloudWatch, and DynamoDB. If you are running agents on other platforms, you will need to adapt the instrumentation and investigation patterns, but the dual-layer concept (separate quality from infrastructure) applies universally.