Seven Layers of Observability: How AWS Bedrock's Managed Knowledge Base Instruments Multi-Agent Retrieval

AWS just published a reference architecture that treats observability as infrastructure, not an afterthought. The pattern deploys seven distinct observability layers for multi-knowledge-base agent retrieval using a single CloudFormation chain. This is the first public AWS blueprint that instruments agent reasoning, routing, citation tracking, and continuous evaluation as reproducible infrastructure.

The architecture uses Amazon Bedrock’s managed Knowledge Base and AgentCore. An agent routes across multiple knowledge bases, returns cited answers, and exposes every decision point through structured traces. Both on-demand and continuous evaluation patterns feed back into agent behavior.

The Seven Observability Layers

AWS breaks observability into seven instrumentation points that map to agent execution phases:

1. Agent reasoning traces: Captures the agent’s internal decision tree (which KB to query, how to decompose the question, when to stop).
2. Knowledge base routing logs: Records which KB the agent selected and why, including confidence scores.
3. Retrieval telemetry: Tracks chunk selection, vector similarity scores, and reranking decisions.
4. Citation chain tracing: Links every answer fragment back to source documents across multiple KBs.
5. On-demand evaluation: Runs quality checks (relevance, faithfulness, answer completeness) per query.
6. Continuous evaluation: Samples production traffic for drift detection and model degradation.
7. CloudFormation deployment state: Treats the observability stack itself as versioned infrastructure.

Each layer emits structured JSON to CloudWatch Logs, with cross-layer correlation via trace IDs. The agent doesn’t just log errors. It logs every routing decision, every retrieval candidate, and every citation link.

Multi-KB Routing and Citation Chains

When an agent routes across multiple knowledge bases in a single query, citation tracking becomes non-trivial. AWS solves this with a two-phase approach:

Phase 1: Routing decision
The agent evaluates the query against KB metadata (domain, recency, authority) and selects one or more KBs. This decision is logged with confidence scores and reasoning text.

Phase 2: Citation assembly
Each KB returns chunks with source metadata. The agent synthesizes an answer and attaches a citation chain: a list of (kb_id, document_id, chunk_id, confidence) tuples. If the answer draws from three KBs, the citation chain has three entries.

The observability stack exposes both phases. You can see why the agent chose KB-A over KB-B, and you can audit whether the final answer actually used the chunks it cited.

CloudFormation Deployment Shape

The entire stack deploys via a single CloudFormation template with nested stacks:

Resources:
  KnowledgeBaseStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub 'https://s3.${AWS::Region}.amazonaws.com/cfn-templates/kb-stack.yaml'
      Parameters:
        VectorStoreType: OpenSearchServerless
        EmbeddingModel: amazon.titan-embed-text-v2:0

  AgentStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub 'https://s3.${AWS::Region}.amazonaws.com/cfn-templates/agent-stack.yaml'
      Parameters:
        KnowledgeBaseIds: !GetAtt KnowledgeBaseStack.Outputs.KBIds
        FoundationModel: anthropic.claude-3-sonnet-20240229-v1:0

  ObservabilityStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub 'https://s3.${AWS::Region}.amazonaws.com/cfn-templates/observability-stack.yaml'
      Parameters:
        AgentId: !GetAtt AgentStack.Outputs.AgentId
        EvaluationMode: continuous
        SamplingRate: 0.1

The observability stack is parameterized separately. You can toggle between on-demand and continuous evaluation, adjust sampling rates, and configure alert thresholds without touching the agent or KB stacks.

What gets parameterized:

• Evaluation mode (on-demand, continuous, both)
• Sampling rate for continuous evaluation
• CloudWatch log retention periods
• Alert thresholds for citation accuracy and retrieval latency

What’s hard-coded:

• The seven observability layers (you get all of them)
• Trace ID propagation (always enabled)
• Citation chain structure (fixed schema)

On-Demand vs. Continuous Evaluation

AWS provides two evaluation patterns with different feedback loops:

Evaluation Mode	Trigger	Latency Impact	Use Case
On-demand	Per query	+200-500ms	High-stakes queries, compliance audits, debugging
Continuous	Sampled async	None (post-response)	Drift detection, model degradation, A/B testing

On-demand evaluation runs synchronously. The agent waits for quality checks (relevance, faithfulness, completeness) before returning an answer. This adds latency but guarantees every response meets quality thresholds. Use it for financial advice, legal research, or medical triage.

Continuous evaluation samples 10% of production traffic and evaluates asynchronously. Results feed into CloudWatch dashboards and trigger alerts if quality metrics degrade. This catches model drift, KB staleness, and prompt regressions without blocking user requests.

Both modes use the same evaluation metrics (RAGAS framework: faithfulness, answer relevance, context precision). The difference is when and how results affect agent behavior.

Failure Modes This Stack Exposes

Single-KB RAG systems hide several failure modes that become visible with multi-KB routing and full observability:

Routing oscillation
The agent flip-flops between two KBs on similar queries. Citation chains show inconsistent KB selection. Fix: Adjust KB metadata or add routing hysteresis.

Citation hallucination
The agent cites chunks it didn’t actually retrieve. Citation chain tracing catches this: the cited chunk_id doesn’t appear in retrieval telemetry. Fix: Increase retrieval top-k or tune reranking.

Cross-KB coherence failure
The agent retrieves from three KBs but synthesizes an answer that contradicts itself. Continuous evaluation flags low faithfulness scores. Fix: Add a coherence check in the agent prompt or limit KB count per query.

Evaluation drift
On-demand evaluation passes but continuous evaluation shows degrading metrics over time. This indicates the evaluation criteria don’t match production query distribution. Fix: Retrain evaluation models on production samples.

CloudFormation stack drift
Someone manually tweaks an observability parameter in the console. CloudFormation drift detection catches this, but you need to decide: update the template or revert the change.

State Management and Trace Propagation

The agent maintains no persistent state between queries. Each invocation is stateless, which simplifies observability but complicates multi-turn conversations.

Trace IDs propagate through three layers:

1. API Gateway generates a request-id on ingress.
2. AgentCore wraps this in a trace-id and passes it to all KB queries.
3. CloudWatch Logs indexes by trace-id, enabling cross-service correlation.

If you need multi-turn state (conversation history, user preferences), you must implement it outside the agent. AWS suggests DynamoDB with TTL for ephemeral state or S3 for long-term conversation logs. The observability stack doesn’t track state mutations, only individual query traces.

Security Boundaries

The CloudFormation template enforces least-privilege IAM roles:

• The agent role can invoke Bedrock models and query KBs, but cannot modify KB contents.
• The KB role can read from S3 and OpenSearch, but cannot write.
• The observability role can write to CloudWatch and read agent traces, but cannot invoke the agent.

Cross-account KB access requires explicit trust policies. If you want the agent in Account A to query a KB in Account B, you must:

1. Grant the agent role bedrock:Retrieve on the cross-account KB ARN.
2. Add a resource policy to the KB allowing the agent role.
3. Update the observability stack to collect traces from both accounts.

The template does not handle cross-account observability by default. You need a custom CloudWatch cross-account sink.

Technical Verdict

Use this pattern when:

• You need multi-KB routing with full citation tracing.
• Observability is a compliance requirement, not a nice-to-have.
• You want reproducible infrastructure for agent deployments (no ClickOps).
• You need both real-time quality checks and long-term drift detection.

Avoid this pattern when:

• You have a single knowledge base and simple RAG is sufficient.
• You can’t tolerate 200-500ms latency for on-demand evaluation.
• Your team lacks CloudFormation expertise (the nested stack pattern is non-trivial).
• You need multi-turn conversational state (this architecture is stateless).

The seven-layer observability stack is overkill for prototypes. But for production multi-KB retrieval where you need to audit every citation and detect model drift, this is the first AWS reference architecture that treats observability as infrastructure.

Source Links

• AWS Blog: Build observable enterprise agentic retrieval using Managed Amazon Bedrock Knowledge Base with AWS CloudFormation