Amazon Bedrock AgentCore Evaluations solves a real problem: you build agents on LangGraph, your teammate uses LlamaIndex, and the platform team is experimenting with the OpenAI Agents SDK. Every framework has its own evaluation story, and none of them talk to each other.
AWS’s answer is to treat OpenTelemetry telemetry as the evaluation contract. If your agent emits the right spans and attributes, AgentCore can score it without knowing which framework you used. The service works with LangGraph, LlamaIndex, OpenAI Agents SDK, Google ADK, Claude Agent SDK, and Strands Agents. It also works with custom stacks, as long as you instrument them correctly.
This is the first major cloud vendor to decouple agent evaluation from framework choice using a telemetry-based contract. Here’s how the plumbing works.
The OpenTelemetry Contract
AgentCore Evaluations expects agents to emit structured telemetry in OpenTelemetry format. The service looks for specific span types and attributes:
- Agent spans: Top-level execution context for a single agent invocation
- Tool call spans: Individual tool invocations, including input, output, and latency
- LLM spans: Model calls with prompt, response, token counts, and model ID
- Retrieval spans: Vector search or knowledge base queries
Each span must include semantic attributes that map to evaluation dimensions. For example:
# Pseudo-code for manual instrumentation
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("agent.run") as agent_span:
agent_span.set_attribute("agent.id", "customer-support-v2")
agent_span.set_attribute("agent.framework", "langgraph")
with tracer.start_as_current_span("tool.call") as tool_span:
tool_span.set_attribute("tool.name", "get_order_status")
tool_span.set_attribute("tool.input", json.dumps({"order_id": "12345"}))
result = get_order_status("12345")
tool_span.set_attribute("tool.output", json.dumps(result))
tool_span.set_attribute("tool.success", True)The key insight is that AgentCore doesn’t care about your framework’s internal state machine or graph structure. It only cares about the observable events: what tools were called, what the LLM said, how long things took, and whether they succeeded.
Framework Adapters and Instrumentation Gaps
Most popular frameworks already emit some OpenTelemetry telemetry, but the coverage varies:
| Framework | Native OTel Support | Instrumentation Gap | Workaround |
|---|---|---|---|
| LangGraph | Partial (LangSmith integration) | Missing tool success/failure attributes | Manual span enrichment |
| LlamaIndex | Good (built-in OTel exporter) | Inconsistent span naming | Span processor to normalize |
| OpenAI Agents SDK | Minimal | No tool spans by default | Wrap tool calls with custom tracer |
| Claude Agent SDK | None | Everything | Full manual instrumentation |
| Custom stacks | None | Everything | Build from scratch |
If your framework doesn’t emit the required telemetry, you have three options:
- Manual instrumentation: Wrap your agent code with OpenTelemetry API calls
- Auto-instrumentation: Use OpenTelemetry’s auto-instrumentation libraries for HTTP, database, and LLM calls
- Span processors: Intercept and enrich spans after they’re created but before they’re exported
AWS doesn’t provide framework-specific shims. You’re responsible for making sure your agent emits the right telemetry shape. The documentation includes example instrumentation for each supported framework, but you’ll need to adapt it to your specific agent architecture.
Evaluation Metrics and Scoring
Once AgentCore receives telemetry, it computes metrics across several dimensions:
- Task success rate: Percentage of agent runs that completed without errors
- Tool accuracy: Whether the agent called the right tools in the right order
- Response quality: LLM-as-judge scoring of final outputs against ground truth
- Latency: P50, P95, and P99 for agent runs, tool calls, and LLM calls
- Cost: Token usage and estimated inference cost per run
The service stores evaluation results in a time-series database (likely Amazon Timestream, though AWS doesn’t specify). You can query results via the AgentCore API or view them in the AWS console.
Retention is 90 days by default. After that, you need to export results to S3 if you want long-term storage.
Deployment Shape and Data Flow
Here’s the typical data flow:
- Your agent runs in your AWS account (Lambda, ECS, EC2, or on-prem)
- The OpenTelemetry SDK batches spans and exports them to the AWS Distro for OpenTelemetry (ADOT) Collector
- The ADOT Collector forwards spans to AgentCore Evaluations via AWS PrivateLink
- AgentCore processes spans, computes metrics, and stores results
- You query results via the AgentCore API or console
The ADOT Collector runs as a sidecar container or daemon process. It handles batching, retries, and credential management. You configure it with a YAML file:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
awsxray:
region: us-east-1
agentcore:
endpoint: agentcore.us-east-1.amazonaws.com
region: us-east-1
service:
pipelines:
traces:
receivers: [otlp]
exporters: [awsxray, agentcore]The agentcore exporter is a custom plugin that ships with ADOT. It handles authentication via IAM roles and forwards spans to the AgentCore service.
Latency and Cost Overhead
Shipping telemetry to AgentCore adds latency and cost:
- Latency: 10-50ms per agent run, depending on span volume and network conditions. The ADOT Collector batches spans, so the overhead is amortized across multiple runs.
- Cost: AgentCore charges per span ingested. AWS hasn’t published pricing yet, but expect $0.10-$0.50 per million spans based on similar services (X-Ray, CloudWatch Logs Insights).
If you’re running high-throughput agents (thousands of runs per second), the cost can add up. You can reduce it by:
- Sampling telemetry (e.g., only export 10% of runs)
- Filtering spans (e.g., only export tool calls and LLM calls, skip internal framework spans)
- Running evals asynchronously (export telemetry to S3, process in batch)
For local development or low-stakes testing, you can run evals in-process using the AgentCore SDK. It computes metrics locally without shipping telemetry to AWS. This is faster and cheaper, but you lose the centralized dashboard and historical comparison.
Security Boundaries
AgentCore Evaluations runs in AWS’s account, not yours. Your telemetry data crosses an account boundary, which raises two questions:
- Data residency: Spans are stored in the AWS region you specify, but they’re not in your VPC. If you have strict data residency requirements, you’ll need to run your own evaluation stack.
- Sensitive data: Spans can include tool inputs, LLM prompts, and user queries. If these contain PII or secrets, you need to scrub them before export. The ADOT Collector supports span processors that can redact attributes, but you have to configure them yourself.
AWS encrypts spans in transit (TLS) and at rest (KMS). You can use customer-managed KMS keys if you want full control over encryption.
Failure Modes
Here are the most likely failure modes:
- Missing telemetry: If your agent doesn’t emit the required spans or attributes, AgentCore can’t compute metrics. The service doesn’t fail gracefully. It just returns empty results.
- Schema drift: If you change your agent’s tool signatures or add new tools, the evaluation metrics may become inconsistent over time. You’ll need to version your evaluation datasets and re-run historical evals.
- Collector downtime: If the ADOT Collector crashes or loses network connectivity, spans are buffered in memory. If the buffer fills up, spans are dropped. You won’t know about it unless you monitor the collector’s own telemetry.
- Rate limiting: AgentCore has undocumented rate limits on span ingestion. If you exceed them, spans are rejected with HTTP 429 errors. The ADOT Collector will retry, but you may lose data if the backlog grows too large.
Technical Verdict
Use AgentCore Evaluations if:
- You run agents on multiple frameworks and want a single evaluation dashboard
- You already use OpenTelemetry for observability and want to reuse the same telemetry pipeline
- You need historical comparison and trend analysis across agent versions
- You’re willing to pay for a managed service and accept the latency/cost overhead
Avoid it if:
- You have strict data residency requirements or can’t send telemetry outside your VPC
- You run high-throughput agents and need sub-10ms latency
- Your framework doesn’t emit OpenTelemetry telemetry and you don’t want to instrument it manually
- You need custom evaluation metrics that AgentCore doesn’t support (e.g., domain-specific accuracy measures)
For teams running heterogeneous agent stacks, the framework-agnostic contract is a real win. But you’re trading flexibility for operational complexity. You’ll need to manage the ADOT Collector, monitor span ingestion, and handle instrumentation gaps yourself.