AWS published a pattern for combining OpenAI-compatible SageMaker endpoints with Bedrock AgentCore runtime. The problem it solves: orchestrating multi-agent workflows where each specialized agent needs a different model, and the orchestration runtime doesn’t instrument all endpoints by default.
This matters when you want a reasoning-heavy agent to use Claude Sonnet, a code-generation agent to use a fine-tuned Llama variant on SageMaker, and a summarization agent to use a cheaper Haiku endpoint. AgentCore handles the routing, but you lose token-level observability unless you wire it yourself.
The Plumbing Problem
Bedrock AgentCore is an orchestration runtime. It manages task delegation, state transitions, and tool calls. When you point it at Bedrock-native models, you get built-in metrics: token counts, latency, errors. When you point it at SageMaker endpoints (even OpenAI-compatible ones), the runtime treats them as opaque HTTP targets. You get request success or failure, but no token-level telemetry.
This creates three operational gaps:
- Cost attribution: You can’t track which agent consumed how many tokens from which endpoint.
- Failure diagnosis: A 500 from SageMaker tells you nothing about whether the model hit a context limit, timed out internally, or failed validation.
- Rate limit coordination: If three agents share one SageMaker endpoint, you need to track token velocity to avoid throttling.
The AWS pattern addresses this by injecting observability at the SageMaker endpoint layer, not the AgentCore layer.
Architecture: Model Routing and Observability Injection
The workflow uses three components:
- AgentCore runtime: Orchestrates agents, delegates tasks, manages conversation state.
- SageMaker endpoints: Host different models behind OpenAI-compatible APIs.
- CloudWatch Logs + Lambda: Capture SageMaker invocation logs, parse token counts, emit custom metrics.
Each agent in AgentCore is configured with a modelId that maps to a SageMaker endpoint. When AgentCore invokes an agent, it sends the request to the corresponding endpoint. The endpoint logs the request and response (including token counts) to CloudWatch. A Lambda function tails those logs, extracts token metadata, and publishes it to CloudWatch Metrics with agent-specific dimensions.
# Lambda function to parse SageMaker logs and emit token metrics
import json
import boto3
cloudwatch = boto3.client('cloudwatch')
def lambda_handler(event, context):
for record in event['Records']:
log_data = json.loads(record['Sns']['Message'])
# Extract token counts from SageMaker response
tokens_in = log_data['requestParameters']['usage']['prompt_tokens']
tokens_out = log_data['requestParameters']['usage']['completion_tokens']
agent_id = log_data['requestParameters']['headers']['X-Agent-ID']
# Emit custom metrics
cloudwatch.put_metric_data(
Namespace='AgentCore/SageMaker',
MetricData=[
{
'MetricName': 'PromptTokens',
'Value': tokens_in,
'Dimensions': [{'Name': 'AgentID', 'Value': agent_id}]
},
{
'MetricName': 'CompletionTokens',
'Value': tokens_out,
'Dimensions': [{'Name': 'AgentID', 'Value': agent_id}]
}
]
)This pattern assumes your SageMaker endpoint returns OpenAI-compatible usage fields. If you’re hosting a custom model, you need to modify the inference container to include token counts in the response.
Model Selection Per Agent
AgentCore lets you define agents with different capabilities. The pattern shows three agents:
- Researcher: Uses Claude Sonnet 4 on Bedrock (native integration, automatic metrics).
- Code Generator: Uses a fine-tuned Llama 3.1 70B on SageMaker (custom endpoint, manual metrics).
- Summarizer: Uses Claude Haiku on SageMaker (cheaper than Bedrock for high-volume tasks, manual metrics).
Each agent’s configuration specifies its model endpoint:
agents:
- id: researcher
modelId: anthropic.claude-sonnet-4-20250514-v1:0
tools: [web_search, document_retrieval]
- id: code_generator
modelId: sagemaker://llama-3-1-70b-instruct-ft
tools: [code_execution, github_api]
- id: summarizer
modelId: sagemaker://claude-haiku-20250320
tools: []AgentCore routes tasks based on agent capabilities. If a task requires code generation, it delegates to the code generator agent, which hits the SageMaker Llama endpoint. If a task requires summarization, it delegates to the summarizer agent, which hits the SageMaker Haiku endpoint.
Authentication and Rate Limit Handling
Each SageMaker endpoint requires IAM credentials. AgentCore’s execution role needs sagemaker:InvokeEndpoint permissions for every endpoint it calls. The pattern uses a single execution role with a policy that grants access to all endpoints:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sagemaker:InvokeEndpoint",
"Resource": [
"arn:aws:sagemaker:us-east-1:123456789012:endpoint/llama-3-1-70b-instruct-ft",
"arn:aws:sagemaker:us-east-1:123456789012:endpoint/claude-haiku-20250320"
]
}
]
}Rate limits are harder. SageMaker endpoints have per-instance throughput limits. If you configure an endpoint with 2 instances of ml.g5.12xlarge, you get roughly 200 tokens/second total. If three agents share that endpoint, you need to track token velocity across all agents.
The pattern does not include a built-in rate limiter. You have two options:
- Separate endpoints per agent: Each agent gets its own SageMaker endpoint. Simple, but expensive.
- External rate limiter: Use a Lambda function or API Gateway in front of SageMaker to enforce per-agent quotas. Adds latency and complexity.
The AWS blog post recommends option 1 for production workloads.
Failure Modes and Observability Gaps
| Failure Mode | Detection Method | Mitigation |
|---|---|---|
| SageMaker endpoint throttling | CloudWatch ModelInvocationThrottles metric | Separate endpoints per agent, or add retry logic in AgentCore |
| Model context overflow | Parse SageMaker error response for “context_length_exceeded” | Truncate input in agent preprocessing, or switch to a larger context model |
| AgentCore task timeout | AgentCore emits TaskTimeout event | Increase task timeout in agent config, or split task into smaller subtasks |
| Missing token metrics | No CloudWatch metrics for agent ID | Verify SageMaker endpoint logs are enabled, check Lambda function permissions |
| Cross-agent state corruption | AgentCore conversation state shows incorrect agent responses | Enable AgentCore debug logging, inspect state transitions in CloudWatch Logs |
The biggest gap: AgentCore does not retry failed SageMaker invocations by default. If a SageMaker endpoint returns a 500, AgentCore marks the task as failed and moves on. You need to implement retry logic in the agent’s tool code or use Step Functions to wrap the AgentCore invocation.
Deployment Shape
The pattern assumes you already have:
- SageMaker endpoints running OpenAI-compatible models.
- Bedrock AgentCore configured with agent definitions.
- CloudWatch Logs enabled for SageMaker endpoints.
- Lambda function subscribed to SageMaker log streams.
The deployment sequence:
- Deploy SageMaker endpoints with logging enabled.
- Create Lambda function to parse logs and emit metrics.
- Subscribe Lambda to SageMaker CloudWatch log groups.
- Configure AgentCore agents with SageMaker endpoint ARNs.
- Test end-to-end workflow, verify metrics appear in CloudWatch.
The Lambda function adds 50-100ms of latency to metric publication, but does not block the SageMaker invocation. Metrics are eventually consistent.
Cost and Scaling Considerations
Running separate SageMaker endpoints per agent is expensive. A single ml.g5.12xlarge instance costs $7/hour. If you have five agents, that’s $35/hour or $25,200/month just for compute.
The pattern suggests three cost optimizations:
- Shared endpoints for low-traffic agents: Combine summarizer and researcher on one endpoint if their combined throughput stays under the instance limit.
- Serverless endpoints for bursty workloads: SageMaker Serverless Inference scales to zero when idle, but adds cold start latency (2-5 seconds).
- Bedrock for high-volume, low-latency tasks: If an agent doesn’t need a custom model, use Bedrock-native models to avoid SageMaker overhead.
The observability Lambda function costs $0.20 per million invocations. At 1,000 agent tasks per hour, you’ll pay $1.44/month for metrics.
Technical Verdict
Use this pattern when:
- You need different models for different agents (fine-tuned code model, reasoning model, summarization model).
- You already have SageMaker endpoints and want to reuse them in AgentCore workflows.
- You need token-level cost attribution across agents.
- You can afford separate SageMaker endpoints per agent (or accept the complexity of external rate limiting).
Avoid this pattern when:
- All agents can use Bedrock-native models (you get observability for free).
- You need sub-100ms agent response times (SageMaker adds network hops and cold start risk).
- You’re prototyping and don’t need production-grade metrics yet (start with Bedrock-only agents, migrate to SageMaker later).
- You need automatic retries and circuit breakers (AgentCore doesn’t provide them, you’ll need Step Functions or custom orchestration).
The pattern works, but it’s not turnkey. You’re trading Bedrock’s simplicity for SageMaker’s flexibility. If you don’t need custom models or fine-tuning, stick with Bedrock-native agents.
Source Links
- Building agentic workflows with SageMaker AI and Bedrock AgentCore (AWS Machine Learning Blog, Aug 14, 2026)