mech.app

The mech.app newsletter

Agentic AI, minus the noise.

Get practical field notes on AI agents, automation, developer tools and security delivered to your inbox.

No spam. Unsubscribe anytime.

AI Agents

Cutting Agent Token Costs by 71%: How Amazon Bedrock AgentCore Handles Multi-Megabyte JSON Without Exploding the Context Window

Three composable strategies for managing large JSON payloads in production agents: S3 offloading, on-demand paging, and dictionary compaction.

Source: aws.amazon.com
Cutting Agent Token Costs by 71%: How Amazon Bedrock AgentCore Handles Multi-Megabyte JSON Without Exploding the Context Window

Production agents hit a wall when they start processing real API responses, database dumps, and structured data. A single tool call can return megabytes of JSON. The context window fills up, token costs spike, and latency climbs as the model churns through irrelevant keys.

Amazon Bedrock AgentCore now documents three composable strategies that cut per-turn token use by up to 71% and latency by up to 80%. These are not model tricks. They are orchestration patterns that change how you wire tools, manage state, and handle large payloads before they reach the model.

The Problem: Tools That Return Everything

Most agent tools return complete JSON objects. A database query returns all columns. An API call returns the full response body. A file read returns the entire document.

The orchestrator serializes this JSON into the prompt, the model processes it, and you pay for every token. When a single tool call returns 50 KB of JSON, you burn thousands of tokens on data the agent never references.

Worse, the model slows down. Processing large context windows increases time to first token. The agent waits longer for a response that ignores 90% of the input.

Strategy One: S3 Artifact Offloading

Instead of returning large payloads inline, tools write them to S3 and return a reference.

How it works:

  1. The tool executes (database query, API call, file read).
  2. If the response exceeds a threshold (configurable, typically 10 KB), the tool writes the payload to S3.
  3. The tool returns a small JSON object with the S3 URI and a summary.
  4. The orchestrator injects this reference into the prompt.
  5. If the agent needs the full payload, it calls a retrieval tool with the URI.

Orchestration impact:

  • Tools need access to an S3 bucket and write permissions.
  • The orchestrator must provide a retrieval tool in the agent’s toolset.
  • State management becomes explicit: the agent tracks which artifacts it has seen and which it needs.

When it fails:

  • The agent must recognize when it needs full data versus a summary. If the model hallucinates that the summary is sufficient, it never fetches the artifact.
  • S3 latency adds round trips. If the agent needs the data immediately, you pay the fetch cost upfront.
  • Artifacts expire. If the agent references a URI in a later turn and the object is gone, the conversation breaks.

Strategy Two: On-Demand Result Paging

For list-heavy responses (search results, table rows, log entries), return the first page and a continuation token.

How it works:

  1. The tool executes and generates a full result set.
  2. The tool returns the first N items (configurable, typically 10-20) and a cursor or page token.
  3. The orchestrator injects this partial result into the prompt.
  4. If the agent needs more, it calls a paging tool with the token.
  5. The orchestrator fetches the next page and appends it to the conversation history.

State management:

  • The orchestrator must track active cursors across turns.
  • If the agent abandons a query mid-page, the cursor expires (configurable TTL).
  • Retry logic gets complex: if a page fetch fails, does the orchestrator retry transparently or surface the error to the agent?

When it fails:

  • The agent must understand pagination semantics. If it assumes the first page is complete, it makes decisions on incomplete data.
  • Cursor invalidation breaks multi-turn queries. If the underlying data changes between pages, results become inconsistent.
  • Token savings depend on the agent stopping early. If it always fetches all pages, you add orchestration overhead with no benefit.

Strategy Three: Dictionary Compaction

For JSON with repetitive keys (common in API responses and database rows), compress the structure before serialization.

How it works:

  1. The tool returns a standard JSON object.
  2. A middleware layer (runs in the orchestrator, not the tool) analyzes the structure.
  3. If the JSON contains arrays of objects with identical keys, the middleware converts it to a columnar format: one array per key instead of one object per row.
  4. The orchestrator serializes the compacted structure into the prompt.
  5. The model processes fewer tokens because key names appear once instead of per-row.

Example transformation:

{
  "original": [
    {"id": 1, "name": "Alice", "email": "alice@example.com"},
    {"id": 2, "name": "Bob", "email": "bob@example.com"},
    {"id": 3, "name": "Carol", "email": "carol@example.com"}
  ],
  "compacted": {
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Carol"],
    "email": ["alice@example.com", "bob@example.com", "carol@example.com"]
  }
}

Where it runs:

  • Client-side: the tool itself compacts before returning. Simple but couples the tool to the orchestrator’s expectations.
  • Orchestrator middleware: a transformation layer between tool execution and prompt serialization. Flexible but adds latency.
  • Model provider: some providers support compressed formats natively. Rare and non-portable.

When it fails:

  • Semantic search breaks. If the agent needs to reason about relationships between fields in a single row, columnar format obscures the structure.
  • Nested objects complicate compaction. The middleware must decide how deep to recurse.
  • Models trained on standard JSON may perform worse on compacted structures. Test before deploying.

Composability and Trade-Offs

These strategies stack. You can offload large artifacts to S3, page through result sets, and compact the first page before serialization.

StrategyToken ReductionLatency ImpactImplementation ComplexityFailure Mode
S3 Offloading60-80% for large payloads+200-500ms per fetchMedium (S3 permissions, retrieval tool)Agent ignores artifacts, URIs expire
Result Paging40-70% for lists+100-300ms per pageHigh (cursor tracking, retry logic)Agent assumes first page is complete
Dictionary Compaction20-40% for repetitive JSON+10-50ms per transformLow (middleware hook)Model struggles with columnar format

Combined impact:

AWS reports 71% token reduction and 80% latency reduction in production workloads. These numbers assume:

  • Most tool responses exceed 10 KB (offloading triggers frequently).
  • Agents stop after the first page 60% of the time (paging saves tokens).
  • JSON contains repetitive keys (compaction applies).

Your mileage will vary. If your tools return small, unique payloads, compaction does nothing. If your agents always need full data, offloading adds latency without savings.

Implementation Sketch

Here’s how the orchestrator wires these strategies together:

class AgentOrchestrator:
    def __init__(self, s3_bucket, offload_threshold_kb=10, page_size=20):
        self.s3_bucket = s3_bucket
        self.offload_threshold = offload_threshold_kb * 1024
        self.page_size = page_size
        self.active_cursors = {}

    def execute_tool(self, tool_name, params):
        # Tool returns raw result
        result = self.tools[tool_name].execute(params)
        
        # Strategy 1: Offload large payloads
        if len(json.dumps(result)) > self.offload_threshold:
            uri = self.upload_to_s3(result)
            return {
                "type": "artifact_reference",
                "uri": uri,
                "summary": self.summarize(result),
                "size_bytes": len(json.dumps(result))
            }
        
        # Strategy 2: Page large lists
        if isinstance(result, list) and len(result) > self.page_size:
            cursor = self.create_cursor(result)
            self.active_cursors[cursor] = result[self.page_size:]
            return {
                "type": "paged_result",
                "items": result[:self.page_size],
                "cursor": cursor,
                "total_count": len(result)
            }
        
        # Strategy 3: Compact repetitive structures
        if self.is_compactable(result):
            result = self.compact(result)
        
        return result

    def compact(self, data):
        if not isinstance(data, list) or len(data) == 0:
            return data
        
        # Convert array of objects to columnar format
        keys = data[0].keys()
        return {key: [row[key] for row in data] for key in keys}

The orchestrator intercepts tool responses, applies strategies based on payload characteristics, and rewrites the response before serialization. The agent sees a transformed result but calls tools using the same interface.

Observability Gaps

These strategies introduce new failure modes that standard agent observability misses:

  • Artifact fetch failures: The agent requests an S3 URI but the object is gone. Does your telemetry distinguish between tool errors and retrieval errors?
  • Cursor expiration: The agent tries to fetch page 3 but the cursor expired. Does your trace show why the page is unavailable?
  • Compaction errors: The middleware fails to compact a nested structure and falls back to raw JSON. Does your cost tracking flag the fallback?

You need separate metrics for:

  • Offload rate (percentage of tool calls that trigger S3 upload).
  • Fetch rate (percentage of artifacts the agent actually retrieves).
  • Page depth distribution (how many pages agents fetch per query).
  • Compaction savings (token delta before and after transformation).

Without these, you cannot tune thresholds or diagnose why token costs remain high.

Security Boundaries

S3 offloading changes the trust model. Tools now write to shared storage, and the agent reads from it.

New attack surfaces:

  • Artifact poisoning: A malicious tool writes a crafted payload to S3. The agent fetches it in a later turn and processes malicious data.
  • URI leakage: The agent logs or returns an S3 URI to the user. If the bucket is misconfigured, the user gains access to other artifacts.
  • Cursor hijacking: An attacker guesses or enumerates cursor tokens and fetches pages from other sessions.

Mitigations:

  • Use pre-signed URLs with short expiration (5-10 minutes).
  • Namespace artifacts by session ID and agent ID.
  • Encrypt cursors and validate them server-side before fetching.
  • Audit S3 access logs for unexpected retrieval patterns.

Technical Verdict

Use these strategies when:

  • Your tools regularly return payloads larger than 10 KB.
  • Token costs are a measurable line item (more than $500/month).
  • Latency is acceptable at p95 (you can tolerate 200-500ms fetch overhead).
  • Your agents can handle partial data and request more when needed.

Avoid them when:

  • Your tools return small, unique payloads (compaction and offloading add overhead without savings).
  • Your agents must see complete data to reason correctly (paging introduces incompleteness).
  • You lack observability into artifact fetches and cursor expiration (you will debug blind).
  • Your orchestrator does not support middleware hooks (you must rewrite every tool).

The 71% token reduction is real, but it assumes your workload matches AWS’s profile: large, repetitive JSON from tools that agents rarely need in full. Measure your own payload distribution before committing to the implementation complexity.