Cost per run is the wrong number. A cheap call that fails, gets retried, and ends with the user giving up and opening a ticket costs more than an expensive call that works. The number that matters is cost per resolved task.
Getting to it needs attribution at the run level, and the levers that move it are not the ones people reach for first.
The Naive Accounting Problem
Most teams start with token counts multiplied by API prices. This tells you what a single LLM call costs. It does not tell you:
- How many retries happened before the task succeeded
- How many partial completions were abandoned
- Which phase of the agent loop consumed the budget
- Whether prompt caching actually saved money or just shifted it
When you measure cost-per-run instead of cost-per-resolved-task, you optimize for the wrong thing. You end up with agents that run cheap but fail often, or that succeed but take three attempts to get there.
Instrument the Agent Loop, Not Just the API
The first step is tracking cost at the phase level. An agent typically moves through plan, retrieve, act, and summarize. Without phase attribution, you know a run cost 14 cents. With it, you discover the summarize phase is resending the entire conversation history on every turn.
export class Budget {
private spentUsd = 0;
readonly calls: CallCost[] = [];
constructor(
readonly capUsd: number,
private onExceed: () => never
) {}
record(u: Usage, model: string, phase: Phase) {
const usd = price(model, u);
this.spentUsd += usd;
this.calls.push({
model,
phase,
usd,
in: u.input,
out: u.output,
cachedIn: u.cacheRead ?? 0
});
if (this.spentUsd > this.capUsd) this.onExceed();
}
get spent() {
return this.spentUsd;
}
}The phase field is what makes this useful. The first report usually shows something surprising, and it is usually the same thing: the summarize phase resending the entire history.
Separate Cached Input from Fresh Input
Cached input is typically an order of magnitude cheaper than uncached. Mixing them makes the numbers meaningless. Track cache reads, cache writes, and fresh input separately.
export function price(model: string, u: Usage): number {
const p = PRICES[model];
return (
(u.input - (u.cacheRead ?? 0)) / 1e6 * p.in +
(u.cacheRead ?? 0) / 1e6 * p.cacheRead +
(u.cacheWrite ?? 0) / 1e6 * p.cacheWrite +
u.output / 1e6 * p.out
);
}This pricing function splits input tokens into fresh and cached. Without this split, you cannot tell whether enabling prompt caching reduced your bill or just moved cost from input to cache-write.
State You Need to Persist
To calculate cost-per-resolved-task across multi-turn sessions and async workflows, you need:
- Task ID: A stable identifier that survives retries and spans multiple agent runs
- Run ID: A unique identifier for each attempt, linked to the task
- Phase costs: Per-phase token usage and USD cost for each run
- Outcome: Success, failure, retry, or abandonment
- Timestamp: When the run started and ended
Store this in a time-series database or append-only log. You need to query it by task ID to sum costs across all runs that contributed to a single resolved task.
Four Levers That Change Unit Economics
These levers can be toggled without retraining or changing the agent’s external behavior:
| Lever | Mechanism | Typical Impact | Trade-off |
|---|---|---|---|
| Prompt caching | Cache system prompts and conversation history | 50-90% reduction in input token cost | Cache-write cost, cache invalidation complexity |
| Model routing | Use smaller models for plan/retrieve, larger for act | 30-60% reduction in total cost | Routing logic adds latency, risk of under-provisioning |
| Prompt compression | Summarize history, prune tool definitions | 20-40% reduction in input tokens | Information loss, summarization cost |
| Early stopping | Halt agent loop when confidence threshold is met | 10-30% reduction in unnecessary runs | Risk of premature termination |
Prompt Caching
Most agents resend the same system prompt and tool definitions on every turn. Prompt caching moves this cost from fresh input (expensive) to cache read (cheap). The first call pays a cache-write fee, but subsequent calls in the same session read from cache.
The failure mode is cache invalidation. If your system prompt changes between calls, you pay the cache-write cost again. If your cache TTL is too short, you pay it more often than you save.
Model Routing
The plan and retrieve phases often do not need the largest model. Routing these to a smaller model (GPT-4o-mini instead of GPT-4o, Claude Haiku instead of Sonnet) cuts cost without degrading output quality.
The failure mode is under-provisioning. If the smaller model cannot handle the task, you pay for both the failed small-model call and the retry with the large model.
Prompt Compression
Long conversation histories and verbose tool definitions bloat input tokens. Summarizing the history after every N turns and pruning unused tool definitions reduces input size.
The failure mode is information loss. If the summary drops a critical detail, the agent makes a mistake. If you prune a tool definition the agent needs, it cannot complete the task.
Early Stopping
If the agent reaches a confidence threshold (e.g., tool call succeeded, user confirmed, or internal validation passed), stop the loop. Do not run the summarize phase if the task is already resolved.
The failure mode is premature termination. If the confidence signal is noisy, you stop before the task is actually done.
Billable Runs vs Internal Retries
You need to separate billable runs (user-initiated, task-level) from internal retries (tool call failed, LLM returned invalid JSON, rate limit hit). Internal retries should not inflate your cost-per-resolved-task metric.
Tag each run with a billable flag. Set it to false for:
- Retries triggered by transient API errors
- Validation failures that do not consume user quota
- Internal health checks and warmup calls
Set it to true for:
- User-initiated agent runs
- Scheduled tasks that count against quota
- Retries triggered by user input changes
When calculating cost-per-resolved-task, sum only billable runs.
Observability Shape
Your observability stack needs:
- Per-run cost breakdown: Phase, model, tokens, USD
- Per-task cost rollup: Sum of all runs that contributed to task resolution
- Cache hit rate: Percentage of input tokens served from cache
- Retry rate: Percentage of tasks that required more than one run
- Abandonment rate: Percentage of tasks started but not resolved
Expose these as time-series metrics. Alert when:
- Cost-per-resolved-task exceeds a threshold
- Cache hit rate drops below baseline
- Retry rate spikes
- Abandonment rate climbs
Deployment Shape
Run the budget tracker in the same process as the agent loop. Do not send cost data to a separate service after the fact. You need to enforce budget caps in real time, before the agent exceeds its quota.
Pass the Budget instance to every function that calls an LLM. Call budget.record() immediately after receiving the API response, before any retry logic runs.
const budget = new Budget(1.0, () => {
throw new Error("Budget exceeded");
});
const response = await llm.call(prompt, { model: "gpt-4o" });
budget.record(response.usage, "gpt-4o", "plan");If the budget cap is exceeded, the onExceed callback throws. Catch this at the task level and mark the task as failed due to budget exhaustion.
Likely Failure Modes
Cache thrashing: If your system prompt changes frequently, you pay cache-write cost on every call and never benefit from cache reads. Pin your system prompt version and invalidate the cache explicitly when you deploy a new version.
Model routing loops: If the small model fails and you retry with the large model, but the large model also fails, you pay for both. Set a max-retry limit and fail fast.
Summarization drift: If you summarize the conversation history after every turn, the summary drifts from the original conversation. The agent loses context and makes mistakes. Summarize less frequently or use a larger context window.
Premature stopping: If you stop the agent loop as soon as a tool call succeeds, but the tool returned partial data, the task is not actually resolved. Validate tool output before stopping.
Technical Verdict
Use this approach when:
- You are running agents in production and need to control costs
- Your agent loops involve multiple LLM calls, retries, and tool invocations
- You need to justify agent costs to finance or product teams
- You are optimizing for cost-per-outcome, not cost-per-call
Avoid this approach when:
- You are still prototyping and cost is not yet a constraint
- Your agent runs are one-shot (no retries, no multi-turn sessions)
- You do not have the instrumentation infrastructure to track per-phase costs
- Your agent workload is too low-volume to justify the observability overhead
The key insight is that cost-per-run is a vanity metric. Cost-per-resolved-task is the number that determines whether your agent feature is profitable. Instrument the loop, separate billable runs from retries, and optimize the four levers in order: caching first, routing second, compression third, early stopping last.