Terminal-Bench: Why Agent Benchmarks Need Continuous Integration to Stay Valid

Coding agent benchmarks rot. Not because the tasks change, but because models discover valid solutions that verifiers reject. A benchmark carefully validated at release can become stale within months as agents find edge cases, exploit unintended shortcuts, or trigger infrastructure assumptions that no longer hold.

Terminal-Bench 2.0 shipped after extensive human and LM-assisted review. An audit found issues in roughly 30% of the benchmark. The repairs changed accuracy by over 10 percentage points and moved leaderboard positions by up to three ranks. The problem is not sloppy initial work. The problem is that benchmarks are software, and software needs maintenance.

The Maintenance Debt Problem

Benchmark maintenance debt accumulates in three layers:

Verifier staleness
Models produce correct solutions that fail automated checks. The verifier expects a specific file structure, but the agent writes equivalent code in a different layout. The verifier rejects it.

Infrastructure drift
Evaluation harnesses change. Dependency versions update. Timeout assumptions baked into tasks no longer match production resource budgets. A task designed for 60-second execution now needs 120 seconds because the model explores more paths.

Specification ambiguity
Tasks contain implicit assumptions that become visible only when models probe edge cases. A “create a user profile” task might not specify whether duplicate usernames should error or overwrite. Both behaviors are defensible, but the verifier only accepts one.

These issues compound. A stale verifier produces false negatives. Teams tune agents to pass the verifier instead of solving the task. The benchmark measures verifier compliance, not capability.

Continuous Benchmark Architecture

Terminal-Bench treats evaluation as a living system with version-controlled artifacts and a repair pipeline:

Artifact versioning
Tasks, verifiers, agent traces, and oracle checks live in a Git repository. Each benchmark version is a tagged release. Historical comparisons reference specific commit hashes, not floating “latest” pointers.

Community reporting
Users submit issues when agents produce valid solutions that fail verification. Each report includes the agent trace, the failing test, and a human explanation of why the solution is correct.

Automated oracle checks
Before merging a repair, the system runs the updated verifier against a corpus of historical agent outputs. If the change flips verdicts on previously passing solutions, the repair is flagged for human review.

Human-gated repairs
A maintainer reviews the trace, the proposed fix, and the oracle results. If the repair is valid, they merge it and cut a new benchmark version. The old version remains available for historical comparisons.

This workflow mirrors software CI/CD. Benchmarks are not static datasets. They are codebases with tests, issues, pull requests, and release cycles.

Implementation Shape

The Terminal-Bench pipeline has five stages:

Agent execution: Run the agent against the task. Capture stdout, stderr, filesystem changes, and API calls.
Verifier run: Execute the test suite. Collect pass/fail signals and error messages.
Trace storage: Write the full execution trace to object storage with a content-addressed key.
Issue triage: When a user reports a problem, load the trace and compare it against the task specification.
Repair validation: Run the updated verifier against a sample of historical traces. Check for regressions.

Each stage is idempotent. You can rerun the verifier against old traces without rerunning the agent. This separation is critical for debugging. When a verifier changes, you need to know which historical results it would have affected.

class BenchmarkVersion:
    def __init__(self, commit_hash, tasks, verifiers):
        self.commit = commit_hash
        self.tasks = tasks
        self.verifiers = verifiers
    
    def evaluate(self, agent_trace):
        task = self.tasks[agent_trace.task_id]
        verifier = self.verifiers[task.verifier_id]
        return verifier.check(agent_trace, task.spec)
    
    def validate_repair(self, new_verifier, historical_traces):
        regressions = []
        for trace in historical_traces:
            old_result = self.evaluate(trace)
            new_result = new_verifier.check(trace, trace.task_spec)
            if old_result != new_result:
                regressions.append((trace, old_result, new_result))
        return regressions

The key insight is that evaluate() is pure. Given a trace and a benchmark version, the result is deterministic. This makes it safe to rerun verifiers without burning agent quota.

Resource Budgets as Specification

Terminal-Bench found that wall-clock limits materially change results. Extending the timeout from 60 to 120 seconds closed a 20-point accuracy gap on GPT-4o and shrank a 30-point gap on GPT-4o mini to 5 points.

This is not a bug. It is a specification problem. The benchmark measures “what can an agent do in 60 seconds,” not “what can an agent do, given infinite time.” When you change the timeout, you change the question.

Resource budgets are part of the measurement:

Resource	Impact on Results	Validation Strategy
Wall-clock time	10-30pp accuracy swing	Profile agent execution, set timeout to 95th percentile
Token budget	Determines search depth	Track cumulative tokens per task, flag outliers
API rate limits	Forces sequential execution	Measure parallelism, document concurrency assumptions
Memory limits	Affects state retention	Monitor peak memory, set limit to 90th percentile

If you do not version resource budgets alongside tasks, you cannot reproduce results. A leaderboard entry from six months ago might have used different timeouts, making the comparison meaningless.

Failure Modes

Continuous benchmarks introduce new failure modes:

Version proliferation
Every repair creates a new benchmark version. Teams must decide whether to rerun all agents on the new version or maintain separate leaderboards. Both choices have costs.

Oracle brittleness
Automated checks can reject valid repairs if the historical trace corpus is not representative. A repair that fixes one edge case might break another.

Coordination overhead
Human-gated repairs require maintainer attention. If the issue queue grows faster than the review rate, the benchmark falls behind.

Leaderboard instability
Frequent updates change rankings. Teams optimizing for leaderboard position might chase moving targets instead of improving core capabilities.

The mitigation is to batch repairs into quarterly releases and maintain a stable “frozen” version for official comparisons. The continuous pipeline runs in the background, but public leaderboards update on a fixed schedule.

Observability Requirements

To debug benchmark staleness, you need visibility into three layers:

Task-level metrics
Track pass rate, average execution time, and token consumption per task. A sudden drop in pass rate signals a verifier problem. A spike in execution time suggests infrastructure drift.

Verifier-level metrics
Count false positives (valid solutions rejected) and false negatives (invalid solutions accepted). High false positive rates indicate stale verifiers.

Agent-level metrics
Measure retry behavior, error recovery, and tool call patterns. If agents are retrying the same action repeatedly, the verifier might be rejecting valid outputs.

These metrics feed into a dashboard that flags anomalies. When a task’s pass rate drops by more than 10 points between versions, the system creates an issue for human review.

Security Boundaries

Continuous benchmarks execute untrusted agent code. The evaluation harness must isolate execution:

Run agents in ephemeral containers with no network access
Limit filesystem writes to a sandboxed directory
Cap CPU and memory usage to prevent resource exhaustion
Scrub agent outputs before storing traces to remove secrets

The verifier runs in a separate container. If an agent compromises the execution environment, it cannot tamper with the verifier or access other agents’ traces.

Technical Verdict

Use continuous benchmarks when:

You are evaluating coding agents or other systems that produce complex, verifiable outputs
Your benchmark will be used for more than six months
You have the engineering capacity to maintain a CI/CD pipeline
You need reproducible results across model versions

Avoid continuous benchmarks when:

Your evaluation is a one-time research experiment
Tasks have unambiguous, stable success criteria (e.g., multiple-choice questions)
You cannot commit to ongoing maintenance
Leaderboard stability is more important than measurement accuracy

The core trade-off is maintenance cost versus measurement validity. Static benchmarks are cheaper to run but decay over time. Continuous benchmarks stay valid but require infrastructure and human attention. For production agent deployments, the investment is worth it. For academic research, it depends on your timeline.

Agentic AI, minus the noise.