When you deploy an LLM as an agent, the model weights are only half the system. The other half is the harness: prompts, tool definitions, control flow, memory management, and orchestration code. You can swap GPT-4 for Claude or Gemini and see minimal performance change if the harness is poorly designed. You can also take a weaker model and outperform a stronger one by tuning the plumbing.
HarnessOpt-Bench formalizes this reality as a benchmark. It measures how well an LLM can iteratively rewrite its own harness to improve downstream task performance under a fixed evaluation budget. The paper treats harness optimization as a distinct capability, separate from base model reasoning or prompt engineering.
What the Harness Includes
The harness is everything that wraps the model:
- Prompts: System messages, few-shot examples, chain-of-thought scaffolding.
- Tools: Function signatures, parameter schemas, tool selection logic.
- Control flow: Retry policies, branching conditions, loop guards.
- Memory: Context window management, retrieval triggers, state persistence.
- Orchestration code: The Python, TypeScript, or YAML that wires it all together.
In production, you tune these components iteratively. You run evals, identify failure modes, adjust the prompt or add a tool, and re-run. HarnessOpt-Bench asks: can an LLM do this loop autonomously?
The Benchmark Protocol
HarnessOpt-Bench gives an optimizer LLM three things:
- A seed harness for a target agent.
- Graded evaluation feedback from running that harness on a task.
- A fixed evaluation budget (number of target-agent runs allowed).
The optimizer edits the harness, submits candidates, receives feedback, and iterates. At the end, it nominates a final candidate. The benchmark scores that candidate on a held-out test partition the optimizer never sees during search.
A trusted execution environment enforces the evaluation boundary. It meters target-agent resource use, prevents the optimizer from peeking at test data, and preserves all candidate versions for audit. This prevents overfitting and ensures the optimizer genuinely improves generalization, not just memorizes the training partition.
Why This Is a Different Search Problem
Harness optimization is not hyperparameter tuning. You are not searching a fixed grid of floats. You are searching a combinatorial space of code, text, and configuration. Each candidate harness is a program, and the feedback is stochastic (same harness, different outcomes across runs).
It is also not pure prompt engineering. Changing a tool’s parameter schema or rewriting a retry loop is a code change, not a text change. The optimizer must reason about control flow, state transitions, and failure modes.
The benchmark reveals three challenges:
- Expensive evaluation: Each candidate costs tokens and latency. You cannot brute-force the search.
- Stochastic feedback: The same harness may pass on one run and fail on another. The optimizer must aggregate noisy signals.
- Compositional edits: Changing the prompt may require changing the tool set. Changing the tool set may require changing the control flow. Edits are interdependent.
Evaluation Results
The authors tested five frontier LLMs as optimizers across four downstream tasks. They ran 111 scored optimization runs, comparing models both under a shared coding harness and under their native harnesses (the environment each model ships with by default).
Key findings:
- Optimizer models separate more than coding harnesses. The choice of LLM matters more than the choice of harness framework. A strong model with a basic harness outperforms a weak model with a sophisticated harness.
- Native harnesses are not consistently superior. Models do not always perform better when using their own default orchestration code. Sometimes a neutral, shared harness levels the playing field.
- Gains vary substantially across tasks and seed regimes. Some tasks see 30%+ improvement from harness optimization. Others see single-digit gains. The quality of the seed harness matters: if the seed is already strong, optimization headroom shrinks.
Architecture: Trusted Execution Boundary
The benchmark enforces a hard boundary between the optimizer and the target agent. The optimizer cannot directly inspect the target’s runtime state, logs, or test data. It only receives:
- Aggregate scores (pass/fail, numeric metrics).
- Optional trace summaries (sanitized logs, error messages).
- Resource usage (token count, latency, tool call count).
This mirrors production constraints. In a real deployment, you do not have perfect observability into every agent run. You have metrics, logs, and cost data. The optimizer must infer what to change from indirect signals.
The execution environment also meters evaluation budget. If the optimizer is allowed 50 target-agent runs, it cannot exceed that limit. This forces the optimizer to balance exploration (trying diverse candidates) and exploitation (refining promising candidates).
Trade-Offs: Optimization Strategies
| Strategy | Strengths | Weaknesses |
|---|---|---|
| Greedy local search | Fast convergence, low eval budget | Gets stuck in local optima, ignores compositional changes |
| Random sampling | Explores diverse candidates, avoids bias | Wastes budget on low-quality candidates, slow convergence |
| Gradient-free optimization | Handles stochastic feedback, scales to large search spaces | Requires many evaluations, hard to tune hyperparameters |
| LLM-guided search | Reasons about code semantics, proposes plausible edits | Expensive per iteration, may hallucinate invalid code |
The benchmark does not prescribe a strategy. It measures the outcome: normalized gain over the seed harness on held-out test data.
Code Shape: Optimizer Harness Example
Here is a simplified optimizer loop. The real benchmark uses a trusted execution sandbox, but the control flow is similar:
class HarnessOptimizer:
def __init__(self, model, eval_budget):
self.model = model
self.eval_budget = eval_budget
self.candidates = []
def optimize(self, seed_harness, task_spec):
current = seed_harness
evals_used = 0
while evals_used < self.eval_budget:
# LLM proposes edit
edit_prompt = self.build_edit_prompt(current, self.get_feedback())
proposed = self.model.generate(edit_prompt)
# Validate and run
if self.is_valid_harness(proposed):
score = self.evaluate_harness(proposed, task_spec)
evals_used += 1
self.candidates.append((proposed, score))
# Update current if better
if score > self.get_best_score():
current = proposed
# Return best candidate
return max(self.candidates, key=lambda x: x[1])[0]
def build_edit_prompt(self, harness, feedback):
return f"""
Current harness:
{harness}
Recent feedback:
{feedback}
Propose an improved harness. Focus on:
- Tool selection logic
- Prompt clarity
- Error handling
- Memory management
"""The optimizer does not see test data. It only sees aggregated feedback from training runs. The evaluate_harness function runs the target agent in a sandboxed environment and returns a score.
Failure Modes
Harness optimization can fail in several ways:
- Overfitting to training partition: The optimizer tunes the harness to pass specific training examples but fails on held-out test cases.
- Invalid code generation: The optimizer proposes syntactically invalid harnesses or harnesses that crash at runtime.
- Premature convergence: The optimizer stops exploring too early and misses better candidates.
- Budget exhaustion: The optimizer wastes evaluations on low-quality candidates and runs out of budget before finding a good solution.
- Stochastic noise: The optimizer chases random variance instead of real signal, especially when evaluation feedback is noisy.
The benchmark’s held-out test partition and trusted execution boundary mitigate the first failure mode. The others require better optimizer design.
Observability Gaps
The benchmark exposes a tension in agent observability. To optimize a harness, you need feedback. But if you expose too much feedback (full traces, internal state, test data), the optimizer can overfit or cheat. The benchmark enforces a realistic observability boundary: aggregate metrics, sanitized logs, and resource usage.
In production, you face the same trade-off. You want rich telemetry for debugging, but you do not want to leak sensitive data or create overfitting opportunities. The benchmark’s design suggests a middle path: structured feedback that is informative but not exhaustive.
When Harness Optimization Matters
Harness optimization is most valuable when:
- You are deploying agents at scale and small performance gains compound.
- The task is well-defined and you can run many evaluations cheaply.
- The seed harness is mediocre and there is clear headroom for improvement.
- You have a trusted execution environment to enforce evaluation boundaries.
It is less valuable when:
- The task is poorly specified or evaluation is subjective.
- The seed harness is already near-optimal.
- Evaluation is expensive (e.g., requires human labeling or long-running simulations).
- You lack infrastructure to sandbox the optimizer and prevent overfitting.
Technical Verdict
Use HarnessOpt-Bench when you need to measure an LLM’s ability to improve its own orchestration code under realistic constraints. It is the first benchmark to formalize harness optimization as a distinct capability, separate from base model performance or prompt engineering. If you are building meta-agents (agents that tune other agents), this benchmark tells you which models can actually do the job.
Avoid it when you are optimizing a single, hand-crafted harness for a specific task. The benchmark measures general harness optimization capability, not task-specific tuning. If you just need to improve one agent, manual iteration with domain expertise will outperform automated optimization.
The benchmark also assumes you can run many evaluations. If your task requires expensive human feedback or long-running simulations, the fixed evaluation budget may be too restrictive.