Multirun: How Running Ten Isolated Agents in Parallel Becomes the Cheapest Eval You'll Ever Run

Denis Kubilyov ran the same task through ten isolated Claude Code agents simultaneously. Each agent got its own git worktree and its own Postgres branch. All ten agents agreed completely on the schema. Then they split 5/5 on two decisions nobody explicitly asked them to make.

That split is the eval. No judge model, no golden dataset, no hand-labeled ground truth. Just ten parallel runs and a diff.

The Problem: One Run Is an Anecdote

Ask a coding agent to do the same non-trivial thing twice and you get two different implementations. Both fine, both defensible. With a single run you read the diff, decide if you like it, and move on. The actual question never surfaces: what would it have done the other nine times?

At n=1 you cannot see the distribution. At n=2 you get a comparison, not a pattern. What Denis wanted was to know where the model lands reliably and where it quietly flips a coin on your behalf.

Traditional evals require infrastructure: a harness, a judge model, a benchmark suite, a score to optimize. That setup takes a week you do not have. Multirun sidesteps all of it by treating consensus and divergence across parallel runs as the signal.

Architecture: Isolated Environments Per Agent

Multirun provisions ten isolated execution contexts in parallel. Each agent gets:

• Dedicated git worktree: separate working directory, same underlying repository
• Dedicated Postgres branch: isolated schema and data, powered by Neon’s branching API
• Separate filesystem namespace: no shared state between runs

The framework spawns all ten agents at once, waits for completion, then diffs their outputs. No agent sees what another agent did. No cross-contamination.

Isolation Mechanics

Git worktrees share the .git directory but maintain separate working trees. This means:

• Disk usage stays low (objects are shared)
• Each agent can commit, branch, and modify files independently
• Cleanup is a single git worktree remove per agent

Postgres branches use Neon’s copy-on-write branching. Each branch starts as a snapshot of the parent database. Writes diverge immediately. The parent remains untouched.

# Simplified Multirun orchestration sketch
import asyncio
from neon_api import create_branch
from git import Repo

async def spawn_agent(task, agent_id, base_branch):
    # Provision isolated environment
    db_branch = await create_branch(
        parent=base_branch,
        name=f"agent-{agent_id}"
    )
    worktree_path = f"/tmp/worktree-{agent_id}"
    repo.git.worktree("add", worktree_path, "HEAD")
    
    # Run agent with isolated context
    result = await run_claude_agent(
        task=task,
        db_url=db_branch.connection_string,
        working_dir=worktree_path
    )
    
    return {
        "agent_id": agent_id,
        "db_branch": db_branch,
        "worktree": worktree_path,
        "result": result
    }

async def multirun(task, n=10):
    base_branch = "main"
    runs = await asyncio.gather(*[
        spawn_agent(task, i, base_branch)
        for i in range(n)
    ])
    
    # Diff all outputs
    schema_diffs = compare_schemas([r["db_branch"] for r in runs])
    code_diffs = compare_worktrees([r["worktree"] for r in runs])
    
    return aggregate_consensus(schema_diffs, code_diffs)

Diffing Strategy: Consensus and Divergence

After all ten agents finish, Multirun compares their outputs across two dimensions:

1. Schema convergence: did all agents produce the same table structure, column types, constraints?
2. Implementation divergence: where did agents make different choices on questions the prompt did not specify?

In Denis’s test case, all ten agents agreed on the schema. Perfect consensus. Then they split 5/5 on two undocumented decisions. That split reveals the model’s uncertainty boundary. The prompt did not constrain those choices, so the model sampled from its prior.

This is cheaper than human labeling because you are not asking “is this output correct?” You are asking “do ten independent runs agree?” Agreement implies the model has high confidence. Disagreement flags ambiguity in the task or the prompt.

Aggregating Signals Without Manual Review

Multirun does not require a human to read ten diffs. It automates the aggregation:

• Full consensus (10/10): high confidence, likely correct
• Strong majority (8+/10): model prefers one path but has some variance
• Split decision (5/5 or 6/4): prompt ambiguity or model uncertainty
• Scattered outputs: task is underspecified or too hard for the model

The framework outputs a summary table showing which decisions converged and which split.

Decision Point	Consensus	Interpretation
Schema design	10/10	Model is confident, prompt is clear
Index strategy	5/5	Prompt did not specify, model is guessing
Naming convention	7/3	Weak preference, consider clarifying prompt
Error handling	10/10	Model has strong prior, likely correct

This table becomes your eval report. No judge model required.

Resource Limits and Cleanup

Running ten agents in parallel can exhaust compute, storage, or API rate limits. Multirun handles this with:

• Concurrency cap: configurable max parallel agents (default 10, tunable down to 3)
• Timeout per agent: kill runaway processes after N minutes
• Automatic cleanup: delete worktrees and database branches after diffing completes
• Rate limit backoff: retry with exponential backoff if API calls fail

Cleanup is critical. Without it, you accumulate orphaned worktrees and database branches. Multirun registers cleanup handlers that run even if the orchestrator crashes.

async def cleanup_agent(agent_context):
    # Remove git worktree
    repo.git.worktree("remove", agent_context["worktree"], force=True)
    
    # Delete Postgres branch
    await neon_api.delete_branch(agent_context["db_branch"].id)
    
    # Clear temp files
    shutil.rmtree(agent_context["temp_dir"], ignore_errors=True)

Failure Modes

Multirun exposes several failure modes you should plan for:

1. Agent timeout: one agent hangs, blocking the entire run. Solution: per-agent timeout with partial result aggregation.
2. Database branch quota: Neon limits concurrent branches per project. Solution: queue agents if you hit the limit.
3. Disk exhaustion: ten worktrees fill /tmp. Solution: use a dedicated volume or stream diffs without keeping full worktrees.
4. Non-deterministic outputs: agents produce different results on identical reruns. This is signal, not noise. It means the task or model is inherently stochastic.

The fourth failure mode is the point. If you rerun Multirun on the same task and get different consensus patterns, your prompt is underspecified or the model is sampling heavily.

When to Use Multirun

Use Multirun when:

• You need to eval agent reliability without building eval infrastructure
• You want to know where your prompt is ambiguous
• You are debugging why an agent gives different answers on identical tasks
• You need to measure model confidence on a specific task type

Avoid Multirun when:

• You need deterministic, reproducible outputs (use temperature=0 and a single run)
• Your task has one correct answer and you already know it (use a traditional eval)
• You cannot afford the cost of ten parallel API calls
• Your task modifies external state that cannot be isolated (e.g., sending emails, charging credit cards)

Technical Verdict

Multirun is a practical eval primitive for agent workflows. It trades compute cost (ten parallel runs) for human cost (no labeling, no judge model). The consensus/divergence signal is crude but actionable.

Use it if:

• Your agent task is deterministic enough that consensus across ten runs indicates correctness, and you have budget for parallel API calls
• You need to identify prompt ambiguities faster than manual review cycles allow
• Your task produces structured outputs (schemas, code, configs) that can be diffed programmatically
• You have access to isolated execution environments (git worktrees, database branches, containerized filesystems)
• The cost of ten API calls is lower than the cost of a single human eval pass

Avoid it if:

• Your task requires deterministic output or regulatory compliance that prohibits stochastic variation
• Isolation overhead (worktree provisioning, branch creation, cleanup) exceeds the cost of running a single eval
• Your task has side effects that cannot be rolled back or isolated (payments, emails, external API mutations)
• You need ground-truth correctness validation rather than confidence measurement
• Your infrastructure cannot support concurrent database branches or parallel git worktrees

The core trade-off is compute for human time. If ten parallel runs cost less than one hour of human eval work, and your task can be isolated cleanly, Multirun is the fastest way to find out where your agent is guessing.

---

Source Links

• Multirun: A Multi-Agent Run Framework (dev.to)
• a2wio/multirun (GitHub repository)