Coding agents get benchmarked on pristine GitHub issues. One prompt, one task, clear acceptance criteria. SWE-bench has become the standard, but it tests a fiction: users who write perfect specifications and never change their minds.
SWE-DIVE (October 2026, ArXiv 2610.30047v1) introduces dynamic user simulation to expose what happens when agents face real human behavior. Vague initial prompts. Follow-up clarifications. Mid-task scope changes. The kind of multi-turn coordination that breaks state management, exhausts context windows, and reveals whether an agent can actually track evolving requirements.
The One-Shot Evaluation Fiction
Traditional coding agent benchmarks work like this:
- Feed the agent a well-formed GitHub issue
- Let it run autonomously
- Check if the final code passes tests
This measures execution capability but ignores conversation plumbing. Real users don’t arrive with perfect specifications. They start with “make the login faster” and iterate toward “cache user sessions in Redis but only for premium accounts and log cache hits to DataDog.”
Static benchmarks miss:
- Context drift across 10+ turns when the user adds constraints
- Conflicting instructions when turn 8 contradicts turn 3
- Clarification loops when the agent asks questions the user can’t answer yet
- Scope creep detection when requirements expand beyond the original task
How SWE-DIVE Simulates Real Users
The framework introduces three orchestrated components:
1. User Simulator
Generates behavioral patterns that mimic real developer interactions:
- Vague starters: “The API is slow” instead of “Optimize the /users endpoint query”
- Incremental refinement: Adds constraints across multiple turns
- Requirement pivots: Changes direction mid-task based on agent proposals
- Ambiguous feedback: “That’s not quite right” without specifying what’s wrong
The simulator doesn’t follow a script. It adapts based on the agent’s responses, creating emergent conversation paths.
2. Conversation Validator
Evaluates whether each turn moves toward task completion:
- Checks if agent questions are relevant to current requirements
- Detects when the agent ignores previous user constraints
- Flags hallucinated assumptions not grounded in conversation history
- Validates that proposed solutions address the accumulated requirement set
This component exposes coordination failures invisible in one-shot evals. An agent might pass SWE-bench but fail SWE-DIVE by losing track of constraints introduced in turn 4.
3. Harbor Orchestration Environment
Wraps the agent, simulator, and validator in a test harness:
class HarborSession:
def __init__(self, agent, user_sim, validator):
self.agent = agent
self.user_sim = user_sim
self.validator = validator
self.conversation_state = []
self.requirement_graph = RequirementGraph()
def run_turn(self):
# User simulator generates next message
user_msg = self.user_sim.generate_message(
self.conversation_state,
self.agent.last_response
)
# Agent processes and responds
agent_response = self.agent.process(
user_msg,
context=self.conversation_state
)
# Validator checks alignment
validation = self.validator.check(
agent_response,
self.requirement_graph
)
# Update state
self.conversation_state.append({
'user': user_msg,
'agent': agent_response,
'valid': validation.passed,
'issues': validation.issues
})
# Update requirement graph
self.requirement_graph.merge(
user_msg.extract_requirements()
)
return validation.passedThe orchestrator maintains a requirement graph that tracks constraints, dependencies, and conflicts across turns. This is the plumbing that one-shot evals don’t need.
Failure Modes Exposed by Multi-Turn Evaluation
Context Window Exhaustion
Agents that rely on full conversation replay hit token limits around turn 12. Symptoms:
- Dropping early constraints to fit new messages
- Summarizing previous turns and losing precision
- Forgetting user preferences stated in turn 2
Mitigation requires selective context compression or external state storage.
Conflicting Instruction Reconciliation
When turn 8 says “use PostgreSQL” but turn 3 said “keep it in SQLite,” agents fail in predictable ways:
- Last-wins: Ignores earlier constraints
- First-wins: Ignores new requirements
- Confusion: Asks the user to resolve conflicts they don’t remember creating
Robust agents need conflict detection and explicit reconciliation prompts.
State Management Across Pivots
User changes direction: “Actually, let’s use FastAPI instead of Flask.” Agents must:
- Identify which previous work is now obsolete
- Preserve constraints that still apply (auth, logging, tests)
- Avoid restarting from scratch
SWE-DIVE measures whether agents can surgically update plans versus thrashing.
Clarification Loop Deadlocks
Agent asks: “Should I use async or sync endpoints?”
User replies: “Whatever is faster.”
Agent asks: “What’s your expected request volume?”
User replies: “I don’t know yet.”
Agents that can’t proceed with reasonable defaults get stuck. The benchmark tracks how many turns agents waste in unproductive clarification loops.
Architecture Comparison
| Component | SWE-bench | SWE-DIVE |
|---|---|---|
| Input | Single GitHub issue | Simulated user with behavioral model |
| Turns | 1 (autonomous execution) | 10-20 (interactive) |
| State tracking | None (stateless task) | Requirement graph + conversation history |
| Context management | Not tested | Explicitly measured |
| Failure detection | Test suite pass/fail | Per-turn validation + final test suite |
| Scope changes | Not possible | Injected by user simulator |
| Orchestration | Agent + sandbox | Agent + user sim + validator + sandbox |
What This Means for Agent Design
If you’re building coding agents for real users, SWE-DIVE exposes gaps that matter:
1. Explicit State Machines
Agents need conversation state separate from LLM context:
- Requirement graph (constraints, dependencies, conflicts)
- Task decomposition tree (what’s done, what’s blocked, what’s obsolete)
- User preference log (decisions made, options rejected)
This state must survive context window truncation.
2. Conflict Resolution Protocols
When new instructions contradict old ones:
class ConflictResolver:
def detect_conflicts(self, new_req, req_graph):
conflicts = req_graph.find_incompatible(new_req)
if conflicts:
return self.generate_clarification(
new_req, conflicts
)
return None
def generate_clarification(self, new, old):
return f"You previously asked for {old.summary}. " \
f"The new requirement ({new.summary}) conflicts. " \
f"Should I: (A) replace the old approach, " \
f"(B) keep both if possible, or " \
f"(C) ignore the new requirement?"Agents that silently pick a resolution strategy fail user trust tests.
3. Selective Context Compression
Full conversation replay doesn’t scale past 15 turns. Agents need:
- Requirement extraction: Pull constraints into structured state
- Decision logging: Track why choices were made
- Turn summarization: Compress non-requirement discussion
The LLM sees compressed history but can query full logs when needed.
4. Progress Checkpointing
When users pivot, agents must identify:
- What work remains valid
- What must be discarded
- What needs modification
Without checkpointing, agents restart from scratch or apply changes to obsolete code.
Observability Requirements
Multi-turn coordination needs instrumentation that one-shot tasks don’t:
- Requirement graph diffs per turn (what changed, what conflicts emerged)
- Context window utilization (when compression triggers, what gets dropped)
- Validation failure reasons (ignored constraint, hallucinated assumption, irrelevant question)
- Turn efficiency (productive progress vs. clarification loops)
These metrics expose coordination bottlenecks invisible in final test results.
Deployment Implications
Running SWE-DIVE-style evaluation in production:
Cost Structure
- Per-turn LLM calls: 3x overhead (agent + user sim + validator)
- State storage: Requirement graphs and conversation logs grow linearly
- Validation compute: Semantic similarity checks on every turn
Budget 3-5x the inference cost of one-shot evaluation.
Latency Constraints
User simulators add 200-500ms per turn for behavioral model inference. Validators add another 100-300ms. Total turn latency: 1-2 seconds beyond agent response time.
For real-time user interaction, run validation asynchronously and flag issues post-hoc.
Failure Mode Detection
SWE-DIVE catches coordination failures before code execution:
- Agent ignores user constraint (turn 6)
- Agent hallucinates requirement not in conversation (turn 9)
- Agent asks question already answered (turn 4)
This enables early intervention instead of waiting for test suite failures.
Technical Verdict
Use SWE-DIVE when:
- You’re building agents for iterative, conversational coding workflows
- Users frequently refine requirements across multiple turns
- You need to test state management and context handling under realistic conditions
- Your agent must handle vague initial prompts and scope changes
Avoid SWE-DIVE when:
- Your agent executes well-specified, one-shot tasks (SWE-bench is sufficient)
- You’re optimizing for autonomous execution without user interaction
- Evaluation budget can’t support 3-5x inference cost overhead
- Your agent operates in domains where requirements are fully specified upfront
The framework exposes plumbing that matters for real-world deployment but adds complexity and cost. If your users iterate and clarify, the investment pays off. If they write perfect specs, it’s overkill.
Source Links
- SWE-DIVE ArXiv Paper
- GitHub Repository (referenced in paper)