RoboArchitect: LLM-Generated Agent Harnesses for Robot Control

Every new robot benchmark requires hand-coded plumbing. A frozen policy that picks and places objects needs a harness: control code that decides when to query the vision model, when to command the arm directly, when to consult persistent memory, and when to retry a failed grasp. Today that harness is bespoke. RoboArchitect automates the design of this orchestration layer, using an LLM to watch rollouts, diagnose failures, and iteratively rewrite the harness without touching the policy weights.

The result is a 4.5x success improvement on unseen scenes and 3.2x on unseen tasks, purely from better plumbing.

The Harness Problem

A robot policy is a function: observations in, actions out. But real tasks need more:

• Tool boundaries: Should the harness call a vision API to detect obstacles, or trust the policy’s internal perception?
• Retry logic: If the gripper misses, should the harness back up and try again, or hand control back to the policy?
• Persistent memory: If the robot learns that grasping 2 cm deeper works better for mugs, where does that knowledge live between episodes?

Current practice is to write this harness by hand for each benchmark. RoboArchitect treats harness design as a code-generation problem. A coding model observes rollouts, identifies failure modes (jammed arm, blocked approach path), and rewrites the harness to route around them.

Architecture

RoboArchitect operates in three phases:

1. Rollout observation: The frozen policy runs inside a minimal harness. The system records trajectories, success/failure labels, and environment state.
2. Failure diagnosis: The LLM analyzes failed episodes, identifying patterns (e.g., “arm collides when approaching from above”).
3. Harness revision: The LLM generates new Python code that wraps the policy, adding control flow (if obstacle detected, approach horizontally), memory reads/writes, and tool calls.

The harness is optimized on a training set of episodes, then evaluated on held-out scenes and tasks. No policy weights change.

Harness Primitives

The generated harness can use:

• Vision tools: Object detection, depth estimation, obstacle maps.
• Memory API: Key-value store persisted across episodes (e.g., memory.set("mug_grasp_depth", 0.02)).
• Direct arm commands: Lift, rotate, or retract without querying the policy.
• Policy invocation: Call the frozen model when the harness delegates a decision.

The LLM decides which primitives to expose and when to invoke them.

Persistent Memory Design

RoboArchitect’s harnesses evolve their own memory schemas. Example from the paper:

# Auto-generated harness snippet
def grasp_object(obj_type, policy):
    # Check if we've learned a better depth for this object
    depth = memory.get(f"{obj_type}_grasp_depth", default=0.01)
    
    # Query policy for approach vector
    approach = policy.plan_grasp(obj_type, depth)
    
    # Execute and record outcome
    success = execute_grasp(approach)
    
    if success:
        # Reinforce this depth
        memory.set(f"{obj_type}_grasp_depth", depth)
    else:
        # Try deeper next time
        memory.set(f"{obj_type}_grasp_depth", depth + 0.005)
    
    return success

Memory keys are strings. Values are floats or small JSON blobs. The harness writes after each episode and reads before the next. No external database; the memory API is a simple dict persisted to disk.

State Corruption Risks

Because memory persists across episodes, a bad write can poison future runs. RoboArchitect mitigates this by:

• Scoped keys: Memory keys include object type and task context, reducing cross-contamination.
• Rollback on catastrophic failure: If success rate drops below a threshold, the system reverts the last memory update.
• Bounded values: Numeric memory entries are clamped (e.g., grasp depth between 0 and 0.05 m).

The paper does not describe transactional semantics or conflict resolution if multiple episodes write concurrently. In practice, episodes run sequentially during optimization.

Evaluation Strategy

RoboArchitect cannot deploy every candidate harness to a physical robot. Instead:

1. Simulation first: Candidate harnesses run in a simulator (the paper uses environments from RLBench and Metaworld).
2. Held-out split: Training episodes optimize the harness. Evaluation episodes test generalization to new scenes and tasks.
3. Success threshold: A harness is accepted if it improves success rate on the validation set by at least 5 percentage points.

The LLM generates 3-5 candidate harnesses per iteration. The best performer on validation data becomes the new baseline for the next round.

Failure Mode Discovery

The LLM identifies failure modes by parsing rollout logs and environment state. Common patterns:

Failure Mode	Harness Fix
Arm jammed against table	Add lift_clear() before horizontal motion
Obstacle blocks top approach	Switch to horizontal reach if depth sensor detects blockage
Gripper misses small objects	Increase grasp depth and slow approach speed
Policy oscillates near goal	Add termination condition when position error < 1 cm

The harness evolves control strategies (lift, rotate, retry) that the frozen policy cannot express.

Performance Gains

Across benchmarks, RoboArchitect improved success rates:

• Unseen scenes: 9.7% → 43.3% (4.5x)
• Unseen tasks: 12.7% → 40.2% (3.2x)

Gains were largest when the benchmark diverged from the policy’s training data. Wrapping a general-purpose policy with an evolved harness often outperformed a finetuned specialist.

Example: A policy trained on tabletop pick-and-place struggled with cluttered shelves (12% success). The evolved harness added obstacle detection and horizontal reach, raising success to 38%. Finetuning the policy on shelf data reached 34%.

Code Generation Details

The LLM (GPT-4 in the paper) receives:

• Policy API: Function signatures for policy.plan_grasp(), policy.move_to(), etc.
• Tool API: Vision and memory primitives.
• Rollout logs: Timestamped observations, actions, and failure annotations.
• Current harness code: The Python file to revise.

Output is a new Python file. The system parses it, runs static checks (syntax, type hints), and executes it in simulation. If the harness crashes or violates safety bounds (e.g., commands outside workspace), the LLM receives the error trace and tries again.

The paper does not specify token budgets or retry limits. In practice, 2-3 revision rounds per iteration were typical.

When Harness Evolution Fails

RoboArchitect struggles when:

• Policy is fundamentally broken: If the policy cannot grasp at all, no harness will fix it.
• Simulation diverges from reality: Harnesses optimized in sim may fail on real hardware due to unmodeled dynamics.
• Memory schema grows unbounded: After 50+ episodes, memory can accumulate hundreds of keys, slowing lookups and confusing the LLM.

The paper recommends periodic memory pruning (delete keys unused for 10 episodes) and sim-to-real validation on a small physical testbed before full deployment.

Deployment Shape

A production deployment might look like:

1. Offline optimization: Run RoboArchitect in simulation for 100 episodes, evolving the harness.
2. Validation: Test the final harness on 20 real-world episodes.
3. Deployment: Freeze the harness code and memory snapshot. Deploy to the robot fleet.
4. Monitoring: Log success rates. If performance degrades, trigger a new optimization cycle.

The harness is Python code, so it can be version-controlled, reviewed, and rolled back. Memory is a JSON file, snapshotted alongside the code.

Observability Gaps

The paper does not address:

• Trace visualization: How do you inspect why the harness chose to lift instead of reach?
• Memory provenance: Which episode wrote a given memory key, and why?
• Failure attribution: Did the policy fail, or did the harness make a bad routing decision?

A production system would need structured logging (trace IDs, decision trees) and a dashboard showing harness control flow alongside policy actions.

Technical Verdict

Use RoboArchitect when:

• You have a frozen policy that works in some settings but fails in others.
• Finetuning is expensive or impractical (limited data, long training cycles).
• The failure modes are observable in simulation or cheap real-world rollouts.
• You can tolerate 100+ episodes of optimization time.

Avoid it when:

• The policy is fundamentally unsuited to the task (no amount of plumbing will help).
• Simulation fidelity is poor (harness will overfit to sim artifacts).
• You need explainability or safety guarantees (generated code is hard to audit).
• Memory requirements are unbounded (long-running deployments will accumulate state).

RoboArchitect exposes a key insight: much of the performance gap between lab demos and production robots lives in the orchestration layer, not the policy weights. Automating that layer is a practical win, but it shifts the debugging burden from model training to code generation and state management.

Source Links

• Paper: RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory
• ArXiv ID: 2610.42141v1