Plover: How Plan-Centric Steering Keeps GUI Agents on Track When Interfaces Change

Vision-based GUI agents fail predictably. A modal dialog appears mid-task. A layout shifts. A loading spinner times out. The agent drifts, clicks the wrong element, or loops indefinitely. Most systems respond by re-prompting the LLM or restarting from scratch, discarding all prior progress.

Plover takes a different approach. It externalizes the task plan as a persistent, editable artifact that both the agent and the user can modify during execution. When the interface changes, you don’t re-prompt. You edit the plan, add a recovery step, or annotate the screenshot to steer the agent back on track.

This is plan-centric interaction: the plan becomes the control surface.

The Core Problem with Opaque Planning

Most GUI agents keep their plans internal. The LLM generates a sequence of actions, the executor runs them, and if something breaks, you get a retry or a full re-plan. You can’t inspect intermediate state. You can’t surgically fix a single step. You can’t preserve the work already done.

Plover’s architecture splits planning from execution and makes both visible:

Planner: Generates a structured task plan from the user’s natural language goal and the current screenshot.
Executor: Steps through the plan, taking actions (click, type, scroll) and observing the GUI state.
Plan Artifact: A persistent, revisable document that tracks completed steps, pending steps, and user edits.

When the executor hits an unexpected state (a dialog, a missing button, a timeout), it pauses. The plan remains intact. The user can edit the plan, insert a recovery step, or provide natural language guidance without losing prior progress.

Plan Representation and State Management

Plover represents plans as a sequence of steps, each with:

Action: The operation to perform (click, type, navigate).
Target: A natural language description of the UI element (e.g., “the blue ‘Submit’ button in the lower right”).
Grounding: A bounding box or screenshot annotation linking the target to the visual state.
Status: Pending, in-progress, completed, or failed.

The plan is stored as a structured JSON object, but rendered to the user as an editable list. Each step is anchored to a screenshot, so you can see what the agent saw when it planned that action.

When the executor completes a step, it updates the plan’s status and captures a new screenshot. If a step fails, the plan pauses at that step. The user can:

Edit the step: Change the target description or action type.
Insert a new step: Add a recovery action (e.g., “close the dialog by clicking the X button”).
Provide guidance: Annotate the screenshot with a bounding box or natural language hint.
Replan from here: Ask the planner to regenerate subsequent steps based on the current state.

The key insight: localized repair is cheaper and more reliable than full re-planning. If the agent gets stuck on step 7 of a 12-step plan, you fix step 7. Steps 1 through 6 remain completed. Steps 8 through 12 may still be valid.

Recovery Primitives

Plover exposes three recovery mechanisms:

Mechanism	When to Use	State Preserved	User Effort
Step Edit	Minor target mismatch (button moved, label changed)	All prior steps, rest of plan	Low (text edit)
Step Insertion	Unexpected dialog, new required action	All prior steps, rest of plan	Medium (write new step)
Localized Replan	Multiple steps invalidated, but goal unchanged	All prior steps	Medium (approve new steps)

Step edits are the lightest touch. The user changes the target description or action, and the executor retries. This handles cases where the interface layout shifted but the logical flow remains the same.

Step insertion handles unexpected interruptions. A cookie banner appears. A confirmation dialog blocks progress. The user inserts a step to dismiss it, and the agent continues.

Localized replanning regenerates steps from the failure point forward. The planner sees the current screenshot, the original goal, and the completed steps. It produces a new sequence that picks up where the agent left off.

Architecture Flow

class Plover:
    def __init__(self, planner_model, executor):
        self.planner = planner_model  # LLM-based planner
        self.executor = executor      # Vision-based action executor
        self.plan = None
        self.screenshots = []
        
    def execute_task(self, goal: str, initial_screenshot):
        # Generate initial plan
        self.plan = self.planner.generate_plan(goal, initial_screenshot)
        self.screenshots.append(initial_screenshot)
        
        for step in self.plan.steps:
            if step.status == "completed":
                continue
                
            # Attempt execution
            result = self.executor.execute_step(step)
            
            if result.success:
                step.status = "completed"
                self.screenshots.append(result.screenshot)
            else:
                # Pause and wait for user intervention
                step.status = "failed"
                step.error = result.error
                return self.wait_for_user_edit(step)
                
    def wait_for_user_edit(self, failed_step):
        # User can:
        # 1. Edit the step target or action
        # 2. Insert a new recovery step before this one
        # 3. Request localized replan from this step forward
        # 4. Provide screenshot annotation or natural language hint
        pass

The executor is stateless. It receives a step, attempts the action, and returns success or failure. The plan holds all state. This makes rollback trivial: revert the plan to a prior snapshot, and the executor picks up from there.

Grounding and Vision

Plover uses vision-language models to ground actions in the GUI. Each step’s target description is matched to the current screenshot using a multimodal embedding or object detection model. The executor returns a bounding box for the matched element.

When the user edits a step, they can:

Provide a new natural language description.
Draw a bounding box directly on the screenshot.
Highlight the correct element if the agent selected the wrong one.

The executor re-grounds the action using the updated target. This handles cases where the interface changed but the user can visually identify the correct element.

Observability and Debugging

Every step execution produces:

Screenshot: The GUI state when the action was attempted.
Grounding: The bounding box of the target element.
Action Log: The low-level operation (click coordinates, typed text).
Result: Success, failure, or timeout.

These artifacts are stored alongside the plan. You can replay the execution, inspect each decision, and understand why the agent chose a particular element.

This is critical for debugging. When an agent clicks the wrong button, you need to see what it saw. Plover captures the visual context, the grounding logic, and the action result in a single trace.

Failure Modes and Limitations

Plover assumes the user can recognize when the agent drifts and knows how to correct it. This works well for semi-autonomous workflows where a human is supervising. It breaks down for fully autonomous tasks where no one is watching.

The system also assumes the plan structure remains valid. If the user’s goal changes mid-execution, localized repair won’t help. You need a full replan or a new goal.

Vision grounding is still brittle. If the interface uses custom controls, non-standard layouts, or dynamic content, the executor may fail to locate the target element even with user annotations.

Finally, plan editing requires mental overhead. The user must understand the task structure, recognize which step failed, and know how to fix it. This is easier than writing automation scripts from scratch, but harder than clicking through the GUI manually.

Comparison to Checkpoint-and-Retry

Traditional GUI automation uses checkpoints: save state, attempt a sequence of actions, and rollback on failure. Plover’s plan-centric approach differs in three ways:

Granularity: Checkpoints are coarse (save every N steps). Plans are fine-grained (every step is a checkpoint).
Editability: Checkpoints are opaque. Plans are human-readable and editable.
Recovery: Checkpoints require re-execution from the saved state. Plans allow localized repair without re-running prior steps.

The trade-off: plan-centric systems require more user interaction. Checkpoint-and-retry is fully autonomous but less adaptable.

When to Use Plan-Centric Steering

Plover makes sense when:

Interfaces change frequently: Layouts shift, dialogs appear, elements move.
Tasks are long: Multi-step workflows where re-running from scratch is expensive.
Supervision is available: A human can intervene when the agent drifts.
Transparency matters: You need to inspect and debug agent decisions.

Avoid it when:

Tasks are short: Single-step actions don’t benefit from plan editing.
Interfaces are stable: If the GUI never changes, simpler automation works fine.
Full autonomy is required: No one is available to edit plans mid-execution.
Speed is critical: Plan editing adds latency compared to blind retries.

Technical Verdict

Plan-centric interaction solves a real problem: GUI agents drift when interfaces change, and opaque re-planning discards progress. Externalizing the plan as an editable artifact gives you localized repair, transparent decision-making, and fine-grained control.

The architecture is straightforward: split planning from execution, persist the plan as structured data, and expose edit operations. The hard part is vision grounding and user experience. You need reliable element detection and a UI that makes plan editing feel natural.

Use Plover’s approach when you’re building semi-autonomous GUI agents for dynamic interfaces. Skip it if you need fully autonomous execution or if your interfaces are stable enough for traditional scripting.

The core insight holds: when agents fail, localized repair beats full re-planning. Making the plan visible and editable is the simplest way to enable that.

Agentic AI, minus the noise.