You can run a computer-use agent that books a flight, files a tax form, or transfers funds. But how do you programmatically verify it did the right thing?
OSReward, a new benchmark from researchers at HKU and collaborators, exposes the unsexy infrastructure gap: verifying agent success across heterogeneous platforms requires reward models that most teams don’t have. The paper introduces a standardized evaluation framework for vision-language models (VLMs) acting as judges of agent trajectories, and the results are sobering. Even state-of-the-art models show systematic leniency bias, labeling failed runs as successes.
This matters because you can’t improve what you can’t measure. Computer-use agents (CUAs) are moving from demos to production workflows in finance, customer service, and operations. Without reliable verification infrastructure, you’re flying blind on data curation, reinforcement learning, and production monitoring.
The Verification Problem
A CUA trajectory is a sequence of actions, states, and reasoning logs. Verifying success means answering: did the agent fulfill the task instruction?
Human-written verifiers don’t scale. You can’t hard-code selectors for every possible UI across web, desktop, and mobile platforms. Human annotators are expensive and slow. So teams increasingly use VLMs as judges.
But are VLM judges reliable enough? OSReward provides the first systematic answer.
The benchmark includes:
- Trajectories from diverse agent backbones executing human-verified instructions
- Multi-stage human annotation for ground-truth verdicts
- Cross-platform coverage (web, desktop, mobile)
- OSReward-Hard, a challenge set of genuinely difficult cases
- OSReward-Multi for fine-grained efficiency and alignment scoring
What the Evaluation Found
The most comprehensive evaluation of VLM judges to date reveals three problems:
-
Systematic leniency bias. Models mislabel failed runs as successes more often than the reverse. This is catastrophic for RL training, where false positives poison the reward signal.
-
Reliability vs. cost trade-off. The few models reliable enough to trust are too expensive to run at scale. Affordable open models trail far behind.
-
Partial success ambiguity. When an agent completes 4 of 5 steps correctly, most judges struggle to propagate the right reward signal back through the decision chain.
| Model Class | Accuracy | Cost per 1K Evals | Leniency Bias |
|---|---|---|---|
| Frontier closed (GPT-4V, Claude 3.5) | 85-88% | $15-25 | Moderate |
| Mid-tier closed (GPT-4o-mini) | 78-82% | $2-5 | High |
| Open models (LLaVA, Qwen-VL) | 65-72% | $0.10-0.50 | Very high |
| OS-Shepherd (35B) | 83% | $0.80 | Low |
Architecture: Reward Models for CUA Verification
A production-grade reward model for CUA verification needs three layers:
1. Trajectory ingestion
- Parse action logs, screenshots, and reasoning traces
- Normalize across platforms (Selenium, Playwright, native APIs)
- Handle missing or corrupted state snapshots
2. Verification logic
- Compare final state against task specification
- Check intermediate checkpoints for multi-step tasks
- Handle ambiguous outcomes (partial success, acceptable alternatives)
3. Reward signal generation
- Binary success/failure for evaluation and monitoring
- Fine-grained scores for RL training (per-step credit assignment)
- Confidence intervals to flag uncertain cases for human review
The key insight: trajectory verification for RL training needs different architecture than production monitoring. RL needs dense per-step rewards. Production needs fast binary verdicts with high precision on failures.
OS-Shepherd: Open Reward Models
To close the reliability gap, the team released OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments. They trained OS-Shepherd models (9B and 35B parameters) that approach frontier model accuracy at 1/20th the cost.
Training data structure:
{
"trajectory_id": "flight_booking_001",
"task_instruction": "Book a round-trip flight from SFO to JFK, departing March 15, returning March 22",
"agent_actions": [
{"step": 1, "action": "click", "target": "search_flights_button", "screenshot": "..."},
{"step": 2, "action": "type", "target": "origin_field", "value": "SFO", "screenshot": "..."},
# ... 18 more steps
],
"final_state": {
"booking_confirmed": true,
"origin": "SFO",
"destination": "JFK",
"departure_date": "2026-03-15",
"return_date": "2026-03-22"
},
"ground_truth_verdict": "success",
"reasoning": "All required fields match task specification. Confirmation number present.",
"failure_modes_checked": ["wrong_dates", "wrong_airports", "one_way_instead_of_roundtrip"]
}The reasoning annotations are critical. They teach the model why a trajectory succeeded or failed, not just the binary label.
Deployment Shapes
For RL training:
- Run reward model after every episode
- Use fine-grained per-step scores
- Batch inference to amortize VLM costs
- Cache verdicts for identical trajectories (surprisingly common in early training)
For production monitoring:
- Sample 5-10% of production trajectories
- Run lightweight binary classifier first (rule-based or small model)
- Escalate uncertain cases to full VLM judge
- Human review for high-stakes failures (financial transactions, compliance)
For data curation:
- Run reward model on all candidate trajectories
- Filter out false positives before adding to training set
- Use confidence scores to prioritize human review
- Track inter-annotator agreement between model and humans
Failure Modes
Leniency bias in action:
An agent books a flight but selects the wrong return date. The VLM judge sees a confirmation page and labels it success without checking date fields. This poisons your training data.
Mitigation: Explicitly prompt the judge to verify each task requirement. Include negative examples in few-shot context.
Cross-platform state inspection:
Task requires verifying a bank transfer. Agent completes the transfer in one app, but verification requires checking balance in another app. Single-screenshot judges fail.
Mitigation: Multi-modal trajectory representation. Include API responses, not just screenshots.
Ambiguous success criteria:
Task says “book a cheap flight.” Agent books a $300 flight when a $250 option existed. Is this success?
Mitigation: Quantify “cheap” in task specification. Use fine-grained scoring (0.0 to 1.0) instead of binary.
Observability Hooks
What to instrument:
- Verdict distribution over time. Sudden shift toward failures signals agent regression or environment change.
- Confidence scores. Low confidence on high-stakes tasks triggers human review.
- Failure mode breakdown. Track which failure modes are most common (wrong fields, incomplete actions, timeout).
- Judge latency. VLM inference can bottleneck RL training loops.
- Cost per trajectory. Monitor inference costs, especially for closed models.
Security Boundaries
Reward models see everything: screenshots, action logs, reasoning traces. This includes PII, credentials, and sensitive business data.
Isolation requirements:
- Run reward model inference in separate VPC from production agents
- Scrub PII from trajectories before logging
- Use ephemeral credentials for agent actions (rotate after each episode)
- Audit reward model access logs (who’s querying which trajectories)
Adversarial concerns:
An attacker who can manipulate the reward model can poison your RL training. If you’re fine-tuning on your own trajectories, validate that reward model weights haven’t been tampered with.
When to Build vs. Buy
Build your own reward model if:
- You have domain-specific success criteria (financial compliance, medical accuracy)
- You’re running RL training at scale (thousands of episodes per day)
- You need explainable verdicts for audit trails
Use a closed VLM judge if:
- You’re prototyping or running low-volume evaluation
- You can afford $15-25 per 1K trajectories
- You need highest accuracy and can tolerate leniency bias
Use OS-Shepherd or similar open models if:
- You need cost-effective verification at scale
- You can fine-tune on your own annotated trajectories
- You want to self-host for data privacy
Technical Verdict
Use OSReward and OS-Shepherd when:
- You’re building production CUA infrastructure and need standardized evaluation
- You’re training RL agents and need reliable reward signals at scale
- You’re curating training data and need to filter false positives
- You want to benchmark your own reward models against a public leaderboard
Avoid or defer when:
- You’re still in early prototyping (human eval is fine for <100 trajectories)
- Your tasks have simple, rule-based success criteria (DOM assertions, API checks)
- You don’t have budget for VLM inference or human annotation
- Your agents operate in a single platform with stable UI (hard-coded verifiers work)
The gap between running an agent and verifying it succeeded is the next infrastructure bottleneck. OSReward makes it measurable. OS-Shepherd makes it affordable. But you still need to design your verification architecture around your specific deployment shape, failure modes, and cost constraints.