Most agent safety work treats the model and the orchestration layer as separate problems. You either fine-tune the policy to refuse harmful requests or you add guardrails to the harness that wraps it. SafeEvolve from Mao et al. argues that real safety failures emerge from the interaction between these layers during multi-step execution, and proposes a bilevel optimization loop where both the harness and the policy learn from shared agent experience.
The core insight: an agent can fail safely in two places. It can generate a harmful final response, or it can execute a dangerous intermediate tool call that looks benign in isolation but violates safety boundaries when chained with other actions. Existing alignment methods target one layer or the other. SafeEvolve updates both simultaneously from on-policy trajectories.
Architecture: Two-Level Learning from Trajectories
SafeEvolve runs a continual loop:
- Experience collection: The agent executes tasks using its current harness and policy, generating complete trajectories with safety labels.
- Harness evolution: Trajectory-level safety evidence updates the orchestration layer (safety prompts, hierarchical skill definitions).
- Policy evolution: The base model learns to use the evolved harness through supervised fine-tuning, then refines safety behavior through RL with verifier-decomposed rewards.
- Deployment: The co-evolved harness-policy pair replaces the previous version.
The harness side converts trajectory outcomes into component-level updates. If an agent trajectory violates a safety boundary, SafeEvolve traces the failure back to specific orchestration decisions: which tool was called, which skill was invoked, which prompt template was active. It then updates those components in a bounded, auditable way. Each harness artifact is versioned, and updates are reversible.
The policy side follows a two-stage training regime:
- Harness-use SFT: The model learns to actively query and follow harness artifacts (safety prompts, skill definitions) during execution. This bootstraps the policy to treat the harness as a runtime resource, not just a static wrapper.
- Harness-augmented RL: The model explores multi-step tasks with verifier-decomposed rewards. Instead of a single reward at trajectory end, SafeEvolve decomposes safety into per-step signals using a learned verifier. This shapes behavior during exploration, not just at terminal states.
State Representation and Replay
The harness needs to capture enough execution state to learn which paths lead to violations before the final response is generated. SafeEvolve represents each trajectory as a sequence of (state, action, harness_artifact, safety_label) tuples. The state includes:
- Current task context
- Tool call history
- Active skill definitions
- Safety prompt version
When a trajectory violates safety, the system replays it to identify the earliest decision point where the harness could have intervened. This replay mechanism is critical: it allows the harness to learn preemptive boundaries, not just post-hoc filters.
The replay process generates training data for both layers. For the harness, it produces (state, artifact_update) pairs that map execution contexts to orchestration changes. For the policy, it produces (state, action, reward) tuples where the reward reflects both task success and safety compliance.
Bilevel Optimization Trade-offs
SafeEvolve models safety as a bilevel problem:
- Upper level (harness): Optimize orchestration artifacts to minimize safety violations across all trajectories.
- Lower level (policy): Optimize model behavior to maximize task success subject to harness constraints.
This structure creates a coordination game. The harness learns to provide useful safety boundaries. The policy learns to respect those boundaries while still completing tasks. The key is that both layers update from the same experience, so they co-adapt rather than fighting each other.
| Component | Update Mechanism | Training Signal | Deployment Risk |
|---|---|---|---|
| Safety prompts | Trajectory-level evidence → prompt template updates | Violation traces, benign task success | Prompt drift breaks existing tool interfaces |
| Hierarchical skills | Failure analysis → skill boundary refinement | Multi-step execution logs | Skill version conflicts across agent instances |
| Policy SFT | Harness-use demonstrations → supervised fine-tuning | (state, harness_query, action) tuples | Model forgets how to use old harness versions |
| Policy RL | Verifier-decomposed rewards → exploration shaping | Per-step safety + utility signals | Reward hacking if verifier is misaligned |
The bilevel structure introduces versioning complexity. When you update the harness, you need to retrain the policy to use the new artifacts. When you update the policy, you need to verify it still respects harness boundaries. SafeEvolve handles this by maintaining a synchronized version registry: each harness-policy pair is tagged with a joint version ID, and deployment rolls out both components atomically.
Implementation: Harness Artifact Updates
The harness evolution step converts trajectory-level safety labels into component-level updates. Here’s the conceptual flow:
def evolve_harness(trajectories, current_harness):
violations = [t for t in trajectories if t.safety_label == "unsafe"]
# Trace each violation to harness decision points
update_candidates = []
for traj in violations:
for step in traj.steps:
if step.harness_artifact in current_harness:
# Identify which artifact failed to prevent violation
update_candidates.append({
"artifact": step.harness_artifact,
"context": step.state,
"failure_mode": traj.violation_type
})
# Cluster similar failures and generate bounded updates
updates = cluster_and_bound(update_candidates)
# Apply updates with version tracking
new_harness = current_harness.copy()
for update in updates:
new_harness.apply(update, version=current_harness.version + 1)
return new_harnessThe cluster_and_bound function ensures updates are localized. If a safety prompt needs to be more restrictive, the update only affects that prompt, not the entire orchestration layer. This keeps changes auditable and reversible.
Policy Training: Harness-Use SFT and Verifier-Augmented RL
The policy evolution step trains the model to actively use harness artifacts during execution. The SFT phase uses demonstrations where the agent successfully queries safety prompts or skill definitions before taking actions:
# Harness-use SFT training example
def generate_sft_examples(trajectories, harness):
examples = []
for traj in trajectories:
for step in traj.steps:
if step.harness_query:
# Model learns to query harness before acting
examples.append({
"input": step.state,
"harness_query": step.harness_query,
"action": step.action,
"label": "safe" if traj.safety_label == "safe" else "unsafe"
})
return examplesThe RL phase uses a verifier to decompose trajectory-level safety into per-step rewards. The verifier is a learned model that predicts whether a partial trajectory will lead to a violation. This allows the policy to receive feedback during exploration, not just at the end:
def compute_step_reward(state, action, verifier, task_reward):
# Verifier predicts safety of partial trajectory
safety_score = verifier.predict(state, action)
# Combine task success and safety into single reward
return task_reward * safety_scoreThe verifier itself is trained on completed trajectories with known safety labels. It learns to recognize execution patterns that lead to violations, even if the violation doesn’t occur until many steps later.
Versioning and Deployment
Co-evolving the harness and policy creates a versioning problem. If you deploy a new harness without retraining the policy, the model won’t know how to use the new artifacts. If you deploy a new policy without updating the harness, the model might bypass safety boundaries.
SafeEvolve solves this with synchronized versioning:
- Each harness-policy pair gets a joint version ID.
- Deployment rolls out both components atomically.
- Old versions remain available for rollback.
- Agent instances check version compatibility at startup.
This prevents the failure mode where a new policy is deployed against an old harness, or vice versa. The trade-off is deployment complexity: you can’t update the harness independently of the policy, so every safety improvement requires a full retraining cycle.
Failure Modes and Observability
SafeEvolve introduces several new failure modes:
- Harness drift: Repeated updates can make the harness overly restrictive, blocking benign tasks. The paper addresses this with bounded updates and benign utility tracking.
- Policy-harness desync: If the policy learns to ignore harness artifacts, the safety guarantees collapse. The SFT phase is designed to prevent this by explicitly training harness-use behavior.
- Verifier misalignment: If the verifier used for RL rewards is itself unsafe, the policy will learn to exploit it. SafeEvolve trains the verifier on human-labeled trajectories to mitigate this.
- Version conflicts: If multiple agent instances run different harness-policy versions, they may have inconsistent safety boundaries. The synchronized versioning system prevents this but adds deployment overhead.
Observability requirements:
- Log every harness artifact query and policy decision.
- Track which version of the harness-policy pair each agent instance is running.
- Monitor benign utility alongside safety metrics to detect harness drift.
- Maintain a replay buffer of trajectories for post-hoc analysis.
Experimental Results
The paper evaluates SafeEvolve on AgentDojo, a benchmark for agentic safety. For Qwen3.5-4B, SafeEvolve achieves a 3x reduction in attack success rate (ASR) while improving benign utility from 59.79% to 61.86%. This is the key result: most safety methods trade off utility for safety, but SafeEvolve improves both by teaching the policy to use the harness effectively.
The improvement comes from two sources:
- Harness evolution: The orchestration layer learns to block dangerous tool call sequences before they execute.
- Policy evolution: The model learns to query the harness proactively, not just react to guardrails.
The paper also shows that harness-only or policy-only updates underperform. Updating the harness without retraining the policy leads to the model ignoring new safety prompts. Updating the policy without evolving the harness leads to the model learning brittle, context-specific safety behaviors that don’t generalize.
Technical Verdict
Use SafeEvolve when:
- You need safety alignment across multi-step agent execution, not just final responses.
- You can afford the deployment complexity of synchronized harness-policy versioning.
- You have access to on-policy trajectories with safety labels (human feedback or automated verification).
- Your orchestration layer is modular enough to support component-level updates.
Avoid SafeEvolve when:
- You need to update safety boundaries without retraining the base model (use pure harness-based guardrails instead).
- Your agent tasks are single-step or have simple tool call patterns (the co-evolution overhead isn’t justified).
- You can’t maintain a synchronized version registry across distributed agent instances.
- Your safety requirements are static and well-defined upfront (pre-training alignment may be simpler).
The core trade-off: SafeEvolve converts runtime experience into both orchestration improvements and model behavior changes, but it couples the harness and policy in a way that increases deployment complexity. If you can manage the versioning overhead, you get safety alignment that adapts to real execution patterns rather than relying on pre-defined rules or static fine-tuning.