Most teams building multi-agent systems write the same adapter code twice: once when they prototype a single agent, then again when they scale to fleets. The missing abstraction is the agent harness, the layer that wraps raw model API calls with state persistence, tool routing, retry logic, and observability hooks before handing control to an orchestrator.
The Practical AI podcast recently broke down this terminology gap in an episode titled “Models, Harnesses, and Multi-Agent Systems.” The conversation clarifies what belongs in the harness versus what belongs in the orchestrator, and why conflating the two creates vendor lock-in and brittle deployments.
What an Agent Harness Actually Does
An agent harness sits between the LLM API and your orchestration layer. It owns:
- State management: Conversation history, context windows, and session persistence across turns.
- Tool routing: Mapping function calls from the model to actual code execution, with schema validation.
- Retry and fallback: Handling rate limits, timeouts, and model errors without bubbling raw exceptions to business logic.
- Observability hooks: Logging prompts, completions, token counts, and latency before the orchestrator sees results.
- Model abstraction: Normalizing API differences between OpenAI, Anthropic, local models, and custom endpoints.
The harness does not decide which agent runs next, how to split tasks, or when to terminate a workflow. That responsibility belongs to the orchestrator.
Harness vs. Orchestrator Boundaries
| Layer | Responsibility | Failure Mode |
|---|---|---|
| Model API | Raw inference, token generation | Rate limit, timeout, malformed JSON |
| Agent Harness | State, tool calls, retries, logging | Tool schema mismatch, context overflow |
| Orchestrator | Task routing, agent selection, workflow termination | Deadlock, infinite loops, resource exhaustion |
When you skip the harness and call model APIs directly from orchestration logic, you end up with:
- Retry logic scattered across every agent definition.
- Tool schemas duplicated in multiple places.
- No single place to swap models without rewriting workflows.
- Observability that only captures orchestrator decisions, not individual agent behavior.
Practical Implementation Shape
A minimal harness wraps a model client and exposes a single run() method that accepts a prompt, tool definitions, and state. It returns a normalized response with tool calls, text output, and updated state.
class AgentHarness:
def __init__(self, model_client, tools, state_store):
self.client = model_client
self.tools = tools
self.state = state_store
def run(self, prompt, session_id):
# Load conversation history
history = self.state.get(session_id)
messages = history + [{"role": "user", "content": prompt}]
# Call model with retry logic
response = self._call_with_retry(messages, self.tools)
# Execute tool calls if present
if response.tool_calls:
results = [self._execute_tool(tc) for tc in response.tool_calls]
messages.append({"role": "assistant", "tool_calls": response.tool_calls})
messages.extend([{"role": "tool", "content": r} for r in results])
response = self._call_with_retry(messages, self.tools)
# Persist updated state
self.state.save(session_id, messages)
return response
def _call_with_retry(self, messages, tools):
for attempt in range(3):
try:
return self.client.chat(messages, tools=tools)
except RateLimitError:
time.sleep(2 ** attempt)
raise MaxRetriesExceeded()
def _execute_tool(self, tool_call):
func = self.tools[tool_call.name]
return func(**tool_call.arguments)This pattern keeps orchestrators clean. They call harness.run() and get back structured results without worrying about API quirks or state management.
State Persistence Without Leakage
The harness owns conversation history, but it should not own workflow state. If an orchestrator is routing tasks across three agents, the harness for each agent tracks its own conversation, but the orchestrator tracks which agent ran last and what the next step is.
State leakage happens when you store orchestration decisions inside the harness. For example, if the harness decides “this task is done, move to the next agent,” you have tightly coupled the harness to a specific workflow. Now you cannot reuse that harness in a different multi-agent system.
Instead, the harness returns a completion signal (like done: true or next_action: "escalate"), and the orchestrator interprets it.
Tool Routing and Schema Validation
Tool calls are the most common source of runtime errors in agent systems. The model returns a JSON blob with a function name and arguments, and your code has to map that to actual execution.
The harness validates tool schemas before execution:
- Check that the function name exists in the tool registry.
- Validate argument types against a schema (Pydantic, JSON Schema, or similar).
- Catch exceptions during execution and return structured error messages to the model.
If validation fails, the harness can retry the model call with an error message in the conversation history, giving the model a chance to correct itself. This keeps orchestrators from handling malformed tool calls.
Vendor Lock-In at the Harness Layer
Switching models is easier when the harness abstracts API differences. OpenAI uses messages with role and content, Anthropic uses messages with role and content but different tool call formats, and local models often require custom tokenization.
The harness normalizes these differences. If you later swap OpenAI for a local Llama deployment, you change the model client inside the harness, not every orchestrator that calls it.
Lock-in happens when you skip the harness and call model APIs directly from orchestration logic. Now every workflow has OpenAI-specific code, and migrating to a different provider means rewriting every agent.
Observability Hooks
The harness is the right place to log prompts, completions, token counts, and latency. Orchestrators should not care about these details.
Typical observability hooks in a harness:
- Log every prompt and completion to a structured log store (JSON, OpenTelemetry, or a database).
- Track token usage per session to detect context window overflows.
- Emit latency metrics for each model call to identify slow agents.
- Capture tool call success and failure rates.
If you wait until the orchestrator to log this data, you lose visibility into individual agent behavior. You only see high-level workflow metrics, not which agent is burning tokens or which tool call is failing.
Error Recovery Without Bubbling Exceptions
The harness should handle transient errors (rate limits, timeouts) and return structured errors for permanent failures (invalid tool schema, context overflow).
Transient errors get retried with exponential backoff. Permanent errors get returned to the orchestrator as a structured response, not an exception. This keeps orchestrators from crashing when a single agent fails.
For example, if a tool call fails because the model hallucinated a function name, the harness logs the error, appends it to the conversation history, and retries the model call. If the model still fails after three attempts, the harness returns a response with error: "tool_not_found", and the orchestrator decides whether to retry with a different agent or escalate to a human.
When to Use a Harness
You need a harness layer when:
- You are running more than one agent in a workflow.
- You plan to swap models (open vs. closed, different providers).
- You need consistent observability across all agents.
- You want to reuse agents in different orchestration patterns.
You can skip the harness if:
- You have a single agent with no plans to scale.
- You are prototyping and do not care about observability yet.
- You are tightly coupled to a single model provider and have no plans to change.
Technical Verdict
Agent harnesses are the adapter layer that most production multi-agent systems need but few teams name explicitly. They decouple model APIs from orchestration logic, making it easier to swap models, add observability, and reuse agents across workflows.
Build a harness when you move from a single chatbot to a fleet of agents. Skip it if you are still prototyping or have no plans to scale beyond one agent.
The harness owns state persistence, tool routing, retry logic, and observability. The orchestrator owns task routing, agent selection, and workflow termination. Keep these boundaries clean, or you will rewrite the same adapter code every time you add a new agent.