Craig McLuckie co-founded Kubernetes. Now he’s building ToolHive on top of the Model Context Protocol (MCP), and the architecture choices reveal how container orchestration patterns are migrating to agent fleets. The plumbing questions are the same: service discovery, identity boundaries, state persistence, and failure recovery. The workload shape is different.
MCP as Service Mesh for Tools
MCP defines a client-server protocol where agents (clients) discover and invoke tools (servers). This maps cleanly to Kubernetes service discovery, but the routing layer is chattier. An agent doesn’t just hit an endpoint once. It negotiates capabilities, streams context, and maintains session state across multiple tool invocations.
Key differences from HTTP services:
- Tools expose schemas and capability manifests, not just REST paths
- Agents need bidirectional streaming for long-running operations
- Context windows create memory pressure that doesn’t exist in stateless microservices
- Tool servers can reject requests based on agent identity or resource quotas
ToolHive sits above MCP and handles the orchestration plane. It routes agent requests to tool servers, enforces policy, and manages the lifecycle of both agents and tools as workloads. The analogy is Kubernetes managing pods and services, but the state model is heavier.
Identity and Authorization Primitives
Container orchestration solved identity with service accounts and RBAC. Agent orchestration needs more granular controls because agents act on behalf of users, carry sensitive context, and invoke tools with side effects.
Identity layers in agent fleets:
| Layer | Container Orchestration | Agent Orchestration |
|---|---|---|
| Workload identity | Service account per pod | Agent identity per instance |
| User attribution | External (JWT, mTLS) | Embedded in agent context |
| Tool access control | Ingress + network policy | Capability negotiation + policy engine |
| Audit trail | API server logs | Tool invocation logs + context snapshots |
ToolHive introduces a policy layer that sits between agents and tools. An agent might have permission to invoke a database query tool, but only for specific schemas or row-level filters. This is finer-grained than Kubernetes RBAC because the authorization decision depends on runtime context, not just static labels.
The failure mode: if an agent’s context includes PII or credentials, every tool invocation becomes a potential exfiltration vector. Container orchestration assumes workloads are trusted after admission. Agent orchestration cannot.
State Persistence and Failure Recovery
Kubernetes treats pods as ephemeral. StatefulSets exist, but the default assumption is that workloads can restart without losing critical state. Agents break this model because their value is in accumulated context and learned behavior.
State persistence strategies:
- Checkpoint snapshots: Serialize agent memory to object storage after each tool invocation. Recovery is slow but reliable.
- Event sourcing: Log every tool call and response. Replay the log to reconstruct agent state. Adds latency but enables audit and rollback.
- Distributed state stores: Use Redis or etcd to hold agent context. Fast but introduces a shared dependency that becomes a bottleneck.
ToolHive uses a hybrid approach. Short-term context lives in memory. Long-term state gets checkpointed to S3-compatible storage. If an agent crashes, the orchestrator spawns a new instance and replays the last N tool invocations from the event log.
The trade-off: checkpoint frequency vs. recovery time. Checkpoint too often and you burn I/O. Checkpoint too rarely and you lose work. Kubernetes doesn’t face this because most pods are stateless or use persistent volumes. Agents are neither.
Agent-to-Tool Routing
In Kubernetes, a service selector routes traffic to pods with matching labels. In ToolHive, routing depends on agent intent, tool availability, and policy constraints. An agent might request a “web search” capability, and the orchestrator must decide which search tool to invoke based on cost, latency, and quota.
Routing decision factors:
- Tool capability match (schema compatibility)
- Agent authorization (policy engine verdict)
- Tool server load (active connections, queue depth)
- Cost and rate limits (per-agent budgets)
This is closer to Istio’s traffic management than basic Kubernetes services. The orchestrator maintains a registry of available tools, polls their health endpoints, and updates routing tables dynamically. If a tool server goes down, the orchestrator can fail over to an alternative implementation or queue requests until recovery.
The failure mode: routing loops. If an agent invokes a tool that invokes another agent that invokes the original tool, you get infinite recursion. ToolHive tracks call chains and enforces depth limits, but this is a runtime check, not a compile-time guarantee.
Deployment Shape
ToolHive runs as a control plane with multiple components:
# Simplified ToolHive deployment topology
apiVersion: v1
kind: Namespace
metadata:
name: toolhive-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: toolhive-controller
spec:
replicas: 3
template:
spec:
containers:
- name: controller
image: toolhive/controller:latest
env:
- name: MCP_REGISTRY_URL
value: "http://mcp-registry:8080"
- name: POLICY_ENGINE_URL
value: "http://policy-engine:9090"
---
apiVersion: v1
kind: Service
metadata:
name: agent-gateway
spec:
type: LoadBalancer
ports:
- port: 443
targetPort: 8443
selector:
app: toolhive-gatewayThe controller watches for agent and tool CRDs (Custom Resource Definitions). When an agent is created, the controller provisions a pod, injects identity credentials, and registers it with the MCP registry. When a tool server is deployed, the controller updates the routing table and notifies active agents of the new capability.
Observability hooks:
- Prometheus metrics for tool invocation latency and error rates
- OpenTelemetry traces for end-to-end agent workflows
- Structured logs for policy decisions and authorization failures
The deployment model assumes Kubernetes as the substrate, but the same patterns could run on Nomad, ECS, or even bare VMs. The key abstraction is workload lifecycle management, not the specific orchestrator.
Likely Failure Modes
Context window exhaustion: Agents accumulate state until they hit model token limits. ToolHive needs a context pruning strategy (summarization, sliding window, or explicit forget operations).
Tool server cascading failures: If a popular tool goes down, every agent that depends on it will retry, amplifying load. Circuit breakers and backoff policies are mandatory.
Policy drift: Authorization rules change over time. If an agent’s cached policy is stale, it might invoke tools it no longer has permission to use. Policy versioning and refresh intervals matter.
State corruption: If a checkpoint is written mid-transaction, recovery will restore inconsistent state. Atomic writes and validation checks are required.
Identity token leakage: If an agent’s credentials are logged or stored in plaintext, an attacker can impersonate it. Credential rotation and short-lived tokens reduce the blast radius.
Technical Verdict
Use ToolHive and MCP patterns when:
- You need to manage more than a handful of agents and tools
- Authorization and audit requirements are non-negotiable
- Agents must survive restarts without losing critical context
- You already run Kubernetes and want to reuse operational muscle
Avoid when:
- Your agent workload is single-tenant and short-lived (a Lambda function is simpler)
- You don’t have Kubernetes expertise in-house (the operational overhead is real)
- Your tools are stateless HTTP APIs (MCP adds protocol complexity for little gain)
- You need sub-millisecond latency (the orchestration layer adds hops)
The convergence of container and agent orchestration is inevitable. The patterns that worked for stateless microservices need adaptation, not replacement. ToolHive is an early signal of where the infrastructure is heading: declarative workload management, policy-driven routing, and observable failure modes. The plumbing is familiar. The failure modes are new.