MCP servers expose tools to agents, but once a session starts inside Claude Desktop or ChatGPT, you lose visibility. You see scattered tool call logs but not the user intent, the agent’s reasoning, or whether the workflow succeeded. Armature instruments MCP servers to capture full sessions, attribute costs, and run evals without changing your tool implementations.
The Observability Gap
When you build an MCP server, you control the tool definitions and the API responses. You do not control the agent runtime. Sessions happen inside the user’s AI client. Your logs show individual tool calls, but you cannot answer:
- What did the user actually want to do?
- Did the agent complete the task or loop on a missing scope?
- Which workflows cost the most tokens?
- Which tools fail silently because the agent retries without telling you?
Standard observability tools track HTTP requests. They do not reconstruct multi-step agent workflows or evaluate whether the session succeeded from the user’s perspective.
How Armature Instruments MCP Sessions
Armature sits between the MCP server and the agent runtime. It captures every tool call, the agent’s reasoning tokens, and the user’s original prompt. It does not require changes to your tool code.
Integration steps:
- Generate an API key in the Armature dashboard.
- Add the key to your deployment secrets.
- Wrap your MCP server with the Armature SDK or proxy.
The SDK intercepts tool calls at the MCP protocol layer. It logs the call, the response, and the latency. It also captures the agent’s internal reasoning if the runtime exposes it (Claude and GPT-4 both do).
What gets captured:
- User intent (the original prompt)
- Agent thinking (reasoning tokens between tool calls)
- Tool call sequence (order, parameters, responses)
- Session outcome (success, failure, loop, timeout)
Armature’s models read the full session and assign a score. A session that completes the user’s task scores high. A session that loops on a missing auth scope or hits a rate limit scores low.
Session Replay and Root Cause Grouping
Every session gets a replay view. You see the user’s ask, the agent’s reasoning, and every tool call in order. When a session fails, you can step through the trace and find the exact point where the agent got stuck.
Armature groups failures by root cause. If 127 sessions loop on a missing auth scope, you see one issue with 127 occurrences. If 64 sessions fail because your search tool does not handle the word “refund,” you see that as a separate issue.
Example issue grouping:
| Issue | Sessions | Trend | Root Cause |
|---|---|---|---|
| Agent loops on missing auth scope | 127 | ↑ 43 this week | Tool returns 403, agent retries without re-auth |
| Search misses “refund” phrasing | 64 | Stable | Keyword match does not include synonyms |
| Export truncated by pagination | 31 | ↓ 12 this week | Agent stops after first page, assumes complete |
| Rate limit hit on bulk updates | 12 | New | No backoff logic in tool wrapper |
This table surfaces the issues that affect the most users. You fix the top item and the next deploy reduces failed sessions by 127.
Use Case Discovery
Armature’s models read every session and extract the user’s intent. Sessions are grouped into use cases and ranked by volume and success rate.
Example use case breakdown:
- Create & send invoices: 38% of sessions, 92% success
- Reconcile payments: 22% of sessions, 88% success
- Bulk refunds (not supported yet): 14% of sessions, 0% success
- Export revenue report: 9% of sessions, 95% success
The third item is a feature gap. Users ask for bulk refunds, the agent tries to call a tool that does not exist, and the session fails. You now have quantified demand for a new tool.
Cost Attribution Across Multi-Step Workflows
Agent sessions span multiple tool calls. Each call has a token cost (input and output) and a latency cost (API round trip). Armature attributes both to the session.
Session cost breakdown:
- User prompt tokens: 120
- Agent reasoning tokens: 1,450
- Tool call input tokens: 890
- Tool call output tokens: 3,200
- Total tokens: 5,660
- Estimated cost: $0.08
You can filter sessions by cost and find the workflows that burn the most tokens. If “Export revenue report” sessions average $0.15 because the agent fetches 10 pages of data, you can optimize the tool to return paginated results or add a summary endpoint.
Eval Integration
Armature runs evals on every session. An eval checks whether the session met the user’s intent. You define success criteria in code or natural language.
Example eval (pseudocode):
def eval_invoice_session(session):
user_intent = session.user_prompt
tool_calls = session.tool_calls
# Check if invoice was created and sent
invoice_created = any(call.tool == "create_invoice" for call in tool_calls)
invoice_sent = any(call.tool == "send_invoice" and call.response.status == 200 for call in tool_calls)
if invoice_created and invoice_sent:
return {"score": 100, "passed": True}
elif invoice_created:
return {"score": 50, "passed": False, "reason": "Invoice created but not sent"}
else:
return {"score": 0, "passed": False, "reason": "Invoice not created"}Evals run asynchronously after the session completes. You see pass/fail rates over time and can track regressions when you deploy a new tool version.
Deployment Shape
Armature runs as a hosted service. You send session data to their API. The SDK handles batching and retries.
Architecture:
- Agent runtime calls your MCP server.
- MCP server (wrapped with Armature SDK) executes the tool.
- SDK logs the call to Armature’s API (async, non-blocking).
- Armature’s backend processes the session, runs evals, and updates the dashboard.
The SDK adds 2-5ms of latency per tool call (local logging + async flush). It does not block the agent’s execution path.
Security boundaries:
- Tool call parameters and responses are sent to Armature’s API.
- User prompts and agent reasoning tokens are sent if you enable full session capture.
- Armature stores data in encrypted S3 buckets (AES-256).
- You can configure data retention (default: 90 days).
If you cannot send data to a third-party API, Armature offers a self-hosted option. You run the analytics backend in your own VPC and point the SDK at your instance.
Failure Modes
SDK crashes or loses connection:
The SDK logs to a local buffer and flushes asynchronously. If the flush fails, it retries with exponential backoff. If the buffer fills, it drops the oldest sessions. Your MCP server continues to serve tool calls.
Eval logic is wrong:
Evals are code. If your eval function has a bug, it will misclassify sessions. Armature shows eval results alongside session replays, so you can spot mismatches and fix the eval.
Agent reasoning is not captured:
Some agent runtimes do not expose reasoning tokens. If you use a custom runtime or a model that does not emit chain-of-thought, Armature only sees tool calls. You lose the “agent thinking” layer but still get session replay and cost attribution.
High-volume MCP servers:
If your server handles 10,000+ tool calls per minute, the SDK’s async flush may not keep up. You can configure batch size and flush interval. Armature recommends sampling (log 10% of sessions) for high-volume servers.
Trade-Offs
| Dimension | Armature | DIY Logging | OpenTelemetry |
|---|---|---|---|
| Setup time | 10 minutes | Days to weeks | Hours to days |
| Session replay | Built-in | Custom UI required | Trace view, no eval |
| Eval integration | Native | Manual | Plugin required |
| Cost attribution | Per-session | Requires custom logic | Span-level, not session-level |
| Data residency | Hosted or self-hosted | Full control | Full control |
| MCP-specific features | Yes (tool call semantics) | No | No |
Armature is faster to deploy than building your own session replay. It understands MCP semantics (tool definitions, parameter schemas). It does not give you full control over data pipelines or custom aggregations.
Technical Verdict
Use Armature when:
- You run MCP servers in production and need session-level visibility.
- You want to track which tools agents actually use and how often they fail.
- You need cost attribution across multi-step workflows.
- You want to run evals on agent sessions without building eval infrastructure.
Skip it when:
- You cannot send session data to a third-party API and do not want to self-host.
- You already have a custom observability stack that reconstructs agent sessions.
- You need real-time alerting on tool failures (Armature processes sessions asynchronously).
- Your MCP server handles low traffic and you can manually review logs.
Armature fills the gap between tool call logs and session-level observability. It works best for teams that ship MCP servers to users and need to understand how agents use their tools in the wild.