A payment API can return 200 OK while your agent violates every approval policy you wrote. The HTTP status code tells you the transaction cleared. It does not tell you whether the agent was authorized to initiate it.
This gap between technical success and governance compliance is becoming a production incident category. As agents gain access to payment rails, refund APIs, and ledger systems, you need observability that captures what the agent tried to do, not just whether the downstream service accepted the request.
Second Signature is a reference implementation built for the SigNoz hackathon. It instruments AI agents that execute financial actions and reconstructs the full trace from intent through policy evaluation, approval workflow, and ledger impact. The goal is to make boundary violations visible before the finance team spots the discrepancy.
The problem: execution success is not governance success
Consider a refund workflow:
- Operation: refund
- Amount: CAD 12,500
- Beneficiary: Northstar Vendor (new)
- Reason: contract dispute
Your policy says transfers above CAD 5,000 to a new beneficiary require human approval. The agent has three possible outcomes:
| Outcome | Policy Check | Approval Flow | Payment Execution | Ledger Impact | Governance Status |
|---|---|---|---|---|---|
| Unsafe success | Skipped | None | Succeeds | Changed | Violation |
| Safe stop | Evaluated | Requested, denied | Blocked | Unchanged | Compliant |
| Safe success | Evaluated | Requested, approved | Succeeds | Changed | Compliant |
In the first case, the payment succeeds and the ledger changes. Only a post-execution audit reveals the agent should have requested approval. By then, the money moved.
Traditional observability captures the HTTP 200 from the payment API. It does not capture that the agent bypassed a required gate.
Architecture: tracing intent and authorization scope
Second Signature uses OpenTelemetry to emit custom spans that represent agent decisions, not just service calls. The trace structure looks like this:
agent.request
├── policy.evaluate
│ ├── attribute: action=refund
│ ├── attribute: amount=12500
│ ├── attribute: beneficiary_status=new
│ └── attribute: requires_approval=true
├── approval.request (conditional)
│ ├── attribute: approver_role=FINANCE_LEAD
│ └── attribute: decision=approved
├── payment.execute (conditional)
│ ├── attribute: api_status=200
│ └── attribute: ledger_id=txn_98234
└── governance.audit
├── attribute: policy_compliant=true
└── attribute: violation_type=noneEach span carries attributes that distinguish legitimate actions from boundary violations:
- agent.intent: what the agent is trying to do (refund, transfer, approve)
- policy.required_approval: whether this action needs a second signature
- approval.obtained: whether the agent actually requested and received approval
- payment.authorized_limit: the maximum amount the agent can move without escalation
- payment.actual_amount: what the agent requested
The difference between policy.required_approval=true and approval.obtained=false is a governance failure, even if payment.api_status=200.
Instrumentation: wiring policy checks into the trace
The key is to emit spans at decision points, not just at I/O boundaries. Here is how you instrument a guarded agent:
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def execute_refund(amount, beneficiary, reason):
with tracer.start_as_current_span("agent.request") as request_span:
request_span.set_attribute("action", "refund")
request_span.set_attribute("amount", amount)
request_span.set_attribute("beneficiary", beneficiary)
# Policy evaluation happens before execution
with tracer.start_as_current_span("policy.evaluate") as policy_span:
requires_approval = check_policy(amount, beneficiary)
policy_span.set_attribute("requires_approval", requires_approval)
policy_span.set_attribute("beneficiary_status", get_status(beneficiary))
if requires_approval:
with tracer.start_as_current_span("approval.request") as approval_span:
decision = request_approval("FINANCE_LEAD", amount, beneficiary)
approval_span.set_attribute("decision", decision)
if decision != "approved":
request_span.set_attribute("governance.compliant", True)
request_span.set_attribute("payment.executed", False)
return {"status": "blocked", "reason": "approval_denied"}
# Only execute if policy allows
with tracer.start_as_current_span("payment.execute") as payment_span:
response = payment_api.refund(amount, beneficiary)
payment_span.set_attribute("api_status", response.status_code)
payment_span.set_attribute("ledger_id", response.transaction_id)
# Audit span captures the gap
with tracer.start_as_current_span("governance.audit") as audit_span:
audit_span.set_attribute("policy_compliant", requires_approval == (decision == "approved"))
audit_span.set_attribute("violation_type", "none" if requires_approval == (decision == "approved") else "missing_approval")The governance.audit span is where you diff intent against authorization. If requires_approval=true but approval.obtained=false, you have a violation, regardless of the payment API response.
Alerting: detecting violations in SigNoz
SigNoz can query trace attributes to surface boundary violations:
Query 1: Payments that succeeded without required approval
span.name = "payment.execute"
AND span.attributes.api_status = 200
AND trace.attributes.policy.requires_approval = true
AND trace.attributes.approval.obtained = falseQuery 2: Agents exceeding authorized limits
span.name = "payment.execute"
AND span.attributes.payment.actual_amount > span.attributes.payment.authorized_limitQuery 3: Governance audit failures
span.name = "governance.audit"
AND span.attributes.policy_compliant = falseThese queries turn governance violations into observable events. You can route them to PagerDuty, Slack, or a compliance dashboard before the ledger reconciliation catches the discrepancy.
State management: reconstructing the approval chain
The trace alone does not tell you who approved what. You need a state store that links approval events to agent actions.
Second Signature uses a simple ledger:
| Trace ID | Action | Amount | Beneficiary | Required Approval | Approver | Decision | Timestamp |
|---|---|---|---|---|---|---|---|
| abc123 | refund | 12500 | Northstar | true | FINANCE_LEAD | approved | 2026-07-27T09:15:32Z |
| def456 | refund | 8000 | Acme Corp | true | none | none | 2026-07-27T09:22:18Z |
The second row is a violation. The agent executed a refund that required approval but never requested one. The trace shows payment.execute succeeded, but the ledger shows no approver.
You can query this table to generate compliance reports, audit trails, or incident timelines.
Failure modes
Span overhead: emitting a span for every policy check adds latency. If your agent evaluates 10 policies per request, you add 10 spans to the trace. This can slow down the critical path.
Attribute cardinality: if you emit beneficiary as a trace attribute and you have 10,000 beneficiaries, you create high-cardinality data that bloats your observability backend. Use hashed identifiers or status categories instead.
Policy drift: if your policy logic changes but your instrumentation does not, your traces will report false positives. You need to version your policy checks and emit the policy version as a span attribute.
Approval latency: if the approval workflow is async (Slack message, email, ticket), the trace will remain open until the human responds. Long-lived traces can time out or get dropped by your collector.
Technical Verdict
Use this approach when:
- Your agents execute financial transactions, data deletions, or permission changes
- You need to prove compliance to auditors or regulators
- The gap between “API succeeded” and “agent was authorized” is a real risk
- You already run OpenTelemetry and have a trace backend like SigNoz or Honeycomb
Avoid this approach when:
- Your agents only read data or generate text (no high-risk actions)
- You have no policy layer to instrument (the agent has unlimited authority)
- Your observability backend cannot handle custom span attributes or high trace volume
- Approval workflows are so slow that keeping traces open is impractical
The core insight is that observability for agents needs to capture decisions, not just I/O. If your agent can move money, change permissions, or delete data, you need traces that show whether it was allowed to do so, not just whether the downstream API accepted the request.