ORCA-bench: How Oncall Root Cause Analysis Exposes the Limits of Agent Reasoning Over Production Chaos

Large language models can write, patch, and search code. Oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began.

ORCA-bench is a new benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. The results are sobering. The best agent achieves 25.3% RCA accuracy on medium-difficulty tasks (the realistic-input setting) and 10.0% on hard tasks. The weakest model hallucinates an implausible root cause in 40% of incident reports.

This is not a code generation problem. It is a multi-modal debugging problem under time pressure, and current agent architectures are not ready.

What ORCA-bench Actually Tests

ORCA-bench pairs a live OpenTelemetry-instrumented microservice system with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios.

The testbed exposes:

Six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, OpenSearch via Grafana)
Full source-code access to the microservice system
Ground-truth symptoms curated and signed off by expert SREs
LLM-as-judge scoring independently re-validated by humans (Cohen’s κ_w = 0.90)

Tasks start from user-facing reports like “checkout is slow” or “search returns empty results.” The agent must correlate telemetry streams, form hypotheses, and identify the root cause. The system is 50 GB across six days. Real production systems are orders of magnitude larger, more dynamic, and more idiosyncratic.

Oncall RCA requires agents to:

Parse ambiguous user reports into observable symptoms
Query metrics, logs, and traces with different sampling rates and timestamp alignment
Correlate signals across telemetry streams (e.g., a latency spike in metrics, an error cluster in logs, a slow span in traces)
Navigate source code to understand service boundaries, dependencies, and failure modes
Rank hypotheses and converge on a root cause

Each step introduces retrieval and reasoning challenges:

Metrics are time-series with varying granularity (1s, 10s, 1m). Agents must choose the right aggregation window and detect anomalies against baseline behavior.
Logs are unstructured or semi-structured text. Relevant error messages may be buried in millions of log lines, and log levels (INFO, WARN, ERROR) are unreliable signals.
Traces are distributed call graphs. A single user request may generate hundreds of spans across dozens of services. Agents must identify the critical path and isolate slow or failing spans.
Source code provides context for interpreting telemetry. Without it, an agent cannot distinguish between expected retries and a cascading failure.

ORCA-bench found that removing source-code access degrades every metric. Agents that cannot read code cannot reason about system behavior.

Agent Architecture Breakdown

ORCA-bench tested five frontier agents. The architecture pattern is consistent:

class OncallAgent:
    def __init__(self, llm, tools, context_window):
        self.llm = llm
        self.tools = tools  # Prometheus, Jaeger, OpenSearch, GitHub
        self.context_window = context_window
        self.state = {"hypotheses": [], "evidence": []}
    
    def investigate(self, user_report):
        # Step 1: Parse report into symptoms
        symptoms = self.llm.extract_symptoms(user_report)
        
        # Step 2: Query telemetry for each symptom
        for symptom in symptoms:
            metrics = self.tools.prometheus.query(symptom)
            logs = self.tools.opensearch.search(symptom)
            traces = self.tools.jaeger.find_traces(symptom)
            
            # Step 3: Correlate signals
            hypothesis = self.llm.correlate(metrics, logs, traces)
            self.state["hypotheses"].append(hypothesis)
        
        # Step 4: Rank hypotheses and select root cause
        root_cause = self.llm.rank_hypotheses(self.state["hypotheses"])
        return root_cause

The failure modes cluster around three areas:

Context overflow: Investigations span hours. Telemetry data exceeds context windows. Agents lose track of earlier hypotheses.
Retrieval precision: Agents query too broadly (millions of log lines) or too narrowly (miss the relevant time window).
Correlation logic: Agents struggle to distinguish correlation from causation. A latency spike and an error cluster may co-occur without one causing the other.

Tool Boundary Design

ORCA-bench uses real telemetry interfaces. Agents call Prometheus, Jaeger, and OpenSearch APIs directly. This exposes a design tension:

Read-only access is safe but limits investigation. Agents cannot run ad-hoc queries, restart services, or deploy mitigations.
Write access enables faster resolution but introduces risk. An agent that can restart services can also cause outages.

The benchmark sidesteps this by keeping agents read-only. In production, you need a hybrid model:

Access Level	Use Case	Risk
Read-only telemetry	Hypothesis formation, evidence gathering	Low (query load on observability backend)
Read-only source code	Understanding service behavior, dependency graphs	Low (no execution)
Write to runbooks	Documenting findings, updating playbooks	Medium (incorrect documentation)
Execute mitigation scripts	Restarting services, scaling resources	High (cascading failures, data loss)

The safest production pattern is read-only investigation with human-in-the-loop approval for write operations. Agents propose mitigations, humans approve and execute.

State Management Across Long Investigations

Oncall investigations do not fit in a single context window. An agent may:

Query metrics for the past 6 hours
Search logs for error patterns across 10 services
Trace 50 user requests to isolate the slow path
Read source code for 5 microservices

The total context is hundreds of thousands of tokens. Agents need external state management:

class StatefulOncallAgent:
    def __init__(self, llm, tools, vector_store):
        self.llm = llm
        self.tools = tools
        self.vector_store = vector_store  # For long-term memory
        self.working_memory = []  # For current investigation
    
    def investigate(self, user_report):
        # Retrieve similar past incidents
        past_incidents = self.vector_store.search(user_report)
        
        # Start investigation with priors
        for incident in past_incidents:
            self.working_memory.append(incident["root_cause"])
        
        # Iterative hypothesis testing
        while not self.converged():
            hypothesis = self.llm.next_hypothesis(self.working_memory)
            evidence = self.gather_evidence(hypothesis)
            self.working_memory.append({"hypothesis": hypothesis, "evidence": evidence})
            
            # Prune working memory to fit context window
            if len(self.working_memory) > self.context_limit:
                self.summarize_and_prune()
        
        return self.llm.final_root_cause(self.working_memory)

The summarize-and-prune step is critical. Agents must decide which evidence to keep and which to discard. ORCA-bench does not test this explicitly, but it is a known failure mode in long-running investigations.

Evaluation Challenges

How do you evaluate agent performance on RCA when ground truth is often “we never found the root cause”?

ORCA-bench solves this by curating tasks with known root causes, signed off by expert SREs. The LLM-as-judge is re-scored by humans with high agreement (Cohen’s κ_w = 0.90).

But this introduces selection bias. The benchmark only includes incidents with clear root causes. Real oncall includes:

Incidents with multiple contributing factors
Incidents where the root cause is outside the instrumented system (network, DNS, third-party API)
Incidents where the symptom resolves before the root cause is identified

ORCA-bench is a lower bound on agent capability. Production RCA is harder.

Performance Breakdown

Agent	Medium Accuracy	Hard Accuracy	Hallucination Rate
Best (Claude Fable 5)	25.3%	10.0%	Not reported
Weakest	Not reported	Not reported	40.0%

The gap between medium and hard tasks is large. Hard tasks involve:

Co-occurring faults (multiple services failing simultaneously)
Delayed detection (incident started hours before the report)
Ambiguous symptoms (user report is vague or misleading)

Even the best agent fails 75% of medium tasks. The weakest agent hallucinates a root cause in 40% of cases. This is dangerous. A confident but incorrect diagnosis can send engineers down the wrong path, wasting hours during an active outage.

When Agents Fail

The paper identifies three failure patterns:

Premature convergence: Agent latches onto the first plausible hypothesis and ignores contradictory evidence.
Retrieval failure: Agent queries the wrong time window, misses relevant logs, or retrieves too much noise.
Correlation confusion: Agent identifies a correlation (e.g., high CPU and slow requests) but misidentifies the causal direction.

Removing source-code access makes all three worse. Without code, agents cannot:

Understand retry logic (is this a transient error or a persistent failure?)
Identify rate-limiting behavior (is the service rejecting requests or is the client backing off?)
Trace dependency chains (which service is upstream of the failure?)

Source code is not optional for RCA. It is load-bearing context.

Deployment Shape for Production RCA

If you want to deploy an RCA agent in production, the architecture looks like this:

Incident detection: Alerting system (PagerDuty, Opsgenie) triggers the agent with a user report.
Telemetry access: Agent has read-only API access to Prometheus, Jaeger, OpenSearch, and source code repositories.
Hypothesis generation: Agent queries telemetry, correlates signals, and proposes root causes.
Human review: Oncall engineer reviews agent findings, approves or rejects hypotheses.
Mitigation execution: If approved, engineer executes mitigation (restart, scale, rollback).
Postmortem update: Agent drafts postmortem, engineer edits and publishes.

The agent is a co-pilot, not an autopilot. It accelerates investigation but does not replace human judgment.

Security and Observability Boundaries

RCA agents need broad read access to production systems. This introduces security risks:

Credential management: Agents need API keys for Prometheus, Jaeger, OpenSearch, GitHub. Rotate keys frequently, use short-lived tokens.
Data exfiltration: Logs and traces may contain PII, secrets, or sensitive business logic. Agents must not leak this data to external LLM APIs. Use on-premise models or data anonymization.
Audit logging: Every agent query and hypothesis must be logged for postmortem review. If the agent misdiagnoses, you need to understand why.

Observability for the agent itself is critical:

Query latency: How long does the agent spend querying telemetry?
Token usage: How much context does the agent consume per investigation?
Hypothesis churn: How many hypotheses does the agent generate before converging?

If the agent is slow, it is not useful during an active outage. If it burns through tokens, it is expensive. If it generates dozens of hypotheses, it is noisy.

Technical Verdict

Use ORCA-bench when:

You are building or evaluating RCA agents and need a production-fidelity benchmark.
You want to test agent performance on multi-modal telemetry correlation, not just code generation.
You need a baseline for agent capability on oncall tasks before deploying to production.

Avoid or defer when:

You expect agents to achieve human-level RCA accuracy today. They do not.
You plan to deploy agents without human-in-the-loop review. The hallucination rate is too high.
Your production system is larger, more dynamic, or less instrumented than the ORCA-bench testbed. Agent performance will be worse, not better.

ORCA-bench is a reality check. Frontier agents can write code, but they cannot yet debug production systems under time pressure. The gap is large, and closing it will require better retrieval strategies, longer context windows, and more sophisticated correlation logic.

If you are building oncall automation, start with read-only co-pilots that assist human engineers. Do not hand over the pager yet.

Source Links

ORCA-bench: How Ready Are Language Model Agents for Oncall? (ArXiv)
ORCA-bench Public Dataset (referenced in paper)

Agentic AI, minus the noise.