mech.app

The mech.app newsletter

Agentic AI, minus the noise.

Get practical field notes on AI agents, automation, developer tools and security delivered to your inbox.

No spam. Unsubscribe anytime.

AI Agents

OpenAI's Performance Factory: Codex Agents That Auto-Fix Production Bottlenecks

How OpenAI uses Codex-powered workflows to detect, diagnose, and deploy performance fixes in ChatGPT without human intervention.

Source: openai.com
OpenAI's Performance Factory: Codex Agents That Auto-Fix Production Bottlenecks

OpenAI just published details on Performance Factory, an internal system that uses Codex agents to detect, diagnose, and fix performance regressions in ChatGPT without human intervention. This is not a research demo. It is production infrastructure that keeps a high-scale service fast while engineers ship new features at pace.

The system runs a closed loop: alerts trigger workflows, agents investigate root causes, generate fixes, test them, and deploy or escalate. The post reveals how OpenAI constrains agent behavior, handles failure modes, and decides when to auto-deploy versus flag for review.

Architecture: Four Core Components

Performance Factory is built on a task service that orchestrates Codex agents through configurable workflows. The system has four parts:

  • Task service: Stores workflow definitions and task state. Engineers write instructions for agents in natural language, specifying tools, constraints, and success criteria.
  • Dispatch loop: Assigns work to agents based on task priority and resource availability. Handles retries and escalation when agents fail.
  • Codex in the cloud: Runs agent reasoning and tool calls. Agents have access to metrics dashboards, code search, profiling tools, and deployment APIs.
  • Subscriptions: Wake agents when specific events occur (alert fires, test completes, deployment finishes). This avoids polling and keeps latency low.

The first production workflow is the Regression Workflow. When a performance alert fires, the system opens a task. An agent investigates metrics, identifies the root cause, generates a fix, tests it, and either deploys or escalates to a human.

Detection: Signal Versus Noise

Performance Factory relies on existing observability infrastructure. OpenAI monitors latency percentiles, throughput, and resource utilization across services. Alerts fire when metrics cross thresholds or deviate from historical baselines.

The challenge is distinguishing real regressions from transient spikes. The system uses two filters:

  1. Temporal correlation: Agents check if the alert coincides with a recent deployment or configuration change. If yes, the workflow proceeds. If no, the agent flags the alert as potentially spurious.
  2. Adversarial review: A second agent challenges the first agent’s diagnosis. It looks for alternative explanations (traffic spike, downstream dependency, infrastructure issue). If the adversarial agent finds a plausible alternative, the task escalates to a human.

This two-agent pattern reduces false positives. The post does not specify how often adversarial review overturns the initial diagnosis, but it implies the pattern is critical for production trust.

Fix Generation: Constrained Code Changes

Once an agent confirms a regression and identifies the root cause, it generates a fix. This is the riskiest part of the workflow. OpenAI constrains the agent in several ways:

  • Scoped access: Agents can only modify specific files or functions flagged as performance-critical. They cannot touch authentication, billing, or data handling logic.
  • Test requirements: Every fix must pass existing unit tests, integration tests, and a new performance benchmark that reproduces the regression.
  • Rollback plan: The system generates a rollback commit before deploying. If post-deployment metrics degrade, the system auto-reverts.

The post describes a real example. An alert fired for increased latency in a text processing pipeline. The agent traced the issue to a recent refactor that introduced an unnecessary copy operation. The fix removed the copy, passed tests, and deployed automatically. Latency returned to baseline within minutes.

State Management and Observability

Performance Factory tracks task state in a persistent store. Each task has a status (open, investigating, fixing, testing, deployed, escalated), a timeline of agent actions, and links to relevant metrics and logs.

Agents log every tool call and reasoning step. This creates an audit trail for debugging and improving workflows. When a fix fails, engineers review the logs to understand what the agent missed.

The system does not retry indefinitely. If an agent fails to diagnose a regression after three attempts, the task escalates. If a fix fails tests twice, the task escalates. This prevents the system from thrashing or deploying broken code.

Deployment Decision Tree

Not all fixes auto-deploy. The system uses a decision tree based on risk and confidence:

ConditionAction
Fix passes all tests, regression is isolated to one service, rollback plan existsAuto-deploy
Fix passes tests but touches shared infrastructureDeploy to canary, monitor for 10 minutes, then full rollout
Fix fails one test but passes othersEscalate to human
Adversarial agent disputes diagnosisEscalate to human
Fix requires changes outside scoped filesEscalate to human

The post does not specify what percentage of tasks auto-deploy versus escalate. It implies the system handles routine regressions autonomously but defers complex or ambiguous cases.

Failure Modes and Guardrails

OpenAI describes three failure modes they encountered during rollout:

  1. Agent fixates on wrong hypothesis: Early versions of the workflow allowed agents to investigate indefinitely. This led to wasted compute and delayed escalation. The fix was a hard limit on investigation steps.
  2. Fix introduces new regression: An agent optimized one code path but degraded another. The fix was stricter performance benchmarks that test multiple scenarios.
  3. Cascading alerts: A single root cause triggered multiple alerts, and agents opened redundant tasks. The fix was alert deduplication based on shared metrics and timestamps.

The system also has circuit breakers. If more than three tasks escalate in an hour, the system pauses new task creation and alerts the on-call engineer. This prevents the system from overwhelming humans during an incident.

Code Example: Workflow Definition

The post does not include full code, but it describes workflows as YAML-like configurations. Here is a plausible reconstruction:

workflow: regression_investigation
trigger:
  - alert: latency_p99_high
    threshold: 500ms
    duration: 5m

steps:
  - name: correlate_deployment
    agent: investigator
    tools: [metrics_api, deployment_log]
    prompt: "Check if this alert coincides with a recent deployment."
    timeout: 2m

  - name: adversarial_review
    agent: challenger
    tools: [metrics_api, trace_viewer]
    prompt: "Find alternative explanations for this regression."
    timeout: 3m

  - name: generate_fix
    agent: fixer
    tools: [code_search, profiler, test_runner]
    prompt: "Generate a minimal fix for the identified root cause."
    constraints:
      - allowed_files: ["src/processing/*.py"]
      - required_tests: ["unit", "integration", "perf"]
    timeout: 10m

  - name: deploy
    condition: all_tests_pass
    action: auto_deploy
    rollback: true
    canary_duration: 10m

This structure separates concerns: investigation, review, fix generation, and deployment. Each step has explicit timeouts and tool access.

Lessons: Protect Attention, Evaluate Changes, Self-Repair

OpenAI highlights three operational lessons:

Protect engineers’ attention: The system must escalate only when necessary. Too many false positives erode trust. Too few escalations risk silent failures. The adversarial review pattern and strict escalation criteria balance these risks.

Evaluate changes to agents: When OpenAI updates Codex or modifies workflow instructions, they run the new version on historical tasks and compare outcomes. This prevents regressions in the automation itself.

Turn recurring failures into self-repair: If the same type of regression occurs multiple times, engineers add a new workflow or expand agent constraints. The system learns from past escalations.

What’s Next: Proactive Monitoring

The post hints at future work. Current workflows are reactive (alert fires, agent responds). OpenAI is exploring proactive monitoring, where agents continuously scan for performance degradation before alerts fire. This requires tighter integration with profiling tools and more sophisticated anomaly detection.

They also plan to expand Performance Factory beyond performance. The same architecture could handle security patches, dependency updates, or cost optimization.

Technical Verdict

Use Performance Factory patterns when:

  • You have high-frequency, well-scoped performance issues that follow predictable patterns.
  • Your observability stack can reliably detect regressions and correlate them with deployments.
  • You can define clear constraints on what agents are allowed to change.
  • You have strong rollback and canary infrastructure.

Avoid this approach when:

  • Performance issues are rare or highly variable (not enough signal to train workflows).
  • Your codebase lacks test coverage or performance benchmarks.
  • You cannot tolerate any risk of automated code changes in production.
  • Your team is still debugging basic observability gaps.

Performance Factory works because OpenAI invested in the surrounding infrastructure: metrics, tests, deployment automation, and rollback mechanisms. The agents are the orchestration layer, not a replacement for fundamentals.