The OpenAI Performance Factory is an autonomous cloud system that uses Codex agents to find and fix ChatGPT slowdowns. Its first workflow tracked more than 1,700 investigations in just over three months and root-caused over 600 of them. That works out to 17 automated investigations a day.
How the system grew
Before Codex, engineers examined metrics and logs by hand, compared traces and debugged live machines. Repeating problems, such as regressions in ChatGPT’s first-token latency, led the team to build more than 30 performance tools, including CPU and allocation profilers.
Codex first became a partner. Engineers could ask why something got slower and work through it together, even in an unfamiliar system. Next the team wrote a shared skill called perf-forensics. It tells Codex which tools exist, when to use them and how to work a slowdown: find when it began, identify the affected requests, and check whether deployments or configuration changes explain it.
After testing the skill against past regressions, engineers could hand Codex an alert and get a diagnosis in one run. But they had become a human proxy, pasting alerts into Codex and results back into Slack. Moving investigations to the cloud fixed part of that, yet engineers still wrote the fix and checked it after deployment. The Performance Factory covers the rest.

Four parts of the factory
- A task tracker with configurable workflows, defined in Markdown files that list states, allowed transitions and which steps need an engineer.
- A dispatch loop that checks which tasks are ready and starts a Codex turn for each.
- An internal API that runs Codex in the cloud, inside OpenAI’s network, with access to code repositories, metrics, logs, traces, CPU profiles and memory snapshots.
- Subscriptions that wake an agent when something changes, such as a GitHub review comment or a finished deployment.
Agents run under a service identity, so permissions can be scoped and their actions traced. Running in the cloud means work proceeds in parallel without anyone’s laptop staying open.

The Regression Workflow
When a configured alert fires, the workflow opens a task. It moves through four stages: needs investigation, root caused, fix proposed and resolved. The task stores its status and findings outside any single agent’s context, so engineers or other agents can pick it up.
The agent works out which metrics worsened, when, and which requests were hit. It also checks whether similar requests got slower or whether more already-slow requests were served. Then it reads deployments, feature flag changes and code.
Early agents sometimes stopped at the first plausible explanation. OpenAI added an adversarial review, a separate Codex session that tries to disprove the diagnosis. In one case an agent blamed image download errors for failing conversations. The reviewer saw that the file-locating service had already recovered while image requests kept failing, so it reopened the case.
In a manual review of 100 root-cause suggestions, 70% were accurate. The other 30% needed more work or did not fully establish causality.

Fixes are harder than diagnoses
After a diagnosis passes review, Codex writes a fix and opens a pull request. An engineer reviews it and decides whether to merge. Each task carries a report with a summary, impact, root cause and proposed fix, plus time series, CPU flame graphs and traces.
After deployment, an agent checks whether performance is back to baseline. Full recovery resolves the task. Partial recovery resolves the original task and opens a new one for what remains. No improvement sends it back for investigation.
OpenAI says fixing has been harder than finding causes. Agents sometimes propose changes that are too complicated or that introduce new problems, so fixes also get an adversarial review. Some choices need an engineer’s judgment: adding a cache may speed things up but serve stale data.
Lessons for other teams
Raw agent output can be a wall of text, so summaries need to be easy to scan. One regression can show up as several incidents, so the workflow now links related ones, groups them and ranks estimated impact.
The team backtests investigations against past production incidents, with the original signal and no answer-revealing details. It scores when the problem began, its size and which requests were affected, and it scores causes separately because timing can be right while the blamed deployment is wrong. A daily agent also scans recent runs for broken tools and access errors.
What OpenAI plans next
Today the regression flow starts from an alert. OpenAI wants agents to compare CPU and memory profiles, heap dumps and request traces across releases to catch regressions alerts miss. It is also building workflows for known patterns, such as a rightsizing skill that spots autoscaling settings keeping more replicas running than needed. The team is exploring agents that ship low-risk fixes without a human reviewer.
Source: OpenAI