OpenAI Performance Factory fixes ChatGPT slowdowns

OpenAI Performance Factory workflow for finding ChatGPT performance regressions

The OpenAI Performance Factory is an autonomous cloud system that uses Codex agents to find and fix ChatGPT slowdowns. Its first workflow tracked more than 1,700 investigations in just over three months and root-caused over 600 of them. That works out to 17 automated investigations a day.

How the system grew

Before Codex, engineers examined metrics and logs by hand, compared traces and debugged live machines. Repeating problems, such as regressions in ChatGPT’s first-token latency, led the team to build more than 30 performance tools, including CPU and allocation profilers.

Codex first became a partner. Engineers could ask why something got slower and work through it together, even in an unfamiliar system. Next the team wrote a shared skill called perf-forensics. It tells Codex which tools exist, when to use them and how to work a slowdown: find when it began, identify the affected requests, and check whether deployments or configuration changes explain it.

After testing the skill against past regressions, engineers could hand Codex an alert and get a diagnosis in one run. But they had become a human proxy, pasting alerts into Codex and results back into Slack. Moving investigations to the cloud fixed part of that, yet engineers still wrote the fix and checked it after deployment. The Performance Factory covers the rest.

The evolution of performance engineering: manual work, reusable tools, engineers working with Codex, engineers dispatching tasks, systems dispatching tasks, and a factory of agents coordinating automated workflows.
Image: OpenAI

Four parts of the factory

  • A task tracker with configurable workflows, defined in Markdown files that list states, allowed transitions and which steps need an engineer.
  • A dispatch loop that checks which tasks are ready and starts a Codex turn for each.
  • An internal API that runs Codex in the cloud, inside OpenAI’s network, with access to code repositories, metrics, logs, traces, CPU profiles and memory snapshots.
  • Subscriptions that wake an agent when something changes, such as a GitHub review comment or a finished deployment.

Agents run under a service identity, so permissions can be scoped and their actions traced. Running in the cloud means work proceeds in parallel without anyone’s laptop staying open.

Performance Factory orchestration workflow: alerts, engineers, and other agents create tasks; the task service sends ready tasks to a dispatch loop, which starts Codex turns in the cloud. Agents record progress, while subscriptions wake waiting tasks after PR comments or deployments.
Image: OpenAI

The Regression Workflow

When a configured alert fires, the workflow opens a task. It moves through four stages: needs investigation, root caused, fix proposed and resolved. The task stores its status and findings outside any single agent’s context, so engineers or other agents can pick it up.

The agent works out which metrics worsened, when, and which requests were hit. It also checks whether similar requests got slower or whether more already-slow requests were served. Then it reads deployments, feature flag changes and code.

Early agents sometimes stopped at the first plausible explanation. OpenAI added an adversarial review, a separate Codex session that tries to disprove the diagnosis. In one case an agent blamed image download errors for failing conversations. The reviewer saw that the file-locating service had already recovered while image requests kept failing, so it reopened the case.

In a manual review of 100 root-cause suggestions, 70% were accurate. The other 30% needed more work or did not fully establish causality.

Performance Factory report showing worker startup time more than doubling, with impact and recovery, root cause, and proposed fix.
Image: OpenAI

Fixes are harder than diagnoses

After a diagnosis passes review, Codex writes a fix and opens a pull request. An engineer reviews it and decides whether to merge. Each task carries a report with a summary, impact, root cause and proposed fix, plus time series, CPU flame graphs and traces.

After deployment, an agent checks whether performance is back to baseline. Full recovery resolves the task. Partial recovery resolves the original task and opens a new one for what remains. No improvement sends it back for investigation.

OpenAI says fixing has been harder than finding causes. Agents sometimes propose changes that are too complicated or that introduce new problems, so fixes also get an adversarial review. Some choices need an engineer’s judgment: adding a cache may speed things up but serve stale data.

Lessons for other teams

Raw agent output can be a wall of text, so summaries need to be easy to scan. One regression can show up as several incidents, so the workflow now links related ones, groups them and ranks estimated impact.

The team backtests investigations against past production incidents, with the original signal and no answer-revealing details. It scores when the problem began, its size and which requests were affected, and it scores causes separately because timing can be right while the blamed deployment is wrong. A daily agent also scans recent runs for broken tools and access errors.

What OpenAI plans next

Today the regression flow starts from an alert. OpenAI wants agents to compare CPU and memory profiles, heap dumps and request traces across releases to catch regressions alerts miss. It is also building workflows for known patterns, such as a rightsizing skill that spots autoscaling settings keeping more replicas running than needed. The team is exploring agents that ship low-risk fixes without a human reviewer.

Source: OpenAI

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.