
News
OpenAI’s Performance Factory puts Codex on ChatGPT regressions
OpenAI’s Performance Factory uses Codex agents to investigate ChatGPT regressions, propose fixes, and check recovery, with engineers still merging.
OpenAI published an engineering post on November 19, 2026, on an internal cloud system it calls the Performance Factory. Okay Zed and Cyrus Forbes, members of technical staff, describe Codex agents that investigate performance regressions, record findings, propose fixes, and later check whether production recovered. The same day, @OpenAIDevs put the current loop in one sentence: "We built the Performance Factory to coordinate agents that investigate regressions, propose fixes for engineers to review, and check if those fixes work in production."
Read that sentence against the post’s stronger wording. OpenAI says the system "automates finding and fixing performance issues," and that the Regression Workflow "has automated the entire process of fixing regressions." The stages still stop for a person. An engineer reviews the pull request and decides whether to merge. Shipping low-risk fixes with no human in the loop is something the team says it is exploring, not the workflow it describes as running. This is not a model launch.
How a task moves
Engineers define workflows with configuration and Markdown prompts. The task service lists states, allowed transitions, and which steps need an engineer. A dispatch loop checks for ready work and starts a Codex turn with the instructions for that state. The turn runs in the cloud, inside OpenAI’s network, under a service identity. It can reach repositories and the observability tools engineers already use: metrics, logs, traces, CPU profiles, and memory snapshots. Work can run in parallel without an engineer’s laptop staying open.
A task that has to wait subscribes to an event, such as a review comment, a failed CI check, or a finished deployment, and does not keep an agent running. When the event matches, the task is ready again and the dispatch loop assigns an agent. Findings stay on the task, outside any one session, so another agent or an engineer can resume.
Alert, diagnosis, challenge, merge, check
The Regression Workflow is the first one built on that machinery. A configured performance alert opens a task. Named stages are needs investigation, root caused, fix proposed, and resolved. The published steps match that path: the system detects the alert, an agent finds the root cause, an agent proposes a fix, an engineer reviews and deploys, and an agent verifies recovery. ChatGPT latency and availability drops are the examples OpenAI says it focused on; the post also describes the factory as machinery for workflows across its services.
The investigating turn uses a shared skill, perf-forensics, which tells Codex which tools exist and how to work a slowdown: when it began, which requests were affected, and whether a deployment or configuration change could explain it. An alert can fire well after a metric first moved, so the agent looks backward. It also separates two causes that point different ways: similar requests got slower, or the service started serving more requests that were already slow. It then reads the relevant code and uses logs or traces to test whether a change could explain what happened.
Early on, agents sometimes stopped at the first plausible story, and some of those diagnoses were wrong. OpenAI added a separate Codex session whose job is to disprove the diagnosis. The reviewer rechecks assumptions and tests counterfactuals. If a rollout is the suspected cause, did requests that were not exposed stay healthy? If the change was rolled back, did the metric recover?
One investigation in the post shows the check doing real work. An agent traced failing ChatGPT conversations to image-download errors and marked the cause as found. The reviewer saw that the internal service responsible for locating files had already recovered while image requests were still failing. That diagnosis did not explain the continuing failures, so the reviewer reopened the investigation and wrote down what still had to be established.
What three months of runs showed
In just over three months, the Regression Workflow tracked more than 1,700 investigations and root-caused over 600 cases. OpenAI’s own pace figure is 17 automated investigations a day.
Accuracy is partial, and OpenAI says so. A manual review of 100 root-cause suggestions graded 70% accurate. The other 30% needed further refinement or did not fully establish causality. Separate turns for investigation, for the challenge, and for the proposed fix are how the workflow tries to improve on that.
Fixes are the harder step. "Getting fixes right has been harder than finding the cause," the post says. Proposed changes have been more complicated than necessary, or have introduced new issues. Some also need a judgment call an agent is not asked to make alone: a cache might cut latency and also serve stale data. After adversarial review of the diagnosis, Codex opens a pull request. An engineer reviews it, can edit it or write a different fix, and decides whether to merge. Comments on the PR wake an agent to address them. The handoff is a task report: regression, impact, root cause, proposed fix, plus charts such as metrics time series, CPU flame graphs, and traces.
After deploy, another agent checks the earlier baseline. Recovery resolves the task. Partial recovery resolves the original regression and opens a new task for the remaining slowdown, which may have a different cause. No improvement sends the case back. OpenAI added a matching adversarial review of each proposed fix, to test whether it addresses the root cause and to catch bugs or side effects.
Still gated on the merge
The post is explicit about what is not shipping today. OpenAI is comparing agent proposals with the changes engineers ultimately merge, and it is working on synthetic tests so an agent can measure a fix before review. It is also "exploring how agents could use our other agentic review tools to identify low-risk changes and eventually ship those fixes without a human in the loop." The present limit, in the authors’ words: "Today, the bottleneck may be waiting for an engineer to review and merge the fix."
Alert-triggered investigation is the live path. Using CPU and memory profiles, heap dumps, and request traces to find slowdowns no alert caught is described as work in progress, and it would still open a pull request for an engineer. A daily agent already looks through recent runs for tool failures and access errors. When the same error blocks several investigations, that check can open a task; an engineer still decides whether to merge the repair.
The pattern in the post is narrower than unsupervised performance engineering. It is a recurring ops loop with explicit states, a second session assigned to break a weak diagnosis, and a human gate on the merge. OpenAI’s closing line, repeated by @OpenAIDevs, is "Codex already helps us build ChatGPT faster. Now it’s helping us keep ChatGPT fast."
Read the November 19 engineering post before treating the factory as a template. The stage diagram, the image-download counterexample, and the 70% review are the parts that show where the loop holds and where it still needs a person.