OpenAI describes the Performance Factory, a Codex-powered system that investigates ChatGPT performance regressions
In an engineering post, OpenAI's Okay Zed and Cyrus Forbes describe the Performance Factory, a cloud system in which Codex agents find and help fix performance problems in ChatGPT. Engineers define workflows as Markdown prompts and a configuration file listing a task's states and transitions.
- Parts: a task service with configurable workflows, a dispatch loop that starts a Codex turn when a task's next step is ready, an internal API that runs Codex in the cloud under a service identity, and subscriptions that wake a task when something changes, such as a pull request comment or a finished deployment.
- Regression Workflow: a performance alert opens a task. An agent investigates, a separate Codex session tries to disprove the diagnosis, and an agent proposes a fix as a pull request for an engineer to review. After deployment an agent checks whether performance has returned to baseline, and reopens the work if it has not.
- Numbers, as OpenAI reports them: in just over three months the workflow tracked more than 1,700 investigations and root-caused over 600, about 17 a day. In a manual review of 100 root-cause suggestions, OpenAI graded 70% accurate; the rest needed refinement or did not fully establish causality.
- Lessons it lists: protect engineers' attention, evaluate changes to agents (its
perf-forensicsskill was refined with evals built from past regressions), and turn recurring failures into self-repair.
Sources
Get news like this as a push.
AI Tracker alerts you when something changes for the tools you use.