When ChatGPT slows down, how does OpenAI know a Codex fix worked?
OpenAI’s Performance Factory can investigate a slowdown and propose a fix. The outcome that matters is verified recovery for affected users.

On November 19, OpenAI described Performance Factory, its internal system that uses Codex to investigate performance alerts and propose fixes. Engineers review the changes, and a separate agent checks production after deployment for a return to baseline. The company reports more than 1,700 investigations and more than 600 root-caused cases in just over three months. Those counts establish that the workflow is running, not how many fixes shipped or how much faster ChatGPT became. Its most transferable idea is that an investigation ends when production evidence confirms recovery, rather than when someone submits a change.
Find out what users are waiting for
A slow response can mean a long wait before any text appears or a long wait for the complete answer. Queuing, the first model output and rendering can affect the initial wait. Answer length and tool calls can also affect completion. Imagine an improvement that puts text on screen half a second sooner but delays completion for long answers that use a tool. The first measure would register a win while some users would perceive a slowdown. That is an illustrative example, not a reported OpenAI result.
The other question is whose experience changed. An unchanged median can coexist with severe slowdowns for one geography or conversation type. Google’s site reliability engineering guidance explains why a latency distribution and high percentiles help make slow requests visible. Even a better overall percentile can mislead if a larger share of traffic now uses short responses or lighter models. Before and after measurements need comparable workflows, traffic mixes and observation windows. Recovery should be visible for the group that experienced the problem.
A promising clue needs independent evidence
An alert in the same minute as a deployment is a useful lead, not proof. A stronger comparison would put similar traffic through the two versions while measuring stages separately. For example, a team could replay sanitized requests, change a single service setting and see whether the wait moves with it. Where replay is unavailable, unaffected traffic can serve as a comparison group to check for a broader load change. An agent may be fast at bringing the evidence into one ticket. Engineers still need to ask whether the two situations are comparable and what other explanations remain.
Diagnostic samples do not necessarily represent the whole population. As OpenTelemetry explains, teams can preferentially retain slow or failed traces to investigate them. The resulting collection is useful for locating a delay and unsuitable by itself for measuring its prevalence. An alert identifies a change, a trace helps identify where it occurred, and representative production measures establish how many users it affected and whether they recovered. Combining those roles can turn a change in sampling into apparent performance improvement.
Review the recovery criteria, not just the patch
A low-risk change can reach a limited share of traffic first. The same workflow should show improvement while error rates, resource use and visible user problems stay in view. If removing a backend read saves time, a reviewer should ask whether it serves stale results, omits a required check or transfers work to another service. If measures disagree, the owner can pause the rollout, preserve a rollback path and wait out the agreed observation window before closing the incident. That decision involves impact and authority, not just whether the code compiles.
To evaluate the workflow, compare the time from alert to diagnosis, deployment and verified recovery with the previous process. Include false alarms, reopened cases and engineering review time. Faster investigations cannot justify their cost if the review queue simply fills with weak proposals. The public account does not quantify that net impact. It does describe a workable separation of duties. Teams can learn more from automating one recurring problem they can reproduce and reverse than from starting with every alert and counting how many tickets the agent closes.
Sources
Information only. No investment, legal, tax, or financial advice.