OpenAI reported 54% falling to 0.2% in one safeguard test. What did it measure?
OpenAI’s figures cover one protected-reasoning test. Deployment needs separate evidence on missed incidents, false blocks and review work.

On November 20, OpenAI reported test results for safeguards against unauthorized model distillation. In one evaluation of protected reasoning from an earlier conversation turn, the average disclosure rate fell from 54.0% with GPT-6 Astra and its original safeguards to 0.2% with GPT-6.2 Astra and its updated safeguards. Distillation is the practice of using one model’s output to help train another; it is not intrinsically abuse. The reported result measures disclosure of particular information under test. It does not show that another model was trained, that a model was copied or that incidents occur at that rate on a production API. The November 17 GPT-6.2 system card adds release context. It comes from the same provider and is not independent verification.
The comparison does not isolate what changed
The experiment suggests a sizeable change between two model-and-safeguard combinations in its tested setting. Both the underlying model and its safeguards changed, so those figures alone cannot apportion the improvement. Where feasible, comparing both safeguards on the same model and then checking a separate, held-back sample would help. Where matched tests are unavailable, a report can name that limitation. Readers also benefit from knowing the number of observations, how the average was formed and how much uncertainty surrounds a low-frequency result. Without that context, it is hard to tell what 0.2% says about conditions beyond the mix tested.
An average can conceal a concentration of failures. Imagine a version that performs poorly only in long conversations: a test set full of short ones can produce a low overall rate while a business built around long sessions faces a different result. Useful summaries can separate language, session length or product capabilities, and flag groups with too little evidence to interpret. That does not require publishing instructions that could enable abuse. An authorized independent reviewer can examine the scope, versions, resources and sampling rules and report at an aggregated level.
Measure the work that gets blocked by mistake
Consider a support team that asks a model to summarize many tickets using a consistent format. Repetition is normal for that workload. If protective measures block it incorrectly, employees may retry the requests or move back to manual work. An evaluation should record legitimate task completion, the false-block rate and time to correction. A completed preventive block, escalation for human review and temporary account restriction also have different implications. Counting them as an interchangeable success would conceal the operational cost.
NIST’s AI risk management guidance encourages documenting test sets and methods and paying separate attention to expensive pockets of failure. An application team can maintain a set of common workloads without sensitive records and compare normal completion across versions. A risk team can maintain a separate authorized defensive evaluation for the behavior the safeguard is designed to prevent. Putting both results in the same release review makes clear whether lower rates of missed incidents come with acceptable friction.
Production monitoring is a different measurement
A provider test helps compare releases. It cannot serve directly as the rate of production incidents, where the request mix, versions and configuration differ. An operations team can retain de-identified, reviewed records that distinguish an unconfirmed alert from a substantiated incident and include scope, time to resolve and outcome. If alerts rise after deployment, the cause could be growing traffic, a change in detection or an increase in validated incidents. The alert count itself does not choose between those possibilities.
OpenAI’s figures are evidence of progress in a controlled setting, not a description of every deployment. To explain the protection that customers experience, teams need to keep the tested scope, harm from blocking legitimate work and deployed evidence visible. Naming the model versions and the remaining unknowns is more useful than carrying forward a single percentage.
Sources
Information only. No investment, legal, tax, or financial advice.