Jailbreak ratings are taking shape. What risk record do AI buyers still need?

FMF released its preliminary cyber jailbreak rating framework, version 1.0, on November 5. Public comments close December 21. Our November 7 analysis examines what enterprise buyers should ask next. Announcement

For enterprise procurement teams, the consequential question is whether a supplier's rating explains the loss exposure of their own deployment. Our assessment is that the framework can help organize remediation priorities, but buyers need a separate deployment-risk record. Otherwise, a lower rating may be mistaken for stronger safeguards, while a capability that is difficult to use may be mistaken for an absent capability.

1. Identify what is rated before reading the color

The specification assesses individual jailbreaks and relative risk, with a comparison baseline that can change. It is preliminary, not operational guidance. These boundaries make the buyer’s own evidence requirements especially important.

Buyers should therefore ask what a score covers before asking how high it is. Connecting the same model to a read-only knowledge base, an internal coding environment or a business workflow with payment permissions could create very different paths to loss. These are deployment-analysis examples, not FMF test results. They illustrate the intervening permissions, tools and human approvals between bypassing a model safeguard and damaging an enterprise asset.

Procurement records should consequently retain two layers of evidence. The first describes the tested model version, jailbreak case and observed performance. The second describes the tools, accessible data and executable actions supplied by the enterprise. Mapping the two lets security teams explain why a model finding matters to a particular workflow, rather than copying a supplier's overall rating into an approval form.

Another procurement recommendation is to retain unsuccessful test records, rather than collecting only striking demonstrations. One success can reveal a gap in safeguards without establishing how often it occurs under specified conditions. Test scope, attempt counts and reasons for failure can support internal review, but a success rate from a small sample should not be presented as the probability of an attack in a real environment.

2. A moving baseline can change the risk story

Consider a hypothetical procurement report that compares a product only with whatever alternatives are currently available. Its relative position could improve because the product changed or because the comparison set changed. Those explanations imply different decisions about repair budgets. This is our analytical example, not a reported FMF test or a calculation of its rating.

This creates an easily overlooked management problem. As widely available capabilities improve, a jailbreak may become less exceptional without improving the attack conditions an enterprise faces. If a technique was scarce yesterday and common today, its reduced advantage over the market does not demonstrate that existing access controls are sufficient. This is a scenario inferred from relative scoring, not a prediction about actual attack volumes.

Quarterly security reports should therefore distinguish downgrades caused by remediation from those caused by a changing comparison baseline. Recording the assessment date, model version, baseline version and results of fixed internal tests would help. The former records explain why a score changed; the latter help determine whether the enterprise's defenses improved. Without them, an attractive trend line might simply show that the comparison set became stronger.

3. Account separately for usability and capability

Our recommendation is to ask two separate questions: what could an attacker accomplish with the capability, and what would be required to use it repeatedly? Discovery costs, expert time and automation are useful subjects for the second question. A difficult workflow can be an obstacle to use without demonstrating that the underlying capability is weak.

That distinction matters for budgets. A case requiring sustained expert involvement may temporarily limit the scale of misuse while still exposing a capability worth addressing. Conversely, an easily reusable case does not automatically establish a substantial capability gain over existing tools. Compressing both into a single claim that the model is dangerous or the attack is difficult obscures where remediation spending should go.

Enterprises can ask suppliers to explain capability evidence and conditions of use separately, while including retesting costs in maintenance budgets. After model updates, changes to connected tools or revisions to safeguards, the continued validity of earlier findings requires verification. This is a recommendation for evidence management, not for publishing dangerous prompts or executable attack material. Auditable scope, methodological descriptions and controlled validation results are more useful to procurement than circulating attack recipes.

Supplier comparisons should likewise avoid comparing only the final color. A supplier conducting broader tests and disclosing more evidence may show more problems; another publishing less information is not safer as a result. Buyers can first compare evidence coverage and retest conditions, then discuss rating differences. This does not rank any vendor. It cautions procurement teams against treating missing information as evidence of low risk.

4. Turn a public framework into an accountable process

For now, buyers should favor evidence that independent reviewers can examine over safety badges on marketing pages. A supplier should explain what was tested, where uncertainty remains and what evidence would change its conclusion. A usable contract should then connect those findings to an owner and a response. These are our procurement recommendations, not provisions that the new framework has imposed.

Buyers can translate this into three contract questions: which changes trigger reassessment, how customers are notified of material changes, and who maps model risks to business permissions. Smaller teams do not need to begin with a large new dashboard. A short record naming the owner, retest conditions and exception-approval process may be easier to put into practice.

For security leaders, a useful deliverable should answer a plain question: if the model or business permissions change tomorrow, who will notice, who will review the change, and who can pause the relevant workflow? A rating can help reduce losses only when incorporated into that assignment of responsibility. A common scorecard may improve communication, but enterprises must still define responsibilities, validation resources and response deadlines.

The framework's future value should be judged by increasing public validation, more comparable rating evidence and faster remediation of consequential problems. It could reduce communication costs between security teams, but it will not decide acceptable losses for buyers. What enterprises ultimately need is a decision trail connecting evidence to an accountable owner.