Distillation defenses need a customer-side scorecard

The news and the evidence boundary

On October 12, 2026, Anthropic reported stronger distillation defenses, describing reduced unauthorized extraction alongside efforts to preserve legitimate use. This is a company assessment, not an independent replication.

A June 2026 preprint frames evaluation around query budgets, training-data budgets and API interfaces. It does not independently validate Anthropic's October deployment. Claude's evaluation documentation separately recommends measurable, task-specific criteria and comparisons with a baseline.

Analysis: two scorecards, not one headline

Open Market Notes' reading is that a procurement decision needs two scorecards. One asks whether a defense raises the cost of reproducing useful capabilities under stated conditions. The other asks whether an authorized customer can still finish its own work. A favorable answer to one does not logically establish the other.

Consider a hypothetical support application. It could produce fluent replies while becoming less reliable at preserving order identifiers or applying refund rules. A broad language-quality score might miss that failure. Conversely, a minor stylistic change need not mean the service has lost useful capability. Neither example describes an observed incident at Anthropic; both illustrate why buyers need their own acceptance criteria.

A defensible internal test would freeze a lawful, non-sensitive sample of real tasks, specify the expected business result, and compare the same workflow before and after a service change. Record the model identifier, date, prompt version, tools, retries and human corrections. Keep successful task completion separate from refusal rate: an answer can be accepted by the API and still fail the business task.

Measure recovery as well as error

Here is an illustrative calculation, not a vendor benchmark. Among 1,000 legitimate tasks, a rise in erroneous refusals from 1% to 2% creates 10 additional blocked tasks. If each needs six minutes of review, the extra burden is one staff-hour. That number says nothing about attack prevention. It makes one possible customer-side cost visible.

The operational question then becomes whether a team can identify, appeal and resolve an error without hiding it through repeated retries. Silent workarounds make the dashboard look healthier while moving the cost to staff. An approved fallback should preserve the original failure record and separately measure the outcome after recovery.

What remains uncertain

These sources do not give us an independent, application-by-application measure of the latest defenses. We therefore cannot infer a universal quality loss or promise durable protection. The useful next evidence would combine a clearly bounded security evaluation with reproducible customer-task results and a documented correction path. For a buyer, the relevant unit is an authorized workflow completed reliably, not merely a successful response.