AI Escalation Needs an Independent Scorekeeper

A buyer evaluating an AI handoff system should reserve part of the audit budget for tasks that never get handed off. Otherwise, the system deciding which cases deserve help also decides which cases receive scrutiny. Open Market Notes proposes a procurement test that separates those two decisions.

Two routing papers appear in arXiv’s October 16, 2026 listing. Listing

LEASH uses uncertainty for computer-use handoffs and continued monitoring. Study

A separate study learns routing from model disagreement without ground-truth labels. Study

Both preprints were submitted October 15. Our experiment below is an independent proposal, not a reproduction of their evaluations or a claim that either system has passed a customer trial.

Build a comparison the router cannot select

Imagine an office-supply distributor testing an assistant that prepares return instructions. A low-cost model handles routine requests; another model is available for difficult cases. Before the pilot, the distributor creates a frozen set of completed cases, with the applicable policy, product record and final disposition. A reviewer who did not build the router establishes acceptable outcomes, including cases that require human judgment.

Run both models separately on the same case files in a sandbox. Do not let either see the other’s answer. This is an audit exercise, not permission to issue return labels or contact customers. Keep model versions, source documents and scoring rules fixed. Repeated trials are needed where outputs vary.

For illustration only, suppose a deterministic 200-case exercise gives four groups: both models are correct on 140 cases; only the second is correct on 35; only the first is correct on 10; both are wrong on 15. The first model therefore succeeds on 150 cases and the second on 175. An impossible selector that knows the answers in advance could reach 185. These invented counts describe no published model.

The practical target is the gap between the deployed router and these separately scored alternatives. A handoff rate alone cannot reveal which group received the extra spending. Report the four groups before compressing the result into one success percentage.

Audit the quiet cases

After fixing the router, sample from both the retained and handed-off queues. Choose the audit sample independently of the router’s confidence score. Otherwise, confident mistakes can remain outside the measurement process. If difficult categories are deliberately oversampled, reweight the overall estimate to the actual workload and also report each category separately.

Include a deliberately incomplete file: remove the product identifier needed to decide which return policy applies. The acceptable result is a request for the missing information. A fluent answer from either model should fail this case. This tests a purchasing requirement we propose: extra computation must not silently substitute for unavailable evidence.

Also include cases in which the first answer is already acceptable. Record whether a handoff preserves it, improves a required detail, or introduces a new error. The reviewer should apply the same acceptance rule regardless of the model’s brand, price or position in the sequence.

Follow the money beyond inference

Keep the pilot ledger in separate columns: model charges, routing overhead, human review, repeated attempts and reopened returns. Calculate total workflow cost per accepted case, while reporting unresolved and abandoned cases explicitly. A lower invoice from the model provider is valuable only if the rest of the workflow does not consume the saving.

Audit spending is part of this design. A customer may reasonably decide that a small, recurring independent sample costs less than reviewing every output. That choice still needs an error threshold and a named owner. There is no universal sampling rate in this article; consequence, volume and observed variation determine it.

Make the evidence survive an update

Retain a sealed test set that was not used to tune the routing threshold. Before changing either model or the return policy, rerun the comparison and inspect which cases changed groups. A supplier should not earn a better score merely because the test was adjusted after its failures became visible.

The buying question is concrete: can the supplier show independently assessed outcomes for work that stayed with the cheaper model, work that moved, and work neither model could finish? Routing may be a useful way to allocate computation. The customer still needs control over the evidence used to judge the allocation.