One Encoder, Two Error Budgets

Original analysis · October 16, 2026.

DMIS’s October 14 UltraBERT release separates pretraining heads by masking regime, targeting classification and retrieval.

The useful question for a business is which mistakes it can afford. A search system that includes an irrelevant document has created extra work. A routing system that excludes the right destination may have removed the possibility of answering correctly. Those costs remain different even when the two systems start from the same model family.

Our central argument is that consolidation should stop at the loss function: the business rule for valuing a mistake. A common technical foundation can be attractive without making one average quality score a sensible deployment target. The most revealing test is a workflow in which classification determines what retrieval is allowed to see.

The error before the search

Consider a hypothetical support product. A classifier first decides whether a request needs the technical knowledge base. A retriever then selects candidate documents for the requests admitted. In this design, the classifier is a gate, while the retriever is a shortlist maker. The gate can discard an opportunity that the shortlist maker never gets to evaluate.

Suppose 1,000 requests genuinely need that knowledge base. The gate admits 900. For those 900, the retriever finds a relevant document in its candidate list 90% of the time. The combined system succeeds on 810 requests, or 81% of the original 1,000. These are illustrative assumptions, not UltraBERT measurements. The calculation uses retrieval success conditional on admission; it does not assume independent errors.

Now improve that conditional retrieval rate to 95% while leaving the gate unchanged. The total rises to 855. By comparison, admitting 980 requests at the original 90% conditional retrieval rate gives 882 successes. The latter comparison assumes the newly admitted requests are equally retrievable, which may be false. Its purpose is to show what must be measured: the difficulty of the requests the gate previously hid.

A team celebrating only search quality among admitted requests could miss this distinction. The system can look cleaner after a stricter gate simply because it stopped attempting the difficult work. We would therefore report successful evidence retrieval against all requests that need evidence, including the ones rejected upstream.

Give wasted work and missed work different prices

Scikit-learn documents precision and recall across decision thresholds.

Our economic extension is to choose a gate by the consequences of admitting and excluding work. A false admission might consume a search, a reranking pass, and some review time. A false exclusion might produce an unresolved case, a repeat contact, or an unnecessary escalation. The decision is not automatically to admit more. It is to compare the marginal costs rather than count both errors as equally undesirable.

For example, imagine a proposed threshold change causes 200 extra unnecessary searches while recovering 20 cases that would otherwise be missed. The extra searches pay for themselves when the average net value of recovering one case, after its own handling costs, exceeds ten times the incremental cost of an unnecessary search. This is our simplified arithmetic, not a price forecast. Any added delay, misleading search results, and downstream answer errors also belong in that cost comparison.

This matters for incentives. The infrastructure owner may be rewarded for fewer searches, while support is measured on resolved cases. Both teams can improve their own dashboards while the overall product becomes worse. A shared encoder should make these competing objectives easier to discuss, rather than disguise them behind one model score.

Retrieval has a different margin

The BEIR research benchmark evaluates retrieval across varied tasks and domains.

For our support example, the commercial question is narrower: how many useful candidates should the next stage receive? One additional candidate may recover missing evidence, but it may also consume attention or introduce a plausible distraction. Its value depends on what the downstream system does with it. An analyst who scans a shortlist and an automated answer generator do not necessarily benefit from the same list length.

We would price retrieval changes at that boundary. Measure whether the added candidate actually helps complete the case, how much extra review it creates, and whether it displaces a more useful item. Keeping a relevant document somewhere in a long list is not equivalent to putting usable evidence where the next stage will inspect it. Nor is a high similarity score a business decision to act.

This leads to two distinct optimization questions. The classifier asks whether opening a path is worth its expected cost. The retriever asks which additional evidence is worth passing onward. They may use related representations, but their operating points should reflect different consequences. There is no general reason the same level of selectivity should be optimal for both.

Test the interaction, not only the parts

Our proposed experiment uses the same held-out requests in four configurations: existing gate with existing search, new gate with existing search, existing gate with new search, and both new. Keep the underlying knowledge base fixed. This separates the gain from each change and exposes interactions that two isolated test reports would miss.

Label the rejected requests too. Otherwise, a gate can improve its apparent efficiency by removing exactly the cases needed to judge its cost. Track the small but expensive categories separately from the common easy cases, and keep enough examples to avoid turning one unusual outcome into a sweeping conclusion.

A shared foundation also creates a question about overlapping weaknesses. If the gate and retriever miss the same kinds of language, a second stage may provide little rescue. That is a hypothesis to test, not a claim about this release. Examine the actual overlap of failures rather than assume that two model calls provide two independent chances.

The opportunity is better reuse of engineering work. The constraint is that a saved operation and a lost answer have different values. A business can consolidate its model stack while preserving separate error budgets for opening the search path and choosing the evidence that travels along it.