Reading the experiment is not running the lab: what BioBasicsBench should change

OpenAI introduced BioBasicsBench on November 6, 2026: 100 raw wet-lab tasks, developed with Ligo. Its reported percentage measures rubric credit, not whole-task success.

That distinction should shape the buying decision before any model ranking does. A laboratory does not buy an average score. It adopts a way of allocating attention, accepting conclusions and deciding what happens next. An assistant may make one part of that process cheaper without making the whole process safe to delegate.

Three different questions, not one ladder

The July 14, 2024 LAB-Bench paper introduced more than 2,400 multiple-choice biology-research questions. The August 24, 2026 BenchBench-Protocol preprint instead draws 149 tasks from scientists’ real protocol modifications. These are different assessment designs, not interchangeable units of progress.

Selecting an answer, interpreting evidence and adapting a plan put pressure on different parts of a system. A good answer selector might benefit from options that expose the intended distinction. An open-ended assistant must decide which distinction matters before it can answer. A planning assistant must additionally preserve the dependencies between choices. None of these observations establishes that one benchmark is universally harder; it explains why a cross-benchmark leaderboard can answer the wrong question.

For procurement, the useful comparison is between the decision a test asks for and the decision a user intends to delegate. If they differ, a strong score is a reason to investigate, not an authorization to deploy.

Partial credit has an economic shadow

Rubric scoring can make incremental progress visible. But an organization still needs to decide which mistakes are tolerable, reversible or expensive. Two answers with equal credit could have very different operational consequences: one may omit a useful explanation, while another may support a conclusion that should have remained unresolved. This is an analytical possibility, not a claim about the benchmark’s scoring of any particular answer.

Consider a hypothetical assistant reviewing a measurement image. It describes the visible pattern accurately but lacks the context needed to choose between two explanations. If it returns a qualified observation, it can help a scientist focus a review. If it turns the same observation into a confident decision, the extra fluency creates work rather than removes it. The value depends on where the output enters the workflow. No experimental procedure is implied by this example.

This also complicates the familiar productivity calculation. Time saved drafting an interpretation is not automatically time saved reaching a trustworthy conclusion. Verification, resolving disagreements and recovering from wrong decisions belong in the same cost ledger. A faster first answer can be a useful product and still be a poor substitute for an accountable final judgment.

An evidence trail is part of the product

Our view is that useful lab assistance should make the boundary between observation and inference inspectable. A reviewer should be able to distinguish what the input shows, which context the assistant used and what remains unknown. This is a proposed design standard, not a capability established by these publications.

Such a record need not expose a model’s private internal reasoning. It needs concise, checkable claims tied to the relevant input and an explicit statement of the decision being requested. A second reviewer can then challenge the claim without reconstructing an entire conversation. That makes disagreements informative instead of merely inconvenient.

There is a competing concern: excessive qualification can burden experts with low-value review. The answer is not to maximize caution in every sentence. It is to measure whether the assistant sends the right cases for review and whether that review changes outcomes. A system that never commits may avoid visible mistakes while contributing little; one that always commits may look efficient until its errors propagate.

What would justify broader use?

A credible adoption case would connect performance to a defined role, a relevant mix of inputs and the cost of the remaining errors. It would distinguish assistance from autonomous execution and show what happens when the input is incomplete or unfamiliar. It would also compare the complete assisted workflow with the existing workflow, rather than compare isolated answer times.

These are criteria for future evidence, not results we have independently measured. The three sources supply different views of assessment; they do not, by themselves, establish end-to-end laboratory autonomy or universal performance across laboratories.

The editorial conclusion is deliberately narrower than an autonomy forecast. Better interpretation can be valuable long before a system deserves final decision authority. The commercial opportunity lies in making that limited contribution dependable and legible. The mistake would be to erase the limit because the benchmark number is easy to advertise.