When AI ranks research ideas, keep the rejection decision separate

On October 27, 2026, Ai2 discussed unstable AI novelty judgments. The underlying paper first appeared October 1 and was revised October 26; generated-idea labels include assumptions. This is an evaluation study, not a verdict on all AI research. Ai2 announcement, paper and version history.

For a research team, the immediate question is not whether to stop using model judges. It is where their output enters the decision process. A low score that moves an idea farther down a reading list is different from one that deletes it before anyone can inspect it. The same assessment can have very different consequences depending on that boundary.

A ranking tool can quietly become a gatekeeper

Consider a hypothetical team that can test only a few proposals each month. It asks an agent to produce candidates, then asks another model to rank them. This arrangement may save reading time. But if the team examines only the highest-ranked proposals, it cannot learn much about the quality of the discarded ones. A neat dashboard can hide that missing evidence.

Our analysis is that the workflow should preserve the rejected pool, not simply the winning proposal and its score. Otherwise, a later successful experiment cannot tell the team whether its filter was good or whether an even better direction was screened out. The relevant unit of accountability becomes the entire queue: what entered it, what moved forward and what was never examined.

That does not mean every idea deserves an expensive experiment. A small, deliberately varied sample from below the threshold can reveal whether the filter systematically overlooks a particular kind of proposal. The sample should not consist only of near-winners. Ideas with unfamiliar methods or awkward wording may deserve a separate audit precisely because the ranking gives them little attention.

An audit record also helps separate filtering mistakes from execution mistakes. A selected project may fail because experimental conditions were inadequate; a discarded idea that later proves useful does not necessarily mean it deserved the whole budget at the time. Preserving the conditions visible when the decision was made helps avoid rewriting the past solely around eventual success or failure.

Ask what the score is comparing

The older MT-Bench study documented position and verbosity biases in model judging, while also finding useful alignment with human preferences in its setting. That is context, not a direct replication of novelty evaluation. Original study.

A research manager therefore needs a more explicit question than “Is this new?” New relative to which literature, for which research community and at what date? A familiar mechanism applied to an overlooked population might be useful without being conceptually radical. An unfamiliar mechanism might be original but impossible to test with the available data. Collapsing these issues into one number obscures the choice the team actually faces.

A hypothetical proposal to reuse an established measurement method illustrates the distinction. The novelty assessment might reasonably be modest. Yet the proposal could still resolve a consequential uncertainty at low cost. Conversely, an imaginative proposal with no feasible observation that could disprove it may be a poor use of the next experiment slot. Novelty, usefulness and testability belong in separate fields, even when the eventual decision combines them.

The judge’s explanation should make that comparison inspectable. A reference list alone is insufficient: a reviewer needs to see which earlier claim is alleged to overlap, and which part of the candidate remains different. Where no relevant comparison was found, the record should say so rather than converting incomplete retrieval into a confident assertion of originality.

This distinction also directs the next task. If the disagreement concerns whether an equivalent experiment already exists, a targeted search may help. If it concerns feasibility, experimental design or domain expertise may be needed. Asking the model the same broad question repeatedly does not necessarily supply the missing kind of evidence.

Preserve a route back into the queue

The NIST AI Risk Management Framework is voluntary guidance covering trustworthiness across AI design, use and evaluation. It is not a novelty certification. NIST framework.

The operational design below is our own analysis. Keep the candidate text, comparison evidence, model and prompt version, and the reason for advancing or holding an idea together. If a model changes, recheck a stable selection of previously reviewed cases before treating its new scores as continuous with the old ones. A different score scale should not silently rewrite the team’s past decisions.

Disagreement can also be an instruction to gather evidence rather than a reason to take a majority vote. If two judges disagree because one compares the method and the other compares the application, averaging their scores produces precision without resolving the question. A human reviewer can instead decide which comparison matters for the project, or commission a targeted literature check.

There is a reasonable competing argument: tight research budgets require aggressive filtering, and human committees also make mistakes. A cautious workflow can become so elaborate that it defeats the purpose of automation. The answer is proportionate control. Use inexpensive ranking for reversible scheduling, and require stronger evidence when exclusion is hard to reverse or when the idea would otherwise disappear.

This approach does not establish that a particular judge is reliable. It makes the cost of being wrong more visible and keeps an avenue for correction. A research team can then assess the assistant by whether it improves the allocation of expert attention, not merely by whether its winning ideas receive persuasive novelty scores.