When the model changes, can a scientific result still be reproduced?

A research team can keep its data, publish its code and still discover that an AI-assisted result cannot be reconstructed a year later. The missing element may be a decision made between the two: which observations an agent excluded, which interpretation it selected, or which external service it called. A fluent account of the experiment does not necessarily reveal that decision.

Recent research makes this a practical design question. In Patterns, Dunn, Schultz and Musser (September 4, 2026) distinguish LLMs building saved analytical artifacts from LLMs required during execution. This distinction matters when deciding what a research group must preserve.

Our argument is that an AI-enabled research product should sell more than convenient execution. It should make the boundary of a claim inspectable: what was observed, what was decided, and what would have to change before the conclusion changed. That is a different promise from producing the same polished report twice.

Start with the decision that can move the result

Consider a hypothetical study of temperature sensors in a warehouse. An assistant helps a researcher compare conditions before and after ventilation equipment is replaced. It labels unusual readings, organizes records and drafts the chart. None of these are findings about an actual deployment; the example illustrates choices a future reader would need to examine.

Suppose a sensor records a sharp rise during maintenance. Excluding it may be appropriate if the sensor was disconnected. Keeping it may be appropriate if maintenance itself explains the conditions workers experienced. Both pipelines can execute successfully. The scientific disagreement concerns the question being answered and the reason for excluding an observation, not whether a program crashed.

A useful archive therefore connects each consequential choice to its input and rationale. The retained object might be a table of excluded observations, their original values, the rule applied and the person who accepted the rule. A transcript is helpful context, but asking another model to interpret a long conversation introduces an additional judgment. A future reader should not need to infer the exclusion policy from conversational tone.

This is where an archive differs from a recording. A recording captures activity. An archive designed around a claim makes the relationships necessary to challenge that claim accessible. It need not expose private data publicly; it can document which materials require controlled access and who can authorize inspection.

Repeating a number and defending a conclusion

The FABRIC preprint, revised October 7, 2026, reports AI assistance with setup but continuing human guidance when analytical workflows were unclear. Its case studies do not establish a universal productivity gain.

For our hypothetical warehouse study, three different checks would be useful. One asks whether the archived inputs and procedure reproduce the reported estimate. Another asks whether reasonable alternative handling of questionable readings changes its direction. A third asks whether the comparison supports the proposed explanation, given changes in weather or operating hours. Success on the first cannot settle the other two.

This separation prevents a costly misunderstanding. A team might spend considerable effort making an output identical while leaving the interpretation fragile. Conversely, small numerical differences may be acceptable if the scientific claim explicitly allows them and its conclusion remains stable. The acceptable difference should be justified in the study, not selected afterward because a rerun happened to miss the original value.

For the warehouse example, changing the question also deserves its own record. A comparison of average temperatures is not automatically evidence about workers' exposure during the hottest hour. Even with identical observations, selecting a different time window can change the meaning of the study. A reviewer should be able to see who proposed that change, why it was appropriate, and whether it was decided before or after inspecting the result.

A conversational interface still needs a stable research object

Paper2Agent, published in Nature on September 16, 2026, presents agents that expose papers' methods through tested executable tools. It demonstrates an approach to reuse rather than proof that every new question an agent answers is scientifically valid.

A natural-language interface could let a warehouse engineer ask for a different time window without learning the entire analysis package. That is useful. But the response should identify whether it reproduced the published comparison, changed a parameter within an established method, or introduced a new analytical choice. These are distinct services even if the interface presents them as adjacent chat messages.

There is a product opportunity in making these differences visible without forcing readers through every implementation detail. A result can point to a fixed analysis record, while an exploratory workspace permits new questions. A comparison view can show which assumptions changed. The familiar report then becomes an entry point to evidence rather than the only object available for review.

There is also a cost. Someone must maintain access, preserve compatible execution environments and decide which changes warrant a new version. A lightweight analysis may not justify a permanently hosted agent. A documented downloadable package may be the better research deliverable. Convenience should be evaluated against the expected life of the result and the people likely to challenge or reuse it.

The institution owns the continuity problem

These choices affect purchasing and staffing as much as research technique. If a laboratory buys an AI research service, the relevant question is not only how quickly it produces an answer. Can the institution retain the evidence needed to explain an answer after the subscription ends, the researcher leaves, or the service changes? Exportable text alone may be insufficient if an essential intermediate decision remains inaccessible.

Responsibility should be divided explicitly. The investigator owns the scientific interpretation; the platform can provide records and execution controls; the institution decides how long materials remain accessible. None of those roles can guarantee that an original claim is correct. Together, however, they can make disagreement more productive by allowing another person to locate and test the choice at issue.

The evidence cited here is recent but bounded: a methodological paper, a limited reproduction study and a framework demonstrated through selected cases. It does not show that AI-assisted science is generally unreliable, nor that one architecture solves reproducibility. It supports taking preservation seriously while AI tools are still being integrated into research practice.

For a useful research product, the final question is therefore concrete: when a reader disputes the conclusion, can they examine the relevant decision without rebuilding the entire conversation? Making that possible is substantial work. It gives institutions a more durable asset than an answer that is easy to generate but difficult to revisit.