After faster AI coding, what should validation infrastructure prove?

Blacksmith announced its Series C on October 21, 2026. The financing brings fresh attention to continuous integration, or CI, the infrastructure that builds and checks changes before they are integrated into a codebase. The announcement is a useful occasion to ask what a team is actually buying when it buys faster validation. Company announcement

For a team using coding agents, a plausible constraint is no longer the time needed to produce a patch. It is the time needed to establish whether that patch deserves to become part of the product. This is an analytical proposition, not a claim that every team has reached the same bottleneck. A small application with quick, reliable tests may remain constrained by product decisions. A large repository may have enough machines but too few people who understand its failure modes.

The distinction matters because validation has two costs. One is waiting for evidence. The other is deciding how much that evidence proves. Additional compute can reduce the first without improving the second. In some circumstances it can even make a misleading result arrive sooner.

A faster answer to an incomplete question

Consider a hypothetical retailer changing the way discounts are calculated. An agent modifies the pricing function and adds a test showing that a qualifying basket receives a discount. The build passes. The missing question is whether a refund, a currency conversion or two overlapping promotions still produces the intended result. If those behaviours are absent from the test specification, running the existing suite more often does not supply the missing rule.

This is not an argument against generated tests. An agent can help enumerate edge cases and turn a known failure into a regression test. The concern is independence: when implementation and expected output are derived from the same mistaken interpretation, agreement between them is weak evidence. A useful challenge comes from somewhere else, such as a product invariant, an existing customer contract or a deliberately adverse example reviewed by a domain owner.

A paper by Chenglin Li and colleagues, revised on August 6, 2026, reports that stronger tests exposed incorrectly accepted benchmark repairs. Its experimental setting was repository-repair benchmarks, not production incident rates. Research paper

The distinction between those settings limits what can be concluded. The research does not establish how often a particular company's releases will fail. For an operating team, the relevant inference is narrower: a passing test is evidence about the behaviours the test can distinguish. It cannot certify requirements that nobody expressed. That leaves test design as a continuing engineering responsibility even when execution becomes cheap.

The queue continues after the runner finishes

A second hypothetical team has a different problem. Its tests cover the change reasonably well, but each completed run creates a failure report that must be interpreted. Some failures concern the patch, some concern an unavailable dependency, and others concern stale test data. Doubling the number of candidate changes can create more diagnostic work without increasing the number of people able to resolve it.

The visible CI queue might shrink while the queue of unexplained results grows. Measuring only the time from job start to job completion would miss that shift. A more useful observation window follows the change until someone can make a defensible merge decision, including reruns and investigation. Otherwise, a supplier can meet its execution promise while the organisation sees little improvement in delivery.

Martin Fowler's April 14, 2011 essay describes how intermittently failing tests erode confidence in a test suite. That is longstanding engineering experience, not a new measurement of AI tools. Original essay

Our economic interpretation is that unreliable signals consume scarce attention. A reviewer who repeatedly encounters unexplained failures may learn to rerun rather than investigate. The cost is not confined to wasted machine minutes: the team also loses a dependable distinction between a bad change and a bad test. More testing is valuable only if the organisation preserves that distinction and gives someone responsibility for repairing degraded checks.

What throughput can and cannot tell a buyer

Google's September 23, 2025 account of DORA research linked AI adoption to higher delivery throughput while noting continuing verification challenges. This is historical survey evidence, not a controlled estimate of any vendor's effect. Research summary

It would be equally mistaken to dismiss faster infrastructure. Shorter feedback cycles can keep a developer close to the reasoning behind a change, and abundant test capacity can make previously expensive checks practical. The point is to identify which constraint is being relieved. Faster execution has direct value when trusted checks are waiting for resources; it has less immediate value when the unresolved issue is what correct behaviour means.

A purchasing comparison therefore needs a stable unit of work. Cost per CI minute is easy to observe but can reward an incomplete comparison. A faster machine may cost more per minute and less per completed suite. A cheaper suite may omit expensive checks. A low price per successful run may conceal repeated failures. For an internal evaluation, the useful unit could be a comparable change that reaches a reviewed release decision, with the scope of tests kept visible.

Even that unit is imperfect. Changes vary in risk, and incidents may appear well after deployment. A cosmetic edit should not be compared directly with a migration that rewrites customer balances. Separating classes of change and recording exceptions is less elegant than a single productivity number, but it makes a claimed improvement easier to interpret.

An operating model, not an unlimited test budget

A plausible design is to let inexpensive checks answer narrow questions quickly, while reserving broader checks and human attention for changes with a larger possible impact. This is an editorial operating hypothesis, not a universal sequence. A safety-critical component or an unfamiliar dependency may justify a different balance. The essential requirement is that a faster path does not silently remove the evidence on which approval previously depended.

There is also a cost to excessive caution. If every harmless edit requires the same expensive review, teams can lose the benefit of automation and create incentives to bundle unrelated changes. The aim is not to maximise test count. It is to make the remaining uncertainty explicit enough that the organisation can decide whether the change, its exposure and its recovery path are acceptable.

The commercial opportunity around CI is therefore broader than selling processors, but narrower than promising trust on demand. Suppliers can make evidence cheaper and easier to obtain. Teams still have to define the behaviours worth protecting, maintain the checks and interpret the results. As code generation accelerates, that division of responsibility becomes more visible in the cost of delivering a change that users can actually rely on.