En
← All articles

Coding agents need a better account of completed work

The McKinsey–Cognition alliance puts measurement in focus. More generated code is not yet evidence of more valuable delivery.

ColumnAI FrontiersAuthorOpen Market NotesTypeArticle
Two stacks of code sheets pass through a red review gate toward two checked acceptance forms.

On November 16, McKinsey and Cognition announced an alliance combining Devin with enterprise transformation services to scale software development and measure business value. Announcement

This is a commercial partnership, not an experiment proving equal returns for every company. It raises a useful question: as code becomes easier to generate, how can buyers distinguish more activity from more completed work? The following is Open Market Notes analysis, not an independent test of Devin.

A pull request is not yet a business outcome

Coding tools leave countable records: generated lines, created tasks and proposed changes. These describe activity, but not necessarily value. A change may be checked and deployed, or abandoned because requirements changed. Both produce activity records; their economics differ.

An acceptance unit could instead be a bounded change, such as an interface migration meeting agreed compatibility and test requirements. This is a hypothetical management example, not a customer case disclosed by the partners. Measurement should include task definition, environment preparation, human review, corrections and post-release rework. Counting only agent runtime leaves other participants' costs outside the account.

That does not dismiss fast generation. It connects local efficiency with the wider process. If a test environment still takes three days to obtain, saving two coding hours need not bring delivery forward by two hours. Evaluation must identify where time was saved and whether later stages can absorb the additional work.

Old conclusions and new measurement problems both matter

METR's randomized study of early-2025 tools found that participating experienced open-source developers took 19% longer with AI allowed. This concerned particular people, projects and tools at that time. Original study

In February 2026, METR said selection effects and difficulty timing parallel agent use made its follow-up experiment an unreliable estimate of current productivity effects. Research update

Read together, these findings resist convenient slogans. Applying the earlier result directly to today's products ignores changing tools. Treating developers' reluctance to abandon AI as a precise measurement of gains also exceeds the evidence. Enterprises need observations suited to their own work rather than a single outside number.

Parallel work is especially easy to miscount. An engineer assigning five tasks cannot simply add five agents' runtimes and call the sum saved labour. The engineer may wait, do other work or face a concentrated review queue. Elapsed time, actual human effort and completed output need separate records.

A pilot must retain its failures

Before starting, teams can choose bounded, manageable-risk work and record existing practices and acceptance criteria. Difficulty, repository familiarity and dependencies should be reasonably comparable. Where random assignment is impractical, the exercise should be described as observational, without attributing every difference to the tool.

Incomplete, reworked and abandoned tasks must remain in the account. Reporting only successes selects a favourable answer in advance. Failure need not mean a product is useless: it may identify unsuitable tasks, missing permissions or inadequate tests. Categorising those causes can improve subsequent allocation.

DORA's 2025 research describes AI as amplifying existing organizational strengths and weaknesses. Report That is not a law guaranteeing a particular result at every company. It does suggest that clear requirements, dependable tests and actionable review rules cannot disappear from the explanation. They are part of the deployment being evaluated.

Saved time needs a destination

Suppose a team saves implementation time. Financial value could take several forms: less overtime or outsourcing, completion of previously deferred work, or more attention to quality. Each may matter, but they are different benefits and must not be counted twice.

Multiplying saved hours by salaries estimates capacity; it does not automatically establish a cash saving. If staffing and spending remain unchanged, the company may have gained capacity rather than reduced its bill. Its value depends on what happens next, whether customers need the additional output and whether it adds maintenance obligations.

The evaluation also needs subscriptions, model use, runtime environments, integration, training and human supervision. Not every cost is easily allocated, but recording uncertainty is better than assuming it is zero. Early pilots can publish ranges and assumptions, revising them with operating evidence rather than prematurely promising a stable return multiple.

The test continues after the project ships

The alliance packages product capability and implementation services into an enterprise purchasing route. Customer projects must establish whether it works. Initially, completed tasks should meet the same acceptance standard. Over time, teams need to examine incidents, review queues and maintenance created by the extra output.

Commercial validation need not mean fewer engineers. A worthwhile project that could not previously be funded, but can now be completed reliably, may also generate a return. Whichever goal is chosen, it requires an agreed definition of completion, full costs and a later review of results. Code generation is the starting capability; buyers need an explainable improvement in delivery.

Information only. No investment, legal, tax, or financial advice.