En
← All articles

Stitch Loop makes post-launch improvement an evidence problem

An agent can propose a change continuously. Proving its effect and authorising its release remain separate decisions.

ColumnAI FrontiersAuthorOpen Market NotesTypeArticle
Looping arrows connect a metrics dashboard and checklist; a hand stops at a red approval gate.

Google announced experimental Stitch Loop on November 18, opening a global waitlist with priority for eligible AI Ultra users. Its proposed changes require pull-request review before merging. Google announcement This is not general availability, nor evidence that an autonomous optimisation system has already improved every participating product.

The significant product question is what happens after a proposed improvement reaches real users. A rising metric may reflect the change, a promotion, a different audience or a broken event definition. An agent that keeps working toward a target needs an equally persistent record of what changed and why the result is credible. The following is an evaluation approach, not a description of undocumented Stitch controls or a hands-on test.

Write the objective before the patch

“Improve conversion” is too incomplete to serve as an acceptance condition. Specify the eligible population, numerator, denominator, observation window and business event that counts. A completed checkout should not silently become a button click because that event is easier to increase.

Keep the measurement definition versioned separately from the code change. If a patch also changes event collection, the team must establish that the before-and-after measures remain comparable. Otherwise, the first achievement may be a new instrument panel rather than a better product.

Microsoft's pre-experiment guidance treats user outcomes, guardrails, feature metrics and data quality as a shared measurement set. Experiment preparation For this use case, our proposed brief would name one primary outcome, the groups it applies to, the minimum useful effect and the evidence needed to stop or proceed. The agent can propose that brief; a product owner should accept its tradeoffs.

An improvement needs a comparison

Consider a hypothetical checkout that rises from 4% to 5% completion after a release. If a concurrent control also rises from 4% to 5%, the movement supplies no positive difference-in-differences for that release. The arithmetic is illustrative, not a Stitch result; causal interpretation still depends on assignment, measurement and other assumptions.

Where appropriate and ethical, a randomised concurrent comparison helps distinguish a change from the traffic environment. Keep exposure assignment stable and record overlapping releases. Where randomisation is infeasible, a staged rollout and historical comparison can be useful monitoring, but should carry a weaker causal claim. Do not rename an observational association “verified impact”.

A low-traffic product may need more time than its team wants. A report saying evidence is insufficient can be more valuable than a daily stream of confident winners. Urgent corrective maintenance should not be delayed merely to preserve a control group; its verification plan can prioritise functional correctness and user protection.

Monitor harm without repeatedly declaring victory

Microsoft's during-experiment guidance distinguishes guardrails from success metrics and warns that repeated early readings need appropriate statistical treatment. Experiment monitoring That supports two separate decisions: stop unacceptable harm promptly, and claim a durable benefit only under the chosen analysis plan.

For a proposed checkout change, use completion as the objective while monitoring payment errors, latency, support demand and accessibility. A shorter flow that removes necessary information can raise a narrow metric without serving the user. Review important segments and delayed outcomes rather than only an aggregate on the first day.

Freeze the success criterion before observing treatment results. If the team changes it, record a new hypothesis and evaluation window. Continuous optimisation should not become continuous editing of the definition of success.

Read, propose, merge and deploy are different permissions

Reading approved telemetry should not imply permission to read every customer record. Proposing a patch should not imply authority to merge it, alter repository protection or change production configuration. Deployment is another decision even when a merge normally triggers it automatically.

GitHub documents protected-branch mechanisms such as required reviews and status checks. Repository controls Their presence in GitHub is not proof that a particular integration has enabled them. Test the actual repository policy, including who can bypass it, and preserve an accountable reviewer.

For an initial trial, our suggested boundary is one repository, a limited telemetry view and a narrow class of reversible changes. Remove credentials and unnecessary personal data from diagnostic material. Grant only the access needed for the accepted task, with a named owner able to revoke it. This is defensive deployment hygiene, not a claim about a vulnerability in Stitch Loop.

Close the loop with a decision record

Every accepted change should connect a hypothesis, code revision, reviewer, exposure period, data-quality checks, outcomes and rollback decision. If several agents contribute, keep their proposed changes distinguishable. Otherwise the team may know that something improved while being unable to identify which intervention deserves credit.

A trial should count accepted improvements that survive review and remain beneficial, alongside review burden and reversions. Pull-request volume is an activity measure. It does not establish product value.

Stitch Loop's announcement makes ongoing product work a useful category to examine. The adoption decision should rest on whether the team can trace a measured outcome to an authorised change, with visible uncertainty and manageable recovery. That is the standard for trusting the loop after deployment.

Information only. No investment, legal, tax, or financial advice.