Stripe tested AI-built payments. A working checkout is only the first gate
Stripe reports criterion scores and all-must pass rates separately. The gap helps explain what merchants still need to verify.

On November 19, Stripe published an evaluation of payment integrations built by coding agents. It covered 69 scenarios and 14 model-and-harness pairs across one-time payments, subscriptions and Connect. Each pair was evaluated with and without Stripe’s plugin. Stripe’s aggregate example puts average criterion performance at 78.1%, against a strict pass rate of 29.4% when all required criteria have to pass. Those figures measure different things. Neither is a forecast of merchant launch success. A build can satisfy many requirements and still miss the one that changes what a customer receives or how the business accounts for it.
Write down what must never happen
Consider a subscription merchant that accepts a payment method with delayed confirmation. The customer completes the checkout form, and the application activates service before the payment outcome arrives. A test with a normal card can sail through. The same application can grant access prematurely for another method. Stripe’s developer guidance distinguishes checkout completion from the later success or failure of a delayed payment. The shop has to decide which event is sufficient to change its own state and who investigates exceptions, rather than letting the browser screen imply that funds have arrived.
Notifications can be duplicated or arrive in a different order than expected. A late refund confirmation is not a reason to send the difference again; a repeated payment notification should not trigger a second fulfillment. Good acceptance criteria specify the result: fulfill once per order, reconcile the remainder after a partial refund, and retain enough evidence to explain a difference between provider and local records. Finance and operations should own those decisions. The examples illustrate what to guard against; they are not claims about individual failures Stripe observed.
Use partial credit to diagnose, not to approve
Stripe used a role-playing model to represent the scenario and a judge model to review observable evidence such as code and responses. Criterion scores reveal where an agent improves. Requiring all essentials exposes how one omission affects the entire scenario, but it is also more sensitive to judging errors: evidence overlooked by a judge can turn a run that worked into a failure. Neither aggregate settles which system a particular merchant should use. A seller with hosted checkout and a platform dividing funds among many sellers do not face the same costs when requirements are missed.
Stripe reports higher criterion scores for all 14 pairings when its plugin is available. Platform-specific context can help an agent locate the right information, but this is a provider testing its own plugin on its own scenarios. The result should not be extrapolated to other providers. An internal comparison can hold the code base and merchant samples constant, then record how much human review each configuration needs. A higher score by itself does not establish better reconciliation or live payment behavior.
Compare tools on the cost of accepted integrations
Model charges are only one part of delivery. Preparing cases, running tests, reviewing outcomes and adding missing reconciliation or support flows take time; mistakes found after launch can add customer-support and accounting work. Compare time and total cost per accepted integration, with failed essential criteria reported separately. Start with common sandbox paths and a few representative exceptions. Decide which failures require a named owner to review, and preserve evidence instead of rerunning tests until the best result appears.
For a small shop, hosted checkout can limit the amount of code it owns, but order and payment states still need to agree. Subscriptions and split payments invite closer scrutiny of ledger matching, allocation and exception ownership. The useful output of an evaluation is an earlier view of which business consequences still need proof, not merely a number that looks good on a deployment dashboard.
Sources
Information only. No investment, legal, tax, or financial advice.