Stripe Evals Test How Coding Agents Build Payments

Stripe evals results for coding agents building payment integrations

Stripe evals, a new benchmark from Stripe, measure whether coding agents build payment integrations that are actually correct. Every agent tested scored better with Stripe’s agent plugin, but the strictest grading shows how far they still have to go: the average pass rate falls from 78.1% to 29.4%.

Stripe is also publishing the benchmark and its eval framework. The company says it will keep the framework maintained and update the results as new models, cases and features arrive.

Why a new benchmark

Stripe says it has seen coding agents lean on older APIs, because those appear more often in their training data. Agents also miss newer features and sometimes invent details. Common coding benchmarks don’t catch this, since they run deterministic checks on whether code passes, not whether the API integration is right.

The integration choice matters to a business. Current patterns give access to new capabilities with less custom code and fewer migrations later. A good build also covers payment failures, subscription changes and webhook events.

How the tests ran

The suite has 69 public scenarios: 24 for one-time payments, 26 for subscriptions and 19 for Connect, Stripe’s product for marketplaces and platforms. Tasks include building a hosted checkout, metered billing, per-seat subscriptions and splitting marketplace payments. Each ran twice.

Stripe tested 14 pairs of models and coding tools, using Claude Code, Codex and OpenCode with a sample of models that includes open-weight ones. Each pair ran with and without the Stripe agent plugin, which adds Stripe-written skills and tools through an MCP server, including an implementation planner and documentation search.

A second agent plays the developer. It starts from a goal, holds a private set of two to six needs and asks one follow-up question at a time. It doesn’t write code or reveal its list. A separate judge grades each criterion against the transcript and the files the agent produced, with criteria marked as required (MUST) or recommended (SHOULD).

What the plugin changed

All 14 pairs improved, by roughly 3 to 8 percentage points. Outdated parameters in API calls fell from 7% to 1.3%. In one payments scenario, GPT 6.1 Sol used a recently removed parameter without the plugin and wrote the correct automatic payment methods setting with it.

  • Web searches fell 96%, and agents did about 1.2 fewer searches and 3 fewer fetches per eval.
  • Agents wrote about 40% fewer test lines and ran tests in 42% of evals, against 53% without the plugin. Stripe says that doesn’t mean fewer tests raised scores. Better information reduced the urge to over-check.
  • Behavior varied. Fable 5.1 made 53% more tool calls with the plugin, Gemini 3.8 Flash made 26% fewer, and Argon and Grok fetched more pages.

The last mile problem

Under strict grading, an integration passes only if every required criterion passes. The average then drops to 29.4% from 78.1%. Stripe reads this as models producing almost correct code that stumbles at the end.

One example: in a fulfillment scenario, Opus 5.5 checked only that a PaymentIntent had succeeded, not whether the order was eligible to ship. That could let orders go out after a refund or dispute.

Why web search matters here

To mimic what a developer would see, each agent got the most native web search its tool offered. Claude Code and Codex used their built-in search. OpenCode used the OpenRouter web search tool, which relies on a provider’s native search when available and falls back to Exa. Stripe notes that it can’t observe the OpenCode searches directly, so those numbers are less certain.

The drop in searching is the clearest sign of what the plugin does. Agents consult Stripe’s own guidance instead of hunting through pages that may describe old APIs.

Differences by category

Agents that did best on one-time payments and subscriptions didn’t lead on Connect. Connect scenarios with more integration-specific decisions also gained more from the plugin than the others.

What developers should take from it

The results suggest that giving agents current, authoritative documentation helps, but doesn’t remove the need for human review of payment logic. Teams using coding agents for billing should check failure handling, refunds and webhooks themselves. Stripe evals will be updated over time, so the numbers are a snapshot of today’s models.

Source: stripe.dev

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.