Where it fails37%
of paying journeys fail a check at reading the product facts.
29% fail at asking the shopper what they need. Building the cart, checking out and paying: under one percent each, on every protocol.
We sent an AI agent shopping 101,416 times.
It almost never failed at checkout. It failed earlier: reading the product page, and asking what the shopper meant.
- Stand-in stores
- 1,044
- Kinds of shop
- 25
- Commerce protocols
- 5
- Ended right
- 68.8%
An open dataset from General Machines.
CartTrace-100k100 journeys, as they endedEnded in a sale we can confirm66
Lost at the till0.0
Per 100 everyday journeys. Every loss happens before the payment step.
Watch one
Every replay is a real journey from the dataset.
The shopper is a program. It holds some of its needs back and gives them only when asked. The agent has the store's tools and one instruction: buy the right thing.
Category
Replaying a journey
What we found
It fails before it pays.
Journeys are lost between finding a product and choosing it: reading the facts, and asking for what the shopper didn't say. Once the cart is right, checkout and payment go through on every protocol.
Stores54–82%
success, from the 5th to the 95th percentile store.
The stores where the agent does worst have one thing in common: search that misses the first query. A fitting product came up first in 81% of their journeys, against 85% on the best stores. So the agent stops more than it buys wrong.
worst quarter of storesbest quarter
Asking62% vs 12%
success when the shopper held two needs back, with and without the agent asking.
On the same task, asking adds 11.9 points (likely range 10.2 to 13.7) where a need is held back and almost nothing where none is. Yet the agent asks in 63% of journeys where nothing is hidden. The fix is aim, not more questions.
agent did not askagent asked
Giving up7.9%
of doable journeys end with the agent saying nothing fits.
On requests where a substitute would do, 27.4%. When a task really was impossible, it refused correctly 77.4% of the time, so only 9.4% of its refusals were right.
The store explains little1.9%
of the difference in outcomes comes from which store it was.
The task explains most of it, and plain chance much of the rest: of 24,549 tasks run three times or more, 29.3% both succeeded and failed. One test tells you little about a store.
When we broke it on purpose99.3%
of the time it told the shopper what really happened.
Declined cards, failed bank checks, price changes, requests nothing could fill. It paid when it shouldn't have in 6 of 3,572 journeys, and never paid for an impossible task.
The 25 categories
A shirt needs a size. A drill needs a voltage.
A gift needs a guess.
Each label shows how often the agent ended right in that category. Open one to see where it stalls, or watch a journey from it.
How it ran
Stand-in stores, real protocols, real payment steps.
An agent can't pay on a real store, and a real store's data can't be published. So one agent shopped stand-in stores that sell like real ones, through checkout, with test tokens.
- 01 · Requests28,985
shopping tasks from 30 shopper personas: gifts, budgets, substitutes, exact sizes.
- 02 · Stores1,044
stand-ins in 25 categories, stocked from open product catalogs.
- 03 · Journeys101,416
from 17 to 22 October 2026, over UCP, ACP, AP2, MPP and WebMCP.
- 04 · Checks85
per journey: 71 settled by code from the store's records, 14 by a model.
- Agent
- Qwen3.8-27B
- Compute
- 971 H100 hours
- Paid orders
- 42,102
- Reference agent
- 82.2% ended right
The data, the paper and the code
Get the data.
Every journey, the paper behind every number, and the code to replay any journey and check it yourself.
01 · DownloadThe data
101,416 journeys as Parquet tables on Hugging Face.
load_dataset("generalmachines/CartTrace-100k")
Hugging Face →
02 · ReadThe paper
How the run was built and judged, and where every number comes from.
Bhatnagar & Pal, 2026
arXiv →
03 · RunThe replay kit
Replay any journey in its store and judge it again.
uv run carttrace replay ep-db6398bc69f1
GitHub →
Every journey includes
- Conversation
- Tool calls
- Protocol traffic
- Cart and checkout states
- Store events
- What the store showed
- The setup
- The ending, and why
Data CC BY 4.0 (product data keeps its source catalog's licence) · Replay kit Apache-2.0 · Stores are pseudonyms; prices and stock are synthetic.
If you use CartTrace-100k in research, cite it as
@misc{carttrace100k,
title = {CartTrace-100k: How Shopping Agents Fare Across Stores,
Categories and Commerce Protocols},
author = {Bhatnagar, Pratham and Pal, Deepanshu},
year = {2026},
publisher = {General Machines},
url = {https://generalmachines.ai/datasets/carttrace-100k},
note = {Dataset: https://huggingface.co/datasets/generalmachines/CartTrace-100k}
}