Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail

What happened

Visa launched its Visa Agentic Commerce Benchmark, testing AI models on realistic shopping tasks that require them to clarify customer needs and avoid purchases when requirements are unmet. In the first round, 22 models from 12 labs completed 15 challenges, and the overall pass rate was 57.2%.

Confirmed

Global impact / market context

The results suggest that AI shopping tools may need to ask better questions before recommending or making purchases. If they misunderstand requirements, retailers could lose sales or face unsuitable orders; stronger performance could help companies convert more shopping conversations into completed purchases.

Analyst inference

Visa’s test shows that AI shopping performance varied widely: the top models passed 82.2% of attempts, while the lowest passed 28.9%. Models that uncovered more hidden buying needs also had higher average pass rates, linking useful conversation to better task completion.

Confirmed

What to watch

1. Further benchmark results could show whether AI models improve at identifying customer needs, a capability that may influence how reliably shopping tools recommend products or complete purchases. Analyst inference
2. The article reports that the five models with the highest average cost per run passed 67.1% of runs, versus 50.2% for the lowest-cost group. Confirmed
3. Retailers and payment businesses could assess whether better-performing AI shopping tools increase completed orders enough to justify the cost of using more expensive models. Analyst inference

Evidence

• Original source link unavailable