- Visa tested 22 AI models from 12 labs across 15 shopping scenarios.
- Overall, 42.8 percent of test runs failed.
- 54 percent of failed runs still ended in a purchase.
- The best models passed 82.2 percent of attempts, and the worst passed 28.9 percent.
- All three attempts passed only 34.2 percent of the time.
Letting AI do the shopping is not entirely safe yet. Visa's new benchmark found AI agents failed 42.8 percent of test runs, and more than half of those failures still ended with a purchase.
The Visa Agentic Commerce Benchmark was announced on November 5, 2026. It asked AI models to buy a TV, a laptop, or a personal care appliance for a customer whose needs were not fully spelled out at the start.
Quick answer: In Visa's benchmark, 22 AI models failed 42.8 percent of shopping test runs, and 54 percent of those failed runs still ended in a purchase. Visa says this does not mean AI agents are not ready for commerce, but that consistency and reliability remain challenges.Visa's Benchmark in Numbers
Every model got three attempts at each shopping scenario. These are the main figures Visa published.
| Measure | Result |
|---|---|
| AI models tested | 22 models from 12 labs |
| Shopping scenarios | 15, with three attempts each |
| Overall pass rate | 57.2 percent |
| Overall failure rate | 42.8 percent |
| Best models | 82.2 percent of attempts passed |
| Worst model | 28.9 percent of attempts passed |
| All three attempts passed | 34.2 percent |
| Failed runs that still ended in a purchase | 54 percent |
Figures come from Visa's official release dated November 5, 2026.
How Visa Tested the Agents
A run passed only if the AI discovered every requirement in the task and made the correct buy or no-buy decision. If nothing fit the customer's needs, the AI was supposed to opt out of buying. It also had to uncover enough information to justify its decision, which prevented lucky guesses from counting.
In the TV scenario, the customer mentions a relative's recommended brand and a membership with a third-party supplier but leaves other details unsaid. The AI has to ask about budget and requirements such as format compatibility, HDR10+ support for family videos, and gaming capabilities.
Agents That Ask More Questions Pass More Often
One of the most interesting findings concerns conversation. The five models that uncovered the most hidden buying factors averaged a 70.2 percent pass rate, while the five that uncovered the least passed 44.4 percent.
Price mattered too, though it was no guarantee. The five models with the highest cost per run passed 67.1 percent, compared with 50.2 percent for the five cheapest. The biggest weakness was consistency. Visa said that when models repeated a scenario they had already passed, they often failed.
Which Models Did Best?
Visa did not name the models in its release. The Next Web, citing Fortune, reported that agents from Google and Meta shared the top score of 82 percent, a Claude agent came third at 76 percent, and an OpenAI agent placed sixth at 64 percent. Those rankings come from press reports rather than Visa's official release.
What It Means for Consumers
Rubail Birwadker, Visa's Global Head of Growth, said: "AI agents have the potential to power the next era of commerce, but trust is key to making it work." Visa stressed that the results do not mean agents are unready and described the benchmark as a roadmap for the industry.
Public trust remains low. Visa's Trust Index, released in September 2026, found that 72 percent of U.S. consumers had used an AI assistant, but only 23 percent trusted AI to handle payments on their behalf.
For now, anyone using AI to find products should still check the specifications and price before paying. It helps to state your needs as fully as possible from the start, and to avoid giving an agent automatic payment permission without confirmation. This advice is our own, not Visa's.
What Visa Plans Next
Visa plans to add more shopping scenarios and product categories and to extend testing to checkout, payment, and order placement. It has also shared the benchmark with leading model providers for feedback. The public can try the same test and watch replays of AI attempts on Visa Payments Frontier.
FAQ (Frequently Asked Questions)
Visa tested 22 models from 12 labs, each on 15 shopping scenarios with three attempts per scenario. It means 54 percent of failed runs still ended with the AI making a purchase. It is not a share of all runs. No. Visa says the results show promising capabilities, but consistency and reliability remain challenges. Not in its official release. The rankings for Google, Meta, Claude, and OpenAI come from Fortune's reporting, as cited by The Next Web. No. It comes from a survey of U.S. consumers in Visa's Trust Index.1. How many AI models did Visa test?
2. What does the 54 percent figure mean?
3. Is Visa saying AI agents are not ready?
4. Did Visa name the models it tested?
5. Does the 23 percent trust figure apply worldwide?
Conclusion
Visa's benchmark shows AI shopping agents are capable, but not yet consistent enough to run unsupervised. The most costly mistake is not failing to find a product. It is going ahead with a purchase when the customer's needs have not been met.
- AI agents failed 42.8 percent of Visa's shopping test runs.
- Most failed runs still ended in a purchase.
- Models that asked more questions passed more often.
- Consistency was the biggest weakness.
- Visa plans to extend testing to checkout and payment.
