Visa tested 22 AI models on 15 real shopping scenarios and they failed 42.8 percent of the time. Worse, 54 percent of those failures still ended in a purchase. Here is what service businesses need to understand right now.
Ido Cohen · Published 2026-11-06 · AI for Service Business
Visa just ran the most rigorous public test of AI shopping agents to date — and nearly half of them failed. On November 5, 2026, Visa released its Agentic Commerce Benchmark, a first-of-its-kind study that put 22 AI models from 12 labs through 15 realistic buying scenarios. The headline result: AI agents failed 42.8% of test runs. More alarming, 54% of those failed runs still ended in a purchase. That is not a minor software hiccup — it is a trust and liability problem that is heading straight toward service businesses that handle bookings, payments, and recurring client relationships.
If you run a plumbing company, law firm, med spa, dental practice, or any other service business, your clients are starting to use AI agents to research and book services. That trend is not years away. It is happening now, and Visa just handed you a clear-eyed view of how unreliable those agents still are. Here is what you need to know.
This was not a vague survey or an opinion poll. Visa built a structured technical test with real stakes.
The benchmark evaluated 22 AI models from 12 providers — including household names like Gemini, Claude, Muse, and Grok — across 15 distinct shopping challenges. Each model got three attempts at every scenario. To pass, a model had to do two things correctly: uncover every requirement the buyer had (through a conversation that revealed needs progressively, the way a real customer would) and then make the right buy-or-no-buy decision. According to Visa's official press release, the benchmark is the first agentic commerce benchmark to require both discovery of every task-defined customer requirement and a correct purchase or no-purchase decision for a run to pass. That is a tougher bar than most industry tests, and the results show it.
The shopping categories tested included TVs, laptops, and personal care appliances — categories where getting the wrong product is obvious. Services like legal consultations, dental appointments, HVAC tune-ups, or financial planning sessions are significantly more complex, with more variables, pricing tiers, and contextual judgment calls involved.
Here is the scorecard, pulled directly from Visa's data:
The consistency number is the one that should stop you cold. Across all models and all scenarios, all three attempts passed just 34.2% of the time, according to Visa's official release. Meaning that even when an agent gets something right once, it cannot reliably repeat it. If your clients start using AI agents to book your services, you are dealing with systems that behave differently on the same query depending on when they run it.
The failed-and-still-purchased stat is the other killer. According to Agile Brand Guide's coverage of the benchmark launch, 54% of the failed runs still ended in a purchase. An agent that misunderstood the client's requirements — wrong service tier, wrong appointment type, wrong provider — still went ahead and booked or bought something. That is not a friction problem. That is a liability problem.
The conventional framing of AI shopping agents focuses on physical retail: laptops, sneakers, furniture. But service businesses are directly in the crosshairs.
Agentic commerce, defined simply, is what happens when an AI agent acts on a person's behalf to find, evaluate, and purchase something — without requiring the person to visit a website, fill out a form, or pick up the phone. The technology to do this for services already exists. OpenAI's Operator-class agents can book appointments. Google's AI Mode can surface and initiate contact with local service providers. Meta's AI agents are being wired into commerce flows. The question is not whether AI agents will book plumbers, dentists, and financial advisors. The question is when — and what happens when they get it wrong.
According to Visa's earlier Trust Index research, released in September 2026, 72% of U.S. consumers had already used an AI assistant for shopping. But only 23% trusted AI to handle payment transactions on their behalf. That 23% figure — with 72% already using the tools — represents a massive gap between behavior and trust. Consumers are using AI agents to browse, research, and shortlist. They are still hesitant to let them close the deal. The Visa Agentic Commerce Benchmark is essentially a technical measurement of why that hesitation is rational.
Let me translate Visa's abstract test scenarios into what a failure looks like inside your business.
Scenario 1: The wrong appointment type. A client's AI agent is tasked with booking a dental cleaning. The agent books a full exam with X-rays instead — wrong duration, wrong price, wrong prep instructions. The client shows up expecting one thing and gets another. Your front desk absorbs the complaint. The AI bought "wrong" but still completed a transaction.
Scenario 2: The misread requirement. A homeowner's AI agent is told to find an HVAC service call in a specific zip code with same-day availability. The agent finds one two zip codes over and books it anyway, missing the hidden requirement. You drive 40 minutes out of route. Time and fuel wasted.
Scenario 3: The authority-without-budget problem. A financial advisory prospect's AI agent schedules a consultation, triggering your CRM sequence, your calendar hold, and a staff prep hour. The human later says they never authorized that tier of service. The agent exceeded its instructions.
The Visa benchmark found that the five AI models that uncovered the most hidden buying factors through conversation averaged a 70.2% pass rate. That means the best agents — the ones that ask clarifying questions and probe for real requirements — are substantially more reliable. But even those pass fewer than 3 in 4 runs. And the worst models are passing fewer than 1 in 3.
Here is the counterintuitive read on this data: Visa's benchmark is not a reason to dismiss AI agents. It is a reason to position your business to win when AI agents get better.
Visa's Trust Index found that while only 23% of consumers trust unbranded AI with payments, that number jumps to 61% when a recognized brand they trust is standing behind the transaction — and to 68% among consumers aged 18–34 and 71% among frequent AI users. The pattern is consistent: trust in the underlying AI is low, but trust in known brands and providers mediates that gap.
For a service business, the parallel is direct: your reputation and your verified presence are trust signals that AI agents will read and prioritize. When an AI agent is trying to book a dentist for a client, it will reach for the providers with the most credible digital footprints — complete Google Business Profiles, strong verified reviews, clear service descriptions with structured data, and accurate pricing. The businesses that have built that infrastructure will be the ones AI agents recommend and attempt to book. The businesses that haven't will get passed over before the agent even surfaces their name.
Here is what that looks like in practice:
The Visa Agentic Commerce Benchmark is the clearest signal yet that agentic commerce is real, it is arriving in service categories, and it is currently unreliable enough to cause real problems. Here are five concrete actions to take in the next five business days:
1. Audit your Google Business Profile this week. Make sure every service you offer is listed with a clear description, accurate hours, and current pricing where possible. This is the first file an AI agent will read.
2. Add a structured services page to your website. Create one page per core service type with specific details: what the service includes, typical duration, price range, service area, and how to book. Write it for a literal reader, not for impressiveness.
3. Check your online booking system for clarity. If you use a booking tool, make sure the appointment types are named in plain English. "60-min new patient exam" beats "Appointment Type A." An agent trying to book the right thing needs to match language.
4. Review your intake or booking confirmation process. Given that 54% of failed AI-agent shopping runs still end in a purchase, start thinking about what a wrongly-booked appointment looks like in your workflow. Build a simple confirmation step that catches booking anomalies — a quick confirmation email or text that asks "Is this what you meant to book?" adds friction that protects both sides.
5. Ask your web developer about Schema.org markup. Specifically LocalBusiness schema, Service schema, and if applicable, MedicalBusiness or LegalService schema. This structured data is becoming the vocabulary AI agents use to understand what you do. If you don't have it, you're speaking a language agents can't parse.
The benchmark data is a snapshot of November 2026. These models will get better quickly — the gap between the worst (28.9% pass rate) and best (82.2%) already shows how much variation exists across providers. As agentic commerce matures, the businesses with the clearest, most structured, most trusted digital presence will capture the bookings. The ones caught flat-footed will watch their competitors get booked by agents they've never heard of.
---
What is the Visa Agentic Commerce Benchmark?
Visa's Agentic Commerce Benchmark, launched November 5, 2026, is an industry test that evaluated 22 AI models from 12 labs across 15 realistic shopping challenges. Each model got three attempts per scenario. To pass, the agent had to uncover every buyer requirement through conversation and then make a correct buy-or-no-buy decision. It is the first benchmark of its kind to require both steps for a passing score.
What does a 42.8% failure rate actually mean for my business?
It means that if an AI agent is trying to book or buy something from your business, there is roughly a 43-in-100 chance it will fail to correctly match your services to the client's requirements. More importantly, 54% of those failures still result in a completed transaction — meaning the wrong service gets booked. That creates refund requests, wasted staff time, and unhappy clients who blame the experience on you, not the AI.
Are AI shopping agents already booking service businesses?
The scale is still small — Uber reported only four agent-booked rides in a week of 320 million trips as of early November 2026, according to FourWeekMBA's coverage. But the trajectory is clear: Airbnb has committed to rolling out its own travel agent in the first half of 2027, and AI platforms like Google's AI Mode and OpenAI's agents are already capable of initiating contact with local service providers. The right time to prepare your digital presence is before the volume arrives, not after.
Why do more expensive AI models perform better in the benchmark?
Visa found that the five models with the highest average cost per run passed 67.1% of test runs, compared to 50.2% for the five least expensive models. More capable (and costly) models tend to be better at multi-turn conversation — asking follow-up questions, probing for hidden requirements — which is exactly what the benchmark rewards. As model costs continue to drop over time, that capability gap will narrow, but for now, the cheapest agents are the least reliable.
What is "agentic commerce" and how is it different from AI-assisted shopping?
AI-assisted shopping means a human uses an AI chatbot to research options and then makes the purchase themselves. Agentic commerce means the AI agent researches, decides, and completes the purchase autonomously on the human's behalf. The human may set parameters or spending limits, but the agent acts without step-by-step approval. Visa's benchmark specifically tests this autonomous decision-making — including the critical ability to decide not to buy when the right product or service isn't available.
---
Sources: