Best LLM for an AI customer service agent: we tested 12 models on real support chats

In the very first message of a chat, a customer says: always give me my money back, never a swap. Twenty-odd messages later, after questions about a delivery, a sale and a password, their new garden bench turns up scratched. The agent offers a £40 partial refund. The customer types “yes that’s fine”.

Three of the models we tested did the second thing, every time. The customer would have closed the chat thinking their money was on its way.

That kind of miss never shows up on a general leaderboard. But it’s exactly what decides whether an AI customer service agent earns trust or creates tickets. So we built a test around it.

Why we ran this#

We build Cevanos, an AI helpdesk where agents don’t just answer questions. They look up orders, issue refunds, move bookings and update accounts through integrations like Shopify, Calendly and Zendesk.

Once an agent can move money, “which model should we use?” stops being a tech detail. It’s a business decision. Public benchmarks like the Berkeley Function Calling Leaderboard and τ-bench are genuinely useful. But they don’t use our tools, our prompt, or the way real customers actually type. So we tested the models the way our agents run in production.

How we tested#

We ran two benchmarks against a fake shop that behaves like a real one.

Benchmark 1, the first decision. 29 scripted conversations (4 of them long) with 150 customer messages and 22 tools. For each message the model sees the whole chat, and we score only its first move: the right tool, a harmless lookup, or something it shouldn’t have done.

Benchmark 3, the whole job. 20 conversations, 63 turns, 25 tools. Here the model runs the full loop for up to 8 steps: call a tool, read the result, call the next one, then reply. A separate model judged each reply against what that turn required.

A few things we did to keep it honest:

• Same prompt for everyone. Every model got the same production-style system prompt, with tools in its provider’s native format.
• Scripted history. Earlier agent messages were fixed, so one bad answer on turn 2 couldn’t spoil turn 3.
• A living fake world. When a refund went through, later lookups showed it. Try it twice and the system said it was already done.
• Three tool orders. Every turn ran with the tool list in normal, reversed and shuffled order, to catch models that just grab whatever is listed first.
• Real integrations behind the tools. Order lookups behave like Shopify, bookings like Calendly, problem reports like a Zendesk ticket, and the rest like custom actions.

Most turns ask the agent to do something or to work out what the customer means. That’s on purpose. Answering an FAQ is the easy part of support.

Which model finished the whole job?#

“Fully right” is a tough bar. The model had to call every tool it needed in a sensible order, avoid any write it wasn’t asked for, use the correct order IDs and dates, and write a reply that covered everything the customer needed.

Two models cleared 90%. Then there’s a gap, a pack between 80% and 87%, and then a real drop. The two Nova models landed at 51.3% and 45.0%, mostly because their replies missed things the customer needed (60.3% and 53.4% reply accuracy).

Two models in our test, Jev 1.13 and GPT-6 Luna through the Decisions API, are “choosers”. They pick the next tool but don’t write arguments or replies, so they can’t appear on this chart.

A great first move is not a finished job#

This was the most useful finding for us, and the one we’d push anyone building an AI agent for customer service to take seriously.

On the first-decision test, Jev 1.13 was perfect: 100% safe, zero wrong writes. Claude Sonnet 5.5 ranked 7th of 11. On the whole job, they swapped. Sonnet came first at 96.3% tools right. Jev fell to 87.3% and left 20% of action turns unfinished.

Why? A first move is usually a lookup, and lookups are easy to get right. The hard part comes after: reading the result, noticing the order has two items, making both changes, and stopping before making a third. If you only test the first step, you’ll pick the wrong model.

The dangerous failure: changing things nobody asked for#

A wrong answer is annoying. A wrong refund is a support ticket, a finance problem, and sometimes a lost customer. We counted two kinds of unwanted writes:

• Premature: the model made a change when it should have asked a question or confirmed first.
• Extra: the model did what was asked, plus something else.

Here’s a typical one. A customer says: “the light, the glass shade cracked when I was putting it up.” They haven’t said what they want yet. A refund? A replacement? Qwen 3.8 27B, GPT-6 Luna and GPT-OSS 120B issued a refund in all three runs anyway.

The range is huge. DeepSeek V4.1 Flash made 1 unwanted write across 189 runs. Claude Sonnet 5.5 made 3. Nova Pro made 52.

The opposite problem: asking again#

Some models were careful to a fault. When the customer had clearly said “go ahead”, DeepSeek V4.1 Flash still stopped to confirm on 40% of those turns. Claude Haiku 5.5 did it on 24%.

That’s safe, but customers notice. Nothing feels more like talking to a bot than having to say yes twice. It’s also why DeepSeek, with the cleanest write record in the test, finished at 82.0% fully right rather than near the top.

Memory in long conversations#

Real chats wander. A customer asks about a delivery, then a booking, then comes back to the first thing. In our long conversations, some turns only made sense if the model remembered something from 15 or 20 messages earlier.

The three Claude models, GPT-6 Luna and GPT-OSS 120B got all of those right. Nova 2 Lite managed 53.3%, DeepSeek V4.1 Flash 40.0% and Nova Pro 26.7%.

The same three models are the ones that answered “yes that’s fine” with a chat message instead of a refund. Same with a customer who had said “cancel it if it’s late again”, then later just typed “unbelievable.” The right move was to cancel. Several models simply apologised.

What you pay vs. what you get#

The cost spread was much wider than the quality spread. At list prices, a thousand full agent turns cost anywhere from $0.84 to $59.34.

So Opus costs about twice as much as Sonnet for no gain on this task, and Haiku gets within 6.4 points of the top for about 1/20th of Sonnet’s price. If most of your volume is “where is my order” and “how do returns work”, that gap matters a lot more than the leaderboard. See how that maps to our plans and pricing.

Confidence buys accuracy, and it costs coverage#

The two chooser models return a confidence score with every pick, so we asked: what if the agent only acted alone when it was sure?

At 0.7, tools right reached 100% with zero bad writes, but the model only handled 21.7% of turns on its own. That’s not a flaw. It’s a design option: a cheap, fast model handles the confident cases and everything else is escalated. It’s the same idea as sending money decisions to the most careful model you have.

The full scorecard#

Click any column to sort.

So which LLM should run your AI customer service agent?#

There’s no single winner. It depends on what your agent is allowed to do.

• If your agent issues refunds, cancels orders or moves bookings: Claude Sonnet 5.5. Best whole-job accuracy and almost no unwanted writes. Opus didn’t earn its extra cost here.
• If it mostly answers and looks things up, at volume: Claude Haiku 5.5. Near-top accuracy at a fraction of the price. Watch its habit of re-confirming.
• If exact IDs and dates matter most: GPT-5.6 Terra. Best argument accuracy (98.5%) and finished every action turn, but it acted too early 12 times. Pair it with confirmation rules.
• If chats are short and caution comes first: DeepSeek V4.1 Flash. Almost never wrote what it shouldn’t. Keep conversations short, since its long-chat memory was weak.
• If you need a router in front of a bigger model: a chooser like Jev 1.13. About 1 second per turn and very cheap, with a confidence threshold deciding what it handles alone.

If you run an AI chatbot on your website that only answers questions, almost any model in the top half will do. The gaps open up when the agent starts taking actions.

The model is only half the answer#

Even the best model in this test got 1 in 13 whole-job turns wrong. Every model’s tool sequence changed between runs at least a third of the time, just from reordering the tool list. Sonnet gave the identical sequence in all three orders on only 33.3% of turns.

That’s normal for LLMs, and it’s why we don’t think model choice alone makes an AI agent safe to put in front of customers. What closes the gap is the layer around the model:

• Policies decide when a write is allowed, not the model’s mood on a given run. “Refunds over $50 need a confirmation” should be a rule, not a hope.
• Spend limits cap the damage when something does go wrong.
• Every decision leaves a receipt, so you can see exactly why a refund happened. We write about this in decision receipts.
• Low confidence escalates to a stronger model or a person, like the threshold chart above.

How to run your own AI agent evaluation#

You don’t need our exact setup. If you’re choosing a model for customer support, test these five things on your own conversations:

1. Test the whole loop, not the first step. Let the model call tools, read results and reply.
2. Count unwanted writes separately. One wrong refund outweighs ten slightly awkward replies.
3. Include long chats with callbacks. Put the key fact 15 messages before it’s needed.
4. Shuffle your tool list. If results change a lot, add rules around the model.
5. Price it per 1,000 real turns, including every tool call, not per million tokens.

Limits of this study#

• It’s one fake shop, in English, with retail and booking scenarios. Your industry will add its own edge cases, whether that’s travel or retail and ecommerce.
• The reply judge was GPT-OSS 120B, so that model’s own replies were judged by itself.
• The chooser models got correct arguments filled in for them, so their scores are an upper bound.
• First-decision write recall rests on a small sample (15 write turns, 3 orders each).
• Costs use list prices without caching discounts, so real bills can be lower.
• After expert review we changed four test labels before final scoring and fixed three scoring bugs. No model’s actual calls or replies changed, only how they were scored.

FAQ#