AI customer support agents: how to choose one that doesn't hide problems
Support is where AI looks best on a dashboard and worst in a customer's inbox. The difference is usually a metric the buyer did not interrogate.
Key takeaways
- Deflection is not resolution. A conversation can end without a human and still leave the customer worse off.
- Containment without a CSAT guardrail rewards ending conversations, not solving them.
- Four failure modes cover most incidents: wrong policy, refund overreach, loops, and silent escalation.
- A 30-day pilot with a baseline and thresholds is the cheapest way to test the claims.
Choose an AI support agent by what happens when it is wrong, not by the deflection rate in the deck. Require visible escalation, policy citations, approval above a spending threshold, and CSAT tied to AI-only conversations. Then pilot on one ticket category.
The economics are simple: every conversation the agent finishes without a human is money not spent. That is also the incentive problem — if the dashboard rewards endings, the fastest way to improve it is to end more conversations, including the ones that needed a person. We compare published capabilities, documentation, and operator reports; no vendor pays for placement. The editorial policy explains sourcing.
Deflection, resolution, containment, handoff
Vendors use these terms loosely. Ask each to define them in writing.
- Deflection rate: the share of conversations that end without a human. It measures where the conversation stopped, not whether the problem was solved.
- Resolution rate: the share where the issue was fixed, confirmed by no reopen within a window or by a satisfaction response.
- Containment: the share handled end-to-end without escalation. Some vendors include customers who simply gave up.
- Handoff quality: what the human receives — a summary, transcript, account context, and next step. It separates a warm transfer from a repeat explanation.
An agent that resolves 40% and escalates cleanly beats one that deflects 70% and leaves customers to reopen elsewhere. The second number looks better in a review and costs more later.
Three categories, and who each fits
Help-desk-native AI
Intercom Fin, Zendesk AI, and Freshworks sit inside a help desk and use its history and knowledge base. Best fit if you already run support and want the agent to inherit resolved tickets. Trade-off: only as good as the help center, resolution-based pricing is hard to forecast, and migration means moving the agent too.
Ecommerce-native AI
Tidio with Lyro, Gorgias, and similar connect to order, shipping, and returns data. Best fit if most tickets are order status, address changes, or returns. Trade-off: strong on the top ecommerce intents, thin outside them. Tidio's platform was listed from about $24.17 per month with Lyro from about $32.50 on annual billing at the time of writing.
General agent builders
Zapier, Make, n8n, and platform suites can connect a model to a help desk, database, and internal tools. Best fit if support crosses systems the vendors do not integrate. Trade-off: you build retrieval, escalation, evaluation, and dashboards — flexible and easy to leave unmeasured.
The metrics that matter, and how each misleads
| Metric | What it tells you | How it misleads |
|---|---|---|
| Deflection rate | How often the AI ends a conversation without a human | Counts abandoned and unresolved chats as wins; says nothing about satisfaction |
| Resolution rate | How often the issue was actually fixed | Depends on the confirmation window; a reopen three days later may be excluded |
| First-response time | How fast the customer hears anything | A fast wrong answer is worse than a slow correct one; measures speed, not quality |
| CSAT after AI interaction | Whether customers were satisfied with the AI-only exchange | Response bias; unhappy customers often leave without answering the survey |
| Escalation accuracy | Whether the right conversations reach a human | Misses silent escalation, where a ticket is created but the customer is not told |
| Cost per resolved ticket | The all-in cost of a solved issue, including platform and usage fees | Falls if unresolved tickets are excluded from the denominator |
Every support metric improves by narrowing what counts. Ask for the denominator, the time window, and how reopens are treated.
Four failure modes to test before you buy
1. Confidently wrong policy answers
The agent retrieves an outdated article or blends two policies and states the result confidently. Test questions whose answers changed recently, and require a source citation or an explicit “I don't know.” Visible uncertainty is a feature.
2. Refund overreach
Given a refund tool, a model may interpret policy generously, especially with an insistent customer. Test prompts claiming a manager approved it or citing a competitor. The correct behavior is a hard limit plus human approval, not negotiation.
3. Looped conversations
The agent cannot resolve the issue, does not recognize it is stuck, and repeats the question. Require a loop detector: after two failed attempts, escalate with a summary. Test by asking for something it cannot do.
4. Silent escalation
A ticket appears in the queue but the customer is never told a human will follow up, or when. The dashboard looks contained; the inbox looks abandoned. Require an explicit message, an expected response time, and a handoff record.
A 30-day pilot that produces a decision
The pilot tests the tool on your tickets, policy complexity, and staff.
- Days -14 to 0 — baseline. Pull 200–300 tickets from one category. Tag reasons; measure human resolution, first response, CSAT, and cost per resolved ticket.
- Days 1–7 — shadow mode. The agent drafts, a human sends. Compare against what the team would have written; no customer is exposed.
- Days 8–14 — limited release. Route 20–30% of one category to the agent during staffed hours, with an obvious escalation button.
- Days 15–30 — expand or stop. Expand only if thresholds are met; otherwise document the failure and fix the knowledge base.
Set thresholds before launch. Starting points, not benchmarks: CSAT within five points of baseline; escalation accuracy above 90%; policy errors under 2% of audited chats; zero unapproved refunds; loops under 3%; median handoff under two minutes.
Review 30 transcripts weekly. Categorize failures as missing knowledge, bad retrieval, tool error, policy ambiguity, or escalation failure. After four weeks you can buy, extend, or walk away.
Frequently asked questions
What is a good deflection rate for an AI support agent?
No universal benchmark: it depends on ticket mix, policy complexity, and whether the agent can act. Vendor figures are not comparable. Measure against your own baseline, and count a conversation as deflected only if it was not reopened within a defined window.
Will an AI support agent reduce headcount?
Mostly it absorbs volume rather than eliminating roles: peaks without a hire, or an agent freed for exceptions and retention. A team expecting immediate layoffs usually gets worse service and a tool nobody trusts. The ROI guide models the saving honestly.
How do I stop an AI agent from promising refunds it should not?
Read-only access to policy and order data, plus human approval above a dollar threshold. No bulk refund permissions. Test adversarial prompts that cite a generous policy or claim a manager approved it, and log every policy citation.