I needed a refund. It should have been simple.
I went online and started with the company's automated chat. Fine by me. If the bot could handle it, I could get my refund and get on with my day. The first answer didn't solve the problem, so I explained again. Same result. I tried again, more carefully this time, and got another answer that wasn't going to help. So I asked for a person. When that didn't work, I typed "Human." Then I tried it again.
This is so ordinary now that we've stopped noticing how strange it is. It happens in chat windows, but it's even more maddening on the phone, where you start saying "representative" or "agent," hoping you've found the word that will finally get you to a person. We've quietly accepted that reaching a human is something we sometimes have to earn.
I didn't actually reach a person until the following day. The person I finally reached refunded me, but by then I wasn't thinking much about the refund. I was wondering whether I'd ever use that company again. The company stole my time, and that's what burned me. The automated system might have saved the company a few minutes of employee time, but it left me with a problem that spanned over two days.
I don't want this to sound like an argument against artificial intelligence in customer service, because it isn't. There are plenty of times when I would rather deal with a machine. If I want to check an order or confirm an appointment, give me the answer and let me get on with my day. I don't need another human interaction just for the sake of having one. That's where automation is useful. It takes something routine and gets it done quickly.
There's also nothing wrong with a company tracking how many conversations AI handles without an employee. I'd want to know that, too, because every question AI can handle successfully saves employee time and might give the customer a faster answer. The trouble comes when reducing human involvement becomes the goal, because then you have to be careful about what you're actually rewarding.
A customer who never reaches an employee isn't necessarily a customer whose problem was solved. She might have given up, tried again the next day, or eventually found another way around the system. Maybe she finally got her answer and decided dealing with the company again wasn't worth it. From inside the business, some of those experiences might not look all that different, and that's where the measurement gets tricky.
Imagine that the company with which I dealt had set a goal of reducing the percentage of customer conversations that reached an employee. Now think about what happened when I kept trying to get through. Every additional barrier between me and that employee could help the company reach its goal. I was trying to get out of the automated system while, from the company's perspective, keeping me there might have been exactly what the technology was supposed to accomplish.
If preventing escalation counts as success, you've created an incentive to prevent escalation. That's fine when the AI can actually solve the customer's problem. Mine couldn't, which makes me wonder whether getting handed to a person should count against the AI at all.
Of course I wanted the system to solve my problem. But once it became clear that it couldn't, the most useful thing it could have done was recognize that and get me to someone who could. Customer service has always worked that way. Sometimes the first person you reach can't resolve an issue, so the conversation moves to someone else. We don't automatically consider that first interaction a failure, so why should we think differently about AI?
If AI handles 100 routine questions without involving an employee, wonderful. But on Conversation 101, success might mean recognizing that this one needs a person. If I owned the company, I'd want to know whether the system was good at making that distinction, not simply how often it managed to keep customers away from employees.
When I finally reached a person, I still needed the same refund. I wasn't the same customer, though. The woman who picked up my chat was stepping into everything that had happened since I'd first asked for help, all of it stacked up and waiting for her, and none of it her doing. Before she could deal with the refund, she had to deal with the frustration the system created.
That's easy to miss when we're thinking about AI primarily in terms of the employee time it saves. Automation can remove a lot of simple interactions from someone's day, and that's valuable. But I also wonder what it's doing to the conversations that remain. The employee might not get the customer who started the conversation. She might get that customer after 30 minutes arguing with a chatbot, several attempts to explain the problem, and a growing suspicion that the company would rather wear her down than let her talk to somebody.
A good employee might handle all of that so smoothly that nobody notices. That's pretty much what happened with me. Once I reached a person, she took care of the refund, and if you looked only at that part of the interaction, everything worked. But she was dealing with a problem the company itself had made harder before I ever reached her.
So when we talk about AI reducing employee workload, I'd want to look at more than the number of conversations employees no longer have to handle. I'd want to know whether we're actually reducing their work or changing the kind of work they're getting, because those aren't necessarily the same thing.
If this were my company, I'd become my own customer and go through the same process customers do without using whatever shortcut the people inside the company know exists. Give the system an easy question and see what happens. Then give it something messy, something that requires an exception or a judgment call.
And then I'd try to get out.
If I ask for a person, what happens? If the AI keeps giving me an answer I've already rejected, how long does that continue? When I finally reach an employee, does everything I've already explained travel with me, or am I starting over? These are fairly simple things to test, but you have to experience the system the way your customer does to see them.
I'd also pay attention to what happens after those difficult interactions. The customer might eventually get an answer, but does she come back? Does she contact the company again about the same issue? When she finally reaches an employee, how much of that person's time is spent solving the original problem and how much is spent dealing with frustration that wasn't there when the customer first showed up? Those are harder things to see on a dashboard.
That's why I'd be careful about looking at a rising containment rate or falling employee handling time and assuming the customer experience must be improving along with it. Those numbers might be telling you exactly what you need to know about efficiency, but they might not tell you what the customer had to do to produce them.
The company eventually gave me my refund, so somewhere in its system my problem might very well show up as resolved. If I owned that company, though, I'd want to know something the resolution number couldn't tell me: How hard did we make the customer work to get that answer, and did we make it hard enough that she won't come back?
Anne Lackey is co-founder of HireSmart Virtual Employees, a full-service HR firm helping others recruit, hire, and train top global talent. She has coached and trained hundreds of U.S. and Canadian business owners in creating more successful and profitable businesses. She can be reached at anne@hiresmartvirtualemployees.com.