** Catch a replay of VOX here: https://poly.ai/vox-2024/ ** Damien and PolyAI CEO Nikola Mrkšić as they rewind to Thursday, November 14th, covering the evolution and challenges of automatic speech recognition in voice AI and automation for contact centers, emerging trends in CX and customer satisfaction, and tips on boosting ops without sacrificing quality. Hear insightful recaps from industry leaders, such as Lauren Sullivan from Howard Brown Health and Jo Lai from Hopper, and get a sneak peek into the future of AI-driven customer service. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Nikolathe natural progression of a tier one agent is to be a tier two agent, a manager, supervisor, point of escalation, someone who over time has done, creating a system, updating the training materials, doing everything to, set up the new generation of agents. Right. So instead they're becoming AI developers,
DamienHello and welcome to another episode of the deep learning with poly AI podcast. I'm your host, Damien. And I am here as per usual with CEO and co founder of poly AI, Nikola Mrkšić. Hey, Nikola.
NikolaHow are you?
DamienI am very ready for the weekend, especially after yesterday, which was a high activity, high emotion action packed day where we just wrapped our 2024 box horizons annual. Conference. It was virtual this year. We stuck with what worked. It was so big. We had two tracks. I thought today it would be great to chat with one, the keynote provider and to the CEO of the company who put on the dang thing to discuss a little bit about some of the concepts that our customer and user participants brought up during their sessions, and maybe just drill down a little bit deeper to get your take as to some of the trends that were brought up, some of the factoids that were mentioned.
DamienAnd if this is typical, if this is novel, or is this something that's on the cutting edge of what we can expect for voice AI and automation in the contact center, what do you think?
NikolaAbsolutely. No, I think we heard a lot of good stuff and we're lucky to have so many people there. So happy to have you to go over it. Yeah, it was a blast.
DamienLet's start at the start, right? Which was your keynote. Really insightful stuff talking about the state of the union with regards to the industry of voice automation and our voice AI and automation in the contact center, one of the things that you really spoke on was how Automatic speech recognition has really not. Progressed, or at least this is the way I interpret is really not progressed as much as we thought it would as being at such a critical and crucial component of the S. L. U. Tech stack. So I'm really curious, and you mentioned something about, 15 years ago, there were two organizations that had two papers on this, right?
DamienIt was like Google and Microsoft. And they were writing about it and said, Okay, we got this mastered and nothing has really come. Well, are still missing out 15 and 50, right? So. Why is ESR so difficult to master?
NikolaYeah, look, I think that it's just about our voices are all different. They have different properties. We all have different accents, styles, and we're capturing it with different devices, right? So there is no single speech recognition. There are many ways to do it, right? There are constraints around speed and latency. There are constraints around the channel, right? So what we do is a lot of speech recognition over the phone. Right. The phone is a top panel, different countries, you different encodings. Right. And like these encodings can make all the difference between the error rate you're going to get, right.
NikolaThen people talk on the phone, they use the speaker phone they use the car surround systems to, to talk. Right. So like you get many different flavors of that input right now, what was interesting, and I don't know if I went into proper detail yesterday, but. In 2014 Google and Microsoft both had the paper saying we have human level speech recognition on this relatively complex data set. Right. And I think as we've worked on the problem and, when I started working on it, people told me, Hey, speech recognition is improving really fast, it'll be, there'll be a sole problem.
NikolaGo focus on the reasoning bit or on the quality of voice produced back. And these are all components that really matter. Right. I think the fact that each of our customers has like a. Super branded voice that sounds exactly the way they want to sound in every interaction or, different voices that sound the way that they want to sound like that helped us with many customers and become what we are today. But the 1 problem, whenever you look at something not working, whenever you look at containment figures, whenever you see a spike in frustration, and you look under the hood.
NikolaMore often than not, about three quarters of the time, someone's getting stuck because they have a difficult accent, or they're saying something the speech recognition is not capturing, right? So to say that it's not progressed dramatically every two years Would be wrong. It has as deep learning came, as transformers came, new architectures, better and better, it keeps getting better. It will be better in two years again. Right. And maybe the
Damiencomponents of that that are getting better, like the contextualization gets better and those sort of applications, or is it as a whole, like it's entirely getting more and more human capable,
Nikolawell, human capable is the wrong way to look at it. There's just is it transcribing everything you say correctly, right? The truth is if you. Listen to snippets of me talking and someone like cut out like different sentences. You might not get every word correctly. You might admit something that I think I've said, maybe I've muttered it out. And it's not really clear what I said. Maybe you, take a punt and guess one thing and it would be another. Right. So like your error rate would not be like zero, right? Similarly, neither it is for machines, right? Not what you have that.
NikolaThese systems don't have yet is the ability to use context to say, Hey, like he's talking about numbers or, he said this in this way, a few times, he probably means this, even though, he's a non native speaker and, you just learned that, I say some things incorrectly, or sometimes there isn't really a correct way. There are many dialects and different people say different words in different ways, right? Is it a schedule is the schedule, right? And with all that done, you basically are at a point that you're waiting for it to be perfect, but it's not. Now the real problem with customer services, many of the inputs you need to capture are things like names, things like,
Damienthey're pretty binary. You have to have them accurate, especially in the alpha numeric stuff.
NikolaYeah. Someone just blurts out like 1314 Sierra Valley road. Right. And did you get that right? Well, what's the post creation? Five barbecue. Hmm. Right. Cool. Right. Like it was at a P or B. Right. And that's why we, we often talk about dialogue, right? Because while, if I happen to say that postcode in a way that's not very comprehensible for that system. Right. And you go sorry, I didn't get that. Can you say it again? What am I going to do? HTML5 hypercube. Right. Maybe I slow down a little. Maybe I get lucky the second time around. But maybe something about my voice footprint with that model doesn't work.
NikolaAnd then, you have to go was it Papa or Bravo? Right now it's then I know that Papa or Bravo is likely following and the classification problem of, have you said Papa or Bravo now that's really easy compared to transcribing alphanumeric characters from any member of the population coming into the system, right? So you basically have to build a robust way to handle those errors, right? And like computer networking has this famous seven layer stack from like the physical layer, all the way to the application layer, you and I are this thing.
NikolaThere are probably milliseconds where the signal is omitted or like something doesn't go right, whether my wifi, your wifi, something in the middle, fiber optic cable, there are many components we can't even enumerate all the components that this technology is touching in this conversation and you and I are having, but even though it's cables are missing bits like there's fluctuations in latency that mean that like some packages arrived before and after others. It's being assembled, it works, right? Yeah, we're still hearing
Damieneach other in real time and having that conversation, yeah.
NikolaYeah, so similarly, we're seeing progress, we're
Damienseeing progress in ASR, and like you said, every two years it's getting better and better, but it's still just a very complex component of a larger, multi layered tech stack that, one that probably gets blamed when things go wrong, Because it's easy to blame when you don't have proper transcript.
NikolaIt is. It is the culprit. Right? Yeah, we showed a pie chart, which basically said, 3 quarters of. Failure modes have to do with speech recognition and, there's a 12 percent to do with dialogue management. So whether you're doing that with an old school dialogue manager, whether it's programmatic or machine learning driven, or if it's just an LLM that you let run amok yeah, it was perfect in our sampling on that data set. Then it's just a data set, right? That's pretty damn big. 12%, right? So that's the lift, right?
NikolaIf you had God mode LLM with no latency, you'd get that uplift, right? And that's important to know, right? Cause even a lot of people think that the admin of LLMs just solved everything. And. Actually, some of the problems that they introduced that 12 percent can be really, really bad. So not all problems are made equal. So yeah, like it's a hard problem. Right. I think that I've been working on this for well over 10 years and, it's still, it's huge progress was made. It is better. The real issue is that we're dealing with like sprints of speech recognition, right?
NikolaThe language models, there are two components of a speech recognizer, the audio model and the language model, at least in old school ones, newer ones are end to end, but a longer sentence allows the language model piece to use context inside that input to say, Oh, he's saying a number or, Oh, that has to be like the name that has to be the word road at the end, because it was like 13, 14, something, something. It's going to be road or drive or street. Right. But especially in what we do, if someone goes yeah, right.
NikolaThat, yeah, it could be whatever. Right. And we've had examples of, Mandarin Cantonese, where like a slight, like added tonal inflection to start can, there's a
Damienlot of tonality affect that will be for two,
Nikolaright. The problems are. For most other languages, a lot more difficult, although English is fantastic for its variance. It's a huge language. We made native dialects plus all of us foreigners speaking English in our own ways. So, English is hard.
DamienEnglish is hard. Allegedly the second most difficult language in the world to learn.
NikolaMaybe my vocabulary, right?
DamienMaybe. All right. So, the opening keynote was great. Take a look at that. If you haven't already. One of the first individuals that we spoke with was Lauren Sullivan from Howard Brown Health and really highlighting some interesting facets and unique nuances to the healthcare industry. But one of the things that brought up was where we envision things like seasonal spikes in contact centers, especially for support and assistance. We usually think retail, approaching the holidays, et cetera, et cetera. But Lauren actually brought up a really interesting piece, which is, Global pandemics and massive health crises and how that can drive a surge of inbound calls into healthcare organizations where it's more than just, I need to return this robe.
DamienIt doesn't fit it's what do I need to do to properly inoculate myself against this global pandemic? I thought it was really interesting that you brought this up. I'm curious from your perspective, what are some other often overlooked contributors to spikes and call center activity across industries that maybe we don't think about? When we think about things like, oh, it's tax season. So of course, CPAs or tax advisors are going to get those spikes.
NikolaYeah, for sure. Look, I think that for many contact centers out there. Lunchtime is a spike because people are grabbing lunch and, calling the utility company, telco, something, so that's a big one. In some cases, you'll see spikes from just people commuting to work and back, especially in the U S where commutes are like effective daily life for many. That's like the mundane. Then of course, like for many businesses, they will have big fluctuations between working days and the weekend. With a lot of financial services, you'll have spikes maybe around the times of month when people get paid and something's happening, they're paying off something.
NikolaMany people still call to pay their mortgage monthly, especially in the US. So there are spikes on specific payments. So there's every
Damienother Fridays or whatever when the paycheck hits.
NikolaYeah. Yeah. Yeah. Well, usually I think like once a month for those payments, people cool to make, right. Then you'll have, then you have events that are like not seasonal, but important, like interest rate changes for some of our clients. Drive behavior quite a lot of them people call have questions about the changes to their payments, their effect, et cetera. Yeah. And then, holidays are always, I think every December across pretty much every vertical sees a huge, huge spike of calls, right.
NikolaAnd then August is interesting, right? Hospitality, some places you'll see a spike and other places you'll see a complete wipe out. So it can be, really interesting.
DamienThe school year effect. Cool. Yeah. A very fascinating talk with Lawrence to check that out again. If you haven't seen it Dan Eddie also from simply health sitting on a voice assistant, that's a year and a half old 18 months. So we did it would catch us up to speed as to how the last 18 months have been. And he said outright and I'm quoting our customer satisfaction scores have never been higher. Our trust pilot scores have never been higher. And then later, almost in the same breath, he said, and we were able to slash 4 million pounds out of our operational costs.
DamienSo this goes back to an episode we recorded a number of months back, which was operational efficiency or low costs, why not both? And it seems that that's working. Is this the new expectation for this kind of automation, the contact center to where you're seeing not only reduced costs, but also an increase in customer satisfaction?
NikolaAbsolutely. Right. I think that unless you're running an operation where your age and nutrition is super low, everyone's super ramped, motivated and happy and you've provided the training, you have a flow of information for all the changes being made and, you have enough people to handle all the spikes and stuff. There's probably a way for you to benefit. Now, if you have that kind of operation, frankly, you're probably not in the market buying software. To do anything. Cause you are one of the very, very privileged few, right?
NikolaThey exist, they exist. Right. Now, if that's not the case, if you're missing some calls, if you have weird seasonality, if you have disputes with your BPOs around handling times and why they're taking too long, then you can probably improve customer experience, right? In one way or another, either by providing more accurate responses, by making sure that no one waits for service. Right. And just by, like having a better experience, right. The more fine grained thing that is an agent who's been on the job for two weeks versus like a voice assistant trained on 18 months.
NikolaAt some point, it's not impossible that voice assistants will be better at most things, right. At least in most things that happen very, very frequently. And with that, as they get calcified and become the yeah, your voice assistant. Is another website, right? And you're building it gradually the same way that you built the website to be like a major way that you do business with the world, with your consumers, right? At first, maybe you just had some information on it. Then you started adding, you create an account and you allow people to log in and do things on there.
NikolaRight. And then you talk very carefully about, I think I want them to be able to, I don't know, transfer money between their different accounts, but I'm not sure I'm going to allow them to transfer. 100, 000 from one of their accounts to somewhere else. I might have to, I might ask them to come into the branch and bring a form of ID. Right. And the same way with a voice assistant, right? You go and you build, right? And the more you build out these flows and things and integrations. The better it gets, right? And the more you build and the longer you tune it to be good at what you need it to be good at, well, the better, and then the other overlooked thing is that consumer behavior takes a bit of time to settle, right?
NikolaSo even if you introduce the world's best voice assistant overnight, if people are used to your cranky old DTMF. And they know that the right way to get to wherever they need to get to is to hit 77332, right? They probably they might have it saved with the number and then I've done that
Damieni've definitely done that program for many people
NikolaYeah, yeah, absolutely. Like a lot of people who Use certain service lines, right? Who call like their dispatch centers and stuff like that? They know exactly what they're doing You Even if you introduce a better experience, it's a different experience. And we all love the status quo, so it takes a bit of time. And there, you got to think like long term, is it going to be better? And if it's not better, well, then don't fix something that's not broken.
DamienRight. It is. He definitely said that things were better. And one of the unexpected outcomes that he had after 18 months is actually taking a second look at the organizational design. Obviously, when you bring in some operational efficiencies, there are some, maybe some lower level positions that can be made redundant. But what he noticed was that he was moving his highly skilled agents to more of a supervisory role, responsible and tasked with overseeing and managing the ongoing fine tuning and training of the voice assistant. to also a point where they were responsible for writing scripts for human agents to be able to validate when, someone was a little, like you said, the cultural difference of being able to talk to talk to an automated assistant.
DamienSometimes people just aren't ready for it with that validated why it was so essential. I thought that was cool to see that. That he was, it was affecting his operation or his organizational design. My question to you is, do we see more of this on the front end of doing our deployments where people are concerned about how this is going to affect the organizational design?
NikolaListen, people tend to move and implement when they have a problem. Right. Yeah. Abandons. They have missed revenue. They have service level issues that are, reaching the CEO and like the board. Right. And at that point they're looking for a fix. Right. So I wouldn't say that most of our customers initially worry about your design. I think what I've always found really interesting is just how excited people in the contact center are to work with this, because a lot of the other CCAS technology.
NikolaIt's a blunt instrument, right? And some is more sophisticated than others, but There's an agent screen, there's a workforce management tool, you set up like the, the skill groups, it's boring, right? And then there's some QA that's going to be a bit more interesting, especially if there's something machine learning powered. But when you think about this, right, the natural progression of a tier one agent is to be a tier two agent, a manager, supervisor, point of escalation, someone who over time has done, creating a system, updating the training materials, doing everything to, set up the new generation of agents.
NikolaRight. So instead they're becoming AI developers, right? And that's kind of like, the whole fear of, are we going to have something to do, in a few years? Well, always with technology, we display some jobs and we create new ones. Right. And I think that this is actually quite a change for the better. For it's, it's mind, it's,
Damienit's mind blowing to think of how these folks who maybe experienced contact center agents now are being told You will have proficiencies in AI, dialogue design. Systems management. It's just it's like the concept of that being the new career path is wild.
NikolaYeah, it's really cool. Right. And by the way people who have been in the contact center for a while are naturally really good at this stuff. Right? Yeah, they are there. Training one by one, sometimes maybe in a bit of a like onboarding session with more humans. That's what they do. So they know what sounds right.
DamienThey know what's on brand. They know what works. They know how to deal with real customers, especially in their business or industry. So it's essential. It's really essential to get the maximum out of something. Absolutely. Yeah, exactly. Exactly. Very cool. Another person who spoke at Fox was July over at Hopper, the travel platform. And Joe said, and I quote, a brilliant AI solution does not a knowledge base make, which is true as A number of organizations are basically having holes punched in their knowledge management programs when entertaining CS automation.
DamienWhat do you see that companies are doing to better organize their internal data stores in preparation for deploying something like a voice assistant in their context? Whatever
Nikolathey can, there's no playbook. Now, here's what's interesting, right? Often in the early days of a deployment, the customer will turn around and start saying, Oh, you did this wrong. You did this wrong. You did this wrong. And then they'll look awkwardly and turn around and say, Oh, we gave this information, right? It's actually a really good way to flush out the stuff in your knowledge bases. That is not correct. Right. And, you could go there and come through it and, maybe there are answers to questions that no one even asks. Right. So when you start running it, I think it's just about, you could polish and polish and polish, or you could just go live.
NikolaAnd then as soon as you see that things are really going off rails, or, that people are complaining that something wasn't wrong, then the issue surfaces at the moment, whatever knowledge base you're going to give an AI agent. It's probably the knowledge base that your human agents use, but you don't know They're providing the bad information. You might find out occasionally But my experience is at least that there are competing incentives, right? Agents are they're trying to take as many calls as possible, especially if there are too many calls at the same time it's in the interest of the organization for these guys to raise their hand and say hey I don't think this is right.
NikolaI just spend quite some time and, they might figure it out. They might go on the website. They might see a different resource with a conflicting piece of information. They might ask their tier two agent. And, a colleague of mine likes to say there's a lot of tribal knowledge, right? So like maybe a group of Asian knows, but another group doesn't. Right. And you just haven't codified it because whenever you look at an Asian screening and contact center, there's 50 things open. Right.
DamienYeah. They're concerned with reducing the average handling time. They're concerned with churning through as many calls as possible. What are the chances that somebody, what if they had a bad day and they're just like, I don't want to take this note right now. I'm not going to submit this. I'm not going to go for some sort of like change order.
NikolaI would just not there. And whoever, like maybe they submit feedback to someone who doesn't understand what that feedback means and doesn't have to do it to push that information out, at least by creating them. The knowledge base that your voice assistant uses, then like you're codifying it. You're agreeing on that single source of truth, ideally you have it already, but if you don't, you should have it. It's good for your organization. If you, even if you don't have a virtual agent.
DamienNo, that's a good point. Joe, also one of the end user organizations in our customer base that is, has been like utilizing a platform to manage their their generative voice assistant and, take more control into their organization as opposed to us building the software and then having them run it, manage and support it, which we still do. But this puts more of that in, in the control of the customer, that was a big piece of our product roadmap, which was announced at the end of Vox and Devidas and Sean and folks in our product team and Megan all went through and talked a little bit about a roadmap.
DamienIt was largely around greater capabilities within this generative platform. Is this going to be the new norm for deployments for voice assistance in the contact center? Will people basically be building and managing their own off of platforms?
NikolaThey'll definitely be managing them, but I think that it's a huge space where, our goal has always been to provide the full experience and then to support all along the way. Right. We tend to do a lot of handholding at the start of that partnership, or like the initial version, and frankly, we provide a lot of consultation around, well, what should your knowledge base look like? So that are generative. Agent can actually make use of it, right? Cause it might be good for human, but maybe it's not in the right structure to be ingested in by, by, although that's usually not a huge problem.
NikolaSo, different companies have different appetites. We have clients that do a lot of the build themselves and, we kick them off and then they're off to the races and they do pushing the baby
Damienbirds out of the nest. You got this.
NikolaWell, yeah, look, and there are other companies that. Or not like that, right? And I think that is the beauty of working with, I think we, we had a slide and we enumerated 19 different verticals depending on how you look at it, right? Like healthcare is not the same thing as insurance. It's not the same thing as a banking and, the challenger bank is not the same thing as a tier one pan European bank either. Right? So technology is messy. It takes people, it takes aligning a lot of people, roadmaps architecture, security.
NikolaSo. There are places where it's easier to fully self serve and there are places where, the customer may not be equipped with the right skillset. And with things evolving so fast, many of them. Can benefit from a quick start. And that's really about time to value. Right. We talked about that piece where, Gartner is now turning quite negative on like the outlook of most generative AI POCs and they're like, most of them go into production. They're saying it like it's a bad thing, but isn't that the point of pilots right to test technology? And if the 10 percent are hugely impactful, that's still a good thing.
NikolaRight. But think that if you want to make the success rate of these deployments a bit higher than 10%, you roll up your sleeves and you do a lot of the work with your customers, right? You show them how you learn from them, right? Because most companies that are starting off, and we were like this once as well. Look, we don't know a whole lot about the contact center, about how they work, about how they prioritize things, how they organize their knowledge, what their priorities are. So by working with them. You're like building it for them and again build a platform where they can do much of it themselves.
Damienit makes a ton of sense. And I guess we'll see what the future holds But like you said everyone has their own flavor right and they're all gonna approach it But it's becoming easier and easier for organizations to manage their own deployments good stuff I think we're running up on time. If you haven't had a chance to check out box I think the recordings are going to be available here really soon. So keep an eye on our youtube channeled for that information and check it out. A lot of really great client stories, and I didn't even get a chance to touch on all the partnership stuff we had with AWS, Twilio, Zendesk, et cetera. OpenTable had a really fun session in the hospitality industry.
DamienSo give it a watch, but thanks for giving us your take on some of the stuff that we covered yesterday, Nikolai. Have a
Nikolagood weekend. Thank you. for hosting all the events.
DamienYeah, it was a ton of fun, but now I'm going to go take a nap.
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...