In this concluding episode of the 'Anatomy of a Voice Assistant' series, CTO Shawn Wen discusses the intricacies of speech synthesis in voice assistants, emphasizing the importance of authentic, human-like voices in improving user engagement and containment rates. Kylie and Shawn chat about the evolution of AI voices from the early days of IVR systems to current sophisticated standards, highlighting the necessity of collaboration across engineering, marketing, and branding teams to achieve desired voice quality. The conversation touches on the ethical considerations of voice cloning and the role of voice actors. It's a good one! Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Shawnwe'll do a lot of these, like a beta testing in our days. We found that just by changing this to. A very welcoming, how can I help was a very human like voice. You increase the containment rate by five, 10%. And it's just not simple.
KylieHello and welcome to Deep Learning with Polly AI. Today is the final episode in our Anatomy of a Voice Assistant mini series with Polly AI CTO, Shawn Wen. Hi Shawn.
ShawnHello.
KylieAnd today we're going to talk about speech synthesis and it's a This part is really interesting because bot builders can get really high functioning text to speech off the shelf today, right? Like it, it works. So it's more a question of what is the quality that you're looking for and how do you achieve text to speech that sounds great? Good or good enough.
ShawnYeah. Yeah. I think speech synthesis, I would say in the history of conversational systems has been part of the field that is, majority ignored because there are so many problems to solve the language, understanding issues, the ability to understand the speech all the fun things around it. So I think historically. Engineers and researchers just focusing on to get things to work, rather than how they actually sound like.
ShawnSo in the early version of a lot of the voice assistants, I think even, I think, when we get to Siri, the situation has already become a lot better, because Siri is the first, but, consumer facing voice assistant, they focus a lot to get the brand right because Apple is all about brand, right? But if you talk to some of the older generation of the IVR systems, the new ones, the systems, even though their customers paying a lot of money for those automations, but the quality is not that great because there wasn't.
ShawnWe are in a new world where generated AI has changed, not just text generation, but also audio generation as well. So we are seeing like new generation of these kind of voices, just as technology become better. However, in poly AI's eyes, we are still, viewing maturity of the solutions out there, not really good enough because we have a history of obsessing with voice quality. And I guess there's just like in our DNA is something that, we are still very proud of and we still continue to do what we do.
ShawnAnd yeah, and then we started to pay attention to these voice qualities in contact centers way before people start paying attention to, I remember when I, when we talked to one of the Gardner analysts, the feedback after several sections of it, the feedback to us was like, you guys are the company that taking the voice To voice quality to like really, really far, like the furthest that they have ever seen in the industry. The concern was like, you probably take it too far that not a lot of people appreciate that.
ShawnGuess what? Once January I take off, like people start thinking about it people realize voice quality is. One of the major thing they are looking for.
KylieYeah. I think when you like IVR is such a good example, right? Because well, sometimes this happens to me, you have the recorded message now, right? So this call may be recorded for training and monitoring and you just tune out. You're not listening. You're not paying attention. You know what that message says. And then you get to You tune back in again and the same voice is saying press three for this and I'm like oh god I didn't listen to the first two options like I just tuned out because I know that this thing isn't going to say anything useful to me. And so I think we've come so much further in understanding that making an experience that people actually engage with.
KylieDoes require that voice to sound really, really good.
ShawnYeah. I remember the first generation of our system, we were using off the shelf TTS voice, like in 2017, 2018, arguably the best Technology is not as good as it used it is today, but it is already a lot better than several years ago, right? Because we started way earlier than generative AI. So we started, we put out the TTS in, engineers in the team were like, Oh, that's fine. That voice is good, including myself. And then all the product managers and also Nicola, our CEO was like, no, that's wrong.
ShawnWe can't do it that way. So In the end, that voice assistant didn't get deployed. We do a lot of work, so came up with the algorithm that allowed us to connecting voice actor voices together to make itself as nature as possible. And that was the first generation of a system. That's how far we are willing to take the voice quality too.
KylieYeah, and I think when we were working with voice actors, there was so much direction that you could give so you'd get a recording back from somebody and. Well, actually, I think one of the things was firstly, you had to teach the voice actors how to not speak as if it were an IVR because they they knew what the role was, and they wanted to sound like a robot because that was what people were asking for, right? That, that had been their job. So we had to say, no, no, no, no, no pretend you're working in this restaurant and you're busy and you're answering the phone, right? And then you're like, hello, how can I help? Sound like a person again.
ShawnOur initial system. study in the hospitality sector, right? So they, voice actor has these issues when they work with contact center. They feel like they have to record at least very rigid sentences. However, our clients in the hospitality sector, they just want it to be warm and welcoming. And as a scientist, we do actually ask ourselves how important these voice qualities are, are actually impacting to our customers. So we did actually do in early stage, we do some, AP tested testing on several things, like including, TTS voice.
ShawnComparing to a very well recorded human, like voice from voice actors. And then we told them exactly how to behave. And we say information or we've, we've revealing it's a agent or not an agent. Right. So we'll do a lot of these, like a beta testing in our days. We found that just by changing this to. A very welcoming, how can I help was a very human like voice. You increase the containment rate by five, 10%.
ShawnAnd it's just not simple. And that just tells you like how important, I think one thing we are compacting in a context and there is that because some people get used to this robotic message and once they hear that, they immediately know that, no, this is not going to work. I'm not going to talk to it. Right. So. Even though the voice should be good, but the system should be good, but they are not willing to give it a chance. So that voice quality almost sounds like a real human gave the voice assistant another chance to be successful, right?
ShawnIf the user started to engage in the natural way in the first turn or second turn, even though the voice assistant didn't pick up things at a third turn, but, and that at that point, the user has realized that it's talking to a system, they are probably Because they have already gone so far in the conversation, they are probably willing to give the voice assistant another chance. And once they try it, it becomes successful. Then the confidence be built up. And then there's this enough trust between two with the user that we can take this conversation to containment, right?
ShawnSo I think that's a very important thing.
KylieYeah. And it, it takes us back to something we spoke about in the SLU episode, right? Which is that a lot of ASR is going to work better if the person actually says more, if they give more context. And so you hear this robotic voice and someone would just shout agent, agent, or you say, why are you calling? And they say credit card. And it's yeah, that that's not enough information. Like you're providing the information for a very rigid bot. But if you were to say, Oh, my, my credit card has, I think it might've been stolen. Am I able to freeze it? You're much better able to help the person.
KylieAnd so you can actually get them through to that second turn of the conversation. So there's something about this, the quality of the voice and sounding human that enables people to engage in the way that gets the technology, what it wants without the person feeling like they were being forced to change their behavior.
ShawnThat's right. Yeah. I think that's just super important to consider that aspect of the human perception, right? In the conversation, that's where you should progress as far as possible. You need to understand human as much as possible so that, the human don't feel frustrated and wanting to drop out. In those conversations, the human also feels that you, the bot genuine or actually doesn't sound like robotic so that they don't feel like, at some point they also drop out of a call because of their voice quality.
ShawnSo we do actually think that, all these aspects are important as well, that's why it should be a really successful voice assistant with a high contentment rate.
KylieYeah. I think my theory is that there's going to be a day when You hear the robotic voice answer the phone and you think, Oh, thank God it's a robot. It's going to understand me. It's going to get me what I want. Right. And we're going to revert back. And we're going to be like, Oh, give us those terrible TTS voices again. Cause they let people know that it was AI, right. That there was something else behind it. But until we get to that point we still have to focus on, delivering voice experiences that something we say quite a lot at Poly AI is that we want to create we don't want to trick customers into thinking that they're talking to a person, but we want to enable them to forget that they're talking to a robot.
KylieRight. So they can actually do what they need to do.
ShawnIn the end of the way, right. We are, the really good voice quality doesn't really means that, you are trying to deceive the customers that, you are the person it's really about Giving that voice assistant, that kind of human level trust to the customers, waiting to engage with it because the system would continue to improve. And I think the regulation, once it comes in, you will actually change the entire landscapes as well, because in some countries you probably have to reveal your voice assistant anyway, but regardless of whether it's a bot or not a bot, whether, they are trying to trick the customer into it.
ShawnI think. All we would agree is that interacting with a more human sounding voice assistant. Will be ultimately the right thing to do anyway, because we all like that human touch in despite the fact that in the future, I probably talking into automatic voice assistant over the phone because of context and center would be highly automated in the future. I still would like to prefer to talk to a more human like robot bot, right? So
Kyliewe'll see. Maybe my future is a. My future and your future may be quite different there, but we'll see who wins. I think when we were talking about this previously as well, you said something quite interesting about how, because a lot of this, a lot of what we've spoken about so far is like a psychology problem, right? It's a user engagement problem. It's not necessarily an engineering problem. And so when you have a team, Working on this stuff, like we did at the beginning, right? When you have your machine learning engineers and they hear the TTS and they're going, yeah, that's fine. That, it does the job. It's good because I know what TTS is and I know this is a good one.
KylieBut then people like me, like marketing come in and they say, no, I don't want it to sound like this. I want it to have a Scottish accent. And I, I want it to be friendly and all of this stuff. And I think it just highlights the importance of having. The right people on the team, like this isn't, and this goes for the whole voice assistant build, right. It's not purely an engineering problem.
ShawnCorrect. Yeah. And I think, that's why I think we have been quite, since we care about this voice quality very early on we have already, we started to think about about this aspect of the dialogue. The design quite a lot, those dialogue designs, it's not just about the traditional sense of Oh, what kind of messages you should say? And also like, how do we actually navigate those entire customer journeys in the contact center? But it's also about the quality of a voice that, in that particular flow, what should have voices.
ShawnAnd sound like we have some more sen, more sensitive customer use cases. Yeah. Where it's the bereavement use cases, right. Where actually the customer lost someone and because the, there someone have some saving with that particular bank.
KylieYeah.
ShawnWanted to get what they wanted to know how do they do with it? Right. So I think that is a perfect, scenario. When we started that scenario, we were freaking out. We were like do we actually want robots to deal with this kind of use cases because it feels a very new in human in essence, but once we carefully put in the effort to design crafting that experiences we saw, we heard we have seen some calls that just that. Phenomenal. And that's better than a human would do. And I was actually quite proud of the product of that use case.
KylieYeah, I think that's incredible. And it's really important. Right. And it's something that a lot of, companies come to us and that's the big concern because you don't want to have this sensitive use case where somebody says. Something terrible has happened and the voice assistant replies with this chipper like great. I can help with that. It's not appropriate. So you need to be able to, to yeah, design these conversations in that way. So Shawn, we've talked about this and we know there are some pretty good TTS voices out there. Some of the.
KylieLet's just say some of the big cloud providers, perhaps some of their TTS is still not very good.
ShawnThat is right. Yeah. And then, I had a lot of these conversations with some of our partners and us, but especially NVIDIA guys, because they have been pushing a lot of these open source models. And then there have been. To them that, Hey, why you guys are not investing in TTS as much as, like for example, they invest a lot on the speech recognition. A, I think Nvidia, all the NVIDIA models on the open source arena, they're probably like, nine out of 10 best performing models are from Nvidia now.
ShawnSo they have been invested a lot in the speech technology, but not in the TTS front. I was talking to them like, why you guys are not. Investing as much as you should on the ASR run, on the TTS phone instead. What they told me is that it's a very conscious decision to make. It's because. You probably heard about this, Scott and Johnson's, social issues with, right. It's exactly that kind of scenario because these technology has become so good these days that you can almost just listen to a one clip of your audio and then just synthesize your voice.
ShawnIt's very, very human like in a way, that is. It's crazy, right? And I think in, because of that, I think some of these are really big tech names or companies. They are, they choose not to push too much on those fronts because that would just potentially associate their brand names with some other, misuse use
Kyliecase. On, on
Shawna web. That's one of the reasons, right? Google cloud TTS, for example we never actually use them in production, but we use it as a fallback in case our system didn't actually work well. But you were surprised, that, Google, I think was a research powerhouse, but on the speech synthesis from their quality has not been very great. I like to joke around is that they have a lot of these different voices selections, but. All of them sounds like from the same family is that they're all similar to each other.
ShawnAnd I think I talked to some of the people work there as well. It's just because that, they are just limitations of data. They can use it, because they need to be compliance. They need to be, have a fair use policy. And that's like the limitations of the cloud providers, right? Because voice. Voice footprint is more sensitive. It's more considered as like personal information and less sort of things. Imagine if I clone you of that voice, if you set up some voice by a metric somewhere with your bank, they hack into the system with your voice.
KylieYeah. Wow. That's scary.
ShawnBecause of that reason, it's you're not seeing, we're not seeing a lot of like good cloud vendors doing a lot there. And on the other hand, we saw the newer generation companies, like pushing the boundary of these voice quality, because that's literally one thing they do.
DamienAnd
ShawnCompanies like 11labs, Play HT, their quality is actually a lot better they probably turn down all the data they can find. Yes.
KylieAnd then they have some really great features, which again, I think comes back to this sort of who's role. Is the voice quality in building a voice assistant. Cause with, I think with 11 labs and I know with our internal models that we've been playing around with, you can record yourself saying something to get the tone, right. To say, I don't want you to say, how can I help? I want you to say, how can I help or whatever. And again, this is something that. me in a brand role. It's good that we're, working on the same thing here. Cause if we were in this enterprise organization, right.
KylieYou'd be building it. And I'd be saying, no, Shawn, I went and enunciate exactly like this. And so having that collaboration is.
ShawnYeah, we are seeing a lot of our enterprise customers as well. They are not happy about the synthesizer voice.
DamienThey
Shawnwanted to have control. They want it to be speaking slower, faster. They want it to be, higher pitch. They wanted to change the tone they are saying. So I think. Those kind of controls would event, inevitably need to be provided to the enterprises.
KylieI think
Shawnwe are starting to see a lot of features around that. And, we also put in a lot of effort to actually, making sure that our in house solution have those kinds of configurability as well. Just like all, every part of the system components we are doing. I think serving enterprises, That level of control is needed, not saying that everyone needs to control. Like a lot of people, they're quite chill. They just wanted to actually work to a certain degree and then they just wanted to, forget about it. But I think there are quite a lot of people focusing a lot on different aspects of the system.
ShawnAnd I think as a vendor in the market, we just need to serve all their requirements like
Kyliethat. Yeah. And I think there's a real opportunity here. What the work that we're doing at poly AI and other, voice assistants are. Enabling brands to, to have a voice over the phone in the same way that they have, a tone of voice on the website and the app, the same way that, you might have a certain person doing your voiceovers on your adverts and the way that you speak there. So really being able to create this brand voice is something that is becoming possible in a really exciting way.
ShawnYeah, exactly. I think, yeah, you are completely right. Actually in a lot of the, in some of the industry sectors, clients are already thinking about that way. Like I remember when we first talked to our first casino clients in the U S right. They have a lot of presence in Texas and I wasn't in those engagement, but I was actually hearing a lot of my colleagues talking about this is that we try to pitch them how good our language understanding is, we pitch them, how good, no matter what kind of accents the system can pick it up and the system will be there, 24, seven, pick up a phone, et cetera, et cetera.
ShawnThey're not interested in all that. The only thing that he, the person was interested was that. Can your voice assistant speak in Texas accent, right? And it's basically, you have to start with the conversation, say, Hooray,
Kylieyeah.
ShawnHowdy, right? Yeah. So, so just like a very different, because like they they consider that as part of their brand. Yeah. Especially for some of these Yeah,
Kyliewe went, I remember we went really heavy handed in that example. We went way over the top Texan. I'm not even gonna like Good on you for trying, but I'm not even gonna go there. That's, we were very lucky. How cowboy about it? And yeah, and the client was definitely like, mm. Well, that's not a brand, but we can see how far you can take this. Yeah, yeah, yeah, exactly.
ShawnYeah. We can find out. I think in our typical engagement now, right? We would always present customers at least three voices for them to run. And this is basically based on what we understand about the customer brand and what they would like. They will be able to, Change into different voices if they don't like it. Right. But I think having those kinds of journey is actually very important. We are one thing that's why this has become more and more important is that we are seeing a lot of the customers marketing team and branding team are getting involved as well.
ShawnEspecially, primary reason is because of JNI, but also they are applying JNI to contact centers and they have the voice. So they are also wanted to get involved in that. So I think that aspect I see is definitely the future. And then that's like how people, the people would study to expect that, Hey, the voice quality just be, need to be as good as possible, right?
KylieYeah. Yeah. Yeah, absolutely. So I think one, one thing that we did touch on earlier, but maybe worth going back to is this idea of ethics, especially when it comes to voice cloning. So if you're, if you're a team in an enterprise in charge of building something like this, or in charge of buying something like this, how do you, how should you be thinking about ethics?
ShawnYeah, I think for. Us, right. We do actually invest into voice calling technology and also, yeah, I think the primary thing is we are not planning to release that for a third party to be able to, clone their voices, because that would open up the Pandora's box, because you don't know who would actually use it to do actually, to do things that are not supposed to be done. A lot of our voice, to be honest, I think, also very important is that the voice actor community is also, are also aware of these issues, right?
ShawnTo them, it's not just about ethical, whether you use my voice is to do certain things that you said you are going to do, but not the other things that you are not supposed to be doing, right? So, We have a lot of these. We have been working on back and forth on this contracting element with better voice actors, because that is a very that's a very strong community. I think a lot of people talking to each other. And if you want to find someone who is like really good voices, this is where, that's exactly the community you should work for. But they are also aware of the challenges, posed to them, not just about ethical issues, but also You know, their future, et cetera.
ShawnWe do actually work with some voice actors. There are a lot, they are very forward looking. They're saying like, Hey, I think this is the future.
KylieI
Shawnwanted to make sure that, in that future it's, if you're you're going to use my voice in that future. I'm going to be part of the future too. So we have structured these like revenue sharings and those kinds of also we structured in some ways is, Hey are you happy for your voice to be used for, third parties or clients, if you don't, that's fine. Are you happy to, for your voice to be used in the hospitality sectors, the utility sectors, that particular sectors, right?
ShawnSo they also have a say on the, their. their own brain and their awareness. Hey, like, where do I want my voice to be exposed to? And of course, like if you expose more, you can probably get more return from it. But then at the same time, if you want it to have a control, that's completely up to you as well.
KylieYeah, I think it's offering that scalability to something that, you can only take so much voice acting work at a time. But also you were doing it on this one to one basis, you'd be making these decisions yourself as you went along, and you want to make sure that you retain control over that.
ShawnThat's right. And I think in the end of the day, right, the way we are thinking about working with voice actors and also applying these latest speech synthesis technology, it's not to replace them and, out of the job, because we do think, because I know our obsession about voice quality would not go easily just to say, Oh, like we clone the voice and it's done. It's never going to be the case. We wanted to continue to work with the voice actor, right? And the voice calming model has its own limitations.
ShawnLike technically, it's still not as good as the real humans. Like we, we done some internal surveys about, a lot of people listening to a lot of different voice voices, a synthesizer comparing to real humans, real humans. Yeah. I think we are, we are doing, we were doing like one to five. I think the voice the AI voice is probably considered consistently rated a 3. 5 and the human is actually about 4. 5, right? So there's still a gap again, it's a subjective, but there's always going to go going to be gap, right?
ShawnSo we do actually wanted to maintain a good relationship with the voice actor community. So that whenever. We need your help to create like craft, crafted voice recording that is actually very on brand that they're there to help us. Right. So it's going to be a partnership relationship.
KylieYeah. Excellent. So I think to summarize a lot of this part of, the sort of spoken language understanding and ASR is going to require a significant investment from an engineering team to really maximize that, really make sure that you're getting the best quality understanding of what customers are saying on the phone. Then you need to have, again, the same sort of resources to really support you with the with your LLMs to make sure that they are. Behaving correctly that they're not hallucinating that they're able to move you through certain transactions.
KylieAnd then when it comes to speech synthesis, it's. It would be easier, to just plug in a Google TTS. And Hey, it'll work. It'll be okay, but it is going to sound robotic and this sort of, this is where you really need to think about design. This is where you really need to think about how you want it to sound, your brand and make sure that you're bringing in the right people from across the company to, to really maximize that. Don't just chuck it on your engineer's shoulders. Cause it's a bigger, you It's a brand problem as well as an engineering problem.
ShawnYeah. Yeah. I would there's a way I try to describe this and then talking to, a lot of my, our prospects and clients and even internal teams is that, Hey, as a developer, you have spent so much time polish your voice assistant to understand every accents, understand every languages, and then not to hallucinate and make sure that, there's a proper guardrail in place, If you have already spent so much time, just get it right, get it to a perfect stage. Just also thinking about the voice quality,
Kyliethink
Shawnabout the brand you wanted to project your, your, whatever your brand, your company, to think about that particular brand identity, choose the right voice. Don't just pick one of the random speech synthesis because I guarantee you it's not going to be good enough in that new world where every voice assistant sounds like a real human. That's definitely going to be the future. So if you have already spent so much effort, that final mile, just finish it properly.
KylieYeah, it's the cherry on top. Like it's going to make all of this stuff come together in a really nice way. And likewise, a really great voice with. LLMs that are running wild and really bad ASR is not going to save you there either. It's a case of all of this stuff working together. Cool. Well, Shawn, thank you so much for taking so much time to sit down with me and talk us through all of the different pieces of making a voice assistant. I've certainly learned a lot. I hope that anyone listening to this podcast has learned a lot too. Thank you once again, and we will see you on future episodes of Deep Learning.
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...