New guest, who dis? This episode, Damien invites Nathan Liu, VP of Deployments at PolyAI, discusses the company's approach to making the world's most lifelike voice assistant, emphasizing the synergy between AI and human voice actors to enhance voice assistant quality while safeguarding actors' interests. The conversation touches on concerns about AI replacing voice actors, recent controversies involving synthesized voices, and the future potential of AI in augmenting the industry rather than replacing human talent. They're also looking ahead at technical advancements in generating authentic-sounding AI voices and while preserving the integrity of voice actors' contributions. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Nathanwe still use that actor to record top up audio and live audio as well. On top of the synthesized voice to make sure the quality is perfect. And that means that there's an ongoing relationship. There's ongoing work for the actor. I think that any fears about, synthesized voices, reducing the need for actors is probably unfounded. It's probably doing the opposite. And actually increasing the work that we've been able to give to voice actors
DamienHello folks and welcome to another episode of the deep learning with Poly AI podcast. The only podcast that has someone who has built and delivered. Hundreds of custom voice assistants and also Nathan Liu. Nathan, welcome to the show. I really appreciate it.
NathanI was going to say you've oversold me there, but
Damiengot to catch you off guard. No, Nathan is our VP of deployments here at Poly AI. He's been with the company for a number of years. Nathan, why don't you tell us a little bit about what you do for the company? And then we're going to dive into what everybody is talking about today, which is the concept of voice as an asset in the bigger picture. We work with voice actors a lot, Scarlett Johansson in the news recently going up against open AI for use of voice. That's awfully similar to hers. So tell us a little bit about what you do and then we'll get into the meat of it.
NathanYeah. So, I'm Nathan. I run our delivery and deployments teams here. That includes dialogue designers who are the ones who make the magic happen for us. So yeah, this voice story is been my life for the past four years. I, as many of our team did started in the contact center space. Some of us started in the tech contact center space. I started answering phones from unhappy people. So I think. I like to think that most of the team here have always had an instinct for what a genuine customer service agent sounds like.
NathanAnd we've been trying to make our voice assistants be like that too.
DamienThat's amazing. Let's talk a little bit about the process that you do internally. Our traditional product is, creating these really lifelike voice assistants for enterprise contact centers, as you mentioned. We do this in a way that's a little bit different than how other people are approaching voice in that we bring in a sort of combination of voice acting. Along with some synthesized voice. So the concept of working with voice actors has been long standard practice here at poly AI. Can you tell us a little bit about your process when it comes to creating a new voice assistant for a customer?
NathanYeah, I think probably one of the first kind of breakthroughs we had in the early days was realizing that when you. When you tell a customer that you're a virtual assistant, that customer tends to react quite negatively. So that led to us starting all our calls with just like a, how can I help? Yeah. And over time that's evolved. It's not a hard and fast rule but that. Feeling of you want to make a customer trust you from the outset led to us really focusing on how lifelike we could make the voice assistance.
NathanPeople trust humans. So if they treat us like a human, then it's going to be better for us. And it's going to be better for the customer as well. So that's where it started. We worked with many voice actors throughout the years. We worked with voice actors who. Are very well trained and eloquent. And then we worked with voice actors who are pretty amateur freelancers. And actually the latter is the better ones. They sound engaging. They sound, they probably sound less
Damientheatrical too, right? They have that, the emotive, like I'm on stage, I'm in the moment.
NathanYeah, exactly. So I think that's where we found yeah, a bit of a niche. Of course you need the tech to back it up. It doesn't work. Being really human, if whatever the customer says you can't understand, but
Damienand then when you say we work with a voice actor, what's the typical process in creating that voice?
NathanYeah, so things have changed over the years. Originally it was, a very personal relationship where we would Record every individual prompt with the voice actor. It meant working with the voice actor to record all 365 days of the year every time of the day to the nearest 15 minutes every number up to a thousand it was a Yeah, tricky. Fortunately fortunately in the last year, things have become a little bit, the tech has got so good that, we can generate these voices and for things like addresses and dates and times we can use an existing voice actor and we can completely generate that.
NathanSlot that date that time freely and we don't have to get the voice actor to record like that quantity of stuff And no one wants to do that. No voice actor wants to do that no company wants to spend that money So, it's winning for everybody. And then of course kind of the third stage of that is Having a synthesized voice do it all and people who have experimented with these Text to speech voices over the years have probably felt is that most of them suited to things like audiobooks and things like that. So we ended up sure Audiobooks my accessibility so we ended up in the same position that we did three years ago where we Have a new challenge in making things as human as possible and making the text right voices as human as possible So yeah, that's our third stage is how do we create a synthesized voice which sounds like a 16 year old on their saturday job in a call center
DamienWhich is what we all strive for. Right. That's the transition. What do we do to put the voice actors at ease when it comes to that concept of, okay, we're not only taking what you've recorded, what you'd know, and if willingly given over as regards to a package of recorded phrases, numbers one through a thousand, et cetera, and the synthesized voice. How do you go about putting voice actors at ease, especially at a time right now where there's. So much speculation around, is someone going to take what has been my bread and butter from a voice actors positioning and now anyone can recreate it and we'll go into some like stuff that's in the headlines and talk a little bit more about that.
DamienBut what do we specifically do? What do you and your team do to put voice actors at ease when it comes to how their voice will be utilized?
NathanSo I think we, we come from a strong standing point in the first place, which is that our customers are a proper enterprises. So, right. And not being used by. Anyone on an app on their phone, they have a specific task, which was something like pack a parcel or book a hotel room. So that tends to, give some comfort to any voice actor we work with anyway, beyond that, with our top two voices, it's not like we're going to create a voice clone of them and then run away.
NathanWe create, we create the voice code and then we work with that actor for a much longer period of time. So most text to speech systems aren't going to be able to pronounce that. Yep. A brand name perfectly or a product name perfectly. So we still use that actor to record top up audio and live audio as well. On top of the synthesized voice to make sure the quality is perfect. And that means that there's an ongoing relationship. There's ongoing ongoing work for the actor.
NathanI think that any fears about, synthesized voices, reducing the need for actors is probably unfounded. It's probably doing the opposite. And actually increasing the work that we've been able to give to voice actors.
DamienYeah, that's and that's good to know. I'm curious and I will be asking a little bit more about where you see the future of AI as a, as like an industry enabler within the voice acting space, as opposed to a direct competition. But let's talk about the elephant in the room a little bit, obviously in the headlines recently, Scarlett Johansson up against Sam Altman specifically calling him out as head of open AI in that. Their voice Sky that was released in the recent iteration of GPT for, Oh, for open for Omni rather it sounded a lot like Scarlett Johansson and not just.
DamienThat happened stanchly or circumstantially. He had approached the actress a few times and asked her to be a voice. So it caused us big kerfuffle a couple of weeks ago with the Drake and Kendrick Lamar rap battle Drake's entree to that was putting out a song that used a synthesized voice of Tupac, the late rapper, his estate got involved and said, no, no, no, you can't do that. Even for this like little beef they had him, they were going to sue him. He ended up taking it down. Now it's caused a lot.
DamienIt's in the headlines, right? This concept of. Yeah, your voice is up for grabs. If you've recorded anything anywhere, you could probably be, it could probably be taken. Obviously we take a proper course of action in regards to how we work with our voice actors, but just looking at the industry as a whole, what do you think are some easy steps that voice actors and the industry should follow to, maintain the integrity of using someone's voice? Because the legislation is fuzzy on it right now.
NathanHonestly, I didn't think that sounded like Scarlett Johansson. It sounded like Scarlett Johansson. So I, yeah, I don't know how much I believe that story, but putting myself in the shoes of Sam Woolman for a second, there's no reason to do anything malicious there at all. There are so many voice actors. There's. So much opportunity to get a voice actor who sounds like you want them to sound and you know I think it would be we would be shooting ourselves in the foot if we were to use a If we were to frustrate the voice actor community, and I think sure I know that as well it's Probably unfortunately got this far as it's scary to voice actors and we're nobody, we're not out there to alienate voice actors.
NathanWe're there to work with them and give them work. So, it's a bit frustrating to see the situation when it probably was was not the intention of opening up.
DamienMaybe a lot of a lot of smoke, but no fire here. Perhaps this is something that's the threat of which is a little bit overblown because of some high profile instances, but otherwise this isn't something that it's really going to behoove anyone to violate that trust between actor and industry.
NathanYeah. Yeah. Also, I think how much money would Scarlett Johansson have made if she'd said yes?
DamienSomething we're all asking ourselves and also what can I do to make for a more compelling voice? That's too funny you had talked a little bit earlier on about you know You see AI is being something that's going to better enable this industry. We rolled out recently a voice model or voice A voice cloning model called Femi. We had an episode about it a couple months ago. Obviously this is something where the technology is there to make really high quality clone voice. Where do you see as being the good combine between, a voice actor, organic agents, et cetera, and AI voice clones or AI voice cloning technology to really maximize Their stickiness within the space again, not AI replacing voice actors, but AI augmenting their capabilities and what they bring to market.
DamienWhere do you see some opportunities for for organic voices and AI voices to combine?
NathanProbably our use cases, enterprise use cases. And Oh, interesting. Surprised and a shock, but it's genuine as well. It's a place where, brands are very sensitive about how their brand comes across. Every big organization has, a 50 page document saying what words you should use and what words you shouldn't use marketing and then whatever we have one as well. So I think that it's a perfect place to, for an actor to have, to give an actor like a limited scope when it comes to cloning the voice.
NathanSo they've got a clone voice. It's only going to do these things. Yeah. And it's tuned specifically to that brand. So when we have our biggest customers, we don't reuse voices for like our real big big, like global brands. They will have a voice, which is theirs. It sounds like them. It's recognizable as them and follows their principles. And that's also exciting for a voice actor as well. Like you are a representative for. For that big name. There's yeah. I think that's probably where it is. That's the perfect use case.
NathanThere are general uses. You could, I don't think there's a, it's necessarily bad for an actor to clone their voice for a general use case, but. It's always going to feel safer to do it for a proper enterprise grade.
DamienYeah. Once you have a, once you have a general clone out in the wild, like it's harder to protect as opposed to within an enterprise use case, like you said, then you're dealing with someone who has an established brand and they're not going to go and violate that and trust of the consumer by misusing or mistreating. An actor, especially when that actor is going to be the voice of their brand ongoing. It seems like it's a loss if they try and do anything
Nathanshady. There's always, in addition to that, just maybe goes without saying with all our synthesized voices, we take steps to ensure the safety of that voice. So yeah. We pay the actor an ongoing fee. And the actor has the opportunity to withdraw their consent at some point in the future as well at that set time. So that's
Damieninteresting. Giving them an out clause at some point where if you ever feel
Nathanlike it's not going in the right direction, no one has ever used that out clause. There's no reason to, we would hope that no one ever would, right. You never know if that's. If you decided to get into politics maybe that's the time. And
Damienyeah, you want people playing clips of how you're endorsing, somebody's cat food or something like that. And it just doesn't help my campaign. No, I can totally see that. I think the concept of we all make mistakes or something as youths or whatever, that we wish we could go back and pull back, but now it's your voices out there and if it's representing a brand and all of a sudden something changes for you, that you have the ability to go back and reclaim that. I think that's. That's highly observational for, from us as a company to say, we, we respect this for you. And so if you ever feel that you're in that position, like you have the right to withdraw your consent, also something to point out, it's never happened.
DamienRight. Yeah. These are cool. Sure. They're having their family call up and say, listen, it's just like me.
NathanExactly. We've all, most of us have cloned our voices internally.
DamienMine's a little fuzzier. I don't know if it really brings my subtle curious nuance, but that's actually a good question. A good segue is one of the things that in the recent unveiling of GPTs four O are four oh. Is the emotionality that the voice demonstrated. And that's seen as like a big leap. I know that that is one of the value propositions that, we had always delivered is that by using voice actors, it really delivered some things that you couldn't program to write those drawn out. Let me see here and Oh yeah, I see that this blah, blah, blah.
DamienIt just sounded, it had that nice conversational flow. And now. We're seeing some attempts at trying to recapture and replicate that. It's not perfect. There's still a distinct uncanny valley. Where do you see that going in the future? Where do you, how do you, first of all, what are your opinions on the 4. 0 emotionality? And is it anywhere close to being able to, I don't want to say do, but convince somebody that it's actually human talking?
NathanYeah, I think there's probably two points there. So when it comes to emotionality and the personality, the voice had, I think we find that it really matters on. The source voice really matters. So the voice, so if you train on a voice, which is very well trained in theatrical, the voice that comes out is going to be very well trained in theatrical, if you trained on a voice that is much more conversational, if you train on a verse, a voice that is flirty, then it's going to, that's the same thing that's going to come out as well, which opening I did. So I think that's achievable. It's probably not rocket science to make that leap when it comes to things like extending vowels and Sounding like you're thinking and ums and ahs, that's a little bit more tricky, but fortunately there are a few tools including speech to speech tools, which allow you to convert your own voice into someone else's voice, which means you can show, you can speak in whatever way you want to speak.
NathanYou can add the ums and ahs that you want to make.
DamienAh, interesting.
NathanSo, yeah, there's, It's, we're a lot closer than I think most people think to really being convincingly human. There's always going to be that sense that it's a little bit uncanny, but that shouldn't last for much longer.
DamienInteresting. Where do you see there is being added opportunities for collaboration on let's say voice actors or, maybe internal industry builders that are looking to create their own voice assistant that want to use voice actors, but also want to leverage the power of AI. What are some more opportunities for collaboration between those two platforms, i. e. human actors and voice technology, voice AI tech?
NathanI think at the moment, the, the voice synthesis side and the Building an AI agent side a little bit different. They're two different things. We have we have Jupiter, right? Which is a generative AI platform to build an agent. And we have a set of standard voices and we have a set of enterprise grade voices, which are used by our customers and they're separate, but it's almost, in fact, it is certain that give it a little bit of time. And that will be one thing it will be here is the contact center is my highest performing contact center agent.
NathanAnd they're going to talk, they're going to talk into a platform for a few seconds and they're going to be there's going to be a voice assistant in their voice with all the knowledge of their contact center. And that's not too far away. It's not there yet, but it's not too far away. But yeah, they're a little bit siloed at the moment and I think they need to be siloed at the moment while everyone, while voice actors build confidence that this is, something, That's going to help the industry and not something that's going to hurt it. And once that confidence there, then we can go the next step. But yeah, we're here to help that voice actor industry that we've worked with for many years, build that confidence and doing something like copying someone's voice without their consent.
NathanWould just break that completely so we're not going to do that
Damienright and you would think you know There have been copyright protections in place for other sort of like tonal properties right like music and even Audiobook recordings which are specific to a work But still carry the same protection. So voice, it's only a matter of time. I guess the question is just for music, like to what extent is a voice similar versus cloned or replicated? And I think it'll be interesting to see from, with regards to legislative protections, where those thresholds are drawn do you have to have a 15 second thing and match like bar for bar, what the note is, what the key is, what octave they're speaking in, et cetera.
DamienAnd assign those tonal qualities that we do for music.
NathanPeople have been able to photoshop whatever they want. Any famous politician or celebrity for a long time. That is a little bit scary, but we as a, we as humans have learned to identify that and learn to be skeptical about things we see online, especially if they are controversial. We're all going to learn, we're all going to learn that about voice as well. The
Damienmodern take. I don't know a deep, I can't define a deep fake, but I know it when I see it. That's incredible. I'm really fascinated at this concept of the sort of speech to speech. Is that something that we're actively deploying or that we're actively utilizing in some of our voice creation that's happening now?
NathanYeah, we have voice assistants that are a hundred percent speech to speech right now where we've used. A synthetic voice, but had one of our team. Speak the script in their own tone and convert to that voice for all of the prompts. So
DamienThat's amazing. And what's been the impact for something like that? Just again, kind of com combining the voice of the best agent for the enterprise, along with now saying they didn't have to record any of this, and here's what it sounds like. What's been the response from those client customers that have had that sort of voice assistant developed?
NathanThe impact has been zero, right? Because you can't tell that it's not a real person. And yeah, we sell our voice assistance as being. Incredibly lifelike and engaging and like real. So for them clients, that's exactly what they got.
DamienThat's wild. Is there a favorite voice assistant you could call it out by name or just some phrase or something that you thought that really stood out from all the ones that you've developed that really like plants itself is one of your favorites.
NathanWe all have our favorites, but it wouldn't be fair to the others to
DamienAll those children you don't want to call out the number one, all right Well, nathan, I really appreciate your time today. It was a great conversation a fascinating topic Hopefully this stuff that's in the headlines will fizzle out as pop culture talks tumult and feuds tend to do. But it's really cool to see that the technology is really better enabling industries, especially in the contact center and in the customer support and service space. So really appreciate you giving a glimpse into what you and your team do. And thanks for being on the show. Thank you.
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...