Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Deep Learning with PolyAI

Deep Learning with PolyAI artwork

013: The Evolution of Voice Generation

Dec 21, 202324 minAudio

Join host Kylie Whitehead as she discusses the transformation of voice generation with PolyAI's head of machine learning, Paweł Budzianowski. They delve into the history of voice generation, discussing its inception in the 1950s and the influence of movies like 'Her' on its development. They also explore the shortcomings of current AI voices like Siri and Alexa and suggest strategies to improve them. The conversation further revolves around the effect of deep learning strategies on voice generation and the imminent breakthrough with speech-to-speech models. 00:21 Discussion on Voice Generation 01:12 History of Voice Generation 04:16 Challenges in Voice Generation 05:26 Role of Neural Networks in Voice Generation 08:33 Impact of Context in Voice Generation 20:52 The Shift from Voice Actors to Synthesized Voices 24:19 Future of Voice Generation: Speech to Speech Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

KylieSo if I'm really angry, if I sound angry, it will be able to adjust its voice to be maybe more accommodating. Whereas if I'm excited, it can be excited with me. Yes,

PawełThat's that's already happening.

KylieHello, I'm Kylie Whitehead. Welcome to the PolyAI podcast. I'm joined today by PolyAI's head of machine learning, Pavel Bansinoski. Hi, Pavel. Hey,

PawełKylie. How's it going on? Yeah, good. How are you? Not bad. Excited about our new topic, which we talked about a little bit before the other end of it, which is voice generation. Yeah,

Kylieexactly. So in episode six, a few episodes ago Pavel and I spoke about speech recognition and like Pavel just said, we're back here today to talk a little bit about voice generation and Pavel, just to set the context here, we've all seen films like Spike Jonze's Her. Right, where the AI has this incredibly realistic voice to the point where Joaquin Phoenix falls in love with it. But then I speak to Siri or I speak to Alexa and it still just sounds like a robot, right? I'm not going to fall in love with Siri or Alexa. They're pretty boring. Sorry.

KylieAnd there are thousands of people working on those voice assistants. And I really want to use today to get to the bottom of why they still sound like that. And what we're going to do to make them better. So maybe to kick us off, you could talk about the really early history of voice generation. Cause I, I think it goes pretty far back,

Pawełright? Yep. Yep. Just as with automatic speech recognition machine translation again, historically speaking, all started in, in deep 1950s. A lot of army projects, a lot of DARPA style like projects was about imitating human beings. And part of that was ability to create human like speech. You can trace back early experiments, even earlier to like 18th century, 19th century, where people physically were generating some sounds just like human just like normal instruments, right?

PawełThey were trying to mimic human human voices but as again, with machine translation or automatic speech recognition, IBM led. They're way in early fifties where it was creating first so called vocoders, which are actually still exist. And so part of all of these really nasty stuff that you hear sometimes since you're Alexa, but not long, we're going to get that to that problem soon. So, fifties is definitely like the early. Early stage of these systems, then 60 70s folks were working on improving that to the point that I think it's worth mentioning here.

PawełWhich is the space. So to say, right? 2001, where that was highlighted quite visibly.

KylieYeah, I so apparently Arthur C. Clarke when he was writing 2001 he visited Bell Labs and he saw a demonstration of their vocoder and it was singing the song Daisy Bell, which is how 2001 ends. So he was so impressed and wrote that in, which I think is really cool. Really interesting. You mentioned vocoder then, right? And so vocoder goes a really long way back. I remember a few months ago you wrote me a really great blog post about scaling the universal vocoder. And like, what has changed?

KylieAre you telling me that you're using the same technology now? That they were using in the

Pawełfifties. tHe early experiments and the early models that are almost can be traced to late 2000s. Essentially they're everything they're evolved around the vocoder, the early idea of the vocoder, where you had the units of speech, just like with automatic speech recognition. We call them phonemes. That are our heuristics to to simplify the problem of of human human voice where you have parts of the speech that we need to alter.

PawełAnd the whole job of a quarter was to concatenate these parts of the speech, these phonemes and make the sounds as similar as possible to the human beings. Now once, once you define the tool like that, immediately, you're going to run into the problems. Because you're concatenating all of the words, all of the sentences in like small units and you're just trying them concatenate them together. Now, by definition this will not sound natural and you would have a four years of a lot of work, a lot of PhDs, a lot of papers, a lot of even companies trying to make this sound a little bit better to the point that, you had these early Google speech synthesis or Siri or Alexa that.

PawełGot us to the point that it was intelligible. You clearly you've, you understood when Sierra was telling you about the address or direction, but definitely wasn't pleasing. Right. So this early stage of all coders was essentially getting the phonemes concatenating together and creating the audio out of it. Then as always. Neural networks comes in, right? Come in and they a flip the table, right? As you can observe in automatic speech recognition around 2010, 2012, with speech synthesis, it was a little bit better later around 2015, you started to see early traces of people start Applying deep learning to Texas synthesis as previously called the whole field.

PawełNow it's moving into the voice generation nomenclature where we are. giving the neural network freedom to generate the whole word or even the whole sentence at once, right? And by definition, that will introduce a great leap in terms of the in terms of the quality. So when I, Wrote about the Vocoder in 2023. It's slightly different Vocoder than it was 2000 in one, 1000, 1,970 in two, in 2001 Space Auto say, there's so many numbers of getting lost into it, But what's the most important bit that since last couple of years, we stopped chunking the speech and we start generating the full sentence and the ones

Kylieright. That makes sense. But I still feel like. In a lot of applications, because of the way the speech is being concatenated, you, it still doesn't sound like you would say a sentence, right? So I would say like Friday the 13th. But when you hear these systems, they say Friday the 13th. Like, why does it still sound like that?

PawełGot you. Part of the issues that a lot of systems still exist and work on old systems that not really using deep learning, even in 2023. So, that could be 1 of the issues when you hear it on the bus or in the underground. I'm sure that probably the code goes back into times where we weren't even born. So that will be swept away. I'm telling you in a couple of months. Cause that's how easy it will be to implement these systems. The other the other problem, if the system already works on the deep learning architectures are coming from the lack of context.

PawełAnd here we go back to the, everything that stands for, which is. It's really hard. If you think about if you think if you were asked actually to mimic somebody's voice and out of the blue, you're required to say specific sentence, which is announced that the bus is coming at at 1 o'clock. And you're sorry for delays, right? Without knowing. What country you're in how long already the bus was delayed, right? All sorts of the questions that are part of the context, you would be just doing a guessing game, how you should utter the census.

PawełAnd that's what the models are doing these days. They do not have that context. So they average out all possible, things all possible ways to say that particular sentence and just making the best guess.

KylieYeah that's really interesting. We so obviously at Poly AI currently, like as of, this moment, we're using a combination, right. Of synthesized voices and voice actors, where we are doing a bit of concatenation to put those together. But when we first started working with voice actors, I remember that. They would record, they would see the script of what we wanted them to record and they would say it in that IVR voice. They'd be like, thank you for calling the company name. How can I help? And it's taken a lot, even just to train people that, that isn't like the most natural way to sound.

KylieIt just, it seems very neutral. It's like we almost learned from machines how professionals should talk. And now we're trying to undo that and make it. People that again, right, really natural.

PawełSo there is, there's a lot of. Issues they're not visible to, to people who are not practicing the deep learning field. And one of them is very trivial. Something that is obvious when you're thinking about large language models or ASR is the amount of data. So text to speech voice generation historically has been deprived of that data. First of all people tend to You know private close also mentally when it comes to like property of their voice, right?

PawełThat's a part of your identity. So, most of folks and I'm including myself would be pretty reluctant, right? If you would tell them that, give me your voice for a couple of dollars and you're going to be, we're going to copy that. And you're going to be heard all over the world, right? That's something that. Okay. It's not as easy as telling you that record your speech and we were trying to understand you, right? That's whole different spectrum. So because of these because of these reasons, we've, while the models in 2022 or automatic speech recognition models were trained on a hundred thousand of hours, the state of the art models, either in production or in research and voice generation was, were trained on 500 hours at best.

PawełSo you see the huge difference. Right. And as we know and probably kids are being taught this in kindergarten these days, right? Neural networks need more data, right? So only last couple of months, and I can see it if you like open up our research research forums and boards you can start to see the models that we're able to train from vast majority of the internet data. To mimic conversational conversational way of human talking to each other, like podcasts or YouTube videos.

PawełAnd historically, it was really hard for us to train with that noisy data, the models that were that were asked to generate the voice that is not noisy. Right. So you have this issue where I'm giving you as an input, very noisy conversational data, like Paul's cast or data, like, like we are having right now. And then you're asked to generate something very clean, right? That's by definition, crazy confusions. And for a lot of years, we had just failed with with training with this data and that left us with some voice recording of actors.

PawełAs I told you, 500 hours of data, that was our typical benchmarks and you see the wave of new research coming in last couple of months. I would say a year highlighting with volley for Microsoft. I think that was 1 of the models that really hit the news where you could basically clone your voice with 5 seconds of your right input data that really blew people away. But what was. What was the most important part of that model wasn't any new architecture.

PawełIt was just classic architecture that we use with large language models, but they train on 60, 000 hours of data as a base model. So you see the, the jump you see the jump in terms of amount of data that they had and hence the quality. Yep.

KylieYou, you were speaking earlier about context as well, which I think. You speak very differently. I'm thinking of an example of one of our deployments where it's for a restaurant group and it answers the phone. Like you'd expect a person to answer the phone. Hello. And how do you get that context into voice data? Like it's so, the context of being able to understand is one thing, but then adding on that other layer of emotional cues, how you change the way the voice is inflected and what that actually sounds like.

KylieYeah, it feels quite

Pawełdifficult to define. I appreciate that you're asking the questions that I was waiting since the beginning of that podcast, because that's one of the most exciting parts of my day to day work, which is essentially contextualizing text to speech and voice generation, which you will observe. And you can get some hints from from from the demos that you see online or even like project products from some companies is that the wave of conversational voice generation is coming.

PawełAnd the way we're going to do it is basically adapting the architectures that we have for a single speaker, where you were given a text right to say and some context of this this speaker voice. And we will transform it in a conversational setup where the training data to put it natively will be two blocks. One is person one a per, and the second block will be person B. And essentially the model will see. The turn before your answer, right?

PawełSo you can see straight away that in terms of the text generation in the example that you were given, if I'm going to call a restaurant and I'll say, hi, how's it going on? I'm Pavel. I'm looking for a table for tonight, right? This will be all inputs to the mobile. Along with all of the data that the model had about the voice that it needs to generate. So this combined will give you the full information that you need in order to say, hello, Pavel, how are you? How can I help you? Unfortunately, we don't have any tables tonight, right? Which is which is completely different setup that we have right now, which is just a text.

PawełUnfortunately, we don't have a table tonight that will be set in a flat average voice. And. As I was, stitching puzzles together, if you think about what's happening right now and which we'll observe soon in, in the production and that we are working as well in poly AI is that we start to have conversationally data, people love podcasts. You're a good example of that. So there is more and more data out there where two people converse or even more. We learn how to train models from that noisy environment. We have architectures, which are deep neural networks.

PawełThey are so called transformer. When you put these things together, right, magic will happen. You can already see some demos out there. I'm not gonna name some of the big companies doing that. Where you start seeing this early traces of just mind blowing mind blowing conversational examples. And you soon going to be probably living in a reality where you might call your mom and she might be busy cooking and should that will be like AI assistant answering all of your all of your questions. I hopefully that don't got to be the case, I'm expecting my mom to do that.

KylieYeah, that's so creepy. It's surely when you have that like huge pot of data, like if you combine my voice with your voice, you're going to get a really weird voice. Yep.

PawełSo

Kyliewhat happens when you have that, that like really big data set, how does it average out because it's so personal, right? Voice is so, it's so much more personal than text or anything

Pawełelse. Got it. So from the modeling perspective. The model the models that we are training, they always have this part that that are called speaker embeddings and as an input to the model, except for the text or the context of the conversation, you're also giving a a sample of your speech and. This will be this will be transformed into so called vector embedding. So this factor of numbers that will specify that I want to generate voice with that specific or specific qualities.

PawełAnd because throughout the training the model observed, I don't know, 100, 000 speakers it will learn that given these speaker embedding thinking about female voice. Age from 20 to 30 having a, let's say British or Welsh accent let's be diplomatic here. And this signal would be way different than a male 20 to 40, right. Having some some non native accent and giving this embedding, the model will learn and know how to generate the right, right quality of the voice.

KylieAnd then in terms of responding emotionally, you did talk about this a bit earlier, but I you mentioned that it would be looking at the previous turn in the conversation. So if I'm really angry, if I sound angry, it will be able to adjust its voice to be maybe more accommodating. Whereas if I'm excited, it can be excited with me. Yes,

PawełThat's that's already happening. You have models that Given the sample of your voice that are it's happy. It will copy that happiness now that's a little bit different that what we really want which is contextually, contextually smart, right? So if the, my interlocutor is happy, then I should be happy as well. Right. Rather than just artificially like putting some happy input and getting happy output, because you can imagine really bad. Experience in terms of, I don't know poly AI deployments where, somebody said something really sad, but we put some happy sample, right.

PawełAnd you have this confusion. And then, the person on the phone will be like, what the hell. Right. So, so we're not there yet, but it's clear that it's coming, that the true true contextuality will happen. And it's, I would say it's just a matter of a couple of months.

KylieYeah, that's really exciting. I think it opens a lot of doors when you think about the context of mirroring in psychology, where perhaps the voice assistant can adapt to the speaker's tone. And that is something that's meant to put the other person in the conversation at ease. Right? Yes. Which will be Very interesting to see how that's applied in like sales situations and other situations where maybe it's a little bit ethically blurry, but should be interesting to watch. A

Pawełhundred percent, a hundred percent. I think it would be worth of us adding later on under, on the website of our podcasts, maybe some samples that we're already having in live productions, right. With this generative voices through already mind blowing. And I'm really excited. What's going to happen. We know when we're going to increase our neural networks. Add more data, add more context. It's just going to get better, which means better UX for the customers.

KylieYeah, I think we've, so we've both been at Poly AI for like five years now. Right. And it, it really feels to me that the last year so much more has changed than had changed in those four years before that. Right. Like we, we didn't have a choice, but to use voice actors because synthesized voices just were not good. And then all of a sudden, wow, synthesized voices are pretty good now. Is that all because of generative?

PawełYes. Yeah. It's the wave of generative AI. That's a simple answer, but also the true answer. It was just like the stepping point of increasing our knowledge, how neural net data is important. And as I told you, right, we had a lot of. Fights as a research community with this noisy data that we have we've overcome with better architecture, our architectures that allows us to like make this, the sleep that, that you can hear and experience, when you call our production systems.

KylieYeah, cool. And it's not just the voice itself, right? Like we've spoken about the way that the voice sounds and voice generation, but also the actual words. That are being generated allows you to add that extra layer of conversationalness to the conversation, right? Because at the moment using the intent based models, you're pulling a pre written utterance back and you have to just keep repeating it or But a generative model can make things up on the fly in the way that a person

Pawełwould. Absolutely. So as you see, like this whole technology stacks out things up together into a really nice puzzle. So, starting from automatic speech recognition, better and better understanding. The conceptuality of the answers with large language models, where, as you said, it's not about retrieving the best answer, but really generating every word as close and as specific that you need to be given the question of the user, and then given that highly adapted answer to specific problem, you've going to generate highly a contextual speech that will.

PawełThat will truly that will truly makes the whole UX much better. Cause let's be honest no matter how smart my ASR system will be, no matter how big and smart my large language model will be, a lot of times I got to. They in the last couple of years, like most of people were saying that the system is dumb or smart, depending on the voice quality. It's a huge factor. It's, I would say by far the biggest and in terms of the, how people receive, receive and feel. When they're talked to the system, so, that all finally comes together and I'm super excited.

PawełWhere are things going to go? Even like going a couple of years ahead excuse me for the research anecdotes, but you can already see some people trying and you're actively discussing this. In our community of building models that will be truly speech to speech on Facebook, sir, for example created literally a couple of weeks ago, both data set and first models that I got to say suck, but they will, I'm sure they will be good at some point. Where essentially we're going to transform from you talking to me and the system will be outputting in one go with the speech rather than dividing into like simple simple, not simple, but specific models.

PawełAnd that will be like another step into into Improving the speed and quality of a role, but that's, that's the story of the future, but you see how the whole field is just moving on fast forward speed.

KylieIs speech to speech then going to kill like voice biometrics?

PawełI would say so.

KylieYeah. If I can just generate your voice.

PawełYeah. Yeah. That's one thing. The other thing is that, it might kill large language models. If you think about it, it was like, why do we care about text in some applications? If we care about speech, like, imagine like you're driving, right? You should never be looking at the text or like trying to write some text. It will be, it should always be like just speech to speech interactions. It's really hard to foresee what's going to happen in the future given the speed ups, but I can tell you one thing the quality of it, it's going so it's improving so much that I would say by 2024, by the end of 2024, you're going to have a really hard time telling whether you're talking to a human being or not.

KylieYeah, I think that's going to open up a lot of very interesting questions that I'm sure we'll keep covering on the podcast. So, I think we'll wrap it up there. You can listen to previous episodes of the podcast on our YouTube channel or wherever you get your podcasts. And you can find us on LinkedIn for all of our latest news and updates. So once again, Pavel, thank you so much for joining me today, taking the time to explain. It's very complicated, but very interesting technology. And until next time, goodbye.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. What if your contact center became a product team?

    Nov 17, 202635 minAudio

    Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...

    Credit the people on this episode

  2. What happens if your AI agents get 1% better every day?

    Sep 10, 202639 minAudio

    An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...

    Credit the people on this episode

  3. Who's coordinating your army of AI agents?

    Sep 3, 202644 minAudio

    Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...

    Credit the people on this episode

  4. Why should CX leaders care about MCP?

    Aug 20, 202636 minAudio

    When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...

    Credit the people on this episode

  5. Can AI really hear a call the way a person does?

    Aug 6, 202622 minAudio

    PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...

    Credit the people on this episode