In this episode of the PolyAI podcast, host Kylie Whitehead interviews Paweł Budzianowski, the head of machine learning at PolyAI. They dive into the past 70 years of speech recognition, moving from hidden Markov models to deep learning transformations and beyond. Plus, Seinfeld and Jordan Peele in the speech zeitgeist? 00:43 The Importance of Voice Recognition 01:20 The History of Speech Recognition 04:18 The Role of AI in Speech Recognition 10:35 The Evolution of Speech Recognition in the 2010s 13:54 The Future of Speech Recognition 20:17 The Challenges of Accents in Speech Recognition Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
PawelIf I'm going to say check in when I'm calling the hotel is a little bit higher probability that you say check in instead of chicken. tHat type of stuff actually happens and you might see the best air service system that's going to go for chicken without check in because it just doesn't have that context. Right?
KylieHi, I'm Kylie Whitehead. Welcome to the PolyAI podcast. I'm joined today by Pavel Budzianowski. Hi, Pavel.
PawelHey, how's it going on?
KyliePavel is the head of machine learning at PolyAI, and he also teaches an NLP course for machine learning master's programs at Warsaw University and Jagiellonian University. Pavel has a bachelor's degree in maths PhD at the Dialogue Systems Group at Cambridge University, working on conversational agents. He's published 40 papers on dialogue systems, and his work's been cited over 2, 000 times. So, who better to cover our topic today, which is voice recognition? And great voice or speech recognition is absolutely crucial to creating a usable voice assistant, but we've all had...
Kyliea bad experience to the point where it's kind of a meme now and I'm thinking of a scene, I don't know if you've seen the Jordan Peele film, Us. Yeah. And somebody's shouting, Alexa, call the police. And it starts playing the NWA song and just carnage ensues. So Pavel, I'd love to understand a little bit more about how we got to this point. So tell me. Engineers have been working on speech recognition for a really long time, right? Yep. Yep. It's been a, it's been a long journey. It's been over, I would say 70 years since we as a computer scientist are looking into this problem.
KylieObvious applications all over the world in every appliance, right? We would like to talk to the to the appliance rather than type it whatever comments we would need to do. But funny enough one of the biggest starting point for speech recognition was actually things that were happening after the second world war and the Cold War between U S A and Soviet union and where scientists basically were tasked to figure it out how to spy on, on, on phone calls or like just emails, that's where speech recognition and machine translation.
KylieAnd that's where we started basically trying to figure it out, how to recognize speech how to recognize speech and multiple languages and since fifties there's been variety of different approaches and over the years, you see all of this all of them in in, in the generation of appliances, including all these funny issues that we had up until I would say 2017, 2018, I would say I would start from one of the most memorable part where IBM released Showbox it's been already seven years, which is, it's crazy how early actually that type of appliance existed there where essentially they created a mini computer that could recognize numbers from zero to nine.
KylieIt actually worked quite okay, and to a lot of people were like first signs of something out of this world. A lot of people were calling it fake. They were basically checking whether there's a human being behind the stage who is recognizing this. But it actually it was working. There was really little AI, but it was a lot of signal processing, a lot of signal recognition because the way we say zero versus nine is much different, right? So the wavelength is much, much different. And thanks to that they could they could understand what's going on there.
KylieBut as you can imagine, communicating with machines through zero through numbers and simple comments is not ideal UX that you can imagine. Especially if you're calling, if you're calling you police, or you need really big help because your internet doesn't work right in your room. That's why we moved to much more complex models. And that AI started coming in. And disguise of the statistical models and here is like a really sophisticated name that pops on the stage, which is called hidden Markov models.
KylieNot going into much of a details because there is a lot of mats that unfortunately is not really needed anymore because we have neural networks that we're going to get to pretty soon, right? Hidden Markov models were first statistical models that we're Very very good at explicitly modeling parts of speech that we say, which we call phonemes or small units of speech that compose that put together, compose words and the whole sentences and essentially since 60s, 70s, All of these models that were behind speech recognition systems were using these hidden Markov models to properly model these phonemes.
KylieThere is around 40, 50, 60 depend, depending how you're going to define them. And essentially the model was trained to properly divide the the wavelength that it was, that was getting. Into these phones and these models were again, a revolutionary because you can go above just digits and simple comments into, in theory, full sentences and in 70s in the 80s. That was a lot of progress in terms of recognition.
KylieAnd that. Led to famous first system, the first commercial system, Dragon, that was released as a product, as a standalone product for speech speech recognition use cases that basically paved the way to what we observe today. So the first version of Dragon, I was reading about this earlier, the first version of Dragon couldn't tell the different couldn't tell when you ended one word and started the next word. And it was. I saw that it was 1997 until we had like continuous speech recognition.
KylieSo that's 30 years ago. Why? Like, why is speech recognition still so bad?
PawelYeah. Since the release of the Dragon NaturallySpeaking, I think that's the one you were talking about in 1997, all of these systems up until 2017 were based on these hidden Markov models. And as With all of the AI models that were released before neural network, deep learning revolution that happened in the last 5 years and and with all of the outcomes that today, like chat GPT. All of these models learn from the data, which is a crucial aspects of AI, right?
PawelJust like humans, these systems learn from the data, but they had this visible invisible ceiling up until they were learning quite. Okay. And then. Later on, once you are adding more and more data, the system didn't really because it had this, the ceiling of capabilities and inherently inside of these models. So when you, when these models were released since 1997 a lot of work was put into tweaking them up, making them better. We got to the point where continuous speech, even with British accents Could be recognized quite easily and favorable condition, right?
PawelYeah, imagine no rain, no much of a noise. No, no other speakers in the room, just you with your with your good clear audio. That's where things were okay, actually. And these systems were deploying a production and they work well. At the moment, you put your voice through the phone very low resolution. There is going to be a lot of people talking around you. systems completely broke. And we couldn't really get passed through that because even though we were throwing more and more data we have internet era.
PawelPeople are sending mails, talking through Skype, you name it. We weren't improving as demand was improving and that has radically. around 2012, actually a little bit earlier than 2017, where we start to see one of the first magical revolution in AI that is somehow overlooked these days because of chat GPT and all of the all of the repercussion of it. But actually the first moment where everyone realized that there's something being coming in through deep learning and not neural networks were improvements that we did and speech recognition through applying deep learning rather than hidden Markov models.
PawelAnd that's where the defining moment that that basically changed the whole scene.
KylieThat's, that's interesting. Did you know that, so I guess a few years after that came out AOL movie phone introduced speech recognition. So you used to have to phone up and it would say press, if you know the movie you'd like to see, press one. And then there was that episode of Seinfeld where Kramer's line gets crossed with movie phone or he gets a number that is similar to movie phones and he ends up obviously can't understand the key, the button key press. So he says, why don't you just tell me what film you're, you want to see?
KylieAnd that was actually the inspiration behind movie phone using speech recognition. It's yeah how Cosmo Kramer invented speech recognition. Yeah really interesting. So I think you touched on this earlier around, we get to this point in 2012 where deep learning has progressed. And the 2010s were a really big time for this, right? Like a lot changed between 2010 and
Pawel2020. Yeah, quite a bit. Where the most important thing is that since people started calling more and more systems. They were more inclined to in, talk to variety of AIs. We had Siri since 2000, I think, 13, 14, if I'm correct, maybe even earlier. People they started to got to accustomed to that, right? These old IVR systems. Were a passé already back then, even though they still exist these days. I don't know why people should be using poly already everywhere, but essentially we started getting a lot of data.
PawelAnd as I said, there is a huge difference between hidden market models and deep learning and neural networks, because the more data that we have, the better. Deep learning based systems will will be essentially performed. And in 2010, 2012, for first research it popped out showing that with more and more data, we actually finally passed the bar of the performance of old type of models with these new models. That's when the history basically has changed because that was a sign that we found a way to train these big neural networks.
PawelWe talking about a million. Parameters right up until now, we were getting to the point where we have more and more parameters to the point that we have as many parameters in some networks as we might have neurons in the brain, which is crazy numbers and still these systems were able to train. And with more data, they were performing even better. So what you will observe if you be following the research papers that were popping up basically every month, it was like crazy race between different universities and also research labs is that every couple of months, there was a new basically record in terms of how good the systems are.
PawelAnd that, that led to the. I would say 2015, 16, 17, when even us back in the uni times the original founders of poly I, we could train systems like like that, we. Deep learning behind these systems that were completely destroying the previous models that required 200 people working on them, a massive amount of annotators, tweaking them up with just one one computer and a couple of gigabytes of transcribed data. And that basically led to the situation that you have these all the players like dragon you have Siri that, not going to understand you well Alexa came out, right?
PawelAnd that's where that was like this moment okay, let's change the situation, at least from our perspective.
KylieCool. And loads has changed since then, right? Like 2017. I know that you've been at Poly AI since the beginning. I wasn't too far behind you. But I, I can see a difference in what we've been able to achieve since 2017. So what, what has changed since then to allow? Companies like us to really hit this quality in terms
Pawelof speech recognition. Yeah, absolutely. One of the things that became a standard that it wasn't the case before 2017, that everybody was a building their own models, their own deep learning models. There were a lot of different ideas how to create these systems in terms of the structure of neural networks. You can think of different way of firing neurons in your brain, but since 2016, 2017, there became like a standard way of building these neural networks.
PawelAnd it was just a matter of right. Correct data. To improve these systems and at poly AI we initially were also doing quite a lot of Quite a lot of research how to build these type of systems. We were competing with different different order providers, but since we are getting more and more clients at different domains and insights about them, we were able to find a way to use Let's say off the shelf pre trained models that are available to basically every research group or every company and adapt to the exactly the way people talk.
PawelWe're going back to this, fanny anecdote that you mentioned before. It's all about the context. It's all about situation where people called, call on the phone, which is a completely different data type that, for example, you have when you're You know, having a podcast where we were explicitly, we're made sure that we're not going to have noisy room or people around us so that people who are watching us going to have the best experience, right? So here we cannot, of course request that from the consumers because they might call hotel restaurants.
PawelWe have so many domains these days that I'm getting lost, actually, in terms of the use cases, but essentially, thanks to the access to this data, we were we knew. What type of data we should throw out our model to make the model, being receptive to that as the ecosystem that typically is involved on the phone calls. And that's became a game changer when it comes to, when it comes to understanding what people really talk talk about through the phone rather than through the perfect environment when they when they, for example, do the podcast or just talk on YouTube.
PawelYeah,
KylieI love that, as you said about us both finding quiet places to record this podcast, like a bus went out, passed outside your window, I think, and it was like, you could hear the background noise, it was like perfect timing. The phone is such a strange even right now, we're looking at each other, you can see my mouth move, you can see my facial expressions, you can, even if you don't catch 100 percent of what I'm saying, you fill in the gaps, right? Yes, that's correct. And you don't have that luxury with the phone there's no visual, it is just the words that are being said, no body language, nothing.
KylieCan can speech recognition catch some of those? Yeah. Queues in like how people sound or.
PawelYeah. That's an amazing question. That's one of the actually research questions that I I'm still dreaming about because we are still not there yet. So let me answer with what we are doing this right now and what's going to be the ultimate goal of these models. Okay. At PolyEI we typically work with multiple with multiple ASR systems at once, in some cases, we are looking at and best lists. So our models are able to provide multiple hypothesis of what you say, as you said, when we were talking on the phone, right? I don't see the, I don't see your emotions.
PawelI don't see your lips. I don't know exactly what might have been said, because there's also a lot of errors over the line. So I might. Do the guessing game, right? If I'm going to say check in when I'm calling the hotel is a little bit higher probability that you say check in instead of chicken. tHat type of stuff actually happens and you might see the best air service system that's going to go for chicken without check in because it just doesn't have that context. And that's how we were, by combining a lot of hypothesis we were able to extract the right one to be used in the systems, but thinking about what's going to happen in the future when we're going to have more and more AI systems combined, the systems that understand pictures, the systems that understand audio, the systems that understand text, you can imagine building a SR system where you Tell the system what type of dialogues or, data that system should expect to get into.
PawelIf you think about understanding the the conversation over the phone line, when people are calling it restaurant, and the model be able to create conceptually, what is the restaurant? What type of sentences might be might be spoken in that scenario. You can imagine that all of these chicken hypothesis will go away because if I explain that the receptionist at the hotel, right? It's, it's almost 0 that you said you start talking about some anecdotes about chicken instead of actually asking about the chicken. Somebody could ask some questions around that, but, yeah, the probability is low that's basically the future that we going for.
PawelIt's clear that there is that's going to happen. There's already some research work. But but it's, we're essentially getting there. And once that will happen my prediction is that all of the last 1 percent errors that we are observing in production will go away. There are
Kylieinstances where, this is not quite the best example, but there are instances where perhaps somebody has an accent and they are saying chicken but they mean check in, but it sounds exactly like chicken. I think accents provide quite a problem, like quite a hurdle for speech recognition, right? Because, if I'm talking to somebody I, and I know. What accent to expect, I think it's easier to understand, whereas if you speak to somebody with an accent that you haven't really, that you're not really familiar with, it can be much more difficult to understand.
PaweltHat's absolutely right. And that's what that's why I mentioned before if we know, for example at PolyEye, that in one of our deployments, there might be coming people calling people from a lot of different geographical places with different accents. We typically we are using multiple ASR systems so that we can combine a lot of hypothesis together. And that's that's typically the best solution.
KylieAmazing. Thank you so much for talking me through all of that pebble. Sure. Do you wanna record another episode soon where we talk about voice generation?
PawelAbsolutely. Let's go on the other way around and let's talk about voice. Yeah.
KylieVery cool. Okay. So keep your eyes and ears peeled for that. You can listen to previous episodes of the Poly AI Podcast on our YouTube channel at Poly AI voice and you can find us on LinkedIn for all the latest news and updates. So once again, big thank you to our special guest. Thank you, pleasure. And until next time, take care.
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...