Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Deep Learning with PolyAI

Deep Learning with PolyAI artwork

072: Introducing Owl: PolyAI's in-house speech recognition model

May 23, 202522 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Guest host Meghan Berton and VP of Engineering Razvan Kusztos cover the release of PolyAI's Owl ASR model, while dishing goss on the intricacies of developing high-performing speech recognition models. They discuss the significance of in-house SLU tech, the challenges of ensuring accuracy across different accents and environments, and the benefits of using synthetic data. The conversation also covers the cultural nuances in text to speech models and how Poly AI leverages partnerships with companies like NVIDIA for enhanced model performance. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

MeghanFor example, we learned with some French clients that the text to speech model, the actual synthetic voice, it needs to be a lot faster, like the cadence. People just, it needs to speak faster, to be more natural, and we need to interrupt. Over a longer time span, whereas some cultural norms are a little bit different. Hi everyone. Welcome back to another episode of Deep Learning with Poly ai.

MeghanMy name's Megan Burton and I'm one of the group product managers here at Poly AI working on some of our technical products and machine learning models. And I'm here with Zen. Would you like to introduce yourself? I.

RazvanSure. Hi everyone. My name Poly. Close to being seven years now.

MeghanAncient, just like owls.

Razvandinosaur.

MeghanWe're here actually to talk about owl, which is our automatic speech recognition model. Tell us a little bit about how we named it Owl. Could you maybe prep the audience or prime the audience with what speech recognition models are? A little bit about how they work.

RazvanCool. Yeah, I guess I'll start with the speech recognition part. As we build voice assistance, a natural first step to, ingesting the audio, the normal conversation that happens over phone is to transcribe it into text. So turning audio into text, that's what speech recognition is. A nutshell, obviously understanding when a human speaking and when they're not which of the participants is speaking at a time, which is called Diarization. And and just transcribing accurately, representing the words into text as accurately as possible Now about the owl that's a fun name.

RazvanWe're keeping a bit of a bird theme going on with our releases now for all these foundational models. I think that started pretty early on in poly ai. I think most of our projects were named with birds. We had Finch, we had, we had peacock, we had swan talking about, our first voice agents with in different sectors. So I think it's quite a nice and nice going back to our roots. Yeah, funny seeing like

MeghanKeeping the tradition alive.

RazvanYeah, for sure. The birth theme does come up a lot in the industry. Actually. We saw that NVIDIA's models are also a version with birds, so it's I feel like a lot of people are on the same page here, just like birds.

MeghanSo I wanna touch on a little bit more on how speech recognition models work. To give a little bit more context, so you mentioned like this is the first piece of the tech stack conversation. AI assistance, they can only respond, their output can only be as good as the input they're receiving. My assumption would be I guess I know'cause we work on the same team, but there's a lot of challenges with these models. Like it's a very tricky problem to solve. Could you maybe explain a little bit about why this is such a hard problem to solve?

MeghanI.

RazvanYeah, for sure. I think I think the problem comes from many different facets here. I think when you're building a conversational assistant, the first thing you might think about is, is latency and naturalness and responsiveness and stuff like that. For that. Purpose. You'd need quite a small model. So you can't use a huge model and a huge GPU with billions and billions of parameters. Just, this is gonna be too slow. We have very we cover a lot of area in terms of use case and geography.

RazvanSo there's many accents, there's many speech patterns. The hardware's different. Phone lines are extremely noisy. we have use cases when people talk directly to kiosks. For example, there might be a car engine in the background. People might be calling

MeghanSomeone's television in the background.

RazvanYeah, absolutely. And it's just I think that's just one of the, one of the reasons for for this being a tricky problem. And then obviously the important part which is, the aspect of form filling that we do, collecting names, addresses so we can actually execute business logic on behalf of our users. Accuracy is paramount. A digit off can break a whole address a digit in a postcode. Break breaks the whole identity of that person so they can't get on with their with their requirement as well.

MeghanSo performance, latency, interruptions, background noise, accents, and many more things. How does having an in-house model, our OWL model, how does that set us up to have a. Performance and to do better on these things than if we were to just adopt third party models.

RazvanYeah, for sure. Having an in-house model has really changed the game for us and I think that's where, our partnership with NVIDIA has come up and we're, we've been able to deploy these models on Nvidia Reva, and we have access to these. Super state of the art models that we can fine tune ourselves. I think Having an in-house model really shines when you first of all, we can guarantee to our users that our versions are fixed with third party APIs. They can change the version at any time. They can break. whole piece of the business logic.

RazvanIf a certain, I don't know, certain words, all of a sudden stops being transcribed like that, that, that company's name, that, and then you break their whole use case. So we, we have this very good versioning that is probably not as interesting, but it's really something that keeps coming up again and again. And we've really gotten rid of that issue with having an in-house model. Second of all, we it's, it really helps for compliance. So again, keeping our customer's data safe we don't need to send our data to third party providers to fine tune we can do that in-house.

RazvanThe data never leaves our environment. So it's it's it's safe and very good for compliance. And, and keeping all the pi in one place. Yeah. And lastly, yeah lastly I was gonna mention latency as well. You are not going over to a different cloud provider that might have God knows how many hops before you get to the actual servers. So we can optimize this this infrastructure in a way that really shines in a natural conversation with our agents.

MeghanAnd maybe something that would be helpful to give more insight into is how we improve performance of our in-house model, particularly'cause, our client base stretches across so many different regions, accents languages. Like how do we approach fine tuning and improving the model so that we can do better on these specific use cases.

RazvanYeah, for sure. I think what's become very apparent in the last few years in the, on the machine learning front is that data is, and data is the most important part of your of your sort of machine learning improvement lifecycle. You need to have data that really targets the exact. Distribution of what people might be talking about in this, in these phone calls. That's why it's really important that we can use our existing data that we've collected from our from our live conversations to fine tune. That definitely boosts our performance substantially.

RazvanThen again, with audio, getting high quality data is quite tricky. You need to in order to train these machine learning models, you need annotated data. So that means pairs of the audio and what was actually said and that audio. And if you could automate that, then. We solve our problems solved. You usually need a human in the loop here to make sure that data is transcribed correctly. And that obviously becomes an operational challenge when you have multiple languages and multiple accents as

MeghanYeah

Razvanso in, in order to

MeghanI'm sure.

Razvanyeah, done.

MeghanNo, I was gonna say, I'm sure our machine learning team would be very happy if we could automate some of this annotation. I think I. We annotate so much data that

Razvanfor sure. We do need to annotate that data for improvement and for evaluation as well. In, in order to really make a dent here, we need to get a bit more creative with how we source this data. So I think. As far as oh, also one thing that I forgot to mention is the the type of data that we can actually train on. We re, we're really reluctant to use PI and this training data just because that might put our users at risk. So we avoid that. And therefore we need samples that represent, addresses, names, unique identifiers, like tracking numbers driver's license numbers and so on.

RazvanSo in order to do that, we we leverage quite a bit of synthetic data generations. So that means that we can, we take the text and we use of the existing tope models. So the opposite problem of turning text into speech. And we can we can generate data sets that way. So you, I'm sure all familiar with this new technology. So you're like, you ask J-G-P-T-A generate, a hundred ways in which you can say a random name from this region.

RazvanAnd then pipe all those responses back into into a text to speech model. And they surround city data, so we go around it the other way around. An another really important tool here is distillation. Like I said, for our live traffic, our models need to be very nimble and swift like owls. But for annotating data, we can use some huge dinosaur birds here. We a exactly. So

MeghanTyrannosaurus.

Razvanwe I think we've had quite a bit of success with Canary actually, which is the Nvidia Riva model that we use for

MeghanOkay.

RazvanIt's pretty slow. latencies are, seconds in magnitude here. But we can get like huge huge samples of conversations, pass them into this into this bigger model. It's not gonna be perfect, but it will be better than a smaller model. So then we can use this data to further improve it. and then we're always on the lookout for open source data sets as well. That might match our use case. As well. It's, that's the most most important thing is having a good good data set.

RazvanAnd I think so far I've only been talking about training evaluations always. I think the trickier challenge here as you release new models and especially in our multi-talented architecture it's really difficult to to, bias the evaluation set in the wrong way. So say if you have a bigger client and a smaller client you have more evaluation points for the bigger client, then you might not see a regression for the smaller client. So that's why it's really important to have have a very good biased evaluation set.

RazvanI've mentioned the client dimension here, but we're also talking about extracting names, extracting addresses, all these like smaller tasks. That are implicit in the model. Say if you're only training on address data, you might get worse at phone data

Meghanso you've mentioned a lot of different ways that we aggregate and pull data for evaluations and then also for fine tuning. I'm curious if. We could go a little bit more into the synthetic data side. You mentioned something about having a broad distribution of data. Are we talking about a distribution of data? On the text side? We're synthetically generating and also the audio side. Because I would assume with text we want lots of different ways people can say things, lots of different words. But then on the audio side, we also wanna capture accents and different ways people say things.

MeghanSo could you explain a little bit more about how we're curating a data set with synthetic data? Why is that better than just doing the annotations ourselves? Or how is it different?

RazvanI think. This distri the distribution of text, I think that's an easier problem to solve. For example, we

MeghanAll right.

RazvanWhich particular terms our clients are unhappy about because they can annotate them on Agent Studio and then we can generate

MeghanAgent Studios are a platform.

RazvanYeah, and thanks for that for your guys know. And and since users done, interact with the conversation history and they can be like, this turn is bad. This turn is bad. We can see where that turn happens in the conversation. We can generate turns like that and all of a sudden we have a surgically designed data set that will improve improve

MeghanSo it's very targeted.

RazvanVery targeted. Yeah, exactly. And if you do that over and over again, you're gonna, you're gonna, even now this the surface of where your model performs. Course, over time. On the audio front, I think that becomes much trickier with In particular. Already all these platforms that provide text to speech services allow allow zero shot cl cloning. So you can provide a snippet of a certain type of speech and then you will try to point something that looks, sounds similar to that.

RazvanThat's a way in which you can generate different accents. I don't think we've had great results with that. what really did improve our performance is adding background noise, adding TVs, adding crying children in the background and all these techniques to like from one single sample you can generate. You can generate 10 or more of, here's a sample with a car engine in the background. Here's a windy sample and so on. And that really creates some sort of resilience to these external environmental factors.

MeghanBenefit of being able to have our models in-house, right? Because third party models, they're not actually fine tuned or improved to take these different things into account. There's lots of different use cases for speech recognition models, conversational ai. It's pretty unique in what it needs

RazvanI think if you start playing around with third party providers, you'll see that. The use cases are so broad YouTube needs a speech recognition model to

Meghanaudio books.

Razvanaudio books. Yes. And so there's so many use cases. I think this this really breaks a bit when you get into latency interruption rates and dealing with a realtime use case. And there's a lot of there's a lot of trade offs we need to make with these models. I don't think, I don't think we can get the same level of faithfulness to what a human-like conversation looks like with the off shelf models.

MeghanCould you explain a little bit about this trade off and help us understand why this trade off is so tricky between interruption rate and latency for these models?

RazvanYeah, for sure. I think when you talk to someone, and especially face to face, there's so many signals. You,

MeghanI.

RazvanThe two parties give each other. Now it's my turn to speak. Now I'm signaling that it's your turn to speak. You might say Uhhuh. Yeah, I'm following you. You might give me an encouragement to continue speaking. And

MeghanI might scoff and give you dis encouragement.

Razvanyeah.

MeghanIs that a word?

Razvanknow too. Yeah, I think so. Yeah. Dis encouragement, discouragement.

Meghangoing. Sorry.

RazvanYeah. So you might, I, I think we've just done

Meghanit is.

Razvannow is, I think it's a very good example of people know exactly when to interrupt each other when it's polite to do the model does not have the same sort of social cultural awareness. What usually happens in these models, there's a window of silence which we refer to as the VAD window, a voice activity detection window in which we we just, the model waits for a certain. Number of milliseconds for silence before it actually processes the speech. When you're building this ensemble of, here's a speech to text model, then there's an LLM model, which are also quite expensive.

RazvanThen there's all these expensive in terms of in terms of latency. Then there's all these other functions I need to run, like an API request to update a booking and so on. And on the background, this might actually increase. Your your perceived latency. So how much a user actually waits before they hear something back from the agent. So that's why it's quite a tricky balance here to figure out when a user is done talking versus when a user might. Say something in their thinking. Say you've asked them for a your, you have asked them for a ID number.

RazvanThey need to be like, hang on a sec, I'm just checking. I just need to take it outta my wallet and read it out to you. So all these nuances of how people interact. They, I think we've seen experimentally, they themselves to be on a curve. If you map interruption rate and latency from what we've seen is if your model is very snappy and it cuts users off, then you have a high interruption rate. And on the other side you

MeghanRight.

Razvanlenient, but then you're just like, Hey, are you still here? Are you still talking to me? On the hand. I think

MeghanYeah.

Razvanthe that's a very interesting curve that we track all the time. And we're obviously trying to minimize both, but we found that there's a trade off, there's a trade off to be made. And I think the industry's not I don't think this is a solved problem. And there's no API out there that says API that says. Conversational, VAD turn on in, in, in all these, all of these providers. I don't think that's a that's a thing yet. So I'm quite excited to see how we're gonna progress with this. There's there's loads of innovative ways we been doing.

RazvanSo like we've been using, so say a user speaks for a bit and we just start processing that partial response. And then if they decide to continue speaking, then we count it. But if they don't, we have something ready to say. So we cut latency that way. We're also very excited about the future with audio LMS and what they allow, where you can actually use a, maybe integrate this cultural element of of how people interact that's encoded in the large language model.

MeghanBe a lot more contextual. I like what you said, calling out a little bit like the cultural. Differences. For example, we learned with some French clients that the text to speech model, the actual synthetic voice, it needs to be a lot faster, like the cadence. People just, it needs to speak faster, to be more natural, and we need to interrupt. Over a longer time span, whereas some cultural norms are a little bit different. So I think there's this balance of we wanna improve all these models generally, but we also have analytics and we have to make projects specific and client specific changes and implementations, which is nice as well.

MeghanCan you talk about a bit how we balance general model improvements versus what we're doing to make sure. Every model is the best for the individual clients, particularly speaking to and a SR, like the project specific piece.

RazvanYeah, sure. I think, it's, again, it's a combination of many factors that I think we've touched on before. So it's building a strong evaluation data set, building fineable data set. And then targeting those areas that really need our attention by using client feedback on the agent studio. I think those are I think critical but also having di open looking at what new models coming up, we can leverage them, what new solutions are there. Like I mentioned, audio, audio, lms so LLMs that take audio as input they have huge they can have huge performance.

RazvanBoost for us. However, they're too slow at the moment. So I think our colleague Matt Henderson posted on on LinkedIn a while back, a graph showing showing difference

Meghangood plug.

RazvanG pt. Yeah. I'm sure he's happy about the plug. Yeah, we keep, really helping us here is our partnership with Nvidia as well. We have access to their latest, greatest models and their inference servers. That really allows us to, to scale this scale, this process. I hope I answered

MeghanSo

Razvanhere. I'm not sure

Meghanyou did. You did.

RazvanYeah.

MeghanYeah. No, that was a great response. I think it's like it's. Something as well, which is I think where like the managed service piece comes in and how we work with our clients in partnership is it's not just appro improving things across the board or why you'd look at the a SR hugging face leaderboard and see the best models. It's also about making sure we're doing all these best implementations for our clients and the tech stack is actually. Quite complex, like VAD, all these different things you mentioned, they all add up together and so we have to make sure it works best for clients.

MeghanSo I don't wanna steal the final wrap, but how should we close this wonderful episode of the Deep Learning Podcast? You've mentioned a lot of reasons why our in-house investment and speech recognition is really amazing for clients. Is there anything you wanna add? Emphasize, or is there anything you wanna say as we close about looking ahead towards the.

RazvanI think, yeah, I. Speak for themselves and like how much, how far we've gone on this on the, on this journey of improving the naturalness and accuracy of conversation. Then, stay tuned. I hope we're gonna be able to post way more graphs showing how owl's performing better than than, similar technologies out there from other vendors and wide data that we, we have inhouse is so valuable for this use case. Really excited about, again, like I said, about future technologies integrating audios into our tech stack as well.

RazvanIt's definitely something that we're actively looking.

MeghanAmazing. It was such a pleasure to speak with you. And hopefully everyone enjoyed the episode and I have to close with our team is hiring. Apologies if I'm not supposed to plug that. But follow Rasin because the team is hiring and if you wanna work on some of these problems, we wanna work with you too.

Razvanyeah, for sure. This is, This is an exciting time to be building Components,

Meghanokay. Thank you everyone.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. What if your contact center became a product team?

    Nov 17, 202635 minAudio

    Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...

    Credit the people on this episode

  2. What happens if your AI agents get 1% better every day?

    Sep 10, 202639 minAudio

    An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...

    Credit the people on this episode

  3. Who's coordinating your army of AI agents?

    Sep 3, 202644 minAudio

    Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...

    Credit the people on this episode

  4. Why should CX leaders care about MCP?

    Aug 20, 202636 minAudio

    When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...

    Credit the people on this episode

  5. Can AI really hear a call the way a person does?

    Aug 6, 202622 minAudio

    PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...

    Credit the people on this episode