It's getting technical up in here! Kylie welcomes PolyAI CTO & Co-Founder Shawn Wen to the show to discuss the complexities of building high-quality voice AI systems. From the limitations of large language models (LLMs) to the nuances across each layer of the tech stack, they'll cover setting up audio channels, streaming speech recognition, voice activity detection, and the uniquely global challenge of handling different accents and languages. Shawn gives away a bit of our secret sauce, while underscoring the importance of adapting technology to user characteristics and enterprise demands. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
ShawnIt's super powerful. It's like nothing comparing to what we had previously, but there's still a lot of limited patients, right? one good analogy is like, you know, LM can get you to 70 percent of what you wanted to do very quickly. Then the last mile usually is like another, get to 95% You still need to put in a lot of work and a lot of effort. And then that's what our clients is demanding in terms of the quality of the part that we build.
KylieHello, welcome to deep learning with PolyAI. Today we're going to get a little bit more technical than usual. And the reason for that is recently Andreessen Horowitz published their AI Sorry, their thesis on AI voice agents, and in it, they give a pretty good overview of the AI tech stack, and it's a lot more than just LLMs, it's actually pretty complicated. I've asked Shawn Wen, co founder and CTO at PolyAI, to join us today. Hi Sean! And help us to get a better understanding of what's involved in making really good voice AI.
KylieSo Sean, just to start us off, for anybody who's thinking about building or buying a voice assistant, could you could you explain a little bit about what that tech stack should look like?
ShawnYeah, that makes sense. Yeah. Yeah, my name is Sean, co founder and CTO of Poly. Yeah, so in terms of buying or building voice AI, So basically, you have to first thinking about the channel, like if it's a phone call channel, which is like predominantly the context center business would run into, then you will have to think about how do you pipe the audio from your contact center. Into the voice assistant and usually the voice assistant should be living in the cloud. Sometimes you can do on-prem deployments as well, but predominantly it's in the cloud.
ShawnIf it's in the cloud, you, the most easy way is probably through the SIPP connection so that you know you will have an IPA address of the voice assistant and you can pipe the OO audio there. Or if your voice agent vendors also provide you to for the PSTN redirect, then they will host a phone number somewhere with a telco partner. And then you just forward that call into that number. That's another way. So once you get this connection, set it up, then You go into the real me meaty bit of the entire voice assistant because it's a voice assistant, right?
ShawnSo you technically need the voice bit, which is containing what we call the speech recognition or speech to text module. Basically transcribing your live audio into text. Usually this. Should be a streaming endpoint so that you will be able to transcribing the speech as soon as you get the O audio chunks rather than waiting for the entire audio to get back. A lot of times when the latency happens, this is like the number one thing that you should check, are you actually doing a streaming endpoint or are you actually just, waiting for the entire, or audio to be sent it to the clock.
ShawnSo that's the speech to text bit, and then When
Kylieyou say, Sean, when you say the streaming audio, do you mean that it, so it's waiting, it's, sorry, as the caller is speaking, it's transcribing it, rather than waiting for the caller to say. What they're done saying.
ShawnSo basically it means when you, when I say streaming architecture, that means that as soon as you get the first audio packet, you should just send it to the clock, right? So you have a streaming protocol that as users speaking over the phone, you don't need to wait for the entire user speech to be finished. To send the entire message into a cloud, you started saying, when I started to say hello, you say hello, peace already to a cloud. So the model has already recognized it so that by doing this kind of streaming architecture, you can actually minimize the latency and then, have a very good user experience upon it.
KylieThat makes sense. And I think it's, More difficult than I thought. I thought you just plug in the ASR, and then it just works, but you have to think about what is the right way.
ShawnThat's right. And
Kyliealso,
ShawnThere's a lot of pitfalls in choosing the speech recognition models for the purposes, despite that all the speech vendors will tell you that, they have the lowest award, right? Which is like usually how they measure a speech recognition systems performance, right? The problem is that this kind of speech models are trained to based on different kind of Data, right? So let's say if you have British accents, voice assistant, because you majority of your colors are actually with that accent, then a model that a trend based on us in English is probably not ideal, right?
ShawnSo there's a lots of these nuances like that. And for example, in UK, you also have a lot of regional accents as well. And those original accents, if it's not part of your training, data, which is pretty normally the case because majority of these speech recognition vendors actually serving us market predominantly, then you are going to struggle to recognizing some of the even basic, simple words like yes or no, or in Scotland, you will say I, right? So these are the problems you will run into.
ShawnAnd then, if you don't know the nuance of it, you don't know how the model is trend, then you are going to hit a lot of roadblocks there.
KylieDo you have to use multiple models then to serve all of your audience?
ShawnYeah. So basically what we do is, specifically for English, right? You know, I think if you are, serving different languages that you definitely need, different kinds of models, despite the fact that now the so called end to end speech recognition model becomes more and more popular, and then you will be able to train a multilingual ASR On top of one single model. This is what the way we whisper model from open AI is doing, they are still some challenges there because, like then in this case, they started to transcribe.
ShawnSome mixed code speeches, so for example, like maybe they will transfer, the full English sentence into English to Mandarin mixed the text and that caused some challenges as well. And also these models, I think we are still at an early stage of those models because the latency and the compute intensiveness of those models are. Still an engineering challenge to become, to be overcome, but I think it's becoming more, but a lot of the production systems you are seeing these days are basically tailored to that particular regional accents or particular language per se.
ShawnFor example, in our deployments, we have British deployments. We have the US deployments, which also align with the data privacy quite well, because you need to. Keep the user data in that particular region anyway, and then you deploy different models for that purpose. And for the, for each of those regions, we actually have a lot of these kind of user data that is available to us, and then we will be able to continue to improve that model in that particular region.
KylieWow. Okay. So a lot to think about just in terms of, speech recognition there, getting the right vendor. That's right. Models. Yeah. Okay, cool. So you've got your, your speech recognition. That's one part of your tech stack. What's next?
ShawnYeah. So then, usually speech recognition would all also come with a small technique is called voice activity detection. This is called VAD. This is basically a model that decides when to stop listening and then tell the next model saying that, Hey, now is the end of that user speaking. Now you can start speaking. That's another challenge as well, because you can imagine if that model is not accurate enough, you'll probably wait for too long. So you cause higher latency.
ShawnIf you are waiting for too short that you are going to interrupt your users and also depends on your users. Demography, right? The older. People, all the people will tend to speak slowly, slower, then you need to factor that in, in, into your, the design of the systems as well. So there's more to consider.
KylieYeah. That's amazing. There are times when I'm on like a zoom call with somebody and I have to go, are you still there? Have you frozen? Cause they're just thinking about how they're going to finish the sentence. And that must be so difficult. And that's with somebody that I can see. Let alone somebody over the phone, you can't see them, you don't know what they're doing.
ShawnThat's right. And then, as humans talking to each other, you will actually adapt to that person's, the way of their speaking as well. I think voice AI, you also need to think about who are your users and then are they pre, do they prefer to have a faster iteration or, a slower pace of conversation? So I think you see a lot of new. Come back platform companies, they're optimizing for latency. All we are saying is. That's, a shorter latency, maybe not what you need, right? Maybe your user would actually tend to post for a very long time, if you think about it.
ShawnSo if you project yourself, as a developer, you want things to be fast and punchy, right? But you shouldn't project yourself when you design your system.
KylieYeah. That's so interesting because you take into account as well the fact that people don't know how to speak to voice technologies. They've been trained to speak to them in keywords or in this really stilted and awkward way. Yeah,
Shawnand in real, in, in reality is that we found that a lot of our users falling into contact centers when they pick up and then they notice is a voice assistant. They pause for a very long time, wondering what else is going on. What's the IVR, this is what I get used to. Shouldn't I press 1, 2, 3?
KylieYeah. Okay, cool. So you've got that speech recognition part you have This part where you're saying, how long should I be listening for? And then what happens?
ShawnThen, once you transcribe it, you, it's basically transcribing it into a text, right? And that text then will be sent in a modern conversational AI architecture with a generative AI. You send it to the gen AI models process. Usually before that, there's also some sort of other things that you wanted to do the safety guardrails, the content filter, you want to make sure that those sentences are actually not harmful or, not intended to do problem injection and things like that. You should have extra safety guardrail there Before LMS, what we do at Polly is that we don't, because our clients are enterprise clients, they are very adamant about how can I gain control over what the voice assistant to say.
ShawnSo before we fit it to LLM, we also have a proprietary dialogue framework to actually. Decide based on, the context of the conversation, the, what a bot should be doing. So we have what we call the, so we have the rack component. So basically it's, it's like a normal like rack kind of system where you put the user queries into the rack and then the rack would actually return you the top cup, top File top 10 results, right?
ShawnDepends on how you configure it. And that result would then be fit into part of the prompts. We say our apps, but then we also have a techniques called talk tracks. So the talk track is we are still thinking about the branding of those terms. Kylie, we will need you. You will have more on that one in the future, but think about When you go on to the speech, to the stage to give a speech, you either wanted to have the major topics you wanted to talk about in that, for that audience and in that conversation.
ShawnSo you take those notes. That's basically what we do. But, the notes that we are taking for the voice assistant is much more structured. So we tell the voice assistant, Hey, In this particular kind of scenario, you should do this. It's almost like a contact center training menu. Hey, if the user is asking for paid refund for this case, you shouldn't do, you shouldn't just refund them. You should first ask why you are requesting refunding. And based on the reason you will triage, which reason is actually valid for the refund.
ShawnSo there's a lots of these. business processes that you are, you wanted to overlay it before you send a query to LLM, because you are not going to just completely lean on LLM to make the business decision for you. I'm sure that a lot of people wouldn't feel comfortable about it.
KylieLike in the old way with an intent based model, it would just the utterance would just trigger an intent, right? It would be the closest thing. So they don't do a certain amount of things. That, that could happen
Shawnas
Kylieyou start using generative models, there's many, many more things that could happen. So you need to apply this logic.
ShawnYeah. It's like a two end of the world where the intent model is super precise, like the user needs to exactly say certain things to trigger that certain behavior, but in the general world is almost super blurry. That whatever you say, the model has something to say back to you, sometimes overly smart about it, right? So you need to find somewhere in between to say that now, what we basically saying is that, based on the. context you are being boxed in, right?
ShawnLLM is free to make a lot of judgments by themselves, but you are not going to let LLM to make the entire business decision for you because that's a way too risky, right?
KylieYeah. That's interesting. So that's like your call center script or processes. It's your agent wouldn't have a chat about the weather, their family for 20 minutes. So sure that your models don't that's interesting. And I think putting that, the guardrails in before you even send The prompt to the LLM wasn't how I'd previously thought about guardrails. I'd thought of it as something that you were stopping the LLM from doing something rather than taking charge of what it actually gets in the first place.
ShawnYeah. There's also some other, there are second layers of guardrails at the end, but we leaning on the guardrails before, because once you do it later, you are actually fixing the problems that shouldn't have happened. So we're trying to be proactive. about a problem to begin with.
KylieYeah. That's again, if you had an agent on the phone, they would, somebody would say something to them and they would say, no, I can't do that. Or that's, that's not what we do here.
ShawnAnd
Kylieit's giving that power again. Okay. So we've understood, we've listened to what the caller has said. We've given that information to the model, the model has figured out, okay, here's what I'm going to do next. What's the next piece in the tech sack?
ShawnYeah. Once you give those prompts, instructions to the model, then the model can make a couple of decisions, right? So the model can decide this is now the right time to generate a text or the con the information I, I need. I guess so far it's still not sufficient for me to reply to the user's query. So in this case, the model would actually trigger, the function call or the people prefer to call tool tooling these days so that once you trigger the function, this is a programmatic instruction usually trigger, in as a lambda function or whatever.
ShawnOr the A API query. Yeah. So that you can actually. Send it to a cloud, pull down the information from your U user, CM system, for example. Yeah. Or you can check the availity of your timetable or a lot of times we actually use the load of function codes just to write checks about. What LLM has already collected, give you an example is that one very popular LLM is a very, very good at not good at that, very prone to hallucinating the function variables.
ShawnSo a lot of times when you, when we say, collect list of phone number. Even the user doesn't say anything about the numbers or digits, you will just hallucinate a number from zero to nine, right? So zero, one, two, three, four, seven, eight, nine, that's it, right? So a lot, this happens a lot. So we also need a second layer of light triggers that in that case we say, we tell LLM, Hey, don't just assume that you have already collected the number. Please triggers a function so that we can have a supervisor walking with you to actually validate these numbers.
ShawnAnd that supervisor is actually just a program that checks exactly. Hey A valid number shouldn't be zero to nine, for example, right? The zero UK, the valid, UK number should be plus four, for a valid US number should be plus one whatever you are. Yeah. So you can mix the program logic of like very precise logic into a very fussy LLM. So that, You make sure the model is working in your favor rather than actually, hallucinating or along the way.
KylieThis is, this sounds like a lot of work. Like
Shawnthe
Kylieidea of generative was that it, it should make this stuff easier. Because we see, we see this all the time, new companies popping up and they've just put this wrapper around an LLM and they say, here you go, build your own voice assistant. But from what you're saying, there's a lot of work. Yeah. Don't get me
Shawnwrong, right? I think this is, AOM does get things a lot easier. Yeah. But then, it's not it's not like a fairytale you just put things together, you magically works. And I think this has been something we have been telling people is that, building a voice assistant is not you have a. chatbot and you just put voice, at the front and the back end and you are, you magically make a voice assistant, right? It's never like the case, but I think a lot of people, I think a lot of these kinds of technology is still in a very early stage, right?
ShawnA lot of people still figure out like what it can do, what it cannot do. So there's a lot of like misconception about like how powerful the system is. And I think We are going to come to a realization, yeah, like model is phenomenal, right? It's super powerful. It's like nothing comparing to what we had previously, but there's still a lot of limited patients, right? And we as a, enterprise facing company, we directly sell these voice assistants to our clients and our clients are demanding a really high quality voice assistant, right?
ShawnSo therefore, I think one good analogy is like, you know, LM can get you to 70 percent of what you wanted to do very quickly. Then the last mile usually is like another, get to 95% You still need to put in a lot of work and a lot of effort. And then that's what our clients is demanding in terms of the quality of the part that we build.
KylieYeah, of course. Cause you don't just want to put something out there that's going to make stuff up or, if you had a person doing that that'd be a different story. Okay. So you've, again you've understood what the person has said. You've checked that, made sure that that, that makes sense. What's the next piece?
ShawnYeah. Then, eventually the bot will probably make several tool codes. It could be, several APIs running at the same time, several checks at the same time, but it will keep looping until the bot thinks that it's the right timing to respond back to the user that I got all the information. So in that case, the bot would basically Response generate a response back to the user. One interesting note to, to say though, is the tool calling, right? We get the API, gets the API result back.
ShawnA lot of times these are, when you get an API, sometimes you get a failure or you get a success, or you get some sort of parameters from it. We would also summarize these API results into the natural language. And then write it back to LMS. So you can imagine it's almost like you have a supervisor whispering in your ear to say, Hey, no, no, no, that's wrong. Like the number shouldn't be that number. You should, instead, doing it again or repeat the process again.
ShawnBut yeah, anyway the, End of it is you generate a textual response, right? Yeah. And that text response would then be sent to a speech synthesizer system. And that's what we call the TTS. And TTS, we're basically converting text into a voice. And then, stream the voice back to the telephony system. And then the user from the other side will be able to hear from the agent.
KylieYeah. Okay, wow. There's a lot. And that is just one turn of the conversation. That's the
Shawncustomer
Kyliesays, I'm having a problem with my credit card. All of that is happening then.
ShawnThat's right. Yeah, exactly. There's a lot of, little things running in the background to actually make magical things happen. Yes.
KylieOkay. Amazing. Sean, before I let you go, there's a lot happening here. This is a huge tech sack. Will we ever get to the point where there'll be just one single model that does all of this for you?
ShawnAbsolutely. Yeah, I think so. I think end-to-end, or this is in the research community. Research community, we say, one model rules it all, or the end-to-end, like research directions. To be honest, that's not this whole thing is not a new thing, right? I think LM that people started to research on that, in since 2014, right? If you see the technical shift in the past, it always been a case that it becomes. Bigger model, bigger data set, and to rule it all.
ShawnThat's just like naturally how technology evolved over time, because this kind of, language is very hard to model by just, writing for the programs, right? And we see that happen with translation models, right? In 2016, we have sequence to sequence for machine translation. In No, recently the speech model also get to and for the fashion, I just say, the whisper of the world, right? Although they are not fast enough for real time speech yet, but it's going to get better and it's going to get there eventually because these are the obstacle that, we know how to deal with.
ShawnSo end to end speech will be there I think in the next year or two, right? So it's coming. Then. You have the LLM is already end to end, right? I remember when we modeling that dialogue systems eight years ago, people has been working on modular systems. And, that's why we have three, three co founders, Nicola working on language understanding, Eddie work on the policy management. I work on the language generation. So we, we have that type of pipeline. At the end of my PhD, I was fascinated about this end to end conversational modeling.
ShawnSo I have already, I have some papers, that was like very early days. The model was very small and the data set was very small, but we will be able to see there was some, very encouraging results back at that time, right? So I actually published those papers to top kind of conferences back then. Yeah. Yeah. Yeah. So this kind of end to end system, and you see now is LLM finally becoming like a production ready system and everyone's talking about it, everyone is using it.
ShawnSo to me, it's pretty clear that end to end speech model, it's going to happen at some point. I think we have already like GPT 4. 0 is already. The first, like for the product outlayer started to showcase that capability. We saw a lot of research papers has been, working on the list in the past couple of years. Yeah. However, do I think that you will be ready for especially the enterprise to adopt in the near future? I don't think so.
ShawnBecause even LLM, since we started working on it was. In 2015, 16, now, like we are eight years in, right? It takes some time, but the thing is these kinds of models, once you crack the initial setting of the model and the data, and if it's just a matter of amount of data, the iteration becomes super fast, right? But you crack that setting to begin with. I think the, with the voice end to end, I'm not sure we are cracking the entire setting yet.
ShawnSo I think we are still in the early stage of the entire development, but I do agree that this is the inevitable. The direction we were moving to.
KylieYeah, great. So with enterprise use cases, don't wait for that technology because it could be years, it could be a long way away, but there are ways to get the same results today that you would expect. Okay. Amazing. Sean, thank you so much for joining me today. We're going to do a series of conversations about different parts of the tech stack. So everything that you've heard today, Sean and I are going to dive deeper into each of those pieces. So if that's what you're into join us again for future episode of deep learning in the meantime, thanks for joining me, Sean, and thanks for
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...