Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Deep Learning with PolyAI

Deep Learning with PolyAI artwork

047: The anatomy of a voice assistant, part 3 - Large language models (LLMs)

Sep 5, 202428 minAudio
ShareXLinkedInFacebookWhatsAppEmail

It's the third in a series of technical deep dives for building IVAs, and for this episode, it's all about LLMs. Kylie chats with Shawn Wen, co-founder and CTO at PolyAI, who explains why LLMs can accomplish 70% of what’s needed for voice assistants, and more importantly: what's involved in the remaining 30%. The discussion touches on strategies to manage challenges through prompt engineering, dialogue management, and integrating third-party APIs, while highlighting the similarities between AI and human behavior. A look into cybersecurity and the future of specialized LLMs rounds out the conversation, offering insights for industry professionals navigating this evolving technology. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

ShawnIf we think that AI is actually getting closer to human and human intelligence, then you kind of have to treat AI as a human as well. And I think there's one special punting techniques. Our engineering team font is that. It's a lot harder to tell the AI what not to do comparing to tell the AI what to do. And it's kind of a lie with humans as well. If

KylieHello, welcome to deep learning with poly ai i'm joined today by Shawn wen co founder and cto at poly ai. Hi Shawn

ShawnHello.

KylieAnd today we're going to carry on our discussion about how to build a voice assistant, and we're going to focus on LLMs. So Shawn, a couple of episodes back, you said, I think I'll paraphrase you. The LLMs are 70 percent are good enough to get you 70 percent of the way in terms of building a voice assistant. So today I want to talk about that other 30%. Why can we not just plug in a really great LLM and have our voice assistant work straight away?

ShawnI think, yeah, that was like two episode ago and I think, you know, LM is actually just wanted to clarify, right? It's actually get you to last 70 per percent in a rec record speed. It's like something that we won't be able to do in the previous generation technology because. The kind of conversation ability you need to be carefully encrypted, and then a lot of these conversations, you kind of have to foresee how it happens and how can the bot answers where it should progress the conversation. So you, as a developer of these kind of voice assistant, you need to foresee a lot of the conversational paths beforehand, which is not quite.

ShawnIt's not really doable because the humans don't actually having a conversation that way. And I think plot plugging in LLM instead, just making it a lot more powerful because these models have learned from billions of conversations. Before in different kinds of context, different scenarios. So therefore, like a lot of the people these days, once you integrate into LLM and you put a speech recognition on top of it, which we also covered why it's not less straight forward from the speech side, but even from the large language model size, also not that easy, right?

ShawnYou plug in a chatbot and then you have a LLM based chatbot. You when you talk to it in all sorts of different ways, it feels really powerful, but there's a lots of pitfalls come with it afterwards, which is that, the, I would say it's like a two sides of the same coin. So basically the previous generation technology, the Intel based the model. It's a very precise. If you want the bot to actually take down a particular piece of logic, you can do it very easily.

ShawnIt's just a very clear that if a user say this, the bot should do this and say that, and the user say this, you should do other things otherwise. So that's very straightforward from a builder point of view, it's very straightforward. Like clear, like what to expect. And then a lot of times the contact center, it managers like that idea because they wanted to have a control of the user experience. And then that's how they have been doing that with their contact center agents is that. They give the agents a manual and say, Hey, like in this scenarios, you can do the refund automatically in last scenario, you have to ask the reason why, and then take them through a process, for example,

KylieI think it. The intent based model. So we used to use intent based models a lot, right. And we still do in a couple of instances where clients want that level of control. But I think even with intent based models, you get that control, but it came at the cost of, it could get very complex very quickly, right? So depending on how many things you want to handle, you ended up with this massive decision tree that becomes complicated.

ShawnExactly. Yeah. So I think that's the real, really the problem of the intent based model. And as you can imagine, as the voice assistant capability continue to expand, the system will become more and more fragile because there are so many different branches that you have to expect and things like that. And then I think that's where LLM is actually more powerful than the intent based models, because now it's just a one single model. It try to match the input. Sentence patterns, and then based on the instructions and the prompt you provide to the system, and it just generate the most reasonable response afterwards.

ShawnNow the issue, it comes with the LLM, and I think that you will probably see a lot of people online already complaining about it. It's just not controllable. It feels like, A very talented student that you tell them to do certain things, but then they figure it out how to do other things instead. And then they just don't follow your instructions. And that's what kind of problems we have to deal with. We do with ALM.

KylieYeah, it's, I guess it's different to a customer service use case, but when I use, I was using chat GPT the other day to try and come up with some like pithy marketing statements, right? It didn't give me anything good, but every time it gave me some responses, I'd say, no, make it more like this or change it in this respect. And it's a lot better than it was being able to understand. That I'm applying that same context from, the first request and then doing that. So it feels like generative AI is getting better at least like multi turn, conversations.

KylieRight. But I guess when it comes to making a booking or making a payment or some other kind of transaction, maybe. Maybe we're not quite there yet.

ShawnYeah. I think, the kind of use cases you're using, you're actually using, um, and, or it is AI models who actually co pilot with you to accommodate, and it's actually very powerful that way because you can actually iteratively prompting the agent and getting closer to what you want, the problem of deploying this in the contact center and it. In a direct user facing applications is this is an auto pilot application. That means that you, because the user or the caller will not be cooperative with their app because they, I mean, first of all is that they probably don't know how to use the tool that much yet, and then this is probably the first time they talk, they are talking to them and they don't know that they have a, issue in mind.

ShawnThey just wanted to get it done and then they wanted to get it done as soon as possible. In that kind of scenarios, it's very hard to imagine the caller or the user would actually to incorporate with LLM.

KylieI was talking to someone at, at poly AI recently about prompt engineering. Right. And yeah, chat GPT is very good. When I say I want this, it's, it does a pretty good job of understanding what that is. But I think there are instances right, where maybe if you repeat something a couple of times, it just performs better. Or if you do something that you maybe wouldn't expect Yeah. To say or do, it actually improves the performance of the LLM. So I guess if you're a customer. You're being asked to be a prompt engineer if relying purely on the LLM to do that.

ShawnThat's right. Yeah. And I think, a lot of these like response generation, a quality of it is still, I mean, people have been talking about hallucination issue of hallucinations and then, fabricating content that doesn't exist. I think this is a Very legitimate concern and a very prominent issue that people have been seeing and then there are some challenges in deploying this into consumer facing applications. There's Other aspect of it, which, in order to make these LLM really powerful in the context and center automation, you still have need the LLM to integrate to the backend systems of your, of yours.

ShawnAnd that's what we call a tool called coding or the function call abilities. It, these abilities basically saying that. Based on a particular conversational context, LLM would decided that the best decision is to not to generate a response is actually query a third party API, getting the information in and then integrating that information into LLM. The response. An example would be if you ask the LM, what is the weather in London right now?

ShawnAnd the LM should know there's an API it can call to get this weather information with London as location and then incorporating this API return result. Thanks for the response.

KylieRight. So I guess again, when you, and I keep going back to chat GPT because it's something that we've all tried and we've all used. If you were to ask chat GPT, what's the weather in London today? I guess it's been programmed to know that when you're asking for, someone's asking for the weather, it's going to go and look at the Met office or whatever.

ShawnYeah. So basically, like in. A lot of these in the back end and then people have already building a lot of applications around it in order to actually make these more powerful is to integrate in real time third party information into the system, let it to be the weather, your calendar. You know, your conversation between you and your colleagues or, the contact center in real time information of the contact centers, for example, and these are all very valuable information to integrate into LMS and making them powerful.

ShawnOne, one of the real major challenges that you, we were, we are seeing in integrating this, even the best models out there, let's say GPT O. The problem is that, we have this metric. We call the function call ARA, which is basically like how frequently that the bot should supposed to call an API with the, a set of parameters, but it actually does the wrong thing. Right.

KylieAnd

Shawnwhat's even worse on top of it is that the bot will say, okay, I will check the weather for you, but actually they didn't call the API at all. And then fabricating an answer back to you. So that, that, that problem started to compound, you know? So imagine, I think whether API is okay, right? You can say that, the bot just hallucinating, but thinking about if you are making a reservation to a restaurant, the bot say, Oh, I have made the reservation for you, but it actually didn't call the API and you show up in a restaurant and the owner was like, Hey, sorry you didn't make it with the booking.

ShawnThat would be very disappointing.

KylieYeah. So how do you get around that? How do you trust that it's doing what it's told customers that it's doing?

ShawnYeah, I think there are, I mean, that is basically how, why we are measuring the function call error rate, because you, a real application, you need to actually drive that function call error rate down. Too close to zero really but I think in a lot of the Just like a very basic implementation, just following the open AI menu of how to prompt in GPE, how to create function calls. What we are seeing that is, in general, the function call error rate we are seeing is about at least a 10 to 20 percent error, which is sign.

ShawnPretty high. And then it's higher than you, you expect it. I think, which is that also reasonable because these kinds of models are very powerful to do certain kinds of tasks, but the kind of tasks that we are prompting them to do, they probably have not seen it before. So imagine if you are not seeing that before, I just put you on the job and then to do the job, you have to figure out along the way. Right. So that's like a situation we are facing there.

KylieOkay. And then is this where things like guardrails come in and dialogue management, like additional work that you do around, The LLM to get that 10 to 20 percent down. Cause obviously if you're a poly AI client and we're, yeah, taking restaurant bookings, we don't want 10 to 10, 20 percent of your customers showing up. And they,

Shawnyeah. So I think there are a couple of like techniques to tackle that particular problem, but. Usually people will start it with a lot of prompting techniques. And then that's why prompting engineers becoming a thing these days, because like you felt that you just are writing a text to LLM, then the LLM should just work, but it's actually not that trivial. So that's why like people started to talk about prompt engineer is like a real job, a real title, for the future, right? So I think there's a lot of like tactics to actually prompt the bot to be able to more reliable.

ShawnOne. For example, one particular pattern we are seeing is that despite the fact that these models have longer and longer context window, the fact is that if you are cramping too much information in one single prompt, Despite that the context screen code can fit those information in the model still have a much higher likelihood to get things, which kind of makes sense. You know, if I send you a menu of 100 pages and then say that, this is like the instructions you have to follow when users say sort of things, you will get it wrong majority of time as well.

ShawnBut if I only give you one page, then it's much more clear. And then, you can follow that instructions more more easily. So what we do internally is that we'll have the style of management come on, on top of it. We haven't actually really figured out what, how we should call it yet, but it seems like the product team, they prefer to call them top tracks. So it's the idea of not to craft every single conversation in every transition, just like the you know, before you go on a stage to actually give a speech, you will probably write down a few bullet points to make sure that, these three bullet points are on top of your hat so that you can talk around those points.

ShawnRight. So that kind of ideas,

Kylieokay. So building an additional framework that can sit on top of the LLM and just check like. Check it's done this check, it's done this. Yeah. Make sure that it follows the path without limiting it to the path in the way that an intent based model.

ShawnThat's right. Yeah. And then just to give you another example is that in a traditional intent based model, it's more about, I create like a very clear flow that with a lot of if else statements. And then if you go into this, you go there and then you do that in et cetera. Everything's very pro programmatic. Yeah. The way we are doing it. So I think then the LLM way of doing it is that I fill you a hundred page user manual, and then you figure it out. So LLM skin through the entire text, like a hundred pages, and then try to come up with the answer.

ShawnThe way we are doing is something in between. It's kind of like now. This a hundred page, I'm going to group these pages together for you. So you have 10 categories you have to think about, and then you put very lightweight rule on top. So say if user talk about payment, you should, refer to this five pages. And if the user is talking about something else, you should refer to that. And then within those page, they are probably like step by step instructions as well, how to proceed that transaction.

ShawnWhat we do is that, think about that. We patronize the steps so that, you just, the bot just needs to follow the instructions step by step rather than, like, Oh, this is like a huge. That document just really an answer.

KylieYeah. Really interesting. I suppose in the intent based world, it was like having a, an agent who's working off the script all the time. Yeah. And in the generative world, it's like an agent who is off script. Right? They've learned it, they know what to do, and now they can be natural flow through the conversation, but you can trust them. With the right dialogue

Shawnmanagement. So basically all we are doing with these talk tracks or dialogue management component is really to bring back the control to the contact center, which they used to have with the intent based assistance, but with JNI they don't and then they feel quite scared and they worry about it. We are trying to create this software there to bring back them the control, but still Not compromising the performance of what the, of the LLM can bring to their bots.

KylieInteresting. And you were talking about prompt engineering and because, it's not as simple as just saying, here's what, do this to the LLM. There's very specific ways that you have to ask, um, prompting. And I guess, a lot of this is it's in the black box, right? Like you don't know necessarily which prompt is going to work. And I think this is getting better documented. People are learning a lot, but I think this concept of the black box is one of the main concerns about LLM. Yeah. What is actually going on? How can I be sure that, that it's following my processes and doing what I want to do for my customers?

ShawnYeah, yeah, yeah. And so, yeah, I think another important concept of this the dialogue management component we are trying to bring it into it is that we are going to help the, you know, because contact centers always have their processes. So we are going to help the contact center managers to say. Break down the entire transaction. Let's say like a reservation, for example, but break it down into several steps and let's say there are four steps for you, for example, each steps, it has its own module or own prompt or instruction inside and within that box, it can do whatever it wants, but at the end of that transaction, you always need to make sure.

ShawnA function called gut trigger so that the entire state of the conversation changed to the next stage so that becomes a checkbox exercise, right? So each of the LOM Subprocesses would have a checkbox at the end to say, Hey, like all the tasks in that box has been accomplished, so then you can move on to the next stage so that when you look at the conversation later on, you know, okay. The bot actually tick all three, but the bot boxes, but the last one didn't tick. That's why the code got transferred. Or like all the bots, all the boxes got checked so that you can proceed to say, oh, I have finished the transaction.

ShawnAnd those check boxes would, includes important. APIs need to be queried, important information, need to be registered in a. Database. There are a lot of these, like processes you can put around it to make sure the bot is actually doing what it says it's doing.

KylieIt sounds like there's a lot of compliance work that needs to happen. So even if you have, you've built this thing you've got a team of people who've built this thing, they're running it, somebody needs to be monitoring it and making sure that it is doing what it needs to be doing.

ShawnThat's right. That's right. Yeah. So I think, therefore like having a. Team of experts who knows how to build this kind of thing. Because I think the fact is that LLM for voice assistant and contact center is still in its early stage. There's a lot of things that, people are still figuring out along the way. And I think it will be quite a very expensive process, try to do this entire thing. By yourself internally.

KylieYeah we've seen some examples out there from big customers. And I won't name and shame, not big customers of ours. I should say big companies who've deployed generative bots that have gone rogue, that have, um, And yeah, I suppose that just demonstrates that there is a lot to learn around how we make this happen. That said, we have a couple of our clients running on purely generative AI deployments and the, and they seem to be. Performing well.

ShawnYeah. Yeah. I mean, those applications we actually did put in a lot of effort to continuous improvement, because it's not really the right thing to say that we get everything's right at the first, the

Kyliefirst time.

ShawnWe also like just through years of experiences and, putting the right play, like the right technology, the right procedures in the right places to make sure that everything is as smooth as possible and those kind of things.

KylieYeah, you want to, if you've been there, done that, seen it all go wrong, you're in a good position then to be able to make these tweaks, build these frameworks, do all this additional work to really. Yeah, make it secure, make it work for your customers. Cool. What else should people know about LLMs when they're thinking about building a voice assistant?

ShawnYeah. I think one thing, there are two. Pretty subjects, uh, we, you, you mentioned a little bit about guardrail, but we haven't actually talked too much around it yet. I think a lot of people talking about guardrails, genetic safety and all things are like, and then there are lots of like governments across the world is trying to regulate, regulating it. I think we, there's one thing I think that we realized maybe like earlier this year is that, genetic safety is a very important topic. And from injection issues is a real issue.

ShawnWe've seen a lot of examples, deployments that they have failed previously. I think it's important to think about this kind of prompt injection issue as a new cyber security frontier. Like if you think about cyber security, how they deal with this kind of threat, it's not to Create a system that is a hundred percent bulletproof because it's impossible. Cyber security is the space where the tech, the defense technology continue to improve, but the attack techniques and technology also continue to evolve.

ShawnSo it's a never ending war that you have to constantly fight. And I think that's the same thing with genetic safety. It's the same thing with these prom injections, because people can't. Constantly figure out a new way to prompt you about to do something that it is not supposed to do. I think the most important thing is that if we have the cypress, the way to treat his genetic safety as a separate security problem, I think the most important thing is not to have a bulletproof or bulletproof systems in the first place.

ShawnIt's more about. How do you do threat detection when a threat happens and how do you do the reporting so you have the visibility around it and also the mitigation procedures afterwards, right? And then you need to build up a continuous plan around it as well. So I think that's the right kind of mindset, right approach to think about the problem. We are very lucky to have, working with one of the largest with the British telco. And their cybersecurity is like really, really good because they just, they actually identified that this is actually, genetic safety is just a cybersecurity issue.

ShawnAnd then, they help us to actually shape our strategy towards that. But I think, that's, Not a very obvious one, but I think that people would actually realize that will be the case.

KylieYeah. It's almost like the better this technology gets, the more you sort of almost move backwards and think about it in the same way that you think about people, right. In that a lot of the big hacks that we've seen recently have been the result of social engineering, which is, prompt engineering, you know, Is almost a type of social engineering right on robots rather than on people. So it totally makes sense that having that arm of cybersecurity that is focused on this is going to be how this works in the future.

ShawnYeah, exactly.

KylieSo I've heard people talking about, training their own models and small language models, is that something that you need to think about if you're going to build this up yourself? And I guess there's two questions there. Do you need to think about the size of the model and do you need to think about how much custom data it is trained on from your use case?

ShawnYeah, yeah, yeah. I think the custom LMS or specialized LMS is a future and I think size is like the smaller size of course is cheaper, is easier to run. It's always, it has a lot of engineering benefits associated to it. But it's not the most interesting points, right? I think the argument to have a specialized, a smaller LLM is like, Because these models can actually do better for that particular special tasks that you have at hand. And I think that's also aligned with the human evolution as well.

ShawnIf you think about in the 14th, 15th centuries, you have Da Vinci's that can actually do everything, right? It's a scientist, also an artist, also a, theoretical physicist or whatever. The person can do everything and he's a very generalist. But then as the human society continues to evolve, Specialization is a key right now these days we have different people in different of their job roles and they are doing a very good job and in their particular role.

ShawnThe only reason just because they were trained to specialize. To specialize those tasks, right? So I don't think there's going to be a model that can actually fits all. Maybe that's the way that we go get closer to, a GI, but like specialize the model would be the primary adoption in industries and in the future, because that's just how the entire world evolve and that, that's just like the theory of it.

KylieAnd that's how, again, that's how. People work. That's how call centers work. They have special ACD groups, of people who have, who can definitely do this one thing really well.

ShawnYeah, that's right. Yeah. So I think, like now I think people are debating about general model, custom model, I think that's, because it's a very interesting topic to discuss, but I think the truth is actually just right there. It just depends on, I think. It's the matter of, like, when is not the matter of how.

KylieYeah. Okay. Thank you very much, Shawn. I think throughout this episode, I have continued to refer to, like, compare generative AI to a person and the way that a person learns. And I think it's worth highlighting that that's a very easy trap to fall into. It's a very easy way to try and understand this technology, but that we're still at a point where there's We don't know how some of this works. We're still figuring it out in academia. It's still being figured out. And there are things that we can do, but don't just think. Oh, this is a really smart person.

KylieI'll just let them do whatever.

ShawnIf we think that AI is actually getting closer to human and human intelligence, then you have to treat AI as a human as well. And I think there's one special punting techniques. Our engineering team font is that. It's a lot harder to tell the AI what not to do comparing to tell the AI what to do. And it's a lie with humans as well. If you have teenager, boys and girls do this and then they were definitely going to do that. Right. So I think it's a very interesting, my career of like, you know, working with AI machine learning just continue to see a lot of another today, but yeah.

KylieYeah, fascinating. I think it's only going to continue to become more interesting and with that more confusing but also hopefully more effective as well. So thank you very much, Shawn for talking about LLMs with us today. I think next time we may talk about speech generation which is a whole other side of things that I'm sure will open a whole new can of worms. But until then, thanks for joining us, Shawn, and we'll see you soon.

ShawnSee you soon.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. What if your contact center became a product team?

    Nov 17, 202635 minAudio

    Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...

    Credit the people on this episode

  2. What happens if your AI agents get 1% better every day?

    Sep 10, 202639 minAudio

    An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...

    Credit the people on this episode

  3. Who's coordinating your army of AI agents?

    Sep 3, 202644 minAudio

    Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...

    Credit the people on this episode

  4. Why should CX leaders care about MCP?

    Aug 20, 202636 minAudio

    When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...

    Credit the people on this episode

  5. Can AI really hear a call the way a person does?

    Aug 6, 202622 minAudio

    PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...

    Credit the people on this episode