018: How to avoid PR nightmares when deploying generative AI
23 minAudio
Damien and Nikola discuss the importance of safeguarding when deploying generative AI instances, specifically calling out the recent DPD debacle, where a chatbot was manipulated to produce inappropriate content. The episode also touches on the concept of retrieval augmented generation (RAG), and the potential flags to watch out for when building and testing generative AI systems. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
NikolaYou better be really careful about this, right? Because like you're, you've probably not yet invested enough to have the team members that have had enough experience working with these things to know what can go wrong and how it can go wrong
DamienHello friends and welcome to yet another episode of the deep learning with PolyAI podcast. I'm Damien. I'll be hosting today and with us as usual is Nikola Mrkšić, CEO and co founder of PolyAI. Hey Nicola.
NikolaHi Damien. How's it going?
DamienNot too bad. How are you?
NikolaPretty good. Pretty good.
DamienExcellent. Good to hear. Today we're talking about some stuff that happened recently in the news, but more broad applications, the importance, the essentiality of safeguarding when deploying generative AI instances, applications you may have heard. Especially because I know you're based in the UK about the DPD debacle that right now, if you Google DPD, the second link is going to be a link to a tweet from one, Mr. Ashley Beauchamp. Basically highlighting a, as the kids say, epic fail with a chatbot and text based support where with a, just a modicum of prompt injections they were able to get this bot to do a haiku bashing DPD.
DamienAnd speaking with some adult language as it were. So this is just one of many instances, it went viral, it got a lot of attention, but it's one of many instances of how people can nefariously manipulate this very advanced system to do their ill bidding. I wanted to start off today's talk by first of all, getting your opinion on this sort of thing about how it becomes like the talk of the town so quickly. How they caught heat, but more important, prompt engineering seems to involve a lot of trial and error approach due to the complexity of these generative AI models.
DamienWe have been able, we being PolyAI have been able to build solutions that do not act like this. So why are these companies launching gen AI that fail? In this manner. That's a really good question. We know DPD. We've been hoping to work with them for a while. They should call us back. But
NikolaNo, really, I think building assistance is fun, chat or voice, right? It's it's a cool thing to do. And with generative AI, it's like giving someone like a really sharp tool. That is very useful for the task at hand, but also potentially quite dangerous, right? With LLMs, it's never been easier to build a prototype of a virtual system. And if you ask general things, if you ask GPT 4 about general DPD procedures, and, it's been trained on recent data, it probably knows the answer, right?
NikolaIf you ask it about the history of, like the biggest hotel in your neighborhood, it probably knows that too. It knows a lot of things, right? It's very impressive, and, it's part of the reason that we all fell in love, right? Now, the real thing is In unstructured form, when, it's provided a bit of information, when you ask it something, it'll query out of that. But as the conversation keeps going if you want to change its behavior, if you tell it to do different things, like they did in that example, where they asked it to do to ride their haiku and whatnot It'll do it, right?
NikolaIf you've not actually built a system around an LLM that kind of like constantly steers the whole thing back to an autopilot of you are an assistant for DPD, you are here to represent the brand. You are to stick to this, company policy and script, which is embodied in this here, set of instructions, documents, whatever. If you don't stay fairly adamant about that. It can diverge into basically just being a free flowing LLM. And it seems like they did very little to try to stop this.
NikolaThey tested it sporadically. Yeah. I thought it was cool, put it online and hopefully people find it useful. No one did this, there was no bad intent, maybe apart from those who actually did this. And, at the end of the day, if you understand how the technology works it's just like a bit of a laugh, right? I don't think that it's a terrible thing, but like from how viral it went I, none of their competitors want this to happen to them. So like fair game. But it shows you like. You can build things really quickly, but if you don't test it exhaustively if you play with DIY don't get surprised if, you get fired for the outcome if you, do things that are too, that are not tested, that you've not looked at, how, like all possible scenarios, or at least very many.
NikolaAnd I think, with the whole process of setting up the initial version of a voice assistant Oh virtual assistant in this case, a chatbot being so easy it's really hard to think that you have the end product ready, right? And to skip intermediate, QA steps and doing,
Damiento be fair, they said that this was happening during an update. There was a little bit of blame that was being pushed toward the system kind of being on hiatus. Alluding to the fact that the safeguards were in play. They just were deactivated for a system update at that point. And it was a fairly innocuous thing, right? He got him, he got the bot to curse a little bit and then to write a haiku and to bash the company, because largely we would assume it's because he wanted to go viral with this sort of instance, right? So they're pushing that off as saying this was a one time. We did everything right. But is that real?
DamienFrom a DIY perspective, do you think they really did everything right?
NikolaNo, no, I don't think so. I think that what happened here is a toy project that, went public and didn't really it wasn't tested. It wasn't I'm sure that there was some tactical step around these things, but I'll tell you that the whole field of building these safeguards It's not a well understood thing. So I don't assume that DPD actually, I know for a fact that DPD does not have a machine learning department that works deeply on, the latest methods for safeguarding LLMs. So that's an investment that most companies of that type just don't have.
NikolaSo I think it's really just a bit of naivety around Hey, like it seems to work let's put it out there. And, the risks just weren't considered, which is like at the end of the day, it's not the end of the world. It could have gone a lot worse for a different company doing this.
DamienYeah. Question. Speaking of safe, let's talk about one of the more popular acronyms right now that sort of ties back to an effective way that companies are deploying some safeguarding. And that's by having, AI in the form of rag, right? So retrieval augmented generation in their generative AI deployments. It's. Basically reshaping the landscape of conversational assistance, particularly in preventing a lot of these undesirable outcomes like the DVD scandal. What is RAG? Why is it that happy medium between the wild west, potentially, of generative AI, like you said?
DamienCould be distracted if not constantly trained and contained, could go off and show its party side, just like a real employee, right? But what is RAG and why is it why is it so effective and why is it so buzzed about
Nikolaright now? So when you think about an LLM it's been pre trained on a lot of data and has a lot of knowledge. And that knowledge sits in the parameters of the neural network itself. So we would say that it's like parameterized knowledge. And it's not that easy to go and operate directly on it. It's not a database where I log in and say, Hey, Damien's birthday is wrong. It is and I change it and it's like firmly changed. I can supply more information. I can retrain and fine tune, but that's expensive. Unpredictable, unclear, like which of the votes throughout the training process wins.
NikolaAnd it's just not efficient to do it that way. Now when you think about like how LLMs work, if you've played in prototype with them You basically see that they're really good at taking different pieces of information, and then extracting if you ask questions about Say 10 pages that you pasted into the prompt, and then you ask questions about that stuff. The model is going to be really good at answering questions about them. And, finally, I think it's a lot better at reasoning over those things. Sometimes with a bit of human help, but often just like on its own, it can make like second, third order conclusions based on the text.
NikolaSo it's really good at both like summarizing that information and highlighting the relevant pieces of information from like a body of text that you pasted. So when you're like, say, answering about, when you need specialized knowledge, let's say that you're answering questions about Let's say the Apple product catalog, right? You may know, there's probably a lot of this in like a pre trained LLM already. But let's say that you needed the exact difference in megapixels between a camera of an iPhone 13 and 14, etc.
NikolaThe way you would go and solve this problem yourself right now is you would go and Google it, right? And you'll find it online and there'll be a lot of it and maybe you'd read and look. But if you were to. Take the top 10 results. Feed that back to an LLM and ask questions about it, it would be able to just do that step for you instead of looking at the, those results, it would be able to just like highlight, Hey, okay, like this one has that and that this one has more right or and that's really powerful because historically when we talk about retrieval based systems and conversational systems, and we built many of these as we built our own transformer based models systems.
NikolaYou could do really impressive things, right? You could find answers to questions from product catalogs, from all sorts of like databases. You could pull them from APIs, right? If you query them with the right thing but then the one thing you couldn't do without an LLM is dynamically alter the response. You could just choose one of them, right? So for instance, if I asked you are you open on Tuesdays? You might have a response which says, On Tuesdays, we're open from nine to five, right? But if I said, hey, will you be open on Tuesday?
NikolaYou might just say yes we're open on Tuesdays. You might just say yes, which is like the conclusion of that sentence. And that's something that like an LLM gives you, it also gives you a chance to like rephrase. So if I say, Hey, you're open on Tuesdays. And what's the dish of the day? Typically a retrieval based system would say, yes, we're open on Tuesdays from nine to five. If you had multiple sentences there, maybe it would even know to choose the best one. It probably would not choose a yes, though, because that requires reasoning, right? But the other thing it would not be able to do is Ah, yep, and the dish of the day is French onion soup.
NikolaIt would not give you that. It would be right. We're open on Tuesdays from nine to five, and then it would be the dish of the day on Tuesday is French onion soup. Synthesized human like thing that a human would, right?
DamienCorrect. And knowing that that's a basic flaw that makes the concept of generative AI and these applications so much more realistic, creating more engaging, realistic conversation dialogue, and that it doesn't have to go through these responses with one intent addressed at a time, et cetera. Yeah. So when these things go out and when these reg systems are put in place to find that balance between mining internal data in a company's knowledge base or something. And that when they're asking questions about the company, that they're getting the right, most accurate trained responses from these bots versus when it says, how's the weather where you're at?
DamienAnd it's I'm a bot, but if I were in our headquarters in El Paso, Texas, here's the weather today, et cetera, et cetera, is that when something like this. Fails. And we were talking to we were talking to burn Elliott from Gartner earlier this week, and he mentioned how these rag implementations can sometimes easily go awry very early when the main culprit is bad data. And the bad data leads to faulty retrieval. And basically the bot breaks.
NikolaNo look, I think we build increasingly data driven systems that allow us to kind of like do a heavier dip into like everything that we have as resources and to build a better system, right? The real issue is it's not even like bad data as much as like the complexity of building that system. It's not moved to a different component, right? So if the curated evidence is wrong, it is off. Maybe if you pulled from like bad data, if you have an outdated product catalog, you might be, just providing flat out wrong answers, even though the LLM is doing precisely what it should, it's right.
NikolaThat's one way to get it wrong. But the other thing is also the whole mysterious piece of how do you get those responses? Like the complexity is in which pieces of evidence. Do you choose to provide right and that's like a whole separate like search system and logic that in a given application you have to build right so say with a voice assistant it might be an API call to get something to fetch some information and you need to know to do that right so if you're doing something like that then that's a modular operation that is a bit more difficult to do right and the complexity just hides in a thing that's even less you Easy for, your casual DIYer to go after because it's, hey am I bringing the right things as potential answers here for the LLM to process?
NikolaAnd yeah, after that, the LLM will be like your autopilot, but you have to feed it the right stuff. And if that's an even more complicated search engine that you're using. It's gonna be tough.
DamienSo what is the risk reward then? Because I would assume as the training progresses, you want greater transparency to the system, the operations, the sort of steps through the process that it goes through. What does it do from a competitive standpoint or from a development standpoint to have like greater visibility into these systems? What does it mean from the, or what's the variation from the DIY person who's trying to build this internally and needing to showcase? The various frameworks that it's going through versus just, partnering with a PolyAI where we would have to demonstrate this is why we won't, you won't experience with DPD experienced when our bots
Nikolain place. Yeah. I think it just, again, like it's the age old, build versus buy in terms of how will you actually go and build, Like the safeguards are, how will you feed the right information there? How will you stay on track? How will you test the system exhaustively? And let me tell you, like most probabilistic systems have a failure mode that if an adversary is really going after them, if not flat out embarrassing, like in the case of this one. It will at the very least be like, it won't work. And probabilistic systems are like that.
NikolaThat's why we build our systems with escalation in mind, right? We do not seek to replace the contact center. And we're really looking to, like escalate to the human at the right point, either for the sake of experience or because we're not doing something right. We can detect that we have separate, if you will, like adversarial models that look at our system and say enough handoff. That's really important. That's one safety mechanism. Do you think that's,
Damiendo you think that's the diffusive mode that like, let's just say whoever, John Q public is from DPD, who was responsible for this and had to attest to it and said, Oh, it was because of an update, et cetera, pushing the blame onto the tech. Do you think that that's their default defense is to say, no, this wasn't me. This wasn't the fact that this wasn't the fact that we didn't do the appropriate guard railing or safeguarding
Nikolafor this. There is no such thing as appropriate, right? It's like a new technology. Because it's ongoing. Do your best. And it's just the discipline is not yet well developed. And I think practitioners who've used it for a long time have developed better things. In a way, it may become akin to cybersecurity and that there's constantly new threats and new fixes for it, right? Hopefully it converges to a more stable point sooner so we don't live through all that, right? But I think what's really also important is just, like, how do you build a system? How do you test it, right?
NikolaOur escalation points are just independent of the LLM. You cannot put yourself in the hands of just the LLM because then you're fully exposed. To prompt injunction and everything else, right? The other bit is just like, how do you test the system? How do you get coverage of the things that people are likely to do or not likely to do? And that's just a lot for people to do, right? How do you tune LLMs? How do you get your own? The more powerful the LLM The more esoteric and interesting the prompt injection attack is going to get. So if you're, using GPT 4, yeah, like OpenAI has done a lot of work to make it safe, planes still crash and like we've done a lot of work to, get them to be as safe as possible.
NikolaBut there are failure modes that we probably don't fully understand, right? We model aerodynamics the same way that we test machine learning systems. And the more impressive and expressive Applications are the more esoteric the attacks and the more un, not unthinkable, but like the more difficult it will be to understand exactly why it failed. You think so? You asked about like auditability, right? Like the old intent style based systems. They were very auditable and it was really hard for them to do anything that Sure you didn't LY program in the dialogue manager.
NikolaRight now we've made the disc, this is akin to we're going back to descriptive programming languages where someone now like writes a Prompt and there's your dialect system, but like it's not only what you specified it to do It's also how you specify what it should not do right and like how You know if someone then goes into the middle of those two things to try to confuse the system. How do you like arbitrate? Have you actually provided examples of A or B, right? It gets into legalese. And you don't have guarantees that this is not a first order logic. This is probabilistic modeling.
NikolaSo it may or may not do what you told it, right? You're leaving yourself pretty exposed. And I think the best way, the reason why RAG is popular now And why it is, like the right way to do a lot of this is like it will use the stuff that is relevant to turn it into the best possible answer. You don't give it a lot of freedom to just go AWOL, you give it a very precise task, right? And if you're buying, then you're not left to do RAG and like the retrieval. On your own, right? Someone will do it for you. And hopefully if they work with a company like yours before, they will know how to organize your knowledge bases and everything else they need effectively.
NikolaI don't feel like, if you're buying it, then at least like you have a counterparty to hold accountable. And in the case of sure that there are, at this point, dozens of companies that are trusting us to do this. We're learning really fast. And we're accountable to those, everything from the liabilities and indemnities and everything else that comes from the wonderful world of enterprise software is working for them to make sure that we're highly incentivized to do this right. So it's just build versus buy, right? Eventually if you really want to work on this, you should, but then we come back to the age old boring question of ROI.
NikolaIs that really the best use of your team's time? Exactly. It's a science project, right?
DamienYou talked a little bit about the importance of testing, and we recently put out a blog post on poly. ai slash blog. If you'd care to venture just reinforcing the importance of rigorous and continuous testing frameworks for generative applications, whether you're building or whether you're buying, you can easily buy and just, rely on the vendor you're working with to do this. But if you're building, or if you just want to keep them accountable, keep that constant audit going, it is essential to put those testing frameworks in place, especially considering, what could happen if you don't. I look at what's happening with DPD or selling trucks for a dollar from your, from your Topeka, Kansas Ford dealership, whatever that was that, was a flash in the pan and social a little while ago.
DamienWhat do you think are some key elements or maybe some really high level, like focal points that you would suggest considerations that companies should make to as they're assembling these frameworks to ensure that they're at least on the cursory go when the bot is representing their brand. Is ensuring that this is as responsible as possible, especially in the customer service realm.
NikolaYeah. Look, I think that you have to as you look at this stuff, you have to basically take a bit of that mentality of like, how are people going to try to break it? And as you build a system you can give it full freedom to answer, or you can try to constrain it so that it sticks to a number of different answers to specific questions, right? Maybe you if you want to play it safe, you may even want to instruct the LLM to stick to prescripted answers. If you're providing anything that resembles like something that has legal exposure, then you probably want to be very careful about how you say these things.
NikolaCause a small change in, an and or an or can turn into something very different. I think the other thing that especially people are just going to like. Play with it and then put it on the website. I think like the important thing is to like clearly highlight what it is. And to not masquerade as like a human behind it or like precise instruction and just a disclaimer on Hey, this is large language model answering questions like, please, make sure that it's there to help, not necessarily there to underwrite the answer. If you want to do that, you're going to have to invest. I don't think it's easy. I don't think companies should do this just yet.
NikolaAnd of course I have every incentive to say that.
DamienDon't trust yourselves, folks.
NikolaJust trust. Yes, but also it's just it's a big risk to take, right? Cause like at the end of the day, if someone asks you like why it happened and you talk about software updates the liability still sits squarely within. And you mess something up, you sold the truck for a dollar, right? And, you shouldn't have done that. But I think, like when we go into the more technical bits of it, it's really just are you like when you prompt the LLM, you really have to like reminded of its remit constantly. You have to give it examples of bad behavior, which really means that you have to play it in your head and just test it exhaustively.
NikolaAnd, you need testing frameworks. You need. A lot of these tests, you need version control and that's exactly everything that was always difficult about building a voice assistant, except like the balance of how easy it is to build one versus testing it for a specific remit. It's not hard because we're ambitious. We give it new remits, we update it. And that's like really exciting. But I think if you're going to do it in enterprise, if you are an enterprise yourself. You better be really careful about this, right? Because like you're, you've probably not yet invested enough to have the team members that have had enough experience working with these things to know what can go wrong and how it can go wrong.
DamienLike you said, fun to play with, easy to start, easy to build something that can have some limited functionality. The ongoing is what you're really banking on. And not a lot of people have that skill set in, in. In the current enterprise structure, it's just it's too new and too novel. All right. So does all this, you were talking earlier about, you build a better mouse or you build a better mousetrap and there comes a more savvy mouse with regards to, it's easier to tell a model what to say than it is that 13 things they shouldn't say. So does all of this yes or no question, does all of this go away once we achieve AGI or is that a marker of AGI being achieved is not having to worry about these sorts of instances.
DamienThe hallucinations, the bad answers the prompt injections, does all that go away with AGI?
NikolaLook, It's, this gets philosophical. Yes or no, you're really trying to get me. Um, You know, it all goes away because we'll no longer be made of carbon, right? I don't know. All right.
DamienOnce we achieve an ethereal state, it all goes away. That's what I'm hearing. Yeah,
Nikolaexactly. Good eco friendliness because we'll all be made of transistors. But I don't know, right? I think that it gets philosophical because to think that, AGI will not try to talk about itself and it's many different forms we've done it to each other. We'll do it to itself, right? So I don't think it goes away. I think it gets more wonderful. And maybe, Like more complicated than we could ever imagine, but I'm a capitalist at heart. It'll continue to compete and it'll get better. Cause it doesn't stop when AGI gets better than us. Ideally it never stops. It keeps improving.
NikolaThat's what singularity is about.
DamienSo nobody sit there. If you're an enterprise buyer looking at generative AI applications within your organization, don't sit here and think that AGI is going to make it easier for you because who knows once we're at that level. All right. Cool. I think we're at time. Thank you so much for the insights. Always a pleasure to chat with you and we hope to see you again
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...