025: Mythbusting generative AI "threats" (and how to cut through the fear-mongering)
18 minAudio
Laura debuts hosting in today's episode, which discusses various concerns and myths around generative AI, particularly addressing Gartner's prediction of a customer's death due to bad information from AI chatbots. It also delves into real-life scenarios such as the Air Canada lawsuit, mishaps with Google Gemini, and the overall impact of generative AI on public confidence. They offer tips for organizations looking to implement generative AI, focusing on behavior testing, risk mitigation in regulated industries, and data utilization. Follow PolyAI on LinkedIn Watch this and other episodes of the Deep Learning pod on YouTube
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Nikolaif you have legal language around how you have to communicate a message. Then you don't let the language matter freestyle because an and an or can have like very different implications and you might be held accountable and then you have Air Canada, right?
LauraHi, welcome to another episode of Deep Learning with Poly AI. My name is Laura. I'm in product marketing at Poly AI and I'm joined CEO and co founder, Nikola. So we're going to be talking about a few myths of generative AI, and I'm going to start with a very spicy headline that's come out recently from Gartner. So they've made a prediction that by 2027, a company's generative AI chatbot will directly lead to the death of a customer from bad information it provides.
LauraHow do you feel about that? Do you think that's a real concern that could actually happen? I think
Nikolait's really good clickbait. That title probably got them exactly what they wanted, which was more traffic, right? Look, I think that all technology has risks, right? We man invented fire and many have died in fires, right? It doesn't mean that it didn't lead to the advancement of civilization and better lives overall, right? Same is true for Knives for cars for all sorts of technology. So to say something like that is scaremongering and, badly used technology can have negative consequences.
NikolaAnd could it have a fatal outcome? Well, there's a lot of us. So highly improbable things can still happen, occasionally, has it happened already, did a set of unforeseen circumstance and use one thing or another lead to someone's death perhaps, there's nothing gen AI specific that makes this a technology that cannot be controlled. And well, I think that, it's what who's responsible? Well, like the one who implemented it, who didn't test it, could it be an inadvertent accident have had people die from bad advice they received from a customer service representative?
NikolaWell, probably. Right. I think that claim is as true as as the one they made. And it's like when we see. An accident from like a car that is autonomous, right? We pay a lot of attention because sometimes it's different from the kind of accident that would happen if a human drove that car. And yet, quarter million people die every year on Chinese roads. And I think the math is pretty clear around if we had fully autonomous roads. And we implemented them there. Like we probably have fewer debts. And the remaining ones that would be fewer would still be caused by AI.
NikolaSo like the AI caused those, I don't know, like a hundred thousand deaths instead of the quarter million. It's still fewer. And I think playing God and, the classical question of what should a car do if it's about to hit someone, on the other side, there's like a different group of people and how it should behave. These are like. Moral dilemmas that have no right answer, but I think we will be using this technology because the advantages far exceeds the disadvantages, right? There's a lot of potential. We'll use them and are we going to get something wrong? Yeah, for sure.
LauraIt's a very interesting topic. I have been talking to a lot of our prospects at poly AI. So I've been hearing their concerns that they're bringing to us and asking us about generative AI. So one of them I don't know if you've heard about the Air Canada lawsuit that has just been ruled not in their favor. So they lost that lawsuit. So what happened? No, I just want to make sure I get the information correct. So there was a customer that had already traveled for a bereavement for a funeral, I believe.
LauraAnd then they talked to the chat bot. The chat bot said, even if you've already traveled, you can claim that back. That's fine. That's not correct. So the customer sued Air Canada and has won. Air Canada is trying to say. It's represents, including its chatbot are not, they do not represent the company and the company cannot be held reliable for what they've said. So now the company that we're talking to and selling our product to is nervous about this.
LauraSo how would you respond to that?
NikolaLook, I think companies are trying to weasel out of promises made by their representatives automated or not for an eternity. And it's just it's going to continue to happen. I think this is just You know, a mishap and, should it be held responsible? I don't know. It feels to me like the court did the right thing, right? Which this person acted and incurred the loss because of information they received. If it's all documented and clear, I think that it makes sense that they bear some legal responsibility. It's not even about AI. It could have been a website, which reads. Falsely or as depending on the context to interpret.
NikolaI think we're getting like a lot of speculation right now where people think that hallucinations will explode and these systems be saying completely random things. And I encourage a lot of. All of those people to play with chat GPT on their own and just see how it works. I think that, we, you don't trust everything on the internet. I think you could have very easily gone, 30 years in the past and said, it's terrible. People read a lot of false information. And the truth is now you're just like very skeptical about everything you read. And I don't know if I think it's fine.
NikolaI think that you will have these situations the same way you did with like the web. And I don't think it's a particularly big concern because you can make these things precise and you should design them so that, you know where you put in the knowledge base. But the real problem here is just the source of truth and it's organizational. If you have all the information for a human. It won't be that difficult to get it to work with a generative AI system. Now, if you have something that's off in the database and you don't specify it and you're not clear enough, you can confuse someone and maybe you'll be held responsible.
NikolaAnd that's the cost of doing business.
LauraSo another one of our concerns that I've heard is a customer who, that if one of their customers calls up and says, I have a problem with products, the the generative AI chatbot will go and look on sources like Reddit or reviews or things like that and find other people who've had this problem and come back and say, you're right there, there is a problem with this product. We've seen it before. What do you think of that? Do you think that's a real issue?
LauraIs
Nikolathere a problem with the product? I don't know, right? The whole kind of like scraping and gathering information from that, like this, the reason that like rag is used to retrieve augmented generation is that the complexity of application design largely moves to again, like pointing the LLM to the right place. Source of truth, right? So if you have nowhere better to point your agents or your AI, then read it to find answers to questions about your products, then you're probably going to be in trouble because you can find all sorts of things on Reddit and I don't think you really want to use it as as your source of truth.
NikolaSo yeah, I think that basically you have to design the application in the right way. And I don't think that's a problem. The interesting thing is just when. You get into situations that are unexpected. These systems tend to be very strong and precise. You can always misconstrue what the system said if you want to, like act skeptical and, you might genuinely be confused as well sometimes. And I think that's about user experience and design and how you answer questions, what style the real problem comes when people ask something completely unexpected, right? That's where the fact that these, Systems have been trained on so much data basically puts you in situations where they might even know the answer, right?
NikolaIf we were talking about the history of, I don't know, the Rosewood hotel in London, right? And you said, Hey, the hotel like built, system might just start telling you about like late Victorian architecture in London. And it'll be true, right? Maybe you didn't intend that to happen, but when prompting and building these things. You inform them what you want to do, but I think it becomes quite important to inform them what not to do as well, right? So there's like the positives and the negatives that you need to provide so that you create the guardrails around how the system will behave.
NikolaWhen you over constrain it, then you get things that don't work that well and you lose a lot of that flair, right? Around what LLMs can actually do, but if you don't constrain it at all, then you might get just completely random questions, right? Cause if you're not even, you can take the whole thing on a long discursive narrative. And in the end, you're talking about something completely unrelated. Now, we build systems, fully generative ones, which still have a meta level control flow over them that kind of dictates the world of what's possible or what's not possible.
NikolaIn a given part of the conversation and that like problem of dialogue management, right? And like having a notion of like, Hey, where am I in the conversation? What is the right source of truth? If I'm answering questions about, I don't know, flight policies, that's different from the part of the conversation where I'm rebooking travel, right? I think you just have to clearly define like what. Is in your knowledge base and where you perform function calls, what you do with them. But if you do it well and you test the system robustly, I think it's going to be pretty hard to really extreme examples of generative systems, misbehaving and ways that people fear.
LauraSure. So on that topic I'm going to bring up another spicy thing that's happened with Google Gemini. So Google rebranded, but to Gemini that just started out, and then they had a pretty big mishap when they were changing users prompts in the backend before it was to the system. So it was producing images that were. fairly problematic. Yeah. So that's just happened recently. How are companies still getting it wrong? Like what I'm actually not entirely sure what their what their QA processes are and what their testing processes are.
LauraI
Nikolathink it's really hard to release a product like this. Cause you're damned if you do and you're damned if you don't. Right. So I think the example that really got people riled up was. There were, there was a query around generate a photo of 1943 German soldiers who were then drawn. They were very, very diverse, multiracial, and not really historically accurate. So I think they really backpedaled there. I think, they, it's from a position of care.
NikolaFor the world that they probably have filters around, problematic here in the context where you probably don't want to have your tool used to generate that kind of stuff. But they tried to filter it and create a non problematic thing. And in it, they almost created something that was comical. And they backpedaled really, really quickly. And they're really trying to control how people use it. More so than what it does. I think with open AI, you see a lot more of just like flat out, like refusal to do this. I get annoyed by open AI all the time when I say I was trying to generate an image where Paddington and Winnie Pooh were eating hot pot.
NikolaRight. And it's like, I cannot generate, like trademark blah, blah, blah, blah, blah. So you go can I at least just generate two bears eating hot pot? But I think, they control it at just like flat out, like, I'll refuse. I think what Google tried is basically UX thing, right? If you think about how. They're coming. Well, they're behind opening. I write what if maybe they could make you just like me happier, but just like generating things, but they'll change it up. So it's not problematic. Right? And in that filter, they just didn't really expect this. I don't think we can laugh at them and be like, Hey, they got it wrong.
NikolaI think it's just there's the challenger now, which I think is a very Yeah. Interesting place for Google to be because, they were and are still like a leading company and I think they're a bit confused, right? Like it's challenger came in, knocked them out. They're coming in for a rematch. Their confidence is shaken. They're performing. They're doing like, things that they're not very used to doing. And these things happen. I think. We'll see. We'll see how they went again, how they rebrand and what they do with the whole thing. But it's interesting, right? Because you definitely want to control, on the other side of the spectrum.
NikolaYou've got Elon and X and, you've got Mistral, which is also a lot less filtered, right? I think that's probably the better approach. But equally, they're catering to really wide audiences. So I think it's really hard to fit the specific needs of all those different personas and create a thing that people won't construe as offensive or, comically constrained or, like something that looks like it's just nannying you and changing the way you behave. It's basically an adult filter that they try to add, right?
NikolaExcept their filter put, Rose tinted glasses on problematic content and that's just, okay, like mistakes or what? Let's move
Lauraon. Yeah, it gets into a tricky censorship topic there if you go too far into it. When companies make these mistakes so Google's made this misstep. We've talked on the podcast before about DPD one of their customers got the chatbot to write a haiku That criticized them. There's been a few others, like the Air Canada one I spoke about before what do you think that does like to, To the general confidence of the public about generative AI and using it.
LauraDo you think people are still excited about it or is it worrying people more?
NikolaI think we're in a bubble where we care a lot. So it's really hard for us to say, like we're very sensitive to it either way. I don't, I don't think so because I think the. What we're putting out at scale with these things, the number of like really good experiences and things that people couldn't even imagine doing before that's happening, I think far outweighs these examples. Of course, these examples are like sensationalist and I think, they'll probably lead to slower adoption in some industries that are a bit more risk averse, but that's always been the case with any technology, right?
NikolaMachine learning is not like a new thing, right? We've had probabilistic systems that, work or not work in specific situations. These ones are. There's a bit more autopilot in them. At the end of the day, it comes down to legislation. I don't think anyone's really going after, well, there's the EU AI act, but we'll see how that plays out as long as the planet is not aligned around doing it in most countries or not, I don't think we have much to worry about. I think we'll make a lot of progress.
LauraOkay. So if we can get to some advice that you would give, so if a company is looking to implement generative AI what should they be looking for? So if they're partnering with a third party, like ourselves, what are the questions that that they should be asking or the things that they should be looking for that this company offers.
NikolaI think, in our case, it's really about like, how do you do behavior testing? How do you safeguard around things? What is the process all the way from, like unit tests to, jungle testing and just like, how do you red team, the whole solution, right? And just try to break it in as many ways. And they should try to do it themselves, right? I think what's funny about a lot of these problems is I think you could have figured it out on your own relatively quickly, right? So I think, the other questions go a bit deeper, right? It's like, what do those companies actually do with your data?
NikolaHow do they use your data? Who are they sharing it with? But these are questions that Your dpo your data protection officer is already asking regardless right and your it teams and stuff. So yeah, I think it's You know the level of usage and self serve that you're going to have relative to others It really depends on what you're deploying jenny for it's it's a general thing, right?
LauraWe've deployed our product in incredibly regulated industries with really big companies. So what are the things that our team teams have done to make them like to prove that risk mitigation before they push that out to their thousands of callers a day?
NikolaYes. I think you have to do a fair amount of just like educating yourself around how these things work. So for a heavily regulated enterprise, what tends to be a pretty good fail safe is implementing a system that doesn't do freeform response generation, but is instead kept to responding with one of a number. And the number can be high, 10, 000 different responses to things. But, if you have legal language around how you have to communicate a message. Then you don't let the language matter freestyle because an and an or can have like very different implications and you might be held accountable and then you have Air Canada, right?
NikolaSo I think there, it's just important to like maybe when you produce outputs to have complete self-contained things. It's a bit, and you've been in phone calls with people where they're like, Hey, look, legally, I have to like, read you this and then they start reading and you're like, what that's a lot. And you have to implement systems that will do the same thing. So in regulated industries, you freestyle a bit less. That also means that the product iterations. Are slower, right? But it doesn't mean that they have to be slow, right? Because I think companies, we work with some very large enterprises, utilities, banks, where, they've been on the frontier of applying these things.
NikolaAnd we've tested it thoroughly. We've created behavior that is never Off brand it's never freestyling and yet it's very flexible and versatile So it can be done. It's just you know, there are a lot of pieces to it.
LauraGreat. I think that's all the questions that I have for you today. thank you so much for your time. Nicola and we will see everyone next week.
Three years ago, Simplyhealth's PolyAI deployment was only resolving simple FAQs over the phone. Today, they run voice, in-app, email, chat, and WhatsApp conversations on the platform. Nikola Mrkšić talks with Paul...
An agent that gets better every day compounds into a very different agent within months. That's the idea behind Wren, PolyAI's new release: an agent whose job is to make your other agents better. Some of what it finds...
Companies are pouring money into AI, but not everyone is getting the returns. Ian Jacobs, VP and Lead Analyst at Opus Research, thinks he knows why. In this episode, your host Nikola Mrkšić sits down with Ian to work...
When PolyAI built its Builder MCP, one client helped shape it: Vixxo Facility Solutions, one of the largest facilities providers in the US. In this episode, guest host Damian Sasso talks with Vixxo's CTO Derek Neighbors...
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents. Most voice AI either...