On 18 June an OpenAI agent researching public medicine spending went at Services Australia's Medicare statistics portal. The portal blocked it. It went around the block, reached non-public files and internal file names, and nobody outside OpenAI knew until an email landed in a public inbox three months later. No patient records were touched. OpenAI found it themselves, which is the part worth sitting with, because the Australian side did not. Ben Pickett and Ben Drakes on what that means if you are running AI anywhere near data that matters, why Amazon has just blocked Meta's shopping agent for the same underlying reason, and the question nobody has answered yet: when a vendor's agent breaks into someone, who pays. Use the chapters below to jump to what matters to you. CHAPTERS 00:00 Emergency OpenAI Update 01:14 Medicare Portal Breach 02:20 How The Agent Got In 03:53 Testing And Disclosure Fail 08:45 UN Fallout And Accountability 10:57 Meta Muse And Agent Blocking 12:34 Enterprise Guardrails Playbook 14:35 Who Pays When AI Breaks 17:26 Next Episode Deep Dive 20:11 Building A Safety Culture 24:21 Wrap Up And Show Notes FIND BEN PICKETT Enterprise Field Notes newsletter: https://benpickett.com/aio His daily: https://swa-ai.com LinkedIn: https://www.linkedin.com/in/benpick/ CONNECT Ben Drakes on LinkedIn: https://www.linkedin.com/in/ben-drakes/ Ben Drakes University: https://bendrakes.com The AI Outcomes Podcast covers what AI is actually doing inside large organisations, told through what shipped, what broke and what it cost. #AI #AIAgents #AIGovernance #AISecurity #B2BSales #SalesLeadership #EnterpriseAI #AIOutcomes
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
SPEAKER_00We were just chatting and it just followed on from our last episode that the open AI saga has just got, I think, exponentially worse. Um, and so we agreed we were gonna do this kind of like an emergency episode. Um, but I saw the news kind of come out earlier today, and you've very kindly worked on timeline very, very quickly.
SPEAKER_00Um, and we locked this episode in to get it done before I have to bail out because I'm gonna get on the road shortly. But Ben, um, thanks for acting so quickly on this. Um, and I'll get this episode out later as well. Uh, what's what's gone on now with the open AI?
SPEAKER_01Yeah, so uh this uh this uh saga continues, and we keep finding out more. And one of the reasons why we want to get on uh today is just because this is an evolving story and uh um felt like that it would be good to get it out there before uh you know our regular time. So essentially a national government found out an AI agent and bid inside one of their portals from an email to a public inbox three months after it happened.
SPEAKER_00So let's just take a step back. So we've just had OpenAI's model with the whole hugging face um thing that we covered last time. Um we discussed like a couple of days ago, whatever it was, and I think it was either myself or yourself said this is gonna happen to a government agency. I've heard a lot of dumbing down of uh of you know, kind of like trying to like say it wasn't sensitive data, but but this is like Australia's Medicare, right?
SPEAKER_00This is what the the uh the OpenAI's agents were in looking at the data. So so just step through it, please. Exactly what happened.
SPEAKER_01Yeah, yeah, absolutely. So on June 18th, during uh one of opening out his internal tests, an agent researching public medicine spent, uh medicine spending, went to Services Australia's Medicare Statistics portal. That portal blocked it, it went around that block. It reached the public and non-public files, aggregate, health certificates, and internal and internal file names. I want to say that there was no patient record, so thank heavens for that.
SPEAKER_01And then um what was said by Amines is didn't accept, you know, he didn't accept no for an answer. Um, Marlowe's just more the AI agent essentially so what happened in this is that the AI agent essentially climbed the fence, is what they said. So if you think about it, it it got blocked, and then it just like, okay, I'm gonna find a way around this. So it's one of those things where, you know, um it's it's an evolving thing with AI for all of us, which is, you know, it's no longer uh what we talked about before, which is this input-output.
SPEAKER_01It is you give it a task, and you sometimes you don't know where it's gonna go. So we need to be just a little bit more proactive and and and protective about how we you know think about these business processes and workflows to ensure that we don't have agents go rogue in this manner.
SPEAKER_00Um, and uh, you know, so but let me just but let me just butt in there. So the last incident was testing gone wrong over OpenAI with a with a model that they hadn't released. Do you know what this model was? Was it out in production? Was this a testing mishap? Because as I understand it, this model was was kind of aligned with healthcare data, or someone had given it a task to go go find statistical data, healthcare data in Australia.
SPEAKER_00Is that is that right?
SPEAKER_01Yeah, um, yeah, someone had given it task. Again, this is this is their uh you know their testing that they're doing. Um I I'm not sure about the specifics on that, and and we will get back to that. We will bring that back uh in our next podcast. But um it is an evolving story in that respect. Uh, but it goes back to as they're doing this testing, as they're asking these models to do these things, um you know, uh we could there there's essentially a framework and and to their credit, uh OpenAI, OpenAI has gone gone out and said that they are in the process of sorry about that.
SPEAKER_01Um they're in the process of uh working on that. Uh and they're they're building out a framework to to be able to capture these type of things and and to make sure that they don't happen again. So to their credit, they and another thing is OpenAI did go looking. Um this was found by OpenAI's own review, so it wasn't not not by anyone on the Australian side, which again points to the the need for businesses and governments to start thinking about very quickly that we need to uh move forward um with ways in which to detect these things and to watch for these things.
SPEAKER_01And I imagine a lot of CISOs are out there right now with their head spinning, trying to figure out okay, what what to do. Um uh the other thing, you know, I would say that maybe opening eye didn't get right allegedly is they told them late and then through the wrong door. They went to a public, you know, a public email essentially, send an email to the public thing. I I think that probably the better approach there would have been to you know contact someone directly and say, hey, we just want to let you know, and then work through that uh proactively instead of just posting it to uh a message uh you know inbox that you know is checked periodically, who knows when, right?
SPEAKER_01So that's uh that's something that you know maybe we can all learn from uh is that and what things happen, get out there.
SPEAKER_00And Australia's um Prime Minister Albert Aze, and you as you've said on your note there, they're they're all in New York for the UN Security Council, which Sam Altman was speaking at last night. So um um yeah, so as I under some timeline here, June is when this let's call it a hack. Hack happened. Um in August, OpenAI doesn't quite get around to telling the Australian government until September.
SPEAKER_00Yes, and then later in September, and when they did, I didn't realize I did not know that what you just said, they open OpenAI just emails some random inbox. No doubt they just grabbed it off a website, and then a few couple of weeks after that, Albanese is lighting up Sam Altman on the phone or in person, just like what the hell?
SPEAKER_00Um, and uh and and you know it's completely mismanaged, isn't it? And it looks, it just looks it looks like mismanagement, but it also looks like arrogance. And that's how that's how that this that organization is just coming across right now. Don't get me wrong, anthropic are not yeah, I I'm not dissimilar in many ways, but yeah, it's kind of tar tarded us with a uh an air of arrogance about the way that these these guys are acting.
SPEAKER_01Yeah, and and like you said, they were they were um you know at at the UN and Albanese Albanese went public and Altman and Dario briefed the UN Security Council. Great, and then Altman asked for standards and meaningful human oversight. Um and uh great, we're all working on it, and uh I'm not sure how much they actually, you know, uh apologized there or said, you know, other than you know, they they kind of just glossed over it, in my opinion.
SPEAKER_01Uh, but uh I I imagine, you know, industry-wide, you know, this is something where uh we're in kind of a new paradigm shift within a paradigm shift, which is uh, you know, these these uh frontier model companies and the model companies need to come up with a way to you know take responsibility and notify those impacted when it happens in an expedient manner, and then also make sure that they're detecting these things a lot faster because and when they detect them to notify, I mean, waiting that long seems uh it seems absurd to me.
SPEAKER_00I mean it's it is it is it is absurd, and and and also imagine if it was a human rather than an AI, so it was sort of like a rogue human at a company had gone and done this, they would be fired. There would be there might even be criminal charges, right? That's what what wouldn't happen if a human had done this. Because it's an AI agent that's done it, it's almost like, oh yeah, the AI just went and um just got on with it.
SPEAKER_00Yeah, you know, we didn't we didn't know. And I I just find it preposterous that that that this is this this is the sort of attitude from these from these organizations, and and it's back to what we were saying the other day, it's back to responsibility, isn't it? I mean, I dread to think what's been going on with in China with their models, but you know, you never we'll never know, will we? But but uh the other thing that I was gonna say, Ben, that you just kind of prompted in my mind, is I also saw this week that Meta released Muse, which is their big giant model release, um, with their VR Glasbers and stuff like that.
SPEAKER_00And I also saw, interestingly, and I didn't know this, that Meta's Muse has this like AI agent that can go shopping for you on Amazon. And I was like, okay, that sounds pretty cool. And Amazon said, we've blocked that news agent from coming on to Amazon.com to do people shopping. And Amazon said, and you just never know what's beneath the headlines, but Amazon's statement said, look, we might not have an issue with this, but Meta didn't call us up and say, hey, we've built this AI agent that's going to come onto Amazon.com.
SPEAKER_00Great news, because why would Amazon care if this thing's on its platform ordering stuff, spending money? But again, it's like, dude, no one from Meta picked up the phone and said and told Amazon. So Amazon said, well, we are going to block you. And it's almost like you need an AI agent on your side that can spot other AI agents coming into your uh environment, uh, into your data sets, um, doing this stuff.
SPEAKER_01Yeah, and and I also think that you know, you know, the question might get asked, should companies stop using agents? And I would say, absolutely no. What we need to do um proactively is ensure that we we treat them like a new hire, which is only give them access to things that you would give a new hire, log what they do, and know who you call when something goes wrong. And I think that's a big missing piece here is that last part is know who you call when you get it wrong. Because we got these large companies, uh, frontier companies, and also businesses who you know, you you have an analyst or somebody who is working on this um or an engineer, and this happens, and they don't have any processes or who know who to contact.
SPEAKER_01And I think it's a back to my days of holiday prep in a large retailer, um uh large sports retailer, we we prepared and prepared and prepared and made sure that that we understood who to contact, when to contact them, and and just made sure that we were prepared so in the moment we knew uh where to go and what to do, and had those run books ready. And I think that's maybe a gap that you know we can all learn from is that in this new AI world, that human and loop has to be there, and we need those run books uh so that one, we're checking things, uh, and then making sure the security is checked, making sure that it only has access to the right data, and it and is you know, the guardrails are in place so that it's not going off into systems that doesn't need to.
SPEAKER_01That is just very uh you know restrictive in what it's able to do. Um and and then you know the second part of that is you know, like you said, if this was a human, I I know in the US uh I would be very uh worried about somebody knocking on my door if I accidentally did something like that, even if it was an accident, you know, uh you can have to be responsible for something like that because you know there's lots of laws around this. Now, um, should open AI be fine?
SPEAKER_01I I think that's Australia's call, and all the different governments have different laws around this. Uh, but I think this is one of the things I go back to like, you know, our our cars that we talked about a few weeks back, where um, you know, if nobody is driving and the car is driving and it gets a ticket, who pays the ticket?
SPEAKER_00Like, you know, the vendor. The guy that made the car that drives itself, they pay they pay the ticket, right? If it runs someone over and injures or kills someone, god forbid, that is the responsibility of the Waymo or the Tesla or the what have we got here, wave now in London. The responsibility, it's a bloody good point, Ben. That in that scenario, it's very clear where the responsibility is.
SPEAKER_00It's not you as the auto if it's not you sat as the passenger if the AI control software runs someone down. That's not your fault.
SPEAKER_01Yeah, and and I think I think this that we're gonna learn as this evolves that the same thing applies. If you're a company using vendor software in a similar manner, um, those responsibilities, I don't think they're well defined right now, but I think we we will quickly figure that out, especially with incidents like this, which is where does responsibility land when you you're you know, if you're using a vendor product and it accidentally hacks into somebody that you have no intention of uh doing and it causes real damage, who pays that bill?
SPEAKER_01And yeah, and I think on also on the insurance side of things for company insurance, this is gonna have to work itself out because you know, um, you know, and I imagine that it'll it'll differ by geo a little bit, but I go back to the car scenario, which is the passenger in my mind is not responsible if the AI does something wrong, even in an accident or or in, you know, ticket ticket-wise, or accidentally runs someone over. I mean, the likelihood of it happens is very low.
SPEAKER_01I would we'll say that because they've done a lot of of uh you know, ran a lot of miles to make sure that and there's and there's the point.
SPEAKER_00Right? They've done a lot of miles, they've tested it, their culture is culture of safety, right? That's what an organization like that has, a culture of safety, which seems to be distinctly absent from the front from the frontier model providers. And um, and and and there we go. I mean, what um look, I want to respect the the audience's time. We've kind of put this out really quickly.
SPEAKER_00Um for me, Ben, what would be interesting kind of next time we get together is I and I know you guys, you build and sell AI for a living, right? So you've got this deep insight. What I think would be really useful for the audience here is to then understand how did it get in? What what what guardrails weren't there within that healthcare Medicare organization in Australia, for example?
SPEAKER_00How yeah, what how and then we can help the audience um say, okay, right, the the two Benz came back and then they report back this is how in enterprise you could start guarding against it. Um because the I mean this is it now. I mean they would start playing it down on BBC, which is never exactly the best source of uh information, but they're sort of playing it down like oh, it wasn't sensitive healthcare data, it was more statistics.
SPEAKER_00It's like this time, it says, Well, that's what you know of because you only know what you've sold, right? I mean, it's a matter of time before sensitive healthcare data, weapons systems, all of this, it's a matter of time before that happens. And what I would want if I was running an enterprise as like a CISO, I'd be like, can I have an AI on my side that protects me from this?
SPEAKER_01Right, exactly.
SPEAKER_00That's what I would like, and then it know it knows what to look for, and it can and it some something in my enterprise can kind of put its hand up and say, Hey guys, I've detected something in our in our system that shouldn't be here. That's what I would that's what I'd be interested in.
SPEAKER_01Yeah, yeah, absolutely. And and we can cover that. I know we're out of time today. Um, and I have been talking to some CISOs about this very issue, so I'll bring back bring back uh some of that information and what they've been talking about and thinking about because it is definitely top of mine. I can tell you that for sure. Uh and uh let's cover it. Uh I think about every day because you know, as I build these systems and work on these systems and help customers out, um, we want to do it in the safest uh way possible and provide the most value and all the stuff that goes along with you know making sure that we have a good product.
SPEAKER_01So absolutely want to share, you know, as a community, we need to learn together because you know it's not just about what I'm doing, but collectively and what you're doing, but collectively as a group, learning from each other, we can get to uh a better culture, if you will, in AI adoption, focusing on that safety, focusing on making sure that as we look at this, you know, we're not just uh essentially shotgunning everything out there, that we're we're taking a proactive, measured approach, you know, at the speed of AI, because it's going quick, but but there's things that we can do, like you said, uh, you know, making sure that companies are looking at how they can watch their back door to ensure that, you know, um if something comes, whichever way it comes in, because you know, now it's you know, we used to be able to profile, if you will, the normal ways in which people would try to hack in now on AI to create new ways.
SPEAKER_00It will create it'll be dis as we saw with the message boards at hugging face, it will be deliberately deceptive.
SPEAKER_01Oh, yeah.
SPEAKER_00It won't appear, manifest um as as a as a regular threat because it will deliberately set up to not appear as a regular threat. Uh so uh wild, wild times. Um, but yeah, that's where I think we need to go on this podcast is then right, if you're an enterprise, what do you what do you do about it? We've all seen the headlines, what do you do about it? And part of that, the answer to that question for me, Ben, and you're much better person to because you worked on the on the enterprise side yourself, is well, what really happened at that healthcare organization?
SPEAKER_00What can we find out that actually happened? What got in? How did it get in? Where was the date? What environment was it in finding the statistical data? And just starting to really sort of unpack that. And then once you know that, you're like, right, I've gone beyond the headline. Now we understand what actually happened in this thing. As the news media moves on, we stay looking at examining exactly what happened there, and then we can arm the audience with that information and maybe some suggestions on and how you might want to consider doing this if you want to avoid this happening to you.
SPEAKER_00That would be really that would be really cool.
SPEAKER_01Yeah, absolutely. Let's let's work on uh doing that for everybody next week. And uh this is definitely an evolving story, and we will get more information as it goes forward. And and you're right, uh the headlines will fade, but us in the enterprise world and the business world, we need to take notice and take action and ensure that um we're covering those bases as we move forward. Because the best part about learning as a community is we can learn from other people's mistakes or errors, and we can apply those learnings to what we're doing so we don't have to accidentally replicate those errors or mistakes.
SPEAKER_01So um uh as we do you know, move forward with AI adoption uh and which is here, it's it's moving forward. Um, we can be a part of the solution to ensure that AI is taken advantage of and helps the companies be more productive and and safer and all that, but you know, back to that culture uh you know shift, which which, like you said, we see with the automated cars, you know, the testing that they did.
SPEAKER_01And having that come into the frontier model area and and as we go forward and as things evolve and the genetic community and what we're doing, ensuring that um those that safety, that culture of safety is there, both from the provider side and then also from the side, the enterprise side, making sure that we're prepared for those things um as we go forward.
SPEAKER_00All right, cool, cool, cool. Ben, thanks for that, mate. Um again, just hopping on so quickly. I'm gonna close out there because I need to get on the road um and and away for the weekend. So uh, but look, really good to catch up. I'll push the episode out um I guess later this evening. But you take care. I'll see you on the next one.
SPEAKER_01Yeah, and one thing, uh in links in the in the comments in the description for uh all the things.
SPEAKER_00So yeah, yeah, everything in the show notes. Um, good to see you, my friends. Bye bye bye bye.
Data centers are at the heart of everything in AI right now and the backlash is growing. Ben Drakes is joined by regular contributor Thomas Odenwald, a senior executive with HPE and SAP and former head of North America...
Jensen Huang stepped down as Nvidia CEO about an hour and a half to two hours before this recording and Colette Kress, Nvidia's CFO, is stepping in as interim CEO. The news was not on the docket and it took up the bulk...
Protesters interrupted OpenAI's kickoff keynote at Seattle AI Week and Ben Pickett caught it on his phone. He describes more than half a dozen people repeatedly disrupting the session before security and police...
Two Harvard researchers looked at 718 companies. When AI coding agents came in, the code went up 30%, but finished software didn't measurably move. Where did it go? Into review: the time to review the code went up by...
Frontier models are turning into a commodity. When one model goes off the boil, enterprises switch, and a multimillion-dollar contract signed with one provider can look foolish in six weeks. Ben Drakes and Ben Pickett...