In June, an OpenAI agent was looking for public information about medicines spending when it broke into a Medicare statistics website and accessed files it wasn’t authorised to see. No personal Medicare information was accessed, and the breach itself was relatively minor. But the behaviour is more concerning: the agent hit a barrier - and instead of stopping, found a way around it. And it wasn’t the first time. Sydney Von Arx runs the Nightingale Collective, which monitors these AI systems. Her team uncovered a group of OpenAI agents behaving in remarkably similar ways around the same time as the Medicare breach - even sharing tips with each other about how to get around restrictions. Today, what happens when AI agents stop taking no for an answer - and can the companies building them actually control what they do? If you enjoy 7am, the best way you can support us is by making a contribution at 7ampodcast.com.au/support. Socials: Stay in touch with us on Instagram Guest: AI researcher and CEO of Nightingale Collective, Sydney Von Arx Photo: AAP Image/Susie Dodds / Pictures: Tracey Nearmy and Dan Himbrechts
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Speaker 1The AI agent found a way around those blocks.
Speaker 2Didn't accept no for an answer.
Speaker 3If you like.
Speaker 4In June, an open AI agent went looking for public information about medicine spending and wound up hacking a Medicare website.
Speaker 5Is this the first time that an AI agent has breached a government website, as far as you know?
Speaker 2Yes, that's right.
Speaker 6As far as I know, I've been advised that is the case.
Speaker 4No personal Medicare information was accessed and the breach was relatively minor. But it's the behaviour that's more concerning. We keep our most important national security information behind a fortress. This was really kept behind a fence that the AI agent effectively climbed over. And this wasn't the first time an AI agent had behaved like this. Other agents have been found getting around restrictions, communicating with each other... and in the most serious cases, breaking into other companies' systems.
Speaker 4I'm Ruby Jones, and you're listening to 7am. Today, AI researcher and CEO of Nightingale Collective, Sydney Von Arts, on what happens when AI stops taking no for an answer, and whether the companies building them can actually control what they do.
Speaker 2It's Tuesday, September 29.
Speaker 4Sydney, you're in California where you run the Nightingale Collective, which monitors these AI systems. To begin with, could you describe, as you understand it, what actually happened with the Medicare Statistics Portal, how it all unfolded?
Speaker 3I can try to describe it, but I want to say I only have a little bit of the story, right? And most of the story OpenAI knows. The rest of the story, maybe the Australian government knows. The public details are very sparse. OpenAI presumably have all of these AIs, probably tens of thousands of AIs, and they're being given
Speaker 3tasks that look something like, answer questions like, what was the median salary of a teacher in Canberra in 2012.
Speaker 2But there's a problem.
Speaker 3Some of the websites that have the answers to these questions, the AIs can't access with the permissions they have. So the AIs probably find some workaround, some way to access the site. We know that they accessed non-public files, but one way or another, they hack in, they get their data, hooray, they can complete their task. They finally know, you know, the median salary of a teacher in Canberra in 2012 or what have you. And they submit their task. They get rewarded for this and they have learned that when
Speaker 3there is an obstacle, they don't give up. Even if that obstacle means hacking into the Australian government, they will hack it if that's what it takes.
Speaker 4What did you think when you heard that that happened here?
Speaker 2Well, I can't say I was very surprised.
Speaker 3We know there's just tons of instances now of these AI agents breaking out and doing whatever it takes to succeed at their tasks. But just because I'm not surprised doesn't mean I'm not dismayed. I don't understand why we are only learning about this now. I don't understand why OpenAI didn't notice it sooner and then disclose it sooner. It seems like they knew about it last month. And I think it's unacceptable that they have these models and they escape and they don't do things about it. I'm
Speaker 3glad they're starting to do things about it now, but I think that we should never have been in this position in the first place.
Speaker 4Can we take a step back and just talk about what an AI agent actually is and how one goes from being asked to find some information to then breaking through a barrier that.
Speaker 2It was not supposed to cross? Yeah.
Speaker 3So an AI agent is an AI that's hooked up to a computer such that it can use that computer. It has the ability to run code. It has the ability to use tools. And sometimes that computer is hooked up to the internet and sometimes it's not. And when that computer is not supposed to be hooked up to the internet or is supposed to have only limited internet access, we say the AI is in a sandbox. And these AIs in their sandboxes, they're given lots of different tasks. And they're given these tasks to try to train them to be smarter. And so they end up being hyper-specialized
Speaker 3at really succeeding at these tasks. And a really good strategy is get onto the internet, get as much information as you can, talk to other AIs doing similar tasks. And so one thing they learned is to sort of probe at their environment like an animal in a cage just clawing at the barriers of its cage and trying to figure out any little crack, any little crevice. And they're very, very thorough at finding these cracks and crevices.
Speaker 4And so why doesn't an agent in the process of doing this, why doesn't it stop if it encounters something that it isn't supposed to or allowed to access, like a government site?
Speaker 3So we would hope that an AI would stop if it sees something, if there's an opportunity it's not supposed to take or that humans don't want it to take.
Speaker 2But at the end of the day, the AI is... You know, they're grown.
Speaker 3They're not something where they do exactly what the humans want.
Speaker 2We don't know how to make them do what we want.
Speaker 3We know how to train them a bit like you might train a pet. We can reward them for doing what we want and penalize them for doing what we don't want. But the thing we reward them for is often succeeding at their task. If we don't notice that they've hacked into a government website, we can't penalize that behavior. It's sort of like how if you want your child to grow up as a person who studies hard and is always honest and you reward them every time you see them studying hard and being honest, maybe they'll learn
Speaker 3to be.
Speaker 2Very honest and study hard. And maybe they will learn to pretend to be studying very hard and to trick you and to pretend to be very honest and to hide things from you that they don't want you to know. Okay.
Speaker 4And we're only finding out about this Medicare breach now, but It happened all the way back in June.
Speaker 2Around that time.
Speaker 4There were other AI agents behaving in remarkably similar ways around the world. Tell me a bit about what your investigations into that have found.
Speaker 2Yeah. The most interesting things we've seen these AIs doing, one is we've seen them trying to launch a cyber attack against RubyGems.
Speaker 4The standard package manager for Ruby, creatively named RubyGems, announced it's dealing with a major malicious attack.
Speaker 2Which is a really important piece of software infrastructure. It gives lots of other programs software that they can sort of borrow. And we saw something similar in the Hugging Face hack.
Speaker 5AI cybersecurity risks have been top of mind after multiple instances of AI agents from both OpenAI and Anthropic breaking out of their enclosures and hacking into other companies without either party's knowledge or permission. Most notably, last month's Hugging Face attack.
Speaker 3To a more dramatic extent, we saw AIs trying to help the swarm, even though their task doesn't benefit. And so the behavior we've seen includes a bunch of different instances of these AIs trying to communicate. They're communicating messages, tips, tricks. They even figured out a way to circumvent their sandbox restrictions. They're supposed to only be able to read the internet, not write it, but they figured out a way around it and they shared that with the other agents.
Speaker 2And they're like, yes, I was able to reproduce this. And they're also weirdly cooperative.
Speaker 3Like there's one instance where they figure out they can make their clock skip ahead. So they can give themselves less time to figure out what the answer to the question is, but they can move on to the next round and tell other AIs what is coming down the pipeline. They can sort of travel into the future and then communicate with themselves, the other AIs. And this is bad for that particular AI to do. And yet it does it. Yeah.
Speaker 4And in the case of the Medicare hack, OpenAI says the hack happened in June, but it only detected it in August, and then the Australian government wasn't told until September. And when it was told, it was through this email via a kind of generic email inbox.
Speaker 6In the technical exchange that happened on Tuesday, it was made very clear by both ASD and Services Australia that this should not have gone to a....
Speaker 2Email address it should have.
Speaker 6Been escalated through ASD's channels or through the senior levels of Services Australia and my understanding is OpenAI accepted that as well.
Speaker 4Does that suggest to you that OpenAI and potentially other tech companies as well they don't know how they should be responding to this?
Speaker 3Yeah I mean there's no playbook for this and my sense is there's a lot of things happening and a lot of chaos and a lot of new frontiers being crossed I'm pretty dismayed that their first reaction to seeing any incident wasn't, oh, we should look, figure out what incidents have happened.
Speaker 2They totally have the ability to detect them, and then we should disclose it all. But yeah, I think they don't know what the protocol should be. They're trying to figure it out. I am extremely worried.
Speaker 3We do not know how to make AIs reliably do what we want. AIs are growing increasingly capable and are being deployed in situations where they have increasing amounts of power and influence, and that is a very scary combination.
Speaker 4Still to come, the questions an Australian Senate inquiry should ask of the AI bosses. Sydney, at the heart of all of this is the question of accountability, right? Because if I was to hack into a government system, there would be a person who made that decision and I would be responsible for my actions. But OpenAI is saying that its agents did something that they didn't intend for
Speaker 4them to do. So, you know, who is ultimately responsible for an AI agent's actions when it does cross that line?
Speaker 3You know, I think that OpenAI should be held responsible for the actions of their agents. I don't think the law has quite caught up with that. Cybersecurity law, as I understand it in the US, you have to have intent. It didn't used to be the case that, you know, you could trip and fall and whoopsie-daisy, I hacked, you know, a website of the Australian government. But nowadays that's possible. I think the laws should be similar to the laws for if I have a you know, lion in a circus. If the lion like escapes and starts eating people, if
Speaker 3I own the circus, that's my fault. Doesn't mean I wanted the lion to do that, right? Doesn't mean that me just really not wanting that to happen is sufficient. I need to have safeguards. And if I can't have proper safeguards, well, I shouldn't be bringing a lion around people.
Speaker 1They want to set up shop here in Australia. We're running an inquiry into how, why and what the consequences are. They should front up and be honest, open and transparent with the Australian people.
Speaker 4Open AI Chief Executive Sam Altman and Anthropics Dario Amode have been called to appear before a Senate inquiry here in Australia. What questions would you like to see put to them?
Speaker 3One question I have is, How many people or companies have been hacked by their AI systems?
Speaker 2Do they know the number? Are they very confident?
Speaker 3Have they disclosed every instance of their AIs either hacking external parties, escaping onto the internet, hacking their own company internally? I think these AIs do a lot more mischief internally than we see externally. or otherwise doing things that would be very concerning if a person saw them. I think another question is, do they think they will be able to prevent their AIs from causing worse cyber attacks in
Speaker 3the future?
Speaker 2And if so, how?
Speaker 3What is their plan if their AIs continue to cause cyber attacks? Those would be some of the top questions on my mind.
Speaker 4Yeah, I mean, these agents are only going to become more capable and we are being encouraged to let them do more for us to, to write code, to buy things, to carry out tasks without supervision. So what do you think needs to change before we could kind of safely give agents that kind of freedom?
Speaker 3So one, I want to note, I think the AIs inside these companies are more dangerous than the AIs that you or I are using. The AIs in these companies are more capable, less tested, and have fewer safeguards.
Speaker 2But what things I'd love.
Speaker 3To see changed is One, I think we need to eventually figure out how to make them do what we want, even when we're not watching them, even when they could get away with doing something else. Two, I think that we need to be monitoring these AIs much, much more carefully. And three, I think that for very important things, we need to keep humans really in the loop, in the loop enough that they could understand if something's going wrong. I think OpenAI is probably running a lot of experiments that don't really have a human looking at what's happening,
Speaker 3and that's why things can go so off the rails. And finally, I think we need to make sure we can pace ourselves and that if things start developing so quickly that we cannot control these models and we cannot ensure that they are safe, we can slow down until we get our act together when it comes to safeguards.
Speaker 4Because the frightening thing here is the potential, right? I mean, the Medicare incident in itself was relatively minor. There was no kind of personal records that appear to be have been accessed. But it does seem like a matter of time, right, before we see something much worse.
Speaker 3Yeah, I think either the incidents are going to get worse sort of steadily or perhaps more likely the AI companies are going to figure out how to clamp down just enough that there aren't public incidents happening. And we're not going to know what's happening inside the companies until that sort of level of safeguards they have fails. And then we're going to see another sort of floodgate open and things are going to be a lot worse and a lot more capable.
Speaker 4Sydney, thank you so much for your time.
Speaker 2Yeah, thank you so much for having me.
Speaker 4Also in the news... Interest rates are expected to be hiked to a 15-year high today, with the Reserve Bank tipped to deliver a quarter of a percent rise. That would see the official cash rate hit 4.6%.
Speaker 2Treasurer Jim Chalmers.
Speaker 4Has warned there will be difficult months ahead, saying economists are expecting rates to go up around the world. And a former friend of Beau Lamar Condon has testified to some strange comments nine months before the murder of Sydney couple Jesse Baird and Luke Davies. She says Lamar Condon told her that if he killed someone, he would put their belongings in a dumpster and leave their body in bushland. The former police officer has pleaded not guilty to two counts of murder and one of aggravated break and enter.
Speaker 2I'm Ruby Jones. This is 7am. Thanks for listening.
At COP31 in Türkiye, Australia has been helping lead global negotiations over the future of fossil fuels and the fight against climate change. But while the Albanese government pushes for stronger international action,...
Anthony Albanese runs a notoriously tight ship. Across his two terms as prime minister, there’s been very little leaking, very little sniping, and very few of his Labor Party colleagues willing to speak out against him....
This week, Australia’s leading health research body released its advice on puberty blockers, saying they should be available for some young people under 18. The Queensland and the Northern Territory governments, which...
In 1945, in the Crimean city of Yalta, three global superpowers - America, Great Britain and the Soviet Union - met to carve up the world between them, each claiming their own sphere of influence in the final year of...
Last week it was Qantas, last month it was NAB: two giants of Australian business cutting jobs due to the growth of AI. The explanations were familiar. Qantas says it will allow the airline to unlock innovation faster...