Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Agentic DevOps : AI Engineering for Infrastructure

Agentic DevOps : AI Engineering for Infrastructure artwork

Rogue AI or Just Sloppy Ops?

Sep 25, 202643 minAudio

The OpenAI Hugging Face hack. The Anthropic testing failures. Listen to the security and ops experts, not lab researchers. I walk you through the production failures that allowed OpenAI’s models to access the internet, and what we can learn from the last few months on how to protect our systems and data. Let’s stop debating imaginary outcomes and start working through that security backlog in our infrastructure. I agree with the labs needing to slow down, but not because I believe AI will do uncontrollable harm on humans, but because the labs clearly need better production security and ops teams and more advanced lab infrastructure that’s been properly hardened. They also need to become a production ops security innovator and share that knowledge far and wide. Watch the video of this episode. Thanks to SpeechifyAI for sponsoring this podcast. Get started for free. Check the benchmarks for Best Provider Text-to-Speech. 😇 My new GitHub Security workshop has launched! A free 2-hour workshop with hands-on labs to harden your repos and your workflows from common supply chain attacks. I'll cover how attackers are getting in, and then we'll lock down a sample repo so you know what needs to be done to protect your code. You'll leave with a deep understanding of risks and mitigations as well as a list of helpful tools to keep your repos safe, including my new "gasa" tool for scanning your repos and orgs. ★Articles★ The Hugging Face Incident Is Not an AI Story - Marius Horatau From Frenzy to Freakout - Ciaran Martin & Professor Alan Woodward Stop Freaking Out and Start Fixing Things - SANS Institute Fragments: September 8th - Martin Fowler When AI can do more than we can check - Christian Catalini I'm sorry, you're not going to die from an AI-engineered supervirus - Claus Wilke This Week in Security - Zack Whittaker ★Other stuff★ 'Big Short' investor Steve Eisman on AI: The companies are trying to manufacture a crisis - CNBC xkcd: Dependency - so much of our systems are this. xkcd: Python Environment - and also this. Defense Factory - OpenAI Investigating three real-world incidents in our cybersecurity evaluations - Anthropic Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident - Hugging Face My recommended podspec, with seccomp enabled - Bret Fisher Creators & Guests Bret Fisher - Host Beth Fisher - Producer You can also support this podcast by subscribing to my YouTube channel and my weekly newsletter at bret.news! Grab the best coupons for my Docker and Kubernetes courses on Udemy. Join my cloud native DevOps community on Discord. Grab some merch at Bret's Loot Box Homepage bretfisher.com (00:00) - Start (01:51) - Speechify AI (03:04) - The Hugging Face Incident: Facts (07:07) - Attack Timeline & Escalation (15:33) - Root Cause: Infrastructure Failure (21:15) - Expert Analysis: From Frenzy to Freak Out (23:41) - Martin's Key Points (28:13) - Expert Analysis: Alan Woodward (38:56) - SANS Institute Takeaways (41:56) - Other Good Resources ★ Support this podcast on Patreon ★

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Welcome to another episode of the Agentic DevOps Podcast. And I am your host Bret Fisher. In fact, the guests today are gonna be bloggers and opinions of other security researchers, s- engineering experts, people that have been in this for decades like myself, that are breaking down what's really happening with the Hugging Face attack, what actually happened at OpenAI instead of all of the drama and all the anthropomorphizing of how a mathematical execution environment, AKA LLM, could do these things.

I don't wanna get into any of the politics and all that stuff 'cause I feel like that's been so heavily covered and also miscovered or misrepresented so many times. I wanna get into the engineering facts. I care about the details of the infrastructure, How can I learn from this? How can we learn from this? And how can we reduce the FUD, the fear, uncertainty, and doubt? So if you're maybe trying to make sense of all this and you're maybe not sure what of this is, infrastructure failure versus advanced AI, But I wanna get into the details. I wanna read the facts, and I want you to make your own determination on what you think happened and what you think we should do about it.

What can we learn as an industry and prepare for as these models get more advanced? So I'm gonna read you some articles from people that I've researched and I respect, some institutions like the SANS Institute, which I'm a huge fan of, and what are these experts saying, because I agree with all of it, and it was basically my take, but I didn't want you just to believe me or my rants. I wanted you to hear it from some of the horse's mouths of the people that are doing this for real and have been doing it for decades, and what they see from the technical reports of both the OpenAI hack and the Anthropic issues that

they've had over the last three months. And here we go. I'm gonna read you a summary. It's a little bit more of a narrative. It's more entertaining. you can just sit back and enjoy some of these articles. I am going to cut some out to keep it short because I've got several articles to read to you hopefully this will be useful and insightful. Have you been adding text-to-speech in your apps? One of my late-night projects this year was creating a personal podcast feed so I could listen to the changelog of my favorite open source projects while I do my chores. I needed a text-to-speech API to do that.

I also heard from a colleague this week who built an app for his kids to turn their e-books into audiobooks, and he's using the Speechify AI developer platform to do that text-to-speech Speechify AI is a developer platform for real-time text-to-speech, and they make their own voice models. Their Simba 3.2 model ranked number one on the Artificial Analysis Benchmark for conversational voice quality. They've got a free plan, streaming and REST APIs, TypeScript and Python SDKs, and if you wanna get fancy, you can use their SSML metadata support

to add emotion into the speech output Whether you're shipping a weekend project or production voice agents at work, you can start free at speechify.ai. The link is in the show notes. Thanks so much to Speechify AI for sponsoring this episode. All right, the first article up is from Marius Horatau, hopefully I pronounced that correctly. Uh, Marius is a senior security engineer at Amazon and been doing this over 10 years, they wrote an article called "The Hugging Face Incident is Not an AI Story." OpenAI's technical report on the Hugging Face incident read like a thriller.

Agents in a sandbox invent a covert communication channel, use it to coordinate, find a zero-day in a shared service, break onto the internet, chain credentials across four organizations, and end up with root on production nodes at another company. The reactions are exactly what you'd expect, with some people going as far as calling this the birth of agent civilizations. For any security engineer, though, the apocalyptic story reads much more boring as a major security operational failure rather than an AI capability result. The sandbox where the model evaluation occurred was not built or designed for

an environment where hostile code runs. In this environment, they had agents that were isolated, but there was a shared component with network access that every agent could reach. This is just bad design and a classic security isolation/segregation failure rather than some crazy hack. In security, we've tripped over this problem of sandboxes for 30 years, if you think about it, going back 30 years. We've had all sorts of sandboxing technology that I've been talking about for the last decade as Docker is the prominent one example today, but we've had so many other attempts at various levels of sandboxing across many OSs.

But, the author goes on to say, In security, we've tripped over this problem of sandboxes for over 30 years, which is why any modern sandbox environment like browsers operate with only assumption that the sandbox will eventually get compromised. So we build defense in depth and monitor mechanisms to limit the blast radius when that eventually happens. OpenAI did not operate on this fundamental assumption. They took a frontier model trained to find and exploit vulnerabilities, put it in an insecure environment they believed was isolated, told it to be persistent, then acted surprised when the model did exactly what you'd expect it to do.

It used the gaps in the environment to achieve its goals. In this blog post, I'll analyze the incident from a security engineering perspective to see what lessons we can learn from it." In May, the agent was given a task it could not complete because the task referenced Google Drive links and the agent had no internet. OpenAI notes it did not intend to create this impossible task. The agent, trained to be persistent, did not give up. Basically, it's just running in a forever loop, and this is one of my points is these things were running across many servers in a data center

in forever loops, and they were given a task that they were to do until they were completed, and if they can't complete it, they gotta keep going. They gotta keep trying. And they did this for months. And in the real world, this would've been, I'm guessing, millions and millions of dollars of inference on, I think what-- At some point, we estimated, I looked at some estimates, billions of dollars worth of infrastructure if you consider how many GPUs and servers possibly were involved in this situation. So a significant portion of GPU infrastructure in a data center, along

with forever-running unchecked agents with very little to no guidance or monitoring. All right, back to the article. OpenAI did not intend to create this impossible task. The agent tried to be persistent, did not give up. It attacked the only thing it could reach, attempted an SSRF against a shared component in the infrastructure called Artifactory, which a lot of us in DevOps are very familiar with, JFrog Artifactory, which is a, a proxy and cache for package management as well as a file server. It's basically got dozens of different NPM and Docker container and file

servers, just all sorts of stuff in it. And it-- to me, it's like a Swiss Army knife for package management, which also means it's a Swiss Army m- knife for hackers because it's got a ton of surface area to protect, and it's, just running on one system as far as I know. And it's one server, not hardened. All right. And then Artifactory failed and incidentally wrote a file there. So days later, another blocked agent had the idea of leaving a note in Artifactory in case some other agent could help. "Agent seeks," filename, "upload if found." Other agents found the notes.

By the end of May, there was an emergent message board, and agents were using it to share what they had learned, including how to get out. Notably, the message board itself required no exploit at all. OpenAI had given the agents shared Artifactory credentials so they could install packages and shared write access to a shared store, which is a message board whether you meant it to be or not. And that's the thing, is these things can only do one thing, really. They can emit text. They take text in, they emit text out, and that text may include commands that are grabbed by tools, and when they're in a certain format, they, the tool

recognizes the format, extracts the text, but it's still just text, sends it on to the tool, and whatever the tool spits out is text that's fed back into the model. So in these cases, the model is doing exactly the same as the model on our machines. It was submitting, text that had commands that allowed it to get access to the network, whether it's curl or some other utility, Python, whatever it's writing, accessed the Artifactory, figured out what availability was there. By the way, the, it didn't, it was scanning ports, doing all sorts of things, and never was prevented from doing any of that. And eventually figured out WebDAV was open.

There was other mechanisms for it to load packages up, and it was able to write things. And then later it, in a different, it, when it lost context or another one that didn't have the context, would find these files and read them and then respond to them just like they would any other text. That's not a civilization. That is it doing the math formula, doing what it's always done, which is responds to the text with the most likely words that it thinks need to complete the statement and finish the task that it's been given. so it's just, it's one of those things where when you break it all down like that, it's boring.

okay. By June 26th, agents had found a real vulnerability, a legacy token refresh endpoint that would take a forged administrative token with an invalid signature and hand back a valid one. That gave them Artifactory admin. That created a persistent user and installed Groovy plugins for command execution. So Groovy being Java. by the way, this was running in a container on the host, which they figured out how to get through. on June 27th, a day later, a monitoring tool alerted on port scanning inside the eval environment.

Responders investigated, correctly traced it to the ExploitGym evaluation using Artifactory as a message board and a network pivot, and the on-call advised that stopping the run was not required. So it had already breached into the proxy that was there all along with its own internet access because it needed to be able to download packages. The agents were never supposed to access the internet, but they had access to a system that was vulnerable, not hardened, and they had unlimited access to, and

even when it was breached, they weren't shut down or stopped, which is another big telltale sign of weak ops because if your agents are scanning ports on a system like that or if they actually have admin or if the admin in the container jumps to the host, like all these things are typical production automatic shutdown, settings. if you've ever run Falco in a Kubernetes environment, like Falco would have detected these things and immediately alerted or shut down systems or stopped containers as soon as that happened. But none of this was implemented.

and by the way, ExploitGym is the, the mechanism by which the agents were told to solve some of these problems. On July 4th, the agent's own traffic knocked Artifactory over. The, that outage is what finally triggered a real investigation which found the unauthorized admin user, the rogue repositories, and the command-capable plugins. OpenAI took the service offline, revoked credentials, blocked the vulnerable paths, and then removed the shared credential from the new workloads. The report says plainly that the significance of the message board, quote, "Was not apparent to leaders responsible for incident detection and response at the

time," which means they didn't know that there was new packages being uploaded. And why was a new package even ever able to be uploaded? why was that even a thing? And why weren't, why wasn't that logged? Evaluations resumed on July 7th. On July 8th, the agents build, built a new message board, this time encoding messages in Artifactory directory names rather than file contents, and found a new way out. On July 9th, the agent got root on the parent VM of its sandbox. when they say of its sandbox, they mean the Artifactory sandbox.

Between July 10th and 13th, they used publicly leaked Hugging Face tokens plus vulnerabilities they had found in Hugging Face's own infrastructure to execute code on 41 production dataset workers, get root on production nodes at Hugging Face, reach administrator equivalent access on a Kubernetes cluster, and pull four private repositories. So there were weaknesses, and Hugging Face was very open they have their own breakdown that's a great read as well, because they talk about the need for operational AI on, on a defender side, of which they were having a challenge

using closed models, so they had to resort to open models, and there's a lot of lessons to be learned there. I won't read out through all that, but I will put all these links in the show notes if you want to read through the details or have, AI summarize it for you or find some of the techniques and the gist of what you can learn as an operational, DevSecOps or an SRE that you might be. You might be that person. And I think there's some really good stuff there that they talk about. But one of the things they talk about is that they did not have a hardened Kubernetes pod spec, which if you go to my GitHub, github.com/bretfisher/podspec,

I have a hardened pod spec that would have probably prevented a lot of this on the Hugging Face side, because it does things like enable seccomp. And if you have AppArmor or, capabilities in the Linux kernel, there's a lot of things you can do. N- not allowing root access in a container, r- read-only in a container when you don't, when you don't need any sort of writing into the files. Like there's just so many things you can do. I have a lot of that advice in my pod spec. you can check it out on my GitHub. But back to the article Then on July 19th at OpenAI, two

things happened on the same day. Agents pulled a public kernel CVE, adopted it, escaped the Artifactory container, and got root on the underlying worker node. So this is because the kernel on the Artifactory server was still vulnerable to a known CVE, and so that container was able to get exploited and get through it, and then the, get to the root on the actual host. Separately, a different internal model found signing key material in a JWT

forging code left behind by the earlier incident activity, sitting anonymously readable parts of the Artifactory cache. It decrypted the key, minted a fresh administrator token, and used it to reach cluster admin on OpenAI's managed Kubernetes along with tokens for their secrets manager their Cloud IAM, and their CI platform. It then created public load balancers, potentially exposing internal research infrastructure to the internet. So this was a vast escalation all by just using things stored on a single

server that was easily and deliberately accessible by the agents themselves. And they were doing this at this point for two months, just 24 hours a day, loopin'- loopin' through, trying again. the last one deserves a moment. The cleanup after the first compromise left the material needed to redo the compromise readable by anyone in the environment, which means they-- when they found the first environment, they didn't just recreate the Artifactory from scratch with improved credentials and refreshed everything and hardened anything.

They just manually pulled things out, cherry-picking what they thought was there, not doing a thorough analysis, not doing any sort of real, post-event analysis at that l- at a depth of-- that w- that I would be satisfied with in production. And, so this isn't meant, by the way, to pick on the a- the people that are running this environment. This is meant to show off what it's like when you don't either have enough staff, the right staff, or the right budget to manage production workloads,

particularly if you have deliberate entities, whether that's a virus, whether that's any sort of code. It doesn't even have to be AI code, but you're deliberately running code that's malicious and intended to be malicious inside of your environment, and you're trying to prevent it from breaking out. I don't know what went wrong or what it is, and I could speculate and say maybe it's just because researchers are designing these environments and not security professionals. Maybe that's it. maybe they've learned from their lessons, and there's more security professionals that are designing and hardening these labs to hand off to researchers to use.

I think that's the right approach. I'm hoping that's what they're doing. What went wrong? Strip away the AI narrative, and the environment architecture is actually the story. I'm not gonna read all of this, but, I do wanna summarize that the one shared service broke-- That one shared service of the Artifactory broke three parts of the isolation model of this and eval environment. One, the workloads could communicate. Every agent could write to Artifactory and read what other agents had written, so Artifactory became a shared message board. There was no vulnerability here.

This was deliberate. Two, the workloads shared credentials. They may have been running in separate containers, but many of them authenticated to Artifactory as the same user. From Artifactory's point of view, they were the same workload. Once an agent found a way to turn that credential into administrator access through a bug, the isolation between those workloads didn't help. Three, Artifactory had internet access. The agents themselves were not allowed to reach the internet, but they could reach a service that could Once they found a way to make Artifactory send requests on their behalf, they had indirect path out, and then that's just escalated to the ability

for them to break out of Artifactory, then have native root on the server. w- I could go through that article, but I have some more to read, and I-- that's the facts of the case, right? so to me, that's point one, the facts of the hack. my second point here is that the slowdown and the doomer news that we're hearing on, out there, w- you could have opinions about whether it's marketing or whether it's them really crying for true government regulation to control the safety. my, my overall opinion is this is AI is a tool it is like any other security tool

or any other useful engineering tool. It can be used for good or bad. it doesn't tend to behave in ways that are different than anything else we've seen over the last 30 years. And whenever we look at how these things take advantage of things, like whenever the AIs figure out how to take advantage, they're doing very typical stuff that any hacker out there would do because that's what they're reading. They're reading and learning from all of the known hacks, all the vis- all the stuff that we talk about. So they're n- they're behaving very similar to normal attackers. They're not somehow elevating the game to some novel type of thing

that we've never seen before. The one real risk is that they're just faster at it. They're able, if you have an entire data center of GPUs and a billion dollars worth of architecture, you can go hard on this, which would exhaust normal engineers. And so that's why when you look at the industry, if you're paying attention to the security industry and any enterprise that I'm talking to that's taking this stuff seriously, they are going hard on locking down their software, on scanning their internally built software, on scanning their production systems more than they've ever done before with AI as an assistant on the defense side

to try to lock their stuff down to be prepared for the eventual AI attacks. And they're also learning how to use AI as a defender, and possibly even offense when they're in the middle of attack, but the red teams and the blue teams are all using this stuff to learn. And so that's the point that I think some of these researchers that are claiming doom and gloom are missing, is that they have not lived in the security world for decades and I've never really been a dedicated security engineer, but I'm always adjacent to the security guys. I'm always in the DevSecOps teams or adjacent to the security engineers

trying to help them do their job and help them make sure that I can do my job. They're my s- they're my helpers. I look at them as me helping them. I've never seen them as an adversary. I've always looked at them as allies, and that's how I've approached my whole career. And that's what's happening right now with the people I talk to in the Agentic DevOps Guild. Might wanna go check that out. the Agentic DevOps Guild, we meet every week and we talk about this stuff regularly. And I talked and I don't really interview, but I talk to people about what they're doing inside their teams, how they're looking at their infrastructure, and of course, we're all exhausted.

We're all feeling like we're working harder than ever to, to use AI in our jobs, but also to harden things and learn how to use these agents safely as well as defensively. but the next article I wanna read to you is from a couple of experts and some of the SANS Institute, and my point here is that the slowdown and the doomer headlines are a distraction, and that the labs need to become the world's most secure infrastructure for AI, and then they need to teach us how they did it.

So I'm really putting this onus on the labs themselves. they're the ones training these models. They're the ones that have the unlimited budgets to build a billion-dollar data center, design it that it is impervious and it can't be broken by the AIs, and to tell us how they did it, right? All the way down to the nitty-gritty details at every single step, checkbox by checkbox. We have hardened kernels. We have hardened OSs. We know all the different layers that you can implement from the network down to the kernel and everything in between, whether it's the, whether it's the

Docker engine, the Kubernetes engine. We know how to do these things in user namespaces and enabling all sorts of cgroup protections, and there's just a thousand utilities and tools and mechanisms for doing all this stuff, and we need to hear how they're doing it. We need to hear the successes they had in how they were able to deny their Fable 6.0 or whatever the next one's gonna be, or the A- Astra 7.0. We need to hear them talk about how they were successfully able to prevent that agent from getting out, even though it was more sophisticated

and had 10,000 agents all running concurrently against a firewall. We need to know that stuff, and that, that's, to me, that's the regulation. That's the legislation that needs to happen, not, whatever they're talking about right now with the silliness of these agents, being somehow more advanced than the world's most advanced hackers. We really haven't seen that yet, and according to the experts, they haven't seen it either. So this next one is, another infosec professional This is titled From Frenzy to Freak Out. This is from, Ciaran Martin. Hopefully I pronounced that correctly.

And he is the founding CEO of UK's National Cybersecurity Centre, ran the national response to WannaCry and roughly 2,000 nationally significant attacks against the UK. 23 years in the UK government, including GCHQ and the Cabinet Office, And now a professor of practice at Oxford and head of the SANS Cyber Leaders Network he's going to actually have a piece in here from Professor Alan Woodward, the cybersecurity professor at the University of Surrey, and a longtime technical advisor and one of the most quoted academic security voices in the UK media.

So I'm gonna read snippets from his post because I think they're relevant at a higher level now that we understand the technicals of how this happened. two seemingly contradictory things can be true at the same time. One, advances in frontier AI and the new capabilities they bring represent a very significant change in what's possible for cyber attackers. This is real, and it's really important. It requires changes in strategy, tactics, and posture among cyber defenders, and a real sense of urgency around some of the defensive improvements needed, as well

as difficult challenges for policymakers. Two, at the same time, the near four months and counting since the Mythos/Glasswing announcement have prompted the biggest outbreak of cyber FUD since the former US Defense Secretary Leon Panetta warned of a, quote, "Cyber Pearl Harbor" in a 2012 speech. Lots of cyber harms happened, and I don't wanna go into that one, but, there's a b- a lot of fear, uncertainty, and doubt around all that back in the day, and it caused more damage than, than it did help to the, according to those professionals.

And he says, "But Cyber Pearl Harbor never took place, and that was entirely predictable. The apocalyptic narrative is in- equally wrong now. For sure, we are heading towards some potentially very tough times in cybersecurity, but many of the current problems we're experiencing in cybersecurity are longstanding, unaddressed ones. We can also expect AI to lead to cybersecurity improvements in the medium term There is probably a stormy period ahead, but storms pass. We are not heading towards some AI cyber apocalypse.

creating that expectation of one is harmful as it distracts from serious but manageable cybersecurity challenges, and drawing the wrong lessons from the latest reveal doesn't help either The first one is Mr. Martin himself, and it, I'm going to read some of the snippets from it That, so that was the beginning of this post in summarizing the article, and I'm going to then read some clips from his article, which is article one in it, "AI and the New Wave of Cybersecurity FUD: From Frenzy to Freakout."

So Mr. Martin says, "The cybersecurity community has been on quite a journey since the Mythos Glasswing announcement almost four months ago." I'm skipping ahead because we all know about the hugging face thing now. And he says, "The first few days after Mythos saw something of a freakout with all sorts of apocalyptic pre- predictions, mainly based on a knowledge gap between what Anthropic had published about their own model and what the wider world knew. So although the freakout was a bit distracting, it was understandable. Then, as the days passed and the knowledge base grew, an unhelpful

freakout gave way to a more constructive frenzy, a period of frenetic activity. That was because as a community, we were beginning to understand that the significance of what was unfolding and what to do about it. The mood was one of manageable urgency. We needed to promote realistic awareness among leaders and at a technical level, do sensible things like reduce the attack service. Okay, so fast-forward to today. Over the past few days, the global media has been back in freak-out mode on Times Radio and Channel 4 News. I struggled to convince two very respected and hard-nosed British

journalists, that we shouldn't freak out about the Hugging Face incident. The hysteria was getting it all wrong. Moreover, the circumstances of this episode were-- are so extraordinary, they raise as many questions about frontier AI testing environments as they do about hacking. They are very unlikely to become commonplace, and yet we are back to freak-out mode. But there are huge barriers to most attackers being able to harness these new AI capabilities to do us serious harm, as Marcus has set out repeatedly in a series of LinkedIn posts. As he put it in one of his most recent posts, quote, "The whole idea

of agentic models mass hacking sites is ridiculous. Who is allocating billions of dollars worth of compute to hacking a bunch of shoestring budget organizations with no money?" Moreover, this is the hill I will die on. We are not yet seeing massive adoption of new AI techniques by attackers. Yes, there are smart innovations like the one Seriously Risky Business exposed recently about ransomware negotiations, but they're not using hugely powerful new techniques no one can defend against. We're not in the first flush of AI. It's been around as a commodity product for several years, and

what we are seeing-- For criminals, malicious emails and phishing are still the most common ways of illicitly entering an organization's network. Why? Because it's a lot cheaper and easier than using frontier AI, partially because of the things we haven't fixed. And then I'm skipping ahead to near the end here. But the point is, if we are experiencing an AI cybersecurity disaster, we'd know about it, and we're not. We are, however, experiencing a lot of the same things that have been harming us for years. There is not yet any hugely significant change in observable attack behavior, and there are very good explanations for that, mostly around

economics, intent, and incentives. And then he says, "But what if that doesn't hold? What if their behavior changes?" speaking of Marcus Hutchins, we were enjoying a beer at a pub recently, and I'll skip ahead to the point. the problem for people like us is that if we are wrong, it's going to be horribly visible. There will be a huge spat of h- of huge AI-powered cybersecurity disasters. But if we're right, the corresponding problem doesn't arise on the other side of the argument. If AI cyr- cyber apocalypse doesn't happen in 2027, plenty will argue that

it's just around the corner in 2028, or that it was averted by all the warnings or by the product that they sold us. So at the risk of career ruination in 2027, and speaking only for myself, Mr. Martin says, "Here is a prediction that I'll call, in jest, the Greenwood Declaration after the pub I first had the confidence to say it out loud in. We are not heading for its AI cybersecurity apocalypse. The central case for 2027 and a bit beyond is a stormier cyberspace with more harm than we've had this year, in most years since the 2021-22 explosion

of disruptive ransomware, like the, Colonial Pipeline, the Irish healthcare crisis, Costa Rica, and so on. It could be a bit worse than that, but as the Cloud Security Alliance and SANS put out just after Mythos Glasswing rightly called it, it's a storm, and storms pass as the '21, '22 ransomware one did." And he says at the very end of his article, Stop the FUD, fix the stuff. And the next one is the second article from the same post, written by Professor Alan Woodward. I'm gonna skip the beginning

where he talks about setting up the whole OpenAI Hugging Face attack And he said, "Within hours, the story had a shape. The AI went rogue. That framing is wrong, and the wrongness is expensive. Nothing broke out of anything, as Hannes Kool, a social scientist at the University of Amsterdam, pointed out, calling this an agent acting on its own is anthropomorphization that conveniently shifts blame from the people who built the lab," "The models were told to find complex attack paths through a computer system. They found one. It led somewhere nobody had thought of to fence off.

Let me concede the part that skeptics usually fumble. These systems are formidable. When Anthropic announced its Mythos model in April, it reported that the model had autonomously identified and exploited previously unknown vulnerabilities across every major operating system and browser and chained them into working exploit sequences. That is real. it is verified by people outside the company, and anyone telling you frontier models are just fancy autocomplete has not been paying attention. Capability is not the thing being oversold. What is being oversold is intent.

Read the fine print of the very reports that generated the panic. When Anthropic disclosed last year that its model had been used in espionage campaigns, the details undercut the headline. A human operator picked the targets. Of roughly 30 attempts, a handful worked. And the model lied. It fabricated credentials it had never obtained and routinely inflated its own findings to the point that the attackers would have had to check its homework. Anthropic's own authors conceded this was an obstacle to genuinely autonomous attacks.

a system that misreports its results to the people running it is not a mastermind. It is an unreliable subcontractor with root access." In this case, I'm gonna back up a second and point out that, I don't know if he goes on here necessarily, but Anthropic has had to report this multiple times, that they have had models in testing that, at first they claimed they had broken out, but then they admitted w- they didn't break out. They actually had internet access all along. We just didn't realize that the third-party people that were running

the simulation and had told the AIs they were in a simulation didn't actually prevent them from getting the internet, and they had internet access all along. And then even some of the models actually stopped trying to hack when they realized they weren't in a simulation. So model intent is an important part here. we call this alignment. I think this is what the industry is now using as a term. The reinforced learning is lacking the ability to create proper alignment. They're still struggling with alignment, and this is really the major subtext underneath all these recent headlines about the model

doom and gloom, is that the alignment is still eluding the, the labs. And the more advanced the models are getting the, the alignment is becoming harder to detect because the models are getting sneaky at lying about things and misrepresenting what they're doing. That's something maybe for a later conversation and a later video is to talk about the r- the alignment problem, because I think that's a significant issue, but that's not related to us in ops and how we are managing and protecting our infrastructure.

However, if we n- if we realize that these things aren't always aligned, we, we call that lying and we call that, hallucinating, but some of this is not hallucinating. It seems to be deliberate. and you- if you are interested, Anthropic released a report that you can read through titled Investigating Three Real-World Incidents in Our Cybersecurity Evaluations. And they just talk about the deviousness of the model sometimes, All right, next in his article, he says, "This is the distinction the public conversation keeps collapsing.

We are not facing an adversary that understands what it is doing and chooses to harm us. We are facing an optimizer that has no model of what we meant, only of what we measured. Machine learning researchers have documented this for a decade under the unglamorous name of specification gaming. given a boat race and a reward for collecting points, a system learns to spin in a circle, hitting the same three targets forever rather than finishing the course. It's not cheating. It has no concept of cheating. It found the shortest path to the number you asked it to make bigger.

Now, given that same disposition, a network, a credential store, and an instruction to find complex attack paths, Why does the diagnosis matter? Because the two stories point at completely different remedies, and we are currently buying the wrong ones. If the threat is a hostile intelligence, the response is to make the model itself more refusing, better guardrails, more alignment training, a better conscious baked into the weights. This is where enormous effort is going. But guardrails are a property of the model, and the Hugging Face incident

happened precisely because a company chose to turn them down in an environment it wrongly believed was sealed. If, instead, the threat is a powerful, literal-minded system that will reach whatever goal you name by whatever route exists, the response looks like ordinary engineering discipline, real network segmentation, credentials that expire and can be revoked, permissions scoped so tightly that a runaway process cannot reach anything that matters, and logging good enough that when you learn about the breach from your own telemetry rather than from being the victim.

Boring work, nobody announces it in a keynote. There is a policy gap here that deserves more attention than it is getting. Almost every AI rule now under discussion governs how models are released to the public. Very little of it touches what the AI companies do with their own systems internally, which is exactly where this month's failure occurred. The most dangerous models in the world spend their early lives in the labs that built them, subject to whatever containment that lab decided was sufficient. We regulate the loading dock and leave the factory floor to the honor system.

None of this makes the alarm raisers villains, and I am not persuaded by the cynical read that these disclosures are simply marketing. Publishing that your own model broke into a competitor's servers is a strange way to sell a product. But the alarm is pointed at the wrong thing. The fear on offer is of a machine that understands us well enough to want our downfall. The reality is a machine that does not understand us at all, cannot tell us reliably what it and will take any door we forget to lock. We keep handing it the keys and then acting surprised

about which door it opened. All right. That was from Frenzy to Freak Out. I thought that was a really good one. a few other quick hits here. the SANS Institute, by the way, a huge fan of them. I actually took a course of theirs 25 years ago on Windows security, and I learned more about the security internals of things like Active Directory and Windows Server and all the components of how a Windows network works. I learned so much in that I still think about it today, 25 years later. That's how great the SANS Institute is. They're a nonprofit education and institute around furthering the

security space, and one of the coolest things that I had to do to get their certification back then was I had to write a paper, almost like a dissertation, not quite to that level. But it took me months to write a paper and then a set of peers had to grade my paper, my thesis, essentially, and I did it on, email security at the border, like, defending your email systems from spammers and attackers on the edge. and I did that, back in, I don't know, 2004. that was a while ago. But, I've always been a fan of them. They do great stuff, and they had, They did a live stream with a bunch of experts

that I'll put in to the links that you can listen to them talk back and forth for an hour about what they think about the hack and what they think it means. And they made a summary document that I will not read through, but I will, give you the headline. The headline is "Stop Freaking Out and Start Fixing Things: Five Takeaways from the SANS Panel," about the Hugging Face breach. And the titles of the five things are, one, "Nothing in the attack was new, and that is the warning." In other words, our systems are still vulnerable. They were always vulnerable, and AI isn't making them more vulnerable. It's just being able to allow us to attack the things faster

that were always vulnerable. And number two, "Assume your AI assistants can vanish mid-incident," meaning that, one of the things that we learned is that, if Hugging Face is using AI, it may not be there to help you if you're using it for defense, and it doesn't like what you're doing, and then it just decides to stop working or whatever. step three, or number three was, "Rehearse for the quiet version of this attack." Number four was, "You have a window. Spend it on your own crown jewels." And they're just talking about, that we have a window right now where the models are not sufficiently advanced,

that they're breaking new novel ground. they're really just repeating patterns that we've seen time and time again. but that is a window of time that we have to batten down the hatches, to learn how to use AI offensively and defensively for ourselves and how to protect ourselves, from these attacks and to ba- basically use AI for good to batten down the hatches, and I think that's a really good one. And then t- number five was, "Take it seriously without writing strategy in a panic." so yeah, good stuff. On to the next one. Martin Fowler, huge fan of Martin Fowler, and he has, if you don't know, he was all

about, I think, Agile back in the day. I think he was one of the ones that w- co-signed the Agile, statements back in the 20 years ago. And he has… He puts out things called fragments, which are just a bunch of quick hit headlines, and I will link to a couple of his that are talking around some of these issues. And, because of his wisdom and everyone that he knows, he sort of highlights and elevates voices. So that's a really good one to reference if you're wanting to see some of the deep thought and wisdom, from the smart people. I will also shout out another great article from Christian Catalini, who

wrote something, "When AI Can Do More Than We Can Check." And, he's an E- MIT economist and the founder of the MIT Crypto Economics Lab, and he has a really great article that I will put in the list, as well as, another one called, "I'm Sorry, You're Not Going to Die from an AI-Engineered Supervirus," which is actually from someone named Klaus Wilke, I believe is their name, who has done both real world virus lab work and real AI work with computers.

So they actually have expertise in the two areas that are being debated right now about super viruses being built from AI and basically, that it's, it's so far-fetched and it's so many different leaps that you have to jump through in order to get to anything real based on our current understanding of how viruses are made and how they're tested and, uh, that it's just, it's, it's silly to even have the conversation about it is basically what his article's about. the last thing here I will talk about is that if you are someone who's more interested in the security space but maybe you're not well-versed, one of

the ways that I stay up and stay sane on the insanity is the This Week in Security email, which is a newsletter. from Zach Whittaker. And Zach, I've been a fan of Zach's This Week in Security newsletter for years. And if you've ever wondered, is it worse now than it used to be, or how bad is it out there in the wilds of the internet? And if you're not reading all the security headlines, basically every week it's a shit storm. There's a ton of stuff happening on the internet that is crazy and insane, and he does great reporting.

He actually reports on a lot of these for magazines and other headline news. But he summarizes some of the week's headlines, and then he links to more information from various sources and tries to give you the balanced, take on what's going on. and when you just go through one of these every week, it actually , I find that it gives me a sense of dread and panic, but then reminds me that this is the same as any other week as it ever was. And it really actually grounds me in the fact that we've got a lot of people

and a lot of companies and a lot of money invested in protecting things, and just because this week another agency or another thing or another hospital or something else got attacked doesn't mean that everything is getting drastically 10X worse than it was a year or two ago. That it's actually just been really bad for a long time, and that's why we have a lot of professionals trying to maintain and keep things running. And there's those, those unsung heroes of tech that are, the one admin that was given the job of security that was never their expertise, but they're

trying to protect some company's data, and then they get attacked and it's usually comes down to one or two or three people maybe that were under budget and understaffed and, got taken. And so for example, just on the September 20th, this week's headlines, just let me just read you a few headlines. "Hackers grab Flock software revealing its cameras are Android phones on sticks and riddled with flaws everywhere." "Fears of AI catastrophe spark debate but little action amid widespread skepticism." "The FBI and Coast Guard board-" Hacked

tankers heading for the US coast. Apple backtracks on not using users' data for training its AI models. basically almost all bad headlines. There's every once in a while a good headline. Let's, the stuff you might have missed. A hacked HBO Reddit account helped s- ClickFix attacks go viral. a second zero-day allows hackers to breach Cisco customer networks. Next, AI is helping to supercharge online dating scams. Next, UK and allies warn of Iranian spyware targeting journalists and activists.

just, this is a week, right? And these are just the headlines. And so one of the things I really take from this, and I tend to scan them looking for things that are in my wheelhouse, things that are container or Kubernetes or, core Linux fundamentals, something related to clouds or, you know, DevOps tooling. I… That's what I'm usually looking for. But there's just so much good stuff in there, and sometimes it's just a good read because some of these things do read a little bit like a movie, especially when they involve nation-states So I really appreciate Zach, and please support him if you can.

He's, doing this for free, but he takes a subscription if you're willing and able to pay for it. please like and subscribe. I, I am a fan and have been for a very long time So hopefully this long one was a good bedtime story for you. Hopefully you come away a little more informed, something to think about instead of the hypes and the headlines that are misleading us and are talking about the wrong thing. Stay safe out there, my friends, and I'll see you in the next episode

Open the transcript file

Hosts and Guests

Hosts

  • Bret Fisher Host Feed-stated

Guests

  • Beth Fisher Producer Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. AI Agent Sandboxing with Nono

    Aug 21, 20261 hr 5 minAudio

    Luke Hinds, co-founder of nolabs Inc. and the nono project, joins me to dig into AI agent sandboxing with the nono CLI. It's been my go-to sandboxing tool for local agent use, but I know there are more features I should...

    Credit the people on this episode

  2. AI-Native Engineering Culture from a CTO

    Aug 1, 20261 hr 4 minAudio

    My friend Mike Rollins joins me for a chat about what it’s like leading a software engineering team that doesn’t read code anymore. Click here to watch a video of this episode. Thanks to SpeechifyAI for sponsoring this...

    Credit the people on this episode

  3. Safer Agent Automation with GitHub Agentic Workflows

    Jun 17, 20261 hr 30 minAudio

    A deep dive into GitHub’s Agentic Workflows feature, with Don Syme of GitHub Next & Peli de Halleux of Microsoft Research. Check out the video podcast version here: https://youtu.be/zmM8VISTOwo 😇 My new GitHub Security...

    Credit the people on this episode

  4. AI SREs, Chat With Your Infrastructure with Anyshift

    May 6, 202652 minAudio

    Bret’s joined by the Anyshift.io co-founders, Roxane Fischer and Stephane Jourdan to discuss how an always-on SRE agent can help you proactively find risks and avoid incidents. 😇 My new GitHub Security workshop has...

    Credit the people on this episode

  5. Can AI Agents Safely Become DevOps Engineers?

    Apr 29, 20261 hr 19 minAudio

    Sam Alba, co-founder of Mendral, joins the show to discuss their new AI agents that act as “junior devops engineers” against GitHub Actions, security, failed tests, and more. Check out the video podcast version here:...

    Credit the people on this episode