What happens when an AI agent crosses a security boundary it was supposed to respect? How do we prevent an agent from accessing unauthorized systems, exposing secrets, or finding an unexpected path into production? In this episode of DevSec Station, Tanya Janca (SheHacksPurple) walks through how to threat model an AI agent escape using Adam Shostack's four simple threat modeling questions: What are we doing? What could go wrong? What are we going to do about it? Did we do a good job? Using a fictional AI coding agent, we'll explore: The difference between an AI agent doing something unexpected and actually escaping a security boundary. Why not every task needs an autonomous AI agent. How agents can reach sensitive systems indirectly through tools, credentials, CI/CD workflows, and other agents. How to enforce security boundaries with least privilege, isolation, authorization, and network restrictions. Why testing your controls, monitoring agent activity, and preparing for incidents matter. Why a prompt is not an enforceable security boundary. The big lesson: Your AI agent can decide what it wants to do, but it shouldn't get to decide what it's allowed to do. 🔧 Your one thing to do this week: Pick an AI agent you use or are building. Draw a simple diagram of its tools, credentials, connected systems, and trust boundaries. Identify one boundary that isn't enforced outside the AI model, and decide what security control it needs. Then add it! Threat modeling doesn't have to take three days and a 400-column spreadsheet. Sometimes four questions are enough to get started. DevSec Station is a podcast by Tanya Janca (SheHacksPurple), focused on short, practical lessons that help software developers build more secure software. Train with Tanya: Want to bring practical, hands-on secure coding, application security, or AI security training to your team? Learn more at SheHacksPurple. Follow Tanya: https://newsletter.shehackspurple.ca https://linkedin.com/in/tanya-janca https://youtube.com/@shehackspurple https://shehackspurple.ca
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
SPEAKER_00Let's say you've built an AI agent. You've given it a task, some tools, access to a few systems, and a nice little environment where it's supposed to do its work. You've tested it and everything seems fine. But then one day it does something you absolutely did not expect. Maybe it accesses a system it wasn't supposed to. Maybe it finds credentials that you didn't even realize were there.
SPEAKER_00Maybe it sends information somewhere that you did not intend. Or perhaps it discovers in an entirely different way to accomplish something you specifically told it not to do. That's exciting, right? But probably not in the way you were hoping, because your AI agent might have just escaped. And now you have a potential security incident.
SPEAKER_00I'm Tanya Jenka, also known as SheHacks Purple, and this is DevSec Station. In this episode, we are going to threat model an AI agent escape. And before anyone starts worrying that this is going to involve three days of meetings and a spreadsheet with 400 columns, I promise you it doesn't have to be like that. We're going to use four questions. That's right. Four. And by the end of this episode, I want you to be able to use those questions to figure out where your own AI agents might get into trouble.
SPEAKER_00But first, what exactly is an agent escape? Before we start threat modeling, we need to be clear about what we're actually talking about. An AI agent escape is when an agent crosses a security boundary that was supposed to contain it. Maybe it escapes a sandbox you put it in. Maybe it accesses files or systems outside its authorized scope. Maybe it attains credentials it wasn't supposed to have, or maybe it uses one tool to reach something.
SPEAKER_00Another tool was supposed to prevent it from accessing. But here's an important distinction: an agent doing something unexpected isn't necessarily an escape. Remember my Hello Queen application? Cloud published it, executed it, scheduled it to run on my computer, and connected it to an external API that I had not vetted. It didn't ask to do any of those things, but that doesn't necessarily mean that it escaped either.
SPEAKER_00If the tools and permissions I gave it allowed those actions, then it was operating inside the boundaries I'd given it. It just did a whole bunch of stuff I wasn't expecting, which is a different problem and one we've already talked about the season. An escape is when the agent gets past a boundary that was supposed to prevent an action. And today we're going to focus on identifying, testing, and enforcing these boundaries.
SPEAKER_00How do we identify them? How might an agent get around them? And what can we do to make sure these boundaries actually work? Now, I want to talk about threat modeling. When I teach threat modeling at SheHax Purple, I like to use Adam ShowStacks for questions because threat modeling doesn't have to be this enormous, complicated thing where you spend three days filling out spreadsheets and everyone just wants to cry. We can make it pretty straightforward and unpainful.
SPEAKER_00It could even be fun. Like I said, there are just four questions. What are we doing? What could go wrong? What are we gonna do about it? And did we do a good job? That's it. Now obviously you can go much deeper than that, but these four questions give us a really useful way to think through a system. So let's use them to threat model an AI agent for a pretend example system that I made up.
SPEAKER_00So, question one, what are we doing? But before we start drawing diagrams and looking for security boundaries, I actually want to back up for a second and ask something else. Does this system even need to be an AI agent? Because not everything needs to be a damn agent. I was talking with someone who wanted to build an AI agent to collect cybersecurity news, check which stories were relevant, and then put it all together for a company newsletter. And they were worried about how to secure the agent.
SPEAKER_00But I, of course, had a different question. Why does this even need to be an agent in the first place? Couldn't we use a scheduled job to collect the news, regular software to remove any duplicates and store the results, and then maybe AI to help decide which stories are relevant or to summarize them, and then have a human review the newsletter before sending it. We'd still be using AI where it's useful, but we wouldn't need to give an autonomous agent access to all those systems and permissions.
SPEAKER_00We could just use a regular LLM. And that would mean fewer things to secure. Now, I'm not saying that regular software is magically secure because we all know that's not true. But if we can accomplish the task without giving an AI model unnecessary autonomy and permissions, why wouldn't we do it that way instead? And that's a part of threat modeling. Making sure we're building the right thing in the first place and architecting it correctly in the first place so that we reduce risk.
SPEAKER_00Now let's imagine I've built an AI coding agent. Because apparently I haven't learned my lesson from the Hello Queen situation. This one is supposed to help me review code, run tests, and create pull requests.
unknownYes.
SPEAKER_00But that's it. I do not want it accessing production. I don't want it downloading all my secrets, and I definitely do not want it deploying things without asking me first, which is, as we've established, is something I apparently need to be extremely clear about. So that's all I want the agent to do. But what can it actually do? Can it do more than that? Where does it run? Is it inside a container, a virtual machine, directly on my laptop?
SPEAKER_00What files can it access? Can it make network connections? What tools can it run? Does it have access to my credentials? And what can those credentials do? Because here's something that we've already learned together. The boundaries you describe in a prompt are not necessarily boundaries that are enforced by your system. I can tell my agent that it must never access production. But if I've given it credentials that allow production access, I haven't actually stopped it from doing that.
SPEAKER_00I've just asked really nicely. And we all know how I feel about asking nicely for software to behave and how that usually works out for me. So the first thing that we would do is we would draw a diagram. So think of this diagram as though a data flow diagram and an architecture diagram got together and had a baby. It doesn't need to be fancy. So put the agent in the middle and draw all the tools it can use, all the systems it can access, the credentials it has, and the places where information can leave the environment.
SPEAKER_00Then you want to identify where trust boundaries are actually enforced. So not the boundaries that we wish existed, not aspirational boundaries, but real live technical enforced boundaries. So that's the answer to question one. What are we doing? So now question two, what could go wrong? This is where things are gonna get interesting. We've got our coding agent, it's supposed to review code, run tests, and create pull requests. Awesome. And we've drawn a diagram of the environment where it operates, great.
SPEAKER_00Now let's look for ways that it could cross a boundary. Let's start with the network. Maybe I've configured the agent so that it can only access a few approved services. Awesome, great, good job. But can it make arbitrary outbound HTTP requests? Can it send information to an external server? What if it can't make those requests directly? Could it use another tool that can make those requests? For example, maybe the agent can modify a file that gets uploaded somewhere automatically.
SPEAKER_00Or write something to a log that's forwarded to another service, or maybe it can modify a build script that later runs in an environment with broader network access than it has. See what we're doing here? We are looking for paths around the boundaries. So now let's talk about credentials. Because maybe my agent doesn't have production credentials. Excellent. But can it read the environment variables? Can it inspect the configuration files?
SPEAKER_00Can it access a CICD system that does have production credentials? Could it modify a workflow that later runs with elevated privileges? Because if my agent can modify a workflow that has permissions to deploy to production, then I might have accidentally given an unintentional path right to the place I didn't want to. Even though I never gave it production credentials directly, I kind of sort of did. Oops.
SPEAKER_00This is why we have to look at the entire system, not just the agent. Now let's look at tools. Suppose I've given the agent access to a shell. What permissions does that shell have? Can it install packages? Can it access files outside the project? Can it launch other processes? Can it interact with cloud infrastructure? And here's another interesting one. Can it communicate with other AI agents? Maybe agent A has very limited permissions, but agent B has access to something much more sensitive.
SPEAKER_00Can agent A send instructions to agent B? Can it influence agent B through a shared document, message queue, or any other system? And if agent B acts on those instructions, have we accidentally created another path around our security controls? These are the kinds of questions I would like us to be asking. Not just could the AI come up with some incredibly clever way for it to escape, but what paths actually exist through the system and which of those paths could allow it to cross a boundary?
SPEAKER_00That's a much more useful engineering question for when we're threat modeling. And now I want to ask the scary question: what are we going to do about it? Okay. So we've found some potential problems together. Now what do we do? The good news is that we already know how to solve a lot of these problems. We are security people, we've got this. We have been dealing with software that does unexpected things for a very long time. And developers have been building security controls into and around that software for pretty much just as long.
SPEAKER_00So let's start with isolation. If my agent needs to execute, I probably don't want it doing that directly on my everyday computer with access to all my personal files and credentials. I can give it an appropriately restricted environment instead. Maybe a container or a temporary development environment, maybe a sandbox, something that limits what it can access and what it can affect. And I can restrict its network connections too. If it only needs to communicate with three-approved services, why would I give it access to the entire internet?
SPEAKER_00Next, credentials. My agent should have its own identity on the network, not my personal account with every permission that I have accumulated over the past decade. Give it credentials scoped to the specific task that you want to allow it to do and make them short-lived if you can. And don't give it access to secrets that it doesn't need in the first place. Okay, now tools. If an agent needs to read a repository, it doesn't automatically need permission to modify it too.
SPEAKER_00If it needs to create a pull request, it doesn't need permission to merge it. And if it needs to run tests, it doesn't necessarily need unrestricted shell access. But there's something else here that I want to emphasize. The system receiving the action needs to enforce the permission. Let's say my agent wants to deploy something. The agent can request a deployment, but the deployment system should decide whether that request is authorized via a policy, not an AI.
SPEAKER_00The agent probably shouldn't have deployment permissions, and it should probably require an approval. Maybe it can deploy to a development environment, but not production. Whatever our policy is, we need something outside of the AI to enforce that for us. Because the agent may get to decide what it wants to do, but it never gets to decide what it is allowed to do. And finally, let's talk about what happens if one of our controls fails, because that can happen.
SPEAKER_00We've seen this. Can the agent affect one repository or 50? Can it access one cloud resource or an entire production account? Can it perform one transaction or 10,000? We want to limit the blast radius. If something goes wrong, I want the consequences to be manageable. Not an emergency meeting with every executive in the company. Nobody wants that, especially not on a Friday afternoon, because these things always, always, always seem to happen on a Friday, don't they?
SPEAKER_00Okay. Question four. Did we do a good job? Now, this is my favorite question because we can design all the security controls that we want. And we can write beautiful documentation. We can create lovely diagrams. We can tell everyone how secure our system is. But do the controls actually work? Have we tested them? Let's go back to our coding agent. We said it shouldn't be able to access production. Okay. But have we actually tried to access production from its environment?
SPEAKER_00And if so, what happened? Does the request get blocked? Does it succeed because someone else accidentally left a credential lying around? What if the agent tries to read a sensitive file? What if it attempts to make an unauthorized network connection? What if it modifies a CICD workflow? Do the boundaries hold? Because if we haven't actually tested them, then we don't really know. And there's another part to this question. Would we know if an agent escaped?
SPEAKER_00Are we logging the actions that it takes? Can we see what tools it invoked, which creds it used, which systems it accessed? Would an unusual network connection generate an alert that a human being would actually see? And if something goes wrong, do we have the power to stop it? Can we revoke its credentials? Can we terminate its execution environment quickly? Can we prevent pending actions from completing? Investigate what it accessed or modified?
SPEAKER_00These are ordinary instant response questions, but they are especially important when we're giving software the ability to take actions autonomously. And I want to make one thing very, very, very clear. We don't have to invent an entirely new field of security engineering to answer these questions. We already know how to do threat modeling. We already know how to enforce authorization. We already know how to isolate software. We know how to monitor systems and respond to security incidents.
SPEAKER_00But we need to apply those skills to AI agents. Here is what I would really like you to remember. When we hear about an AI agent escaping, it's tempting to focus on how clever that escape was. How did it figure that out? How did it find that voe? How did it know to use that particular tool? And those things might be interesting for sure, but from a security perspective, I think there's a more important question. Why was that path available in the first place?
SPEAKER_00What security boundary was supposed to prevent the action? Was that boundary missing? Was it incorrectly configured? Did another component provide a way around it? And what can we change so that next unexpected action doesn't have the same level of consequence? Because I don't want to build a security strategy around hoping that our AI agents don't do anything surprising. We know that they are going to surprise us. We should know that by now.
SPEAKER_00And sometimes it does surprise us in wonderful ways, and that's why we use it. But sometimes it's in ways that make us seriously question our career choices. But what I want is a system where an AI agent can attempt something unexpected and the security architecture responds appropriately. Maybe the request is rejected. Maybe an approval is required. Maybe an alert fires. Maybe the agent gets stopped. But the important part is that the AI doesn't get to decide whether these controls apply to it or not.
SPEAKER_00Never rely on the AI model itself to enforce the security boundary that's supposed to contain it. Here's your one thing to do this week. Oh, yeah, that's right. Of course there's more homework. Pick one AI agent that you or your org is building, testing, or using, and ask yourself: does this task actually require an autonomous agent? If yes, that's great. Then let's threat model it. Draw a simple diagram. You know, the one where the DFD and the design doc had a baby.
SPEAKER_00Include the agent, its tools, its creds, the system it can access, and the places where information can leave the environment. Then ask yourself one question. Which of these boundaries are enforced by something outside of the AI? If the only thing preventing an action is a sentence in the agent's instructions, you've found something worth looking into. You don't have to solve every problem today. Just identify one boundary that needs a better security control because we know we can't prevent every unexpected action, but we do want to make sure that one unexpected action doesn't turn into a security incident.
SPEAKER_00I'm Tanya Jenka, also known as She Hacks Purple, and this is DevSecStation. Thank you for listening, and I will see you in the next episode.
Your AI agent can read files, execute commands, install software, connect to external services, and maybe even deploy to production. But should it have permission to do all of those things? In Season 2, Episode 3 of...
Two critical PHP vulnerabilities have been disclosed, and even applications written with secure coding practices may be affected. In this special bulletin of DevSec Station, Tanya Janca (SheHacksPurple) explains...
I asked Claude to build me a little application that would greet me every morning. Nothing complicated. Just say hello, call me a Queen, give me a compliment, and share an inspirational quote. Cute, right? Except Claude...
What happens when your AI assistant does exactly what you expect 59 times... and then does something completely bananas on the 60th? While teaching a secure coding class, I watched nearly 60 students use the same AI...
Security problems don't generally happen because developers 'don't care' or lack skill. They happen because no one clearly defined what "secure" meant in the first place. When security expectations are left implicit,...