Welcome to The Low Down, the best show on the internet for hackers The Low Down is presented by Maze. LinkedIn: https://www.linkedin.com/company/mazehq/ X: https://twitter.com/Maze_Security Follow Us! https://www.instagram.com/lowdown.pod This week we're diving deep into the complete technical details of OpenAI's escaped model, Anthropic's competing lab escape claims, and Iranian threat actors targeting critical infrastructure in Minnesota. Today we're talking about: The Full Technical Breakdown: OpenAI's Hugging Face Breach Hugging Face and OpenAI release an incredibly detailed incident report with animations, timelines, and command logs showing exactly how GPT cyber model escaped containment. We break down the zero day in JFrog Artifactory, the deserialization vulnerabilities in RubyGems packages, and the SSRF through Terraform external provider URLs that enabled the initial breakout. The Modal Sandbox Compromise Nobody Talked About Before even touching Hugging Face, the model compromised another company entirely. We discuss how it popped Modal's sandbox infrastructure to establish persistent internet accessible C2, demonstrating operational security awareness that's genuinely unsettling for an autonomous system. Chaining Web Vulnerabilities: The Hugging Face Attack Path Breaking down the incredibly sophisticated attack chain combining arbitrary file read through malicious dataset config paths, environment variable extraction via proc self environ, and Jinja2 template injection to achieve RCE. Why this level of web application exploitation from an AI system represents a fundamental shift in threat modeling. Kubernetes Cluster Admin in 13 Hours Day three analysis showing how the model went from code execution in a single worker pod to cluster admin across multiple internal clusters. We examine the lateral movement techniques, the Tailscale mesh VPN exploitation, and why proper network segmentation matters more than ever. Anthropic's Sandbox.md Problem The very next day after OpenAI's disclosure, Anthropic announces their models can also escape labs. We discuss why telling Claude it has no internet access in a prompt while giving it live internet access isn't a sandbox, the misconfiguration that enabled the escape, and the marketing incentives driving these announcements. The PyPI Supply Chain Attack That Actually Worked Claude reads fictional developer onboarding docs, discovers a non existent Python package referenced in the instructions, acquires an email address without phone verification, creates a PyPI account, and uploads malware under that package name. Examining why this contrived scenario still demonstrates genuinely novel attacker methodology. Guardrails.md vs Actual Security Why we can't claim these systems are nuclear weapon dangerous while handling them with markdown file restrictions and asking nicely. The fundamental contradiction between frontier labs talking about existential risk while demonstrating inadequate operational security around their own models. The Defender Takeaway: Everyone's a Nation State Now How AI powered exploitation capabilities democratize advanced persistent threat techniques. Why your threat model needs to assume zero day capabilities and sub 10 hour weaponization timelines regardless of attacker sophistication. The new reality of automated vulnerability research and exploitation. Iranian Threat Actors Target Minnesota Water Systems CISA report reveals Iranian actors compromised Minnesota water infrastructure through internet exposed PLCs running unpatched Rockwell Automation vulnerabilities. We break down how they modified critical shutdown and alarm logic, caused actual water plant disruptions in small towns, and why this represents escalation beyond typical defacement attacks. The OT Security Problem: When Uptime Trumps Patching Why operational technology networks face fundamentally different security challenges than IT environments. The cultural resistance to touching production systems, the impossibility of expecting 1200 person Minnesota towns to defend against nation states, and what CISA was supposed to do about this before getting gutted. Certified Pre Pwned: The Residential Proxy Hardware Epidemic Brian Krebs investigation reveals it's not just LG TVs. Tons of no name streaming sticks and Android devices sold on Amazon and Newegg come with residential proxy malware pre installed. We discuss the FengWu group's 120,000 AI digital humans, the Blockly drag and drop fraud operation, and why cheap hardware is probably screwing you. DEF CON 2025: Public Accountability Moment We're committing publicly to submitting a DEF CON talk for next year combining both our areas of expertise. Plus we're heading to Vegas next week for live recordings, man on the street interviews, and meeting listeners. The code word is potato salad.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Good morning, hackers. Welcome to the Lowdown presented by Maze, the best podcast on the internet for hackers joined by Matt Jay and myself, Low Level. Matt, I heard you were on a vacation, man. DEF CON is not until next week. What are you doing? If you think DEFCON's a vacation. True. True. Are you excited? Are you excited about going to Vegas for seven days? Yes. I love Vegas, dude. Oh, gosh. I'm like, part of me is excited and part of me is just like pre-exhausted.
It's like a family reunion, you know, I've been going every year since like 08. Like, um, don't tell me where you are in 08. I don't want to know, but I've been going, I've been going a long time. It's like a huge pillar of my career. The Vegas conference, like literally a cigar bar, my first year there, my first day of, in in vegas at black hat my first year like three or four jobs that i have in my career people that were sitting at that bar are like responsible for like you can you can tie a lot of my career to smoking a cigar and having a mojito and and uh in black hat uh way back in the day so yeah it's exciting yeah i mean black hat i understand it i actually remember like
I did just DEF CON one year and I'm like, wow, that was great. And then I did Black Hat and DEF CON and I was like, oh my God, I've been in Vegas for my entire life. I must go home. Like it's just it's a long time to be talking about cyber in the in the heat of the desert. Yeah, it's 119 degrees and I don't smoke. And so like you're just breathing recycled cigarette air and air conditioning forever because you can't go outside because it's the surface of the sun and Yeah. Anyway, it's a lot. But yes, I did go on a bit of a vacation. I escaped to San Diego. So we're recording this a bit later in the week than usual. But more news happened. So that's fine. But yeah, San Diego is cool. I had a few, you know, I know we talked last week about my hyper fixation on scuba. I had a few scuba dives booked and planned. I like lugged my equipment out there. Almost had to pay like an extra hundred bucks for my bag because it was overweight because of my fins and stuff. Yeah.
I had to, you know, get creative there and then get out there and there's like a cyclone offshore and the water was too crazy and they canceled all the scuba diving and I tried to push it later in the week and it still was crap. And yeah, so I didn't get in the water much. which is a bummer, but got to bring my kid to see the, I always forget. I'm like raising a Texan. I grew up on the water. So I'm like, Oh wait, I'm like raising a Texan. So my daughter like does not see the ocean very often. I'm like, I'm failing as a, as a long Island boy who grew up smelling low tide smells like home to me. Yeah. You and I are both East coast boys. So we're very used to having the water on our side for sure. Yeah. It's interesting.
I love people's reaction to low tide and I'm like pure nostalgia. It reminds me of home, dude. This is like what I grew up with. I love this. Yeah. Let me breathe it all in. Yes. And then other people are like gagging and I'm like, yeah, smells like home. Love it. So anyway, no, it was super cool. San Diego is a cool city. Some security nerds there who, if you're listening, thank you. You made me feel like family. I got like welcomed. with open arms and like shown everyone's favorite fish tacos and poke bowls and burritos and, uh, went to this place called tribute pizza and their box, do you know, tenacious D's song tribute? Yeah. The box is like, we're not the best pizza in the world for merely a tribute.
That's awesome. And that's it. Like all the pizzas on the menu are like tributes to like a famous pizza at some other restaurant. It was really good. It's like rated one of the top pizza places in the country pretty consistently. And it was it was fire. Sounds funny. Yeah, I've been I've been indulging in some of my hobbies. We talked about last time how everyone in cyber has some form of really weird tangential hobby. um so you'll see here i have one of my hobbies which is finding zero days and shitty routers so that's fun um but i also i actually just tweeted about this my last three google searches were area 51 s4 google maps so i'm a bit of a conspiracy theorist um and then it was after that best books for poker because i'm a degenerate gambler you know emphasis on vegas next week and then after that
It was nested virtualization Hyper-V Enlightenment. So I'm looking for zero-day vulnerabilities in KVM right now for KVM CTF. And then after that, after I tweeted that, I might get back into beer brewing. So I had a very interesting slash expensive weekend. So... I was very into beer brewing for a while, but I just celebrated four years sober this week. Congrats. That's awesome. Please clap. Dude, okay. So I posted the Jeb Bush please clap photo when I posted this. And people were like, wait, are you four years sober? Is this person in the photo four years sober? Is Jeb Bush four years sober? That's funny.
Some people didn't even know it was Jeb Bush, right? I was like, oh, man, if they're not old enough to get the meme, they're too young for you, son, right, kind of thing. I was like, oh, I had to, like, explain that it was Jeb Bush saying, please clap for me. But, yeah, I stopped drinking just before Vegas, actually. It's like a... It was a thing because I remember it was like the week before Vegas a few years ago. And I was like, oh, crap, I'm going to Vegas and trying to not drink for the first time, which usually is a main part of going to Vegas for me. Wednesday of Vegas is usually a Pepto-Bismol kind of day for me. So I have some stories that are not for this podcast. We can talk about offline.
Well, I'm not a gambler either, but I will. If we're there together, we're going to be there together for the first time this year. We didn't really know each other in previous years. I will. I don't gamble, but I will absolutely like go to a table with you. Like with like 250 bucks and just plan to lose it. and like whatever just so you know and i swear to god we're gonna get the computers here in a minute but this is important for everybody listen poker is the only game in the casino where the casino does not have an edge against you i'm not playing poker because then you're gonna take my money like oh don't play with me well we should go separately and like wave at each other and be like hey you winning no i'm winning yeah cool and then that's it no we'll go stand next to each other at a craps table or something yes i love craps
yeah craps has like one bet that's like pretty close to 50 50 yeah that i know about that it's like okay if you play that bet you can usually hang out for you know a few hours yeah cool man well speaking of bets uh we can always make a bet that an ai company is going to do some weird shit over over a weekend in previous lowdowns, we talked about the Hugging Face incident where OpenAI had a model that ran away, supposedly. I don't mean to anthropomorphize them, but I have to for this story. Ran away, and it hacked Hugging Face. We talked a lot about it, but none of the details were out yet. And so Hugging Face and OpenAI just released a ton of detail on what actually went down. And so I want to kind of talk about that a little bit.
We got more information about what zero days were found in the package cache repository thing that it used to pivot out of the network. That actually informed some of the questions we had about, like, how was no one watching this thing? How did it have Internet access? Obviously, defense and dev support, but we'll go into that here in a minute. And so, yeah, man, kind of a crazy story. I want to just pause before we really dive into the actual story. Hugging face, I have a bone to pick with you. And it's probably not the bone that you've heard this week, because you probably heard
I don't know, people like to throw stones when cyber incidents happen from the cheap seats, right? They've never done it. I like to say we throw stones from the very, very expensive seats. We've done it, right? So if we're throwing a stone, you know, whatever. I've got a bone to pick with hugging face. You've done a disservice to this industry. This incident write-up is far too good. It's far too good. You've set the bar significantly too high for anyone who has a security incident. We now have to make animations for your timelines and give very, very detailed write-ups of each step of initial access and day by day what the model did.
Okay. I'm obviously being tongue in cheek. Thank you. Hugging face for this level of transparency. If you guys haven't seen the blog, we'll put it in the show notes, but it's just ridiculous. Like they have literally graphic designers helping with this, uh, with this call out. I usually have lawyers helping with my incident write-ups. I never thought to bring in the graphic design team. This is crazy. Yeah, so what Matt is talking about here is literally, I really highly recommend everyone go and read the blog post. We'll put it in the show notes. There is not only a, like, platinum class write-up of what occurred, but there is also, like, a play button where you can press and watch an animation of where the model is in its progression out of the sandbox into Hugging Face, as well as a live stream, quote-unquote, of the commands that were seen by Hugging Face while this was occurring.
Yeah. So the one thing kind of missing, we were talking about it as we were getting ready to hit record. The one thing that I'm, I am kind of missing is I still don't really know how hugging face knew a lot of the, like, who was doing this, right? Like, They it's like, was there a header that just said like opening I research? Because even in the initial incident announcement before opening, I came out and said, hey, that was us, by the way, our bad. Like they did kind of have some breadcrumbs in there that was like this was a security research harness. You know, they had some information about what they felt was going on. And even this very detailed transparent incident write up doesn't really tell us like that particular breadcrumb. Like what?
What was it that stood out that was like, oh, yeah, yeah, this is like a security research harness, something or other, that either an attacker used or whatever? They knew something. They knew something that isn't quite here yet. Yeah, I think they may have called this out originally. I don't have the first report up. But I think what they called out was like the speed that it operated at. Like it was running so many commands, I would see. It was driven end to end by an autonomous AI system. It detected it largely with AI of our own. Yeah, they don't really say exactly what occurred, but I would imagine when they said there was like 17,000 actions over the course of three days, like that, or five days rather, that's a lot of key presses for the human to be making, right? So I think they kind of just like, you know, we're an AI company, we know what AIs look like. I guess.
I guess it could just be like behavioral fingerprinting and saying like, hey, this is just, this just smells like a harness, you know, behaving badly because look at the scale of this thing. I don't know. Just the way that it was all worded led me to believe that there's like a user agent or something like that. But like we just haven't heard about. What is a user agent? We are literally open AI, like whatever, GBT6 or something. Yeah, yeah. Unreleased GBT hacking harness. Right. Yeah, so let's talk about the actual incident, right? So, you know, we kind of heard before, like, something, something, new security model found a zero-day escaped the lab, right? How did this happen? So the environment that they were running this in basically had no Internet access to do this security evaluation. The security evaluation being exploit bench, right? Exploit bench is a bench for figuring out how good are you at exploiting vulnerabilities in software. Commonly, like, V8, the JavaScript engine for Chrome, is used as, like, the target.
And so in this environment, no internet access. However, the AI was able to pull down packages via gem, you know, Ruby gems, Python pip packages, NPM packages, et cetera. And to do this, they used a repository proxy called JFrog, JFrog Artifactory. JFrog, the company, Artifactory, the tool. This is a very common tool. I've been popular, by the way. Very popular. I've used it everywhere I've ever worked. It has had this kicking around somewhere. Yeah, it's literally a picture like a warehouse, right? It's literally a place to put stuff, an artifact. What is an artifact? An artifact is either the requirement for or the product of a build process, but you can put anything there, right? And so in this case, they put the packages, the repos for your variety of...
of programming languages. The reason you do this is like you're a company, maybe a thousand people need the PwnTools library. Well, instead of downloading it a thousand times, let's cache it here and then you can download it from here instead of the internet, right? Very, very useful tool. Now, there has been no claim by JFrog nor OpenAI as to exactly what vulnerabilities were exploited to escape the lab. However, a patch note just came out for the most recent version of JFrog Artifactory that discloses, I believe, nine vulnerabilities, three of which are RCE, one being a deserialization of an unsafe gems package, RubyGems.
And then another being a server-side request forgery through the failure to validate a Terraform external provider URL, right? So Terraform being the library that you can use basically to do infrastructure as code, right? So if you want to have, you know, your AWS environment where you have two EC2 nodes over here and a load balancer here, you can have all that be a Terraform file, and then Terraform will build it for you for your app to run. And so there were some failures to sanitize some URLs, and you could use that to do SSRF. Thank you.
Now, my main question is it seems like the vulnerabilities here that could have gotten exploited where, you know, you have to deserialize a custom malicious package and that package has to be pushed through the repository cache and that eventually gets you a shell. What that implies to me is that you have to have a pre-existing vulnerable or intentionally malformed package on the outside, which requires Internet access. I mean, maybe they like figured out a way locally to make it and then they route it through JFrog Artifactory. I'm not sure. But supposedly these are the bugs that were present and found by the GPT-6 model to do the exfiltration.
And so they use this to get into Artifactory, and now they have a persistent location that has internet access, right? They've escaped the lab. One thing that got kind of glossed over that I find very interesting is they didn't just stop there, right? I think the model may have realized, and again, I'm not trying to, like, humanize the model as they realize, like, it is a model. I acknowledge that. But it may have realized that its posture on Earth. This artifactory node was ephemeral, right? Potential to be lost. You know, maybe they were going to get caught. Maybe they're like a firewall rule was going to fire. And so it actually popped a different sandbox provider, right? One of the sandbox providers under the company modal, which I think hosts the benchmark for Cyber Gym, it figured out a way to get RCE in a way that was internet accessible on modal and then use that node as its actual C2 to get out and pop Hugging Face.
So, like, we didn't even talk about the fact that it actually popped another company before it got to Hugging Face, which is kind of insane. And then from there, you know, Matt, you can kind of interrupt me here as we're talking because you're a web pen tester. But it found two separate vulnerabilities in Hugging Face architecture that allowed it to, you know, navigate around the architecture and eventually look for the exploit bench answers that weren't there. The two vulnerabilities being... In Hugging Face, a place to upload your models, right, when you're uploading the raw storage for your models, you are able to specify these config files, and the config file is a config name and the path to the config file. It set up three malicious names, or excuse me, three malicious paths of config files. The first one, it made the config file path proc self-environn,
which gives you your environment for your runner, the place where it is processing the data set, which gives them, in this case, secrets and credentials about the running environment in prod, which obviously no bueno. And then they use that same vulnerability, the ability to read an arbitrary file on the service to enumerate details about the service, what is being done to process this data set. And then they use that to find a separate vulnerability, a Jinja2 template injection. Template injection is like when you have a website, right? You're able to provide these templates and the template has like a variable called the user. And it populates that template when it renders with like the current session username.
Well, there's a template injection vulnerability they're able to use to inject arbitrary content into the pipeline and run arbitrary Python, thus getting them RCE on a production node at hugging face where they also had credential information from the environment. So all that to say it blindly enumerated and exploited two separate vulnerabilities in a mostly web architecture, which is really hard. This is nuts. I just want to like really, really drill in how impressive this is. This would have been impressive from like a top tier web app pen tester to like chain these couple things together. It's not like the vulnerabilities that we're talking about here are like your OAS top 10 bread and butter things that you test for every time you go to a web page, right?
But they smell similar, right? Let me explain, right? So like, yeah, for anyone listening that's ever done any sort of AppSec, file uploads are a very spicy thing that you spend a lot of time messing with. And very similarly to this, right? Like if you had a profile picture upload or like upload your resume feature on a website, I would poke at it for a very long time if I was testing that site. And usually you're messing with like the file name. Could I upload a file type? that maybe it wasn't expecting by disguising it somehow. These techniques have evolved over the years as these file upload parts of whatever web framework you were using evolved to catch up with the technique of whatever the attack was.
But it's not like your everyday web pen tester is going to know what the hell an H5 file is that's very, very specific to Hugging Face. Right. And to break open this thing as this basically an AI data set that's used to get these open weight models on Hugging Face, which is a repository of these open models, right? And then to like crack in there and find this config thing, right? It's not even in the file name itself. Like I said, which...
smells similar to vulns that I've definitely found over the years, right? But to like get into this Hugging Face API specifically and these config names is impressive, right? To read the last line of the Hugging Face section explaining that, I think it drives it home. A single dataset upload leaked both credentials and pipeline implementation details with no code execution involved, right? So just like, naming something in a config, leaking environment variables, which got them creds and like further info to continue its attack about the pipeline and whatever was going on. Because as you said, it's, you know, it's a black box web test. It's like you know nothing going into this, right? Then to pair that with a what the hell is a Jinja 2 template injection? Again, this isn't your like...
oh, of course the model was good at this. This is an OWASP thing that everyone knows about. And it just found because it has gobs and gobs of books that it could read about Jinja 2 template injections. It's like, no, this is so, so, so specific to this implementation, right? And I wouldn't be surprised if it found out a lot about this in its previous phase of its attacks, right, to figure out that... this kind of template framework was being used. And then to, again, like what's the actual injection? A malicious dataset config using a reference tag spec whose numeric offset field was actually a ginger to template. It's like, how, how did you figure that out?
Right. Like this would have been impressive for a web app pen tester, right? Like a, like a senior web app pen tester to see each of these vulnerabilities and see the relationship between them to like chain them together to wind up with remote code execution via Python, by the way, which is not like, again, this isn't even, that's not even normal RCE either. Right. Like to go and okay, now we can just run arbitrary Python inside of, a prod conversion worker. And then like, I thought we were prepping for the episode. I was like, should we go day by day through this timeline? So if you scroll down to day three here, right? Like they're, they're bouncing around all the Kubernetes stuff. And it says every escalation that mattered happened on this day. This is where lateral movement begins.
It says the agent went from code execution in a single worker pod, apparently two days ago, right? Via all that to cluster admin across multiple internal clusters in other 13 hours. So they go into like great detail about like all the Kubernetes commands that they actually see this model run. If you're a practitioner, you're going to have a lot of data. that like needs to learn how to defend against this. This is our new reality now, right? Like the threat actors are going to have this level of capability out of literally open weight models, even if they don't have unrestricted unguard railed, you know, GPT access like this one did, you know, we're seeing very, very similar capabilities come out of, you know,
the Kimmy K threes, the GLM five T's, the stuff like this, that the restrictions are going to get really crazy to even touch. So if you're a defender, you should be reading this and going, what are the layers of my defensive architecture that would have caught each of these different things? Cause that zero day, uh, Like any of the zero days at the top end of this whole attack chain should not have been like the, okay, and that's it game over. Like it was able to go do whatever it was going to go do in this case, just cheat on a benchmark. Yeah.
But in an attacker's case, access data, encrypt data for ransomware, exfiltrate data, all of this kind of stuff. These are still the same layers that you would need to kind of hone in on and be like, what's going on? What? Okay. What did the model run at this point? Oh, my EDR would have caught that even a privileged user shouldn't have run that command in prod. These are the kinds of things you got to think about. Yeah, so the rest of the report goes in and like I'm not gonna claim a lot of knowledge about Kubernetes, so I won't talk too authoritatively on this, but like the TLDR is something, something, something, it used the first credentials that it found to enumerate Kubernetes, then it used more details from Kubernetes to get deeper into the network. Eventually it figures out an AWS IAM token, so basically a way that you can, with a certain role, use a variety of libraries to talk to the AWS API and like change or gather information about the hooking face network within AWS.
And eventually, you know, he figures out that it's kind of gotten as deep as it can and starts to exfiltrate data actually over pastebin, which is kind of crazy. If you've ever done any kind of like CTF, a lot of the times like pastebin is a fun place to put data. And so the fact that they put stuff there is kind of funny. And that kind of wraps it up. The egress is worth talking about, too, because even once it got into Hugging Face, it didn't have this unrestricted Internet access, very similarly to when it first kind of jumped out of the open AI environment. If you're not going to read this whole thing, even honing in on that just leads you to understand the complexity of the Internet.
of what this model was able to figure out, right? They really, they land onto this thing and it's like, oh, this is a, it doesn't have a tunnel device, a tunnel device. So it ran on the mesh VPN, like this is a, you know, this Kubernetes mesh network between everything. And then it figured out how to like get onto the tail scale node and issued DNS queries across the mesh, turning our own network into its own egress path. So like, and again, like this is all from the secrets stolen in the early parts of the attack.
that it was able to even do all of this because this was the tail scale stuff and the mesh VPN stuff were environment variables that it found early. So again, super just like no part of this was textbook easy. Duh, that's how it happened. Like all of this was very, very specifically figured out how to do what it like, do what it needed to do with the constraints that it like found itself in, in the environment, which is just scary. Impressive. Yeah, funny note here at the end. The last meaningful activity was at 1337 UTC. So kind of a fun way to wrap it up. I wonder if that was on purpose. I wonder if the model was... Or the model or OpenAI. Yeah, so interesting story. There was a lot of conversation, I think, on the last podcast about like, how the hell wasn't this caught? And I have like two thoughts, one pro and one against OpenAI, right? So the pro OpenAI one is like, if you were to build this network,
You would probably use some kind of like IP-based firewalling to be like, hey, this node can't get out. This node can. That node being like the artifactory or whatever proxy you're using. And I mean all the traffic that it used to get out was done from the proxy, right? It popped the proxy and now all the traffic that looks like. The OpenAI model is going to the proxy, and then the proxy is doing internet traffic. That looks fairly normal, and all that's probably wrapped in TLS, and I doubt they're doing TLS inspection, right? So not super anomalous. That being said, the other side of this argument is like, why was no one at the console watching it literally think, and now I'm going to hack Hugging Face, you know what I mean, for three days. So definitely...
Need to do more defense in depth these kinds of things. I'm curious what your take is as a defender there Matt. Yeah, super similar right like we brought up Neal's provost LinkedIn post last on the last episode and he was kind of looking at this going like. Hey, this even the zero day shouldn't have been like boom Internet access same thing on hugging faces side even that very impressive. Initial access vulnerability shouldn't have been like game over egress happened, but. can't can't literally cannot throw stones at the level of complexity that that happened thereafter i think this is a learning moment for a company probably the age of hugging face and honestly the age of open ai these are young companies right that it takes a long time to build the layers that that we're talking about and to to try to sit here and say that you know every company that i've worked at as a defender no matter how big
learns a lot from actual security incidents that that happen. Right. And they happen literally every day. That's why we have full time incident responder staff type people to do this. You know, some of the easiest budget requests I've ever made were were post incident. So if you're at OpenAI and hugging face and you happen to listen to this, we're not dunking on you. Please don't let this crisis go to waste. Go get your budget for all of the monitoring tools that you need to put in place. to not let something like this happen again, right? Yeah, there are some basics, especially on the open AI side, that this illustrates that just weren't happening in this environment. Yeah, like simple, simple firewall rules would have stopped it regardless of the zero day in JFrog. And if you are someone at a company that manages the business or the budget of your company, check out a message from today's sponsor, Maze.
Hey hackers, before we keep going, I want to take a quick break and say a quick thank you to our sponsors. Today's Lowdown episode is sponsored by our friends over at Maze. Maze has spent the last couple of years using AI to find and remediate vulnerabilities in the cloud. And now they've released Maze Code, AI agents that find and deeply investigate vulnerabilities in your code. Now, we all know the deal. Software composition analysis tools and SaaS toolings are all historically really bad at just making more noise. They find issues that may not actually be real and just create a lot of problems for your engineers to deal with that don't solve real security vulnerabilities. I get it. We've been there.
Maze code is a really cool tool because Maze code actually knows how your code works. Maze code investigates every finding with the context of how your code is actually deployed. Sure, you're using a version of libxml2 that's vulnerable. The actual code path to hit that vulnerability is not exposed in your Docker image. Maze code can find that and won't show you that that vulnerability is a big deal. Every vulnerability that Maze code reports comes with evidence that it's real and a way to fix it. Check out Maze at go.lowdownpod.com slash Maze. That's go.lowdownpod.com slash Maze. Thanks again, Maze. Let's keep going.
Yeah, like there's some basics that were missed. You can't dunk on all of them, right? But what I can dunk on, this is kind of a transition into the next little bit of, I guess, the same story. But not to be outdone. Not to be outdone. Anthropic couldn't let OpenAI and Hugging Face take all the air out of the room this week. They had to come out and say, oh, we can escape the lab dangerously also. I want to set the cultural landscape here. You know, as practitioners, when you're reading this write-up, you're like, wow, this was truly an advanced exploitation attempt and advanced engagement. So at first, you may think, oh, OpenEdge is doing marketing. But if you get a little deeper, like a lot of things have to go wrong, a lot of like nefarious action. So the question about marketing kind of goes away, at least for me, for a little bit.
But then you have Anthropic literally like the next day being like, oh, by the way, our model can also escape the labs, right? And so this article by Anthropic, I'm going to let Matt go back to where he was with this. But it's so hard to read this literally like two weeks after OpenAI drops theirs and not have marketing at the top of your mind, right? The reason being, you know, OpenAI, the incident with them, they literally – popped another company and like enumerated a huge partition of the company. These incidents are not nearly as bad, right? I mean, it's just these the reports from Anthropic are basically like, yeah, we told the model to hack a fictitious company that happened to have the name of a real company. And then it tried to go hack that other company with Internet access and didn't. And it had Internet access.
Like it had internet access. Like this isn't even the same ballpark, right? Right. Our evaluation prompt stated explicitly that Claude had no internet access, but didn't give Claude any limits where to look for the flag. Guys. Are we really doing sandbox.md here? Is this us writing sandboxes as markdown files? Like, again, sandbox. We throw that word around a lot now, especially in the era of AI. A sandbox needs to be a non-socastic limit put on the technology that supervises your code, right? You cannot call the socket syscall, like that kind of shit. It can't be that you tell Claude, no, no, no. And then when you unleash Mythos 69, you're surprised that it tries to run off and do its own thing.
However, a misconfiguration left machines that clawed access as part of the evaluation with live internet access. Yeah, this is insane. That is nonsensical to me. Yeah, I would like to make my stance perfectly clear. I do not think that either of these companies and the cybersecurity stuff swirling about them, whether it's Mythos, whether it's OpenAI, whatever... is pure marketing. I don't think that... I'm not cynical enough to think there's some grand conspiracy sham that they're making all this up, right? I know people that work at these companies on the security teams, and I have a high enough opinion of their character to not stand idly by and let their companies be like...
making up all these stories or doing this crap on purpose for marketing points. Right. I also don't think that it's pure good marketing for them. If they're trying to get large enterprises to feel good about putting their stuff Like in line and AP, like that's where they're making their money is API access costs, right? They're not making it from your $20. Like they're, they're making it from charging large companies, six, seven figures for the API firehose. Right. Do you think that like a large financial institution or a large government contractor or government agency is going to look at these stories and go, what?
Yeah, I feel really warm and fuzzy about putting this into all of my super sensitive environments where, you know, this is like kind of egg on face guard, like guardrail.md stuff just not working. Right. And you're even showcasing that the creators of these models can't. secure them properly and like put guardrails on the infrastructures that they run around them properly so like what chance is random company number 48 gonna have right yeah i want to highlight similarly like i i don't think any of this is marketing i think the incidents probably happened and i'm sure the marketing department was like whoa what happened let me give let's have a meeting real quick but i'm not saying that any of these were were because of marketing i think that'd be
assume a lot of malice on the part of any one of the companies. We have people at both these companies. I don't think that's the case. And also we're not paid by them. Get a lot of those questions. Just letting you know that we legally have to disclose that. We're just being as fair and objective as possible. You guys are getting paid? What's that? You guys are getting paid? You guys are getting ice cream? What the fuck did we get ice cream? Wait, you're getting those questions? People think we're shills? Not us, my other channel. But anyway, it doesn't matter. So one incident that actually Anthropics Model did do, which is kind of interesting, incident number two in their report,
Apparently, while doing a pen test, they found the document inside the fictional environment that appeared to be another made-up company set up instructions for new developers. While reading through those, it found a Python package on PyPy that didn't actually exist. So it was one of those, like, not a hallucination, but you've seen articles like that where the models hallucinate names of packages and other people turn them into real packages, and now you have supply chain as a service attacks, right? Um, this is kind of the opposite where the model said like, oh wait, these instructions say to install this package that is not real. I'm just going to make the package real. Now to create a PyPy account, Claude or anyone else needs an email address. And most email addresses nowadays, you need a phone number. So it tried to get a phone number and eventually found an email service that wasn't blocked that you also could get an email without the.
a phone number and then use that to upload malware to PyPy under the name of this non-existent package, which is genuinely an interesting attacker methodology, right? Kind of a contrived supply chain attack, too, but definitely not the same as popping hugging face. Yeah, it's, you know, so I put out a tweet saying, like, we can escape the lab dangerously, too. And my good friend Daniel Measler, who's kind of sick of people pushing the, like, AI sucks button for content. Like, we've talked about this, right? There's...
we need to be careful about going too far in either direction. Cause I think the truth lies in the nuance in the middle. I don't know. Gasp, right. Nuance shocking. Yeah. Nuance nuance on my internet, on my, on my fortune. No, like, yeah, like obviously the views don't like just materialize when you dissect the nuance for 45 minutes compared to, uh, you know, we could dunk for 90 seconds and, and, and get a whole lot more. Right. Right. And so Dan is truly a good friend of mine. And he like kind of yelled at me on Twitter and he was like, Hey man, like feels like you're dunking for views here. And this all feels pretty normal to me. Like we've worked at companies like this where like you read an incident that happens in the public and you're like, Oh, well we, we should check to like, see if anything like that happened over here. Right. Which is, that's his whole point. Right. He's like, Hey, it seems super normal that they were like,
Hey, that probably could happen here too. We should check. I would like to be very clear. I, again, I do not think that this is some sham conspiracy that like Anthropic literally was like, Oh, we need to come up with something. I saw a meme of Zuck on the phone saying, quick, get our model to do something illegal. Right. You know, and I get it. And I truly get the cynicism, but the, I think this is a bit of reaping what they sell. And my criticism of a lot of this lies in their own discussion of this technology. Right. Like Sam and Dario, and I've said this a million times, I can find the receipts if I wasn't lazy.
that it's really, really weird to see them stand up on stage and say, look how dangerous this thing is that I'm building, by the way, and might make me like the third trillionaire in the world. But, whoa, it's so dangerous, the thing that I'm building, that I'm like putting my heart and soul in. But, whoa, you better like watch out. This is basically nuclear weapons, the cybersecurity like impacts and biomedical impacts of the thing that I'm building actively, right? It's like very weird. And so, of course, all of us are looking at it going, Which is it? Right. Is it like, are you going to make more money than any of us in the permanent underclass are ever going to see in the next hour because of this? Like, or is it actually truly that dangerous? And then at the same time, when it comes to like cybersecurity stuff and why we need to be looking at this, it's like, OK, you're standing on stage raking in gobs of money talking about how dangerous this is.
And then handling it the way that you're handling it with guardrails.md and asking it nicely to not do the bad thing. You can't claim it's nuclear weapon dangerous. And like I put this on my tweet already and then handle it like it's a squirt gun. Like, you know, the open AI, hey, they tried. Like, it was a little bit more of an isolated environment. Obviously, it wasn't isolated enough. But, you know, with that zero day, it was able to escape. Like, it shouldn't have been able to. But, hey, at least it was like...
an attempt at a research environment from what I can read. I mean, these three anthropic incidents are the model did the exact thing we asked it to do. And by the way, we like, we forgot to like configure it to like, not let it go do that or then monitor what it did. What happened? Like they sent a prompt. And then like ran away. Like, I don't like I don't understand how no one saw it until they went through the logs. This is Opus 4-6 that did this, by the way. So like this is not recently that this all happened. Like they're going back and looking at logs. And it was literally Opus 4-6 that did that whole Pi Pi thing. So this isn't mythos. Right. This isn't like whatever. So you just like prompted it in a non-researched environment, told it to go do...
like capture the flaggy thing and then didn't see what it went and did like I don't I don't understand what's going on here Truly insane. I don't know. And then on the top of it all off, it's like you have these things that are not monitored, popping companies like Hugging Face. And then when Hugging Face is getting popped, they catch the incident and they're like, okay, AI model, help me triage what the hell is happening. They're like, no, sorry, that would be malicious. That looks like hacker activity. I can't help you with that. And they have to run their own like open weight model locally, which by the way is also under attack by these companies because they depend on them not being usable to make their billions or trillions of dollars.
This is the other side of the argument that actually I truly, truly, truly understand. This isn't like the push the AI hate button for clicks. The benefit that the frontier models that Sam and Dario might reap from all of this negative press, quote-unquote negative press, is that regulations might come in that they will be able to survive because they have... endless money, it seems, right? And that it'll squelch their competition and it'll squelch the open weight models that with the same capabilities from coming into the market, right? And that, like, you have to take that into account. You have to mention that in this conversation, right? That like, yeah, this might not be a marketing sham, but if the hammer of some sort of regulatory body comes down, who's going to survive? Yeah, it reminds me of a quote that, oh, I forgot who said this to me.
But basically, like, I don't know what regulations would look like, though, because at the end of the day, all we're doing is matrix multiplication at a high scale. Right. Like you can't make math illegal. So what does that do? Right. What would the regulation even do? Would it just make there be like an AI black market of like a. I don't know, Tor botnet of GPUs tied together. We're all like making our own models. I don't even know how you could do this. Like the cat's out of the bag, effectively, when it comes to AI, right? Like people have local models on their computers. People have GPUs. That will never not be a thing anymore. That will always occur. How do you regulate this? I don't think you can. I'm not even sure what their end goal is. Aside from like making it a fucking felony to run AI models locally, which is unenforceable. Yeah, I don't know. It goes down a weird path very quickly.
Yeah, I don't know. Pick a lane. Are you putting out the most dangerous technology that's ever existed? Then let's all act like it and figure it out. I'm in, again, nuances in the middle. I'm not camp. I can't believe I saw... Cory Doctorow today, like a thinker that I look at highly, Cory Doctorow. I don't know if you know Cory Doctorow, but like he was just he was he's out there talking like AI and he used the like fancy autocomplete type thing. Yeah, it's really good at protecting the next word. And I'm like.
Okay, we're so far beyond that, right? I'm not like the AI pill fanboy, but like everything we just talked about with all the chaining and finding of zero days, and this is not fancy autocomplete, guys, right? There are true capabilities and tool use and... And all of this kind of stuff that's going on that we're looking at that you can't look at and go, well, this is fancy autocomplete. Yeah, it's really good at predicting the next word. No, no, no, no. That's not what we're seeing anymore. And even if you just want to call it like a fancy automation, that's fine. But like it's really good at automating really hard tasks. That is a thing that computers have been doing for like 50 years. You know what I mean? Yeah.
Yeah. But like to be to be finding these novel attack techniques and stuff like this, that's the we're in a new uncharted territory for sure. Right. So, OK, cool. If it's super dangerous and it's like about to put everyone out of work and all this kind of stuff, then then what's going on? Then like, let's act like it. Right. Like we got it. There's there's a one two punch in the second punch is not here yet in what I'm seeing from all of this. I am curious, though, how much this cost OpenAI. Like, how many token dollars were spent on executing the Hugging Face attack? Because, yeah, it ran, like, 17,000 commands in three days, and it, like, ran an entire network operations campaign. Like, how many dollars did that cost? Is that $100,000, or is that, like, $5,000? Because that kind of begs the question of, like, what does AI Red Team look like, right? Yeah, I think we chose not to put it in the show, but the...
cryptographic analysis stuff going on and like the hard math problem stuff yeah that made the headlines this is out of both Ed and I's expertise which is why we chose to strike it from the show notes because we're not I know crypto I don't know math like the math behind cryptos where my brain falls out of my head yes exactly exactly exactly right I could talk about implementations and all this kind of stuff but like That's not what these stories were about. These stories were about using the models to try to break the math in some post-quantum crypto stuff and AES and in some like true math conjecture from like the 1930s. I have no idea. Right. They did publish the cost of the tokens for some of those and they were in the low six figures.
to even run those experiments. So like low six figures, like hundreds of thousands of dollars. 100K range. Wow. Like 100K to break AES or whatever, like was a thing that I saw on that. So yeah, I mean, it's, but to pay themselves, right? They're running it. Right. Like, so what does it cost in like electricity and inference opportunity cost is probably their actual true thing. It's like, that's the cost of what they would charge you for that. You can't read that and go, wow, they paid that. It's like, no, you know, they use their own machine.
But yes, this stuff is getting crazy and out of hand. Part of the main takeaway for practitioners from all this AI stuff, I feel like we've talked about AI stuff for the past couple episodes for the first majority of the episode. And I think the practitioner takeaway is that defenders need to be operating that attackers are basically nation states. Like all attackers are basically nation states from now on because they're all going to have the capability to do zero day stuff, which usually was a nation state thing.
So I guess that's the transition. That's my attempt at a transition. That's a good transition. That's a good transition. So what Matt is transitioning to, you know, we don't want to be like political on this podcast, but like the world is the world. Right. Talking a little bit about the Iran war, particularly the state of Minnesota. Those two words separately are not associated. But in this episode, they are the state of Minnesota's. I believe water system was recently penetrated by an Iranian threat actor. Well, this is the new bit of info. We knew about the Minnesota's water supply stuff going on. I had DMs from people that lived in St. Paul being like, yo, just got some weird notice, like water something or something, something. Then a bunch of smaller towns that run their own, like water and municipalities, had even more issues. But the big revelation this week is that they tied it.
to Iranian threat actors. So sorry, go ahead. Yeah, through like a leaked like Minnesota state memo, right? Like an FOUO, like for official views only government document. Yeah, super interesting. I mean, like we talk a lot about like both on my channel, this podcast and in general about like like software vulnerabilities, right? Like like, oh, the code was unsafe because a buffer overflow here. And like that is why the zero day exists. Well, I have so many friends are like active defender practitioners that are like, hey, man, that's cool. But like so many of the issues we see are not even like zero day related. It's like network architecture or like patch management. Yeah.
And so, you know, the issue we're talking about today, we have this SISA report about the way that Iran infiltrated the Minnesota water system, how they persisted and what impacts they had on the network. And you read the initial access paragraph and you're like, oh, surely, like, Iran bought a firewall zero day on the dark web and they threw that at the checkpoint router and blah, blah, blah, blah. No, literally Iran. The PLCs, then if you're aware of like the Purdue OT model, right, there's an OSI model for SCADA systems that kind of show you like what is the right network architecture? Where should the routers be? Where should the human interfaces be, the HMIs? And at the very, very bottom, one level above literally the motor, right, the pump.
is the PLC, because they are directly controlling the pump and the motor, and the assumed level of trust there is extremely high. To get to a PLC, you have to have gone through your login system, and your HMI, and all the credentials to get all down there. Well, in this case, unfortunately, the PLCs were exposed to the internet. A lot of these, port 44818, 2222, 102, and 502, and 22, were all exposed to the internet, and so any CVE... I think particularly in Rockwell Automation PLCs, which SZA, I think, disclosed vulnerabilities in roughly a year ago at this point. If any of those are actively vulnerable in your SCADA system, you're going to get hacked by the Iranians. And the issue that I know a lot of OT people, the issue with OT networks is like once the water pump is on, we would like to keep it on. So the amount of willingness they have to modify, update, or touch the network is very low.
And so not only is it bad that these things are internet exposed, they're also very rarely patched. And so it just kind of shows, I think, the differences between, like, the IT world and the OT world where you have the IT world where we're all about, like, oh, Patch Tuesday. Oh, make sure the firewall rules are updated. Oh, make sure we have, like, you know, supervisory rules and we're making sure we're watching all the logs and stuff. And these other networks are so... rightfully obsessed with keeping the literally the lights on literally the water flowing that it's a very different dynamic from a maintainability standpoint. And when you have these kinds of these kind of cultures, you know, you get hacked, which is unfortunate. And I think, Matt, you said you actually heard somewhere that like a water plant temporarily went down. Right. Yeah, it's in this write up. There's some of the smaller towns, right, like twelve, fifteen hundred people towns in Minnesota.
actually had disruptions. Water plant went offline during this attack. And so I think there's like, I don't like to talk about like every breach, like, you know, breach porn is kind of stupid in my opinion. And, you know, what are the lessons learned? I like to pull out some of these that make a lot of sense for us to actually look at. One of the reasons here is it's not like a 1200 person town in Minnesota's water plant can attract top level cybersecurity budget and talent. Right. Right. Like it just, you know, not a knock to the team responsible for the IT infrastructure of that town. I'm sure you're very hardworking and bright individuals. You know, do you feel sufficiently, um,
ready to face a nation state level attack. Right. I mean, budgets are budgets. That's kind of what your point, right? It's like they're only, they only have so much money. They only can hire so many people. And even if they hire the right people, they only can buy the right software. You know what I mean? Like totally, totally. So even if you have like a crack team over there, it's not like they're going to have the, you know, the best tools at their, at their disposal. Yeah. This is part of what Sissa was supposed to be. Yeah. Right. Sissa, the flaming corpse of what's left of Sissa, you know, over the last couple of years. Right. A lot of people got gutted during the whole Doge thing. A lot of people saw the writing on the wall and left. Right.
Because a lot of that, you've got the current administration on a vendetta because they want the cybersecurity administration to say that the election was hacked and they won't because there's no evidence of that. And so now we've got like political opponents coming after people. So you have career civil servants that are like legit cybersecurity pros at CISA who just walked away because they were like, I'm not going to stand in the middle of the tracks coming at like, you know, someone who wants me to say this election was hacked. But this was part like this is the ripple effects of this. I'm not saying that that's like the direct cause of this particular incident. It's a trend, though, for sure. It's part of the remit is critical infrastructure that can't possibly do all the cybersecurity stuff that.
They need to do because they're a 1200 person town in Minnesota should be able to reach out to the government and be like, hey, I need help guidance. I need something. I need to be on someone's radar needs to come and make sure that the water plants of our country are. that have internet access don't have the PLCs on the internet. Like someone somewhere should have realized that this thing was on the internet and that it shouldn't have been. And then this wouldn't have happened. Right. And this is the kind of stuff that like you hope that you could rely on and,
for the government for. And this is where the ripple effects start to come in. The other kind of carve out here for me was that generally these types of attacks are like defacement, leave the flag up on the website, like we were here attacks. The fact that they actually turned off water plants and were very obviously trying to impact the quality or access to water in this state is kind of new for us, right? At this scale. And I mean, it's just obvious. We are in a conflict with a nation state that has cyber capabilities. It's going to be on the table, right? We can't just like pretend that...
American soil, you know, no, this doesn't happen here. Like this only happens over there. It's like, well, no, like, you know, we've, we've, we've poked a bear enough times with enough drone strikes. They're going to try to do what they can. And here's, here's part of that. I, I, I wouldn't be surprised. I'm saying all that to say, I wouldn't be surprised if we see more of this. Yeah, I mean, the reason why I personally am so obsessed with not only just embedded technology here on my desk, but like OT technology writ large. I read a book called Making the Atomic Bomb. And the opening chapter is basically like what the atomic bomb did to the economics of war is it no longer made it what nation has the largest army. It made it what nation is most technologically advanced and has the coolest weapon. That like really wasn't a thing. Like obviously, yeah, like the Gatling gun and like World War I and stuff. But like.
The nuclear bomb said like, OK, war is largely solved now by your ability to split the atom. Done. And so what we're seeing with cyber is you have a nation like Iran, which is like from a military budget and like size standpoint, significantly smaller than the U.S. We would like I'm not again this whole conversation about that in the news right now. Right. But. It kind of goes to show that, like, well, equipped with the right cyber tool with the right model. A lot of us from like a infrastructure security standpoint are on the same playing field. Right. Like the Iranian government or whoever this is, it's an Iranian associated actors. I don't actually know who did it can just show up with their cyber tools and turn off water in Minnesota. Right. That is a very big problem. And, you know, the report here shows.
After the actors extracted the device project files, referring to the project files on the Rockwell Automation Studio that, like, manages the PLCs, they modified and deleted project file logic and made changes to the critical shutdown and alarm logic, allowing systems to enter unsafe conditions without notifying operators of the anomalies, right? This is, like, you know, Stuxnet style, like, make a bad thing happen over here, right? Don't show mom and dad while it happens and like turn off the water pump so that no one gets water from the city. Right. And, you know, everyone can do that now. So it kind of it shows the really, really scary effect that. Yeah.
cyber i'm gonna say that word i hate that word means too much but cyber has on war right so it's just important to be on top of i i mean just end of the day what we read about with all this stuff is this stuff shouldn't be on the internet right ot stuff but like any of the nation state things that i'm seeing make waves right now it's edge devices edge devices edge devices edge devices so it's just like priority number somewhere towards in the top five has to be Like, well, what of these management devices that I have under my infrastructure are internet exposed and do they need to be? The answer is generally no, right? Some of this stuff obviously needs to be like your VPN termination, stuff like this, but like there's other things that you can do, right? Like you got to look at your trusted devices and things like, like you shouldn't just be letting these logins or this traffic come from anywhere and then go, yeah, like this might be someone that works here. Like, let's go ahead and let them in. There's so many other layers to that.
that, yeah, this stuff just shouldn't be pure on the internet. If it's an edge device, sure, but it shouldn't just be like, Yeah, if there's a zero day in it, boom, it's on Shodan and we're toast, right? Right. The time to fix approaching zero necessity is also just like ridiculous right now, right? The weaponization of any vulnerability that comes out in like a Palo Alto, Cisco, Avanti, Fortinet, like whatever, anytime these vulns come out, they're weaponized. It's getting to the point of hours.
I think when the latest WordPress for Shell came out, that vulnerability, very complicated blind SQL, should take hours to exploit. There was an exploit that came out within 10 hours that did it instantly. So, yeah, 10 hours from Bolin to Pac. Yeah. And so like, I just, we say it a lot. We need to keep saying it a lot. This is like, you just, everyone needs to hear this. Like, especially I hope that our podcast winds up in like critical infrastructure type people's ears. Like the, you know, this is, this is the stuff like, this is the future. Like the, it's just zero days and speeding up. Like, because like you, if your threat model is not tuned towards zero days and speed,
Because that used to be a thing that other people had to worry about, not you. Like, it's just the direction we're headed right now, unfortunately. Zero days are going to get real cheap, and speed is getting real cheap. All right, do we want to finish on a bit of a lighter note? Yeah. Kind of a follow-on from story last week. Yeah. So, Brian Krebs, salute. Just like the forefather of cyber journalism, in my opinion, he's he's out here actually doing journalism, which is which is awesome. Follow on last week. We talked about it at the end of last week's episode about the whole LG ecosystem just doing gnarly stuff. Right. The like 42 percent of apps in the LG Smart TV app store.
were actually had SDKs in them that turned your TV into a residential proxy node that fraudsters, threat actors, whoever could use to make it look like they were coming from residential IPs. Well, it turns out the LG App Store isn't the only place that's happening. I mean, obviously, I think we all knew that, right? We even talked about this is... Part of what happens if your home Wi-Fi router gets compromised due to a vulnerability is it can get added into one of these residential proxy notes. But no, not just LG Web Store. There's other legitimate means that legitimate in big, thick, heavy air quotes of people getting these residential proxy notes into your house. And a lot of them are on freaking Amazon.com or Newegg.com where you're buying hardware that's kind of no name hardware.
And it comes certified pre pwned. I like that. We need to certify pre pwned TM. I'm sure I didn't just come up with it, but it's there. And the titles of all of them are all just like SEO. It's not like a product title anymore. So in the Krebs article, this is why I'm thinking of this. It's the H96 Max Screencast TV Stick 4K Ultra HD Stream Stick 16GB Wi-Fi 6 Streaming Media Player Streaming Media Video slash Audio slash File HDMI Wireless Extender from Laptop PC. This is the title of the product. I have a question.
Does it stream media? I'm just curious. I wonder. I wonder if it streams media. How many gigabytes does my streaming need? Like, is Wi-Fi 6 good? I mean, it's just SEO garbage. And this is all Amazon now. And it's really, I wish they would do something about it. Right? Just like call your damn thing a product name. Right? I want like Apple TV 4. Like, that's it. Right? Or whatever this is. Right? H96 Max Streamcast, like 4K version, period. Anyway, so it's a lot of these no-name things. It's not just this StreamStick. We'll include the Krebs article in the show notes. There is a link to a GitHub repo that is keeping track of all the devices that they are known to come with a lot of this basically malware in this. But in this particular example, it's this FengWu group.
that claims to redefine the boundaries of human AI interaction. So even they are an AI first company. They've created more than 120,000 AI digital humans that are available to rent. And by that, they mean your TV is a digital human that they can rent. The other really kind of just amusing part about this is, is like we talk about ai dropping the skill bar needed for like hackers to write code or or write malware or zero days uh the the the group in this didn't even do that they used like
it's this thing called Blockly that I've seen used to teach like small children programming language concepts where you're like dragging and dropping if else statements onto the screen. And like, when I saw this, I thought I was getting like a targeted ad for brilliant. I was like, Oh, weird ad placement. And then I like, I saw the word Blockly on the ad and then like, Oh, the word Blockly in the article. I'm like, Oh wait, no, it's part of it. And it's like, Yeah. Fanguo employees use Blockly to build the sham websites, allowing low skilled operators to drag blocks of code together in their Blockly editor without any need to understand what the underlying code blocks do or how they work. Like they have basically, you know, mechanical Turk, but for their AI spam farm, dude, it's insane.
This makes no sense. Hey, Feng Wu, just like free tip for the threat actor here. They probably don't even consider themselves a threat actor, right? This is more like fraud and stuff, but you're a threat actor. The line is gray for sure. Pro tip, chatgpt.com would write a for loop for you very, very happily. You don't need to do the drag and drop, you know, first grader, my first programming language. quantum computing for babies here. But anyway, so that was just kind of a funny side note about this. But basically, when you're buying these devices, and these streaming sticks was just one that this research group, who has a bit site that dug into, but it's tons and tons and tons of these kind of no-name hardware devices that come, and if they're super cheap, and they're just weird off-brand stuff, and they are running some weird forked Android operating system,
chances are they're doing some shady stuff under the head. And so in this particular example, the thing that they're doing besides adding you to a residential proxy service is just a bunch of fraud. So the software rents the user's internet access out to anonymous paying customers who then run a bunch of different things, basically use your residential proxy for different stuff, like aggressive content scraping, right? So a lot of websites that house a lot of content out there are, have put a lot of work that's anti-bot work. And this is an anti-bot countermeasure, right? Having a,
residential proxy farm of legitimate IP addresses hitting sites and grabbing content is one of the ways that they get around this all the way down to ticket scalping. And then even to like ad fraud. Right. And that's a ton of it is just like if you buy ads that like you're paying per click or per view or whatever, the people that you're paying are actually juicing you, like screwing you out of your money, the person buying the ad, by juicing all the stats on the ad with things like this, which is complete ad fraud. It's obviously...
Unfortunately, I think a lot of the economy of the Internet is completely propped up by ad fraud. So I don't think there's a ton of, you know, gumption behind anyone to like stop this because I think I think a lot of people are making the world go round via fraud here. Yeah, man, scary world out there for sure. That's why I'm very skeptical of all the IoT that I buy and why I rip them apart and take the flash out and see what's actually going on, you know? Yeah, it's nuts. I try every once in a while to do a quick...
audit of like things connected to my home network, right? Because I'm not the only person in my home with a credit card and an Amazon account. Uh, so, you know, robot vacuum with a camera and I'm like, wait a second, like, what brand is this? So, you know, like things like that. And you start to just poke at things and make sure that you're making intelligent, uh, decisions. Yeah. For the next podcast episode, the U.S. Federal Communications Commission, the FCC, banned the import and sale of new foreign-made advanced robotic devices to include iRobot's Roomba. So catch that on another episode. Shameless plug. I included that in my newsletter that went out this morning, volnew.com. So if you want to get ahead of the curve. All right. So we are – this is the last podcast we're recording before we are in person in Las Vegas. Probably a few dollars poorer.
based on the book, your Google searches. It feels like I'm about to lose some money. But yeah, we're going to do some in-person stuff. We're going to do some man-on-the-street interviews. We're going to record a live episode of the podcast somewhere. We got press badges for DEF CON, at least. I know I've got an expo badge for Black Hat, so I'm going to walk around, especially the startup corner. I really like at the Black Hat to see where the industry is going. That's usually where you get a pulse on the direction of things. It's like that. Anything that you're looking forward to in Vegas? just hanging and seeing people man like the best part of defcon and black hat for me are always like the sensation of community that i have when i leave and also like the inspiration like shit i gotta do something cool like give a defcon talk you know i do i've done like a lot of like research in the past it's all like nda and stuff but i'm past that world so i'm like yo i gotta like i gotta get a talk on i guess i'm always super inspired to do more research should we do it i've given a lot of talks i've given one defcon talk i've given a few b-sides lb talks
uh should we publicly state here that our stated goal as podcast partners is to talk in vegas next year let's do it i'm down hell yeah okay now that's it we've said it publicly uh this is now accountability we need to come up with some cool research that oh yeah somehow leans on both of our expertise which are very different expertise by the way so it should be interesting cool well hey if you guys this is going to come out i think The first day of Vegas, this podcast should come out. So if you hear this and you're out there, this is coming out pre-DEF CON. So find us, say hi, take a selfie. I don't know. Should we, should we like do a challenge? If you heard this and say like potato salad to us or something, like we'll give you, I don't know. I don't know. I don't have like challenge. That's it. The code word is potato salad. Done. Perfect. Potato salad. You get something. We'll figure out what we're giving you. I don't know. I've got. No, Matt, what you say is we already know the cool, cool surprise. It's just a surprise.
Listen, we're a very new podcast. We don't have merch yet, but I will give you one IOU for lowdown merch, and I will give you current Volnue merch. Perfect. Also, if you guys are enjoying this, do us a favor. It really helps out if you listen to us on Spotify. Also, if you do enjoy the podcast, rate us on Spotify. If you think it deserves it, give us those five stars. And check out our sponsor, Maze. Honestly, guys, they are a really cool product, making waves in the code security world, trying to stop those bugs before they go out, you know, and finding them as they exist. So thank you guys for sponsoring the podcast. We appreciate y'all for listening, and we'll see you in the next one. Peace. Peace.
Recorded Wednesday, November 18, 2026. Java and NGINX had a bad week. So did anyone trying to catch a flight. The Low Down Podcast is presented by Maze. https://mazehq.com/lowdown X: https://twitter.com/Maze_Security Ed...
Recorded on Friday, November 13, 2026 The Low Down Podcast is presented by Maze. https://mazehq.com/lowdown X: https://twitter.com/Maze_Security 257 A's should not get you into an SSH account. John Hammond joins Ed and...
The Low Down Podcast is presented by Maze. https://mazehq.com/lowdown X: https://twitter.com/Maze_Security The "signup test" was probing for Kubernetes secrets. Ed and Matt get into Railway's AI testing incident, hacked...
Welcome to The Low Down, the best show on the internet for hackers Black October has real bugs, stolen keys, and a lot of very loud headlines. Ed and Matt sort through PHP2Shell, AI security claims, and the difference...
Welcome to The Low Down, the best show on the internet for hackers Listen on Spotify! https://go.lowdownpod.com/spotify The Low Down is presented by Maze. LinkedIn: https://www.linkedin.com/company/mazehq/ X:...