Gen AI is helping cyber criminals become more efficient. And we're only at the beginning.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
A disturbing, yet not really surprising article or post was published by Anthropic yesterday. An article about an AI-orchestrated cyber espionage campaign, a cyberattack carried out with help of AI, with help of Claude code. And it's an interesting article. I already read it together with you here. So Anthropic, in this article, describes a, a cyberattack on various companies that was carried out almost entirely with
help of AI, almost entirely with help of Claude code, by jailbreaking Claude code, by getting it to do things it normally shouldn't do. And let's take a closer look at how that played out. "In mid-September 2025, we, Anthropic, detected suspicious activity that later, investigation determined to be a highly sophisticated espionage campaign. The attackers used AI's agentic capabilities to an unprecedented degree, using AI not just as an advisor, but to
execute the cyberattacks themselves." And that's, that is huge that you can already use AI, AI that is published by Anthropic, AI models by Anthropic, and the Claude code tool by Anthropic to carry out cyberattacks. And, uh, here's why that is important. Today we're living in a world where the most capable models are the models by OpenAI, Anthropic, X,
Google, and of course we've got some very capable too, but we're still at a point in time on a, timeline, we are still at a point in time where most of these models and the, the most capable models are controlled by companies, most of them by companies in democracies. Now, I'll not say that these companies are saints or don't do bad stuff, but they're not bad
actors in the sense of this article. They're not cyber, um, attackers obviously. However, in the future, in the not too distant future, we will be at a place where these models, these very capable models will also be owned by bad actors themselves. So right now here in this article, about a cyberattack that was carried out with help of Anthropic and Claude code. And it's bad enough that this is possible,
trick those tools and models into doing stuff they shouldn't we'll get back to how that happened, of course, but that is today. We still have to apply tricks, uh, to abuse these models. In a future, these models will simply belong to the bad actors themselves. There will be open models that are capable enough of doing that, so then certain control mechanisms, get back here in this article, won't even be there anymore.
We'll be in a future where bad actors have direct, uncontrolled access to very capable models can be fine-tuned for their purposes, that can be trained for their purposes, that can use tools that were purpose built for malicious stuff. That is where we're heading to, and even today where we're not there yet or where this is still a niche, even today, we have fully or almost fully AI-controlled,
uh, cyberattacks. So this is definitely a scary article and, and a scary future, which makes it very clear that cybersecurity and preventing attacks will be a super big challenge. It always has been, but now with AI where everything can be quicker and more automated and harder to trace back to individuals, it will be an even more important topic. But back to this article. "The threat actor, whom we assess with high confidence
state-sponsored group, manipulated our Claude code tool into attempting infiltration into roughly 30 global targets and succeeded in a small number of cases." So it was not just an attempt. They succeeded. "The operation targeted large tech companies, financial institutions, companies, and government agencies. Uh, we believe this is the first documented case of a cyberattack executed without substantial human intervention." This is so big, without substantial human
intervention. And again, that is why this is such a scary future where bad actors don't even need to rely on, um, these controlled AI models, uh, which they still have to rely on today. "Upon detecting this activity, we immediately launched an investigation understand its scope and nature. Uh, over the following 10 days, as we mapped the severity and full extent of the operation, they were identified." So again, these were really regular
Claude accounts. These people were using Claude, the model hosted by Anthropic. They did not kind of steal it, run it on their own servers models. They used the models you can use too via via Claude code. This campaign has substantial implications for cybersecurity in the age of AI agents, as I just said, because everything can be automated and it's already possible today, uh, where there are guardrails in place. And again, think of that future where we have no guardrails
actors don't have guardrails. "Uh, these attacks are likely to only grow in their effectiveness. To keep pace with this rapidly advancing threat, we have expanded our detection capabilities and developed better classifiers to flag activities." And that's the important part here.This is what Anthropic is trying to do today to make sure that their models can't be abused for malicious tasks, that Claude code can't be abused and ultimately, their APIs, which Claude code uses in the end.
This all won't matter at all in the future because in a future where bad actors themselves have their own models running on their own servers, these guardrails won't matter. Well, obviously they will still matter. Obviously, you still want to make it easier than it's obviously not all bad actors will have their own malicious models, but especially if we're talking about state-controlled bad actors or big cyber attacker groups. Let's, let's be
real. Of course they will have access to their on their own servers, so this will not matter at all in that future. Obviously, it will matter still because you don't make it easier and it will at least filter out a significant group of potential bad actors that don't have access to their own models. So yeah, it's important, but it will not be enough. Companies themselves need to ramp up their cybersecurity game, which is easier said than done because
it has been true for the last 10 years, of course, even without AI, but it's becoming even more important in the age of AI. So yeah, that is, that is the big problem here. Now, let's see how the, uh, cyberattack worked. Uh, the attack relied on several features of exist or were in much more nascent form just a year ago. Intelligence: Models' general levels of capability have point that they can follow complicance- complex instructions and understand
context in ways that make very sophisticated tasks possible. Not only that, but several of their well-developed particular software coding, lend themselves to being used in Sure, because now with models that are smarter, and just to be very clear here, we're still talking about models that just generate tokens, but of course by generating these tokens that are able, they are able to describe the usage of tools tools, they become more capable. With all the
fine-tuning they received, they also generate more tokens that are likely to be or more likely to be the tokens you want to generate, so that is what intelligence means here. They're not really intelligent, but they have been tuned especially for software development such that they are much more likely to generate meaningful output and especially also output that allows them to describe tool use and then use those tools, so execute code that does something. Uh, for example, send an HTTP request and so
on, and it's that combination that makes them more capable in the end. And of course, yeah, that's exactly what you need for automated cyberattacks, because you need a model that's able to follow your instructions related to that. You need a model that's able to send HTTP requests, phishing emails, whatever, and all that is what these models can do quite well in the end. Agency: Models can act as agents. That is, they can run in loops where they take tasks and make decisions with only minimal occasional human
input. That's another important step. This is what allows these, um, systems or these attacks here in this case to work with only minimal human input because these models and the software that uses these models can go for so much longer and it's important to differentiate here. The model is, is still just the thing that receives a prompt and sends back some tokens. That, that has not changed. But it's the software, Claude Code, for
example, that then takes that output and sends back another message to the same API with that output, with the original task, with some meta instructions like, "Please check if that output answers the question by the user. You got these tools available, please tell me if you want to use a tool." how the software around the models in the end makes these models more capable, not because the model does everything on the model is capable of giving the software the result it needs. The software then feeds these enriched
the model and it's this loop that keeps the whole system going and that leads to these agentic systems that can go on for longer, that can use tools and that require less human input. And yeah, tools, that is therefore the other missing piece here, of course, that models have access to a wide array of tools. They can now search the web, retrieve data, perform many other actions that were previously the sole domain of human operators. In the case of cyberattacks, the tools might include password crackers, scanners and other security-related software.
Because again, it's not all just GitHub MCPs. It can be all kind of tools you could, uh, expose to your, um, model or to the software that uses these models and that runs these agentic tasks. So they got a nice diagram in this article, but in the end the attack played out relatively, uh, simple. They, they describe it in greater detail down there. They convinced Claude Code to do stuff it normally shouldn't be able to do.
They had to convince Claude, which is extensively trained to avoid harmful behaviors, to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails, and that's the part where Anthropic back better, not just in Claude Code but in the models themselves, so on their API where they scan all the requests that reach their models, so to say, and take better... eh, they try to do a better job at detecting injections in the end because these attackers broke down their attacks into
small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm and was being used in defensive testing.That's good old trick. I think that is how jailbreaking was already done two years ago with the early ChatGPT models. Eh, you tell it that you need this information and it'll happily expose its system prompt. Kind of a simplification but that's still how prompt injections can
days. That you try to apply various techniques are very interesting techniques when it comes to that, eh, including the use of special tokens you message to tr- to get the model to generate output it normally shouldn't generate. The attackers then initiated the second involved Claude code ins- inspecting the target organization's So yeah, that's then essentially what Claude code did. Then with minimal human input, um, it
is in the end then used its agentic capabilities, its tools, to really, um, scan networks, write code, and do all that stuff without a human telling it exactly what to do. So, it was, as mentioned earlier, uh, a fully or almost fully automated attack. So, in the next phases of the attack, Claude identified and tested security vulnerabilities in the target systems by researching and writing its own exploit code. Having done so, the framework was able to use Claude to
harvest credentials, usernames and passwords that allowed it to further then extract a large amount of private data. So, it did really research, write the code to get into systems, uh, of other companies and then in those systems write more code to extract data, um, and- and- and compromise these systems and- and- and do bad stuff in there once it was in there. The highest privileged accounts were identified, backdoors all the stuff that happened after it was in the systems and data were exfil-
exfiltrated with minimal human supervision. In a final phase, the attackers had Claude produce the attack, creating helpful files with the stolen analyzed which would assist the framework in planning the next stage of the threats actor, eh, of the threat actor's cyber operations. Overall, the threat actor was able to use AI to perform 80 to 90% of campaign with human intervention required only sporadically, perhaps four to six critical decision points per hacking campaign. That is nothing. That is nothing. That is such
a scale at which you can run these attacks and again, especially in a future where you don't have to work against certain guardrails, where you can just focus on getting the job done and you have to spend energy on getting around guardrails. That is really a scary future. This degree of automation is really, really, uh, scary here. Claude didn't always work perfectly. It occasionally hallucinated credentials or claimed to have extracted secret information that was in fact publicly available. This remains an obstacle to fully, uh, autonomous cyber attacks and this is
not just an obstacle for cyber attacks, this is of course an obstacle for everybody, for us developers too because problem and will stay a problem because as I mentioned before, it's so easy to forget but these are token generation machines. Always have been, always will be. The large language models, I mean. They are generating tokens and they are generating the most likely token as the next token based on all the tokens that came before it, and that is something that can and
always will have the danger of hallucinating. So, that of course is a problem in general. Good to see that it can then also be helpful when it comes to defending against malicious tasks because those also are hurt by hallucination but of course that is a general problem, uh, we face, uh, and it will not be a significant, uh, defense mechanism unfortunately. Because in the end if everything's automated, it's just a question of scale and if some attacks fail because of
hallucination, well, does that really matter if you can run thousands of attacks in parallel? I'm not sure it does. Cybersecurity implications. The barriers to performing sophisticated cyber attacks have dropped substantially and we predict that they'll continue to do so. With the correct setup, threat actors can now use agentic AI extended periods to do the work of entire teams of experienced hackers. So kind of the same thing that applies to normal software also be more productive with AI, and I got a video coming up on
way, just that that is the case for malicious tasks, even worse because there you don't even have to care about things like code quality. Obviously you want to have a successful attack but in the end if everything's automated, you also can care a lot about just the scale. And if you can automate thousands of attacks to run in parallel, it doesn't really matter to you if you might have code quality problems or anything like that. So, uh,
already with the systems today where you could argue about potential problems they have when it comes to generating code, matter for attacks like this because you need a result that's just good enough. And again, chances are definitely high that results will also get better in the future and we'll be dealing with systems that don't even have guardrails. And as they say here, less experienced and resourced groups can now potentially perform large-scale, uh, attacks of this nature. This attack is an escalation even on the wipe hacking findings we
reported this summer. In those operations, humans were much, uh, still in the loop, uh, directing the operations. Here, human involvement was much less frequent. And although we have visibility into Claude usage, this case study probably reflects of behavior across frontier AI models and demonstrates how threat actors are adapting their operations. By the way, this is one case that was caught by Anthropic.... not sure if all the cases are being caught, also by Google, um, OpenAI and so on. This raises
an important question. If AI models can be misused for scale, why continue to develop and release them? The answer is that very, that very, a- abilities that allow Claude to be used in these attacks also make it crucial for cyber defense. Well, (smacks lips) uh, that's kind of a weak argument, I'll say, because if you have one thing that makes a problem much bigger, saying, "Yeah, but it can also help with the solution," kind of bad, right? So, uh, I- I'm not really sure about that. If we would not have these models...
And just to be clear, that is not (laughs) something that's going to But if we would not have them, it would probably be better than in the context of cyberattacks and defense because will always be one step, uh, behind. So, uh, I definitely see these tools more as an advantage for the attackers and a big disadvantage, uh, for the, uh, companies that have to defend against these attacks. So that's kind of a weak argument, my argument would be, it doesn't matter if Anthropic, OpenAI and so on
continue developing AI models, and obviously they will, just to be very clear. And there are way more arguments to be cybersecurity. This is just one very i- important and problematic field, but there are tons of discussions, including philosophical discussions we could have about AI and if it's good that it's there or not, but they all don't matter. It is there, it will stay there, these companies will continuing to develop these models. And even if they wouldn't, the technology is there. Bad actors will have access to their own models in the
future. It does not matter at all if companies like Anthropic or OpenAI continue. The technology is there and the problems with it are also there, therefore, and they will stay here. That would be my argument. This argument here doesn't make too much sense to me. And therefore, definitely scary. A scary world also from a cybersecurity perspective. As I mentioned, I only see that, uh, becoming worse in the future, and therefore, maybe a career in cybersecurity is worth a second look because yeah, that will
GPT 6 Astra was released. It's a good model, I have to say. Especially for 3D work it seems to be a real game changer. But due to all the hype, you may be disappointed...