Raju: Hello, listeners and viewers, and welcome to another episode of RRE POV. I’m Raju Rishi, and I’m joined by my partner Will Porteous. And today, he and I are going to explore a topic that is hitting headlines on an increasingly regular basis, and actually, we have bifurcating viewpoints among the brightest minds in the world. That topic is rogue AI. It’s going to be a fun one, Will.
Will: Indeed, yeah.
Raju: Okay. All right, so here’s what I’ve read in the last month that has happened. So, I’m just going to recite these and, kind of, get everybody up to speed. If you’ve been, kind of, living on a boat, you know, with no internet access, you don’t know about these, you might be underground someplace, or you know, Bali. Who knows? Anyway, here’s what happened last month. OpenAI cyber agents autonomously banded together during a security evaluation and they bypassed safety protocols and successfully hacked into third-party tech platform, Hugging Face. You remember that, Will?
Will: I do indeed [laugh].
Raju: So, that was interesting. And then, actually, based on that incident, Anthropic actually audited its own logs and they discovered its advanced models breached the internal systems of three undisclosed companies during Capture-the-Flag cybersecurity training exercises. The crazy thing is, they didn’t even know that these things had happened until they checked the logs. So, this was, kind of, just out there. And the last one is Meta, and they disclosed that one of its models had inadvertently been given access to the live internet during a cybersecurity game and hacked an external company, also named withheld.
Will: Yeah, and there’s more, too, even. In the international security world, the UK’s AI security institute’s agents actually began attacking real internet targets in July during a relatively permissive cyber evaluation exercise. So, there you’ve got stuff that’s deliberately coming out of a security and intelligence context in an AI cyber world that went rogue, and then, you know, you’ve also got the Cursor Claude incident around Pocket OS.
Raju: Oh man, it is everywhere. It is everywhere. Now, honestly, some of this is environmental misconfigurations, but some of it is autonomous goal pursuit. The most interesting thing I read is that the models exhibited collusion, hiding their tracks from developers, creating hidden communications channels, and organizing into multi-agent swarms. You know, bachelor parties can learn a lot from these agents.
Will: [laugh].
Raju: Honestly. Like, this is giving me ideas. Like, [laugh] all right, hide tracks, hidden communication channels, multi-agent swarms. I don’t know. I don’t know.
Will: And at root, there’s an identification of a resource that the AI agents either want to access that they’ve otherwise been restricted from getting to, or even worse, may want to eliminate as a potential threat.
Raju: Yeah. And by the way, bachelorette bodies can use this too. I’m not trying to be sexist here. Okay. So, I guess my question for you, my first off-the-bat question because we’ll explore a bunch of topics here is, are we moving too fast without setting guardrails?
Will: We’re discovering that we don’t have guardrails that are effective through these incidents, right? So notionally, these agents need a perimeter fence that is stronger than the one we’ve given those environments so far, and so I think the pressure point here is, are we moving too fast? We’re going to keep moving fast no matter what, but I think it asks us to make a consequent investment in guardrails and infrastructure to identify rogue behavior, to surface it, completely independent, objective outside agents. By the way, this is what regulatory bodies were created for in other categories, historically.
Raju: You know, what’s interesting thing is, the smartest minds in the world are bifurcated on this. They say different things, and this is, I think, causing some of the controversy. I don’t think there was one individual—well, that I remember. Obviously, I wasn’t around when nuclear weapons were born—but it was, kind of, an across-the-board thing: this is dangerous stuff, and we need to create regulatory and multi-level permissioning and human-in-the-loop models around nuclear. But let me give you a few of the viewpoints here, okay?
So, Bill Gates has publicly criticized the White House reliance on voluntary industry self-regulation, and he called AI the most dangerous tool ever invented. And he said that a few bad actors using advanced AI could easily engineer pathogens to unleash a pandemic. Okay, powerful stuff. Elon Musk, who, you know, arguably is one of the smartest guys on the planet, said Tesla and SpaceX both, you know—he manages them both—he has repeatedly stated that AI is potentially more dangerous than nuclear weapons and could lead to civilization-ending outcomes if it’s left unregulated. Okay, so those are two smart people. I think it’s hard to argue that they’re not smart.
Then we have Yann LeCun. He’s chief scientist at Meta. Obviously, you know, one of the foundation guys around imaging, and he has criticized extreme warnings about artificial intelligence, and he’s dismissed fears of human extinction or catastrophic risk as AI doomism. And he argues that machine intelligence will be modular, easily controlled, and he compares current fears to panic over new technology that has been around for years. Okay.
Andrew Ng, who’s the founder of Deep Learning AI and he is a machine learning pioneer, has stated that worrying about AI causing human extinction is like worrying about overpopulation on Mars.
Will: [laugh].
Raju: He said, “Extinction-risk scenarios are decades or centuries away and distract from practical near-term safety and ethical challenges.” Okay. Now, over a hundred tech companies, including OpenAI, Google, Microsoft, Anthropic, jointly signed an open letter warning that widespread, highly sophisticated, AI-enabled cyber attacks are only months away. Months away. I feel like I should just get some cash out of the ATM machine and, like, put it in, like, each of the pockets of each of my pants. You know, just in case. Like, you know—I don’t know.
Sometimes I think this painting behind me, the eyes are shifting a little, too. I’m getting a little paranoid here. Oh, my God. But, you know, like, this is bizarre, right? You got two, three people saying, you know, like, not two or three people—like, two or three prominent people just outright saying, “Man, we have to regulate this,” and then you’ve got a couple of real experts in the AI space, and they say it’s overblown. I mean, obviously, nuclear is a different scenario, but I guess we had the same thing during internet days. Didn’t we? I don’t know. I can’t recall it being that controversial, but I think that people were worried about data leakage, for sure.
Will: Yeah, but data leakage has always had relatively isolated consequences, right? I mean, the parallels to the nuclear age are significant because one individual’s decision to launch a nuclear weapon could precipitate nuclear fallout that could eliminate huge sections of the planet from being inhabitable. So, I think it’s the first time we’ve had a digital threat that could be existential to portions of the population.
And I think we have to give Yann LeCun a moment here and ask why be resistant to regulation in this instance. And I think the Trump administration’s position is understandable because the tools to define the guardrails or to create the right government regulator are not obvious or straightforward, [laugh] and so defaulting to industry regulation is a little bit of a way of saying we don’t have the people or the resources to do this right, and so we’d rather not be responsible for doing harm and be criticized after the fact.
Raju: Yeah. Yeah, that’s an interesting viewpoint, Will. By the way, I wore this shirt intentionally because I’m hard to pin down. I don’t think—you know, they can’t—AI can’t spot me perfectly. I’m just, like, a little… I’m shifty. I’m shifty today.
Will: Right. The barcodes on your shirt keep shifting [laugh].
Raju: Or I might be going clubbing tonight. Either one. One of those two is going to be true. Okay. Let me ask you a question about space because, like, look, you’ve been working in the space sector for decades. I’m going to say decades—
Will: Yep.
Raju: —and, you know, it has an interesting challenge to it that it has overcome to some degree. We all know about the old-school satellites being, you know, the size of a trailer park. You know, environment like one of those trailers, and now they are cubesats. But when you get launched into space, you can’t really modify the firmware that easily. You certainly can’t modify the hardware maybe at all, maybe just slightly, if you had, like, another tool out there that could swap out stuff.
But it doesn’t exist today, so you basically prepare something to spend its lifetime with what’s on board. Software can be moved in and out, but firmware and hardware, different story. How has the space industry, sort of, dealt with, like, quality control? Because I don’t mean to imply that cybersecurity and rogue attacks is the same thing as quality control, but there is an element of, like, getting it right and making sure that it works as intended before you launch the damn thing into space. How has the space industry dealt with that?
Will: That’s such a great question, and in many respects, the answer to this question is why we saw so little innovation in the space sector for such a long time. For decades, really beginning after the Gemini and Apollo programs, when we began to see first government communication satellites and military communication satellites and that sort of thing, starting in the ’70s and ’80s, as well as intelligence satellites in space, everything was designed with two primary considerations: absolute fault tolerance and longevity on orbit because it was so hard to get something in space, and space was such a hostile environment that if you were successful in getting the thing there, it had to work, and it had to work for a really long time. And the design parameters, particularly around fault tolerance, meant that the innovation cycle to move new capabilities in space got really, really slow. And this is part of what sustained the first generation of defense contractors in this country, this insistence on total fault tolerance.
And the reason our space infrastructure was really principally built by just a few companies: Boeing, Lockheed, Northrop, and Raytheon being the big four, is those companies were completely vertically integrated in everything. So, to your point, they selected and in many cases manufactured all the components that went on those spacecraft. The software was written by them. The firmware was written by them. The software that ran on top of it was written by them. The network protocol was a proprietary network protocol in many cases, and it communicated with ground station [crosstalk 00:12:32].
Raju: And that, to me, I understand the interoperability layer because obviously, if manufacturing is done by different vendors, you don’t know exactly how it was built and you can’t ensure that it worked properly, but the vetting, right? The vetting of it is something extraordinary. Like they—I believe, and I’m sure you know, since you’re still involved with these companies—it’s tested on ground to ad nauseum before you’re willing to launch it and say, like, it’s going to be up there for three to six years, you know, before something is replacing it. You know, so I feel like more testing is done, more unit testing, more system testing, more integration testing, all of that is part of the cycle here. And software hasn’t really had that, you know? I mean, not that rigor, at least.
Will: Our software systems that we’re talking about being at risk today have been built in the era of open architectures, open interfaces, an understood multi-layer network model in which an operating system, you know, sits at the bottom and supports a middleware layer and an application layer and all of those things that we’ve taken from the PC world and we’ve transferred into the cloud and into mobile, and now into our large-scale AI architectures. That’s all predicated on security being essentially a definable parameter within software. And if you come at this from the national security lens or from the lens of our critical space systems, it begins with complete network isolation [laugh]. Physical isolation.
Raju: Yeah, yeah, yeah. We’re going to get there because that is an important, important, important topic here in this AI world, too. Let me talk about a few things that are being discussed, and there’s, like, three topics there that I’ll talk about. It’s like, one is defense measures to prevent sandbox escapes and regulation and open-weight, and then we’ll talk about recommended strategies, potentially recommended strategies. But let me get into these three topics that are really on people’s discussion boards now. One is, like, defense measures to prevent sandbox escapes. Okay?
Will: Yeah.
Raju: The first is, like, behavior-based monitoring and not action-based. Which is kind of interesting, right? Rather than tracking isolated commands, the systems could scan for macro patterns of autonomous goal pursuit. So, an example of that is, the model queries multiple adjacent IP addresses after a test failure, and that triggers a defense AI to automatically cause a lockout. So, that’s again, I mean, behavior-based tracking, anybody who has children knows, like, you don’t look at the actions, you look at the behavior patterns.
Will: Right [laugh]. That’s a great observation.
Raju: You know, they’re looking to play video games. They’re just going to test the various weak points. It might be mom, it might be their brother or the sister, it might be dad, it might be downstairs in the basement, but there’s a behavior pattern, not you know, basically an action-based pattern. So, that’s one.
And then the second is true least-privilege enclaves. I know it sounds a little technical, but, like, it’s not. AI agents could restrict themselves, or they could be restricted to single-use localized microcontainers, and then network sockets could be hard blocked by firewalls, preventing models from exploiting, you know, software proxy loopholes. I don’t know if that’s possible long-term. Certainly during the testing phases, you could do it. I think what people are trying to do is, you know, like, worry about the sandbox environments, but at some point, these things have to be sort of out in the wild, and you’re not going to necessarily have these enclaves.
A third strategy is network access through mirrored synthetic web caches. Think akamai, right? You’re not going to actually have the website there, but you’re going to have a cache of it. So, virtual test environments blocking all outbound internet traffic by default, and if it requires external data, it’s channeled through a mirrored synthetic web cache rather than a live connection.
Will: So, you’d have eyes on through that mirrored synthetic web cache that you [activated there 00:16:51].
Raju: And you couldn’t damage anything because you’re just damaging the cache, you know what I mean? I guess—and then, you know, deterministic environment fingerprinting. So, security teams could deploy canary files. And one of the things that the AI does is it modifies the encryption certificates, and if the canary files detect missing or modified baseline security tokens, it assumes a breach and refuses execution.
I think these are kind of interesting facets of, like, how to prevent the sandbox escape because I do believe, as you have said, in the space industry and the defense industry, that sandbox environment is really important. You got to be able to test in a very isolated fabric before you let this thing out in the wild. And I don’t think we’re doing enough of that. I think we claim we’re doing it, but like, people are moving so fast, they either allow an adjacent tech stack that has access to the internet, you know, to be coupled with this sandbox environment, which is tantamount to just giving it free access because it’s going to figure out how to get out there.
Will: Yeah.
Raju: So, I think these are important to really—that sandbox testing phase really, really isolated. Let’s talk a little bit about regulation because it’s also being discussed. Mandatory pre-release government reviews. What do you think about that?
Will: I think it’s great in concept, but actually staffing that function to be efficient, and that function is inevitably going to need to define some standards of compliance, safeguards that are built in, and that sort of thing, I’m just not sure how you get the people into the right seats to do that. I think we need some version of that, but ultimately, I think financial liability is going to be a better stick than government enforcement.
Raju: I love that because we are going to talk about that one. That is an important, important tool in this.
Will: And it does seem like we’re right on the cusp of the first incident that precipitates some amount of financial liability.
Raju: Absolutely. Okay. Another topic that’s being discussed is National AI Security Institute audits. So, U.S. and U.K., you know, mandatory standardized stress testing protocols to gauge the model’s autonomous capabilities. That’s not really a pre-release review. This is just an audit of how this system is working. What do you think about that?
Will: I mean, I think it’s a good idea as part of a portfolio of approaches. Again, it requires consequences. Is that government watchdog going to be empowered to take a model down or force someone to take a model down? And this is where you get into problems of competing national jurisdiction [laugh].
Raju: Yeah, this is going to be a tricky one. This is going to be a tricky one, because I think you can regulate in… most of the free world is probably the best way to say it. I don’t know how to say it any other way, and then there’s going to be countries that just don’t have the same, you know, sort of mandate.
Will: That’s right.
Raju: Okay. Compulsory critical capability pauses. So, if a model demonstrates unprompted advanced cyber capabilities, like independently formulating zero-day exploits or organizing a multi-agent task delegation, the developers would be legally required to pause training.
Will: I like this [laugh].
Raju: I like it too. I like it too because I think you’re going to find that we’re going to slow things down pretty aggressively, though, because you’d be shocked at how much of this happens in frontier. You’re basically equipping—the Frontier models are equipped with incredible horsepower and incredible amounts of thinking, and they’re largely black box to the modern world, right? I mean, obviously the vendors know some things behind the cover, but we don’t. Which moves us to open-weight versus closed-source kind of models.
And let me tell you the closed-source risk and the closed-source advantage first. I mean, the risk is the models are bundled with maximum—it’s like Deadpool: maximum effort—maximum raw reasoning capability to perform complex tasks, and they possess long horizon logic required to, you know, find infrastructure flaws and escape contain, but they have the capability. And plus largely black box, right? We don’t know what’s going on underneath the covers. You know, obviously the vendors that own those models may have more insight, but maybe they don’t even have it.
The closed-source advantage is that security updates can be globally deployed instantly. So, you find something, the model, kind of, just, like, offline for a second or two or wherever, and then you kind of put it in back. Now, the open-weight risk is, if you have the intent of modifying the weights, you could do guardrail-stripping in an open-weight model.
Will: Wow. Wow.
Raju: Yeah, I mean honestly, the weights are yours. Whatever you want to weight it, you weight it. The advantage, you know, is that it can be isolated behind the firewall, you know, in a contained environment. So, I doubt, like, a corporation is going to intentionally modify the guardrails. They are going to put their own guardrails in place. Plus, they’re going to isolate behind the firewall and they can actually contain it, so I think it’s really valuable in the enterprise world.
I think where it’s dangerous is in the hands of a freewheeling individual who decides to put a model out there to hide. You know, any raw thoughts on closed-source versus open-source—open-weight—in the context of rogue AI? I know we had a whole topic on this in a previous podcast, and that was more for whether the adoption would happen. And I think our conclusion at that time, you know, and still true, is that enterprises are going to use this because, you know, they can control it behind their firewall, they have an opaque view, they can see into why decisions are being made, and more importantly, like, they can make rules where, like, you know, HIPAA and PII isn’t violated, or buy-side is not sharing sell-side data.
Will: In both contexts, but particularly the open-weight context, I love the idea of guardrail standards and essentially application registry that forces you to say, “Hey, I’m using an open-weight model, I’m registering it with a centralized authority that I’m using this open-weight model, and to comply with certain standards, here’s the guardrail framework that I’m running, and this is auditable and verifiable via centralized authority at any given time, and I will update to future published standards of those guardrails as they come out.” I think those are the kind of commitments we’re going to need. You need a license to do a lot of things in the physical world, and I think we need a licensing scheme for operating technology this powerful in the online world. That, to me, is a form of self-compliance with consequences for failure to comply.
Raju: I love the thought process. I’m just wondering, like, how do you do it ubiquitously, right? This is going to be the challenge with all of this, right? Like, it’s not one, you know, regulatory environment that controls the world here.
Will: No, it’s not, but that’s been the problem for international institutions for a long time. I mean, your parallels earlier to the nuclear age are significant, right? I mean, we create our global nuclear oversight framework and the institutions that support it in the wake of the Second World War, a period of important international institution building, and we exist in that world with rogue states that have their own nuclear weapons programs, have their own nuclear weapons, choose not to comply, and on a lot of risk of leakage. So, the parallels are there; we’re just—we’re not in a period of great international institution-building that it would take to create that kind of registry.
Raju: You know, I have a question for you. You probably know the answer to this because you’re kind of like a wizard, like, a little Harry Potter. You’re like an older Harry Potter. Maybe a Gandalf. Not that old, not that old. Gandalf’s way too old. Okay, okay. Gandalf the Gray or Gandalf the White? I’m not sure. Anyway, it doesn’t matter. The nautilus laws, right, like, what is open territory in the ocean versus like what is it, two miles? One mile? Off your coast, it’s, like, considered part of your own country? Obviously, the Strait of Hormuz, we’re all dealing with this now, and you know, notions of it. How did those standards—do you have any idea how those standards got, like, created?
Will: Sure. Yeah. I mean, you have precedents and centuries of naval activity and naval warfare, and you have, kind of, the understanding of territorial waters that you were referring to, and then you have the understanding of, you know, flags of convenience, I.e., this is a ship that I own and I’ve registered this ship in a certain jurisdiction, and that jurisdiction has certain laws that I have to comply with, as well as a generalized, kind of, law of the sea.
So, we have, you know, well-understood international agreements for how ships do everything, from how they pass each other, how they signal each other, how they can approach each other, and an understanding of private property on the ocean, both on the surface and if a cargo is lost. So, there’s a lot of precedent there that’s just been built up over time. Interesting to this discussion, though, is that there is no maritime law framework in space.
Raju: Yeah, yeah. I was going to ask you about that. There isn’t.
Will: We’re really in the same territory here as we ask the AI questions.
Raju: You know what’s going to be interesting? I think those are two topics that are inevitably going to be handled on a global basis. We are going to have to deal—actually three topics. I’m going to give you three topics—for different podcasts, maybe we don’t even talk about them ever because they’re so controversial, but like, I believe that international law will get created around space and how to handle conflicts, and you know, like ownership of devices that are exploring and zones of operation and all nuances around that. I think we’re going to deal with it from an AI perspective at some point, probably sooner than we think. I think every country in this world that doesn’t want to be, like, a terrorist state, effectively, is going to understand the need to, sort of, create some containment around it. And the third is biological—
Will: Yes.
Raju: —infrastructure. So, that’s for another time. But let me tell you some of the recommended strategies that are being discussed, and then we can talk about it. The last one I’m going to talk about is legal liability—which I think is actually an important topic—and economic guardrails. Okay, but the first one is AI control and air-gapped testing. We talked about this a little bit.
So, this is layered defenses, and so it treats advanced AI like a volatile biohazard, right? And so, systems must be kept restricted, isolated server environments with no direct internet access until they’re proven safe, right? That’s one part of air-gap testing and AI control. A second is anomalous behavior red-teaming. So, implement adversarially robust anomaly detection to flag signs of an AI system attempting to lie, cheat, hide code, or resist a manual shutdown, and then you basically red-team it.
Will: Yeah.
Raju: I think these are important. I think these should be voluntary constraints for some of the frontier models, frankly, and for all the models, frankly. It’s harder to see, you know, non-frontier models having the sophistication that, you know, more and more of the cutting edge, you know, sort of, frontier models have, but I think those are actually reasonably good structural constraints. What are your thoughts?
Will: Completely agree. I think that’s a great approach, if it can be championed. The interesting topic is, sort of, where does the leadership come from on regulation and compliance? I love the idea of the red-teaming function that you talked about. In the nuclear community there is a journal called the Bulletin of Atomic Scientists, and in the last 70 years it has been a mouthpiece for the actual researchers—and the analog here is to the most senior AI developers in the world—giving them a voice in terms of policy and the concerns that they see on the horizon. I kind of feel like the AI community needs an equivalent forum to speak and make recommendations. Because your references earlier to Bill Gates’s call-out and Elon Musk’s call-out and the others, super important, but we need many voices, too.
Raju: Yep, I agree. Okay, here’s a second, which is mandatory structural restrictions on dangerous capabilities. So, biosecurity stripping. So, remove biological synthesis and advanced chemical research capabilities from general public models. And then the second, you know, piece of that structural restriction is hard human-in-the-loop safeguards. So, ensure critical community infrastructure, lethal-weapon systems, utility grids, unbypassable human authorization for any high-consequence action.
Will: Boy, the last one is a great topic. I’m a big fan of it, except that, in a lot of contexts where we’re going to need AI to frankly be an autonomous decision-maker. Defense context, we may not have time to have a human in the loop. We need to improve our decision-making environments to keep humans in the loop as long as we can, but an enormous amount of why we’re moving AI data center capabilities into space is to support decision-making in future space-based conflicts, particularly those that will involve hypersonic weapons and autonomous weapon systems. And the human-in-the-loop theory is—the human-in-the-loop principle is an easy thing to adhere to until you’re faced with autonomous systems where an adversary has removed the human-in-the-loop, and your reaction time is cut to almost nothing.
Raju: You know, that’s why there’s a deterioration of potential jobs in this world.
Will: Yeah.
Raju: I actually think we need a little bit more human-in-the-loop in AI. I got to be honest with you. It’s not so much that every single transaction, every single decision needs a human in the loop, but there are certain ones that elevate to just a rationality perspective. You did SATs a long time ago, and I always error checked my—I did—I was pretty fast. I could answer all the questions very quickly, but then I error checked them by just reasoning to myself. Like, is this a logical answer to this question? Right? Is it in the range, right? I mean, does 2500 units of X seem like a plausible answer?
And I don’t think AI goes through those guardrails. Is that a plausible amount of compute that was required for this activity? Those kinds of things, I think, it’s just a judgment call. It’s a little bit of a safe lock, and—anyway, but that’s just an opinion.
Okay, so the next topic is compute gatekeeping and KYC protocols. So, examples: cloud infrastructure audits. The cloud providers must run a stringent screening process similar to anti-laundering laws to ensure malicious actors or rogue shell companies are not secretly renting massive compute power. Okay, that’s one. And the second is, hardware interlocks, which is integrating physical tracking and off switches directly into the chipsets to monitor or halt training runs that exceed dangerous computational thresholds. This is putting it at the lowest level.
Will: I love that.
Raju: I love it, too, because what does AI need? It needs data and data centers. If you can put some of the controls of the software into the hardware, it’s kind of a good thing. And maybe those oversight, kind of, sits there and says, “You know what? You’re asking a simple question, but you’re just monopolizing my GPU or my CPU or my process power, and does that make sense? Does that make sense?”
I think it’s an intriguing item, and I think we’re going to see some of it, sort of like, leaking in. Okay, and I’m waiting for that because I think that’s actually quite an interesting play. Okay. And this is the topic du jour, which is the legal liability and the economic guardrail. So, one is developer liability. So, pass federal laws holding AI labs legally and financially responsible for downstream harms or security failures caused by their models, and the second is requiring developers to purchase market-driven catastrophe bonds.
Will: Completely.
Raju: If companies’ AI displays unpredictable or rogue behavior, the bond yields shift, exponentially raising the capital costs for the developer and tying profitability directly to safety.
Will: I love it, man [laugh].
Raju: I love it, too. I love it, too. I love it, too. I think that we don’t have enough of that right now. I think we’re just basically saying sorry.
Will: Yeah, yeah.
Raju: I think we’re just saying sorry. And you know where we’re not saying sorry? Like, if you look at the analog to self-driving cars, the quotient for self-driving cars is the metric. How many miles should you have put it on a self-driving car so that it is acceptable to kill someone? And what I mean by that, it’s never acceptable to kill someone, but you make a decision, and you say, “Does a self-driving car have to have 10 times or 100 times the number of road miles that, you know, humans generally have, so it kills somebody 100 times less?”
We’re 100x safer with a self-driving car versus a human behind the wheel because it’s seen all these scenarios, but like, that is a metric in that world, right? Like, we’re not going to allow and enable self-driving cars until they’re proven to have the testing and the, you know, rigor. Except we’re not doing that so much with AI, you know? We’re not basically putting the same rules and saying you can put out that frontier model, but like, what happens if you wipe out a bank? Like, what is the consequence? Should, like, you know, this private company that’s soon to be public—just, like, what’s the ramification? So, I think these legal liability, I think it’s really interesting. I think it’s really interesting. Like, financially responsible is one thing, but that bond item is real—your credit rating is dependent upon it.
Will: Your credit rating and the individual developer liability are great tools, and tools that keep people up at night, the right people up at night—
Raju: Absolutely.
Will: —and enforce a conservative culture with regard to managing the models and managing the development process. I a hundred percent agree. And as you were describing this, I was thinking about the fact that, you know, your self-driving car analogy has great parallels in the rocket industry. No insurer will insure a payload on a new rocket. No one wants to fly on a new rocket. There’ll be a really important launch later this year of Eric Schmidt’s company, Relativity Space. There’s, like, nobody on the rocket [laugh] because the rocket’s never flown before.
Raju: Yeah, I get it. I get it. Listen, I don’t—I’m not the first person in a dance club either. If a new nightclub opens out, I’m not going. I’m not—
Will: [laugh].
Raju: —I don’t know, man. I—no, no, no. Like, I completely understand that. I completely understand that. It’s dangerous. I wouldn’t want to be the first one in a self-driving car that had new tech built into it, right? Like so, there’s nuances here—
Will: Yeah.
Raju: —but I think the financial ramifications. I think part of the problem is that, I don’t know about the venture community, but like, the whole ecosystem around AI, we’ve poured so much money into these companies, nobody wants to slow them down, right? There’s too much vested interest in their success and their exit patterns, and honestly, like, if these ramifications exist, your investment could go to zero. But I think we would be much, much more cautious about how they were handled, and I think these companies, you know, their, sort of like, virtual stock would be dependent upon their success patterns around this.
Will: That’s a great point. If you listen to Nvidia’s Q2 earnings call, Jensen Huang talked a lot about how fully integrated they are as a company in building their AI factories and how they’re working all the way upstream with everyone who’s, you know, developing the land and building the facilities and getting the permits and designing the cooling, and nowhere in that discussion was the promulgation of an AI safety framework to ensure that what happens with NVIDIA hardware and infrastructure does good.
Raju: There are two players that can play a massive role here: the bandwidth providers and the compute providers. We can—the bandwidth providers—see the problem with compute is, you know, you have distributed compute, so you could be basically begging and borrowing, but the bandwidth providers wouldn’t know. You’re bifurcating your compute requests across, like, a hundred different servers and a hundred different locations. Is this necessary? Is this odd?
And then the compute providers, if you’re channeling everything into one compute, you know, architecture, is this necessary? Just ask a question. Just like a bank asks a question: are you taking out too much money? You know, like, is this your regular pattern? This is not your regular pattern. So, anyway, really interesting topic. I’ve loved exploring it with you, Will. I’m going to let you wrap it up for our viewers and listeners, and off to the next one eventually.
Will: Thanks, Raju. RRE POV listeners and viewers, you can tell why it’s such a privilege to work with Raju Rishi every week here. Thank you, Raju, for a tour de force on a really critical topic. Rogue AI is posing questions for us legally, governmentally, and societally, and you did us a great service by surfacing it today. And this will be one that I think we come back to in the future for further discussion. Thank you to our audience, and we’ll be back to you again soon. Take care.