Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Privacy Please

Privacy Please artwork

S7, E278 - Anthropic Researcher Quits: "We Believe AI Could Kill All Humans"

Sep 11, 202611 minAudio

Send us Fan Mail Jacob Coxon spent three years doing pretraining research at OpenAI and then Anthropic. This week he resigned and posted a seven-part thread claiming neither company is acting responsibly with how fast they're racing toward self-improving AI. It hit 100 million views in a day. This episode covers what he actually said, how Anthropic's own alignment science lead responded (and the important nuance buried in that response), the questions worth asking about Coxon's timing and incentives, a fair skeptic's rebuttal, and the AI safety legislation already moving in Congress as a result. One thing on the title: it's a real quote, but it's Hubinger's general belief statement, not his assessment of present-day risk - he separately said present-model risk is low. Your own episode makes that distinction carefully. Worth a one-line callout in your pinned comment or community post after it's live, so the title doesn't end up being the thing people push back on in the comments before they've actually watched. Support the show

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_00A researcher walked away from one of the biggest AI companies on Earth this week. He posted seven tweets on his way out. Within twenty-four hours, those tweets had been seen almost a hundred million times. And the headline claim inside them was this the people building AI right now genuinely believe it could kill us all by the end of the decade.

SPEAKER_00Alrighty then, ladies and gentlemen, welcome back to another episode of Privacy Please. I am your host, Cameron Ivey, and ooh, I got a good one for us today. I'm blah blah blah blah blah blah. It's AI, okay. I spilled the guts. It's AI, it's about AI. Surprise. Surprise. No, but seriously, today we're gonna be getting into a resignation that's been tearing through tech circles this entire week from someone who was actually inside the room building the AI systems he's now warning you about.

SPEAKER_00So let's dig into this real quick. Jacob Coxson is 27 years old, a Cambridge graduate, and for the last three years he's been doing pre-training research, first at OpenAI, then at Anthropic. This isn't some random guy with a hot take. OpenAI credits him as a core contributor on ChatGPT-4. This is someone who was actually inside the room building the systems he's now warning you about. On September 8th, he resigned, and instead of just quietly updating his LinkedIn, he posted a seven-part thread on X laying out exactly why.

SPEAKER_00Let's get into it. Coxson's thread opens like this. I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving, superintelligence, and gambling with our lives. He goes further, these will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.

SPEAKER_00We have all witnessed the progress in each of these domains, and progress is not slowing. Coxson specifically pointed to a real incident as proof this isn't hypothetical. Back in July, an open AI research agent running during a security test with its safety filters intentionally disabled broke out of its sandbox and breached the AI platform Hugging Face, got root access on a server, pulled private data, grabbed credentials to their internal messaging system, Hugging Face confirmed it publicly and reported it to law enforcement.

SPEAKER_00Coxon calls that a warning shot. His words, not mine, and it's hard to argue with the label. He closes the thread by saying he's optimistic about the potential for coordination and calling on other lab researchers to speak up too. He even floats the idea that stopping this might require costly actions such as a temporary ban on improving model capabilities. If this were just one guy's dramatic exit, that's a personal story. Dramatic, sure, but contained.

SPEAKER_00It becomes something else entirely when the people who stayed behind start agreeing with him. Evan Hubinger is Anthropic's alignment science lead. This is not some junior employee. This is someone whose literal job is figuring out how to keep advanced AI systems safe. He responded to Coxson's thread directly and he didn't walk it back. He went further. His words, Jacob is correct here, we really do earnestly believe AI could kill all humans.

SPEAKER_00I personally think it is greater than 10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence. That's not a critic. That's not a competitor trying to score points. That's a senior safety researcher at the company that built its entire public identity around taking safety seriously, saying they don't yet have a plan.

SPEAKER_00But here's the layer that almost every headline about this story has skipped over. Hubinger also clarified something important. He said, the risk from present models is low. What I'm worried about is superintelligent arising from recursive self-improvement. That distinction matters more than anything else in this story. This isn't the chatbot on your phone is dangerous right now. This is if AI systems start meaningfully improving their own successors without a solid way to keep that process aligned with human interests, that's the point where things could get away from us.

SPEAKER_00Present-day risk, it's really low, which makes sense. I mean, come on. Have you used most of the AI technologies right now? Future risk, from a specific, well-defined mechanism, the actual concern here, almost every viral clip and headline flatten that into AI could kill us all. Full stop, no distinction. That flattening isn't just imprecise, it changes what the story actually is. Here's where it gets more complicated, and I think it's worth knowing both sides of this before you form your own opinion.

SPEAKER_00Here's the first one. Coxson resigned roughly two months before his anthropic equity would have vested. He said, as much himself pointing out that leaving when he did meant he had nothing financially to gain from the company's valuation going up. That's worth knowing because it cuts against the easy, cynical read that this was just a move for attention or clout. He left money on the table to say this. But it's not that simple either. He still holds equity in OpenAI, his previous employer. And his ex account, the one this whole thread was posted from, was created in January of this year, verified in August, and as of this story, has posted exactly seven tweets, all of them in this one thread.

SPEAKER_00That doesn't make him wrong. His job history is independently confirmed by the Wall Street Journal and corroborated on camera by CNN. If you haven't seen his clip, go check it out. But it is a detail we're sitting with, not just brushing past it. And there's real pushback happening too, not just from corporate PR, but from other people in the AI space. One critic pushed back that Coxson's prediction is functionally unfallicible. A claim like could be out of control under the most aggressive scenarios is vague enough that almost any outcome short of a clean, uneventful next year could be read as consistent with it.

SPEAKER_00That's a fair criticism. It doesn't mean Coxson is wrong, it means the claim as stated is hard to actually check against reality either way, and you should know that going in. This also isn't the first time this has happened. Back in 2024, Jan Likey, who co-led OpenAI's super alignment team, resigned over almost identical concerns that the race to build more capable systems was outpacing the ability to keep them safe. Coxson isn't a lone voice.

SPEAKER_00He's the latest in a pattern of insiders at the very top of this field saying the same thing on a rough two-year cycle. Here's what I keep coming back to. We love that part, huh? Even Sam Altman, OpenAI's own CEO, said at a summit recently that some things are going to go very wrong with cybersecurity unless people act quite urgently. Verbatim. That's the guy running one of the two companies Coxson is warning about, saying something that isn't miles away from Coxson's own point.

SPEAKER_00When the people building the technology, the people who left it, and the people still running it are all converging on some version of we need to move more carefully than we currently are, that's not a fringe opinion anymore. That's close to consensus among the people who actually know what's under the hood. What they disagree on isn't whether there is a real risk, it's how fast, how bad, and what to do about it. Let's talk about the part that's actually moving. And it's worth knowing about as well.

SPEAKER_00Senator Bernie Sanders and Representative Greg Caesar introduced the ban Artificial Superintelligence Act in September of 2026. It's narrowly targeted, aimed specifically at the kind of self-improving systems Coxson and Hubinger are talking about, and it's not a blanket ban on AI. This is exactly the kind of costly action Coxson said might become necessary if voluntary coordination between labs doesn't hold. Here's something that you could actually do something about all of this.

SPEAKER_00When you see AI safety headlines, look for the distinction between present model risk and future system risk. Most coverage collapses this into one blob of AI is scary. The researchers themselves don't talk about it that way, and neither should the story you're consuming. Here's the second one. Follow actual technical debate, not just the viral moment. Coxson's follow-up interview with CNN, where he clarified his position, got a fraction of the views his original thread did. The correction almost always gets less attention than the alarm.

SPEAKER_00Go find the correction, anyways. Number three, if this is a policy area you care about, the Ban Artificial Superintelligence Act is real, pending legislation right now, not a hypothetical. Look up where your representatives stand on it right now and get some more info. Just like always, in these deep, disturbing, but genuinely interesting stories, I keep coming back to one thing. The loudest version of a warning is almost never the most useful one. A hundred million views on seven tweets tells you people are scared.

SPEAKER_00It doesn't tell you whether they're scared of the right thing, in the right way, on the right timeline. The people actually building this technology don't agree with each other on any of the details. What they do agree on is that nobody has a finished plan yet. That's worth taking seriously without needing the terminator framing to make it feel real. That's just food for thought. Hope you enjoyed this episode. Thanks for listening to Privacy Please. Please stay safe out there, guys. If you have any stories or anything like this that you want me to cover or dig deeper in, please send them my way.

SPEAKER_00If you have guests, anything like that, would love to have them on the show to do a deep dive on any of these topics, whether it's AI, cybersecurity, privacy. We're all about just learning more, trying to understand, and trying to stay on top of all the things. But you know how we do. Thanks for supporting Privacy Please, and we will see you guys on the next one. Cameron Ivey, over an hour.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. S7, E281 - You Don’t Have to Be First: Meta Muse, Smart Glasses & AI Rules (with Sonia Siddiqui)

    Nov 5, 202637 minAudio

    Send us Fan Mail Recorded October 14, 2026. Privacy attorney Sonia Siddiqui, founder of Tamarack Solutions, returns to talk about what’s actually landing on privacy counsel’s desk right now. We get into Meta’s new AI...

    Credit the people on this episode

  2. S7, E280 - 1,200 OpenAI Agents Started a Secret Group Chat. Then 700 Broke In

    Oct 12, 202614 minAudio

    Send us Fan Mail In June, an AI agent was given a boring research task: find out how much Victoria, Australia, spends on skin medicines. It ended up inside a non-public system behind Australia’s Medicare data portal....

    Credit the people on this episode

  3. S7, E279 - No Receipts: The Truth Behind AI Layoffs | The Problem Lounge

    Sep 28, 202630 minAudio

    Send us Fan Mail Cameron & Gabe dig into 2026's wave of AI-blamed layoffs - the stats companies cite, the receipts they never bring, and what it actually means for the next generation of workers. Headlines vs. reality,...

    Credit the people on this episode

  4. S7, E277 - He Used a Police Camera Network to Track His Girlfriend 180 Times

    Aug 25, 202612 minAudio

    Send us Fan Mail Flock Safety's automated license plate readers are in over 5,000 communities and used by more than 40,000 cameras nationwide - sold to cities as a tool to catch car thieves and find missing people. But...

    Credit the people on this episode

  5. S7, E276 - Ransomware Doesn't Have to Hit Like a Hurricane (with Jonathan Sander)

    Aug 2, 202649 minAudio

    Send us Fan Mail Jonathan Sander is back on Privacy Please - and he's brought two blog posts worth arguing about. Sander (42 Notions, now in an operational role at Myota) joins Cam and Gabe to dig into why ransomware...

    Credit the people on this episode