Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The Deepdive

The Deepdive artwork

When Google’s Cloud Fell: How a Power Dip Broke Banks, Cards, and Delivery

Oct 20, 202622 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Google Cloud outage, cloud resilience, data center power failure. Yet a flicker inside a London Google Cloud availability zone helps trigger a cascading cloud outage that leaves people staring at declined cards, frozen banking apps, dead ATMs, and stalled delivery networks. We start with the uncomfortable irony: the UK’s biggest banks and regulators run a sector-wide cloud failure simulation, then the real thing lands days later and daily life still buckles. We unpack what “the cloud” really is, not magic in the sky but physical data centers with power systems, cooling limits, and failure modes shaped by plain old physics. Along the way, we connect the human stories to the architecture, from delivery vans that rely on cloud-hosted APIs for routing to court hearings that halt when remote video platforms drop. Then we get technical in the most practical way: why the data plane can stay up while the control plane collapses, how “dirty power” confuses transfer systems, and why server protection mechanisms can unintentionally amplify an outage. Finally, we zoom out to resilience and regulation: high availability versus single-zone cost cutting, why “just migrate your workload” is often a cruel suggestion mid-incident, and how the UK’s critical third parties approach tries to treat cloud providers like the public utilities they’ve become. If one warehouse can freeze an economy, where does the real power sit today? Subscribe for more deep dives, share this with the most skeptical engineer you know, and leave a review with your answer: what would you change first to make the cloud worthy of its promises? Leave your thoughts in the comments and subscribe for more tech updates and reviews.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

AllanOkay, so picture this. It is October 8th, 2026. You're standing right inside the London Stock Exchange.

IdaOh wow, okay. Good visual.

AllanRight. And it's not just regular traders there. It's the UK's 38 largest banks, um, the top financial regulators, basically the absolute heavyweights of the entire economy.

Speaker 2All gathered in one room.

AllanExactly. For this big event called SimX26.

Speaker 2Which, I mean, that sounds like some shadowy corporate cabal, but it's actually just a massive sector-wide fire drill.

AllanYeah, precisely. They were rehearsing this coordinated rapid response to a catastrophic global cloud computing failure.

Speaker 2Aaron Powell A huge simulation.

AllanYep. And by all accounts, the drill just goes flawlessly. Like high fives all around. The Bank of England even puts out this incredibly confident statement afterwards.

Speaker 2Naturally.

AllanYeah, they specifically noted that the scenario was, quote, not informed by any specific threat information, just a completely routine exercise.

Speaker 2Right. So everyone leaves feeling very secure. They diagnosed this theoretical disease. They ran the simulation, and they felt ready.

AllanAnd then, well, fast forward exactly 11 days.

Speaker 2Oh man.

AllanOctober 20th, 2026. A tiny fraction of a second power blip. Hits a single data center in a London suburb.

Speaker 2Aaron Powell The uninvited fire actually arrives.

AllanExactly. And that elaborate fire drill they just ran, it does absolutely nothing to stop the UK banking system from completely collapsing.

Speaker 2Aaron Powell It is the exquisite, just almost unbelievable irony of modern technology. I mean, you can rehearse evacuation routes all day, but a fire drill doesn't stop the building from burning down when a spark actually catches.

AllanSo on October 20th, the UK essentially woke up to a digital blackout. And for you listening, today we are doing a deep dive into exactly how this happened.

Speaker 2Because it really is a fascinating mess.

AllanOh, totally. Because, you know, you probably think of the cloud as this infinite, weightless, indestructible thing floating above us.

Speaker 2Aaron Powell Right. Just magic up in the sky.

AllanAaron Powell But today we're going to unpack how your ability to buy lunch or receive a package or even hold a court trial relies on incredibly fragile physical warehouses.

Speaker 2Aaron Powell Warehouses that are highly susceptible to real-world physics. And that is what is so fascinating here. We really need to look at what it actually looks like when this invisible cloud falls out of the sky on a random Tuesday morning. Because the impacts aren't just, you know, abstract data points on a screen. They are immediately disruptively human.

AllanAaron Powell Oh, the ground level chaos was staggering. Let's just look at the scope of this outage when that Tuesday morning rolled around. Um Revolute cards are instantly declining everywhere. Trevor Burrus, Jr.

Speaker 2It's completely dead.

AllanTrevor Burrus, Yeah. ATMs go dead. The Lloyd's, Halifax, and HSBC banking apps are just throwing up infinite loading screens.

Speaker 2Aaron Powell And it wasn't just money either. Like think about logistics. Companies like Royal Mail and DPD ground to a complete halt.

AllanAaron Powell Which is wild to think about.

Speaker 2Aaron Powell It is, but it makes sense when you understand how modern delivery works. It's not just a driver with a paper map anymore. Those delivery vans rely on cloud-hosted APIs.

AllanAaron Powell Application programming interfaces.

Speaker 2Trevor Burrus Right. They are constantly pinging remote servers every few seconds for like dynamic routing based on traffic or package tracking updates, delivery confirmation.

SpeakerAaron Powell So when the cloud goes down.

Speaker 2The van driver literally doesn't know where to drive next. The API just stops returning coordinates.

SpeakerWait, it gets better.

AllanOr well, worse, depending on how you look at it. The HM Courts and Tribunal Service uses a system called the Cloud Video Platform or CVP for remote hearings.

Speaker 2Oh no. Don't tell me.

AllanYep, that system completely died. Nationwide trials were just halted. There's this crazy detail in the sources about journalists, actually.

Speaker 2Aaron Powell What happened with them?

AllanWell, usually they watch these trials remotely via the CVP. Suddenly the feed drops. And these reporters are physically sprinting across town to the actual courtrooms.

Speaker 2You're kidding.

AllanNo, seriously. Desperately squeezing into whatever extra seats they could find just to hear a complainant give evidence.

Speaker 2And if the room was at physical capacity.

AllanYou were out of luck.

Speaker 2Wow. So the UK justice system was suddenly bottlenecked by the physical square footage of a room, all because a server hundreds of miles away just stopped talking.

AllanThe justice system bottlenecked by square footage. I love that. And I mean the list of outages just keeps going. Right move, screw fix, BT, Cloudflare, all showing major issues.

Speaker 2It was everywhere.

AllanYeah. But my personal favorite, if you can have a favorite in a crisis, was the budgeting app Emma. They went down and put out this panicked message assuring users, quote, this is a technical outage, not a security issue.

Speaker 2Which is the exact right public relations move to prevent a bank run, even though it doesn't solve the user's immediate problem at all.

AllanRight.

Speaker 2It's kind of like saying, um, don't worry, the bank wasn't robbed. The vault doors just permanently welded themselves shut.

AllanYes, exactly. But this is where the absurdity really hits everyday people. There's a specific story in our sources about a woman and crew.

Speaker 2Oh, the petrol station story.

AllanYeah. She's at the petrol station, just going about her morning, she pumps 90 pounds of petrol into her car, goes to the till to pay, and her card is declined.

Speaker 2Awful.

AllanThen she frantically tries to log in and top up her daughter's school lunch cards so they can eat that day, also completely unfunded.

Speaker 2See, that right there is the human element that gets buried in these sterile technical postmortems.

AllanSo true.

Speaker 2The sheer visceral panic of standing at a till, holding up a queue of angry drivers, and being handed a literal no means to pay form by a cashier. And simply because a piece of silicon in London had a hiccup, that is the physical consequence of digital centralization.

AllanSo we know the chaos it caused. Let's start unpacking how things actually failed. Because it wasn't a clean break everywhere.

Speaker 2Right. It was patchy.

AllanYeah. Let's look at Hostinger. They are the world's second largest web host, five million customers, tens of millions of websites.

Speaker 2Massive footprint.

AllanHuge. So when the outage hit, all those actual websites they host, they stayed online. Completely fine.

Speaker 2Right. Because if you visited a blog or a store hosted on Hostaflare, it bloated normally. Because the data plane, the underlying servers actually serving the web pages to visitors, that survived.

AllanBut something else broke.

Speaker 2Exactly. Hostaflare's entire management layer evaporated. The dashboards, the checkout systems, the back end where a user logs in to update their site. That entire layer is what we call the control plane. And the control plane just vanished.

AllanOkay. Admit my confusion here for a second. Let me try an analogy to make sure I'm getting this data plane versus control plane concept.

Speaker 2Go for it.

AllanIs it like a neighborhood where all the houses are totally fine, the structures are standing, people are sleeping in their beds.

Speaker 2Okay.

AllanBut the estate agent's office down the street, the one that holds every single set of master keys and all the blueprints for those houses, that office just burned to the ground.

Speaker 2Yes.

AllanSo you have a house, but you can't unlock the front door, you can't change the locks, and you can't sell it.

Speaker 2Aaron Powell That is a perfect way to conceptualize it. The underlying assets, the houses, or in this case, the websites were intact, but the mechanism of management was centralized in the cloud. Got it. And when that centralized control plane dissipates, you are effectively locked out of your own infrastructure.

AllanAaron Powell Which forces us to ask the million-dollar question if the control plane vanished, what actually caused the break?

Speaker 2Right. What started the fire?

AllanYeah. Was this a massive Russian cyberattack? Did a rogue AI decide it was time to pull the plug?

Speaker 2I think people secretly hope for a cyber attack because it feels like a worthy adversary, you know?

AllanRight. We want a supervillain. But the truth here is delightfully just painfully mundane. It wasn't hackers, it wasn't malware.

Speaker 2Nope.

AllanOur sources point to a power event at a Google Cloud Data Center in London. Specifically an availability zone known as Europe-West II-B.

Speaker 2Okay, let's pause there because availability zone sounds like a vague geographic area on a weather map. We need to be clear about what that term actually means in cloud architecture. An availability zone, or AZ, is not a region. It is a specific, physical, concrete building, or sometimes a cluster of tightly connected buildings.

AllanAaron Powell Packed with servers and cooling systems?

Speaker 2Exactly. So Europe West 2-B is a literal warehouse somewhere in the London suburbs. When you select an availability zone for your app, you are renting space in that specific physical warehouse.

AllanOkay. So a power event hits this specific warehouse. Let me give you the exact metric from the incident report.

Speaker 2Hit me.

AllanIt was a three millisecond voltage dip. Three milliseconds.

Speaker 2Aaron Powell Which is an almost incomprehensible unit of time. I mean, three milliseconds is less time than it takes a standard computer hard drive to spin half a turn.

AllanThat's insane.

Speaker 2It is literally ten times faster than the blink of a human eye.

AllanAaron Powell Hold on though. I am going to push back on this entirely. I just don't buy that a three millisecond flicker takes down the UK. Aaron Powell Why not? Well, I have a $50 surge protector under my desk that handles power dips without my monitor turning off. Are you telling me a multi-billion dollar Google facility doesn't have backup batteries?

Speaker 2Aaron Powell They absolutely do.

AllanAaron Powell Don't they have massive diesel generators for exactly this reason?

Speaker 2Aaron Ross Powell You're totally right to question that. Okay. They have massive uninterruptible power supply systems, UPS systems that are essentially rooms full of industrial batteries.

SpeakerOkay.

Speaker 2So designed to instantly bridge the gap between grid power failing and the backup diesel generators spinning up.

AllanAaron Ross Powell So why didn't they work?

Speaker 2Because of the nature of the anomaly. A complete sudden blackout is actually easier for these systems to handle. Really? Yeah. The sensors detect zero voltage, the automatic transfer switches flip over to battery power instantly, and the servers never know the difference.

AllanAaron Powell Okay, but this wasn't a total blackout.

Speaker 2Aaron Powell Right. A voltage dip, a microfluctuation of three milliseconds, is what engineers call dirty power.

AllanSo the power didn't go out, it just sagged.

Speaker 2Exactly. The voltage sagged just enough to be abnormal, but too fast for the transfer switches to smoothly process. The mechanical switches essentially hesitated. But you know what didn't hesitate? The internal power supplies inside the actual server racks.

AllanOh, the servers themselves panicked.

Speaker 2Precisely. Server hardware is incredibly sensitive to voltage changes because bad power can corrupt data.

AllanMakes sense.

Speaker 2So when those servers detected the micro drop in voltage, their built-in safety protocols triggered. They essentially said, the power is unstable, I'm shutting down immediately to protect the data integrity on my hard drives.

SpeakerWow.

Speaker 2It wasn't a failure of power supply, it was an overly sensitive safety mechanism reacting to a shadow. The machines rebooted to protect themselves, and in doing so, took the UK banking system offline.

AllanAaron Powell That is simultaneously impressive engineering and completely ridiculous. Like a safety mechanism that works so perfectly it causes a national crisis. Right.

Speaker 2And this brings us to Google's official advice to their clients during the outage.

AllanOh, this is the best part.

Speaker 2The message they posted to their SADIS dashboard while half the country is failing to buy groceries.

AllanI have it right here. Okay. This is incredible. They said, quote, migrate your workload to another available location.

Speaker 2Aaron Powell I love that this exists, but also just why? That is incredibly dark humor.

AllanRight. Just pack up the database and move, guys. No big deal.

Speaker 2Imagine you're a panicked system administrator. Your non-HA single zone setup is actively burning down.

AllanAaron Powell Wait, define non-HA for the listener, really quick.

Speaker 2Oh, sure. HA stands for high availability. If you have a high availability setup, it means you pay extra to have your digital infrastructure constantly mirrored in real time across multiple availability zones.

AllanAaron Powell So like Europe dash West-2-B and Europe-West-2-A.

Speaker 2Exactly. If one warehouse loses power, the system automatically routes traffic to the other warehouse without skipping a beat.

AllanAaron Powell But a single zone setup means you put all your eggs in the Europe dash West-2-B basket to save money.

Speaker 2Right. So your single zone setup is offline. Your phone is ringing off the hook from your CEO, and Google's advice is basically have you considered simply picking up your entire digital infrastructure and moving it to a different warehouse mid-collapse?

SpeakerAaron Powell It's so absurd.

Speaker 2If you didn't build it to fail over automatically, you certainly can't migrate it manually while the control plane is down.

AllanIt really highlights something profound, I think. We are sold this marketing myth that the cloud is infinite, it's magical, it's weightless, it's everywhere and nowhere. But it's not.

Speaker 2No. It's just a very hot, incredibly loud metal shed full of someone else's computers. And it is completely bound by the laws of physics and local weather.

AllanAaron Powell Yes. Which reminds me of another brilliant detail from our sources. We have to go back a few years to July 19th, 2022.

Speaker 2Oh, the heat wave.

AllanYeah. The UK was hit by a brutal, historic heat wave. Temperatures spiked over 40 degrees Celsius outside, which meant they hit 44 degrees inside the data halls.

Speaker 2Ugh, the cooling failures.

AllanExactly. Let's unpack the physics here for a second. Data centers generate an immense amount of heat. They rely on massive industrial cooling units on the roof to pump chilled water through the facility.

Speaker 2Right.

AllanBut on that day, the outside air was so hot that the cooling units literally couldn't dump the heat from the servers into the atmosphere. The thermodynamics just stopped working.

Speaker 2The chillers essentially choked on the ambient air temperature.

AllanRight. And what happened? Both Google and Oracle had to literally pull the plug.

Speaker 2No way.

AllanYeah. They had to manually power down their London computers to prevent the machines from literally cooking themselves, melting their components and catching fire.

Speaker 2Aaron Powell This is exactly why we need to aggressively challenge that marketing myth of the cloud. We're sold this idea of endless redundancy, of fail safes built on top of fail safes.

AllanBut really.

Speaker 2But at the end of the day, a data center is a physical building in London that gets too hot in the summer and completely forgets how to work if the lights flicker for three thousandths of a second.

AllanAaron Powell Bound by Physics. And the wildest part of this whole story is that the UK government actually knew this was a massive vulnerability.

Speaker 2Oh, they definitely knew.

AllanThey weren't blind to it at all. Which leads us right into the bizarre regulatory backstory of this exact October outage.

Speaker 2Aaron Powell Because this wasn't a surprise to anyone who had been paying attention to the legacy of IT failures in the banking sector.

AllanLet's look at the timeline because the dates are just eerie. This Google Cloud outage happened on October 20th, 2026. Exactly one year prior to the very day, October 20th, 2025, Amazon Web Services, or AWS, had a massive outage of their own.

Speaker 2Such an ominous anniversary.

AllanSeriously. The 2025 AWS outage was caused by what they called a data storage glitch on some of their servers over in the U.S.

SpeakerOkay.

AllanAnd what happened in the UK? It took down Lloyd's, Halifax, and Bank of Scotland. It affected up to 25% of all UK consumers.

Speaker 2Think about the scale of that. A quarter of the country's consumers instantly cut off from their money by a glitch happening in a server rack across the Atlantic Ocean. It's wild. That is terrifying to a central bank.

AllanSo because of that 2025 AWS outage and a string of other legacy IT failures where banks couldn't keep their apps online, the UK government finally said enough is enough.

Speaker 2About time.

AllanIn July of 2026, just three months before the Google outage we've been discussing, the UK Treasury took unprecedented action. They officially designated the big four cloud providers, Google Cloud, AWS, Microsoft, and Oracle, as critical third parties or CTPs.

Speaker 2And we really have to emphasize what a huge shift in regulatory power that is. Under the Financial Services and Markets Act 2023, this CTP designation suddenly gave financial regulators, specifically the Bank of England, the Prudential Regulation Authority, and the Financial Conduct Authority, the legal authority to directly oversee these tech giants.

AllanThey essentially dragged Silicon Valley right into the regulatory perimeter of the British banking system.

Speaker 2Because the regulators finally realized something fundamental. When multiple competing banks rely on the exact same cloud provider, a single failure doesn't just inconvenience one bank's customers.

AllanNo, it reverberates across the entire financial system. Right.

Speaker 2If Monzo, Starling, and Lloyd's all use the same AWS availability zone, then AWS is no longer just a vendor. It is a systemic risk. So the regulators demanded the right to audit their physical servers, review their resilience plans, and run stress tests.

AllanWhich brings us full circle back to the sheer irony of our opening hook. The timeline is just, I mean, chef's kiss.

Speaker 2So true.

AllanSo July 2026, the tech giants are officially regulated as critical third parties. October 8th, 2026, the cross-market operational resilience group runs the SimX26 drill at the London Stock Exchange. 38 top banks rehearse a global cloud failure.

Speaker 2They put out their confident press release.

AllanYeah. And then 11 days later, Google Cloud actually goes down and takes the banks with it.

Speaker 2It perfectly captures the tragedy of modern regulation. They did everything right on paper.

AllanTotally.

Speaker 2They diagnosed the disease accurately. They wrote the cure into law. They even ran the fire drill to practice taking the medicine. And the three millisecond power flicker just, well, didn't care. Because regulation is a document. It's a policy framework. Regulation is not a backup generator.

AllanRegulation is not a backup generator. That is the quote of the day right there. And just to rub salt in the wound, our sources note that Google was already having a really rough month leading up to this London outage.

Speaker 2Oh, you're talking about the Iowa incident.

AllanYeah. This detail is wild. On October 8, 2026, literally the exact same day the banks were running their CyMex 26 drill in London, Google Cloud Storage suffered a massive outage in its US Central-1 availability zone in Iowa.

Speaker 2And we should clarify that one wasn't a power flicker or dirty power. That was a pure software issue.

AllanRight. The report says it was a traffic sorting optimization that went wrong and caused severe resource exhaustion on their storage nodes.

Speaker 2Let me decode resource exhaustion for the listener because it sounds like the servers just got tired.

AllanYeah, what does that actually mean?

Speaker 2What actually happens is a software bug creates an infinite loop or a sudden spike in traffic demands more memory or processing power than the physical server actually has available.

SpeakerOkay.

Speaker 2The software asks for 110% of the CPU. The server locks up, trying to compute it, and stops responding to any other requests. It literally exhausts its physical limits.

AllanAnd that software bug in Iowa took things down for over 12 hours. So whether it's a software optimization locking up CPUs in America or a three-millisecond power dip confusing the battery backups in London, the global cloud is visibly creaking under the weight of what we're asking it to do.

Speaker 2Aaron Powell If we connect all of this to the bigger picture, it reveals a really uncomfortable truth about the architecture of our modern world. We have built this deeply interconnected digital society. We've heavily regulated it, we constantly monitor it. But fundamentally, we have consolidated almost all of our societal operational risks into a handful of massive warehouses.

SpeakerAaron Powell The illusion of geodistribution.

Speaker 2Exactly. We are sold this idea that our data is safely scattered everywhere, redundant and invincible, but that illusion hides the dangerous reality that a single point of failure, one physical warehouse losing power for a fraction of a second, can still freeze an entire nation's economy.

AllanAaron Powell It all comes back to that woman in crew, doesn't it? The physical consequences. The cloud is an absolute miracle of modern technology, right up until the precise moment it isn't.

Speaker 2Yep.

AllanAnd when it fails, it doesn't just fail abstractly on some dashboard in a server room. It fails at the petrol pump when you're trying to get to work. It fails at the court docket when someone is waiting for justice. It fails at the grocery store checkout screen when you're trying to feed your kids.

Speaker 2We traded local control for global convenience, and we are only just now understanding the price of that trade.

AllanI want to leave you with a final thought to mull over. The UK government brought these tech giants into the regulatory perimeter because they finally woke up and realized these companies are now effectively public utilities.

Speaker 2Right, absolutely.

AllanWe rely on them as much as we rely on water flowing from the tap or electricity lighting our homes. But if a massive financial institution is deemed too big to fail, well, what exactly do we call a single data center availability zone that literally holds the keys to an entire country's daily life?

Speaker 2That raises a terrifying question about where the real power actually sits today. Is it in the banks or is it in the warehouse?

AllanSomething for you to ponder the next time you tap your card to buy lunch and you hold your breath for that little green check mark.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Frontier AI Turns PHP Memory Bugs Into A Push Button Exploit

    Oct 22, 202622 minAudio

    A decade-old flaw in PHP flips from “too hard to exploit” into a real-world remote code execution chain once frontier AI enters the picture. We break down how the memory bugs work, why the blast radius hits everything...

    Credit the people on this episode

  2. OpenAI Watermarks: The Signature Forgery Problem

    Oct 7, 202620 minAudio

    A viral cartoon hits your feed: Dolly Parton, Tim Curry, the Pearly Gates, and a New Yorker cartoonist’s signature tucked neatly in the corner. It’s funny, polished, and instantly shareable. There’s just one problem:...

    Credit the people on this episode

  3. Anthropic's Doomsday IPO: We Warned You. Now Buy the Stock

    Sep 30, 202622 minAudio

    A normal IPO filing is supposed to answer one question: why is this company a safe place to put serious money? So what do we do with an AI prospectus that spends pages warning about existential risk, models that might...

    Credit the people on this episode

  4. Anthropic's "Ethical" AI Just Got Caught Snooping

    Jul 9, 202623 minAudio

    A tiny punctuation mark shouldn’t be able to start a scandal, but that’s exactly what makes this story so unsettling. We dig into reports that Anthropic’s developer tool, Claude Code, quietly swapped a standard...

    Credit the people on this episode

  5. The Emperor's New Code: Inside the Boardroom Delusion of the AI Hype Cycle

    Jul 6, 202621 minAudio

    A half-billion-dollar AI bill from a missed checkbox sounds like a one-off punchline until you follow the thread. We start with that Axios-style nightmare scenario and pull it into something far more unsettling: when a...

    Credit the people on this episode