Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

DevOps Academy Podcast

DevOps Academy Podcast artwork

DevOps Academy Deep Dive S1E16: When Incidents Last for Days — Are DevOps Teams Prepared for the Long Outage?

Sep 21, 202620 minAudioSeason 1, Episode 16
ShareXLinkedInFacebookWhatsAppEmail

Most incident response plans are built around the first few hours of an outage. But what happens when the incident continues into the next shift, the next time zone, or even the next day? In this episode of DevOps Academy Deep Dive, we explore the challenges of long-running production incidents and why they test much more than technical recovery. From responder fatigue and shift handoffs to decision logs, changing hypotheses, stakeholder communication, and organizational single points of failure, we look at what it takes to keep an incident response effective when the original team can no longer stay online. 🔑 Key Takeaways Why long-running incidents require a different response strategy How responder fatigue becomes an operational risk Why incident teams need structured shift handoffs What should be captured in a live incident document and decision log How to prevent new responders from repeating previous investigations Why incident command may need to rotate How to balance mitigation with root-cause investigation Why long outages expose knowledge and ownership gaps Where AI can help preserve incident context and prepare handoffs Why incident exercises should test people and processes, not just systems How to manage temporary emergency changes after recovery Why resilient incident response should never depend on individual heroics 🎧 Tune in to explore how DevOps and SRE teams can prepare for the incidents that don't end after the first few hours. 💬 Could your incident response process survive a complete shift change without losing critical context?

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

You know, usually when we think about a system failure, there's this expectation of a really clean, predictable rhythm to it. Oh, right, yeah, like a... Very specific sequence of events. Exactly. Like, imagine it. An alert fires at, I don't know, two in the morning. The on -call engineer gets jolted awake, grabs their laptop, and jumps into the incident channel. You find the jagged red line on the dashboard. You deploy the fix. Production recovers. And, well, everyone just breathes this massive sigh of relief. It's comforting, right? Because it feels so binary. You know,

the system was down, and now the system is up. The machine was broken, and we fixed the machine. Yeah, but then you look at what happens when the fix actually doesn't work, when that initial adrenaline rush wears off, and suddenly you're looking at a diagnostic landscape that's just entirely murky. Right, which is incredibly common, honestly. Yeah, and hour one turns into hour six, and then hour six bleeds into hour 12. The root cause is still a complete mystery. Your original engineers are just running on firms and the customer complaints are piling up. It is the absolute definition of an organizational

stress test. And it's exactly where standard operating procedures just completely fall apart. Because they're built for the short term. Exactly. Most teams are incredibly adept at the technical discovery phase. You know, they run drills on how to revert a bad code deployment or like restart a cluster. But they are completely unprepared for the human challenges that emerge in a really prolonged outage. So we're talking about like cognitive fatigue. Severe cognitive fatigue. The massive loss of context between shift changes.

and the intense leadership pressure that just compounds every single hour the system stays down. OK, let's unpack this, because time fundamentally changes the nature of the emergency, right? That's our mission for today. We are taking a deep dive into the source material to explore how a two -hour technical glitch tests your software, but a two -day outage actually tests your entire organization. That's a great way to put it. I was thinking about this like baseball. keeping your original engineering team on a multi -day incident just because they have the most context,

it feels exactly like leaving a starting pitcher in for 20 innings just because he quote unquote knows the batters. Yeah. Eventually his arm gives out and he just starts throwing wild pitches. That is honestly a perfect structural analogy because at the beginning of an incident you have peak energy. The strongest subject matter experts jump in, and the focus is absolute. Right, everyone's laser focused. But human biology simply does not allow us to operate at that level of intensity indefinitely. If we connect this to the bigger

picture of risk management, we normally think about operational risk in purely technical terms. Like servers crashing. Yeah, network failures, bad deployments, hardware degradation. But as time drags on, the human beings operating the system actually become part of the operational risk model alongside the machines. Wow. Wait, really? So the humans literally become the risk? Without question. A tired engineer is going to miss... subtle clues in a log file. They might accidentally execute a destructive command or

repeat an investigation someone else already did five hours ago. And this leads to a massive psychological trap. We call it the hero trap. Right. There is this very specific internal monologue the engineers go through when they're totally exhausted. They tell themselves, I have been investigating this complex web of services for eight hours. If I log off right now and go to sleep, the next person is going to need an hour just to understand everything I have in my head. And that logic sounds incredibly reasonable in the moment. So they stay for another two hours

and then another two, until eventually, The person with the most context about the incident is simultaneously the person with the absolute least energy to do anything about it. Which is the worst possible combination. Exactly. The solution to a long outage isn't physical endurance. It has to be a process where the knowledge actually survives the handoff. Which brings us to the ultimate antidote to all this fatigue -induced amnesia. Because tired humans inevitably forget things, the team has to build an external memory. Yes,

an external memory. But I do have to push back slightly on how teams usually do this, because my first thought is, wait, doesn't all of this just get recorded in the incident chat channel, like Slack or Teams or whatever they use? Why can't the next shift of engineers just scroll up and read what happened? Well, this raises a really important question about how we store operational knowledge. because there is a vital fundamental difference between a conversation and a memory. Think about a chat channel after 10 hours of a SEV -1 outage. And for anyone listening who might be unfamiliar, a SEV -1 is a severity

one crisis. It means a catastrophic drop everything system -wide failure. Right. Thank you. So imagine looking at that chat channel. You are staring at thousands of messages. Oh, it's a nightmare. I mean, as a listener, just imagine trying to catch up on a group text that isn't going off for 12 hours. But instead of weekend plans, It's just a wall of panic, code snippets, and automated server alerts. It's impossible to parse. Crucial decisions get completely buried between random screenshots, people asking clarifying questions,

chaotic side conversations. Chat is fantastic for real -time collaboration, but it is a terrible source of truth. Makes sense. After hours of troubleshooting, a structured decision log isn't just nice to have, it is an absolute requirement. And the structured log isn't like, novel. It's punchy. Like 14 .20, increased database capacity, no measurable improvement, or 15 .05, ruled out network saturation based on traffic data. It's just the facts. Notice the specific phrasing there. Ruled out. Documenting your ruled out

hypotheses is arguably more important than documenting what you are currently trying. I want to make sure I understand the mechanism here. Why is ruling things out so critical? Let's use the example of database connection pool exhaustion. Think of a database like a restaurant. And the connection pool is the number of tables available. If too many apps try to get data once all the tables are full and new customers just have to stand outside and wait, the whole system grinds to a halt. OK, yeah, that makes perfect sense. Now imagine the first shift of engineers spends two grueling hours pulling logs and analyzing

traffic. They definitively prove that the database connection pool is not the issue. There are plenty of tables available. Got it. So they move on to other theories. Six hours later, they're exhausted and they log off. A fresh engineer joins the call. looks at the overarching symptoms, which feature a sluggish system, and says, hey, this looks exactly like connection pool exhaustion. Ah. And if that two -hour investigation wasn't explicitly documented as a dead end in the external memory. The new team will waste another entire

hour pulling the exact same logs to re -prove something the previous team already disproved. Wow. The organization keeps forgetting its own investigation. A good handoff has to transfer the known facts, the ruled out theories, and fundamentally it has to transfer operational control. Everyone needs to know who is actually holding the steering wheel right now. Here's where it gets really interesting to me. Let's say you have a great decision log, the hours are still ticking by, frustration is mounting, customers are incredibly angry on social media. Oh yeah. If your primary revenue generating system

is down for a day, shouldn't every single person in the engineering department be laser focused on finding the root cause? Like, shouldn't it be all hands on deck? It is so counterintuitive, but absolutely not. And this is a trap that even very mature organizations fall into. Fix the impact and understand the failure are two completely different work streams. But wait, how so? Because you have to understand why it broke so you can fix it, right? Like, you can't fix what you don't understand. During an active customer impacting outage, your first objective is restoring acceptable

service, not achieving intellectual enlightenment. OK, I like that phrasing. You do not need intellectual certainty about a software bug to mitigate the bleeding. Let's say a massive new feature update is crashing the servers. You don't need to find the specific line of broken code. You just roll back the update or you route all the web traffic to a different geographic region that hasn't been updated yet. So you just bypass the problem entirely? Yes. You move customers onto a degraded but functional path. The root cause analysis

can happen next week. Do not pursue intellectual certainty while your customers are still suffering. That makes total sense. It's like putting a bucket under a leaky roof. It doesn't fix the hole in the shingles, but it stops the hardwood floor from rotting today. Precisely. But when teams get frustrated, they abandon that discipline, don't they? You hit hour 10, no one knows what's going on, and suddenly engineers start getting really aggressive. Oh, they do. One person wants to reboot the entire primary database. Another person is simultaneously changing firewall rules

just to see if it does anything. Which is exactly how you fall into the three problems trap. Right, the three problems trap. You start hour one by saying, we have one unknown problem. By hour 12, If you lack discipline, you are saying we now have three problems because of the chaotic things we changed while desperately trying to solve the first one. Yeah. That's wild. Changing a firewall rule blindly might suddenly cut off your own monitoring tools, so now you're blind and broken. So the only way to enforce that discipline is through a really strong incident commander,

someone calling the shots. Yes, through controlled experimentation, hypothesis, action, observation, decision. But, and this is crucial, the incident commander has to rotate too. Because they get tired just like everyone else. Exactly. Incident command is an incredibly demanding cognitive load. You're tracking multiple work streams, managing escalations, and constantly watching for those risky cowboy style changes. And if leadership becomes ambiguous because the original commander just quietly went to sleep. Then you have uncoordinated teams making independent changes,

which is why explicit handoffs of command are mandatory. And to your earlier point about all hands on deck, team sizes actually need to shrink as the incident drags on. not grow. Wait, shrink? I mean I'm gonna guess that throwing more bodies at the problem just creates too much noise? It creates noise, yes. But more importantly, it wastes a finite cognitive resource. Think of engineering capacity like a battery reserve. OK. If you burn 20 of your best engineers on a massive bridge, call for 12 hours while you're

literally just waiting for a cloud provider to fix a hardware issue on their end. Oh, right. Who's going to be awake to verify your system when the cloud provider finally finishes? You have to preserve fresh people. Keep the active response team incredibly small. It's the complete opposite of human instincts. You want to rush the burning building, but you actually need to hold people in reserve. It's really hard to do. But while the engineering team is trying to manage their own fatigue and shrink their footprint, there is this massive external storm brewing. The communication black hole. Executives, the

CEO, major clients, they are all breathing down your neck. Constantly. Now, if the CEO is demanding to know when the site will be back up, isn't the natural instinct to just give them an estimated time of recovery to calm them down? It's a very natural instinct that will completely destroy your credibility. This is what we call invented certainty. Invented certainty. If you do not know whether a complex, multi -system failure will take 30 minutes or six hours to fix, Do not manufacture an ETA just to satisfy a frantic executive. Because when you miss that invented

ETA, you look incompetent. You're basically writing a check your engineers can't cash. Exactly. You're trading five minutes of temporary relief for a massive credibility crisis later. Trust evaporates the moment that arbitrary deadline passes. So what is the alternative? Do you just stare at the CEO and say, I have no idea? No, no. You provide predictability about the communication, not the technology. You say, we do not have a reliable restoration estimate right now. The next update will be provided in exactly 30 minutes. Oh, that's smart. It creates a reliable rhythm

for stakeholders without pretending you can predict the unpredictable. Which means you absolutely need a dedicated communications role. You cannot have the engineer who is elbows deep in database logs stopping every 15 minutes to draft a corporate friendly update for the board of directors. The communication role acts as a shield. They translate the technical reality into useful stakeholder updates, keeping the executives informed so the executives don't interrupt the debuggers. But even if you have the perfect communication shield,

your local engineers still need to sleep. That brings up the logistical nightmare of passing this massive, messy context across time zones. Ah, the fall of the sun model. If you're a big enough organization, Europe hands the incident off to North America. North America hands off to Asia. But I've always wondered how that actually works mechanically. Because the team in Tokyo hasn't been feeling the 12 -hour panic. They're coming in fresh, but totally cold. There's a specific four -part framework for this handoff that I loved from the source text. You hand off...

What we know what we've tried what happens next and what we're afraid of what we're afraid of completely shifts the incoming team's tactical approach. How so? Give me an example of what that actually looks like in practice. Well, let's say the London team hands off to New York. If London just says, we put a patch on the memory leak. New York is just like, cool, good job. Right. New York might just sit back and monitor. But if London adds, we're afraid that if traffic spikes when the US East Coast wakes up, this patch will shatter and crash the whole cluster.

Well, New York changes their behavior. They get proactive. Exactly. They don't just wait. They preemptively spin up extra servers. They set up. specific early warning monitors. You're handing off the risk, not just the state of the system. Now, to manage all of this, the global handoffs, the shifting risks, the thousands of Loglines teams rely heavily on their tools. Which brings up the tech question everyone is probably screaming at their audio player right now. Oh, I know where this is going. With all this complex data piling up over 48 hours, couldn't an AI just take over?

Like, could an AI sit there, ingest all the telemetry, and autonomously fix the system while the humans go to sleep? We have to seriously temper that enthusiasm. What's fascinating here is the distinction between a known failure and an unknown failure. Automation and autonomous remediation are incredible for known patterns. If a specific server type crashes due to a known bug, and we have a predefined script to reboot it and scale up, Great, will the machine do it? But long incidents are long, precisely because the problem is unknown. Ah, of course. If the machine knew how to fix it,

it would have been a five -minute incident, not a five -day incident. Exactly. By hour 20, you were dealing with bizarre edge cases, cascading failures across third -party vendors or compounding errors that literally no one has ever seen before. That requires human judgment, intuition, and negotiation. Right. AI's true value in a multi -day outage isn't as a solver. It is as the ultimate scribe. The ultimate scribe. An AI can ingest those thousands of chat messages and maintain the context. You can ask it, summarize what changed

in the network configuration during the last four hours, or draft the handoff summary based on the decision log. It drastically reduces the cognitive overhead. But there's a catch, right? Oh, definitely. Accuracy is paramount. An AI cannot hallucinate a ruled -out hypothesis. If it incorrectly tells the incoming team that the database was checked when it actually wasn't, you are in serious trouble. And there is another tooling trap that jumped out at me in this deep dive, the observability risk. If your fancy monitoring dashboards and your incident chat tools run on

the exact same infrastructure that is currently failing, you lose your visibility at the exact moment you are blindfolded in the dark. Your tools need their own separate resilient strategy. It's a really painful lesson to learn. If your primary network goes down, and your alerting system relies on that same network to send emails. No one gets the email. Exactly. So if we can't rely on AI to magically fix unknown problems, and we know we can't rely on human endurance to just tough it out indefinitely, we have to

fundamentally change how we train for this. Absolutely. But how do you actually practice a multi -day outage? I mean, you can't exactly force your employees to stay awake for 20 straight hours just for a game day drill. HR might have a few things to say about that. No, you don't simulate the duration. You simulate the disruptions. In a standard disaster recovery exercise, you test the technology. Can we restore from a backup? In an organizational resilience exercise, you test the human process. So, you start a simulated

incident. Halfway through, you tap the incident commander on the shoulder and say, your internet just went out. Leave the room. Wait, if you kick the incident commander out of the room, aren't you just simulating pure chaos? Like, how does that actually test a process? Because it tests the external memory. If taking the expert out of the room causes the entire recovery effort to collapse, it means your decision logs and handoff notes weren't written well enough for the B team to execute. Oh, wow. It forces the team to rely on the documentation, not the individual.

If your entire recovery depends on one specific person being awake and available, you don't have an incident response plan. You have a hostage situation. That is such a crucial mindset shift, and it doesn't end when the drill or the real outage is over. I want to talk about the messy cleanup. Because we focus so much on getting that green light back on the dashboard, we completely forget that returning to normal is its own massive project. The return to normal phase is chronically ignored. During a panic, you make all sorts of emergency compromises. Let's say you have a firewall

rule that tightly restricts vendor access. During the outage, you temporarily disable that firewall rule just to force the system back online and restore service. Just putting the bucket under the leaky roof. Exactly. But if you don't aggressively track those temporary fixes, they slowly ossify into permanent architecture. Six months later, your company suffers a massive security breach. And it's because someone forgot to turn that emergency firewall rule back on the moment the incident was declared resolved. The bucket is

still sitting in the living room and someone finally tripped over it. That's exactly it. Which means the postmortem for a multi -day incident has to look way beyond just the technical root cause. If we connect this to the bigger picture, a multi -day incident is rarely just a technical failure. It is an organizational failure. The postmortem needs to ask, Where did fatigue actively degrade our decision making? Did we discover that critical knowledge was concentrated in just one person's head? Were our escalation paths clear? Or did we waste three hours trying to

wake up a vendor? So what does this all mean? When we really look at everything we've unpacked today, we spend vast, unbelievable amounts of time and money designing our software for failure. We have redundant databases, multiple availability zones, automatic failovers, but we completely forget to design our incident response processes for human limits. True resilience means your organizational process can survive the exact same way your servers do without relying on a single point of failure or, in this case, individual

heroics. You have to build a system that assumes the people who start fighting the fire will not be the people who finish it. And you know, even if you the listener aren't running global DevOps teams or managing cloud infrastructure, this applies so directly to how you manage any project in your world. It really does. I want to leave you with this thought to mull over. Think about your everyday non -emergency work culture right now. If a major project you are working on stalls or completely falls apart simply because one key person takes a two -week vacation, is your

standard operating procedure actually just a slow -motion fragile incident waiting to happen? Are you building hero dependencies into your daily life? It is a profound question every leader needs to ask themselves, regardless of their industry. It really is, because at the end of the day, when that alert fires and the jagged line on the graph won't go away and the clean binary expectation shatters into a murky, chaotic reality, endurance isn't going to save you. Only a resilient system will. Thank you for joining us on this deep dive into the source material. We will catch you next time.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. DevOps Academy S1E22: Platform Engineering 2.0 — Are Internal Developer Platforms Becoming AI Platforms?

    Nov 16, 202628 minAudio

    Internal Developer Platforms were built to simplify software development. But as engineering teams begin building AI-powered applications, are traditional developer platforms ready for what comes next? From integrating...

    Credit the people on this episode

  2. DevOps Academy S1E21: Shadow AI Is Becoming the New Shadow IT — Do You Know What Agents Are Running in Your Organization?

    Nov 9, 202623 minAudio

    Your engineering team may be using more AI tools than you realize. But do you know which ones have access to your code, credentials, and production systems? From AI coding assistants and locally running models to...

    Credit the people on this episode

  3. DevOps Academy S1E20: MCP in Production — Are AI Agents Creating a New Infrastructure Security Boundary?

    Nov 2, 202622 minAudio

    AI agents are moving beyond generating answers. They're connecting to real infrastructure, accessing internal systems, and interacting with production environments. But who controls what they're allowed to do? The Model...

    Credit the people on this episode

  4. DevOps Academy Deep Dive S1E19: AI Workloads Fail Differently — Is Traditional SRE Enough for Production AI?

    Oct 26, 202623 minAudio

    What happens when your AI application is running perfectly, but its answers are wrong? Traditional Site Reliability Engineering focuses on availability, latency, error rates, and infrastructure performance. But...

    Credit the people on this episode

  5. DevOps Academy Deep Dive S1E18: AI Is Making Developers Faster — Is Infrastructure Becoming the New Bottleneck?

    Oct 19, 202623 minAudio

    AI coding assistants are helping developers write code, generate tests, and build applications faster than ever. But is the rest of the software delivery process keeping up? In this episode of DevOps Academy Deep Dive,...

    Credit the people on this episode