Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

DevOps Academy Podcast

DevOps Academy Podcast artwork

DevOps Academy Deep Dive S1E15: Who Owns Production Now? DevOps, SRE, Platform Teams—or Developers?

Sep 14, 202624 minAudioSeason 1, Episode 15
ShareXLinkedInFacebookWhatsAppEmail

As engineering organizations become more specialized, production ownership is getting harder to define. Should developers own what they build? Should SRE teams own reliability? Should platform teams own the runtime environment? Or should DevOps remain the connective layer across everything? In this episode of DevOps Academy Deep Dive, we explore how modern engineering teams divide responsibility for production - and where unclear ownership creates delays, bottlenecks, and incident confusion. The conversation looks at shared responsibility, service ownership, on-call, platform engineering, SRE, self-service infrastructure, Infrastructure as Code, incident response, and the growing role of AI in operational workflows. 🔑 Key Takeaways Why production ownership should be shared but never ambiguous What “you build it, you run it” really means Why central DevOps teams can become operational bottlenecks How platform engineering changes the ownership model Where SRE fits into production reliability Why application teams still need operational accountability How clear incident roles reduce confusion during outages How self-service platforms support distributed ownership How AI can automate operations without taking over responsibility Why autonomy must come with guardrails and clear ownership boundaries 🎧 Tune in to explore how modern teams can distribute production responsibility without creating uncertainty over who owns what. 💬 Who owns production in your organization today?

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

So it is 3 .0 AM. The absolute worst time. The worst time. And the primary checkout service for your e -commerce site is just completely down. Yeah. Millions of dollars in revenue are literally vaporizing by the minute. You jump onto the emergency Zoom bridge, and there are, I mean, there's like 30 people on this call spanning. three different engineering teams. Oh, I've been on that call. We've all been on that call. We all have. But instead of actively rolling back

a deployment or actually troubleshooting the database, they're just aggressively arguing over whose fault it is. Right. Like the app developers are blaming the infrastructure. The infrastructure team is blaming a misconfigured container. And the security team is, well, they're just asking why nobody checked the firewall rules in the first place. Yeah, welcome to the battleground of production ownership. Exactly. It's a total mess. And that is exactly our mission for this deep dive today. It's such a crucial topic. It really is. Whether you're an engineering leader staring down a massive organizational redesign

or a developer who is just completely exhausted by generic alerts waking you up for systems you don't even control. Or if you're just someone fascinated by how modern tech companies actually keep the digital world running. Yes. This one is absolutely for you. We are pulling some incredible insights from the DevOps Academy research today. Right. Because as engineering gets more specialized, we've got, you know, backend developers, DevOps engineers, site reliability engineers, platform teams. The list just keeps growing. It does.

And the boundaries of who actually holds the bag when things break have become incredibly blurred. They really have. And I think the foundational thesis we have to establish right out of the gate here is that the only sustainable answer is a model of shared responsibility. But, and this is a huge but, it has to be enforced by strictly mathematically defined boundaries. Right. Because the great failure of modern tech organizations right now is interpreting specialization as, well, basically permission to avoid accountability. Okay, let's unpack this. Because to really understand

why that 3 a .m. Zoom call is such a mess right now, we kind of have to look backward. We do. We have to look at how the industry's previous attempt to solve this exact problem somehow accidentally recreated the exact dysfunction it was trying to destroy. Right, yeah. To understand the current mess, you really have to understand the traditional IT model that preceded it. Historically, You know, companies operated with this massive impenetrable barrier between the people writing the software and the people running the software. The classic

throw it over the wall approach. Exactly. Developers would write the code on their local machines, package it up, and literally just hurl it over to a completely separate operations team. Right. It's like an architect designing this massive skyscraper, handing the blueprints to a construction crew, and then just immediately leaving the country. That is a perfect way to look at it. Right. Because if the building collapses three months later, the architect is already, you know, designing a shopping mall somewhere else. They don't care. Yeah, they're totally disconnected. The developers

had all the context about how the application logic actually worked, but zero responsibility for its uptime. And then the operations team had all the responsibility for the uptime, but zero context about the internal logic of the code they were running. It was just a recipe for catastrophic downtime. And massive organizational hostility. Absolutely. Tons of hostility. So the original DevOps movement was born to challenge that dynamic. The philosophy was really simple

and elegant. The people who build the software must be intrinsically involved in operating the software. Right. But. You know, philosophy rarely survives contact with the actual corporate org chart. It never does. Because DevOps didn't stay a culture or a shared mindset. Companies just looked at it and said, well, cool, let's just hire a DevOps engineer and make it a job title. And that right there is the DevOps job. It became a job title. And worse. the DevOps team became this centralized dumping ground for absolutely everything that developers didn't want to do.

And everything traditional IT didn't understand, right? Exactly. Suddenly, this one specialized team is responsible for... building CI -CD pipelines, provisioning cloud infrastructure, managing Kubernetes clusters, handling secret rotation, setting up monitoring, executing deployments, and somehow also managing database migrations. That is just wild. It honestly makes me think of a high -end restaurant kitchen where you have this team of specialist chefs who only like... chop vegetables

and sear meat, and then they hand all those prepped ingredients off to one incredibly stressed out sous chef who is suddenly forced to cook every single meal, plate the dishes, wash the pots, and magically fix the broken HVAC system in the dining room. Yeah, just because their name tag says operations. Exactly. So if developers are just submitting JIRA tickets to a DevOps team to get absolutely anything pushed into the real world, haven't we just created a much slower, much more expensive version of the exact same IT operations desk we had 10 years ago? We absolutely have, and it mechanically kills software velocity.

When infrastructure knowledge and deployment capability become entirely centralized in one specific team, that team mathematically becomes a bottleneck. I mean, they have to. Yeah. If every single change to production requires a human DevOps engineer to manually approve it or run a script or, you know, provision a server, The true spirit of DevOps is just dead. You haven't destroyed the silos. You've just given the silo a trendier name. OK, so we have this massive bottleneck. Centralizing operations creates this

massive traffic jam. But I have to push back here on behalf of the developers listening. Fair enough. Because you can't just swing the pendulum entirely in the other direction. You can't just walk up to a front -end developer who specializes in React and say, hey, congratulations. You now own the cloud architecture. Please configure the virtual private cloud routing by Tuesday. No, you can't. They would quit. They absolutely would. Nor should you ask them to do that. That developer's cognitive load is already totally maxed out just trying to keep up with front -end frameworks. And that tension is exactly why the

industry has shifted toward platform engineering. It is the structural answer to the DevOps bottleneck. OK, tell me more about that. So instead of a central team manually executing tasks via a ticketing system, a platform team builds an internal developer platform. basically an automated self -service product for the developers. Okay, I want to understand the mechanics of this. What does that actually look like for a developer trying to just do their job on a Tuesday afternoon? Mechanically, it

means a developer logs into an internal portal. They click a button to spin up a new microservice. Just a button. Just a button. And behind the scenes, the platform automatically provisions the cloud infrastructure, injects the necessary security credentials, sets up a standardized deployment pipeline, and configures all the default monitoring dashboards. Wow. OK. Yeah. The developer didn't have to write a single line of Terraform or Kubernetes configuration. The platform team builds and maintains that automated machinery. And the app team just uses it. Exactly. The boundary

is super clear. The platform team owns the platform. The application team owns the application. Which brings us to the famous industry catchphrase, you build it, you run it. But based on what you're saying about the platform doing all the heavy lifting, that phrase actually feels a little misleading. The developer isn't really running the infrastructure right. The platform is. This is really critical distinction. You build it, you run it out. It is not about forcing developers to execute operational tasks. It is fundamentally about psychological and organizational accountability.

Accountability. Yes. The people making the architectural and coding decisions must feel the immediate consequences of those decisions. If you build an application that is incredibly difficult to deploy, fundamentally unmonitorable, and crashes every time user traffic spikes. You can't just hand that mess to an operations team. Exactly. You are a cannibal. for fixing the architecture. But, you know, real life is incredibly messy. Let's look at a realistic failure scenario. Let's say that newly spun up checkout service has a

massive memory leak in the code. The app starts eating RAM. Eventually, it consumes so much memory that the underlying cloud server actually crashes. Now, from the application team's perspective, they just get an alert that the server died. So they're going to point at the platform team and say, hey, your infrastructure is unstable. Oh, they definitely were. Right. So how do you mechanically enforce a boundary when the symptoms cross over like that? Doesn't the platform team just end up getting paged for every vague production issue anyway? That is exactly the failure mode

that breaks organizations. And it is exactly why the observability tools built into the platform are so incredibly vital. OK. How so? Well, the platform team's responsibility is to provide the forensic evidence. If the platform routing is healthy, the Kubernetes control plane is stable, and the memory spike is perfectly isolated to a single container running the application team's code. Ah, I see. Right. The platform tooling should automatically route that alert to the application team, not the platform team. But

you can only enforce that boundary if the developer experience is flawless. Right, the capability has to match the accountability. You can't just point your finger at a developer, hand them a pager, and say, you own the reliability of this app if you give them terrible tools. Exactly. If they have to submit a ticket to get access to their own production logs, or they don't even have the permission to adjust their own auto -scaling rules to handle a traffic spike, I mean, they don't actually own production. No, they don't. You've just weaponized the concept of ownership to blame them. Precisely. Ownership

without capability is just blame. If you demand that a developer owns their production environment, you must provide what we call a paved road. The platform has to make doing the right thing, the secure thing, and the reliable thing, the absolute easiest path. OK, so let's talk about the sharp edge of this. We have the paved road. We have the boundaries. But it's all just theory until the system actually goes down and customers are actively impacted. Right, when the revenue stops. Exactly. In the crucible of a major incident,

how do specialized disciplines like site reliability, engineering, SRE, and security fit into this matrix? Because surely they have a massive stake. production. Oh, they absolutely do. But their functions are often deeply misunderstood by the wider organization. Let's take SRE. SRE teams are not just advanced sys admins who fix broken apps. Right. They are software engineers tasked with engineering reliability. And they do this mechanically through concepts like service level objectives or SLOs and error budgets. OK, wait,

error budgets. Let's dive into that mechanism for a second, because that sounds like financial accounting, not software engineering. It does. But it's really about mathematical enforcement of ownership. Let's say an application team agrees to a 99 .9 % uptime SLO for their service. OK. Mathematically, that means they're allowed exactly 43 minutes of downtime per month. That 43 minutes is their error budget. Right. If they push buggy code and the service goes down for 45 minutes, they have blown their budget. And the SRE mechanism dictates that the CICD pipeline physically locks. Wait, really? It locks? It locks. The application

team is mechanically blocked from pushing any new feature code until they dedicate their time to fixing the reliability issues and earning that budget back. Wow. OK. So it's not an SRE manager yelling at a dev manager. The system itself enforces the boundary. The math stops the deployment. That completely changes the psychological dynamic, doesn't it? It changes everything. Because the SRE isn't the bad guy. The error budget is. Exactly. It removes all the interpersonal friction. An SRE team cannot indefinitely compensate for

poor application design. If the core architecture of an app is fundamentally flawed, SREs cannot magically make it reliable. They provide the guardrails and the metrics, but the app team has to do the actual engineering. That is brilliant. And I imagine security functions similarly. You can't have a security engineer manually reading every line of code before it ships. You can't. You never ship anything. Security provides the automated policies, the vulnerability scanners built right into the pipeline, and the mandatory controls. But the developer is still the one writing the logic. If a developer introduces

a SQL injection vulnerability, The security team's automated tools should catch it in the pipeline, but the developer is the one who has to fix it. Okay, so when that 3 a .m. Checkout service failure happens, it really shouldn't be a disorganized Zoom call. No, absolutely not. A mature organization defines these incident roles long before the outage ever occurs. When an incident kicks off, It is a surgical procedure. How so? You have an incident commander coordinating communications. You have the application owner investigating

their specific application logs. You have the platform investigator verifying the shared infrastructure. There is no debate about ownership because the organizational boundaries were mapped to the technical boundaries in advance. But let me ask you about the human cost here. We say developers should be on call to feel that feedback loop, to learn from the consequences of their architectural decisions. Yeah. But if forcing developers into a rotation means they are getting paged six times a night for noisy alerts, aren't we just going to cause massive burnout? Like, if you punish

your best talent with a lack of sleep, they're just going to leave for a company that doesn't. That is a critical point. On -call must never be used as a punishment. It has to be sustainable. If a system is paging engineers six times a night, the solution is not to expand the rotation and throw more human bodies at the problem to spread the pain around. Which is what a lot of companies do. Right, and it's terrible. The actual solution is to stop feature development entirely and fix the broken system. So the pager is really just an information delivery system, not a cattle

prod. That's a great way to put it. The goal of developers being on call is to teach them realities that a pristine whiteboard architecture diagram never could. reality is messy. They learn how their application actually behaves under the stress of real -world network latency. They learn which alludes are actionable and which ones are just noise. And that visceral experience directly influences them to write more resilient code the very next morning. That makes total sense for a single application. But let's shift to the reality of modern architecture for a second.

If clarifying ownership for one monolithic application is hard, how on earth do organizations manage this when they adopt microservices? It gets complicated. I mean, when a single customer clicking by triggers a cascading chain of events across 50 specialized independent services managed by 30 different teams. Microservices are a fascinating architectural choice because they are essentially a double -edged sword for ownership. On one hand, they make explicit ownership incredibly easy. You can draw a very clean box and say, Team A owns

the authentication service. Team B owns the inventory service. Right. But the interconnectivity is where the nightmare begins. Yes. Because no microservice exists in a vacuum. Yeah. Let's look at the mechanics of a cascading failure. A customer clicks buy. Team A's authentication service experiences a slight database slowdown. Just a little blip. Right. But because it's slow, Team B's checkout service keeps retrying the connection. Those retries overwhelm the network, which causes Team C's cart service to drop connections, which eventually crashes the payment gateway. One team's minor

localized latency issue instantly spirals into everyone's massive site -wide incident. So you are sitting on that emergency bridge, the entire site is down, and every single dashboard is flashing red. How do you even know who to yell at? Like, how do you map the ownership of a failure that is moving that fast? This is where a service catalog becomes non -negotiable. And a service catalog is not just some static Excel spreadsheet or a corporate directory. Right. It can't just be a wiki page. Exactly. It is a real -time, dynamic map of your dependency graph. During

a crisis, the service catalog tells the incident commander instantly. Service C is failing. Service C depends on Service B. Service B is owned by this specific repository, maintained by this exact team. And here's a one -click link to their specific run book and their current on -call engineer. It traces the blast radius back to the owner. OK, since we are talking about tracing failures through massive dependency graphs at speeds humans struggle to comprehend, I have to bring up the elephant in the room. Artificial intelligence. I was wondering when we'd get to

that. With AI agents advancing this rapidly, could an AI eventually just take over production ownership? Like, if an AI can read a cascading microservice failure a thousand times faster than a human incident commander, shouldn't we just give the AI the pager? What's fascinating here, it is a very tempting thought, but we have to draw a very sharp, unyielding line between operational execution and accountability. AI is already demonstrating incredible capabilities in execution. For sure. Imagine an AI agent that

detects that cascading failure, reads the dependency graph in milliseconds, identifies that Team A deployed a bad configuration 12 minutes ago, and automatically rolls back that deployment while rerouting traffic. That sounds like an absolute dream for an exhausted engineer. It's a massive leap in operational velocity. But, and this is crucial, a machine cannot absorb responsibility. The AI is an incredibly fast tool. But the human teams? the engineering leaders, and the organization remain fully, entirely accountable

for the policies, the permissions, and the guardrails that allow the AI to take that action. Because things can go wrong. Exactly. If the AI hallucinates and accidentally shuts down the primary database instead of rolling back the app, you cannot stand in front of your customers or your board of directors and say, well, nobody owns this outage. The AI handled it. Yeah, it's not going to fly. Automation executes tasks. it does not and cannot replace human accountability. That is a profound distinction. You can delegate the execution, but you can never delegate the accountability. Never. So having

mapped out this ideal state of automated platforms, error budgets, and clear boundaries, let's bring it back to practical reality for the listener. What are the mechanical warning signs in a real -world organization that the ownership model is fundamentally broken? The red flags usually manifest as cultural friction before they show up as technical downtime. If your developers habitually say, Oh, I don't know why that failed. Ask DevOps. Or if centralized production access is guarded by two highly -stressed senior engineers

who are the only ones allowed to touch the database. The heroes. The heroes, yeah. Or if runbooks are either completely missing or haven't been updated in three years. Or, honestly, the most glaring symptom, if teams spend more time in post -mortem meetings arguing over whose responsibility it was to fix the issue, then they spend actually fixing it. The classic pointing fingers meme. So how do we fix this safely? Because you mentioned giving developers self -service capabilities, but if you just hand a junior developer the keys

to the kingdom, they might accidentally delete the production database on their second day. That's a valid fear. And this is where the concept of infrastructure as code, or IAC combined with policy as code, becomes the absolute game changer. Okay, break that down. Instead of clicking around at a cloud console, you define your infrastructure using code. That means every single change the developer wants to make is versioned in Git, it can be peer reviewed, and it is fully auditable. But the magic happens when the platform team embeds policy as code into the pipeline. How

does that actually shift the dynamic? It makes the compiler the bad guy, not the platform engineer. Let's say the platform team writes a policy that says, no database can be provisioned without encryption enabled. Okay. A developer tries to deploy an unencrypted database. The automated pipeline instantly rejects the code. The platform engineer didn't have to say no or argue in a Jira ticket. The automation mechanically enforced the boundary. You achieved developer self -service. but safely within a defined sandbox. But let me ask you this. Does company scale completely

change the math on this? Absolutely. Because if I'm listening to this and I work at a 12 -person startup, do I really need strict bureaucratic boundaries between a platform team, an SRE team, a security team, and the application team? I probably only have four engineers total. Yeah. Scale changes the math entirely. At a 12 -person startup, one engineer might literally wear all of those hats before their lunch break. You don't need distinct organizational boundaries because the communication overhead is zero. You're all sitting in the exact same Slack channel. Right,

you just turn around and talk to them. Exactly. But as a company hyperscales to hundreds or thousands of engineers, enterprise specialization becomes a strict requirement. But it's a dangerous requirement, right? Because every boundary introduces friction. Incredibly dangerous. because every time you specialize, you create a new organizational boundary. And human boundaries naturally require meetings, synchronization, ticket handoffs, and approvals. This brings us to Conway's law, which states that organizations design systems that mirror their own communication structures. If your organizational

structure requires six different teams to communicate just to ship one tiny feature, your software architecture will inevitably become a bloated, complex mess of interconnected bottlenecks. Conway's law is undefeated. Okay, so for the engineering leaders listening right now who are realizing they have a serious ownership problem, what is the actionable, immediate advice? Where do they start on Monday morning? Start with a ruthless inventory of your architecture. List out your critical services. For every single one, identify

the technical owner, the code repository, and the emergency run book. Just get it all on paper. Yes. Because you will inevitably discover gaps, critical microservices that the original creators abandoned and nobody actually owns. Eliminate those orphaned services by officially assigning ownership. And then what? then audit the friction. How many manual tickets, how many human approvals does a team have to wait for just to deploy a routine update to their own service? Find that human friction and engineer it out of existence through platform automation. So what does this

all mean? When we boil down all the models, the error budgets, the platforms, and the endless debates over who gets paged? It means the future of production belongs to organizations that can systematically distribute ownership without making it ambiguous. The absolute golden rule for engineering leadership is this. Never, ever confuse shared responsibility with shared uncertainty. Teams can only move fast and collaborate effectively when the boundaries are mathematically clear, when the capability matches the accountability,

and when everyone knows exactly what they own. Whether you are the developer writing the application logic, the platform engineer building the automated paved road, or the leader managing the teams trying to navigate it all, enforcing these boundaries is the master key. It's how you unlock faster, safer software delivery, and more importantly, it's how you finally put an end to those frantic, finger -pointing midnight Zoom calls. But before we let you go, I want to leave you with one final kind of mind -bending thought to chew on. We talked heavily today about how infrastructure

as code, policy as code, and AI driven automation are continuously perfecting our deployment pipelines. They are making environments reproducible and infinitely safer. Well, what happens in a few years when our testing environments become totally flawless, real -time digital twins of production? If the test environment is a mathematically perfect clone of reality, will the very concept of a high -stakes shipping to production moment disappear entirely? Will development and operations just blend into one continuous, completely risk -free state, forever changing who owns what? Until

next time, keep digging deeper.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. DevOps Academy S1E22: Platform Engineering 2.0 — Are Internal Developer Platforms Becoming AI Platforms?

    Nov 16, 202628 minAudio

    Internal Developer Platforms were built to simplify software development. But as engineering teams begin building AI-powered applications, are traditional developer platforms ready for what comes next? From integrating...

    Credit the people on this episode

  2. DevOps Academy S1E21: Shadow AI Is Becoming the New Shadow IT — Do You Know What Agents Are Running in Your Organization?

    Nov 9, 202623 minAudio

    Your engineering team may be using more AI tools than you realize. But do you know which ones have access to your code, credentials, and production systems? From AI coding assistants and locally running models to...

    Credit the people on this episode

  3. DevOps Academy S1E20: MCP in Production — Are AI Agents Creating a New Infrastructure Security Boundary?

    Nov 2, 202622 minAudio

    AI agents are moving beyond generating answers. They're connecting to real infrastructure, accessing internal systems, and interacting with production environments. But who controls what they're allowed to do? The Model...

    Credit the people on this episode

  4. DevOps Academy Deep Dive S1E19: AI Workloads Fail Differently — Is Traditional SRE Enough for Production AI?

    Oct 26, 202623 minAudio

    What happens when your AI application is running perfectly, but its answers are wrong? Traditional Site Reliability Engineering focuses on availability, latency, error rates, and infrastructure performance. But...

    Credit the people on this episode

  5. DevOps Academy Deep Dive S1E18: AI Is Making Developers Faster — Is Infrastructure Becoming the New Bottleneck?

    Oct 19, 202623 minAudio

    AI coding assistants are helping developers write code, generate tests, and build applications faster than ever. But is the rest of the software delivery process keeping up? In this episode of DevOps Academy Deep Dive,...

    Credit the people on this episode