What happens when your AI application is running perfectly, but its answers are wrong? Traditional Site Reliability Engineering focuses on availability, latency, error rates, and infrastructure performance. But production AI introduces a new challenge: a system can be technically healthy while delivering unreliable results. In this episode of DevOps Academy Deep Dive, we explore why AI workloads require a broader approach to reliability engineering. From hallucinations and model drift to inference latency, retrieval failures, and unpredictable outputs, we discuss the challenges of keeping AI-powered applications dependable in production. We also examine how SRE teams can adapt service-level objectives (SLOs), monitor AI output quality, manage model-provider failures, and design fallback strategies that protect the user experience. 🔑 Key Takeaways Why AI applications fail differently from traditional software The difference between infrastructure reliability and AI output quality Why successful API responses don't guarantee successful outcomes How to define meaningful SLOs for production AI systems Measuring model accuracy, task completion, and response quality Understanding model drift and silent quality regressions Why inference latency requires different performance metrics GPU capacity planning and AI workload scaling challenges Monitoring retrieval-augmented generation (RAG) applications How to safely deploy, evaluate, and roll back AI model updates Why continuous evaluation matters after deployment How SRE and machine-learning teams can collaborate on AI reliability 🎧 Tune in to discover why keeping AI systems available is only half the challenge, and what it takes to make them truly reliable. 💬 If your AI application returns 200 OK but gives the wrong answer, is it really working?
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Um, imagine for a second that you are, you know, in charge of a massive mission -critical application. Right. You're staring at your monitors, looking at your engineering dashboards, and everything is a beautiful comforting green. CPU utilization is perfectly normal. Memory usage is completely stable. Sounds like a good day so far. Exactly. Your latency is well within acceptable limits. You're hitting that golden 99 .9 % uptime metric. Technically... Mathematically, according to every
tool you have, everything is working perfectly. But. But, uh, out in the real world, the product is a complete disaster. Oh, yeah, the worst feeling. Seriously. Your AI customer support bot is confidently inventing return policies that just don't exist. Or your automated document processor is, like, quietly deleting key financial facts from executive summaries. You're basically staring at a green dashboard while the house burns down. It is a terrifying scenario for anyone who manages software.
And unfortunately, it's becoming incredibly common. Traditional site reliability engineering, or SRE, is still absolutely essential. But introducing AI into production systems has a whole new dimension of chaos. A lot of chaos. We're finding that the old rules of monitoring just don't catch these new types of failures. And that is exactly our mission for this deep dive. In today's stack of engineering white papers, SRE blogs, and recent incident postmortem reports, we are taking a hard look at why traditional reliability engineering
is no longer enough. We really have to rethink the playbook. We do. We need to explore how engineering teams have to fundamentally rethink what failure actually means when AI is involved. To really understand this shift, I mean, we have to look at the underlying mechanics of these systems. In conventional deterministic software, think of a standard payment processing API. There is a very clear, rigid contract. Right. It's predictable. Exactly. The system receives a request, it validates the data against a set of rules, it updates a
database, and it returns a defined result. If you send the exact same inputs, you're going to get the exact same outputs. It's essentially just rule following. It's binary. The code either executes the logic or it throws an error. Precisely. But AI outputs are probabilistic. They're driven by generative weights, not rigid logic paths. You can send the exact same input or a very similar prompt and get wildly different responses based on the temperature settings and the statistical
probability of the next word. So it's basically guessing. Right. So when your traditional infrastructure logs an HTTP 200 OK response, all that really means is the TCP handshake worked. an adjacent payload was returned. It just means the pipes are connected. Exactly. It tells you absolutely nothing about the semantic value of that payload. It doesn't tell you if the AI followed instructions, if it was accurate, or if the user actually achieved their goal. OK, let's unpack this. Because treating an AI like a traditional API is like judging
a restaurant purely by whether the kitchen didn't burn down, completely ignoring the fact that they served a picture of a pizza instead of an actual pizza. That is a perfect analogy. Does this mean our existing monitoring tools are functionally blind to actual user outcomes? That pizza analogy hits the nail on the head. Yes. In many ways, traditional tools are functionally blind to the outcome. We're now forced to separate two distinct concepts. Okay. First, there is infrastructure reliability. Is the system available? Is the
network connected? Is the compute running? Second, there is behavioral reliability. Is the system producing an acceptable, safe, and accurate outcome? Let's ground this a bit. There was a really good, specific example in the post -mortem reports about document summarization service. Oh, that's a perfect case study. So an enterprise uses an AI API to summarize dense financial documents. The API receives a 50 -page PDF, the model processes
the text, and it returns a neat three -paragraph summary. Sounds like a success. The infrastructure dashboard logs a complete success. Perfect latency, zero network drops. But in that generated text, the model hallucinated a decimal point and left out a critical quarterly revenue figure. Wow. From the infrastructure perspective, it was a triumph. From the user's perspective, it was a massive, potentially costly failure. If we can't rely on that basic infrastructure metric to tell us if the AI is actually working, how
do we measure behavioral reliability, especially when every single answer the AI gives uses slightly different wording? It's tough. In the past, we relied on service level objectives or SLOs. We could just say, we want a 99 .9 % successful API response rate. And our alerting was built around that. Well, we don't abandon those traditional SLOs, but we have to expand them significantly. We have to introduce new quality metrics that measure the properties of the response rather
than just the exact text. Like what kind of properties? We're talking about tracking dimensions like correctness, relevance, how well the answer is grounded in approved source material, and whether it adhered to a requested format, like outputting valid JSON. Okay, wait, I was reading through the sources and there's this mind -bending paradox for these new SLOs. Sometimes the AI completely refusing to answer a question actually counts as a successful outcome. Yes. So wait, the AI
throwing its hands up and saying I don't know is sometimes the gold standard for reliability? What's fascinating here is that an outright refusal is often much safer than an unwarranted guess. Right. Let's say you have an internal HR assistant. and an employee asks a highly specific question about a new tax policy that just isn't in the company handbook yet. Okay. If the documentation doesn't contain the answer, the most reliable behaviorally sound action is for the AI to admit it lacks the information. If it generates a plausible but completely unsupported guess, that's a severe
behavioral failure. Even if the grammar is perfect and the API was blazing fast. Exactly. But how do you... automate the checking of that. You can't just write a simple Python script to look for a specific keyword or an invoice number. Two completely different sentences can both be factually correct. Or, you know, one could be a dangerous hallucination masked by polite phrasing. You can't use exact text matching. For complex generative tasks, it completely fails. So engineering teams are increasingly relying on LLM as a judge frameworks. LLM as a judge? Yeah. They use smaller,
highly -tuned AI evaluators to review the outputs of the primary AI against representative datasets. You ask the evaluator AI, given the source document and given this response, is the response grounded entirely in the source text? Answer yes or no. Hold on. If we can't use exact text matching and we have to use AI to evaluate the AI, aren't we just creating a giant feedback loop of hallucinations? You're definitely not the first person to ask that. Like who watches the watchers? If your
automated quality monitoring is hallucinating, your entire reliability metric falls apart. You have just identified the single biggest risk in modern SRE for AI. You cannot blindly trust the evaluator models. Teams have to constantly test these evaluators against human baselines. How does that work? You create a golden dataset of queries and answers that human domain experts have manually created. You run your AI evaluator against that golden dataset. If the AI judge agrees with the human experts 95 % of the time,
you can trust it to monitor your production traffic. Wow, okay. But if it starts drifting, your monitoring is broken. It is a constant calibration effort. Which sounds exhausting. But the sources dig into something even crazier. We have to talk about why these failures happen in the first place. In traditional software, if something breaks in production nine times out of ten, an engineer deployed some bad code. You look at the git commit history, you find the bug, you roll back the deployment, and you go to sleep. It was simpler times. But in the AI landscape,
the system can completely fall apart while the application repository remains totally untouched. If the code hasn't changed, how do you even begin an incident response? Are engineers just chasing ghosts? They kind of are. They are dealing with a much wider, more volatile set of variables. We call these the silent breakers. AI behavior can degrade without a single line of application code changing. Give me an example. A common culprit is the external model provider. They might quietly update their default model weights to improve
safety, and suddenly your specific prompt template doesn't work anymore. Just silently breaks. Yep. Or user behavior shifts organically. They start asking different types of questions or submitting longer documents, pushing the model out of its comfort zone. And then there's RAG, which just multiplies the chaos. Oh, retrieval augmented generation. Yeah. This is where most enterprise AI failures actually happen. RAG is when an AI searches an external database, usually a vector database, for relevant information before it
generates its answer. Right. This introduces a massive chain of silent failure points. Let's walk through the mechanics of that. because the postmortems highlighted how easily this breaks. Let's go back to that employee policy example. An employee asked the internal AI assistant about the company's remote work policy. Now, the correct policy exists in SharePoint, but the retrieval system, the vector search part, fails to find it. Maybe the search index is stale because the sync job failed overnight. Or maybe the document
was chunked poorly, so the search algorithm didn't recognize its relevance. Or a permissions thing. Exactly. Perhaps there's a permissions error, and the search agent isn't authorized to read that specific folder. So the retrieval system comes up empty -handed, but it still sends a prompt to the main AI model. It basically says, answer the user's question about remote work based on this context, but the context box is totally empty. And this is where we have to understand the fundamental nature of LLMs. They are prediction engines designed to complete patterns. They abhor
a vacuum. They just want to talk. Right. If you give them an empty or irrelevant context window, they don't naturally stop and say error 404. They draw on their baseline training data, the entire internet, to confidently guess the most statistically likely next words. Oh no. So it invents a convincing generic remote work policy that has absolutely nothing to do with your company. The API succeeds, the latency is great, the user gets a polished paragraph, and it's totally wrong.
The model usually gets the blame for hallucinating, but the failure was actually in the retrieval system. The model did exactly what its mathematical weights told it to do with the bad hand it was dealt. Exactly. And this gets even more complex with multi -step AI agents. If an AI is making decisions, calling external tools, writing a draft, checking policies, you have to trace that entire path. It's a huge chain of dependencies. Right. If an agent successfully calls five different APIs but creates the wrong database record at
the very end because it lost the context from step one, the whole operation is a failure. If we pull back and look at the operational question for SREs, it is fundamentally changed. The question is no longer just, did the recent deployment break something? It's, is the AI ecosystem still meeting its expected quality level under current real -world data conditions? The quality of the answer literally shifts under our feet. But the sources also focus heavily on the physical reality of running these systems, the compute power,
the GPUs, and the latency. We're used to measuring response time as just one big number on a dashboard, total completion time. Right. Here's where it gets really interesting, because relying on average total response time for AI is like judging a book's pacing by only looking at its total page count. That is a great way to frame it. If one AI takes eight seconds to finish processing but starts streaming words to the user's screen almost instantly, the user feels engaged. But if another AI is totally frozen for seven seconds before
dumping the entire block of text on the screen all at once, the math on the SRE spreadsheet looks identical. Eight seconds total. But the user experiences night and day. Exactly. The second one feels completely broken. That behavioral difference is why SRE teams now have to split latency into distinct stages. The most critical metric for interactive applications is time to first token, or TTFT. Time to first token. Yeah. That's the physical time it takes for the model to process the prompt and spit out the very first piece of text. After that, you measure intertoken
latency, how fast the subsequent words appear, and finally, total completion time. And this all ties directly back to the physical constraints of GPU capacity, right, which operates very differently than traditional CPU servers? Completely differently. If you're listening to this and you've ever wondered why your company's internal AI chat bot sometimes takes 10 seconds to answer a simple question in the middle of the day, this is why. It's not a network lag. It's a traffic jam in the GPU memory. Yep. Managing GPUs isn't just about watching a CPU utilization dial. With GPUs, capacity is
constrained primarily by memory bandwidth, specifically something called the KV cache. What does that do? It stores the context of the conversation. When a user submits a massive 100 page document, the GPU has to hold all of that in its video RAM to calculate attention across all those words. Oh, wow. During that time, the GPU might not be doing heavy compute math. It's waiting on memory. So an SRE looking at a standard utilization dashboard might see low GPU load and think the
system is healthy. In reality, the system is memory starved, requests are queuing up, and it's on the verge of dropping requests entirely. And all of this specialized compute is incredibly expensive, which brings us to the economics of reliability and the dependency risk. Right, the cloud bills are massive. A lot of applications rely on external model providers like OpenAI or Anthropic. When those APIs go down, it's a massive outage. In traditional microservices, if a premium service fails, you might have a lightweight, cheaper backup service takeover.
Can SREs just flip a switch to route traffic to a small, cheap, open source model to save the day? If we connect this to the bigger picture, doing that without extensive preparation is a very dangerous trap. Why? Different models are not interchangeable commodities. They interpret prompt nuances differently, they have vastly different context window limits, and they enforce different safety guardrails. Okay, that makes sense. Let's say you have a complex reasoning task running on a massive, highly capable model. If that provider goes down, Blindly routing that
complex prompt to a smaller, cheaper backup model is a recipe for disaster. Because the smaller model just doesn't have the reasoning capacity to handle the complex instructions. Precisely. The backup model might technically accept the request and respond with a 200 OK, but if it completely misunderstands the task, ignores the safety guardrails, or truncates the document because its context window is too small, your attempt at cost optimization just became a severe behavioral reliability failure. Yikes. Engineering
teams have to know before an outage which specific tasks can safely degrade to a smaller model and which tasks simply cannot. So we have all these invisible, unpredictable, and highly expensive ways this technology can fail. How does an engineering team actually build a safety net to catch this? It takes a lot of restructuring. Let's talk about deployment. In traditional software, a canary release is just routing 5 % of traffic to a new version to see if it crashes or throws error codes. How do you canary an AI model if crashing isn't the problem? It requires redefining canary
testing entirely around quality, not just uptime. Imagine a provider releases a new model version that is 20 % faster and 30 % cheaper. Sounds amazing. It looks fantastic on an infrastructure chart. You route 5 % of your traffic to it. It doesn't crash. But your LLM as a judge evaluators flag that the new model is producing less accurate answers for a specific edge case, or it's suddenly formatting its JSON outputs inconsistently. Oh, I see. That is a product regression. It requires an immediate rollback regardless of how fast
or cheap it is. But rolling back an AI system isn't as simple as just reverting to yesterday's Docker container, is it? If you roll back the model, don't you have to roll back the entire data pipeline with it? You do. Because over the weeks you were preparing for the new model, your engineers likely tweaked the prompt templates to work better with its specific quirks. Right. You might have changed how you chunk documents in your vector database to match its context window. If you hit a quality issue and have to revert to the old model, you have to ensure that your props, your retrieval data formats, and
your tool configurations all roll back synchronously. The whole ecosystem is intertwined. Which is why the sources lay out a new comprehensive approach to dashboards. The days of looking at a single screen of CPU charts are completely over. Totally. The white paper suggests a five -layer approach for AI observability, and it makes a lot of sense when you see how complex this is. Layer one is infrastructure health, your compute capacity, memory bandwidth, and network. Layer two is model -serving performance, that time -to -first token and intertoken latency we talked about. Right.
Layer three is dependency health. Are the external API providers and the vector databases actually responding? Layer four is application quality. Is the AI actually grounded? Is it hallucinating? Is it passing the automated judge evaluations? And finally, layer five is user impact. Are users giving a thumbs down? Are they actually achieving what they set out to do? And tying all five of those layers together requires rethinking error budgets. OK, how so? Error budgets are a classic SRE concept where you allow a certain small amount
of acceptable failure, say a few minutes of downtime a month to maintain a fast pace of development. But with AI, you have to rigorously separate minor behavioral issues from severe catastrophic failures. Because an AI error isn't just a dropped connection anymore. An AI error could be a massive liability. Exactly. If an AI gives a slightly incomplete product recommendation on an e -commerce site, that is a minor quality issue. It eats a tiny bit of your error budget, but you can
keep deploying. Right. But if an AI generates dangerously incorrect advice on a high -stakes financial platform or, you know, hallucinates medical guidance, that is a catastrophic failure. They cannot carry the same operational weight. That makes a lot of sense. A single severe failure might require immediate intervention and a halt on all deployments regardless of what your overall uptime score says. This leads into the concept of graceful degradation. In traditional distributed systems, if a backend service fails, we try to
degrade gracefully. Like, if the recommendation engine on Netflix goes down, you don't break the whole app. You just show the user a generic list of top 10 shows. Right, the user barely notices. Exactly. But how do you gracefully degrade an AI? Do you just swap it for a dumber AI? Interestingly enough, in many high -stakes scenarios, graceful degradation for AI actually means removing the AI entirely from the user experience. Wait, really? You just pull the plug on the feature? Yes, because the goal is to protect the user outcome, not to artificially maintain the illusion of an AI
feature. If a critical use case like a legal contract analyzer detects that the model is degraded or hallucinating, generating a low -quality misleading answer is far worse than doing nothing at all. Oh, wow. Yeah, giving bad legal advice is definitely worse. Exactly. The safest fallback is to temporarily disable the chat interface, show the user the raw documents with a standard search bar, or just route the query to a human support queue. Protecting the user from bad information takes absolute priority over maintaining the appearance
of availability. So what does this all mean for the people actually doing the work? Listening to this, it's clear that the reliability contract is no longer something that infrastructure engineers can figure out in a silo. You can't just hand an SRE a pager and say, keep the AI up. That is the most significant paradigm shift we are seeing in the industry. SRE teams deeply understand infrastructure, latency, bottlenecks, and operational risk. But they should not be the ones inventing the definition of correctness. Right. That requires tight cross -functional collaboration. You need
machine learning engineers to interpret the model's probabilistic behavior and temperature settings. You need product teams to define what a successful user journey actually looks like. And the domain experts. Crucially, yes. You need domain experts, financial specialists, medical professionals, legal experts to look at the outputs and determine what an acceptable answer actually is. The SREs then take those human definitions and engineer the systems to measure them operationally. It is a massive team sport now. To wrap this all up, if you are listening to this and you manage
software, the next generation of reliability engineering requires answering two very distinct questions. Not just the old infrastructure question, is the system running? But the new, much more difficult behavioral question, is the system actually doing what it is supposed to do? If you only answer the first question, you are only measuring infrastructure. You are staring at a green dashboard while the product fails. If you can answer both, you are finally measuring real user -centric AI reliability. It changes the entire engineering culture. I want to leave
you with one final thought to mull over, building on this idea that you have to continuously evaluate these systems because they change based on silent updates and shifting user data. To big shift. It really is. If an AI system's behavior is constantly morphing without a single line of code being written, are we moving toward a future where software isn't really engineered? and deployed as a finished static product, are we entering an era where software has to be parented and managed over time almost like a living employee?
And if so, is your current engineering team actually prepared for that kind of management?
Internal Developer Platforms were built to simplify software development. But as engineering teams begin building AI-powered applications, are traditional developer platforms ready for what comes next? From integrating...
Your engineering team may be using more AI tools than you realize. But do you know which ones have access to your code, credentials, and production systems? From AI coding assistants and locally running models to...
AI agents are moving beyond generating answers. They're connecting to real infrastructure, accessing internal systems, and interacting with production environments. But who controls what they're allowed to do? The Model...
AI coding assistants are helping developers write code, generate tests, and build applications faster than ever. But is the rest of the software delivery process keeping up? In this episode of DevOps Academy Deep Dive,...
Modern engineering teams collect more telemetry than ever, yet getting meaningful visibility into production can still require extensive application instrumentation. In this episode of DevOps Academy Deep Dive, we...