Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Pop Goes the Stack

Pop Goes the Stack artwork

Five nines of wrong: Detecting drift and errors in AI systems

Dec 9, 202522 minAudio

Uptime used to mean reliability. But in the LLM era, five nines just means your liar is always available. Real reliability now includes correctness and that means probing models in real time with prompts that have known answers. When those slip, your delivery fabric has to reroute before customers find out. In this episode F5's Lori MacVittie, Joel Moses, and returning guest Garland Moore dig into why availability isn’t enough anymore, and how research like “Get my drift? Catching LLM Task Drift with Activation Probes” shows where semantic health checks fit in the new definition of reliability. How do you keep AI outputs accurate even when external data sources introduce bias, errors, or malicious prompts? Listen now to find out. Read the paper, Get my drift? Catching LLM Task Drift with Activation Deltas: https://arxiv.org/abs/2406.00799

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show.

00:00:05:05 - 00:00:29:14 Lori MacVittie Welcome back to Pop Goes to Stack, the podcast where disruption is less of a buzzword and more of a ticket in your queue. I am Lori MacVittie and I am bracing right alongside you, really. Now I'm going to kick this off because our co-host Joel decided he didn't want to do it. I'm not sure why, maybe because he'd get roped into it all the time, but he really does have,

Joel Moses Okay.

00:00:29:16 - 00:00:32:27 Lori MacVittie he's got a voice. So you're you're stuck with me and

00:00:32:27 - 00:00:33:15 Joel Moses Well, thank you Lori.

00:00:33:15 - 00:01:03:25 Lori MacVittie Yeah, well. Anyway, we're here to talk about uptime today. So uptime, of course, is one of those measures in, you know, boring data centers that, you know, SREs and operations has to worry about because they're measured by it. And it used to mean reliability, right? Availability. Are you, you know, your five nines. Are you up? But in the LLM era five nines might just mean that your liar is always available right.

00:01:03:28 - 00:01:29:14 Lori MacVittie Wrong, five nines of wrong could be the new standard. So, we've seen changes in time for ops and how they're measured. We've gone from actually availability and uptime to things like, hey, let's measure how fast it takes to recover. MTTR is a big thing, especially among SRE and DevOps. But AI is going to change the game yet again.

00:01:29:14 - 00:01:55:13 Lori MacVittie And we think that that means it needs to include things like correctness. Right? Detect drift. Notice when it's lying and going off the rails. So because that's bad, right? I'm right, it's it's bad when they do those things. So you gotta reboot them, you know, start over. So, right, have you turned it off and back on?

00:01:55:15 - 00:02:16:26 Lori MacVittie I mean, it's it's cliche, but it works all the time. So this was kind of spurred by a paper again, titled Are You Still on track? Catching LLM Task Drift With Activation Probes, which Joel will break down for us because it sounds really lofty, but I think it's it's really just pretty

00:02:16:26 - 00:02:19:09 Lori MacVittie pretty fundamental.

Joel Moses It's pretty simple. It's fundamental.

00:02:19:11 - 00:02:31:09 Lori MacVittie Yeah. Semantic health checks. That's a word everyone should get familiar with, right? Semantic health checks. It's about correctness. So today we brought back Garland Moore. Welcome, Garland.

00:02:31:11 - 00:02:34:01 Garland Moore Thanks for having me. Glad to be back.

00:02:34:04 - 00:02:44:12 Lori MacVittie After the last episode he's like, mmm I don't know. We got him to come back anyway. So let's kick it off. Joel, you want to kind of explain what the paper's saying.

00:02:44:17 - 00:03:04:01 Joel Moses Yeah. Well, first off, let me let me start by saying this paper does have a few red lines for me. You know, any time a paper starts with a kind of a cutesy thing, you know, a cutesy title and also includes things like near perfect as a, as a result set that that that that causes my eyebrows to raise.

00:03:04:01 - 00:03:38:09 Joel Moses And I have substantial eyebrows, so there's a lot to raise. The, the paper though, if you read through it, it actually makes some pretty valid points. Let me describe, first of all, what what task drift is in the context of this paper, because I think it's important to, to kind of set the stage. Think of this like like you're driving down the road and you're listening to the directions coming from your GPS and you stop and you pick up a hitchhiker. And while you're driving the hitchhiker decides to enter some new secret routes into your GPS.

00:03:38:12 - 00:03:59:18 Joel Moses And suddenly your GPS is no longer taking you to your office; it's taking you to the ice cream shop. That that sort of thing is something that can happen with LLMs where something that is in the information that they are absorbing will change what their normal output mode would be. It'll change the answer in some cases quite dramatically and in ways that you may not expect.

00:03:59:18 - 00:04:25:01 Joel Moses And so, if you want to keep on the road and pointed towards work, the paper makes the case that what you should do is compare the expected outputs before you incorporate new external information with the information with the output after the information is incorporated, and see how semantically far apart they are. And it's, it proposes something called activation probing to do that.

00:04:25:04 - 00:04:48:18 Joel Moses It's an interesting approach. It's it's kind of similar in some respects to the judging, the model judging criteria that we talked about in previous podcasts. But, but it, it promises decent results. And what's really interesting about the approach is that it uses the simple linear mechanism, and it doesn't necessarily need another LLM to, to, to sample, sample the context.

00:04:48:18 - 00:05:01:21 Joel Moses It can tell you when you are way out of spec without necessarily spending a lot of energy to do so. So, so anyway, it's an interesting paper, an interesting approach. Garland, what did you, what did you think?

00:05:01:24 - 00:05:10:08 Garland Moore Yeah. Well, first of all, I don't know that I'd mind going to the ice cream shop. I like ice cream, so you know

Joel Moses Yeah, nothing nothing, says professional

00:05:10:08 - 00:05:14:14 Joel Moses then you know, I need a PowerPoint presentation and suddenly you're getting ice cream.

00:05:14:16 - 00:05:15:10 Garland Moore Yeah, yeah.

00:05:15:12 - 00:05:17:09 Joel Moses Perfectly professional, Right?

00:05:17:12 - 00:05:47:03 Garland Moore Exactly. But, but for the sake of our conversation, this is this is, this is huge, right? Because a lot of organizations or what have you that are implementing LLMs will probably exhaust RAG before they do any fine tuning, right. So if you're pulling information from external data sources, you need to know what's happening before responses get returned to end users.

00:05:47:03 - 00:05:51:05 Garland Moore So, this this could be this could be huge.

00:05:51:08 - 00:06:14:02 Lori MacVittie Yeah, that's, the whole idea of checking seems to be common. Like we know they lie, we know they hallucinate, we know there's a thing as context drift, and we hear it in different contexts. Wow, that's that's becoming a bad word now. Right, but just how much they carry, what they drop, what they fixate on, right, attention matters in more than one way.

00:06:14:02 - 00:06:38:15 Lori MacVittie So what they get from RAG, you're right. All of these things can combine to give you bad results. And somehow you need to be able to detect that because it's wrong, right? You do not, "Hey, we've been up 99% of the time," but it also answers wrong 98% of the time. You can't have that. And it's difficult when it's semantics.

00:06:38:18 - 00:06:41:02 Lori MacVittie Right? It's, cause it's words. New world.

Joel Moses Yeah.

00:06:41:04 - 00:06:42:29 Garland Moore Right. Yeah, yeah.

00:06:43:01 - 00:07:03:00 Joel Moses Now this is of course a, this is a of course a method to try to prevent what, what we've been talking about for a long time, which is prompt injection. I mean, that's, that's effectively what this is, this is simply prompt injection by ingesting something in inside the data that you're getting through RAG, that is a new instruction that may be embedded in the content that you're ingesting.

00:07:03:02 - 00:07:25:14 Joel Moses You know, sometimes this can this can occur maliciously, but sometimes it can occur just just because of the type of data that you're ingesting and how it's structured. Like maybe the content has something in it that says, you know, pay special attention to this particular special offer and, and it uses overblown language in order to focus your attention,

00:07:25:14 - 00:07:52:26 Joel Moses and remember everything is around attention, on a specific area of the content. And then the weight will shift to whatever is behind that, that prompting. Think about any of the, the, you know, the the polemics that you've read, think about any of the ads that you've read. And when these these this type of content is ingested, it can change the weights fairly dramatically and in ways that you you may not you may not notice until they happen to you.

00:07:52:28 - 00:08:21:09 Joel Moses So, so something something that can semantically check like this, and more importantly, activation probing works without doing effectively running an LLM to analyze the LLM afterwards. It's actually a simple linear mechanism. It offers the ability to, to do some interesting things. However, I would point out that in the paper, they're still not entirely certain how activation probing really accomplishes this.

00:08:21:11 - 00:08:38:01 Joel Moses You know, it's much, much like everything else in, in the world of AI, you know, we're on the we're on the bleeding edge of this type of technology. And so, you know, there's still more study to be done before this is a truly reachable bar for us.

00:08:38:03 - 00:08:51:24 Garland Moore Yeah. And Joel, you you bring up a good point. You mentioned attention, right? A lot of things are based on attention. And, Lori, I think you mentioned hallucination earlier.

Lori MacVittie What? Huh?

Garland Moore So.

00:08:51:27 - 00:08:53:15 Lori MacVittie Yeah, I want there.

00:08:53:17 - 00:09:20:27 Garland Moore So we have a compound problem here. AI models hallucinate and the people that are generating output from these models aren't checking the validity of this information. They put it out there and then what happens? Our LLMs are using RAG to pull in that information that's already inaccurate. So, this can help address that compound problem of we're ingesting information that's already wrong.

00:09:20:27 - 00:09:30:22 Garland Moore And then now we, the adversaries know that so they put malicious things in that information that the LLMs are pulling in to generate output. So,

Joel Moses Yeah.

00:09:30:25 - 00:09:48:23 Garland Moore it's a compound issue.

Lori MacVittie Yeah. Poisoning, poisoning the well has never been easier than

Garland Moore Oh yeah.

Lori MacVittie with AI, right? Like it's like, oh, I could just like set up entire websites that are just wrong and let it ingest it. No, go ahead, you can have it for free, really. Go ahead, take it, take it, right. That's

00:09:48:24 - 00:09:50:25 Garland Moore Yeah. Yeah, absolutely,

Lori MacVittie kinda scary.

Garland Moore absolutely.

00:09:50:25 - 00:10:11:11 Joel Moses We've said it in the past, garbage in garbage out still holds true with LLMs. And so if you feed them with bad and unsanitized data, you're going to get bad and unsanitized output. And that's, that holds true even in this particular case. I would point out that the authors of this study have released the toolkit; they call it Task Tracker,

00:10:11:13 - 00:10:29:18 Joel Moses which, which spans, you know, it includes a sample data set and testing instructions, so that you can play with this yourself. Now, I will be perfectly frank, I haven't had time to do that yet, but it's it's definitely on my list for a weekend activity, with with with a beer in one hand and a keyboard in the other.

00:10:29:25 - 00:11:05:01 Lori MacVittie That, that sounds scary. I, honestly, well maybe, maybe you need it for sanity though, testing these things. Because this and the the one use case is something malicious has been inserted or something wrong has been inserted. But the reality is that LLMs often hallucinate with perfectly consistent input. You can give it the same question five times and it might come back and tell you the glue on pizza is perfectly fine.

00:11:05:03 - 00:11:07:12 Joel Moses Right.

00:11:07:15 - 00:11:28:09 Lori MacVittie What? Right,

Garland Moore Yeah.

Lori MacVittie I mean, yeah, it can happen. So being able to check this, whether it's with this tool kit or some tool kit, right. Some method to be able to say, "has it has it lost the plot?" is going to be very important for for operations for businesses just for anyone using this.

00:11:28:12 - 00:12:00:03 Joel Moses Yeah. Now we've been talking about why it's, the mechanism that's been used to to to judge the accuracy of this output, but I, I think you make a really good point, Lori, at the very beginning, which is that the new standard for availability is not just being on, it's being accurate. And to some degree that is has always been true in our space, in the application delivery controller space, that the better health check is not to check to see whether the port is open, it's to check to see whether the output of your API is correct.

00:12:00:05 - 00:12:21:26 Joel Moses And, and so, so writing checks that that prove out accuracy is actually kind of the best practice for health checking for availability. And it's, and it's good that we just remember that for, for the days of AI that that accuracy and not just simple availability is actually the game that we're trying to play here.

00:12:21:28 - 00:12:23:17 Lori MacVittie Yeah, yeah. We have to.

00:12:23:17 - 00:12:30:24 Garland Moore Do you think they have something similar for humans? I need somebody to health check me when I'm hallucinating.

Joel Moses Oh, yeah.

00:12:30:27 - 00:12:35:26 Joel Moses The first time I read this paper was, like, task drift? I'm pretty familiar with that.

Garland Moore Yeah.

00:12:35:29 - 00:12:42:28 Lori MacVittie Wait, isn't that just distraction? Right. Ooh, squirrel.

00:12:43:00 - 00:13:18:04 Garland Moore You know, one of the one of the interesting things that I, I noticed when reading this paper is that you have to have access, right, to those activations and most foundational models, or probably all of them, don't allow that. So, as you know, AI becomes more critical and, you know, people want to check these activations, I think it's going to put us in a place to where we have to really consider, you know, deploying our own models, self-hosted so that we have access to do this type of semantic checking.

00:13:18:07 - 00:13:43:04 Joel Moses Yeah, that's that's actually a really important aspect. These checks require insight into the state of mind of the LLM. And by state of mind, I mean the metrics, the, the, the actual values of some of the some and temperatures of some of the, the, the settings within the LLM. And they compare the distance between the first activation and the second activation and subsequent activations,

00:13:43:06 - 00:13:59:20 Joel Moses but you're right, like for, for a lot of hosted LLMs, you don't get access to that information.

Garland Moore Yeah.

Joel Moses So so this is either going to be something that has to be built into the services that are offered on top of LLMs, or you're going to have to run your own in order to get this, this approach accomplished.

00:13:59:22 - 00:14:27:07 Lori MacVittie Yeah, but we see at least, right, the research and the data says that's happening. And especially as people move to to agentic AI and they start using more agents to actually automate tasks inside the business, those are necessarily going to be things that they deploy themselves, whether they put it in a cloud hosted environment that they control or on premises in an environment they control.

00:14:27:07 - 00:14:48:20 Lori MacVittie I think that control is the key word and always has been. And it's why we see hybrid multicloud is normal because they're going to have to do it where they can control it, where they can change, you know, the amount of information they're seeing, the telemetry they get, and so that they can see inside what's going on, because that will become critical.

00:14:48:22 - 00:15:03:18 Lori MacVittie It starts running important business tasks, you want to know why it's making decisions to buy 5000 Swingline staplers. I need to know why'd you do that? I mean, I like a stapler too. Good stapler, right?

00:15:03:21 - 00:15:08:05 Garland Moore That's a lot of paper.

Joel Moses Yeah.

Garland Moore Yeah, we got a lot of papers to staples.

Lori MacVittie That's right

00:15:08:05 - 00:15:11:24 Lori MacVittie Like maybe that's too many. I don't know.

00:15:11:26 - 00:15:13:00 Garland Moore Yeah.

00:15:13:02 - 00:15:33:27 Joel Moses Yeah. Now, one of the things that that I took away from this paper that I think is really exciting is a lot of these solutions that gauge accuracy of output actually have to sit in line and, and in fact, they have to process the output itself. This one doesn't necessarily need the output per se, it needs an insight into the state of mind. Which means this, the activation

00:15:33:27 - 00:15:57:25 Joel Moses probing can actually be done passively. Which is an important distinction to make from prior judge, judge based systems that we've seen. This this this can be, as long as you have access to the information about the, you know, to feed to the linear classifier, this can be done passively. So, again, a really interesting move

00:15:58:02 - 00:16:03:00 Joel Moses and, and, and, and some interesting technology that we'll be watching pretty closely.

00:16:03:02 - 00:16:33:07 Lori MacVittie Yeah and that's, that's important because we forget that a lot of the way that we monitor systems, whether they're apps, APIs or now inferencing is through cert-, you know, health probes, right. We ask, you know, give me this answer, give me this thing, let me check, Are you up? Are you correct? We do those kinds of things. But it's been pointed out to me many times that doing so, especially with AI, can actually impact the state of the LLM. By actually like asking it off to the side,

00:16:33:09 - 00:17:08:02 Lori MacVittie you're actually changing things more than you so than you would with a normal system, right? If I send a health check to a normal traditional app, I am opening a connection therefore I am changing the state of the overall system. But it's different than that's not going to make it go wrong. It's just, right? With an LLM I could send it off the rails by doing a health check. Which then, consequently, might become another security surface that we have to worry about.

00:17:08:02 - 00:17:18:23 Lori MacVittie Like is my health probe, you know, a vehicle for getting inside and changing things in an LLM. Well, well, I know. I I'm sorry, I went there, too.

00:17:18:24 - 00:17:26:08 Joel Moses I'm getting I'm getting uncomfortable memories of my quantum physics classes back in the day, and, yeah.

00:17:26:11 - 00:17:32:25 Lori MacVittie Just, yeah. Just saying

Garland Moore Yeah.

Lori MacVittie everything is a surface area with AI. It just, everything.

00:17:32:28 - 00:17:53:05 Garland Moore Yeah, Lori, and to that point when you talk about security, you know, one thing that I thought was interesting with this approach is, you know, people are or should be really good at blocking things that they know are bad. Right? But what about things that we don't know are bad, right? Can this help us with quote unquote zero day?

00:17:53:12 - 00:18:08:13 Garland Moore Right? We didn't know this particular attack was bad for the other LLM, but we saw the drift. So we either alerted or blocked blocked the output. Right. So I think that's an interesting use case that we have here with this type of approach.

00:18:08:15 - 00:18:12:14 Lori MacVittie Yeah. Wait, wait. So it could be, it could be good.

00:18:12:17 - 00:18:13:02 Joel Moses Yeah,

00:18:13:05 - 00:18:14:02 Lori MacVittie Ohhh.

00:18:14:08 - 00:18:17:20 Joel Moses positive security if you can get to it is always great.

00:18:17:22 - 00:18:25:12 Lori MacVittie I just, I like it when we're not always like doom and gloom. Like, oh this is just terrible and you're like, no, this could actually be good.

Joel Moses Definitely.

00:18:25:14 - 00:18:33:21 Garland Moore Now to to Joel's earlier point though, you know, they they use the term near perfect. Only time will tell how close to perfect it is, so.

00:18:33:27 - 00:18:37:00 Lori MacVittie Time and a lot of metrics.

00:18:37:02 - 00:18:49:08 Joel Moses All that means is that these researchers have not, have not learned the lessons of of Chief Engineer Montgomery Scott. You always, you always under-promise and overdeliver.

00:18:49:10 - 00:19:00:28 Lori MacVittie Yeah. Yeah. Oh yeah. Oh, absolutely. Well, Joel, you always have great summaries and takeaways. What should we take away from this episode.

00:19:01:00 - 00:19:22:06 Joel Moses So I think one of the things that I've learned from this paper is that, you know, like you said, Lori, the definition of available, has changed. Availability has changed. It now includes not just is it on, but is it accurate. And I think that that's important. It's an important change to realize and something that you need to design into your AI use.

00:19:22:11 - 00:19:54:21 Joel Moses Another thing that I learned is that this solution actually has real promise in terms of detecting task drift in a passive manner, that other systems, require a lot of heavy compute in order to do. Instead of sampling things semantically, you're actually using a linear classifier and peeking into the state of mind of the LLM, and that that has promise to, to be fairly accurate, while at the same time not taking a lot of time out of the, away from the AI system.

00:19:54:24 - 00:19:55:22 Lori MacVittie Garland?

00:19:55:24 - 00:20:27:20 Garland Moore Yeah. So a lot of lessons here. I think one thing that really caught my attention was why this is more important than output based detection, right. So us humans, right, we think before we speak, or at least we should think some of us don't think before we speak, but most of us do, right? But if we can catch drift during that thinking process, right, we catch the the malicious or adversarial behavior before the output actually gets to back to the end user.

00:20:27:20 - 00:20:36:03 Garland Moore So,

Joel Moses Yeah.

Garland Moore that was one thing that really, really caught my attention. And that's why it's more important than output based detection.

00:20:36:05 - 00:21:00:11 Lori MacVittie Awesome. Yeah, and I took away this, it has security angles. A lot of this is intertwined. I think AI more than anything else, is showing how security is is really part of like the entire thing. That malicious content could be actually malicious, someone trying to exploit the system, but it could also just be a bad query or a mistake.

00:21:00:11 - 00:21:24:15 Lori MacVittie Right? It doesn't have to be malicious. And being able to detect both is important.

Garland Moore Good point, yea.

Lori MacVittie So security is really becoming intertwined with application delivery concerns in a way that before they were just kind of, well, we're best buds, but now they're like, you know, together really intertwined. So I think that's important. We'll see more of that as AI continues to evolve.

00:21:24:21 - 00:21:35:15 Lori MacVittie You know, as organizations build it out and find all the problems. Well, we're we're kind of at time. So,

Joel Moses Yeah.

Lori MacVittie thank you both. As always, Joel

00:21:35:17 - 00:21:36:27 Joel Moses Absolutely, yeah.

00:21:36:28 - 00:21:41:06 Garland Moore Thanks for having me. Yeah. Love being here. Appreciate it.

00:21:41:09 - 00:21:54:07 Lori MacVittie Wonderful. Well, then that's a wrap for Pop Goes the Stack. If your uptime is still intact after that, celebrate by subscribing. Next time we'll see what other seamless solutions fall apart under scrutiny.

Open the transcript file

Hosts and Guests

Hosts

  • Joel Moses Host Feed-stated
  • Lori MacVittie Host Feed-stated

Guests

  • Garland Moore Guest Feed-stated
  • Tabitha R.R. Powell Producer Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Agent ecosystems: Identity, control, and the shadow AI problem

    Nov 17, 202623 minAudio

    AI agents are no longer “features,” they’re becoming enterprise citizens with identities, permissions, runtimes, and a very real need for governance. In this episode of Pop Goes the Stack, Lori MacVittie and Joel Moses...

    Credit the people on this episode

  2. Vibe coding is manufacturing tech debt at machine speed

    Nov 10, 202622 minAudio

    Vibe coding is moving fast, but is it moving software forward or just piling up a new kind of debt. In this episode of Pop Goes the Stack, F5's Lori MacVittie and Joel Moses are joined by Narayan Veeramani to dig into...

    Credit the people on this episode

  3. AI governance: Shadow AI, real access, real risk

    Nov 3, 202621 minAudio

    AI outputs may be polished, but in this episode of Pop Goes the Stack, F5's Lori MacVittie, Joel Moses, and returning guest Ken Arora focus on what happens when confidence meets real-world access. The discussion moves...

    Credit the people on this episode

  4. AI broke your app? It’s probably DNS (Again)

    Oct 27, 202622 minAudio

    AI may be rewriting workflows and reshaping infrastructure, but this episode of Pop Goes the Stack argues that the future still depends on an old, unglamorous constant: DNS. F5's Lori MacVittie and Joel Moses are joined...

    Credit the people on this episode

  5. Agentic fingerprints: Detect, debug, and control AI traffic

    Oct 20, 202620 minAudio

    AI is everywhere, but success metrics and security controls haven’t quite caught up. In this episode of Pop Goes the Stack, hosts Lori MacVittie and Joel Moses are joined by Lee Ennis to unpack “agentic fingerprints,”...

    Credit the people on this episode