In this episode of Data Analytics Chat, we welcome Ryan Stevens, Senior Director of Applied Science at Ramp, the financial technology company helping businesses save time and money across cards, bill pay, expense management, travel and more. Ryan leads a team of machine learning engineers and product data scientists, and before Ramp he worked on ads ranking at Meta. He also holds a PhD in economics. Ryan shares how Ramp evaluates AI once it is live, from structured outputs like invoice OCR to unstructured chat agents like Stack, Ramp's AI accounting agent. The discussion examines why labels sit at the heart of every evaluation problem, how synthetic data can stand in before launch, and why so many teams reach for an agent before they actually need one. The episode explores the three layers of production monitoring (errors and latency, business outcomes, and adoption), how much evaluation can be handed to LLM-as-a-judge versus human reviewers, and how leaders can tie evaluation back to business value by starting with the task and weighing its value against the cost of solving it. 00:00 Cold Open 01:02 Introduction to Data Analytics Chat 01:37 Meet Ryan Stevens and Ramp 02:59 How Do You Know Your AI Is Actually Working? 08:29 Accuracy, Quality, Safety or Business Outcomes? 11:26 Why Is AI Evaluation So Hard? 14:29 Has Generative AI Made Measurement Harder? 16:42 Good Evaluation Before Going Into Production 19:13 You Probably Don't Need an AI Agent Yet 20:01 Evaluation at Enterprises vs Startups 23:00 How Evaluation Changes Once a System Is Live 25:39 How Often Should Teams Check Performance? 27:05 Early Warning Signs Your AI Is Failing 28:11 Can AI Really Evaluate AI? 30:12 Using AI Agents to Test and Monitor Other AI 32:11 Your AI Works Technically. But Does It Create Value? 34:39 What a Strong Evaluation Process Looks Like 37:20 The First Thing to Put in Place Connect with our guests: Ryan Stevens: https://www.linkedin.com/in/ryanstevens3/ Thank you for listening!
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
ryan stevensSo we know how well our models are doing by just looking at precision and recall. Did we say it was fraud? It was actually fraud. That's precision. Did we catch all the fraud? That's recall. Those are what I would call like business outcome-based metrics. That's one set of criteria that you should always be judging your models based off of. A calculus you might have, which we often do, is like don't cost optimize. There's whole- organizations whose job is to cost optimize instead focus on building things with frontier intelligence and know that the cost is just gonna come down in the future, which is what we have seen.
ryan stevensI think too many people are like,"I wanna build an agent," and then you ask why? Like what's the customer pain you're trying to solve? What's the task you're trying to solve? Oftentime you can be like maybe let's just do something deterministic first. Let's do something that we have full control over. Like the car works, it's functional. Turns out it, you built a great car that no one wants.
ben parkerHello everybody, and welcome back to Data Analytics Chat. Today, we're looking at how do you actually know your AI is still working. So it's one thing to build and launch an AI system, it's another to continuously evaluate whether it's accurate, safe, useful, and still delivering the outcomes the business expected. So in this episode, we're going to look at what good AI evaluation actually looks like. And today joining me is Ryan Stevens from RAMP.
ben parkerWelcome, Ryan.
ryan stevensThanks, Ben. Happy to be here
ben parkerIt's gonna be an interesting conversation. And like I said, just mentioned before, a couple of candidates we've had into have, yeah, been obviously lacking in the evaluation side. So I think it's gonna be an interesting topic for our listeners. So I guess before we dive in, do you wanna just give the listeners a quick introduction to who you are and the work you've been leading at Ramp?
ryan stevensYeah, that sounds great. Let me just do a little brief intro of Ramp. So Ramp is a financial technology company. We have a suite of products. Our goal is to save businesses time and money. We offer card, AP, AR, a lot of back office technologies. We have a banking-like product. We also do expense management, which is one of our core offerings, travel. Sort of anything you can think you need to do to spend money at a company, we probably offer that as a service. I am a senior director here. I lead applied sciences.
ryan stevensThat is a team of, historically MLEs was our big focus area, as well as product data scientists. Obviously, with AI and agentic engineering in the last two years a lot of my folks are thinking about how to build better agents and like we're gonna talk about how to evaluate those agents better. Prior to Ramp, I was at Meta working with ads ranking, and I also have a PhD in economics. So if we wanna talk anything economics related, I'm also happy to go that direction as well.
ben parkerBrilliant. Cool. And yeah, like you said, the last, the last two years it's just gone crazy, isn't it?
ryan stevensAbsolutely
ben parkerSo I guess I'll, let's dive in then. So I guess once a AI system is live, how do you actually know whether it's still working properly?
ryan stevensYeah, that's a great question. So once it-- We will talk about it being live, but I think we should also definitely talk about like what to do before it goes live. So an AI system, it's just a probability machine making a decision. Now, it depends on what kind of decision your AI system is doing. I think there's like really two ways we can categorize these structured outputs versus unstructured outputs. So let's do a very exa- simple example of what a structured output is. So every day at Ramp people will upload invoices onto Ramp, and our job is to OCR those, to get the amount on that invoice right get all those little line items that you might see correct.
ryan stevensThat's a structured output. There is a correct answer. Someone will go review that, and we can see we said$100, they said$100. Great. You should be making sure if there's a human in the loop and it is structured output, success means that the output that you have is actually the output that the human expects. Now things get a lot more difficult with unstructured output. These are, typically your chat agents. People will go in, your Claude, your Codexes. At Ramp, we have an AI accounting agent called Stack. People can go in, try to do their accounting in a chat-like interface.
ryan stevensIn those examples, things get a lot more difficult, but we can definitely dive into what good evaluation is there. But what does good evaluation look like? It says you take the interactions of the agent with the user, whether it be unstructured, you run some grade on that. Now, grade can be a very generic term, and you do that continuously for every chat, for every interaction that comes through. So high level, that's what it is. I'm very happy then to go into the details of what it means to grade and monitor these systems if we think it's a good next place to go.
ben parkerYeah, it's five. Let's go deeper
ryan stevensOkay. Yeah, let's go deep. Okay, so let's split up the two sort of things you want to monitor. There is-- I'm an MLE by training. Historically, I would do things like assess fraud on cards. So we know how well our models are doing by just looking at precision and recall. Did we say it was fraud? It was actually fraud. That's precision. Did we catch all the fraud? That's recall. Those are what I would call like business outcome-based metrics. That's one set of criteria that you should always be judging your models based off of.
ryan stevensSo let's go to the example of unstructured output. That's a tough one. What you usually have to do is know the domain that you're working in. So I'm gonna talk about AI accounting, our stack product. So what are the people doing? They're trying to solve accounting tasks in there. And so we have criteria around, first, what type of tasks do we think they're trying to solve, and then what is the verifiable output, i.e., what can we see from the conversation that told us it did the right thing? There are some specific actions we can observe within our AI accounting products.
ryan stevensThey could do things like sync to their ERP. That's a success. So we have a criteria that if they sync to their ERP, that is a success. But sometimes we have more fuzzy things that they're doing. They might be doing something like manipulating an Excel document. So we're not really sure if there was a successful outcome for that type. So instead, we have to look at like behavioral signals there. Did the user show frustration? Did the agent take too long beyond some like guardrails that we think is good sort of response time? Was there errors that appeared to the end user that we don't think is a good experience?
ryan stevensSo it actually becomes almost a product experience measurement that you have to think about for these chat agents. You have to come up with a good rubric. You have to do your best to say what is verifiable, when can humans give me feedback. If they can, always ask for that feedback. If you can put a thumbs up, thumbs down in there, please do. Oftentimes, you're not gonna have that, so then you have to think about, okay, what are-- what can we infer from the data that we see? So frustration, agent doing obviously bad things, it's too slow, it looks like it's too costly, that sort of stuff.
ryan stevensSo when you're doing online evaluation for these unstructured outputs, it's really like step one, get your humans to tell you as much good or bad as possible. Often not possible. Step two, come up with the right criteria for the task at hand. So that's what I call business outcome metrics. Now, I should say for any agent, you should be doing what is normally called application performance monitoring, APM monitoring. How long do turns take? How much does it cost? What is the tool error rate? How often are tools erroring? Actually, that basic core monitoring we noticed early on is not natural and intuitive to most engineers.
ryan stevensAnd honestly, models are getting really smart. If you can make sure the harness that those models are executing in are executing it fast at low cost with low error rate, then usually outcomes do follow. So definitely, make sure you get the plain vanilla things right that you focus you eat your vegetables, as you would say, and you do that basic application performance monitoring in conjunction with what I call business outcome monitoring. So is it solving the task it's supposed to? And, is it operating efficiently?
ryan stevensThat APM monitoring, it's a very intuitive idea to people that have been building backend systems for a long time. The types of things you monitor for agents are slightly different, but it's ultimately the same type of problem. And it's a thing we notice there's a lot of standardization you can do. There's a lot of value you can extract from that.
ben parkerBrilliant. Listen, yeah, of course I like to go deeper because then they it's share insights. It's help people. So I guess what should organizations be eva-eva-evaluating? Should it be accuracy, quality, safety, business outcomes, or all of it?
ryan stevensYeah, that's a great question. Obviously all of those things are important but you can't maximize everything at all time. I will say for-- it depends on where you are as an organization, I should say the first thing. So Ramp operates an environment where like customer data, protecting it is very paramount. We run our agents on a harness that we call Laminar. We have a security team that has worked to lock that stuff down to make sure that we protect our customers' data. We, when we build our agents, don't have to worry as much about like data safety and customer safety because our engineering teams, our production engineering teams have already done that for us.
ryan stevensThey've maximized safety for the organization. That may not be true at all organizations. So when you're building your agent, you always should be asking those sorts of questions like, we're going to be running this data through a given provider. Do we have ZDR? Is it a provider we trust? All of that sort of thing. Definitely like a first and paramount concern, and I don't want to ignore it, but I do want to say that like at Ramp, we have teams that have thought deeply about that. So when we are building agents, what we're thinking about are, does it meet the task at hand, i.e. is it accurate enough?
ryan stevensConditional on that accuracy it depends on what kind of product surface area it is. Does it need to be fast? So if it's something that's happening in the background, if it's something that the user does not have to wait on, it doesn't have to be that fast. And that's great. You can run things over the weekend, at night, or on what are called flex tiers that are cheaper. And it depends on the product experience. Now, if it is a synchronous chat experience, it needs to be fast. And then the question comes up is like, how much does cost matter to you today versus tomorrow?
ryan stevensFor some products, if you're passing through cost to customers then cost is the, price sensitivity of that customer, so cost is very paramount. In other cases, what we, do is we know it's a technology whose cost is like really going down in the future. So we're not gonna try to optimize cost today because we have, multi-trillion dollar frontier labs whose job it is to bring the cost down for for the products that they ship, i.e. these frontier models. A calculus you might have, which we often do, is like don't cost optimize.
ryan stevensThere's whole- organizations whose job is to cost optimize instead focus on building things with frontier intelligence and know that the cost is just gonna come down in the future, which is what we have seen. So that's like a counterintuitive way of saying like oftentimes best not to obsess about cost for a nascent new technology where cost is coming down rapidly unless there's some constraints that are happening, i.e. like you're passing costs on to customers can be quite price sensitive, and then maybe you think a lot about cost itself. But I gave you a little bit of like an everything answer, but that actually is the answer depending on the situation.
ben parkerYep. No, that's good. It's a good all-round answer. And so then also you mentioned like, obviously, the models are getting smarter. So why is continuously evaluating AI so difficult in practice?
ryan stevensYeah, that is a, that's a great question. So there's like a few fundamental reasons. I think like the biggest reason is you often don't have good labels for the things you're actually evaluating. Let me start like there. That's the most basic problem that most organizations are trying to confront. So let's talk about those chat experiences. I've already described that oftentimes we don't have direct human feedback that this was a good or a bad chat session. In fact, humans might just leave that. So like what most organizations will underst- will learn very quickly when they stand up things is actually like labels is the core of your problem.
ryan stevensThis is not news, by the way, to any like machine learning engineer. A large portion of a machine learning engineer's time, especially when they're going zero to one on new models in an organization, is actually thinking about how do you actually label the data. That is something that like agentic applications are learning after they've deployed. But for MLEs, because they were training models using labels, they actually had to answer that question first. But now since you can just deploy an agent quickly, people are answering that question second.
ryan stevensSo like the first reason it's hard to continuously evaluate is you have nothing to evaluate against. And so organizations need to really think deeply about how they're gonna do this well. Now, people do things like LLM as a judge. That is a strategy, but you need to calibrate that judge. And the interesting thing you'll see if you've worked with it enough, is it turns out, new models will come out. So let's say, there's like GPT Sol six or whatever, and you had maybe GT- GPT Sol five or something on there, five Sol. Oftentimes like the newest version as a judge is slightly different.
ryan stevensMaybe it's more correct. Usually it's like on average more correct. But it does give different answers. And so like you need to continuously calibrate these judges with like very high-quality data. That's something that organizations don't do enough of, and they need to do a, a lot of going forward. I think the second thing is, and this is more of an organizational thing as opposed to a technology thing, evaluation is hard It's a very hard skill. Measurement is a hard problem. Not everyone in your organization is going to know how you rigorously measure something.
ryan stevensThat intuitively makes sense, right? People don't know everything. You have entire data departments whose job was and is measurement. I think one of the big challenges for data organizations going forward is that the thing you need to measure now are these agentic systems. And, we've had a whole playbook, a whole industry for how do you measure, y- events going to marketing platforms or how you measure funnels on a given webpage. It turns out a lot of the data professionals do have the necessary skills to think rigorously about measurement but it's now gonna be a new domain, which is agentic measurement.
ryan stevensSo I actually think evaluation and continuous evaluation is a, it's always a measurement problem, and how are you measuring that effectively is something where the skill set matters a lot here. Yeah.
ben parkerSo have you felt that it's got harder because of generative AI?
ryan stevensAbsolutely. Yeah, I think the, the problem of measurement has gotten a lot more difficult because before you can think when you had an ML model, I'm always gonna go back to an ML model, its job, its function was one thing. So we have a card transaction fraud model. Its job is very clear. It's in the name. Its job is to predict whether or not this card transaction is fraudulent. I've talked about Stack, our AI accounting agent. Its job is to do bookkeeping. Bookkeeping is a ton of different discrete tasks that can be solved.
ryan stevensReconciliation, schedules and accruals, variance analysis, fluctuation or flux analysis. Those are all individually different tasks. And so thinking about how you can rigorously evaluate something across multiple tasks is very difficult, especially when the agent can decide what task it's trying to solve. So the, the way I've talked about this internally is actually it's best to, when building an agent, focus on one task.
ryan stevensWhat is the one task you want that agent to do? Measure the heck out of that. Make the agent do that thing. In fact, I recommend just do a prompt. You can get get far with just a good prompt with the right data in it. No, no agents, just LLMs LLM calls. Okay, and then do a second task. Okay, and then do a third task, and then do a fourth task. And as long as you follow this task-based paradigm, then you can have good measurement along the way. And then once you have enough tasks, you're like,"Okay, now it's time to have an agent because it can do many things, and I don't wanna do these individual sort of LLM calls that I'm doing.
ryan stevensI'm building an agent from first principles." But the, the problem that a lot of people will run into with measurement here is they're like,"I want an agent that does a function, that does a whole set of things." And then they wake up six months later and they're like,"What does good look like? It turns out it's trying to do 12 things, so I need to measure 12 different things." Whereas opposed to if you iteratively start from a task paradigm you're able to get measurement for free as you go. But yeah, TLDR, it's hard. It's gotten harder. The tech is great but it's opened up a lot more measurement challenges.
ben parkerOkay, let's move on to the next question then. So I guess what does good evaluation look like before an AI system goes into production?
ryan stevensYeah, that's a great question. I'll talk about the experience with Stack. Before it goes into production the problem you really run into is does-- you have no data. And so you need to figure out,"How do I evaluate something without any data?" So I'll give the experience of Stack, and then I'll give some thoughts about what you can do in sort of different situations. So like I've talked about with Stack, Stack is a AI accounting agent. It's supposed to do a set of tasks. And what we realized early on is Ramp didn't have a product that did that, so there wasn't any proxy data that we could, look at internally and try to create some synthetic data from. And it's also a very expert-driven field.
ryan stevensAccountants are well-trained, they have years of experience, they do a complex task. And so what we did is we worked with a company to create a synthetic data set or a set of synthetic businesses. And these synthetic businesses were like real businesses. They had everything in their books, so they had their GL accounts, they would have bills, they would have invoices. They'd have a whole set of documentation that looked like a business. And then we'd have a set of tasks that a real-life accountant would have to try to solve for each of these businesses. And those, those tasks would get graded by accountants.
ryan stevensSo we had a rich offline data set that we could build and evaluate an agent against. And this was a very useful exercise because it allowed us to tune our harness, tune our skills, tune our prompts and also figure out which models were best at this. A lot of people underestimate the time one should spend before they launch a model online. And this is one of those examples, is synthetic data set generation is a pretty good exercise. There are other scenarios where I think that's actually probably overkill.
ryan stevensAnd those are examples where you think that the model out of the gate... there's a few reasons. One is if the cost of error is low enough, test in prod is an idea. Roll it out to some, see how it goes, and then go from there. So if it's, a low cost of error, if you're just maybe showing ads to people you can show some wrong ads. As long as they're, like, high enough quality and they meet certain bars, that's okay. But let's say the cost of error is high enough that you don't just wanna test in prod but the, the task is not that complex. You probably don't need this full offline data set approach.
ryan stevensYou don't need to generate synthetic data before you go in. So if there's a lot of precedence for what you've done and the task is not that complex, then definitely go ahead and launch stuff online, at least into a shadow mode, which is what we do a lot of. It's not It's using live production data, but it's not making decisions. That's the intermediate that I recommend a lot of people inevitably do. And if I go back to the example I gave to the prior question You also don't have to build an agent out of the gate. I think too many people are like,"I wanna build an agent," and then you ask why? Like what's the customer pain you're trying to solve?
ryan stevensWhat's the task you're trying to solve? Oftentime you can be like maybe let's just do something deterministic first. Let's do something that we have full control over. Let's see if that actually provides like a monochrome of value. Let's prove it can work," and then great, now we have a baseline that we use to generate some data, and then we can compare an agent against. So my recommendation almost always when someone comes to me and is"I wanna build an agent for something," is like,"You don't need an agent for that today. You need to show that you can do something simpler here." That simple thing will generate the data that you need, and then you can use an agent and you can horse race an agent against that problem
ben parkerSo would you... Obviously, you worked both, didn't you? Like enterprise and I guess startup, scale-ups. What is there a difference between the two, would you say, when doing evaluations?
ryan stevensYeah. So for su- for sure there's different considerations. I think one of the big things obviously when you're smaller is just the blast radius of your changes is smaller. So you're a company that has, millions of customers and you launch a system it could affect 1%, and 1% of a million 10,000. That's still a lot of people. Whereas let's say you have 1,000 customers and 1% is 10 people, those are people you can literally individually go talk to, right? I think the interesting thing at a startup scale, which Ramp is no longer a startup scale, we have 80,000 plus, but when I joined we were much, much smaller is that you can actually have personal relationships with clients and you can use those as design partners when you're launching things.
ryan stevensNow, obviously enterprise grade still does a design partner. They still do a slower rollout. But you can roll out things quite wider, and as long as the error rate is low enough the actual number of people that you're going to affect is low and you can personally talk to them, figure out what the difference was, and then tweak the system from there. That's like like one of the big things that you think about is does the error rate the total number of customers actually just overwhelm your company is like a, a really big problem to think about. I obviously think the, the thing that we've all learned about AI over time is that context is king.
ryan stevensAnd so being able to connect disparate pieces of data and let the agent reason through that is hyper important. What enterprises are learning today is that they've built data silos for a myriad of reasons, some good, some less than ideal. Good reasons being security is obviously a good reason you want to silo and make sure that there's the right like permission control. Some less than ideal. It's just two people had the same idea. For whatever reason, they were able to build the same thing, and now you have two of the same thing in different parts of the org and they don't talk to each other.
ryan stevensAnd those silos, those disparate pieces of information that no one in the org can get access to, is a hindrance at enterprise companies, right? You can't actually get the right data to the agent to help it reason through the right decision. And like, when people talk about AI native companies, what they're actually, I think, referencing or where the, the potential comparative advantage comes from is that they're built in the idea of agents being able to integrate into data early on. And silos don't exist as much. And so because of that because you have access to more data, you can also evaluate these things a lot easier.
ryan stevensYou're able to actually be like, okay, like we're building an agent that's supposed to do multiple different things. I can see all the different data points. The company is small enough. I can actually go understand how it works across these places, where as opposed into siloed organizations that's a lot more difficult to do.
ben parkerBrilliant. No, thanks for that. I thought we'd just get... So I remembered you, you obviously covered both, so it'd be good to get a perspective from both the enterprise and I guess you're not, you're no longer like a s-startup. I know it's more start scale, but you've been in both, so it'd be good to get a perspective of people. So moving our focus off to production. So once the system's live, what changes and how should evaluation continue over time?
ryan stevensYeah. So once it's live then it becomes like any other application that you're trying to monitor. You wanna be proactive versus reactive is, I think, like the name of the game there. Similar to any application, there's like Maslow's hierarchy of needs. You wanna know if things are failing, is the first thing. What people have learned, by the way, is that, your Frontier Labs APIs, they do go down. They are resilient, but they're not, as resilient as some other type of technology. When it goes live, you need to know errors. Errors is an important thing.
ryan stevensYou need to have good fallback mechanisms, by the way. That's why routers, which Ramp has a router we have a great router product, but that's why a lot of customers demand routers is because you need to have good fallback models. Like your first concern is errors. Are endpoints going down? Are tools you're calling not working? Tools are basically just an interface to access internal databases, but they're the main thing that fails with agentic calls. The first thing you need to do is am I tracking error rates? Do I have good alerting for that? Do I have good organizational hygiene around treating these, like, serious issues with escalations, incidents, all that sort of stuff?
ryan stevensThat's like the easy stuff, but the stuff that if you don't do correctly you're going to learn it from a customer later than you otherwise should have. Next is like the business outcome monitoring. Every trace that comes through our system gets evaluated. It gets evaluated by LLM as a judge that will then create a metric, like the percentage of user sessions where frustration is expressed, and then we have monitors on top of those, did those spike recently? Usually, those will spike for the errors reasons I told you about. Could be that a tool started failing all of a sudden or an API provider.
ryan stevensBut it's a second line of defense that you wanna have here. Every trace gets an LLM as a judge. That turns into a metric. That metric then we can put an alert on top of. That thing we can then review. That's like the next layer of defense. The final thing is usage, adoption, your standard sort of product tool set, you obviously wanna be evaluating here. People will use an agent if they find value from that agent. I've talked a lot about this as like an engineering system, as one, that needs to work. The wheels need to not be flat, the tires need to not be flat, the engine needs to be like in good working order, like a car.
ryan stevensBut did people go to the lot and buy that car? So you still need to treat it like a product, and you still need to have all of your usage and adoption metrics, and that's the last hierarchy of need. Like the car works, it's functional. Turns out it, you built a great car that no one wants. That's the last thing you always need to answer. Those are like kind of the three like layers of things that matter once it's in production.
ben parkerNice. And then, how frequently should teams be checking performance?
ryan stevensIt's a good question. The errors should be like on a normal on-call rotation. No one should have to systematically check it. Those things should proactively be coming after you. It really depends on two, two factors. What scale are you at? How many customers do you have on this? If the N is small, then, the error of any performance metric you have is going to be very high. And then the second is the delay between it committing an action and you getting some label. So let me give you an example of something where there's some delay. When we do OCRing for invoices we have a review process.
ryan stevensSomeone goes and looks at that invoice, reviews it but they may not review it for 30, 60, 90 days. Some-- So there's some lag between when the agent takes the action and when we actually get a good label. Definitely don't review things before you don't have good labels. And then make sure you have a big enough N so you have tight standard errors. Once it's at scale, once it's in production you should review it like you would any other product. Like weekly. You have weekly posts, updates, stand-ups. If it's an important product area, people should be obsessing over metrics on a weekly cadence. Now, of course, life cycle of the product obviously matters.
ryan stevensIf you just launched this thing, you should be obsessing about it hourly, daily, at a lower, like at a much higher frequency. But, I think that the mistake that a lot of people make about agents and the agentic system is it's no different than any other product. So whatever your rituals and cultures are around product evaluation, it should be the same for agents.
ben parkerGood. So is there like warning signs that an AI system is beginning to fail even if say your users have not started to complain?
ryan stevensYeah. Internally the, the big thing we look at is latency and error rates. So if something's taking too long people will start to complain. And if there are high error rates, people will start to complain. So tool errors is the one we talk a lot about. That's like a very common one. A tool could error for many reasons. It could have issues with accessing the database. It could be that the tool is too complex for the agent to actually use. The latency is a more difficult one, but one that we track quite obsessively. So things could be getting slower for a lot of reasons. Like you could swap a model, and it turns out that model is really bad at using all the tools that you have in your harness and therefore taking a lot longer per task.
ryan stevensWe do see that happen, like from time to time, that like a new model gets launched for the same model family we're on. Ooh, we switch to it. It turns out it's actually like way worse on the task at hand, and so it's like taking a lot longer to answer user questions. So like error rates, usually tool error rates is what we like to see. And then latency are like two leading indicators that are pretty important to like customer experience.
ben parkerOkay Then I guess let's talk about the human side of things then. So how much of the AI evaluation still needs human judgment, and how much can realistically be automated?
ryan stevensYeah. That's a great question, and one that I think if you ask 10 different people, you might get 10 different slightly answers. My opinion could be stale in six months given the the pace of the technology. But we'll-- w- what I will say is this: when you initially launch something, you for sure should have a good golden data set that you've calibrated with humans. You could use model assistance to help you as well. I'm like, everyone does it. Everyone uses like a quorum of models to help them label things. But you should very rigorously go label some data for anything and everything that is gonna be important for your business.
ryan stevensAnd then you should build an LLM as a judge, and you should calibrate that LLM as a judge against your human labels. It should match the human labels down to a T. Once something is in production my strong opinion is you should have human evaluation on at least some subset of traffic at all times so that you then can calibrate any LLM as a judge. So what we historically see with agents is like once an agent is live the distribution of things that it does, of tasks it's trying to solve, is gonna be slightly different than what you initially evaluated on.
ryan stevensIf you don't have some human evaluation going on over time then you'll not actually have calibrated those LLM as a judge. And all I mean by that is like the LLM as a judge might be saying, things are great." But it turns out you calibrate against at a, a different distribution of tasks and that LLM as a judge doesn't, know what's going on. So my strong prior is like there needs to be some human validation somewhere here. Sometimes you get it for free as a product. I talked about the invoicing example. You get that for free because people review the OCR because they need to go pay the bill, and they need to write all that information for you. So if you get it for free in your product, you have a human there.
ryan stevensThey're already annotating and evaluating for you. And if you don't, then my strong opinion is that you should have some continuous evaluation by humans so that at least those LLMs as a judge or your AI systems have something that you can judge and grade against.
ben parkerYeah. So then can AI agents themselves be used to continuously test, challenge, and monitor other AI systems?
ryan stevensIn some respects, yes, in some respects, no. AI systems are good, I would say at things like testing for edge cases. So like coming up with slight variants of prompts and seeing what the output is or like, things that you could call like taste, they can be pretty good at. You're like,"We prefer, shorter versus longer answers. We prefer answers that are bullet points versus not bullet points." You could certainly put all that into like a function that parses the text to try to understand that sort of things. But a lot of those taste things, it's pretty good to give to an agent to be able to do.
ryan stevensBut in my experience you ask Fable, you ask Astra correctness, business outcomes and you can get wildly different answers between the models, and it takes a human to come in to ascertain whether or not which one is correct. So there's still definitely a jagged frontier, as they like to say, i.e. some models-- it's unclear where models are best at, and especially for labeling data or for ascertaining whether or not it was correct. There's still a lot of limitations there. Another way of thinking about this is actually quite simple.
ryan stevensIf an agent is good at determining whether or not another agent successfully executed a task, why did the first a-- Why do you even need the judge in the first place? 'Cause shouldn't the first agent have been able to successfully execute the task? Now there's some like holes in this logic, but I think generally that's where a lot of my head goes, which is if it is true that agents could judge other agents like with 100% accuracy, then you actually wouldn't even need judges in the first place'cause the first agent should be able to self-referentially judge itself and figure out like what's right, what's wrong and then always do the correct thing.
ryan stevensAnd you don't see that in experience. So there's, there, there's definitely something missing here. And a lot of that is there is like ambiguity that models aren't fully able to reason through, and you sometimes do need humans to at least evaluate and figure out where those gaps are.
ben parkerOh, that's some good information there. And then I guess obviously, look, AI and business value, it's, there's always a lot of talk there. How should leaders connect AI evaluation back to business value rather than, just treating it as like a pure tech exercise?
ryan stevensYeah, that's a good question. It's a I think it's more difficult for people that have not thought about ML systems, I would say. Let's return to my task-based approach. If you are successfully articulating what tasks you're trying to use agents for early on, the value automatically derives from the value of that task. Now, the value of that task could be, time saved for our customers. So we're able to save, an hour for our customers and express that as interest. It could be an hour for an internal person, so you're able to say wage and cost time.
ryan stevensIt could be an experience that is new for the customer, and so you're actually, like, expanding the realm of the things that your application is able to do, so you're actually able to fulfill a need that you otherwise weren't able to. So I think a lot of people struggle at this exercise when they don't think about the the just the pure tasks it's trying to solve. 'Cause if you have the task it's trying to solve, then value automatically flows from the value of that task. And if, it turns out that task is extremely low value, you should not be, maybe throwing fable-priced intelligence at it. There are much cheaper models you can do to try to solve the task at hand.
ryan stevensSo yeah, i- if you don't know what it's supposed to do then value becomes different. And we should also obviously talk about the cost of solving that thing,'cause really I think about net profit or, net value. It's really value to the customer or value to the firm minus the cost to the firm or the cost to the customer. And so you should really be, like, thinking about, okay that's a high-value task. We can, pay money to solve it. It could be... Or it's a task that requires someone that we pay a lot to solve, so we can probably use a fairly expensive model to try to solve that problem.
ryan stevensI think it's also pretty good to just, think about, like, where in the quadrant of value and cost are you, and then you select the right model for the right problem. Or sometimes, and, you determine actually that's a pretty low-value task. It doesn't cost us very much to do it, a- and we're gonna either throw a very cheap model at it or, an agent isn't the right choice. I think the thing is the, the purpose of good leaders is to look at opportunities break out the value of that opportunity, look at the different cost options to solve that given opportunity, and come up with something that like maximizes the, the net value of things.
ryan stevensYeah.
ben parkerOkay. And so I guess building this process is a challenge, and it's-- and complex. So I guess what would a sort of strong evaluation and monitoring process look like inside an organization?
ryan stevensYeah. So I think w- one is it's like a cultural sort of thing you have to ascertain. It's like, do they have good on-calls? Are those on-calls being alerted about agentic incidents? I think that's I love to go back to my hierarchy of three things, but that's like the most important thing. It's like, if they have a strong culture of on-call rotations, if those on-calls are like very data-driven, i.e. they have good alerting for it then I'm gonna say that organization is most likely going to be like ahead of the curve when it comes to other sort of companies.
ryan stevensI think the other thing that's important and we've started to really lean into that here is I have this, this fundamental belief is that no one wants to label data. No one wants to bring good labels for your model. So some organizations will say,"Oh, okay, we build an agent and maybe some of our employees will go label this data." No one wants to do that. And so good evaluation at a company also is about how do I bring the cost of label creation down as much as humanly possible. So at Ramp, one thing we're investing a lot of time in is like a internal evaluation platform.
ryan stevensSo what that means is every agent already runs on a common harness at Ramp. That's like important standards, and platforms are very important, especially in the agentic era. All of those can run and build evaluations on a common centralized platform, and we're making it super easy for, bad interactions that aren't effective, to get bubbled up to engineers to then turn into good evaluations on that agent. So how do you as an organization think about building platforms that allow you to create labels at low cost for the most like important interactions that you have?
ryan stevensAnd so I think like good organizations will think deeply about like evaluation as a service and what sort of platform to- tools you do. And then a lot of it is like how much are you educating the organization on the importance of evaluation, what a good evaluation looks like? Like I said, everyone is doing this for the first time. Evaluation, agentic evaluation is new. And so it is a learning exercise for everyone in the organization. And like distributing learning, upskilling everyone at companies, it's guerrilla warfare. And what I mean by that is it's a lot of one-on-one conversations.
ryan stevensIt's a lot of seminars. It's a lot of posts. It's a lot of like documents. It's a lot of things to just try to like, get in the mind meld of the organization. But good organizations don't just treat this as a tech problem, but they think of it as a people problem. And so how are you trying to upskill and uplevel your people is like one of the most important things that I think a lot of companies and a lot of leaders underestimate here.
ben parkerOkay. So then I guess to finish on, what-- say if a team's already got an AI system in production but d- but does not have a strong evaluation process, what would be the f- first thing you'd tell them to put in place?
ryan stevensOh I would have to take a deep breath. That's a-- it de- I think, first and foremost if they're not evaluating it w- let's split up evaluation in- into the two camps. Let's do the basic thing first, like I said, like error rates, latencies, costs. If they don't have that monitored, do that first. That is almost deterministic, it's easy to do, it's low cost to do, and it'll save them a lot of time in the future. So let's say they have that. Let's say they're a good or- engineering organization, so at least they know how to monitor applications.
ryan stevensThe second thing I would then-- the very first thing I would tell them to do is go figure out what good looks like here. What are your labels? That's always the first question I ask any internal team when I come across an agent or they bring an agent to me or my team, and they're like,"We wanna try to figure out like, is it doing, what it's supposed to be doing?"'Cause then the question I naturally ask them is,"What is it supposed to be doing?" There's an AI agent in production. The first thing I would ask them to do is,"You need to write out the five top tasks it's supposed to solve, and you need to tell me what does success look like for each of those tasks.
ryan stevensAnd then you need to determine, do we have something in the interaction with the human at the other end that tells us they did or did not do those things?" And from those like, from that criteria, then you're like,"How do I collect my labels?" So it's all about labels. If you don't have labels, you don't know what success looks like, so that would be the exercise I give them.
ben parkerPerfect. And I said you've, you've-- Yeah, I love your passion, and you've had a... Yeah, it's been a pleasure having you on the podcast. Thanks, Ron
ryan stevensThanks, Ben. It's been great chatting today
In this episode of Data Analytics Chat, we welcome Saadia Khilji, Global Chief Information Officer at Coverys, a medical professional liability insurer that has served the industry for more than 40 years. Based in...
In this episode of Data Analytics Chat, we welcome Lior Barak, Author & Founder of Impact Operations. After 15+ years in data organisations, Lior now helps data teams and their leadership treat data and AI as a capital...
In this episode of Data Analytics Chat, we welcome Erica Howard, Head of Consumer Data Ownership at TD. Erica has spent 18 years at the bank, starting in the call centre before working on TD's early data governance...
In this episode of Data Analytics Chat, we welcome Mary Alfheim, VP of Data and Analytics, and Joseph Kroon, Data Architect, both from Headspace. Mary previously led analytics teams at Amazon Prime Video and HBO, while...