Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Data Analytics Chat

Data Analytics Chat artwork

They Tried 3 Times. Then AI Changed the Enterprise Data Build

Oct 6, 202646 minAudio

In this episode of Data Analytics Chat, we welcome Mary Alfheim, VP of Data and Analytics, and Joseph Kroon, Data Architect, both from Headspace. Mary previously led analytics teams at Amazon Prime Video and HBO, while Joseph spent time in Cigna's data organisation and, earlier, as a Black Hawk helicopter crew chief in the National Guard. Mary and Joseph share how Headspace rebuilt its enterprise data model after three earlier attempts failed to keep pace with the business. The discussion examines the trust problem behind the project and why organisational alignment and technology have to move together. The data topic explores how AI changed the build: generating the data model from data contracts, mapping end-to-end lineage, and letting business users self-serve trusted numbers through AI agents. They also cover keeping humans in the loop, evaluating accuracy, and why leaders should agree on the use case before choosing the tools. 00:00 They Tried 3 Times: A Paradigm Shift in Building Data 01:22 Introduction to Data Analytics Chat 02:04 Meet Mary Alfheim and Joseph Kroon from Headspace 04:16 The Original Problem: A Crisis of Trust in Data 14:02 Three Attempts and the Lessons Learned 19:44 When AI Became Part of the Solution 28:43 Combining AI with Human Expertise 32:44 What Building Enterprise Data with AI Means in Practice 35:06 Making Sure AI Output Is Accurate Enough to Trust 39:37 Driving Adoption Across the Organisation 40:24 What Changed for the Business 41:46 What They Would Do Differently Next Time 43:17 Lessons for Other Large Organisations 44:09 The First Step for Data and Analytics Leaders 45:37 Where to Connect with Mary and Joseph Connect with our guests: Mary Alfheim: https://www.linkedin.com/in/mary-alfheim/ Joseph Kroon: https://www.linkedin.com/in/joseph-kroon-902243109/ Thank you for listening!

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Josephquite a few years back, I was a, a Black Hawk helicopter crew chief in the National Guard. Yeah, if you don't trust a Tableau dashboard, then you're probably not gonna trust a Claude response.

MaryInstead of writing all this code by hand, with Joseph's approach as, as the architect, we decided to automatically generate our data model from our central data contracts using AI code development. So instead of building the data model, we generated the data model, which to us was really like a paradigm shift and a really different way of thinking about how to build. Joseph is underplaying it. I think he was so excited, Ben.

MaryLike, I started to worry Joseph was getting AI psychosis. I was

Josephworried I was too.

MaryFor the first time I can see the full lineage I can generate the data model, the pipelines, et cetera. it really was mind-boggling and there was this kind of sense of awe because I think we all just deeply internalized that the way that we've built things in the past and how we've thought about these projects just fundamentally changed. Things are different now. We have to build differently.

ben parkerHello, everybody. I'm your host, Ben Parker, and welcome back to Data Analytics Chat. And today is quite an exciting one. I've got two guests, and we're looking at a interesting challenge that a lot of companies are dealing with right now, is how to modernize enterprise data in a legacy environment, and I guess where you can get the benefit of AI. And today's guests have actually done that, and so it's a success story. I know they've gone through multiple starts here, so there's gonna be some good lessons learned.

ben parkerSo yeah, I want to welcome two people from Headspace, Mary Altheim and Joseph Kroon.

MaryThanks, Ben. Nice to be here.

JosephYeah, absolutely. It's a pleasure.

ben parkerYeah, do you wanna just both give a brief introduction to yourself and see the work you're doing at what the types of work you've been doing at Headspace?

MaryYeah. I can kick us off. But I'm Mary Altheim. I'm VP of Data and Analytics at Headspace. I get to lead the team that we call DNA not unique in the data space, but I still love it. But we cover data engineering, analytics engineering, BI, data science, data product management and governance. And the company today, and, the data that we'll be talking about, it really pertains to mental health. Headspace is a, a mental health company, so our data, it spans consumer app engagement, but also coaching and clinical care.

MaryAnd we have kind of a D2C business, but we also have enterprise partnerships with health plans and employers. That's where I am now. Before Headspace, I led data and analytics at a company called Script, and also spent some time leading analytics teams at Amazon, Prime Video in particular, and HBO.

JosephYeah. And my name's Joseph Crone, and I've been with Headspace for a little over a year now, and I've been their data architect, and I've been working on rebuilding their platform. Before before Headspace, I spent quite a bit of time at, at, at Cigna in their data organization, and consulted around for a bit. And quite a few years back, I was a, a Black Hawk helicopter crew chief in the National Guard.

ben parkerSo quite the career pivot Joseph. Yeah.

JosephIt continues to be.

ben parkerGood. Good. Now it's great to have you both on. And so you obviously look, you're, you're dealing with really confidential data, aren't you, at Headspace?

MaryIt's about as sensitive as it gets. So it's PHI as well as PII, absolutely.

JosephYeah, it really the nature of the sensitivity kind of colors every almost every design decision that we're making within the data org. So it's always... It- it's either top of mind or at least in the background in, of almost every conversation.

ben parkerBrilliant. Cool. And I'm, like, I'm obviously gonna love the story we're gonna discuss today because obviously it's such a challenge for businesses with their enterprise data. So I guess what was the original problem you were trying to solve with your enterprise data environment?

MarySure. Let me take that one, Joseph. So what we're gonna talk about today basically is, yes we rebuilt our enterprise data model, and we used quite a bit of AI to do that. But essentially, we had a huge trust problem. We as a, as an organization, we did not have common definitions for just key metrics to understand how the business was running, what our member experience was, and, how to think about outcomes for our members. So we had fragmented numbers across different teams.

MaryWe had no sources of truth or strong governance for key tables that c- we could perpetuate data through different endpoints, not just reporting. And just to put it really simply our leadership team and our stakeholders, they just had no faith in our data. They didn't believe anything, and they started to learn to make decisions without data, which is about, about the worst place that a data pl- a team can walk into

ben parkerYeah. No, we hear, we hear that a lot, but it's trust in data is, yeah it's a growing theme, isn't it? Across, across every in- industry, if I'm perfectly honest. And so g- guess what was not working with the existing approach? Was it... Yeah, just I guess it'd be good to get a bit more feel for there.

MaryOoh. I'll talk about the people piece, and then Joseph, maybe you should talk about

JosephYeah. Yep.

Marythe build. So the first is organizational alignment. Our stakeholders our humans in our loop did not agree on definitions of common metrics. So we might have different views in different parts of the company that spun up different metrics for what signup rate really was, or how retention was calculated, or what should be included in something like covered lives. There were different definitions that people had used and spun up, small tables to quickly get something out the door.

MaryBut what that meant was a metric could be named the same thing, but have very different numbers floating around the organization, depending on what it was. So the first thing that we did is we really took a step back, and we thought about how to think about metrics like a product, like a data product. And we wrote, with cross-functional stakeholders, PRDs, product requirement documents, like metric requirement documents, we started calling them like MRDs, to align organization from finance, clinical, product whoever needed to be involved, really make sure that we got the definitions right and commit to those.

MaryAnd that was our starting place to say,"Okay, if this is truly what the business needs to run, this is how we agree that it has to be defined, now we know what we need to go build or rebuild." But we didn't start... We didn't have that. We just did not have that agreement, and so we really had to start there with that org alignment. But then that brought up another problem with then the actual technical implementation. And Joseph, maybe you can talk about the state of the warehouse.

JosephYeah. That-- I think what was not working involved so... it's such a complex problem. But on the technical side the reporting tables that had been built over time had organically grown. And similar concepts like what Mary was talking about, that sort of had this similar or close to the same definition or s- close to the same source systems that fed it they'd been blended together so many times that untangling, untangling those the business rules or the transformations or where different decisions had been placed within the data model, navigating those intricacies while also building at the speed of the business was, it just wasn't possible.

JosephSo even with alignment. Alignment is such a, a transient state. It can disappear very quickly. And so if w- what was happening was even if we were getting alignment in these smaller areas or these, these targeted areas, the, the, the teams that were building the, the tables weren't able to move fast enough in order to in order to realize th- the decisions that were being made about how the data should be landing.

JosephSo that w- that's probably one of the biggest things that, that wasn't working. The int- the platform was, had, was, had grown organically, and on the technical side, we weren't able to deliver fast enough to capitalize on the alignment that that we'd be able to get from time to time.

ben parkerBrilliant. So then I guess why did it become important for you to solve now? I know lots of the... Now the obviously the boom in AI, especially LLMs data needs to be more, yeah, precise. Was that a key driver or was it just you wanted to solve the problem?

MaryWe got to kinda kill a lot of birds with one stone. So just fundamentally, you have to know how your business is performing. You have to be able to feel confident in metrics that you share with your board, with your senior leadership team that you use to make decisions. But essentially we wanted to make sure reporting was in place, but two other like really huge use cases were coming up, maybe three. The first is experimentation. We partner really deeply with product, and we have an incredibly like science forward product team.

MaryWe're very lucky to get to work with them. But they wanna be able to run experiments and self-serve. That means you have to have the same kind of metrics, inputs, outputs, and guardrails put into our system so that everyone can make the same decision based off of data that they can trust, dial something up or dial it back if it's harming the business or our members. So that's a key one. The second might be around personalization. We know so much about our users, but there's really... Or, and our members, but there's not a great way or there wasn't a great way to tap into that for trigger campaigns, in terms of like personalized marketing messages or in terms of a personalized experience within the app.

MaryAnd then of course, like last but not least, yes, of course. Like of course we wanna be able to use AI. And I think we're very much in line with and we're building right along with like what you might have seen from Anthropic a few weeks ago. It's back to basics. You have to not only build data models and documentation and lineage that in the past like maybe humans needed to read, AI needs to read it if you wanna get accurate numbers, a deterministic output back out of that using AI.

MarySo we knew that we really had to start with that data model if we were ever going to get to a place where our users could make use of that data and meet them where they are, which is, through Claude or through ChatGPT or, through Cursor, through the AI that they really want to be able to self-serve with.

ben parkerYeah, no, it's like, it's amazing you've tackled this'cause so many businesses have overlooked this, and I think now it's gonna get to the point where you, like you need to get it in order. It's been... It's just the way it is now, isn't it? 'Cause you need, like you said, you need the accurate numbers, and I think it's... It is the dirty work, isn't it? It is... I guess you pro- probably discuss it in a bit, but it is challenging. It is tough to obviously un-wrangle all your data.

MaryIt really is. And, we used to talk about things like data governance and data management, and now for AI, what we have to manage is a semantic layer and a context layer. And that's, tho- these are terms that sound so sexy and exciting, and we only think about it with AI. But if you really peel back what that means, it means going back to understanding what is the data supposed to mean? Do we have the right definitions? Are we aligned on those? And what is what are all of the pieces of context that we need, including a, a model that makes sense where you can understand your sources and your data contracts?

MaryYou have to have all that context built in if you're ever gonna get something usable back out of it.

JosephYeah, if you don't trust a Tableau dashboard, then you're probably not gonna trust a Claude response. It's and so many of these things are becoming table stakes. What, what used to be like really-

MaryThat's a really good pun, Joseph.

JosephYeah.

MaryData table stakes.

JosephThey are ta- table stakes. Because if basically what used to be these really competitive like key differentiators are now just what you need to start playing the game. And the rate of acceleration that we're seeing across the data industry is is really, like mind-boggling. It's absolutely transformative. So the organizations that are able to build these practices, to establish these data foundations earlier are going to be accelerating faster than anyone else.

JosephSo it's really... it really is in my view, a moment that, that has to be met, and it's not guaranteed to be met, and that's that, that gives me a lot of energy.

ben parkerGood. And then I know Mary, before we had the obviously the initial call, you mentioned the project had effectively started three times. So I guess what happened during those early attempts, and what did you learn from it? And obviously you've touched on people and the technology and obviously, and I guess over time, yeah, technology's got better. So is it... Yeah, if you can just elaborate there.

MaryYes. So we so I wasn't here I wasn't a part of this, but my understanding is we tried three different times between 2023 and 2025 to get something like this model going. Leaders knew, I think instinctually, that this was something that they had to tackle, but essentially they couldn't keep up with moving business needs. And they couldn't execute fast enough If, like if you think about how development has worked in the past for a conceptual, logical, physical data model, these cycles can take months, if not years.

MarySo even if you can get something down on paper, that's great, but if the execution, like the, the commitment and the resource allocation to build this, even putting aside having organizational alignment on what data is important to build, if the execution takes too long, the business will change again. And maybe you'll even get drift from what are the right metrics or what is the right use cases and data that you need anyway. So I think we learn like technology, organizational alignment, they have to move together.

MarySo even if we had the capacity and the fast platform without the metric agreement, then it's just, delivering data faster for people to fight about. But if you have great plans and all that alignment, but it takes like months or years to really get something usable out the door, that, that platform is just never gonna be useful. The business will change by the time you get something there.

ben parkerSo I guess what changed the third time then? Was it just'cause of, was it just, yeah, trial and error? Or obviously now, obviously, I guess a lot more of the tech now, there's a lot that is heavy lifting is done for you, isn't it, in, in data and AI. Obviously we've got the heavy lifting done. Do you think that was a key factor or- Oh, yeah was it more just getting people on board as well? I guess the education piece is pretty crucial as

Marywell. Both. Yeah, I- I'll talk really quickly, and then Joseph, I want... Joseph, I want- I've got some strong

Josephopinions on this, yeah.

MarySo I started at Headspace at the beginning of January, and by the end of the first week it was just clear to me that everyone said data was bad. They couldn't trust it, but it was just data. All data, all things across this really large and complex business with all these different metrics and use cases. So what did we have to do? We had to come up with some sort of scope quickly that we could iterate from. So I spent my second week proposing what I thought were the P0 metrics for the entire business, and in some ways that was easy.

MaryIt was what we, it was what we were goaling ourselves on across different lines of business for the year. So I said, and I said to the entire leadership team,"These are the most important metrics. We are gonna rebuild the data that powers these metrics," and that can be everything from understanding engagement and subscribers to our B2B kinds of pipelines and the health of our care marketplace, how people are getting to clinical care. And I didn't really open it up for a ton of conversation. I th- I had to draw a line in the sand with defined scope and really give those as those first requirements down to Joseph and say,"Okay, if this is where we're starting, if this is what we have to build, and this is what it's going to be, now what?

MaryNow what?" And Joseph, maybe you wanna talk about what came next.

JosephYeah. So with those requirements laid out, the the, the that's where, that, that was once again a moment and I had spent a lot of time in conversations with longtime Headspacers digging into documentation and design specs from previous attempts. And so it really was a little bit of a like,"Okay, here we go. Are we gonna be able to do it again this-- like, or actually do it this time?" And one of the, the blockers that had really gotten kinda the worst piece at that point of time was traceability, observability, and lineage.

JosephWe didn't have a, a unified picture of our data platform. We suffered quite a bit from what I call tooling bloat. So different orchestrators different-- we were like mid-migration to different transformation platforms and, And what I was able to do, and this is where this is where technology had caught up to, to meet that moment. I was finally able to get a clear end-to-end lineage of our entire platform across all of the different tools and present this back to stakeholders and engineering teams and other folks in the data organization that knew pieces of the platform, but not the entire platform.

JosephAnd collectively, we were able to step back look at what existed, compare that to where we needed to be with Mary's metric definition work, and and really have a, like a good sense of what the gap was end-to-end. And that was such an important piece

ben parkerGreat. And obviously, Mary you hit the ground running straight away and it's your second week of work.

MaryPattern matching now, Ben.

ben parkerSo I guess I guess at what point did AI become part of the solution, and what did it allow you to do differently from, a traditional data transformation?

MaryThis is the exciting part. I know, it's like we buried the lead. Instead of writing all this code by hand, with Joseph's approach as, as the architect, we decided to automatically generate our data model from our central data contracts using AI code development. So instead of building the data model, we generated the data model, which to us was really like a paradigm shift and a really different way of thinking about how to build.

MarySo we can talk a little bit about putting the AI on top to, interpret the model. But Joseph, please talk about how you did this. This was the coolest part.

JosephYeah. I'll I'll do my best. Because it's such a, it's such an iterative process and it collects from so many different areas of the technology industry, the data industry and even some of my own academic background and and some of my experiences in the military working at, working on helicopters. It, it all started with the question inspired by my boss, Marlissa. I borrowed this phrase from her, and it has just tremendously helped my, like agentic development.

JosephAnd the phrase is,"Help me understand." So rather than gen- throwing the problem at AI and saying,"Hey, build this house for me," digging in and h-help me understand this, this data model, help me understand this lineage. What can we do to draw a better picture? So using AI just in, in the environment and saying"Let's build a parser. Let's build a parser for these PySpark notebooks, and let's just get like an ERD out of what these PySpark notebooks look like.

JosephAnd then let's build another parser for the, the dbt catalog, and let's see where these where these models connect. And oh, we've got, some things in Looker over here, so how do those connect?" And building the tools that you need in order to perform the analysis that you need to perform in order to make the decisions that you need to make before writing a plan to actually build it.

JosephSo it's it's breaking down the problem in-into small enough pieces that you would trust an agent to kinda do it o-on its own. And that trust is like a constant conversation or a constant kinda watermark, a kinda up or down, depending on the complexity of the problem that I was trying to solve. So- Iterating through design, leaning on AI to capture decision documentation rather than maintaining like tech specs and and decision records.

JosephPulling in conversations from Slack from two years ago in order to understand a decision of how a table or a, a report was being made. Pulling in Jira tickets from six months ago and adding context to the agentic session in order to inform the, the both why does this exist? Do we still need it? And maybe what could we build better in, in its place? And so my approach to using AI, I think is I think because there's so much conversation around it, it's hard to tell, but I think that it's fundamentally different from how a lot of organizations are looking at using AI

ben parkerSo then Joseph, was it like, was AI part of the original plan or did that sort of evolve through and obviously, I guess you've had other attempts?

JosephYeah. Headspace is one of the most like AI forward companies that I've worked with. But I've been in some organizations that are like,"Hey, it's kinda out there. Experiment with it if you want." I've been in some organizations that have been like,"No, this is not for us'cause we can't trust it." But the like encouragement to experiment with AI led to this natural kinda just spot on my desk where it sits. So I wouldn't say that it was the plan but it was something that I ex- that in a wide range of experiments was the most successful.

JosephIntegrating like agentic co-development into the process led to a really awesome lineage diagram. And then integrating AI into the like planning and documentation piece led to some really interesting experiments. And then finally, when it got to the point where the platform or the models had been built out and the context of the tables, like the descriptions and what they mean was already part and embedded in the documentation for the project.

JosephIt was really just a natural next step to say, hey, what would a Databricks Genie agent be able to do with a like dedicated domain star schema Kimball model? Let's start experimenting with that and see if we can get some interesting answers. What would it be like if if we put Claude on top of all of that as kind of a routing agent?" And so AI wasn't necessarily like Agentic Self-Service wasn't necessarily the goal when we started building, but it became a natural next step as the build became more mature

ben parkerAnd I bet you're glad you done it.

JosephYeah. Yeah. Yeah.

ben parkerCool. So I guess where did you get the most, Yeah, where do you get the most v- value from AI then?

JosephI would say that it's in the, it's in the context that's built as the project develops or unfolds. That's where I personally get the most value. And there's just so much exciting things that, that you can do with proper context. So one of, one of the practices that I've built involves a development journal. I've got a kind of a, a workflow intake, research decision, plan, implement, review as distinct chapters or sections within this, this journal.

JosephAnd whenever I think this is like an... this is interesting, this is worth looking at or digging in a little bit more saving off like I have a skill that I've been layering on over time where it processes these, these journal cycles. And then I've got a branch on a GitHub repo where it collects these, these documents. And r- rather recently started opening this up to the rest of the, the data engineering team, and as other engineers have adopted this practice, I'm able to then have the agent have the team's context available as I'm making architectural decisions.

Joseph"Hey what's my colleague over here doing with like ABAC, and how does that layer in with some decisions I'm making on catalog naming?" And it's it's just really powerful. And there's other examples out there in the industry of how powerful open context can be. The Shopify's got a really good example. Mary, you just sent me an article on, I think, DoorDash with Vera. And that Vera I'll let any listeners go in and read more about that. I I won't try to speak to it here.

JosephBut it's one of the most exciting use cases for AI that I've come across so far.

MaryI think, so we're talking so much about how we used AI to build, but I think the other place where it's really come in, Ben, though, is for users then to ask questions of the data that we built. That's the part that we haven't talked about just yet. We also built a series of Databricks Genie spaces and our own agent like more of a routing agent towards those spaces so that now any user in data, in product, in finance, can also now use like natural language and ask for a number or ask where data came from or how they should interpret it, and that is a really big unlock for us.

MarySo right now we're using that to get to know the new data, continue to interrogate it, make sure we have everything that we need, but that's really gonna be the basis for so much more self-serve going forward too.

JosephI love that word interrogate. It really is an interrogation. It really is.

ben parkerSo obviously we touched on a lot. Obviously, one of your challenges was trust. And so I guess, how do you get to that level of combining AI with human expertise when building a solution? 'Cause that's, sorry that's a hurdle for many to overcome, isn't it?

MaryIt sure is. I don't know if we have a perfect answer, but we can tell you how we're approaching it. I'll let Joseph talk about like evals and how he's looking at some of the success metrics from a technology perspective, but let me talk a little bit about the organizational perspective. Humans still need to be in the loop, and there is like a governance or development life cycle that still requires cross-collaboration, a real understanding of the business, and a real understanding of your customers.

MaryAnd so here's what I mean by that, and here's what we're developing. We're talking a lot about setting up context, right? And the context is really what is what g- it's what is going to allow us to get a good number back out of our data, right? Without context, maybe you get forty-five percent accuracy. You can get close to eighty or ninety percent with the right kind of context built in. But we also need rules, and this is where we need humans to also come into play to help build like more guardrails. We have use cases where it's okay to have eighty or ninety percent accuracy.

MaryMaybe it's like oop, I don't know, a s- a simple metric. That's not okay if we're talking about something like member like outcomes for for their actual health, changes in a depression scale or anxiety scale. That threshold and that rule has to be set with humans, with clinicians in our case. And if we're going to update context or anything does get updated, there does need to be like a review cycle to talk about who adds more files, who adds these rules, what does good enough mean, where do we put like almost like kill rules or stop rules into effect so that people can't get certain pieces of data back out of this?

MaryAnd that's again, it- it's like a committee. That's not fun, and everyone wants to throw AI just on top of everything, but we really firmly believe that not all use cases are the same, and context can be updated. It has to have its own development cycle with real review processes and mechanisms in place that doesn't just live within the data team. Cannot just be Joseph deciding to update this and maybe the maybe different success metrics come back with different levels. That is not something he should be doing in a vacuum.

MaryIt really does depend on what the use case is, how sensitive the data is, and what kinds of decisions you're trying to make. But we're very much trying to open this up to not just a DNA thing. We want finance involved, again, product, and for our case, clinical as well.

ben parkerAnd then so I guess it's the human role, is it constantly changing? I know, like obviously, that with data projects it's it's a constant evolution, isn't it? I guess it's not like software projects, you can deliver it and it's done. With soft- sorry, data and AI it's constant, isn't it?

MaryIt is,'cause the context keeps changing, and the business keeps changing, right? It goes back to before your model, it's... It can be impossible for a data model to keep up as your b- if you have a really growing or thriving or just changing business. And so context itself can get really stale. The people... You can't set your context up once and just say it's done.""It's done. Woof, glad that's over with. Now we can move on." It does have to be reviewed. It, and there does have to be evaluations about when that has changed and what does it mean to roll back if those evals or those thresholds come back and something that just doesn't work for your business.

ben parkerYep, it's like you're constantly chasing your tail.

MaryYou are. You're more productive and you can do more, but there is still work to manage. It doesn't go away. There, there is still gonna be some operational overhead.

ben parkerSo I guess s- for someone that's, outside the project, what did building enterprise data with AI actually mean in practice?

MaryBoy, I-- we're so excited about this. We can-- you can trust your data. This the number that shows up in your portal to tell you if you should roll your experiment out to 100% or not will be the same number and the same data that you get in your weekly dashboard. If you don't like the way your dashboard looks or you need other cuts, now you have an ability to go through an agent and ask for it to look a certain way.

MaryRebuild your own dashboard, rebuild your own piece of data. It-- what it's doing is it's letting more people use data. We're driving more adoption, not less, through AI, more adoption of data into decision-making, into actual workflows, and it's really, it's personalized now. We've never been able to keep up with all of the different asks for dashboards, something as simple as that, right? Everyone wants to see something a slightly different way or has a slightly different view or visualization that they're looking for.

MaryWe don't necessarily have to do that now. People can now use the same data. We're not perpetuating different data across different parts of the org. People are able to build and get what they want and do it on their own timeline. Even if we were able to fulfill every kind of custom request for every single visualization that someone may want across the org there's still a capacity issue and a timeliness issue. And data sometimes is only useful if it is timely.

MaryNow we're really unlocking that capability for such a wider scale than we were ever able to do before.

JosephYeah. And so- I'd say it's from someone outside the project, my, I I like to think of less caveats. There's-- right? There's always caveats on any report, on any dashboard. There's asterisks. There's like,"Hey, this number is slightly different from that number because of XYZ," and it's written down or it's in somebody's head. And with fewer caveats, with fewer exceptions to the rule, I think it just makes the data more understandable, digestible and approachable

ben parkerSo how do, how do you make it, make, So how do you make sure the output is accurate enough for the business to trust them?

JosephI will Mary, this might be somewhere where I can jump in and talk about the eval process.

MaryYes.

JosephYeah. So it foundationally, it isn't that different from typical like testing frameworks or data quality frameworks. Those have to be in place first. So that looks like y- unit testing for transformations, integration testing for for like orchestration and and composability promoting that up through a production UAT process QA analysis on any changes or net new tables.

JosephAll of that is still like step one. Then on top of that, it has to be on top of that bringing in, so I've adopted a agentic evaluation framework pulling from a, a couple different places, but a, a static set of questions or a frozen set of natural language questions that are submitted to the agentic interface or whatever agent is sitting on top of your your platform.

JosephAnd then it's just like a call and response. I really lean into a give and when then kind of framework, and having a, a, a set natural language question, a set of keywords that should or should not be included in a SQL query if the agent is expected to generate their own. Or if the agent is expected to use an existing query, make sure that they're not improvising that. And that can all be captured in in the agentic response. So a, a frozen natural language test set with a known input, known output, and then just running those cycles against those agents to see the outcome.

JosephAnd I try to avoid like a Monte Carlo like hitting it a thousand times to see. That, that can be pretty expensive. Usually 10 times per frozen question per change. So then we- We don't get the feedback from the user. Our first point of feedback isn't from the user. So that's like the foundation of trust. And then there has to be a feedback loop. Being able to... the next step that I'm trying to move towards is being able to intercept users' questions to the agents and evaluating the output, and that's a tricky balance.

JosephThere's like chat privacy and all of the other data sensitivity that we've talked about already. But that would be the, that would be the next step in, in that trust framework. Yeah.

ben parkerOkay. So I wanna talk about obviously adopting it,'cause it is, obviously that's the big thing. But this before, how did it feel once you've got the technology working?

JosephReally good. It felt really good.

ben parkerYeah. Was it just did you call Mary and go,"I've cracked it," or how did it...

Josephwhat- Kinda. Ki- it was kinda like, because building out the platform was the ask. The agentic on top was like the bonus. And so it's really"Hey, are we ready for this? Hey, I've got really something... I got something cool to show you. What do you think? Who else should we talk to? What are next steps?" Yeah, that, that was what the experience was like from my end. I'm not sure, Mary, if you wanna...

MaryJos- Joseph is underplaying it. I think he was so excited, Ben. I started to worry Joseph was getting AI psychosis. I was

Josephworried I was too.

MaryFor the first time I can see the full lineage and I can generate I can generate the data model, the pipelines, et cetera. So it really was mind-boggling and there was this kind of sense of awe because I think we all just deeply internalized that the way that we've built things in the past and how we've thought about these projects just fundamentally changed. Things are different now. We have to build differently. So we're pretty excited about it. In terms of like our users and our internal stakeholders, we are still at the beginning of the journey of getting like much further adoption across the org.

MarySo we ob- we've started with, again, like data forward partners and people with a lot of use- like very clear use cases, and people who have been used to even digging into the old messy data with us as well. So frankly, like that's a story that, that is still being written how to continue to drive this at scale, at least for us in, in our current org.

ben parkerAnd I guess how have you started that adoption? 'Cause obviously, yeah, everyone knows they wanna get A to B, but then obviously the squiggly bit in the middle is the hard bit, isn't it? Change management, dealing, like you, you've touched on it, people. How have you gone about trying to adopt it across the wider organization?

MaryBasically beta users, right? Just like the same way that you- Okay a product manager would release would re- maybe dog food or release like a small test of their product. So we're starting with the same group that we included from the beginning to build a lot of the context. So finance, product, and clinical. They've been part of, we did a, a whole RACI as we got started. They knew that we were starting on this months ago, and these are the first folks that we're really rolling this out to and asking them to use it, and then requesting more feedback so that we can keep iterating like you would with any other product

ben parkerSo I guess what's changed for the business when... Obviously, I know you're going through the adoption bit. I guess what's, for the better people what's what's been their, what's been the change for them?

MaryIt, again, beta group, right? We're just getting started in some ways. But they really can self-serve now. Our, our partners have the expectation that they should be able to self-serve, and they can, and they don't have all the caveats. They don't need all of the other information that they may have gotten in the past as if, one analyst pulled a particular number for them. And they're able to get it on the timeline that they need, again, to make a decision, to understand the efficacy of a new release, to understand their members so that they can prioritize their roadmap.

MaryThey are not stuck in some big, long queue where we try to pull some number and it's not exactly what they needed, or it doesn't match something that they've seen in the past. So I think we're able to deliver in ways that we weren't able to before. Con- maybe, I don't know counterintuitively, I think it's leading to more questions. This isn't now, oh the data team can just kinda take some vacations and, you know-

JosephYeah go to Hawaii. Definitely a change in expectations. Definitely a change in expectations.

MaryIt... Yeah. Yeah. Abso- absolutely. But that's great. That means people are asking more questions and they wanna use the data more, and that's really what we're here to drive.

ben parkerYeah. I think expectations, you always have, and even when ChatGPT come out, obviously people's expectations kept getting more... Even I guess when a new iPhone come out, so first of all, the changes were massive, and then obviously it's, the changes are not so different now, I guess so. Yes I guess it's way life is, so I guess knowing what you know now, what would you do differently if you had to start the project again tomorrow?

JosephThat's a big question. I think I would've I would definitely have started like the journal frame. There there's tools and practices that I've developed along the way that I wish I had started using immediately. The, it like exploring with a graph database the lineage as like an experiment. Now that experiment's proven, and so all of the lessons learned and the proven experiments, I would love to just pick up and put in place six months ago, nine months ago, a year ago.

JosephSo that includes the context management system the graph of the lineage and and this, this concept of being able to decompose a problem into small enough pieces to build a coherent plan and then work agentically to implement that plan. I-- Lots of-- There's like toe dipping I would call it, lots of toe dipping. But at this point, h- if I were able to take those proven experiments shifted earlier in the process, that would be I would love to have that.

ben parkerA classic start yesterday.

JosephYeah. Yep.

ben parkerSo what, what do you think other organizations can learn from your journey if they want to use AI to modernize their own, like, enterprise data?

MaryReally tactically, Joseph and I are talking about putting some of this stuff out and sharing more widely, open source, so we can provide that information. I, I'm just gonna go back to the basics again, though. Whatever you want to do, just agree on your use case first. Even if it's a couple of just top priority metrics, get the owners in a room, agree on the exact definitions or what it is- Absolutely that you need to build. And'cause because the AI, like this can all speed up the engineering, but you might just speed up building the wrong thing again.

MarySo just pl- just make sure you're building the thing that your organization actually needs. Don't do it just because it's fun, just because it's the new thing, just because it's using AI. You have to start with some level of consensus and make sure you're building the right product

ben parkerYeah. You saw that with... when obviously the LLM boom, generative AI, everyone was building chatbots, but then obviously the ROI on that is not obviously gonna benefit the stakeholder too much. So yeah, no, I think obviously that's a good point. Like focus on actually what's the most key importive, im- key important initiative that you need. Then I guess, so for any AI or analytics data leaders listening and wanting to start a similar journey, what is the first thing you would tell them to do?

MaryIt's real- it's truly just what I said. Agree on your use case first. Okay. You have to start with what your org needs. Don't just start building a bunch... Don't just start building a data model. Don't just start, like, throwing a chatbot or AI and context and semantic layers together on top of bad data. Really take the time to focus on what it is that you want this to do. Otherwise, it's just a bunch... it could just turn into a chatbot without a real adoption plan, rollout.

MaryYou don't know who to involve for the level of governance or kind of updates and upkeep that it truly needs to keep the context valuable and useful. So just go back to basics. What is the data product that you need to build? And then figure out how to use these as tools to do it. Don't start with the tools and then figure out where to jam in what you built into your organization.

ben parkerBrilliant. And I guess, finally, so if anyone's got any questions for you, is it best to connect with you on LinkedIn?

MaryYes, LinkedIn would be the best place.

JosephYep, LinkedIn for me. And if enough people are interested I'm- I've got a blog series that has been sitting in my hard drive for a little bit looking at publishing that out.

ben parkerOkay. Brilliant. Cool. Thanks for your time, both. Yeah, and it's been a, obviously, amazing journey. And obviously you get onto the adopter now,

Josephabsolutely, man.

MaryMore to come. More to come. Yeah. But thanks so much, Ben.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. What It Takes to Become a CIO Today

    Nov 10, 202638 minAudio

    In this episode of Data Analytics Chat, we welcome Saadia Khilji, Global Chief Information Officer at Coverys, a medical professional liability insurer that has served the industry for more than 40 years. Based in...

    Credit the people on this episode

  2. Are Companies Wasting Money on Data & AI?

    Nov 3, 202648 minAudio

    In this episode of Data Analytics Chat, we welcome Lior Barak, Author & Founder of Impact Operations. After 15+ years in data organisations, Lior now helps data teams and their leadership treat data and AI as a capital...

    Credit the people on this episode

  3. Your AI Is Live. But Is It Actually Working?

    Oct 27, 202639 minAudio

    In this episode of Data Analytics Chat, we welcome Ryan Stevens, Senior Director of Applied Science at Ramp, the financial technology company helping businesses save time and money across cards, bill pay, expense...

    Credit the people on this episode

  4. Nestlé’s Chief Data & AI Officer on Scaling AI

    Oct 20, 202649 minAudio

    In this episode of Data Analytics Chat, we welcome Jessica Matheron, Chief Data & AI Officer at Nestlé. Jessica brings a rare mix of experience to the role, from a French engineering and mathematics background and a...

    Credit the people on this episode

  5. AI Is Moving Fast. Is Your Data Ready?

    Oct 13, 202646 minAudio

    In this episode of Data Analytics Chat, we welcome Erica Howard, Head of Consumer Data Ownership at TD. Erica has spent 18 years at the bank, starting in the call centre before working on TD's early data governance...

    Credit the people on this episode