Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Agentic DevOps : AI Engineering for Infrastructure

Agentic DevOps : AI Engineering for Infrastructure artwork

AI Hype vs. Reality. Real Stats from Laura Tacho, CTO of DX

Sep 25, 20251 hr 18 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Bret and Nirmal are joined by Laura Tacho, CTO at DX and long-time friend of the show, to discuss AI usage and success in teams adopting AI and Agents to generate code and perform tasks. ๐Ÿ˜‡ My new GitHub Security workshop has launched! A free 2-hour workshop with hands-on labs to harden your repos and your workflows from common supply chain attacks. I'll cover how attackers are getting in, and then we'll lock down a sample repo so you know what needs to be done to protect your code. You'll leave with a deep understanding of risks and mitigations as well as a list of helpful tools to keep your repos safe, including my new "gasa" tool for scanning your repos and orgs. We finally have some real data on this topic, and Laura Tacho in her role at DX, the developer experience company, has been studying AI successes and failures in the industry and has released a framework to measure AI impact in an orgs software lifecycle. Check out the video podcast version here: https://www.youtube.com/watch?v=0G_TWLHkj7U โ˜…Show Linksโ˜… AI Measurement Framework DX Core 4 Framework Applying the Core 4 Framework Creators & Guests Bret Fisher - Host Beth Fisher - Producer Cristi Cotovan - Editor Nirmal Mehta - Host Laura Tacho - Guest (00:00) - Teaser (03:05) - Welcome (04:58) - AI Measurement Framework (07:40) - Distilling Fact from Fiction with AI (27:51) - Skepticism and Adoption of AI Tools (30:29) - Measuring AI Impact on Developer Work (51:39) - Assisted vs Agentic (58:31) - More Gains from Training Humans on AI (01:07:08) - Measuring AI's Impact (01:12:47) - Context Switching (01:16:12) - Navigating AI Hype and Reality You can also support this podcast by subscribing to my YouTube channel and my weekly newsletter at bret.news! Grab the best coupons for my Docker and Kubernetes courses on Udemy. Join my cloud native DevOps community on Discord. Grab some merch at Bret's Loot Box Homepage bretfisher.com โ˜… Support this podcast on Patreon โ˜…

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Laura Tacho: Why is it that a developer can only spend one hour a day writing code? It's that they are in meetings all the time, having interruptions. they are waiting on CI, they're trying to navigate through code bases We see this reflected in the data and when we look at AI time savings right now, which is median, like three hours, 45 minutes per developer per week, and then we look at how much time they could be saving by reducing dev environment toil, by having better documentation, by reducing meetings.

AI doesn't even cover a 10th of what they could be saving if the other parts of the developer experience were also increased. So, speaking to your kind of. Deep seated fear, maybe about like the over promise of AI. I think one of the ways that AI has been overpromised is that it is a silver bullet that is gonna solve so many problems when in fact developer experience is still a way bigger lever for almost every company than AI assisted engineering at least at this point in time.

Bret: This is one of my most important and favorite episodes this year. I'm so excited to let you listen to this. If you are currently using AI or even thinking of using AI in software engineering, you'll wanna hear from our guest on this show. Laura Taco, who's a friend of the show, been on many times. Laura is the CTO of DX, that's the DevEx and dev productivity company where they help organizations measure development team work. And in this episode, we're focusing on her team's work on the gains and losses

of using AI in your software lifecycle. My co-host, Nirmal is back adding his experience from what he's seeing on the topic at AWS and their customers. And if you haven't heard Laura before, she's been on this show many times talking about containers and DevOps, but years back she started focusing on engineering management and improving productivity and measuring outcomes, which happens to be very important when you start to introducing a new workflow to everyone's job like AI is doing.

Her pivotal work at DX has her focusing exclusively on how professional software teams are actually adopting ai, where the snags and rough spots are, and how your team can properly adopt AI today for maximum productivity benefits. Also, a quick reminder to smash that like button and review the podcast in your player. If that's your chosen way to listen or watch, because that's the way to help this episode cut through the hype and noise out there on the internet around AI adoption

so that other engineers might find Laura's breath of fresh air on the subject. It turns out it's all not doom and gloom and we all are probably still gonna have jobs and AI is not as good as they claim it is. That's a spoiler spoiler on there, TLDR, but thanks so much for watching and let's get to the show. Hey, welcome to the Agentic DevOps podcast. I am Bret, I am here with my cohost today, Nirmal Mehta, welcome, Nirmal.

Nirmal Mehta: Thanks for having me back, Bret. I'm Nirmal Mehta. I'm a principal specialist, solution architect and containers tech lead at AWS and These are my opinions and not of my employer, Amazon Web Services. Bret: I'm glad to have you back. And we're, we're completing the trio today. Laura Tacho is back with us. She's been on so many podcast episodes I can't keep track anymore. welcome Laura. I'm glad to have you here. Laura Tacho: Hi, Bret. Hi, Nirmal. It's so nice to have the trio back reunited let's hope the audience can brace themselves for what's about to come.

Nirmal Mehta: You're so kind. Can't wait to get into it. We have such interesting topics to dig into. Bret: Laura. you work at DX. Tell us about DX and what you do there. Laura Tacho: So I am CTO at DX DXs, a developer intelligence platform. That means we are getting data insights qualitative self-reported from developers about developer experience about velocity. Productivity metrics and helping organizations use that data in order to improve, in order to make and validate their AI strategy, in

order to make developer experience better and help engineers get more work done with less friction and less drag across their systems. a very exciting role. I get to look into so many different organizations. Netflix, ADP, Vanguard, Pfizer, you name it. I've had a chance to work with some really exceptional leaders who believe that developer experience is an incredibly important driver, and that is such a nice environment to work in. Bret: So Nirmal and I have got to watch your rise of fame recently around AI

because you have been speaking heavily for months now, if not a year, around, productivity specifically related to AI. Can you tell me, you recently launched something significant. What was that? Laura Tacho: The AI measurement framework. Bret: AI measurement framework. What does that mean, Laura Tacho: What does that mean, Bret? Bret: Does it mean the AI is good or bad? Laura Tacho: yeah. Well, we help you find out. one of the things that I enjoy about my job at DX is that we are not affiliated with any AI vendor or any other kind of vendor for a tool in the SDLC.

What we try to do is get you the data so that you can interpret it and make decisions on what you should do for your developers. because of that, we're in a really unique position to. guide the industry on measuring the impact of AI because I have a front row seat to 400 plus companies and how they're using AI, what's worked, what hasn't worked. I have a bunch of longitudinal data. I've been working on developer productivity metrics, as my sole focus for four or five years now. And then before that, I've been in developer tooling for like, 15 years. before Docker? Before, yeah, before we knew each other.

back in January of last year, we wanted to answer this question of like, how should I measure developer productivity? And that's where the DX Core 4 Framework came out. So I worked on that with Abi Noda, who's the co-founder of, DX. We worked with Nicole Forsgren, other researchers and put together that guidance. Well, it didn't take long before AI really. kind of blew everything up late, you know, late 2024. A lot of companies were adopting it and they were looking and thinking, do we need totally new metrics? Like how do we measure the impact of AI?

How do I know if I'm having a return on investment? How does this all relate to things like DORA Metrics, the DX Core 4? we came up with the AI measurement framework, which is really targeted specifically to measure how and where AI is being used. when you use that together with the DX Core 4, you get a comprehensive picture into, speed, quality, maintainability, developer experience, business impact, all of those factors that AI is supposed to be helping us with. maybe you can find out if it is or if it isn't, and decide what to do next. Bret: Nirmal and I started this podcast at London, which we were

all at London for KubeCon earlier this year, I think in April. And that's when Nirmal and I recorded the first episode of this new podcast. at the time, we were still trying to figure out, I feel like so much has happened since in the last five months. At the time we were still trying to read the tea leaves around how AI is useful or not in the software lifecycle, and where is it most useful? Where should we putting our energy and attention? And that was really one of the major premises for even starting this thing was everyone's talking about, frontier models and dev tooling that

adopts AI and everything's shiny. Everything's flashy, and we were really interested in, where is it actually useful today? Because you can easily just sit down and be overwhelmed with everything. But since then you've come out with this framework, you've been talking over the place, multiple conferences around the truth, and we've started to get some sensational headlines that you talked recently on the Pragmatic Engineer Podcast around everything from 30% of Microsoft Code is AI to.

now we see things of despair, of there's actually negative productivity in certain teams, and it just feels like it's all over the board. can you distill all that down to what you're seeing and what's fact from fiction Laura Tacho: let's rewind. There's a couple things that you said that maybe we can kind of paint the picture of what's reality here. you mentioned one thing I wanna dig into a little bit. You're talking about frontier models and doing everything on the bleeding edge? all of us and our collective gray hair, we were around for containerization. before the show started, we were talking about how some companies were okay

with pressing snooze on Kubernetes. They were okay with pressing snooze on containerization. Maybe they're even okay with pressing snooze on the cloud. And that just doesn't seem to be happening with AI. there's really like a lot of hunger to like adopt it, adopt it now, get the maximum gain. and you know, thinking about that and what you said about the frontier models, it very much felt like what we were doing How long ago was that? 12, 13 years ago. With Docker, that very much was on, on the frontier. And what we found was actually the hardest problems are kind of the most boring problems. It was a lot of lift and shift. It was a lot of replatforming.

It was a lot of just like enterprise legacy. that is the majority of what everyday developers are doing. They're not doing frontier work with models they are trying to navigate and Cut through the spaghetti code in their 10, 20 plus year code base. just showing up to work every day, doing the best they can and enjoying the craft and, trying to help their company hit their business objectives. when we talk about a lot of this, like what's in the news, there is a lot about

the frontier and the bleeding edge. What's really tricky though is there's a big gap between the frontier and the bleeding edge and like the experience of an quote unquote everyday developer and what they do every day in the same way that we saw that in Kubernetes Docker time, there's quite a gap between those things. So that's one thing I didn't totally answer your question. Nirmal Mehta: so, Laura Tacho: a good place to start though. Bret: it. Nirmal Mehta: so there's a difference that you're seeing, right? we have that traditional gardener hype cycle, Like early adopters. and then the late majority and the laggards.

I think you were mentioning that you've not seen some, situation where. The laggards and the late majority are actually pursuing this technology aggressively as if they're early adopters without understanding deeply the value proposition why is that happening? Laura Tacho: I think it usually is, it comes down to money. there is a lot of sensationalization sensational headlines about. What AI can do and its impact on the marketplace.

and its impact on sort of like bottom line P&L on the, the economic situation. And I think that's been a huge driving force for a lot of these companies that previously for any other technology would've been very fine to be in the late adopter, or late majority laggard kind of place. But the promise, the financial gain for AI is just so promising. because it not only helps them operate in a leaner way, but when their competitors also have access to that technology, they will outpace them.

I think that's the real threat when I talk to CTOs and senior. exec teams from companies that previously, would've been in that late majority, is that it's, it's not really about, it's not about cutting headcount. It's not about necessarily cutting costs. It's that their competitors have access to this technology and they're so worried about being outpaced they want to catch up as fast as they can to not lose footing in the marketplace and their competitiveness, because that's the risk. and that is a huge disadvantage if that would come true.

Nirmal Mehta: So that implies that the CTOs and the leaders that you're talking to in companies that do face, real competition, right? if they're representing industries where there's not necessarily that much competitive, pressure, then that, Laura Tacho: Mm-hmm. Nirmal Mehta: that might not be necessarily true and they might be able to hold out a little bit longer and truly see if there's an advantage. Laura Tacho: Yeah, I think I see competitive pressure on two ways. One is like in the marketplace, right? Like capitalism, but the other places in the marketplace for talent.

And that is also a very big concern for these companies that you know, they feel like they're behind on AI, because they're worried that because they're not adopting these tools, they're no longer an interesting place or rewarding place for their developers to work. And that they're gonna have a brain drain people leaving their company and going to work for places that are adopting these tools, embracing them more. I think the market as it is right now isn't very conducive to that happening, but it's definitely something running, in the background of a lot of executives minds about, making it.

attractive for talent to stay reducing attrition risk, that kind of competition in the talent marketplace along with, competition, in the marketplace for their software. Whatever businesses, they're selling. Bret: one of the thoughts I've had early that we've discussed earlier was if there's any marketing win for. Big tech, specifically the AI companies, which is all of big tech now. it's, they convinced us all from average Dev to CIO or CEO, that AI

is going to speed up everything and that they need to adopt it everywhere. we talked earlier where's the precedent? where's the previous precedent of what we're dealing with right now? I struggled to come up with one because when the cloud started, it was one company then Docker came out, it was one company and there were other ones trying to, play around with containers and stuff. But there was this singular focus on a single company. And now we have an entire industry whether it's CIO Magazine or the Pragmatic Engineer, you know, newsletter, like they're all telling

us that these tools are being adopted by everyone else, not us. So we have the feeling of being left out, the feeling of being left behind, and the main reason that everyone believes this is better is because it improves the speed of business. we've all been hypnotized by it and obviously we have people that are and, waiting and all that stuff, but it feels like they are so, a minority as opposed to other trends that it does feel like.

It's this just tsunami of tooling, of solutions, of promises that I just don't think we're really prepared to an industry to like separate the truth from the, fiction and the hype from reality. And we're just not really well equipped right now. I feel for that Laura Tacho: Mm-hmm. I think you're very levelheaded in your approach. you ask a lot of good questions about wait, but what about X, Y, Z? where do you put yourself on that? skeptic versus advocate? I mean, you can be both, right? This isn't mutually exclusive. Like I'm skeptical about a lot of things 'cause I see.

The real data, but also like you two truths, like we can be hopeful about something that will be there in the future, but also skeptical about the current capabilities. how do you reconcile that? Especially 'cause you're steeped in this every single day. Nirmal Mehta: great question and thanks for throwing it back at us. I think, one thing that we've learned, all three of us, in our journey adopting like bleeding edge stuff is, new technology paradigms is to be, to have a healthy dose of skepticism, especially around value proposition around the capabilities of technology, right?

Like early containerization full of sharp edges that have, have now been dulled, Laura Tacho: Mm-hmm. Nirmal Mehta: over the last 15 years. But, spinning up a container was full of, issues back a long time Laura Tacho: Mm-hmm. Nirmal Mehta: and, we all had a healthy dose of skepticism, but we also saw a path to value for folks that were adopting that technology and immediate use cases, right? Like, going from spinning up a VM and waiting an hour, as your chef scripts go through and install apache server, to in two seconds having 20

containers with the same configuration. It's a very direct line from seeing the technology and the value proposition in terms of cost savings or whatever, right? If you wanna follow the money. My healthy dose of skepticism with AI right now is that it seems like we're in a perfect storm of other factors that are going on, and is a lot of pressure there's so much. Money involved, and so much pressure from all sides a tremendous amount

of fog over this technology. on one side, having adopted these tools, I'm privileged enough to have almost unlimited access to a lot of these technologies. And enablement from some of the world's best folks. my colleagues Laura Tacho: Mm-hmm. Nirmal Mehta: create this stuff every day, that use this stuff every day. it's hard to push back against that feeling of magic. When I first started doing some of the code generation with, ChatGPT last year, and then moving over to, Cline earlier this year and doing some

agentic, work with Cline and setting up multi-agent to do operational tasks, to do code generation and prototyping. it does provide value, Like even, helping, come up with, questions to ask guests on a podcast. there's value there. At the same time, we have the ending of zero interest, in the United States economy. We have global economic turmoil. The promise of these tools to, improve productivity in not only

developer, code generation, but all kinds of different avenues. and a kind of, under, under the breath, like, oh, that also means we don't need as many people. And it's, I'm, I am healthily skeptical of what if the technology will actually deliver on like these hyped up, premises, like 60% productivity gains, At the same time, I don't know if that matters, I don't know if the leaders at these

companies will actually care about whether there's value there versus using it as an excuse to execute against other defensive maneuvers to navigate all these other things happening in the world. I end up very confused and, I really appreciate that in your latest talk at lead dev, you have this big slide that said, data beats hype. I think it's not only that data beats hype, which I really appreciate and I would love for you to get into, but We have to present that data as fast

as possible because the hype seems to be accelerating decisions are gonna be made before the data is even there. And I'm really worried that damage is gonna be done maybe a year from now we're like this fever dream of adopting these AI tools didn't come to fruition. But you know, it doesn't really matter because we're all not working anymore. Right? Laura Tacho: Mm-hmm. Nirmal Mehta: Like, does that even Laura Tacho: Yeah, Nirmal Mehta: great, we have data that shows that this is not like, this isn't me, all the things that, all the hype, but it's kind of too late.

I feel like that's where I am. And if I feel that way, I feel like our audience is probably feeling that same way or will be soon. Laura Tacho: I think there's already decisions being made based on hype for example, knowing how much code that's running in production was authored with AI. Is a very important thing to understand when it comes to your security posture. Understanding like changes to SRE, understanding how you might need to change or harden your build and test. it's an important thing for businesses to know, like how much of their

code is actually AI generated code is actually making to production. Because there's different decisions you'll make if it's 0% or if it's 90% Up until three weeks ago, there was no way to know, there was no technical solution. you know, broadly speaking, there were some pockets of technical solution of being able to track acceptance. So a suggestion, like a code suggestion in an IDE through an accepted suggestion. So that's in your IDE through human authored edits, committed open.

PR merged PR in production. There's just no way to do that. Sort of like provenance for code. But still, we see headlines six months ago saying, oh, 30% of our code is being written by software. from my position on the inside, I sit in AI data all day long. This is literally, it's my, it's all I do. It's all I do. It's all I think about. There is no way, there is no way to know that reliably. You can, of course you can do really structured sampling. You can do self-reported data and self-reported data does get us most of the way there.

but that's not what this was. This is coming from, this is coming from like, you know, snippets or an executive estimating in some cases. How much of the code from their teams is being written? Just very sort of unreliable. very lossy. Very lossy. Yeah. Vibe, vibe, metrics. And so, I do really have a lot of empathy for A CEO who does not have a background as a developer and cannot distinguish claims. due to lack of subject matter expertise about what is actually authentic

and what, what is technology really capable of versus what it's not. of course the CEO should be seeking out subject matter expertise, and it is the role of the CTO or VPs, of engineering data, to bring that knowledge into the organization. But I a hundred percent believe there are decisions being made about AI strategy and investment on, like, we can call them vapor metrics. They're just sort of like, or vibe metrics. I like that. Nirmal, it's like vibe metrics. it's about gut feel and not about. measurement, not out of Ill will or malice, but just out of lack of telemetry because this has been accelerating so fast.

can you imagine, having to decide on your budget for Docker spend or cloud spend for the next fiscal year when Docker couldn't even tell you like docker ps, um, how many, how many processes were running or what the size of an image was and you were trying to forecast your, image registry costs that's kind of what this would feel like. which doesn't make sense, right? It's very dangerous. but because of the hype, it's where we find ourself. right now, at least in, in a lot of pocket. It is getting better. I have to say that in the last couple months has gotten a lot of better, but.

Nirmal Mehta: Yeah, just one last comment on that. At the end of the day, data will win, right? data that backs up the value proposition will win. there might be some pain as folks adopt these technologies preemptively to address competitive pressure. your approach at DX and doing the hard work of collecting data, you know, even if it's anecdotal and not the best, quality data at, at this time, it's still the only way that, both leadership and folks that are being

asked to adopt these technologies and, and find those productivity gains or else we'll be able to really navigate, getting something out of these tools. 'cause at the end of the day, I do. unlike some of these other technologies that have come out in the last 10 years. I do see, you know, in, in my own adoption, I do see a path to value. I have received gains from using these tools. So there is something there with respect to the percentage. I think your approach with the AI measurement framework and collecting

that data is gonna be key to figuring out where the actual value is and also changing how, you know, I think there's another, uh, another aspect to this, from a potential developer or platform engineer perspective, I'm seeing a trend of, their leadership telling them, they Laura Tacho: Mm-hmm. Nirmal Mehta: use these tools. And a lot of folks who have tried these tools like maybe once or twice at some point in the past year, didn't work for them, maybe don't really know how to use

it properly or at that time was not the most effective way or effective model. And because things have changed so much. we've gotten so many more best practices. They're, skeptical. Like, why do I have to try using this? Like, I'm being forced to use a tool? and that's never a great position, but data is the way for those folks to push back against leadership and potentially adopt those tools in a way that will actually be productive for them. Laura Tacho: absolutely right. I think, I've seen lots of companies use the measurements that come out of the AI

measurement framework, not just to show the ROI to their leadership, but also to bring that back to the developers to show, oh, you know, you might have been like thinking, sensing, feeling, perceiving this about your own productivity, but here's actually what's happened. There has been a 30% lift in pull requests or merge requests for those using AI daily. you're also reporting that code maintainability is not slipping and that you're still having the same amount of effort to review in AI assisted diff versus one that wasn't AI assisted.

And that can actually be really powerful for companies to have a validation loop and get people interested in maybe giving it another try because it's not hype. It's data. It's not someone saying, oh, this is gonna give you a 10x productivity boost. It's like, well you probably are gonna save a couple hours and it's gonna make it a lot more fun and reduce your cognitive load. So why don't you give it a try? Or we have this new use case that helps you, modernize this old bit to, the new bit. with 70% of the way there, you can just do it with AI and then you only

have to take care of that last 30%. doesn't that sound much better than doing it all by hand? Yeah, maybe I'm gonna give it a try. data is important both for leadership, but also it's very important for developers themselves. Bret: in order to get the advantages you actually need to do more than just add AI to your IDE. it's almost like a spray and pray has been the attitude of, the AI hype cycle as long as you've got AI code generation happening, as long as you just throw AI into your CI, like just, it's somehow going to magically make everything faster. we're now seeing evidence that, it's more than that.

which is true of anything in tech. I don't know why we stopped believing that for a moment 'cause it's always been that way. what is some of the stuff that's happened lately? Laura Tacho: Yeah, let me show you actually, I can just talk you through some of the metrics. We can look at some of the new stories and then I'll show you what's really going on out there. this is a good. Juxtaposition of what we're seeing a lot of open up LinkedIn, it'll take you 30 seconds to find something like this. So we've got sensational headline. I mean, this one's not even that sensational. It's like pretty grounded. It's only saying 10% more productive at Google.

and then talking through their measurement methodology TLDR on that one. They're measuring time saved per developer. 10% more productive is on the low end. We're seeing other reports saying, it's writing 30% of our code, or it's saving our developers 20% of time per week. There's so much variability here because there's so much variability in what actually is being counted when we say AI and developers in the same sentence. For some, we're looking at only code gen. For some we're looking at AI assisted engineering, which can be anything from

refactoring, debugging, production, debugging, lots of other things. Atlassian, their state of, DevEx report, which I'm actually gonna talk about. I think this episode will come out after. but I'm doing a live event to talk through their state of DevEx report. Atlassian has put AI in Jira, Confluence, all of their productivity suite of, project management tools, task management tools. they see a lot of developers saving a lot of time when you add AI to Jira, for example. But that's really different than adding AI to an IDE, you know? And so we have to ask the question, and those of you listening, when

you look at a number, ask the question, what's the boundary? What's the start and stop condition for this metric? what are they actually measuring? who's being measured? Is it, a random sample? Is it customers of a particular company? ask yourself, does that company sell tools with AI? Are they trying to. show me favorable results so that I might be interested in buying their tool. Like there's, there's a lot of questions to ask, like data literacy wise. So over here we've got these headlines about, 10% more productive. this graph here that we see is a study for MITRE, and what they did was take

16 developers and they kind of like followed them around for a period of time. They are working on open source projects and they have a pretty, already pretty good, like moderate to heavy familiarity with AI tools. They're all professional programmers and they tagged some. Tasks as AI allowed and AI not allowed. And so the ones that were AI not allowed, they definitely couldn't use AI on the ones that were AI allowed. They could use it however they wanted to. And they had them kind of do some pre-estimation of how much time do you think this is gonna take you to do this task if you use

AI versus if you don't use AI? And they had them do some kind of predictive, estimations then they observed them actually, screen recordings. They also did some exit interview style kind of things. What they found was that the tasks the developers used AI were actually 16% slower than the ones they didn't use AI for. So even though their perception of the change in time, if AI allowed was positive, the actual result was 16% slower.

And this is very valid data gathered in a rigorous way. It's hard to find 16 developers and follow them around for 240, tasks. this was a very big footprint. Bret: Yeah. Laura Tacho: The the thing though, this is about open source developers, um, and specifically they're talking about open source developers who are working on tasks that are a little bit more like on the bleeding edge on the frontier, where code generation was a huge part of the workload that they were trying to use AI for. That is not very similar to, again, that every day engineer or working

in the enterprise who is not doing bleeding edge frontier kind of work. and so while the MITRE study is very interesting and, and definitely points out, we need more research. When we try to generalize this and say, well, using AI for AI assisted engineering is gonna give your developers a 16% slowdown across the board, irrespective of what type of task, irrespective of type of developer, tenure company. That also oversimplifies to the point of being incorrect as well. and so we're definitely seeing this kind of from both ends.

the skeptics using the MITRE study to say, AI is a bunch of BS and you shouldn't use it to the people on the other side saying, but you know, Google's saving 10% of time. It's just really hard to know and there's not a lot of literature on it because it's so new and we don't have longitudinal data because it's not been around for very long. We've only been able to study this for like two years. It's not like we can look back to the 1970s and look at patterns and see how things have changed over decades. Nirmal Mehta: So you just brought up a really interesting thing, which is the digging into that MITRE study.

It was around a specific use case, code-gen. what in your, framework and when you were talking to your customers or to your survey respondents, Laura Tacho: What Nirmal Mehta: the Laura Tacho: yeah. Nirmal Mehta: outside of code-gen? Laura Tacho: Yeah. Nirmal Mehta: is maybe not the majority of what people are doing every day, right? Laura Tacho: Yeah. And I think that's a really important thing. So I'll also be transparent when we do research. A lot of it, like if you see me post something on LinkedIn. That's most likely aggregated customer data that I have access to. When we publish reports and guides like the one I'm about to show you, we do seek

out external, participants as well to not only include people that are customers of DX because that's, very self-selecting, in a certain profile of company that really cares about DX developer experience. Bret: they're already tip of the spear. I've never been lucky enough to work at a company that had anyone focus exclusively on DX. Laura Tacho: Yeah. Bret: I've always been fascinated in your work because it makes every job that I've had and every contracting client I've had, feel like they're amateurs. when I read the reports and the information that you all put out in

Dora and other, sort of, tip of the spear, what are the elites doing? it always gives me like, what are we even doing? The rest of us are just winging it. we don't have these methodologies. We don't have these pipelines of data that we're pulling out of all of our utilities in order to gather stats to change behavior, which I feel like is a lot of this. It's like, if it's not changing behavior, then is it worth anything? so it's exciting to see some real numbers come out of this Laura Tacho: Yeah. Bret: showing us that, yeah, this is like any new technology. We have to figure out where it works and where it doesn't.

It's not universal, it's not a panacea. so yeah, let's jump in. Laura Tacho: Yeah. you say that so eloquently, Bret, like if it's not changing behavior, is it even working? what's the point? Absolutely, and that's why I get up every day is to like the companies that you work with that don't have this, you know, trying to get them, giving them tools and resources, enablement so that they can have this because you don't need crazy ETL pipelines. to have information like this, you can get so far with self-reported data from a survey that you run in Google forms, it's remarkable how well those methods work,

even at scale, the overhead does get, cumbersome at some point, but you don't need to wait for perfect ETL pipelines. Anyway, we looked at 180 different companies and talked to developers who were saving the most time with AI. I just asked them like, what, what are you doing? stack trace analysis and refactoring existing code were the top two use cases that we're saving the most time mid loop generation. So like, code gen didn't come in until number three. And then we have test case generation that's also up there.

Like those top four are kind of the biggest time savers. Actually, I think this is from AWS Nirmal. We talked about this at KubeCon when we were in London. The average developer at AWS only spends how much time of their week coding? it's double digits but it's in the low double digits. that's very consistent with what we see across the industry that developers are spending about 25, 20 ish to 30% of their time actually coding. a lot of it is planning, doing maintenance work, doing other stuff. So, applying AI to other tasks, yeah.

Nirmal Mehta: yeah, I did find it. AWS published in December 3rd, 2024, a post that said that Amazon developers, spend most of their time on tedious undifferentiated tasks, such as learning code bases, writing and reviewing documentation, testing, managing deployments, troubleshooting issues, or finding and fixing vulnerabilities. the stat from the article is no longer in that post, but it was, developers reporting, spending an average of just one hour per day. So this was at reinvent 2024 in the

Bret: Yeah. Nirmal Mehta: by Matt Garmin, where he said that they were spending on average one hour a day on actual code, development Laura Tacho: absolutely. Yeah. Bret: lines. Nirmal Mehta: Yeah, Laura Tacho: that was talking about what are some of the use cases that can bring the most gain? I wanna show you this other thing, which is really, this is like aggregated, kind of dummy data. This isn't data from any customer, so I would never do that. but the spirit of this is accurate. and I've actually seen way more pronounced versions of this, but this is looking in DX.

So, we do quite a lot of, visualizations to help you analyze the data in easier ways. But when you look at. Why is it that a developer can only spend one hour a day writing code? It's that they are in meetings all the time, having interruptions. they are waiting on CI, they're trying to navigate through code bases Nirmal, like literally exactly what you just said. We see this reflected in the data and way down here. it's not always this low for every company, but when we look at AI time savings right now, which is median, like three hours, 45 minutes per developer

per week, and then we look at how much time they could be saving by reducing dev environment toil, by having better documentation, by reducing meetings. AI doesn't even cover a 10th of what they could be saving if the other parts of the developer experience were also increased. I think speaking to your kind of. Deep seated fear, maybe about like the over promise of AI. I think one of the ways that AI has been overpromised is that it is a silver bullet that is gonna solve many problems when in fact developer experience is

still a way bigger lever for almost every company than AI assisted engineering at least at this point in time. Nirmal Mehta: That's super interesting. So another way to interpret that data, from what I'm looking at is that, the traditional DevOps, like the DevOps maturity of the organization, CI/CD automation becomes even more. It was already critical Laura Tacho: Mm-hmm. Nirmal Mehta: competitive Bret: Mm-hmm. Nirmal Mehta: and value this AI world get the value out of these AI adoption.

It becomes even more critical. you have to have that automation in place. and, that's what I'm seeing here, right? CI, wait time, deployment, lead time, those are, that's just pure CI/CD, right? Laura Tacho: Yeah. Nirmal Mehta: automation. so that, that tooling needs to be in place. So you think that, folks that are working on are, are, are pushing for AI, tooling adoption with their developers? net benefit is that, they'll, they'll drag themselves into modern DevOps

because they need to do that first, to create the foundation for getting the value out of adopting these AI tools. Is that a fair Laura Tacho: Yeah. Nirmal Mehta: See that Laura Tacho: that is a really fair way to see that. I posted about this before. Here's, a post from Abi. I think what you're describing Nirmal is that like DevEx is AgentEx. I think there's, you know, there's some subtleties. I think what you're saying is, maybe by using AI these companies can drag themselves into the future, which is great because the reality is that a lot of what will benefit.

Agentic models, agentic workflows, good documentation, organized code, fast feedback loops, clear project requirements. Those are things that also benefit people. And it's too bad that a lot of companies are only willing to invest in those things for a robot and not for the people that were there before. But our opportunity now is that wallets are open and the end result is the same. And so if the robot and pleasing the robot is what's gonna get companies to

spend money on these things that are existential and critical and important, then you know what, I'm fine with that. Let's keep the wallets open and let's keep moving into the future. Bret: The reason is less important than the outcome. Laura Tacho: Yeah. Bret: uh, mean, to be fair, we weren't all pitching better documentation as a business productivity boost. that's never how I positioned it or any other developer I worked with. Laura Tacho: Yeah. Bret: again, like we're going back to this, that the hype cycle is promising AI will make business faster, it makes us write code faster, supposedly.

Therefore, business goals will be met faster. And I mean, this podcast, we, we actually like to think of this podcast as focusing on AI after the commit, where so many other podcasts are focused on before the commit. That's why we don't talk about which model we're using. And if we do Laura Tacho: Yeah. Bret: dev tooling, we're usually just talking about it because we're all into it. It's fun, not because it's the focus of an episode, we've had, other guests on this show and our, and Nirmal and i's other show, DevOps and Docker talk.

We've had multiple shows now over the last four, eight months talking about the idea that AI is going to improve. the standardization and expectations of teams when it comes to the rigor of everything from documentation to automation, to standardizing on infrastructure as code, like all the things that we've all been focusing on for over a decade Laura Tacho: Mm-hmm. Bret: to others and share our opinions. But we didn't necessarily go at it with everyone knowing that, this is gonna save

us money, this is gonna save us time. And I feel like, you're right, that AI, I guess the negative is it's hyped, but the positive is we might be able to use that hype to actually get business to move on some of these things that we've been trying for so long. Because if I go to the team and say, Laura's new project shows, oh, look at this new talk for Laura. It shows the stats that say, if we have better confluence, if we have better in repo documentation, if we have better project planning that's linked into other things, our AIs will actually perform better for us and save us money.

Maybe that's what will actually move the target. So you heard it Laura Tacho: Yeah. Bret: just go spin that to your leadership. maybe you can get more, AI tooling and more of the, stuff that you actually wanted to work on. we need more documentation, but I don't know anybody that's clamoring to make more. So I'm excited that AI might help me write it better and keep Laura Tacho: Yeah, yeah, Bret: find the outdated stuff better. there's many opportunities for improving it. Laura Tacho: yeah. I think documentation is a really good use case for AI in a way that saves.

Time as a second order consequence, like a second order indirect effect. It's harder to estimate because, the reason that documentation has never been, it's always been like a nice to have, right? It's been a vitamin. not a painkiller. And I think that's been the experience in general of developer experience It's something you do to take care of yourself because you're good. It's not like my arm is cut off and need to bandage it. in reality that's what developer experience is. It's that your arm is cut off because on average, developers waste 20% of

their time each week because they can't find the right documentation. They're waiting for builds to finish, they're waiting for all this stuff. 20% of time, if you can imagine a roofer putting shingles on your house, throwing one out of every five shingles down. to the ground or one out of five nails did not work. That company would go out of business and no business leader would tolerate it. But because it's hidden in knowledge work and behind developers and we can kind of brute force our way through it, we don't know the boundaries of the problem. And that makes it squishy.

It makes it seem like a vitamin of oh, give the developers nicer documentation. They'll like that when actually, if you were to measure and, and use some, you know, measurements like we, we do in DX and try to identify how many times per week does a developer lose 30 minutes or more because they can't find information due to outdated or non-existent documentation. look at that number now. That number is your arm cut off? Right? That's not a vitamin anymore. there is a bit of marketing around developer experience and a lot of talking about what it is. 'cause it's not ping pong and beer.

This is like existential problem. And I know that we all know this and that's why we have been working on DevOps and developer tooling because we know the real business impact of it, but it has been historically really hard to tie that to Dollars and cents, which is what the business needs in order to keep their wallets open. I think AI, if anything, as you said, it's overhyped or, hyped the right amount. I don't know how much hype is the right amount of hype, but, people sure have their wallets open and maybe now's the time to, take advantage of that.

Bret: I was just gonna reiterate and repeat a little bit back to Laura, that, in my entire career, the three things on every project that get cut first, and I always like to say this in project meetings when we're all about to cut something, it's documentation monitoring and disaster recovery testing. Like those are the, or just testing. Those are the three things I see on every project that left, on the cutting room floor when the budgets are tight or the timeline didn't get met. if AI could improve, just those three things, Documentation, monitoring and testing. If we could just improve those three things, I feel like a bigger portion of

that graph that you were showing, and for those that weren't seeing it, it was a line graph of all the different places that devs spend their time. And at the very bottom code generation. Like the very bottom of it was ba pretty Laura Tacho: Yeah. Nirmal Mehta: so interesting. It's Laura Tacho: I. Nirmal Mehta: like a, an outcome of this data is that, if you're out there, if you're listening to the show and you're a dev or a platform engineer, engineering team, and you're in your sprint, retro, and you're looking at your backlog, if you scroll all the way down, there's probably a line that says, improve documentation

that has never made its way up. Or, you know, refactor this critical little piece of code Bret: Yep. Nirmal Mehta: improves the speed of CI/CD Bret: Increased test coverage. Nirmal Mehta: into, or, one of the monitoring ones. There's probably these, These user stories that have been on the backlog at the bottom You just gotta scroll all the way down. It seems like you, and, and if your organization, if you're sitting there and you're sprint retro and you're figuring out, you're going through your backlog and planning for your next sprint, and your organization is asking you to adopt AI tools, to, show an outcome of 30% plus productivity

gains with using these tools, right? Regardless of how you're gonna use the tool, you might take this opportunity to take a look at those items on your backlog and push 'em up into the new sprint because you can now say that those are tied with this data. You can show that those, backlog items, those forever backlog items are valuable in actually seeing those productivity gains from these AI tools. Is that the right interpretation? is that what you're saying, Laura?

Laura Tacho: I think that is, a really good interpretation. I think where things get tricky is building the business case around them, Nirmal Mehta: Okay. Laura Tacho: it's, really been difficult for developers to say, sorry, we're not gonna do. X so that we can work on our documentation increase test coverage build out different test cases, work on our CI to reduce flaky tests or make it run 15% faster. It's been really difficult 'cause this shouldn't be extra work, right?

we don't wanna overload devs with extra work. We need to take something away. That's where things get really tricky because when we have a feature that we know is gonna bring in, you know, we have a, it's $500,000, let's just say that I can't hold up documentation improvements or, or CI/CD improvements next to $500,000 and make it as close to an apple's to apple's comparison. It's so difficult to do that without data because we can't.

conceptualize how much money that is costing our company. And so often what we end up is like, sure, that feature might bring $500,000 of new revenue in the door, but this is gonna allow you to accelerate delivery for everything else in the future by 20%. And how much is that worth in terms of salary, cost avoidance, and time to market realization for your future investments. we can do the math and figure that out. but that's where platform engineering teams, where software development teams, like app development teams are having the hardest time right

now is making those arguments. And that's where DX, I mean that's our really, our backbone is like drawing the boundaries around how much it's costing you to have these problems. Helping you figure out how to prioritize them so that you can reduce the friction and stop wasting 20% of your time every week. 'cause that adds up really fast and it's just not fun. So, why don't we, why don't we talk more about AI stuff? Nirmal Mehta: Yes, assistive versus agentic was another thing, but Bret, go ahead. Bret: Yep, So

Laura Tacho: Um, Bret: agentic. What does that even mean? Laura Tacho: what does that even mean? So I think that. When we talk about AI or like when AI is talked about in general, it's kind of like this big pool AI is such a broad term. even thinking about those use cases that I talked about AI for code gen versus AI for refactoring code versus AI for running after the commit pipelines, that's already a huge variation. And then when we talk about the interaction modalities of AI, we've got, am I talking to ChatGPT in a separate window asking it like, oh,

how would you plan, this feature? Like what are the questions I'm missing If I'm gonna do a pen test on this? Like what are some edge cases that I might have missed versus code completion in the IDE versus agentic workflows that need human approval versus ones that are running in swarm mode with many agents doing work. AI is such a huge term. And so when we talk about The shift in the spectrum from assisted to agentic. We're talking about human in the loop ness, maybe I would call it like that. with assisted engineering, we've got chat modality, we've

got auto complete in the IDE. as we get closer to agentic, maybe we're relying more on workflows that are defined, and running, you have an agent doing each of the workflow for you, but you're still saying, yes, do this and verifying and then saying proceed and verifying and proceeding very much human in the loop versus like a totally agentic workflow, which is like, I need an app that is going to analyze my Pokemon card collection tell me which one has the highest market

value and send me a push notification every morning with the highest value card, build it and deploy it to AWS. And you don't look, you know, you're not in the loop at all. that's a very big spectrum versus like pair programming with ChatGPT to giving something production access. So as an industry, if I were to imagine all the companies I'm working with and see what they're doing and put them on a scatterplot, we're definitely more toward the assisted with a little bit of dabbling in the agentic.

But I would say more like workflow is starting to be that early adopter, early majority kind of bleeding edge. I've seen some interesting examples of agentic, but nowhere near industry wide adoption of ag agentic models, or patterns of working right now Bret: That Laura Tacho: September 1st, 2025. Bret: yeah, Nirmal Mehta: because Bret constantly is asking me, but Nirmal, our, our companies and customers actually using these agents to do like SRE tasks or operations or DevOps. Like are they, where's the actual use cases?

And I, Bret: me the money is, yeah. Show me Laura Tacho: Show me the money. Nirmal Mehta: I'm gonna deflect Bret's question that he's constantly asking me to you, Laura, I know you just mentioned that, on that scatter plot. it's definitely not the majority, but are you seeing some early signs of, agentic workflows, not only for business use cases, but also for DevOps, tasks like operations, SRE, CI/CD, things like that Laura Tacho: yes. I would say as a short answer, yes.

I think the complexity is the variable influencing, how and to what extent that's being used. looking at SWE-bench, this hasn't changed since I, looked at it recently, but we've got 67% of tasks resolved and a lot of these models are hovering in like the 55%. So like the technology is also just like not. There. So this is, for those of you that don't know about this leaderboard, this is like testing the model's capability to complete a task end to end and have it be correct a coding task.

Bret: Yeah. Nirmal Mehta: one Laura Tacho: um, Nirmal Mehta: or is it through a workflow Bret: It Laura Tacho: um, Ooh. Bret: tool. if you look at the list, each one might be like. Just this model or this model plus some additional tooling that turns it, like warp Laura Tacho: Yeah. Bret: You know that their agent model that's using a particular LLM like sonnet four gets this percentage. So just to clarify, when Laura says 70%, which she's saying is that that particular setup, like that LLM plus that tool

Laura Tacho: Mm-hmm. Bret: 70% success rate on hundreds, potentially of GitHub issues that it's trying to solve for a particular software project. And that's what we've talked about before we've actually mentioned this is I'm so glad you brought it up. Because Laura Tacho: Mm-hmm. Bret: best models are only getting three out of four or two out of three PRs Laura Tacho: yeah. Bret: Correct. Laura Tacho: Yeah. Bret: it's, it's not fixing. And so, and that, and a lot of us. we are all very lucky. We have access to the best models, to the best tooling.

we live in a western country where we can afford all these things. I do hear on occasion feedback that a lot of the world isn't here yet. A lot of the world has to use either free models or local models, or free stuff like the free ChatGPT equivalent or whatever tokens they get for free with GitHub Copilot. And they don't necessarily have the best tools unless their company buys them because they're just so expensive. Unless you live in a, in a country that is indexed to the US dollar, essentially Laura Tacho: So I think the technical capability isn't really there for organizations to rely on agentic workflows for things

that are gonna touch production. Have I seen lots of experimentation with agentic workflows for stuff used internally and doesn't necessarily hit end users Absolutely, definitely. there's a lot of experimentation, but when we talk about it widespread, especially in the enterprise, we're still away ways away, from that being reality for, for a lot of folks, I would say like one step back toward assist, you know, one step away from Ag agentic, which is like workflows. I have seen many kind of enterprise, fast scaling, high growth organizations

have their platform team responsible for creating workflows that, for example, do legacy modernization or dependency updates and make those available to other development teams, their customer developers internally as part of their platform offering. So this is not quite agentic. There's still more human in the loop, there's more human verification. but companies that are really investing in AI have done a lot of training and enablement to. Get all of their developers with not just an understanding of like how

the, the tools are used, but like organizational understanding of how to apply it to their business context. they are experimenting with, you know, agentic adjacent, we'll call it that, agentic adjacent, workflows. that's exciting and promising. I think, by the end of the year, maybe in Q1 of next year, my answer will likely be very different. Nirmal Mehta: Well, Bret: Yeah. Nirmal Mehta: have to bring you back to give us an update on what you've seen. so Bret: a moving target Laura Tacho: I wanna share just one, you know, kind of thinking about how AI can't make up for all of the devex gains.

I think agentic has a lot of promises, but there's still so much work to do when it comes to identifying organizational use cases for AI to help deliver software faster. This is, um, I'm gonna publish this I think this week actually. Um, this is looking at data from over a hundred companies ish that have developer populations between 105 hundred. And I'm looking at their annualized cost savings. these companies have said, this is what a full-time employee costs. This is how much our developers are using and saving from AI each week.

And so this is the annualized cost savings in US dollar from using AI. obviously there's gonna be a trend line. The more developers you have, the more you can save because math. but even if you look at some of these like. 200 developer, 300 developer mark. We've got companies saving over 6 million, in recovered salary costs. And like, you know, this company not that much of a different size. It's like not even breaking a million. It's not that these companies up here are using Ag agentic

workflows to do a bunch of stuff. It's that they have done really good training and enablement. They've found organizational use cases. they've equipped their organizations with AI instead of spraying and praying to their individuals and hoping that curiosity and grit will take it the rest of the way. there's so much room for optimization all around. Ag agentic workloads are incredibly interesting and I'm very optimistic. the reality is that I just don't see them being widely adopted, especially in customer facing environments and in the enterprise just yet.

But again, my answer will probably change. Nirmal Mehta: So this is interesting. Is Bret: Yeah. Nirmal Mehta: normalized based on, the developer's salaries? Laura Tacho: the companies that I've included in here have given us information about how much a full-time developer at their median salary point. this is adjusted. I have another version of this, which I'll publish along with this, where I pivot basically on region. and you can see like the, the lower bands there's a lot of Europe down here. 'cause the salaries in Europe are just generally not as high.

these are US companies and then like Asia companies are kind of spattered, across, but that seems to be like a pretty in aggregate, like a representative trend, for what we're seeing, also extending to bigger companies Nirmal Mehta: I would love to see that pivot. on the left hand side, the annual cost savings. per, normalized, developer salary or something like that, like a little bit normal normalized, just so that it's easier to compare. this is fascinating. Laura Tacho: yeah. Nirmal Mehta: information. so what are some of the, you kind of mentioned, I mean these

Bret: It Nirmal Mehta: are actual, this is like real money that we're seeing, right? Bret: Yeah. And if I had to describe this to the audio listeners, looking at a graph that is. all over the map. there is nothing here that is certain, there is no clear location where like, this is where everybody, like the majority of people are seeing this level of Laura Tacho: Yeah. Bret: It's basically every company of like this size you said, right? 100 to 500 engineers, I think is what you Laura Tacho: Mm-hmm. Bret: and they're all over the map. I mean, as you can might have guess the ones that are saving the most money are rare.

there definitely seems to be more concentration on the low end of benefit, but there is no clear cluster here yet. this doesn't look like a typical sort of Gartner, approach where everyone's huddled in one place, I think this is just indicative of any new technology that we don't really know how to use yet. We don't really know the patterns of Laura Tacho: Mm-hmm. Bret: know a lot of people are trying, and you might be wasting your time. You might not. Laura Tacho: Absolutely. that's exactly how I interpret this. And I use this to set up, like it's, just look at the, for the audio

listeners, we've got on the X axis engineering organization size and on the Y axis annualized cost savings in USD, but only for companies from a hundred to 500 engineers. So if we look at companies that have. 200 to 250 engineers, We've got all the way from, five and a half million down to 800,000 for the same exact size of engineers. And so what that tells us is I mean, maturity, we can look at how long has AI been introduced, but what it comes down to in my experience, kind of like anecdotally and I'll put together the dashboards to prove this. 'cause I do have data to prove it as well, but it is developer experience and

it is training and enablement technique. if a company makes licenses available to engineers and says, okay, go for, it encourages people. Even make a Slack channel, share your good practices that will not bring as much, effectiveness as actually Nirmal borrowing, like your experience-based accelerator, like Amazon's kind of method of bring a real business problem and then bring the technology together and like, let's figure out how to solve the real business problem as a team. And then create a blueprint for that and spread that out.

That has been incredibly useful. booking.com, for example, they have 3,500 engineers. they have a 30% increase in merger requests, throughput, but they were able with that technique of the experience-based accelerator doing modernization projects and getting 70 to 80% of the way there with AI using that technique from Amazon. the median adoption for developers in an organization is a little over 50% of developers using AI on a daily or weekly basis for AI assisted engineering.

booking has 60 plus percent using it on daily and weekly, and of those 70% are using it on a daily basis. So they've been extremely successful. They have a huge ROI, it comes from the training and enable. and treating it as an organizational tool and focusing on the workflows that AI helps to enable and not just an individual productivity tool where Nirmal can type twice as fast Now. Nirmal Mehta: Interesting. So this mirrors the same kind of conversations I was having with

organizations two decades ago, or a little under two decades ago, Bret: Hmm. Nirmal Mehta: With the adoption of DevOps practices, organizations back then, I think Target famously had like a dojo, a devops dojo where they would bring teams in with these um devops. Subject matter experts take a workload, and re-architect it and train the trainer and get the developers and the platform engineers up to speed

around that specific workload. migrate it, modernize it, and then. Put it through production and then take the next team. And so what you're saying is this is no different, right? AI adoption, developer experience with, AI tooling, organization, if they want to be successful with these tools and really see some gains, they need to take the same approach, of standing up some kind of, outcome oriented, what do you call it? Like a, uh, dojo if you want.

We could probably call it dojo, like an AI dojo, for adopting these tools to be successful and not just say, oh, here is an X, Y, Z license. report back to me in a quarter where, you know, how you've been 30% more productive regardless of Laura Tacho: Here's Nirmal Mehta: doing. Laura Tacho: the bottom line and maybe a good, um, bottom line to, wrap our conversation up, which is if organizations wanna have organization-wide impact increased productivity across the board, they need to treat AI as an organizational tool, not as an individual productivity speed up.

And the organizations that do that, that treat it, like you said, like have an AI dojo or treat it with, you know, enablement in the same way you would with, with any other kind of organization-wide tool, they're seeing big impact. And the ones that are just spraying and praying licenses aren't gonna see that, that impact because it's, there's nothing like, there's nothing new under the sun. Right. this is a tool. As any other developer tool, no one would've bought. Jenkins licenses are like Docker licenses and just expect like, oh, we're

gonna modernize our whole application. It's gonna run in Docker now. here's a license. Go figure it out That didn't work then. It doesn't work with AI now. even though it's sparkly and shiny and there's a lot of hype, like the fundamentals in the physics haven't changed. and I think there's a lot of level-headed leaders, like both of you, Bret and Nirmal, treating this with the right amount of scrutiny and skepticism and optimism. leaders who can do that are gonna be the ones that come out ahead than the ones who are, you know, spraying and praying and following victim to the hype, unfortunately.

Bret: So you said something about two or three weeks ago, what happened? Two or three weeks ago. Laura Tacho: this has been a core concern for so many of our customers and so many like CTOs and VPs. 'cause again, what decisions would you make if you knew that 60% of your developers were using AI for, for AI assisted coding, but 0% of that code made it to production versus if you had 10% of your developers using. AI tools daily, weekly, and 50% of the code going to production was written by AI. those are two wildly different scenarios, and it's important

to figure out where you are. we at DX have a tool that runs on the file system layer so it can track in any IDE in the terminal, whatever, code completion from AI tools as well as copy paste and then track the human editing and then do that on a per line basis, at the commit level so that we can look at the ratio. Of AI code to human authored code in a given commit, and then we can track that commit all the way to production. So there's like pockets of tools that could do this before, like Windsurf had

some pretty good telemetry into what was going on, but before we just sort of had acceptance rate, which is, you know, accepted suggestions and really no insight into what happens even like before the commit, after the acceptance, and then definitely after the commit and into customer facing environment. So. excited about this. It's technically very cool. Bret: Is this. Nirmal Mehta: scary to me. Laura Tacho: industry Nirmal Mehta: That, Bret: Is this a part of DX? is this a part of the DX platform, or is this an open source project?

what is this specifically? Laura Tacho: it's proprietary, Bret: Okay. Laura Tacho: the DX Nirmal Mehta: sorry, that shot should not have been my first reaction to that, it sounds really cool, but it does. So the scary part to me is it sounds very, Laura Tacho: Brothery. Nirmal Mehta: yeah, like survey and, you know, we could have another whole entire episode on this, but this kind of comes back to, incentives by leadership to use this hype for ulterior motives, let's just say that. Laura Tacho: Yeah. Nirmal Mehta: I think it is important for us to know how much of that AI

code is out there in production. I think that's a very legitimate concern on all aspects, especially from a platform Bret: yeah, Nirmal Mehta: and a DevOps team perspective. I think very crucial information, but at the Bret: to be fair, Nirmal Mehta: for monitoring copy and paste from a developer like, oh boy, you know. Laura Tacho: Yeah. Bret: Yeah. I mean, it's not actually uploading the copy, right? we're not talking security concerns. We're just talking. Laura Tacho: no, Bret: Because this is, this sounds way better to me than a decade ago when I was freelancing through certain websites on the internet to random

companies when I was first starting my freelancing business 15 years ago. back then you were billing per hour and the tool you were using for consulting would screenshot your computer every five to seven minutes randomly and send it to the client to prove you were working on their tool because you were billing per hour. That's way more invasive than this. I had to put up with that 15 years ago. But, anyway, this sounds a lot better. Laura Tacho: Well, again, like the double in the details, so change failure rate is one of the four key DORA metrics that organizations have been tracking for a decade. in order to get that number.

We ask a question to individual developers, this is how it's done for the Dora report as well. And then accelerate, you know, in the last month or whatever timeframe, how many of the changes you pushed to production resulted in degraded service. so we can have a data point for the individual and how many of their changes resulted in degraded service, but we don't look at them on an individual basis. We look at them in aggregate. the point is about measuring the system, not about measuring the individual. Nirmal Mehta: Mm-hmm. Laura Tacho: And the same is true for these AI tools. Like we're not. Looking at, Bret had 70% AI authored code and 30% where Nirmal had 10%

AI authored and 90% human authored. We're looking at it in aggregate to figure out what's the shape of the code that's running in production? What's the security risk? How is AI being adopted? Is it being adopted by, you know, like bearded men are using AI a lot more, you know, like, different attributes and demographics. Unfortunately, I have to tell the audience out there, DX does not have the attribute of bearded men. Bret: it's not a checkbox. Laura Tacho: actually that's not true. We can do any custom attribute. So, Bret AI Bret: Okay, Laura Tacho: if we were on our own team, I would definitely

have that as an attribute for Bret: great. Gray beards. I want the gray beard checkbox. Yeah. Laura Tacho: totally. but it's all, the same threats and the same kind of concern about Big Brother surveillance. measuring the system versus measuring the individual. They all still exist. And even more so with AI because the, the threat of like, my job is gonna be at risk because this technology is gonna replace me. I feel like we're in a pressure cooker and there's pressure coming from all Nirmal Mehta: Yeah, that's Bret: Yeah. Nirmal Mehta: I feel like we're in a pressure cooker right now.

Laura Tacho: Yeah. Bret: Yeah. And so if the audience can feel anything, we're with you, we feel the pressure, you're probably listening to this podcast 'cause you probably have a little bit of that pressure that you're feeling to stay up to date and be on the leading edge. part of it's curiosity and passion around that new tech. Part of it's the obligation that we feel like our jobs are somehow maybe at risk someday. I mean, there's so much confusion and news about everything. I go up and down in emotions every day around AI. it's probably generated more frustration and emotion in me than

anything in recent memory that I can think of in terms of tech. Laura Tacho: Mm-hmm. Bret: So, given all that we've talked about what people can do from here, obviously it's a complicated answer. I feel like what we're saying is, really anything in your software life cycle potentially could be beneficial with AI. But you need to make it a project and you need to take it seriously and you need to train people and do all that stuff. I feel like that's the big take. But I wanted to switch a little bit to a slightly adjacent topic that we haven't discussed yet. Context switching.

one of the potential hopes I have and I think others have for AI is that it may actually help developers with all this context switching. And you showed a graphic earlier of like the top 15 activities a developer does on a given week. Everything from meetings to staring at Jira to improving documentation to, oh yeah, I also get the code right? Laura Tacho: Mm-hmm. Bret: all of those context switches. Nevermind the idea of switching between one code base and another. Um, do you have anything on con?

Is there, are there, are there lessons learned from this? Is there tools we can look at? Because it's one of my top goals to improve my ability to context switch with AI. I don't have any good tools. don't have any good methods or any good information on it. So I was hoping we could talk about that. Laura Tacho: I wanna, just take this time to highlight that focus time as a driver of developer experience is capturing exactly that, like cognitive load context switching kind of tax that developers pay from like bopping around from different tasks, but then also having their flow interrupted,

like, you know, I'm coding and then. I have to go to a meeting. We don't get enough focus time to really get into the flow state. And so that is one of the kind of hypotheses is that if we use AI and we can automate the toil that's interruptive, do developers actually get more focused time? And I wish I had like a more conclusive answer because I did look at this very recently and like the answers are a bit all over the place right now. Even controlling for. usage and different things. we track it as, a first level, primary metric on this AI, impact board

where we can do group comparison. So if we wanna compare like all AI users to non-AI users, we can look at how that impacts their focus time, because that's really important in this case. the group that's using AI actually has a lower amount of, Focus time, which isn't Bret: Rent. Laura Tacho: isn't great, but we can see how that trend evolves, across like the different usage of like infrequent monthly to weekly to daily and how the focus time is impacted because that is, again, like the mechanics and the physics of.

Good devex and like good team habits, like focus time is still important even when we have AI. And a lot of companies that I work with do have the hypothesis that AI can automate, toil give more time for focus and increase the innovation ratio. So this percentage of, time spent on feature development, we see this, kind of sample group has 4.8% higher allocation toward feature development. So that's, that's great. Their innovation ratio, we're looking at things like quality, speed, velocity, maintainability, and kind of making sure all those things stay moving together so

that we don't over index on one metric. use AI to preserve context, I leave little notes for myself in chat. GPT. And within my five hour Claude code, session, I find it so helpful to not feel the pressure to have to write a bunch of post-its to myself. Like I can just Mm. from my task and come back to it. And I've like really accelerated the time it takes to get back into things just because I have this like programmer who never gets tired and is always there for me, except after five hours when he magically goes away.

Nirmal Mehta: Okay. as we're wrapping up, for our listeners, we have individuals trying to navigate these rapid changes, that pressure cooker kind of feeling. what's some advice you would give our audience, that they could use today to navigate that? Laura Tacho: I think first thing is sort of a mindset. Switch, like average median time savings per engineer per week from using AI tools for AI assisted engineering is a little under four hours.

that data comes from more than 38,000 developers. When you see a headline, know that that's a headline for a reason and like get curious about what might be going on behind it. So think that's step one is just to like identify what hype have a better. idea of identifying what type. I think the other thing is, it's. Really difficult to make an impact with AI if you can't measure it. you know, if you don't know what's actually happening. And so getting instrumentation in place for measuring things like maintainability,

DevEx, speed, throughput, quality, these can be done lightweight with a survey. it doesn't need to be this huge robust ETL process and data lakes everywhere. You can do so much for getting visibility into how your organization is operating now and how AI is changing it. Do that, easily for very low cost other than the time it takes to do it for you as a person. that data is gonna be instrumental in helping you make arguments for or against certain strategic decisions. on a leadership level, I think that's another thing.

and I think the maybe more tactical way is if you are feeling the pressure for AI more in your organization. Look at how AI is being presented. Is it an individual productivity tool where people get, a license and then they're sharing interesting things that they do? Or are you doing this AI dojo kind of scenario where you're looking at problems that teams have and problems that the organization has and finding ways that AI can.

Solve those problems organization wide, that's where you're gonna get the most leverage. focus on those things. if you do those three together, it's gonna take some time, but this sort of gets you out of the fog a little bit. truth be told it doesn't actually matter if Microsoft is writing 30% of their code. Because you're not Microsoft. it's really difficult to not get a bit of FOMO or not feel the pressure, especially if you're getting pressure from your exec team about, well, I saw this in the headlines, why aren't we doing that? But coming back to them with data about what our business needs and what

we're doing, and how I know that we need to do this and not this next. That is what is gonna get you ahead and stay ahead. instead of just chasing metrics, vanity metrics that you see in a. LinkedIn post on a random Tuesday afternoon Bret: Unless it's yours, Laura Tacho: unless it's mine. Bret: unless it's your post. Laura Tacho: none of my metrics are vanity metrics. they all have adequately sufficient n sizes, or I call it when they don't, when a median might be unstable. But yeah, it is really hard to differentiate that because not everyone, you know, LinkedIn is for half of a brain and half of a thumb.

and a lot of, the shiny graphs get the, most shareable. Bret: speaking of that, so, uh, thanks so much for being here. Where can people find out more about what you're doing? where are you chatting? On the internet. Laura Tacho: I am only on LinkedIn these days Bret: Okay. It's an easy place to be. Laura Tacho: I tried to do Blue Sky. I really did for you, Bret, and for you Nirmal. And I just really couldn't do it. Bret: I don't get likes there, so I'm the same way. no one sees and pays attention to my stuff other than Nirmal and a few other people. Laura Tacho: other than.

That's very cute. Um, find me on LinkedIn I'm the only Laura Tacho out there. and you can read about more of like my research and guides on getdx.com if you go to the research tab. I'm doing a lot of research and white papers, and I heard from Nirmal, if you Google me, you'll come up with a lot of interesting, talks that I've done Nirmal Mehta: Yes. Laura Tacho: I've been, at the Enterprise Tech Leadership Summit in Vegas, talking about booking.com and how they scaled AI adoption. I'm gonna be doing some, other keynotes. I'll be at Cloud native Austria. Actually, it's not, it's a CNCF, like adjacent, not

Bret: Hmm. Laura Tacho: but I'm very excited for that. I'll be keynoting there as well in October. Bret: Ooh. Laura Tacho: I'll also be at KubeCon. I'm gonna be at. Platform Engineering Day, talking about the real adoption rates of AI tools across the industry to help dispel the hype a little bit. we'll be at Backstage Con, and I'll see both of you there Nirmal Mehta: Yep. Laura Tacho: exciting. Bret: that's gonna be an exciting talk. we talked the last, KubeCon we were hoping for more talks, this turnaround on the actual using of AI for DevOps and platform engineering rather than the running of

AI and GPUs which is what everything's been focused on for three years. it'll be fun to see that kind of stuff. Nirmal Mehta: Yep. Laura Tacho: totally. Bret: where can people find you? Nirmal Mehta: with you, Bret. so Agentic DevOps fm, our podcast that you're listening to right now. on top of that, I'm on LinkedIn. Metaverse hq, on Instagram and Blue Sky, and then, mostly on LinkedIn, Nirmal K Mehta. you'll find me. Bret: Yeah, we're all on LinkedIn. Laura Tacho: kind. Nirmal Mehta: Yes. Laura Tacho: Yeah. Bret: LinkedIn as much as anywhere else now, so yeah, it is what it is. well, Laura, thank you so much for being here.

We're gonna have you back, actually, this is a spoiler that we already are planning about maybe another episode, but we won't talk about that yet. We'll talk about when it actually happens. I'm excited to have you on this show. We're so glad to have you back and I love this new focus of yours. Nirmal Mehta: Yes. Bret: Yeah, we were on the container bandwagon for a while, but I think this might just be your moment and, Yeah, we're here for it. Nirmal Mehta: we're here for it. Thanks, Laura. Bret: All right. Laura Tacho: Thanks, Nirmal. Thanks, Bret.

Open the transcript file

Hosts and Guests

Hosts

  • Bret Fisher Host Feed-stated
  • Nirmal Mehta Host Feed-stated

Guests

  • Beth Fisher Producer Feed-stated
  • Cristi Cotovan Editor Feed-stated
  • Laura Tacho ๐ŸŒฎ Guest Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Rogue AI or Just Sloppy Ops?

    Sep 25, 202643 minAudio

    The OpenAI Hugging Face hack. The Anthropic testing failures. Listen to the security and ops experts, not lab researchers. I walk you through the production failures that allowed OpenAIโ€™s models to access the internet,...

    Credit the people on this episode

  2. AI Agent Sandboxing with Nono

    Aug 21, 20261 hr 5 minAudio

    Luke Hinds, co-founder of nolabs Inc. and the nono project, joins me to dig into AI agent sandboxing with the nono CLI. It's been my go-to sandboxing tool for local agent use, but I know there are more features I should...

    Credit the people on this episode

  3. AI-Native Engineering Culture from a CTO

    Aug 1, 20261 hr 4 minAudio

    My friend Mike Rollins joins me for a chat about what itโ€™s like leading a software engineering team that doesnโ€™t read code anymore. Click here to watch a video of this episode. Thanks to SpeechifyAI for sponsoring this...

    Credit the people on this episode

  4. Safer Agent Automation with GitHub Agentic Workflows

    Jun 17, 20261 hr 30 minAudio

    A deep dive into GitHubโ€™s Agentic Workflows feature, with Don Syme of GitHub Next & Peli de Halleux of Microsoft Research. Check out the video podcast version here: https://youtu.be/zmM8VISTOwo ๐Ÿ˜‡ My new GitHub Security...

    Credit the people on this episode

  5. AI SREs, Chat With Your Infrastructure with Anyshift

    May 6, 202652 minAudio

    Bretโ€™s joined by the Anyshift.io co-founders, Roxane Fischer and Stephane Jourdan to discuss how an always-on SRE agent can help you proactively find risks and avoid incidents. ๐Ÿ˜‡ My new GitHub Security workshop has...

    Credit the people on this episode