Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow artwork

Datadog Deleted All Its AI Context. It Worked.

Aug 4, 20261 hr 1 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Datadog's Language Foundations team deleted an entire folder of AI context files that had been carefully written and maintained for over a year, expecting a performance hit. Instead, their evals got better. Simon Boudrias, who runs Language Foundations at Datadog, walks Guy through what that taught his team about context rot, and the full journey of scaling AI coding agents to 4,000 engineers. What we cover: - How Datadog scaled Cursor and Claude Code to 4,000 engineers in under a year - Why Datadog deleted all of its AI context files, and what happened to eval scores - Building an eval-driven code review system that replays old PRs to catch real incidents - Where open-weight models like GLM 5.2 stand against the frontier - Rethinking hiring and career ladders now that AI can run a real codebase interview Chapters: 00:00:00 - Introduction 00:03:19 - Simon's role and Datadog's 4,000-engineer org 00:04:31 - The Cursor rollout that took off overnight 00:07:53 - How Claude Code entered the picture 00:10:51 - Building dedicated Signals and Flows teams 00:11:58 - Why Datadog bet early on evals 00:33:44 - Deleting all their AI context and getting better evals 00:47:51 - Where open-weight models stand today 00:51:45 - Rethinking hiring and career ladders for AI 00:59:24 - The real prize: better decisions, not just productivity 🌐 Tessl: https://tessl.io 🔔 Subscribe for weekly episodes on AI-native development What's the oldest file in your AGENTS.md or CLAUDE.md that you're afraid to delete? Tell us in the comments.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_02The punch we had is like maybe actually the context that was written like a year ago, you know, when it was like cursor Sonnet 35, it's still irrelevant today. And so we started to look into it. And then we're like, well, what if we delete the whole context from the repo? Like, what would happen? And what happened for us is actually like the ELAD started to perform like much better, which was counterintuitive, but it's just like, yeah, that file was pretty big and was listening to stuff that was not like listening useful and that was really interesting to see.

SPEAKER_00The AI Native Dev is a podcast with developers and engineering leads at the cutting edge of AI energetic. Join your hosts, Guy for Germany, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. We just wrapped up two amazing days at AI DevCon in London.

SPEAKER_01But the great thing is that we get to do it all over again in New York City this November.

SPEAKER_00You're absolutely right. We're gonna be back in the city that never sleeps on November 3rd and 4th for more amazing sessions, really engaging, hands-on workshops, and much more.

SPEAKER_01Yep, all that great networking, partying, eating and drinking that you've come to expect from AIDEVCON. We think we have one of the best hallway tracks in the business, and it's the perfect compliment to our incredible speakers and presenters.

SPEAKER_00We'll both be in person and virtual with live-streamed access to all mainstage keynotes and talks.

SPEAKER_01Sign up right now for our Super Blind Bird ticket for just$100, only available for a limited time.

SPEAKER_00We're really excited to be headed back to the big app. We hope to see you all there.

SPEAKER_04Today we have a really interesting story about how to build out agent enablement and learn from it, uh, looking at Datadog as an organization that really kind of dove in and invested in agent enablement uh and has a bunch of learnings to tell from the journey. And to guide us through all of that is Simon Boudrien. Uh, I hope I'm not sort of misrenound that too badly. Simon, thanks for coming onto the show.

SPEAKER_02My pleasure.

SPEAKER_04So, Simon, we're we're we're gonna do this justice and sort of go in chronological order on it. But just to sort of give people a little bit of a teaser of some of the highlights I know in the uh, you know, as we as we talked through the journey that I really liked. We'll talk a little bit about context, but actually a fair bit about context. You know, where did it work for you when you created it, where did it create kind of context rot uh and exaggeration. And in a way, uh I don't know if it's dramatic or standard case, but uh a learning about cases where you actually an eval showed you that if you deleted all the context, it actually actually did better uh than before.

SPEAKER_04So that'll be great. We'll talk a little bit about org and how do you structure and who owns what, uh, the importance of evals, and uh, and and we'll we'll talk about where the bottleneck is, you know, what are the what are the problems really that are worth solving because there's infinite work uh in the world of where we're doing it. So I'm keen to share all of these stories. Uh for starters, can you just sort of tell us a little bit about what is it that uh that that you do, kind of the scope of responsibilities, and maybe a couple of words about Datadog and uh you know that the large group of engineers that you're trying to sort of optimize their work for here?

SPEAKER_02Yeah, for sure. So uh I'm a director at uh Datadog. Uh I manage a part of the uh SDLC org uh work called for the language foundations. Um and so uh we are serving about 4,000 uh engineers here at Datadog. Uh and the way that I've described our scope is like basically kind of the front end of the uh SDLC. It's like everything people uh interact with day to day and like how they do their work. A lot of that falls within my group. Uh and uh and a year ago now, uh we launched like a new uh AI devx uh group to focus on AI uh uh tuning basically here at Datadoc.

SPEAKER_04Yeah, that's a little sort of a small initiative there of uh, you know, maybe maybe this thing has legs on it. Uh cool. Okay, and I like the definition of like the front end of the sort of the developer tools on it. I guess when you have 4,000 developers, there's a there's a depth to that uh as well. So uh walk us through a little bit of the storyline of you know, as you started to to invest in Trulli or in general, like how did uh the kind of engineering organization and datadog kind of approach or start opening up uh the use of agents and a little bit of that timeline of how it evolved it today?

SPEAKER_02Yeah, yeah. So like uh for us it really started in the beginning of uh of like 2025. Um when you know vibe coding as a term was like coined. There was much more talk about it, about this being viable and like starting to be something you know that like really we could use AI to write code. Uh and so I'd say like for us the big driver there uh has been cursor at the beginning. Uh and so at the beginning of 2025, uh we launched like a POC internally with like Cursor.

SPEAKER_02Initially, we were aiming to you know get like a hundred, maybe two hundred devs to use cursor and just like get some signals of like how that you know, how like it worked for them. Um and it just like took off from there. Like we had so many more than like 200 people. Uh, we quickly had like a thousand devs using cursor almost day to day uh within like another month of like launching for this PC. And so like for us it was a pretty big signal that like, yeah, there's truly something there. Um there was a lot of people that like liked it full length value in it.

SPEAKER_02Um this is a pure pull, right?

SPEAKER_04Because you opened the door, you said you can use cursor and everybody came running in.

SPEAKER_02Correct, correct. Everybody wanted to get in and like try it out basically. Um and so that's where we started. And and like with like that pull, you know, towards more uh AI tool, um, we were like, well, yeah, this is gonna be transformational to this industry and like how we do our work as engineer. And so uh and so at that point we decided to, okay, we'll we we will probably need to like create kind of a team to to like own those tools and like own for the uh experience around the around AI used for development.

SPEAKER_02Um and so at that time we started to recruit a team uh that pretty much were like kind of the whole summer.

SPEAKER_04Um and before before the recruitment of the team, the sort of the we over here is basically you and the people looking at like developer experience and and front are the uh are the people, I guess before it was designated AI.

SPEAKER_02Yeah, yeah, yeah, correct. Uh correct. It was like folks within my my org we ran for the POC. There was a like a staff engineer male who like took for the shepherdship uh of that uh of the cursor POC.

SPEAKER_04Yep. And I liked how you sort of told me that at some point this was uh demand was so kind of off the list that you had a product manager that found themselves manually because like you weren't ready for the avalanche, and so someone was like manually adding people's emails uh in to sort of add them to the cursor list of allowed users.

SPEAKER_02Yeah, yeah. That was with like claw code over the summer. Uh when like claw came out, like there was a second wave of uh adoption. Uh and like that tool. Um so at first we we had issue to set up for the SSO. I'm not like exactly sure why, but that didn't work. And so we we needed to add manual email to like to like claw code basically to get access to people. And yeah, a PM took that role of like basically managing access to that tool over the summer until we could figure out sort of SSO bug we were running into.

SPEAKER_04Somewhat ironic in the uh the sort of your AVI. So sorry, I kind of disrupted your flow though a little bit. So you had cursor, you decided to hire someone, and then you went on to start flood code. I guess how did flood code enter the scene here?

SPEAKER_02Yeah, so uh well, so uh because the product were uh already working uh with the uh entropic models, uh we had connection already in like in the contract. And so just like someday Clock Code came out, then people could basically use it because we had uh API keys. Um and so uh and so yeah, it it just was available overnight, and I think you know people pay kind of attention to the news cycle with like people on like singing, there was a lot of hype around clock code at that time.

SPEAKER_02And so it basically uh organically grew from that point.

SPEAKER_04Would you say like the cloud code adoption? This is uh history at this point, you know, versus present, but would you say cloud code adoption was was you know as uh as fast as uh as the cursor one? Because cursor they they really nailed the kind of low friction, you're already in the ID, we'll give you superpowers, you know. So, you know, why wouldn't you want uh cursor? Well, Cloud Code was more you know, like more of a big change to it, but clearly super compelling, but um still the appeal was the same, or was cursor still adopted faster?

SPEAKER_02Uh that that is hard to say. I'm not sure, you know. I'd say like both at that point, you know, towards the end of the summer of like 2025, because I think the story changed later. Uh, but uh at that point they they pretty were like pretty similar, and I think they were catering to like different personas. It's like like people, uh especially for us, where uh maybe piece of context, like we have huge motorepos. And so we know like VS Code on those tends to perform pretty badly. And so the base of cursory and some other monorepos was causing issue for uh adoption of just like being not uh performant enough uh for the large repos we're having.

SPEAKER_02And so there was like some of those people uh had a much better kind of experience with with clock code running in a terminal where like you don't need to have the whole uh IDE running. Uh and so for a lot of people that was more compelling, I think, just from like a performance reason. Um and so I think like that drove a lot of people there, and then uh all of our power user quickly flipped over to Claw Code uh because I think what they were seeing is like they don't need the IDE open. You know, they could do most of their work without an IDE open.

SPEAKER_02Uh and so for them that uh adoption for the power user was like pretty clear early on.

SPEAKER_04Got it. Yeah, and so this is so we're about a year ago, we're sort of like approaching summer of 25 uh at this point. Okay, so talk us through so cool. So you've had that, you had a cursor just sort of lumped in, you decided to hire. While you're hiring, you sort of have this sort of cloud code, you know, land in your lap, uh, and you're managing, sort of added, you know, with that sort of SCS uh SSO. Uh and then you said you started to build a team?

SPEAKER_02Yeah, yeah, yeah, yeah. We started to build a team. Uh so like uh one of our uh early champions was an EM here at Datadog. Uh we offered him to do kind of a team switch and like build this new team. Uh we started to recruit people for that team. We had a few transfer come in as well. Um yeah, and so yeah, we s we like started to put that team together. Um we wanted to do everything, but you gotta decide where you start. Uh and so for that team initially we started really like focused on like metrics and just getting more metrics into like what is going on, what is the impact, uh, what is the cost as well.

SPEAKER_02Uh and on the other side, uh, we knew already that like AI was pretty expensive. Uh and so and so we wanted to start to be able to like benchmark different models, what is the best solution? Uh, and so we created a team or like that team started to work on evals uh and like building kind of a uh agent encoding tool, eval platform, uh to help us make decisions, you know, about like which model, which tool we should onboard uh at DDVOP.

SPEAKER_04And this is uh so this is like an early uh embrace of evals, which I appreciate. You know, we're sort of big fans here at Tesla. Uh and how did you how did you apply that? So like what are examples of things that you deval? We'll we'll we'll dive in more. So I'm still in like journey mode, but these sort of earlier days, yeah, it was sometimes hard. Like you might draw a conclusion from an eval, but it wasn't it's not always easy to think about how would you apply that learning. So, what are examples of like should we do X that you've decided?

SPEAKER_02Yeah. So I'd say like the first application of evals for us very concretely was for code review. Um, and we had a bit of back and forth on like you know, what's the value of AI in code review? Uh and I think you know, if you work with a tool that is very nitpicky, you know, it's not really fun to work with a nitpicker that is not a human. Uh and so we saw pretty early on that like really for us, the distinction here would be like uh we wanted to put AI in the loop for code review to really like prevent an incident or like kind of a major bug and like really like focus on that, you know, as like kind of the last uh guardrail before you go to production.

SPEAKER_02Um and so the first evals we did was like replaying PRs we knew had caused incident, and then like catching uh you know, are they gonna figure out that this is gonna fail actually? Um and so you know, we do track a lot of data around uh incidents here at Datadog in a dataset, and so we had all that data already kind of there for the picking. And so we were able from there to like devise and like kind of basically build a platform that would replay PRs and then run the whole uh agent tick review, and then with an agent judge being like, well, does that does that review comment point out what is gonna cause an incident later on?

SPEAKER_04Yeah, yeah, super smart because it's like a clear use case because in evals oftentimes the difficulty is defining what correct is that if you have those historical incidents. So I want to dig into evals, but let's just sort of complete maybe the timeline. So you went ahead, you sort of hired uh hired or like moved this internal person and you started building a team. And I know you know, maybe fast forwarding to today, as that team evolved, uh you now have two teams, right? Like now that you're sort of a bit more mature, maybe tell us a little bit about current setup and then we'll move on to evals.

SPEAKER_02Yeah, yeah. So what it says is like we had a lot of uh ambition, you know, around like changing the way we work with AI. Um, but towards the beginning of the year, what we saw is like that first team that we built, uh they had a ton on their plate with evals, with metrics, with like cost management, uh, and all of that. And so it was really hard to then like you know carve out a piece of you know time for us to actually work on like rethinking how we work.

SPEAKER_02And at the same time, like the beginning of the year, I think everybody saw with uh Opus uh 4.5 uh that came out just before the holidays, the like adoption from there really like took off like really, really fast. You know, pretty everybody says like the the amount of like PR pretty much doubled overnight, you know.

SPEAKER_04Yeah.

SPEAKER_02Um and it started to really show you know kind of uh bottlenecks in our flow in the uh SDLC that are way more serious than they were before. Um and then it was hard to carve out time to actually like focus on that. Maybe for us right now, it's like really code review is like pretty like the big bottleneck that we're trying to tackle. Um and at that point we're like, well, we need more people to to work and like really focus on the experience itself of like you know the SDLC with AI and not just more kind of the governance and like signals side of the world.

SPEAKER_02And so we we like uh launch like a second team. Um I forgot exactly when, but towards the beginning of the yeah.

SPEAKER_04Yeah, no, very cool. I love uh and I like the names you sort of mentioned to them as well, which is signal and flow. It's like very uh that's what it says on the TN type names. Um and just I guess for uh it's useful sometimes as people calibrate for them to sort of think about rough sizes of those teams. Like it's kind of you have circa 4,000 engineers in total. What are sort of the rough uh sizes of uh these two teams?

SPEAKER_02Yeah, yeah. So we're uh aiming to have six people, around six people in like an boat team and with some recruiting uh for that second team.

SPEAKER_04Got it. Cool, cool. Yeah, well that's still a good ratio, you know, in terms of uh of multiplications and leverage.

SPEAKER_02Yeah, yeah. So about you know, 12 people for 4,000 engineers.

SPEAKER_04Yeah. And the the I guess as the organization flows, this may be like one last question on the team thing before we continue. Um as you change the the way that you work and the adaptation, didn't did a need for new signals come out? Like because I think the previous team, like the signal team, is very much around current way of working. Let's sort of measure output so we see if productivity improves or you know, costs change, etc. But now that you change it, did you find the same metrics work uh over time?

SPEAKER_04Or do you did you need to kind of invent new metrics, right? Or sort of start measuring new things?

SPEAKER_02Yeah, so I don't think we've had to change dramatically how we measure you know productivity. Uh but I'd say like we're using a lot like Dura metrics, and these tends to be more um, you know, they're not like leading metrics, they're a lagging metric. And so and so looking more into like uh the leading metrics and like what's in there, uh, I I think usually we need a bit more creativity, and and those are usually kind of less like true signals of like they're a piece of the pie, and it's hard to uh isolate exactly what move those metrics.

SPEAKER_02Um so so like what I'd say maybe like what has been more visible is more like how we think of costs uh and like the amount of like token we're using. And then um so like you know, just like being more professional and like handling cost of AI, like like you know, with now everybody at the company basically using AI, using more of it, uh there's a bit, you know, it does uh increase for the budget that goes towards AI. And so like we just need to be more cognizant of this and just make sure that is the skills reasonable.

SPEAKER_02Um and so and so definitely more on the cost aspect of it that we've done. And then the other thing is like with evolves, and now that we are that we're getting a bit more data, then like the question comes on like, well, how like do you turn that into adge and all like to know what to do with the evols? And so and so like probably like those two aspects have changed more in terms of productivity metrics themselves. We're still mainly looking at like Dura metrics overall to get the to get signals.

SPEAKER_04Yeah, yeah, interesting. Yeah, makes sense. And I guess did you ever get into token maxing uh of it, or has that been because you mentioned cost, like you're sort of aware of how much it cost before. That sounds a little bit counter, like I want this successful, but I want it responsibly successful.

SPEAKER_02No, no, uh, not like token maxing. Uh we never did quotas, for example. Uh, and so and so we're trying to like really not go that route because you know, if you need a lot of token because of what you're working on, you know, we we kind of do want people to have the uh optionality, you know, and be uh intentional about for their for their own use and their own needs. Um and so you know it's like maybe a bit like breeding hot and cold at the same time, uh, but we're really more looking into like a systematic way that we can um reduce token or like compress output, you know, uh things like that.

SPEAKER_02Cool, cool.

SPEAKER_04Cool. Okay, thanks for sort of uh the rundown there, sort of on the team who've done it. We've started sort of teasing out a little bit the sort of investments in in evals, you know, and and creating it. And I think I thought that was really interesting because this is very much like a world of vibes, right? I'd say even until today, uh, you know, evals, like if you're building an AI product, everybody will say evals. But if you are talking about sort of development with uh with agents, or if you're talking about even sort of like a genetic workflows as a whole, uh I guess kind of uh showing my bias a little bit, it's like I I feel like there's a lack over there, like the vibes are are quite dominant on it.

SPEAKER_04Um so I like that you that that you kind of created those. Can you tell us a little bit about like what works? So like you've invested in evals. Um we don't really need to go through the whole journey of, but like where you are today, what is it that you do today with evals? Maybe tell us a little bit about the tech stack, but also what's what is the value that you sort of derive out of them and maybe where are they not in yet? Yeah.

SPEAKER_02Okay. So uh so like for us it's a fully custom solution written in Go, uh, with like single boxing. Um so I don't know, you know, when we started if there was a lot of uh open source option, uh we pretty like looked at a few, you know, but nothing was like really mature, and so like we just went our own way since like it was not like overly complex for a pure use case of like AI agent. And then Datadog has a pretty good metric platform, so like it was easy to just feed the metrics in there and then consume it from Datadog directly. Um so uh in terms of like what we do today uh with the evolves, uh just like mainly like two things that we're looking at really specifically.

SPEAKER_02Um on one side it's like just like tracking the performance of like new models, especially for the uh open waste model, and like trying to get a sense of like when could that be a viable alternative or in which fashion could could they be? Uh and on the other side is like how does the uh uh what you know uh things like steering documents um are they uh are they working? Uh when we change it, does it change the performance of the evolves?

SPEAKER_02Um and so like we more on the on the performance side where like teams that are not Us are gonna own stewing documents or maybe should own the steering documents because we can talk of ownership there sometimes is a bit hard within repositories. Uh and just like tracking a bit the performance on like is there an issue and like should we change it in and in which form should we change it?

SPEAKER_04And and did you um so maybe there's like two different sort of conversations here. Sorry, I started both together there. You know, maybe one is like let's talk a little bit about the stack and then we go to the substance. So on the stack, you built your own because you know you have your sort of Datadog platform, kind of very good at collecting data uh and aggregating it. And I guess at the time nothing else was mature. Do you do you think people should build their own eval system? Or would you say like if you were to do this again today, you'd probably pick harbor or sort of one of the existing uh frameworks uh for it? Or do you think that there's sort of aspects of how you do the eval that you feel like they need to be custom, they need to be yours?

SPEAKER_02Yeah, so I don't feel like strongly about custom versus hosted platform. Uh I think what's important is like to be able to support multiple harnesses, uh multiple models from like different sources, I think it's pretty important. Uh but as long as you have that, then I think like most of the platform do offer it. Uh, I think going with like with like an off-the-shelf solution is pretty pretty good. Uh but I'd say also coding being less of an issue today, for us it's been fairly easy to put it together. It wasn't that hard.

SPEAKER_04Yeah. Yeah. To build the system. So I guess uh so so I think I think that that sort of makes sense. Like technically, just to be able to run an agent and sort of create have a card uh or like some sort of scorecard uh and run that and aggregate the data, you know, those are kind of within core competencies. And I guess you kind of always weigh that a little bit against, okay, but why reinvent it? You know, like if you if it's not that custom. So it's a decision I guess all developers are sort of fairly familiar with. The second question that I have is maybe a bit more in the substance, is at the beginning you talked about use cases that made a lot of sense, that are central and have clarity about what the truth is.

SPEAKER_04So I guess my my question is as you approach use cases that are sort of more elaborate and you talk about using models and all that, like what is correct is not always as clear-cut. So I guess how do you how do you handle that? And and maybe like who even writes that down, you know, today, right? Like who creates the eval scenarios?

SPEAKER_02Yeah, so in terms of evals uh and like writing those evals, uh there's a few different things that we're doing and unlike new stuff that we're trying out now. Uh the status quo so far has been we've wrote the first ones with like very common use case to just like have a baseline that we can go through. And then we went out to pretty much like all of the platform teams and like asked them to like add evals, you know, to like cover their their like uh own platform use. And so really spread out for the uh ownership of those evals across a company and just like get more support.

SPEAKER_02Um the downside of that though, uh is that yeah, it is limited in a way of like if as a dev you're using like like a harness or like an agentic tool, like you're probably not only asking them to do very specific things. There's a lot of the work, it's more on the creative side, finding ideas, what should you do, what's the right approach to your problem. Um this we don't try to cover with the e-vales we have um because it's it's just like so broad that it's just like really hard to know what exactly you should try.

SPEAKER_02Um and you know, if it's always the same question, then like you only have this like a small slice, but like kind of the creativity aspect, I think is somewhere or like something that's like really hard to gauge with an agent.

unknownRight.

SPEAKER_02And so I don't have a great answer to that, but I know it's a hole that we're having and we're like fine to live uh with this gap that we're not sure how to fill yet. Maybe someday I can come back and like uh maybe we figure it out, but uh for now uh when we're not too much trying to like tackle that.

SPEAKER_04Right. Um so you just to sort of echo back over here, you're saying at the moment where you're sort of seeing the value in a bit of a concrete fashion is in finding cases that have clarity about the truth, the repeatable processes, you're doing the thing again and again. Uh and generally the scenarios and all that are written fairly centrally, like not just by you, but by you mentioned platform teams. So like it's not really not going to the end developers, they're not really writing evals. These are extractions out of real-world scenarios of evals that change at a slightly lower pace, uh uh, broadly speaking.

SPEAKER_04And do you think eventually, like if you're if you were kind of a betting man, right, and you'd sort of say in a year's time, do you think all developers write evals, or do you think it stays this way? This is like a crystal ball question, so no, it's not gonna hold me to it.

SPEAKER_02No, but I have a good answer for you here. Uh, I don't think every dab is gonna be writing evals. Uh though what we're doing right now, uh, so you know, TBD if it's gonna work, but I'm pretty confident that it's gonna be the right solution for us going forward. Uh, we are tracking for the uh trajectories of agents and what they're doing. And so we're tracking a lot of data today for the use of agent within Datadog. And so what we want to do is like go from that to raw data of like, you know, well, this prompt came in, the agent did that, etc., and like extract from there common scenarios that tend to recur, or like a common type of uh interaction that a user have with a given platform, and then using an agent to turn that into actual ELON.

SPEAKER_02And so basically taking like real data of like how people use the agents and make sure that we're covering a fair amount of uh overlap with actual use. And we're pretty confident that we can uh automate uh most of that.

SPEAKER_04Yeah, yeah. No, I love that. That is actually very aligned with sort of our experience at uh at uh at Tesla, which is the more you anchor it in concrete cases, the more real it is. And then, you know, like what the developers don't like doing, they don't like writing docs, they don't like writing tests. And so now you're sort of saying, hey, write docs and tests, you know, that doesn't really work. Uh and and so probably the best ease of use aspect to it is instead of asking uh devs to write those tests or write those eval scenarios, you you observe them, you sort of extract them out of the loop and you extract these scenarios.

SPEAKER_04I guess there's still it's an unavoidable uh future or sort of evolution to that that you will still need. Like these things are dynamic, and so you still captured that scenario, that sort of hypothetical aggregate PR. Uh at least that's the case in our world, right? You've observed a bunch of PRs, you've created an eval scenario, you're running it, you're still capturing that as some files that's in some repo. Uh and over time it will rot, right? Like over time it will no longer represent your system, and so there has to be some maintenance. So at some point, someone needs to own that, right?

SPEAKER_04Like that someone is hopefully someone who can kind of uh represent developing in this repo, right? Or or something like that.

SPEAKER_02Yeah. Yeah, uh, I agree. Uh but I think it just, you know, it's hard to catch Roth and it's hard to come back to to like case you created in the past because like the amount usually that you end up with is like fairly large. And so we're really focused on uh on uh automation here. So like really having any agents to do for the sweeping and to really bring up the signal, but eventually maybe even make the decision about like which case are still relevant today.

SPEAKER_04Yeah, yeah, I think uh I I buy into that. I think in general, like whenever you introduce a human, you you create a bottleneck. We'll talk about that in a sec. Uh but uh whenever you introduce a human, and so so you have to plan ahead for this. And in that sense, the sort of the users, both the authors and the users of the evals uh are very likely to be uh to be agents. So you still need to create them, but it's not necessarily the person that creates it, might be the uh the someone else. And I I guess the before we kind of move on a little bit from evals on it, how um how often do you run the evals?

SPEAKER_04Like once you once you agree that you know something is correct, is this is this something you you deem after the fact do you run them on every context change? You know, do you run them I guess uh how do how do the how does the execution of the evals integrate into this SDLC?

SPEAKER_02Yeah, yeah. So today we're doing nightly evals. So we're running basically once a day. And then for PR that might specifically target evals, like they're changing steering documents uh in like a heavyweight, really trying to sort of change for the results. Uh for those people can do like a ad hoc run. Uh and you're probably gonna want to do a few since like evals are not always super stable to get the statistical significant results.

SPEAKER_04Got it. Very cool. So this is uh you run nightly, you look at relevant changes, you know, some steering docs that have sort of modified. Uh if those occurred, you run the eval, even if there's a an apparent regression, you you you point that out to the to the person. Uh so you don't get in the way and you run them. I guess it's sort of consistent with the previous theme, which is you run them on the things that you you feel like you truly know, as opposed to trying to speculatively uh anticipate ahead. Correct. Okay. And that probably results in something that like costs-wise is not that big a deal.

SPEAKER_04How expensive are these evals getting?

SPEAKER_02Uh it's a good question. I'm not sure, you know, but evals are more expensive. Like they're they're cheap enough that they haven't doubled your sort of uh compared to the coding agent hues, uh, yeah, yeah, yeah. It is uh it is nothing compared to, but it's definitely something that we're cognizant of. And like the decision to run it lightly is obviously part of that. We're like, well, if we run per PR, obviously it's uh that is going to be too expensive.

SPEAKER_04Yeah. Yeah, it's interesting. No, we've um we've been experimenting with this, of course, sort of in uh in Tesla like Tesla itself, but also kind of with customers. And the um one of the uh observations is that you can actually use uh cheap models, like open, cheap open models, to assess regressions. So we found that Deep Seek uh actually correlates pretty well in terms of the skill uplift uh for a variety of tasks. And so what you can do is it doesn't need to be as good at overall the tasks, but if you measure it with skill, now that one is like a lot, it's fast and it's cheap.

SPEAKER_04And so maybe you can run that sort of deep seek v4 flash uh as part of your CI test for some sort of selection, and so you find it because it has all the sort of the natural problem of if someone made a change yesterday and you're coming back to them tomorrow, it's annoying, it's rework, it's better if it's in line, you have to weigh that. So I don't know precisely where it lands, but I feel like there is an evolution of evals uh uh expanding as tests, right? Just like we have some tests that we run in the CI and some tests that are too expensive or slow to run in the CI, I think we'll have the same for evals.

SPEAKER_02I I think that it is exactly correct, and it's also the way we're thinking about it, right? Like what you can run after the fact, unlike you know, what's what's like the cost of a revert or like a rollback versus the cost of like that like you pay now waiting before you can merge something? And so it's not always a clear trade-off. Uh both we need to just try to be kind of cognizant of the cost of making a mistake versus the cost of waiting for things to run or paying for the infra to run it.

SPEAKER_04Yeah, exactly. Very cool. Okay, cool. So I love the passion about evals and I can kind of uh geek out about evals. I've uh built a passion for them. Uh but let's talk a little bit about context. So in the process of that, you know, you referred here to sort of steering context and all that. What is the sort of the state of context typically uh uh within Datadog? You know, are people like is there a central context? Let people share it, is mostly in repo. Like how would you describe like skill adoption right now? Give us a bit of a picture of where is context being used, focus on dev uh uh within uh within the organization.

SPEAKER_02Yeah. Yeah, so it's like there's like two two big sources. One is files that are just committed to to the repositories themselves. Uh I feel like skills is now mainly what there is. There's not a lot of other stuff that is not skills, skills has been pretty well adopted, yeah. Um and then we have the cloud marketplace with a lot of uh skills there as well, uh, that is centralized. Uh and uh I can talk of like mistakes and like all of those as well if you're interested. That's one talk.

SPEAKER_04I think you had uh multiple sort of insights. So yeah, yeah, yeah. Yeah, what did it work? Let's start let's start positive. Where do you sort of see it working?

SPEAKER_02Yeah, yeah, yeah. So I mean what what worked really well is like everybody contributed early on, and especially when we started, you know, without a central team, it was like a PUC, just like get people on boarded, you know. I think it was good to not put too much structure early on so we can try and see what works and like have learnings out of it. And so I think uh for the for the barrier and the friction early on was like really low, and I think that was probably the right move to do.

SPEAKER_02Uh because we want people to be contributing and we want the best idea to win and those ideas to be easy to share.

SPEAKER_04Yeah, I love that. The uh kind of core principle about transformation is at the beginning you're trying to drive adoption on efficiency. Yeah, just sort of get a little in the boat. Absolutely.

SPEAKER_02Uh but then what we saw through time is the more user we've had, the harder it became to go and like change them. Because the impact you're having then, you know, instead of like being on like 200 people doing kind of a PC with you like using the same tool, you're gonna impact like 4,000 people working on the repository. And so like it's just like much more uh stressful to break things for like so many people. Uh and then also uh the size became kind of uh uh hard to manage. Uh like kind of the cloud marketplace has hundreds of plugins.

SPEAKER_02So, you know, if you're once you uh install the marketplace, like do you know which plugins are good, which one should you use, which one are gonna perform better. Uh and so what we've steered people to do is more team-based marketplace for them to share for their own flows within the team things that are specific to their team uh or marketplaces extension for specific platforms. So if you work with Platform X, well, we'll we will suggest in the documentation that you install that marketplace and those plugins for you.

SPEAKER_02Or we'll refer to it from context document uh where the agent can suggest to install the marketplace and stuff like that.

SPEAKER_04Yeah, very interesting. So like a group of context so people know where they're approaching uh that and they control that. Yeah. I think it's kind of slightly hinders uh maybe like reusability. You might sort of have different teams, you probably do have different teams that do that, but you slightly control the chaos uh in this fashion.

SPEAKER_02Yeah, yeah, yeah, yeah. Correct. Correct. And then what I'd say we found out at the beginning of the year, uh Pi became like pretty popular recently. And so with so many people like starting to hint at like the fact that like Pi somehow was just performing better than the alternative we had.

SPEAKER_04And like to clarify for those who don't know that Pi is the uh kind of very highly customizable uh custom hard like harness, but it's open source, but then it can customize uh on top of which OpenClaw is built.

SPEAKER_02Exactly. And so one thing is like by default, it's not gonna read a lot of the context. And so and so kind of the hunch we've had is like maybe actually the context that was written like a year ago, you know, when it was like cursor, sonnet 3.5, is it still relevant today in the world of uh of uh Opus for 8 and G of uh GPT-5.5, right? Like maybe not. And so we started to look into it, and then we're like, well, what if we delete the whole context from the repo?

SPEAKER_02Like what would happen? And what happened for us is actually like the eval started to perform like much better once we deleted the whole context, which was counter uh intuitive, but it's just like, yeah, that file was pretty big and was listing stuff that was not like necessarily useful or you know, that should not go into like a steering document. Yeah um and so that was really interesting to see.

SPEAKER_04I I I love that sort of insight. It's uh it's kind of a real-world manifestation of how context rots, right? And we're sort of so familiar with the software rots that it goes from kind of being useful to being useless to being harmful uh on it. And for context, we're like, we don't have that muscle memory yet on it. So now you're seeing it with the skills. I guess what was the result of so the insight is great. It sounds like you still ran those evals essentially because it was like you or your team is kind of thinking you'll do that. You ran the evals, you found out it's actually outperforming uh having context.

SPEAKER_04I guess what could you do about this? Could you sort of go off and say, cool, let's delete the context? Uh how uh uh how do you do that?

SPEAKER_02Yeah, yeah. So I think if we didn't have data to back it, it would be pretty controversial. Uh and just like to be clear, this was owned uh by the front-end team that run uh for the front-end motorepo. They wrote eval, they did the folks for the testing there, uh, and like kind of they wrote new guides of like what should you do instead of context. And so like, you know, that was fully something that like was owned by by the sort of team within my organal, but but but like not like by the uh AI team. Uh and so and so yeah, uh with the data backing this decision, a lot of people were surprised by this decision, but the data was like really clear that it was performing much better, and so and so people agreed with it like very easily once they saw the footed data.

SPEAKER_04Yeah. So I think like two two sort of interesting tidbits over here is like one is you removed the context, but it's because you have started forming a harness. So, right, you're saying this was not just because the models got better, but also because you started working with more custom harnesses.

SPEAKER_02Oh, uh, this is something we're doing now.

SPEAKER_04Uh this is still primarily led to that decision, yeah. Okay, so this uh I guess it's sort of still primarily either the model improvements or just those context files were created at one point in time and they no longer represent reality for the agent. So anywhere between the new models don't need them and and they're just wrong at this point.

SPEAKER_02Uh correct, correct. I don't think they were wrong per se, but I think it's early on there was a lot of like basics that like you would try to retrain the agent on yeah. So in the case of the front end repo, there was a like a whole paragraph on like how to use yarn. And I'm pretty sure now it's part of the training set of like the models out there and saying we don't necessarily need to retrain it.

SPEAKER_04Yeah, yeah, that makes sense. And context is a scarce resource, and so over time, like if you accumulate enough of that, uh you uh you you take away uh you know information. We had uh I loved I saw one talking where someone talked about how the uh the model gets into dumb mode, uh you know, because it uh Yes. Okay, cool.

SPEAKER_02And so in term of uh governance, you know, around context, if if you want me to go a bit in there, is what we saw is like actually like I think we have an incline to just go and add something to like an agent's MD, but very often it just has easy as like is the error message from that tool clear? And do you provide next step of like what should you do instead? And so we've been also moving uh or like pushing people that like you know when they run into it like kind of an issue where the agent doesn't do what you want it to do, uh well where where where like can we provide that context when it's necessary instead of like trying to train the agent ahead of time into how to do everything, especially within a modern repo where like there's so much going on, uh you just cannot provide all of the context up front.

SPEAKER_02And that seems to be detrimental.

SPEAKER_04Yeah, yeah, no, very good. So I think a whole bunch of lessons from here. One is the importance of evals, you know, so you can make data-driven decisions, you know, two is maybe a reminder around how context does rot over time. So you have to have some sort of plan around how would you sort of reassess it, which comes back to evals. But even if you're just vibing it, like still do something, like be proactive around updating it. Uh and then, you know, maybe just a little bit more as you write new context, solve problems, don't sort of anticipate them uh as you as you go, which I guess comes back to the observability and the loops, right?

SPEAKER_04Because today probably the best practices don't actually solve problems, like just sort of solve a problem in the spot and create a loop to identify and manage that context. Um and you, like when we spoke, you talked about how uh you think one of the theories of how the context got accumulated in the repo uh was because of this sort of loss aversion. Like, you know, once someone wrote it, it was hard. And maybe this comes back to eval's, is you know, it's it just sort of feels fearful to delete it. Like maybe it is useful somewhere on it. Uh is that do you want to go through?

SPEAKER_02Yeah, yeah, no, absolutely. You know, uh I think like as people propose to remove stuff from from like that root agent MD file, like even in like PR review comments, people are like, well, are you sure you you that like you shouldn't be doing this? Because like maybe it provides value to somebody somewhere. And so I think yes, it's uh when you remove something and you don't replace it with something new, it is uh just hard to do, uh especially when you're having an impact on like hundreds or thousands of engineers at once.

SPEAKER_02It's like it's hard to have certainty. And so like if you don't have the data to back those decisions, it is really hard to sort of to like take the step to go ahead and do it or like. feel the permission or feel that you're that you have a good reason to do it.

SPEAKER_04Yeah. Yeah yeah. I I'm 100% sort of uh in agreement and it's uh I love to see it in manifest. So I guess like in those marketplaces that you have, in the small ones, coming back a little bit to what do you create evals for, I mean this sort of logic will imply that whether now or soon, like are you starting to feel that within each of these teams marketplaces, they need to have evals for how how well these skills, like for the things that are shared, like it goes to reason to say like they also need evals because otherwise they'll have the same problem.

SPEAKER_02Yeah. Yeah, yeah, for sure. Uh they will. I think like maybe one of the challenges also just like the distribution um of those skills. Does everybody have them install when they need them? And so there's a bit of a challenge there of like making sure that people have the right tool for the task that like they want to do. And I'm not sure the way we're working today with like local agent only and like everybody set up for their own thing. There might be limits to that eventually.

SPEAKER_02And so I I'm not exactly sure what what is going to be the right solution going forward, but I think meta harnesses for example are pretty interesting to that of like you know what if we we have kind of a classifier and we know well this is touching platform A. So let's run this with an agent that has those skills pre-install. Yeah. And and like you know if this was centralized for example like I think that might be able to provide a much better kind of experience where like we have a better baseline that don't rely on like tribal knowledge or like somebody having read documentation.

SPEAKER_04Yeah yeah no I agree and I feel um I think generally like people underestimate the importance of environment definitions and evaluations. And so I think that's at least how we translates in our world we created these like project evals versus sort of scale evals. They're actually all just the same evals, just sort of names uh and the idea was the scale eval is a thing you test in isolation. And so it's a good way to sort of test if it regresses or not. So that that that needs to tackle that first problem which you correctly pointed out is a bit hard which is define what correct is because it runs together.

SPEAKER_04And then and then you get into the problem of like okay that's nice and well but I actually have 30 skills installed when I'm running this not one. And so does it work in that environment then that basically gets infinite right like which which skills do you have at what time and how do you and so I think environment management like the the thing that we've sort of seen extend in flexibility but then you it's also in complexity you have to think about how to write that in is just like all these definitions of how do you define the environment within something runs. And I guess they both boil down to trying to eliminate works on my machine scenario right like the one is define what correct is and define the machine of like work like define works and define machine and you can step out of that so it's interesting.

SPEAKER_04And it sounds like that's still a sort of a uh a work in progress like at the moment in practice these little marketplaces I don't have label but like team marketplaces don't tend to have evals. Yeah correct you ship it out. Yeah. So uh these are fascinating conversations we're starting to run out of time here so maybe uh sort of chat a little bit about um about kind of open models on it maybe I'll sort of uh uh open models and and bottlenecks so um we you started by sort of talking about how uh the the evals were useful to assess models and uh you you know you you've alluded in our sort of previous conversations how today the bottleneck is very much in the review phase uh of it and and where we work over there.

SPEAKER_04So I guess maybe I'll kind of ask you a little bit of a joint question of it which is you know where where are you and what are your sort of views around the review bottleneck, you know, what to solve it and have you tried open what open models on those?

SPEAKER_02Yeah so so yeah based on our evals I think open weight model to be like fully viable assuming the bar does not move as progress happened in the space would be that they would need to perform about 50% better than they are today.

SPEAKER_04Okay interesting this is a this is a statement even for like GLM52 and like uh the latest or is it just like a notch below?

SPEAKER_02No uh a notch below uh we didn't yet run the Evals on those but we're very excited about this. Yeah yeah uh yeah um and the caveat being like also for the uh for the creativity of those models with like tasks that are unknown or you know larger tasks instead of pretty the uh execution part of it yeah um and so this we don't have signals from our evolves but we're looking at public evolves on to get some signal there uh and then getting user feedback.

SPEAKER_04Yeah. So at the moment when you run like say code reviews or it is like highly repeatable uh processes today you're still with the frontier models running those.

SPEAKER_02Yeah uh for code review today we are um we're having the background agents um that for example for code review the first one we release is like if your CI fail uh you know and it's like a linting error a formatting error stuff like that then like that's pretty easy to offset to like an agent to just go fix it for you so you don't need to like come back and like have to deal with it yourself. And so we're trying to like remove friction from from like different places. And unlike a lot of those tasks can be done because uh well by like like an open weight model that is like much cheaper because it doesn't need creative ED.

SPEAKER_02It's like a very clear pad like you need to run a command or you or you need to do some kind of something specific. And so our thinking now is like we probably can encode a lot of what we what like you would do out of claw code to background agents that are just going to take action on your behalf to like nudge things along. And then here is like finding the right balance between like what can we trust you know and like not changing core decisions of a PR you know like really more into like like nitpicks and like small fixes without changing the meaning of a of a code change that is being proposed.

SPEAKER_04Yeah. Yeah and this is consistent sounds like the your your view is there's a whole bunch of places in which uh it's inarguable that the agent helped like let's apply the agent in all of those places. Don't get into the argument of whether it's right or not like get it into all those places in which it can do it. And clearly there's always like a little bit of a halo like you know you might veer a little bit into judgment but mostly lean into the areas in which you can inarguably say this is correct. So can the agent do it? Can it not? Does that sound right?

SPEAKER_00Yeah that sounds about right yeah yeah yeah and I think it's like super practical and kind of uh and high impact and re and it sounds like in review world and I guess kind of in the repo world as a whole you're just sort of looking for these uh um surrounding annoyances uh to sort of resolve those and there's so much kind of meat on that bone that you're plenty busy sort of doing that you don't need to get into features quite yet yeah yeah yeah yeah correct hey everyone hope you're enjoying the episode so far our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development whether that's talking about the latest tools the most efficient workflows or defining best practices but for whatever reason many of you have yet to subscribe to the channel.

SPEAKER_00If you're enjoying the podcast and want us to continue to bring you the very best content please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

SPEAKER_04Alright back to the episode I'd love to talk a little bit about uh about um uh people and around sort of you know hiring so I know you've you've had a lot of learnings about how you hire and uh and and what is it that you uh advocate or like expect people now when you think about promotions or kind of their career ladder. So can you share a little bit about the learnings there and like where you are today and maybe some key learnings.

SPEAKER_02Yeah yeah yeah so um so personally I always hated leap code type of interviews where I think like they were low signal a way to like filter out some like candidates but I think like very often you know could just lead to like the wrong conclusion. So I've never thought that it were a nice way to hire people um I've never liked doing them personally and I've personally always felt like it it was kind of low signal basically so um so I think AI opened the door to uh to make a case to actually like review how we do interviewing especially coding interviews um and so um and so what we've done is we is like we've built a few squads to own different questions um and with AI in the interview it allow us to do much more than we could before and so it allows us to basically uh work in a much larger code base and asking like actual tasks that like somebody might be doing in their day to day and like assessing more signal than like yes they could write the code and in like 40 minutes and and like pass all all all of the tests but more like how's their how's their judgment how's their trade-off did they understand the code and so like with AI basically the new flow for most of the uh interview is going to be like here's a code base it's fairly large uh use AI to understand uh what it's doing what is going on what is this code base about uh and then afterward like let's like talk of an engineering problem so like maybe you want to uh extend this code base maybe there's like a diff, like basically kind of a code review that like somebody wants to merge should they what should we bring back in terms of like feedback to the code review.

SPEAKER_02Yeah um and so more like kind of real world scenarios and I think this can only happen with larger code base uh and and without the AI it would just not be possible to go through and understand the whole context you know it's like it would be more like kind of left to lock almost and so uh and so I think it it like really allowed us to like step away from lead coding into more actual real-world scenarios in the interviews.

SPEAKER_04Yeah no I love that it uh first of all it's closer to the reality and there's actually like a bit of a hidden question about your sort of agent foo you know like how how manageable are you at kind of navigating the agent and so you're you're sort of testing two things at once you know your your exact your depth in the in the agent usage and your ability to understand those. And I guess it does imply though that uh this is like an in-person type element you know there's no uh this isn't a kind of take home uh code review uh or like coding exercise it has to be inside and is that uh you just sort of accept I guess that's a uh a slight inefficiency in the sense that you need to talk to more people that you just take these with correct correct yeah and and how uh um I guess on the other side of it like when you think about people uh moving up the ladder and you think about people already in the job and doing it what has changed in terms of what do you uh assess them for for promotion?

SPEAKER_02Yeah yeah uh so we've reviewed our current guideline this year too um and so we got together with a group to like brainstorm about like what changed in terms of uh expectation like what is more kind of important today uh and so a few things to call out at the high level of like things that like we changed um one is for uh is for more junior roles I think we saw for everybody that like we had a higher kind of expectation about like the complexity of the work that they were going to own and like kind of push forward and so and so we uh updated you know kind of those roles um to actually reflect like the new kind of uh baseline expectation of just like you know yeah you can take a whole project you probably want some help from like somebody probably kind of understand asking the right questions um learning about you know the uh sort of core competency of like being an engineer uh but compared to before which was usually somebody would be in a support role on the larger project we're finding that like most of the time now they are they they are owning like kind of a whole work stream by themselves because they can now uh and then they just need to make sure that they're asking the right question.

SPEAKER_02And so like here we change a bit that definition for more junior roles. In terms of more senior roles, I think what we saw is like before uh well uh two things um one is in term of like POCing and uh experimenting and the second is like on on like judgment you know like should we do this? When should we stop doing something? Is this solution uh the right one to the problem the customers are like having and so like there's much more of like a judgment call being done. Yeah and then in terms of an architecture based right yeah yeah correct correct and on the POC side of things is uh I think before we would spend a lot of time on like documents and uh RFCs to talk about like is this the right solution for that problem and all this and I think like there's there's like definitely value to those questions but I think uh now we can know more certainty around like is this gonna work or not if we do a POC first.

SPEAKER_02And so you can validate assertion in practice by doing POC. And because it's so easy to like POC something with AI is like you can do a few POC and like compare those options before you decide into like which one you want to actually invest in long term and like really make production ready. And so it's like we have much more of like of like kind of a try things first and then bring the learnings from that to make the proper decision. And so that's a bit different from before where like you would need to think of everything because it was costly to create a POC.

SPEAKER_02But today yeah try something you run it and see what happened and then bring those learning back to then inform the right decision you want to do.

SPEAKER_04Yeah no I love that both as a behavior and as an expectation you know from uh from people and I I often say that like like now that the cost of building is lower the relative cost of alignment is higher because it takes just as much time to get people in a room and have a conversation. And so investing and building more to reduce the cost of alignment because by the time you align there's like something concrete and kind of more proven uh is clearly like the right trait for it. So I love this.

SPEAKER_02Yeah and I think it helps a lot in discussion to just have something more concrete to talk about. Or like you might have concern oh this is not going to work but if you had it in a few scene and it does work, you know, it might prove the point that like okay no this is possible actually so like give you a bit more certainty in your direction. So I think like it it changed with you the whole game in terms of like discussion of like what's the right design for a solution.

SPEAKER_04Yeah. So this was excellent. It was like just jam-packed with useful advice. I was gonna ask you what's your sort of single biggest lesson or takeaway but frankly you sort of shared a whole pile of sort of useful lessons here and learnings for it. So really really appreciate you sharing it. I'm sure the listeners are instead maybe I'll ask you like one different question which is what what are you excited uh on for for for what's next? You know what uh what are you looking for forward the most?

SPEAKER_02Yeah yeah yeah I'm I'm really excited to basically like being able to make better business decisions. And like that's kind of where we uh ended up on but like that's really my belief around AI. Like productivity is nice and all this you know but it's not like really kind of a game changer. Like but yeah being able to know and make better decisions I think is really uh the the key thing we are we are like gaming with AI and this is like really what I hope we will see happening in the business world is like hopefully we ship better product because we were able to try multiple things and see what works and what doesn't instead of like maybe spending kind of a whole quarter on a project that is like off ready then you ship it in a way because of you know you've like uh invested all of that time you need to show something for it and I think that like calculus here of like what can go out and like when you can be like this doesn't work let's like start from scratch and do something different with the learnings we've had I think it's much more possible today than it's been in the past.

SPEAKER_02And so I'm really excited about that.

SPEAKER_04Yeah that is super exciting. I love the uh we did a roundtable at AI Native DEF CON uh recently and uh someone said there that it's not about increasing productivity it's about increasing ambition uh and kind of I loved that sort of phrasing and it sounds like that's very much what you're talking about here.

SPEAKER_02So I love it. Yeah.

SPEAKER_04Cool. Simon this has been excellent. Thanks a lot for coming onto the show and sharing sharing all these uh learnings uh and for our listeners thanks for tuning in and I hope you join us for the next one.

SPEAKER_00The AI Native Dev is brought to you by TESL the package manager for skills and context your hosts are Guy for Germany and me Simon Maple. Our producer is Tom Dambler the AI Native Dev is not just a podcast it's a community and we host monthly meetups at the TESL offices in central London. Visit TEL.io forward slash community to learn more and we hope to see you there.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Your Software Factory Needs a Bill of Materials | Daniel Meppiel

    Nov 19, 20261 hr 1 minAudio

    Developers are about to write skills instead of code, and Microsoft's Daniel Meppiel argues the software factory that runs them needs versioning, a bill of materials and signed receipts before enterprises will trust it....

    Credit the people on this episode

  2. Paul Stack: I'll Never Write Infrastructure as Code Again

    Nov 17, 20261 hr 2 minAudio

    Paul Stack helped build Terraform. Now he says he'll never write another line of infrastructure as code, because an agent harness runs his team's production infrastructure instead. Watch all the AI DevCon NYC 2026...

    Credit the people on this episode

  3. Edith Harbaugh: Code Is Cheap. The Last Mile Isn't.

    Nov 10, 202649 minAudio

    Automated software delivery is moving faster than the people who used to approve every release. LaunchDarkly co-founder and CEO Edith Harbaugh tells Guy Podjarny why rollback should be autonomous, why one-shot coding...

    Credit the people on this episode

  4. Maggie Appleton: Agentic Coding Has Outgrown the Text Box

    Nov 3, 202652 minAudio

    Agentic coding has outgrown the text box. Maggie Appleton, staff research engineer at GitHub Next, argues that walls of CLI output and AGENTS.md files serve coding agents far better than the humans steering them. She...

    Credit the people on this episode

  5. Caleb Sima: AI Attackers Move at Machine Speed

    Oct 27, 202655 minAudio

    AI attackers now move at machine speed, while most security teams still move at meeting speed. Former Robinhood CSO Caleb Sima has seen an AI attacker up close, and he argues the only way to keep up is to take humans...

    Credit the people on this episode