After DevOps, Patrick Debois has a new map. The godfather of DevOps returns to walk through his AI patterns research: how he tracks where AI-native development is heading by mining social signals instead of surveys, why the stack is settling into patterns worth learning, and why your AI budget problem is really an optimization problem. What we cover: - Inside the AI patterns site at Tessl.io/patterns and how it's auto-generated - The four layers of AI enablement: agent, team, platform, and organization - Solo, shared, and multiplayer modes: the compounding effect of shared context - Managing agent costs: budgets as a forcing function for optimization - Scaling adoption: find the success story, don't fight the skeptics - Hiring system thinkers instead of elegant coders Chapters: 00:00:00 - Introduction 00:02:28 - Why AI grips Patrick Debois 00:04:33 - Agent, team, platform and org enablement 00:06:36 - Is AI-native development maturing? 00:11:19 - Inside the AI patterns site 00:18:14 - Quality, security and the IDE as review interface 00:22:45 - Changing roles and harness engineering 00:26:32 - Hiring in the AI era 00:30:31 - Scaling adoption across the org 00:42:03 - Cost management and ROI 🌐 Tessl: https://tessl.io 🔔 Subscribe for weekly episodes on AI-native development Where is your organization on the maturity curve? Drop your take in the comments.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
SPEAKER_01When you mention the word dark factory, it's like blasphemy. It won't work. I think technically it will work, but you're not organized around it. The continuous learning is where the knowledge compounds. And that seems to be becoming the moat of a company. Like, how fast can you learn? How fast can you adapt to a new ID? How fast can you do it? Like even if that means rewriting the whole code base, it doesn't matter. But like, how fast can you get that? And the people who uh contribute to that metric are the ones that are really helping you on the journey.
SPEAKER_01Not the token billionaires.
SPEAKER_03The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Pajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. We just wrapped up two amazing days at AI DevCon in London.
SPEAKER_00But the great thing is that we get to do it all over again in New York City this November.
SPEAKER_03You're absolutely right. We're gonna be back in the city that never sleeps on November 3rd and 4th for more amazing sessions, really engaging, hands-on workshops, and much more.
SPEAKER_00Yep, all that great networking, partying, eating and drinking that you've come to expect from AIDEVCO. We think we have one of the best hallway tracks in the business, and it's the perfect compliment to our incredible speakers and presenters.
SPEAKER_03We'll both be in person and virtual with live-streamed access to all mainstage keynotes and talks.
SPEAKER_00Sign up right now for our super blind bird ticket for just$100, only available for a limited time.
SPEAKER_03We're really excited to be headed back to the big app. We hope to see you all there. Hello and welcome to another episode of the AI Native Dev. We're still here in San Francisco enjoying AI Engineer. Um, and we've managed to grab Patrick Dubois a third or maybe fourth time for the podcast. Okay, yeah. I'm not sure which, a regular, yeah, yeah. Um so actually I've I've I've talked to a number of people who have said that their favourite episodes are with Patrick Du Bois on the on the podcast.
SPEAKER_03You know that?
SPEAKER_01Do you need something from us?
SPEAKER_03No, no, no. I'm just uh I'm just I'm just mentioning it. Thank you for the observation. Well, it's not business, it wasn't my compliment. I prefer the guy ones personally, but uh uh no no, that's fine. So uh Patrick, uh you are obviously someone with a wealth of experience in a number of places, of course. Should I mention DevOps? The uh you're the stepbrother, I think, of DevOps, right? Um and AI has really like AI has really gripped you like more than anything else that I've seen since the DevOps days.
SPEAKER_03Uh and DevOps days I use as a as an uh a pun there on in terms of the the the conference, the set of conferences that you that you built. What is it about AI that that really grips you so much?
SPEAKER_01So I think I've always taken when there's like a new technology emerging, and so that you know happened actually early on with the internet, with cloud, kind of with kind of the collaboration agile, and then leading to DevOps and then DevSecOps and mobile. So anytime there's something changing, uh, and I would say I I like kind of the this emerging chaotic phase of a new technology, but not just for the technology, but also kind of the impact on the organization and the way we work, like uh, and that's kind of the typical narrative within DevOps is the socio-technical kind of ecosystem.
SPEAKER_01Yes, and uh I kind of had a hunch that this AI thing was like automation, kind of blowing doors of kind of the ways that we were working, and that's kind of been you know captivating around it.
SPEAKER_03And one of the things that you've done time and time again with these kind of major shifts in the industry is the kind of like deep thinking and understanding of what needs to change and how it needs to change. Obviously, there were a lot of these changes for cloud, for DevOps, and and and these these types of uh you know motions that that we've experienced. With a gentic development, with uh with AI generally, exactly the same has happened. And and you've done a bunch of research and work to try and help people understand not just the the technical challenges and changes they need to make, but also the process changes, the the organizational changes.
SPEAKER_03And in fact, that's what you're speaking about at AI Engineer. Tell us a little bit about your talk that you're giving tomorrow.
SPEAKER_01Yeah, um, so every new technology will have the impact in the org. And I feel right now um a little bit of the technology, I'm not saying it's maturing because that's a broad overstatement, but we're we kind of get a sense of direction now within AI coding, how it's going, you know, from context to harness to loop to kind of what is mentioned, loop craft, loops of loops, and and so on. So the technicalities are improving, and what you're now seeing in the org is the shift to what does that mean for a developer on the day-to-day?
SPEAKER_01And and that's kind of one way, and that's often talked about. But what does it mean for a team and a team leader to kind of deal with that change? Like maybe like one of examples is well, if your team gets more performant, um, are you running dry on requirements? And can people who depend on you for the marketing and the sales, if you have a hundred features a day, how do they package this to the end user?
SPEAKER_03So there's an impact on the without massively confusing them every week or yeah.
SPEAKER_01So you have that on a team. I can see that already a little bit like happening, maybe more in a shared component, like a platform team who's kind of taking there, like the if the loop of loops evaluates to a harness and that becomes the more of a factory. Is that like reusable across teams instead of each team doing their own kind of reusable components? So their life is changing. What do they need to provide for kind of the new developers and the new teams doing the work? And then the VP of engineering kind of how do they kind of structure the org around that?
SPEAKER_01So for me, level zero, technical agent enablement, this is the whole conference, right? Team enablement, platform enablement, and kind of uh making sure your organization is enablement. I think those are like not enough talked on.
SPEAKER_02Yeah.
SPEAKER_01But it's also because the one is changing so often that we have all their narratives in all the other layers. And that's kind of where I want to bring the stories at.
SPEAKER_03And it's really interesting that you know, we when you talk about you kind of like mention the word maturity there, and it's like, where are we on this maturity curve? Because, you know, for me looking at it, there is no maturity. Because every single time we try and get used to something, there's something else that changes, whether it's prompts that we need to start looking at, then it's context, then it's harnesses, then it's you know, loop engineering. And it's like every single time we're we we think we know what is the area that we we need to spend time on and do research around, there's always something round the corner. So when we look at maturity, are we just in a continuous exploration and continuous change?
SPEAKER_03Or do you feel like there are things which we can longer term say, hey, these are things that we need to really focus more on to actually get more mature practices and processes that we expect to keep a long time? Or as teams, do we need to actually build that muscle which is more about continuous change and how do we continue to adapt? What what what would you say the goals are for teams in that respect?
SPEAKER_01So I think there is a settling in, like yes, prompts and specifications will be a keeper. Um, yes, harnesses and context will make sense, the loops will be so I I don't think this is changing all the time. It is adding on top of kind of the stack that it's building, and I think that's quite common in this emerging times where you're thinking like, oh, this was the first slice and that was revolutionary, and then they build on top of that, and that that is kind of like building more and more.
SPEAKER_01I think I don't know what we're you know, technically in the code generation. If you have teams of agents building on the whole thing, then then kind of that is already uh you know one way of kind of solving the coding thing. Where we might now see things is, but um, how do we prove that the agent did the right thing? So something that we've been so focused on the generation will move the bottleneck over there, or how can we uh assess the risk of things that are coming in?
SPEAKER_01So the the as the craft matures, it's normal that you feel a little bit lost because every time there's a little bit of a different attention level, but I think it's the industry maturing as what is actually the kind of the way the technology kind of works. Now that's a different thing from an organizational maturity level, which might not be in the known of the latest, which could be in like still working in the old kind of version of completion and think it doesn't work for me and making bad decisions on this.
SPEAKER_01Um, and that's kind of a challenge in that transient period of kind of like while the stack is maturing. Now, will it keep evolving? The rate has been astounding, that just kind of change over change. Uh, but you can also see some it's it's not it wasn't unpredictable that we'll reach in kind of the you know an agent loop, agents of agents. That was kind of predictable, but you need the industry to start building that layer out so it becomes accessible for most of the people instead of having to do everything themselves.
SPEAKER_01Yeah, uh, but that's the race that we're in, is keeping up, and for some that is keeping up the uh competitive advantage, like almost being in the edge, and where others it's more like, yeah, you know, we want to work the new way, but like we're not that fast, we don't have the risk, or we have a lot more things around there.
SPEAKER_03So where would you say kind of like average organizations are in terms of maturity? Are we are we toddlers? Are we early stages of puberty, maybe?
SPEAKER_01Where would you say we are we're we're I think on average, but not that high in in a big org because the the friction of scaling things in an org are different than doing it on a solo or maybe in a team of five. Uh, there are stakeholders, there's like processes in place that will kind of cause that. So I think it's normal that um an enterprise is kind of slower. Now, the typical pattern there is they might have one team who's kind of like you know leading the charge, and then that becomes like multiple teams, and then they kind of work on the friction and overcome it.
SPEAKER_01Um when maybe you know, with the comparison of DevOps, when they said, well, continuous delivery, that doesn't work for us. It was basically they were saying we're not ready yet. It wasn't that the tech was impossible for them to use, but the org wasn't ready. And I think when you mentioned the word dark factory, it's like blasphemy, right? Like similarly, like it won't work. I think technically it will work, but you're not organized around those.
SPEAKER_03Yeah, so yeah, super interesting. Now I'd love to talk about um a new um initiative that you that you've built up over the recent months uh called the patterns, agentic patterns. Um and folks can take a look at this, tesle.io forward slash patterns. Now, this was uh a little research project that you built, and I'd love to actually maybe begin by talking about how you actually built up all the information that's here. Um, first of all, tell us what is the what is the patterns site?
SPEAKER_01So as the change of the industry is faster and faster, what you typically have is a fragmentation of kind of news, and there's a fire hose of news and things that are changing, and there's the old information, there's the new information. So my job is to listen into the emerging things that are happening and kind of also understand how it's like stabilizing as a thing. So the way that I would normally do that is almost like I would scroll the social media all the time, listening to like conference talks, finding like new IDs and kind of solidifying that.
SPEAKER_01Now, then I would maybe create a talk of that. But what I'm now trying to do is to build this almost in an auto-generated website and similar to code, it has the same problem. Like if I would just have it generate, it will be rubbish. If I had like my own context to it, like my curation, my taste, my harness, it would be say, well, not vendors or kind of like things or like it has to have four voices. So I can put this up. So I I kind of built almost like the research chain.
SPEAKER_01It's not the deep research per se, but kind of curated and kind of make this uh uh a tool that I hope is useful for people to say, like, hey, I need to talk about the defending the ROI to my CFO. What's the latest thinking about that? Because there's been a few iterations, like one is like you know, spent uh through the roof and then it changes, so kind of keeping that up to date. And one of the other reasons was that if you do this through surveys and you ask, like, what is changing in the industry, I feel that is too slow.
SPEAKER_02Yeah.
SPEAKER_01Uh, because first of all, imagine you would do a survey now in an organization about loop engineering, everybody's like, uh what? Yeah, while there is already the evolve uh like evolution within the industry, that this is where the thinking is going. So I think it is, I find it more valuable of picking up social signals on the socials to kind of see where it's heading and kind of start extrapolating that and um yeah, consolidating that for people, everybody to access it.
SPEAKER_03Yeah, and you added it, uh I think in the the implementation that you used to actually pull that all together, you used the Carpathi uh wiki uh things asked it a whole bunch of questions and pulled pulled data from that, which is super interesting. Um, and actually we can we can take a look at the pattern site here. Um, and and from from all of that data, that you then displayed it in these in these kind of categories of agentic coding. Walk us through, walk us through, I guess, the sections here that you have, uh five sections uh in total, and and and tell us a little bit about each.
SPEAKER_01Yeah. One thing to note is that um this index is hand created, and I I found that if you learn it uh through a bunch of socials, the naming varies and it kind of like you know, everybody has a different slant on it. So I think I I kind of settled into these different groups where you know, agentic development, uh everything maybe more what the solo developer is doing, so from vibe coding, spec coding, all the way up to a dire factory. Like, okay, what are it's almost a progression as we talked about, like when you're kind of building this out, the first thing is yo, you start with some prompting, oh, you get a better spec.
SPEAKER_01And I think this is also how the industry has built up the narrative over time on how to do things more technically. But then the second category is the platform, which I feel the more we're heading into reusable components, whether that's context or harness or the centralized pipeline of the Dark Factory, I have a belief that that will be something provided by a platform team or the developer uh experience team as a central piece. So they have already pieces, maybe the MCP proxy and kind of those parts, but slowly they now have a new uh set of things like you know, centralized evals for everybody, for all the teams, a registry.
SPEAKER_01So they're building up kind of those uh components around that.
SPEAKER_03And this feels like quite uh an immature section in terms of adoption from larger organizations, but this is very often the piece that unlocks large organizations and helps with scaling, particularly when we think about you know those big questions that people you know need to ask about or how are we protecting against X? Well, we need these guardrails, we need these, we need these policies in place, and and this is all about that kind of like distribution across a large organization. So this is something that we probably need to, as a as an industry, get better at.
SPEAKER_03We'll see more and more solutions. Whereas that first category, it feels like you know, there are people who are getting more and more. Yeah, exactly. That's driving a change. And people who are successful here in the agency development, that that drives the need, the adoption and the usage across an organization, which then mandates, well, we need this agentic platform to be able to support everyone who's doing things in this agency development.
SPEAKER_01Correct. Yeah, so it's also the difference between uh some explain it by saying there's a solo work, uh solo mode, there's a shared mode and there's a multiplayer mode. Yeah. So you you unlock a different level if you go from solo to shared, and then you unlock the thing. So there's a compounding effect in if everybody starts using the same components, there's a refinement. So if one team kind of optimizes it, everybody has an optimization if you add context for those pieces. No, I think you're right in a way that it maybe is a little bit less mature, but on the other hand, it's not.
SPEAKER_01Um like AI product, AI inside the product is probably two years ahead from AI coding on components. They already have the eval system, they already have an observability system. So it's not foreign to platform teams per se, but the use case for using it for coding is probably unique in there. And then it has a different way for a proxy, how you do evals, it's not the model, it is the coding agent. And so there is some newness uh building up.
SPEAKER_03So it's almost like we need to potentially reuse existing muscles that we already have and know-how to do things, but just apply them slightly differently to the to the newer ways of working. And then quality and security again, that's that's something that is is is more painful as an organization as you scale out if it's not done well and if it's not done in a way that is, you know, it is whereby people know what they need to do. Uh it's similar, I guess, with writing code, right? If you quality and security, it's it's it's painful if you don't know the guardrails and the ways of working. Yeah.
SPEAKER_01Um, talk us through this section. So I was thorn between putting quality and security in a separate section where on the one hand, it does belong in agentic development in the harness, in the specification as non-functional, and so on. But I I found that it kind of leads a little bit of its own life. And I would say that while agentic development has about cranking out the code as a focus and making that code correct, um it wasn't about verifying that it was kind of good code.
SPEAKER_01Now, harness and loop kind of changed that a little bit, but there's still QA testing, there's the code security, so all the kind of non-functionals probably belong in its category. And I think that will lead up to um if we remove kind of the bottleneck from AI coding, we solve the specification problem, which is the input. Now we need to solve the output verification thing. So I believe this will go on its own path, where maybe some of the latest trends are that after your harness has done things, it's the job of the agent to uh uh convince you that it did the right thing.
SPEAKER_01So not just giving you the PR and looking at the code, but it needs to start like helping you do that verification. Now, within evals, you have an evolution as well. So imagine you are you write your uh your your code with your prompt, you have it evaluated with an LLM as a judge, that gives you a lot of like input and output. Like, yeah, is it correct? Or you have to look at the app. What you see the more mature people do is they build a verification and an almost like a custom tool to deal with all the e-files, like a fast way of saying yes, no, where I agree that it proves to you.
SPEAKER_01And I expect something to happen in that quality section as well, where uh there was a lot of tension between like will we still need an IDE? Yes, no. I believe the IDE for writing the code has moved into the CLI, but it's now popping up again as the review interface, which is the IDE is yeah, interesting. And I think that's a move that's gonna be made where that verification and kind of that proving, and it's almost like situational awareness for the user saying yes, no.
SPEAKER_01Yeah, very interesting.
SPEAKER_03And why would you say why would you say that is that it would go more to uh uh an ID interface?
SPEAKER_01Is that just because you would expect um someone who is reviewing it to want to look through the code and understand well it might not even be the code, but the fact that they can verify certain features of the um API of the website more easily, that there's screenshots, that there's like uh you know recordings of what the agent clicked on, and so there's a lot more visually interaction than you would say, like here's a checklist in my my terminal. Uh I personally still use kind of like the CLI inside of the IDE just to you know, sometimes to review, sometimes to that.
SPEAKER_01And but I think that is the well, just the current evolution is there's an orchestrator thing uh within the IDE. And then there's the evolution about like jumping in when the agents are failing. Uh, I think intent from augment is kind of doing that, that you can step in when it fails, and then it's like about having the review uh making it easier. Again, it's all extrapolations. Uh, it's not that we have a crystal ball where the industry is going, but but that's why I think I granted quality and security, it's kind of own section.
SPEAKER_01As that I think there will be an evolution there. So very much.
SPEAKER_03Earlier you talked a little bit about how us humans, the the meatbags, are harder to change in terms of adapting to a new process or adapting to use new technology. The last two are very, very interesting because changing roles is key. And I think this is this is probably the hardest thing to predict as well, in terms of how we adjust will and how fast we adjust will very much depend on what the short-term view and shape of the org should look like, and that will be different to something longer term.
SPEAKER_03And then scaling the org again, um how do you drive adoption through that, you know, those new roles and folks. So let's let's talk a little bit about, I guess, the changing roles and changing shape of the development organization. How do you see that going?
SPEAKER_01Yeah. So there's obviously more of the pessimistic view of AI can't do this, or we should not be using AI. Okay, that we shouldn't use AI, that's more of a conviction that than kind of something else. But it it's not there yet. I would typically say it is a maturity in a way that you can add more and more rules and context and harnesses to get it in a certain way. So one thing I would say is that a skeptical person get them actually to write the context and have them build a harness, right?
SPEAKER_01Where before it felt like it doesn't work. What can we do? Don't use it. But now I think we're up to a level where we can say, well, you can make it better if you put the effort in. So it's also one of the reasons where people are saying, like, well, we're not using it because it's not good enough, because I haven't been working the problem. Now the other problem in the industry is that they would say, Well, we like to do the coding. We signed up for the coding, we didn't sign up for a glorified prompter as specification.
SPEAKER_01And um, for a while this stagnated into yes, we'll all become the perfect spec writer and the code is being generated and so on. For me, harness engineering and loop engineering maybe click that that requires a technical component to be built. So maybe the people who didn't sign up only for kind of the context, they will be working on the harness and the loops and kind of working on that. So that kind of maybe caters for both audiences who were okay to jump, maybe more in requirements and now if you scale this out into an org, the question is how many people will work on the harness?
SPEAKER_01Uh maybe that's a subset of all the people that we currently have because it's a shared component. Now, if a component gets shared, it becomes more complex.
SPEAKER_02Yeah.
SPEAKER_01So there's kind of some things, and it was uh similar to DevOps. Well, you know, DevOps will automate themselves out of a job, but on the other hand, they could do like things at a scale we didn't imagine before, and then we pull the people back. So that that is still torn, but I think there's now a better outlet for people who still want to do technical pieces uh and kind of make sure that the agents do a good job and kind of there. There's still a part in the review that I mentioned that they can probably start building that review and kind of helping in that piece as well.
SPEAKER_01So with that, that hasn't been solidified into organizations as well. So but that's kind of where the roles could go. Um as a team lead, I think the trend is that you would instead of uh encouraging the solo people to uh improve their own setup, is to push them to a shared thing. Like, can we reuse your skills, my skills? Can we put like improve that like as a team?
SPEAKER_01Yeah, so that driving is something I think the team lead does, but it was similar to can we build a shared library instead of everybody inventing their own library? Uh, and that kind of is a collaboration feature which which I would encourage as a as a team lead uh to go to. And then kind of you know, will be they become more product engineers. Uh that that's debatable. Uh, but they have options, they could do more product engineering. I think at Lovable they would say 80% is working with customers, 20% is improving the technicalities, but it's it, I guess it's it's not for everybody in there.
SPEAKER_03So and one of the things here that I kind of like, and we'll go maybe we'll jump into a few of these, um hiring in the AI era. This is you know, what this is super interesting because what what's what what's really kind of challenging, I guess, is what made a good developer three years ago no longer makes a good developer today. It's useful to have very you know key insights into you know what good architecture, what good reliability needs uh mean we need to kind of like add into into the application or into our design.
SPEAKER_03Um, but in order to actually be a productive engineering team uh to use AI to the extent that we can today and look to continuously change to continue using it to the to the full extent, it feels like we need a very different type of person, one that is very open to throwing themselves into AI and challenging themselves to continually be better, uh trying all the new cool stuff and adopting where they think it can benefit.
SPEAKER_03What what what would you say a good if you were to hire an engineer today? Um what what are the key things that you're looking for in an engineer today that you wouldn't have said these would be key things five years ago, let's say?
SPEAKER_01The way that I would uh phrase it is somebody more as a system thinker. Uh is that architect? Is that kind of like you know, they they care about different things, not just about the code. So that often qualifies probably more senior people, but it doesn't mean that it has to be senior. You can be as a junior that you're interested in as well. Um probably not the person that will write the elegant code that that is kind of you need to have an eye for taste, but it it's not the main thing.
SPEAKER_01You'll be improving the system there, and then kind of the collaborating skills and the improvement skills that you're not there for the solo craft. Like if somebody says, like, I like language X, okay, that's great. Are you only coming if we do this language? Yeah, okay, no, you you need to switch languages uh in that way, similar to a full stack that we defined before, front end, backend, it doesn't matter. You can navigate a bit of the different spaces. Now the problem is like we used to be learning this maybe by having scars like step by step in our career.
SPEAKER_01So that's why uh it was very similar again with with the DevOps where it were like people in their 30s, 40s that kind of got enough from the scars, learned enough that they were adaptable uh from there that we see here as well, picking it up. Now, if you have two rigid ways of working, that like that is probably not where you get hired. Uh but yeah, being open-minded, discoverable, learning, I think the the the willingness to learn. So I would typically probe it's okay, how do we keep how do you keep up what communities are you're interested in?
SPEAKER_01And if they only say their own language, their own stack, that that's very limiting uh there as well. So very interesting.
SPEAKER_03And and and to go back to the um uh to go back to to the the patterns, I think it's that type of person that actually, when we look at scaling the org and driving adoption, it's that it's typically you don't when you look at driving adoption, the org, as you mentioned, they they mature at different rates, and you will have teams, typically that style of individual that you just mentioned, you'll have a team which has probably full of those types or has enough of that type to have uh a big enough momentum that that will be the team that is on the bleeding edge, that's always trying new things out, that will learn, oh, these are the best ways of doing things that they can then educate the other teams, they will get the the the ROI that other teams will then strive for.
SPEAKER_03Uh so let's talk about scaling the org. Um how um there are a ton of enterprises that will have, uh, or even kind of like medium and large companies that will have that will want to be able to scale their AI adoption faster than they are doing today. Um, what in your experience do you see as some of the some of the biggest things that are holding people back in in the kind of area of driving adoption that you feel they can make?
SPEAKER_03I don't want to say easy wins because this isn't an easy, this isn't an easy thing to to adjust, right? But that people need to focus more on in order to unlock greater adoption. Yeah.
SPEAKER_01Um it's a it's the question that I get all asked all the time, how can I speed up my organization? And um there's I guess there's two parts of it. Like some parts you can't speed up. Um you have to have gone through a little bit of the prompt engineering to know, oh, I need more specifications, I need more kind of context, I need more uh harness. And so there is a learning phase to kind of understanding parts of those uh components uh that people go through. So all of a sudden saying you're going to the dark factory, like people are like scrambling, they can't do that.
SPEAKER_01So that's not per se, uh you can speed it up by giving people information, learnings. Maybe as a team lead, you can kind of say, hey, let's start doing this first uh step in there. And I think that's where they can make a little bit of the pace from the team, where they say, um, okay, now everybody's got a fair understanding of what prompting is. Let's move and put all the context and the skills in the repo. That's that's like a forcing function they can do that.
SPEAKER_01All of a sudden, oh, if we do that, we need to have tests, and I don't trust your context. And and so they can pull a little bit of that pace, how the team jumps to the next thing. So it's almost like an automation that unlock yeah, like you know, but they control the kind of area. Well, I think everybody's ready, just jump. If they don't give the signal jump, everybody will keep on doing the same ways. Like, oh, that's limiting. Oh, we're using completion, we're not using rules. So I think there's a um a value in kind of making that cadence of learning, uh, like either the team or the whole org, like how do you jump in from solo to more shared, for like from simple practices to more advanced.
SPEAKER_03Uh and when you talk about these changes, is it more important for an organization to focus on the teams that are the most advanced? Or do you feel like there's a greater win with the majority? Because typically the way I look at it, you'll have an organization where you'll have a high inertia teams, teams that don't want to change. This is how I've done it, get off my lawn. Um, you've got the very fast-moving teams, the teams that just really want to experiment and they they want to push technology as far as they can go, they're happy to change. And then you've got the majority, like the 80% of teams that are willing to change, but when they see value, and almost like they are led by examples, good examples of teams that are leading a certain area.
SPEAKER_03So, where would you say like an organization, an organization can only spend you know its effort effectively in certain areas, where would you say it's most effective for an organization to focus to get the best benefit longer term to improving their overall adoption?
SPEAKER_01I think regardless whether this is now AI or like agile or any change, um, as hard as it sounds, do not spend time on the negatives in the beginning. Find the success story. If you can't make that work, the rest will not work. So it's the super high Yeah, because they will show what it could be, they will also show the benefits uh and attract like other people trying this to get like the same benefits, and maybe they'll look up to these people, like, hey, what did you do?
SPEAKER_01and so on. If you spend time on the negatives, you're not gonna get that like same thing. So that becomes later when you have a more adoption already, then you're like, okay, why are you still resisting? We've offered you all the information, we've seen scale like parts of it in the org. What is holding you back? And that could be a variety of reasons in there. Now, it probably makes sense is not to over-rotate on one team going all the way and nobody else. So I would say there's like a leveling up, they spearhead the thing, and then you bring others maybe on kind of like parts of the journey already, and then they kind of like drive kind of towards what the one team is doing.
SPEAKER_03So is it a fair assessment to say you want to make the fast teams go as fast as they possibly can and allow then, and then the the enablement is more for the teams that are following to say, let's share the and let's share and educate them based on the best practices that these fast teams are going. And how can we enable you to take practices that work for your teams that you know learning from these from these almost like leading teams?
SPEAKER_01But it's a typical transformation play. I have a hackathon, I share stuff, I have a lunch and learn, I share the success stories, I celebrate like whatever team is doing the best thing. So that that that's kind of very common in kind of driving adoption. This is DevOps, this is cloud, this is developer security, it's all the same styles. Correct. Now we're given all the success stories, what and that's the reason why I talk about the organizational enablement. A lot of what happened in the organizations is that well, we'll give everybody the tools, yeah.
SPEAKER_01And some will pick it up and they go all the way and kind of they build their own future and and they become very proactive. Um initially, everybody was measuring adoption by saying, well, is everybody using a tool? Okay, so that's one way of saying, like, okay, there we have some kind of uh scaling within the org. Then people said, Well, we're gonna look at like the people who go crazy on the tokens, that's usage, right? It's a substitute proxy kind of measurement where you say, Well, you know, the people are using more tokens, they're like at least enthusiastic about it, but they might not be using this quite efficiently.
SPEAKER_01If I were to put one measurement right now is how much these people are contributing to the shared components and they're fixing the system instead of kind of using more or having a tool. Because if you that gives the more uh buildup of like people improving the shared context, because one context improvement impacts other teams, and that is a typically another skill from like I can be personally very effective to this is the skill that like helps the whole org, and that's a different measurement.
SPEAKER_01And the measurement there is how many touches does the human still need to do for an agent to do the right thing, and so kind of measuring that becomes more effective than measuring uh you know, kind of adoption just by having license counts. Uh, and if you put that into practice, you'll see like like sharing uh across actually helps that metric to because if you fix it for one, your numbers go up in all the teams.
SPEAKER_01But it's uh it's I guess that's a driving towards that like more almost like autonomous factory that we're building. But I think that is the new metric. Uh, who the people who uh contribute to that metric are the ones that are really helping you on the journey. Uh not uh the the token billionaires.
SPEAKER_03Yeah, yeah, no. So a lot of the conversations that we've had here, it kind of like reminds me of um some of the work that we're doing on Tesla Agent, which is which is you know really maps nicely to some of the ways that we that you're talking about the growth and the scaling. Um, first of all, if we go back to when you were talking about creating context, sharing that context uh and making that context the best it can be. I love the way that you kind of mentioned for people who are who don't want to move to agentic because they feel like you know it's not good enough, it's not where it is.
SPEAKER_03I love the fact that you say make them do a lot of work on the skills because that's how you actually get the most out of it. Now, Tesla Agent, of course, uh is is an agent that allows you to uh to really make those changes well, not just um you know you allowing the agent to make those changes with your guidance, but also based on facts from logs and pull requests that have been uh historically done in those projects. Um so that that feels like a really strong way to kind of like map Tesla Agent to some of those things, but also the way that Tesla Agent doesn't try and provide you with a software factory out of the box.
SPEAKER_03What it does is it provides you with a path so that over time you'll get closer and closer and closer to a software factory. So a big piece of this growing step by step and and and you know, optimizing as much as we can do uh in our context and our skills and things like that, as valuable as that is, we need some level of automation to make sure that this is being done continuously. Um so so how can we, you know, what what's a good process of being able to build that that automation of the optimization into our in into our daily work?
SPEAKER_01Um maybe I'll first say kind of the parallel with DevOps, right? A lot of it initially was like, oh, can we deploy faster? Can we kind of make sure it's happening? But then the whole monitoring became observability, and ultimately that was the feedback channel of the automation. Like, what do we need to improve? Right. So, and and I think part of what you see with the agents, whether that's your your skills optimizer that you can put in the loop that it makes you better, uh, whether that's your harness tool that detects uh things you're doing over and over again that could benefit maybe from better context or becoming a script or a deterministic thing.
SPEAKER_01So kind of those feedback loops uh will help you scale this out because you can only do so much in like you contributing the system, but if it's almost like a positive feedback loop that improves this over time and gives you hints uh like what to improve, I I think that's really powerful. Now, the danger of a loop is that it could be a positive enemy. So, you know, once a rule is accepted, it could go wrong, but that's why you need testing, the regression testing, uh, also for harnesses, for anything you put in there, that it's not just I edit and I pray whether it works.
SPEAKER_01Uh, but it's it's crucial because I I I often say if you have continuous delivery and everything's automated, and now with AI, you know, another part is being automated, the continuous learning is where the knowledge compounds. And and that seems to be becoming the moat of a company. Like, how fast can you learn? How fast can you adapt to a new ID? How fast can you do it? Like, even if that means rewriting the whole code base, it doesn't matter. But like, how fast can you get that? And and you can't do this with without a feedback loop uh helping you to improve that.
SPEAKER_03And this is a great target, essentially, to make sure we can become as effective as we can be, but it all comes at a cost as well, right? And and and there's a section here, obviously, in on scaling the org, which is cost management. Now, last year people were just kind of like very much free to let's try and experiment as much as we can, let's try and get as much usage as we can to understand what works, what doesn't. This year, budgets are a little bit more constrained, and people are thinking more about the return on the investment and how you manage the cost of everything that you're doing. Tell us a little bit about um how important I guess the optimizations are and this automation to make sure we're reeling in a little bit of that cost and we are getting a return rather than just you know those billionaire token users that we hear of.
SPEAKER_01I've been in the position as a VP where I had to say, like, oh, look at my budget, and the first year that AI came out, there was no budget for AI, so where am I looking for it? Um, so I know it's a struggle, and especially with like a lot of the vendors raising their pricing models or changing their pricing models that we didn't cater for. Now you can have the instant reaction, say, Oh, well, everybody stop doing, let's remove, kind of like the like reduce the budget and uh kind of make sure that like we we we keep within limits.
SPEAKER_01The problem then is you won't be going on that journey of learning because if you're limiting people, they they kind of like work wrong, oh I'll still do it manually because I don't have enough tokens and kind of go from there. Now, others people have kind of start assigning maybe more use cases. So instead of it saying it the agent got better now, is it actually useful for that? So I think the way you have to look at air or and and cost management is you know, I need to have a budget for people learning, and I need to have a budget that's useful, but the constraint of a budget should actually drive you in optimizing that loop because a lot of the people are blindly spending tokens, they're using Biggest model out of things, they are doing the same thing over and over where there could have been a context change or a harness change that would save them a lot of tokens.
SPEAKER_01So if your organization is not your mature, you feel like you're over, you know, using things. And um, the parallel in the cloud was like when everybody went to the cloud, every system was a VM.
SPEAKER_02Yeah.
SPEAKER_01And everyone's like, this is proving more costly. And then we learn techniques to put it like on the same instance and kind of like work from there. So I think we're in that phase, but it's it's very risky now for maybe a VP to say, let's close down the full budget and kind of stop the usage, because then your organization doesn't have that. I think the the right reaction is where are we open spending? But you need to have like observability for your FinOps, for your coding agents, and so on.
SPEAKER_01So you need to work on the telemetry and then say, well, there's no habits, don't use this for that. And kind of have a team work on the optimization instead of just closing the budget off.
SPEAKER_03And let's say, you know, as a VP of engineering, uh, would you say, actually, of all my of all my children, all my teams, I do have preferences over where I want to spend money. And actually the teams that are the highest performing, the ones that are experimenting, I almost want to give them free reign. Whereas the others, I do want to make sure they're that they're not just you know being ridiculous with their spend because we're not actually learning things as much from them. We're just trying to it is about ROI, it's about can we do things better in a cost-effective way?
SPEAKER_03Whereas these other teams are actually, even if they're spending money and not doing that, we're learning, okay, that's not a path to go, but it's money well spent.
SPEAKER_01Yeah, yeah. And I think that's the struggle you have with the CFO who says, like, oh, where's the ROI, where's the ROI? Like they're willing to spend if there's a return. And I think it's not per se that I would say the you know the best agentic team will get the budget, but I'll put the best agentic team on the most important business project to get like a return uh back. And and I think that's kind of the the challenge is uh they're not always the teams because the the biggest use case might be the more most risky use case as well.
SPEAKER_01So there's a lot of nuances to go there.
SPEAKER_03So yeah, no, it makes ton of sense. Um, Patrick, this this session's gone very fast. I love I love these, I love these chats learning from you. Um, and for for our listeners, uh Tesla.io forward slash patterns. Yep. And you can read all the wonderful um you know research and data as well as as well as some of your takes and your your categorization there. This is not a static site though, this is changing over time. Yeah, um, and there's a lot more that I think you're planning on adding to it over time. Um, so take you know, absolutely take a look at that site, very, very useful set of resources for you.
SPEAKER_01Yeah, and if you find anything missing or you have an interesting story, I'm also very interesting to listen and learn from you.
SPEAKER_03So thanks very, very much. And I hope and I'm sure everyone at home enjoyed that session. Extremely insightful. TESILIO forward slash patterns to learn more. Thanks for tuning in and see you next time. Bye for now. Thanks. The AI Native Dev is brought to you by TESL, the package manager for skills and context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the TESL offices in central London. Visit TESL.io forward slash community to learn more, and I hope to see you there.
Developers are about to write skills instead of code, and Microsoft's Daniel Meppiel argues the software factory that runs them needs versioning, a bill of materials and signed receipts before enterprises will trust it....
Paul Stack helped build Terraform. Now he says he'll never write another line of infrastructure as code, because an agent harness runs his team's production infrastructure instead. Watch all the AI DevCon NYC 2026...
Automated software delivery is moving faster than the people who used to approve every release. LaunchDarkly co-founder and CEO Edith Harbaugh tells Guy Podjarny why rollback should be autonomous, why one-shot coding...
Agentic coding has outgrown the text box. Maggie Appleton, staff research engineer at GitHub Next, argues that walls of CLI output and AGENTS.md files serve coding agents far better than the humans steering them. She...
AI attackers now move at machine speed, while most security teams still move at meeting speed. Former Robinhood CSO Caleb Sima has seen an AI attacker up close, and he argues the only way to keep up is to take humans...