Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow artwork

850 PRs a Week: How Tessl Runs a Software Factory

Sep 2, 202651 minAudio

The software factory that put up 850 pull requests in a week, with 85 to 90% of them handled by agents end to end. Tessl's Head of Product Dru Knox on how it actually got built, why it started with skills rather than automation, and the two things nobody expects a factory to change. Guy Podjarny digs into the skills-to-loops-to-factory path, and what breaks when you skip a step. What we cover: - What a software factory actually is, and how it differs from a pile of disconnected automations - Loop engineering: turning a skill into something that gets better every time it runs - Why context engineering has to come before automation, and what stalls when it doesn't - The two gains nobody forecasts from extra agent capacity: quality and fungibility - Verifiers and evals, or how you enforce a standard that agents are free to ignore - Context-driven code review, and why a general-purpose reviewer plateaus Chapters: 00:00:00 - Introduction 00:03:23 - Kikimora: Tessl's own dark factory 00:04:45 - Skills, loops and factories, defined 00:09:09 - 850 PRs a week, and the road to it 00:11:27 - The surprises: quality and fungibility 00:13:16 - Why a factory has to be context centric 00:16:40 - Context-driven code review and the wall teams hit 00:22:51 - Skills inventory across your whole code estate 00:28:38 - Verifiers, evals and enforcing what you wrote down 00:37:30 - Loops, automations and owning your software factory Build your software factory, one workflow at a time, with Tessl: https://tessl.co/coi 🔔 Subscribe for weekly episodes on AI-native development If you've hit the plateau Dru describes, tell us where it stopped compounding for you.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_03We hit a peak of something like 850 PRs in a week, and 85-90% of those were handled by agents entirely. A lot of people, when they think software factories or agents, they're sort of expecting a speed up or hoping for a speed up. So we've actually seen that now that we have all this extra capacity, you don't turn all of it towards cranking out new features. One of the members of my design team was saying, I just put up and landed 13 PRs that were all about fixing consistency in copy, in brand voice, and like removing borders that weren't supposed to be there.

SPEAKER_03Things that would have just lingered on a backlog can actually now get done.

SPEAKER_01The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Fajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. Yesterday, you were writing code. Today, you're prompting agents. Tomorrow, you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash.

SPEAKER_01This June, AI DevCon London sold out. Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub cut through the hype and get to what actually works. This November, we're bringing it back to New York. Bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.

SPEAKER_01Our early bird pricing ends on September 30th, so make sure you lock in your place before prices increase. As a podcast listener, you can get 15% off your ticket using promo code POD50. We have so much planned and we can't wait to have you join us in New York.

SPEAKER_04Hello everyone, welcome back to the AI Native Dev. Thanks for tuning back in. Today, uh myself and Drew here, our head of product at TESL, uh, will uh want to catch you up a little bit on what we've been up to at TESL, what we've been building over the last sort of two or three months. I think we had a bit of this episode talking about the TESL agent, which we'll kind of mention.

SPEAKER_03A year and a half to get me on the podcast, back in three months. So I'm trending.

SPEAKER_04Once you're getting on. So really like there's a lot that we've been sort of building here at TESL, but I think the the key thing we're most excited by is this sort of expansion uh of TESIL from uh just sort of focusing on helping you own your context, right, and having a skill registry and build skills correctly and optimize them, which was still thinking is like an enormous uh important piece of this sort of new development paradigm, to uh starting to build loops. We talked about that a little bit with the uh Tesla agent launch on it, to uh to now really increasingly thinking about Tesla as a composable factory or as a platform that helps you build factory.

SPEAKER_04Uh and we talk about, and we'll sort of mention this a lot today, we talk about how Tesla helps you get to your factory from skills to loops to factory, which is sort of our view of the world of the sort of context-centric factory.

SPEAKER_02Yeah.

SPEAKER_04Yeah. So uh before we we dive into that, uh we we actually released this sort of super fun little mini documentary uh about Kikimora, the uh the sort of the spirit, the Eastern European spirit.

SPEAKER_03Eastern European spirit, it will clean your house at night. I'm quoting Maria, so thank you, Maria, for letting me know this. Uh it cleans your house at night, but only if you're good. If you are making a mess, if you are being disrespectful, it makes things worse, which we felt like was a perfect analogy for agents. Like if you're keeping your house in order, agents give you superpowers. If you're being irresponsible, they give you anti-superpowers, right? They go.

SPEAKER_04They amplify whatever it is that you've sort of given them to uh to amplify. Yeah, a little bit of some Christmas illusions there for the uh naughty list. Yeah. Um so that's the name of our sort of dark factory. And we have a really fun uh mini documentary about this uh that was very well produced on it, you know, by a friend here, Tom, who's our uh our producer, uh, and uh definitely it's about maybe like uh 13 minutes or something like that, 14 minutes. Uh uh definitely a fun watch, uh recommend it. We're not gonna force you to sort of repeat the whole thing over here, but maybe before we dive into the sort of factories and how we build it, maybe Drew, you can just sort of remind us a little bit of Tesla's own kind of dark factory, what do we build inside?

SPEAKER_04So a few words about you know what what is it that that uh we have and like what what's the value we're seeing in it?

SPEAKER_03Uh maybe I'll I'll start very, very briefly. I feel like every one of these podcasts has to start with a bit of just like terms. What do they mean these days, you know, two weeks later. Uh so really quickly, I think a couple things. First, we've mentioned skills, loops, and factories. So skills are, you know, obviously skills became quite popular at the beginning of the year. They've sort of expanded. Some people will call them plugins. Think when we say skills as like a unit of work. It's either a workflow that you want completed or it is a policy, a standard that you want agents to adhere to.

SPEAKER_03That includes skills. It also includes things like hooks or MCP tools, like anything you could bundle in a plugin. It's just that skills is the most popular term. It's what everybody thinks of as the unit for agents, so we sort of use that uh interchangeably. Right. So that's I think one. The second is loops, think of these as effectively automated skills. It's things that are running without immediate human intervention, at least to start them. And then as they run, typically there's some kind of meta process where the loop is uh getting better, right?

SPEAKER_03Like as it runs, it is making things better for the next run. And so you start to get compounding gains, you start to see agents sort of inflecting your productivity upwards faster than just linear.

SPEAKER_04Yeah. And this is like the hence the loop, right? Because it goes from an agentic workflow to something that loops with some form of learning back into the beginning. So hopefully the second time around or the hundredth time around is better than the first one.

SPEAKER_03Right. Yeah, and loops are where you really start to see your first glimmers of that sort of inflection point or your trajectory really taking off. And then a factory is in many ways, it's a way of working. It's where you've moved almost all of your development to the creation and maintenance and monitoring of loops. And the loops are now what produce most of your software or the product. Uh one way of thinking of this is that your factory will have some code. That code will largely be loops, and then your product that you ship to users is also some code.

SPEAKER_03Where are your humans spending their time? When you move to the factory, you're mostly spending your time maintaining the loops uh and thinking about how to make them better.

SPEAKER_04Yeah. And in a way, it's like the entirety of your SDLC, right? If your loops are kind of a kind of evolution of your pipelines, you know, of some automation that we come now. So the better.

SPEAKER_03I'm trying to make factory development lifecycle a thing, FDLC. It hasn't caught on yet, but we'll see. If I get to do a term segment at the beginning of the next one, hopefully that'll be one.

SPEAKER_04Maybe the FDLC.

SPEAKER_03Yeah.

SPEAKER_04Um, if you have a list that might be larger the SDLC versus the FDLC.

SPEAKER_03But so with that, with that as groundwork, um, what is Kikimura? What have we been building? So Tesla's really been trying to go on this same journey from the start of the year. Uh our development internally was very skills-driven to start, right? We wrote down what good looks like, how we wanted code review to function, how we wanted uh if you're building something in the CLI versus in the UI, right? Just like writing things down. And then once we felt like we were in a good place there, and we'll talk about why we think that's important to start there.

SPEAKER_03We then wanted to really get into the business of handing work over to agents. And so we started thinking about how we can automate more and more of our tasks so that a human isn't having to click buttons or type text into a text box to kick it off. Uh, and we we started that and very quickly, like the minute you get your first couple loops in place, you start to realize this is gonna get out of hand really quickly, and you have to move to a different way of working uh that has a bit more of the solution baked into the problem itself.

SPEAKER_03And so that was where we made the decision like we're gonna go full factory, we're gonna build out a system where basically all of our work, all of our PRs are created by agents. We focus more on shaping work, so we create issues, uh reviewing work, so we will look at PRs as they're put up. Increasingly, we're trying to have agents review more and more of that code, and we only invest in the sort of like high-leverage points. And then where we're doing our software engineering, our observability, our monitoring is in the loops themselves.

SPEAKER_03And so oftentimes you're creating loops to monitor loops, or you are creating dashboards to control plane your loops and things like that. And so that's been the flow that we've moved to uh internally at TESL. I think the last stat we saw, uh, we hit a peak of something like 850 PRs in a week, and about 90, 85, 90% of those were handled by agents entirely. So that's been our journey. Again, we won't make you watch the whole uh documentary again here, but if you're interested, go check it out.

SPEAKER_03Uh, it's a lot of fun. Yeah. Um, lots of great nuggets for folks to think about.

SPEAKER_04And and the pace, uh, I guess maybe what I would add is before we kind of leaned into the factory, what it felt was uh that we had a bunch of high automation environments, but they were very kind of uh disparate you know, environments. Very different developers. Like some members of the team, first of all, were sort of more, I guess, into automation and automated their kind of work uh in uh to a large degree, while others were a bit further behind, just sort of using it in a slightly more sort of standard way. Uh but then also the ones that were automating were automating their work and iterating, and it was quite siloed.

SPEAKER_04So I think the factory is also a bit of that path to go from single player to multiplayer, right? And go into that collaboration. And I think from like from a security lens perspective, uh there's also the advantage that once you start running things in the cloud and in kind of these defined uh workflows, then uh then you start kind of having visibility to it, you know what's installed, you can identify risks and contain them, you know, you can you can create more protected environments.

SPEAKER_03It's a transparent system as well. You can observe everything.

SPEAKER_04Yeah, exactly. And I've I've uh I'm one of those who who thought the cloud IDEs uh will catch on a lot sooner. It's like they're just they're just better. Like, you know, you have a stronger machine behind there and you can work from any spot on it and you can do it. And you know, lo and behold, they they they didn't really, like they're used somewhat. Yeah, uh, but there's all these advantages. Uh, I think as you get into agentic workflows and and really you're interacting through chats. I mean, developers' machines look more like the matrix and kind of the stereotypical you know hacker in movies than they ever did before, uh, then uh uh then I think it uh it makes sense.

SPEAKER_04And we're seeing, I guess I will add maybe with a management hat on on it, is that we're seeing some incredible speed on it. Again, there's a bunch of stats uh uh that we mentioned uh in in a variety of spots and in the in the movie, but you know, we're seeing just sort of compounding paces and uh now increasingly indeed the org is is leaning into it and more of the GTM is in it.

SPEAKER_03Yeah, I think maybe if I could add one quick the sort of increased feature velocity, great to see. It's the one you sort of are targeting. Maybe a lot of people when they think software factories or agents, they're sort of expecting a speed up or hoping for a speed up. Two of the things that we've seen that were uh interesting, I may not have guessed up front, for the big one was an increase in quality. So we've actually seen that now that we have all this extra capacity, you don't turn all of it towards cranking out new features. Some of it, like one of the members of my design team was saying, I just put up and landed 13 PRs that were all about fixing consistency and uh copy and brand voice and like removing borders that weren't supposed to be there.

SPEAKER_03Things that would have just lingered on a backlog can actually now get done. Uh and then the other one is just better fungibility. Like people, yeah, fungibility is maybe uh such a boring business way of saying it, but everyone is able to do a little bit of everything, right? Like I just mentioned, a designer putting up PRs that are getting merged, but also our GTM team is able to make changes to the marketing website without waiting on support from engineering. Engineering can write their own specs. You just have this ability for people to work on whatever's high leverage and not thinking about like who's ready to pick this up, how can I hand it off, all of that.

SPEAKER_04Right. And I think so. Factories are in vogue. Like uh you hear a lot of companies talk about sort of their path to it, and I think that's exciting. This is the new development paradigm. Uh and at the same time, there's there's sort of this world of skepticism around uh being able to, you know, produce quality software in that process and is it ready? Uh and a lot of sentiment of how you move from the bottleneck being the production of the code to being the review of the code, and is the code that's being generated any good.

SPEAKER_04And at TESL, we've long seen the world through this sort of lens of specs and context and skills, right? Like our kind of roots uh are very much around, you know, start by defining a spec, right? That sort of says, what is it that you want to do? Um and we we have this core conviction that to automate the thing, you first have to sit down and do this bit that is you know defining what correct is. What is it that you want to do, right? And I would use, I think since the beginning of TESL, this example of saying, hey, what do you want for your birthday?

SPEAKER_04It's like, I don't know, you decide, you know, like vibe it on it. You may or may not be happy with it, you know, if I you know use that sort of approach to buy you every year, you know, that gift, you know, some of them will be disappointments. Uh and I I think I think this is what we're seeing with uh with a bunch of that software. And so we we think of context as the as the core, right? As this sort of manual for operating your company, operating your workflows, you know, a bunch of those definitions. And and when we when we talk about building the factory, we have this uh uh perspective on how a factory should work, uh, which is that it should be context-centric, right?

SPEAKER_04It should and hence kind of the sort of repeated terms of going from skills to loops to factory. Uh and uh and it was interesting uh to sort of see that manifest in code. So it uh uh a couple of months ago or so, we launched uh a little over, we launched the Tesla agent, right? And the Tesla agent kind of helps you uh build uh build loops, you know, that was a core part of it. You run it and it helps you find automations and improvements and optimizations in your system and starts creating a loop that would sort of run that and then kind of make it go again, again.

SPEAKER_04Uh and we had someone on our on our team, Tom, uh build uh an automation that says, like, okay, let's let's uh build, I forget actually the the specific workflow. I think it was uh it was around um uh uh test coverage or something in that domain uh and created that workflow with sort of plain vanilla claude uh and with the Tesla agent and with sort of uh another sort of uh factory-related uh agent out there. And what uh what we saw was that these sort of two other ones, they they actually built the solution, the cookie-cutter solution, a little bit faster because they just sort of wrote a whole pile of code, like a lot of that output of that process was creating code that just created the pipeline to run a bunch of those sort of tests, right?

SPEAKER_04And uh and uh and run the process around them. While the approach that the Tesla agent takes, and was almost unintentional, it's just sort of our life view, our sort of uh our view on kind of this domain, uh, was first it created this sort of chain of logic. So it's like this is the definition of what correct is, this is sort of the specs of how it would work, uh, and then it built on top of it. And so it was it was a much more context-driven workflow, which on one hand took a moment longer and was stronger on some things, it was more resilient, you know, more uh uh malleable, um, and and could find and you know more cases and and and and make them work.

SPEAKER_04At the same time, it you know, it it it it again was a little bit more effort, uh, but it was more resilient now to be able to evolve and to uh and to grow. And so this sort of context centricity uh was so ingrained in how we build. It was a so and this is like a little bit of like I I I like that story because it just sort of naturally emerged from how we think about the world. And I think you you you had a similar view when we think about that context centricity uh when it comes to code review on it. You know, maybe you can say a few words about that.

SPEAKER_03I think that code review is actually both a good example of how context centricity is sort of crucial, but it's also hopefully one that hits close to home for folks. So maybe to start with the way most people will adopt some form of agentic code review. There's a bunch of great tools out there that make it really easy to get started. You install a GitHub app, or you click a few buttons here and there, and it's boom, you've now got agents reviewing your PRs, you're finding bugs, you're maybe merging faster, or you're at least catching more issues.

SPEAKER_03Uh, and it feels great. It's kind of like this dopamine hit. Yep. As is so often the case with AI, it's so easy to get to that first demo or that first wow moment. Yeah. How do you scale it from there? And what a lot of people see is that amazing, I'm doing a gen to code review, like I'm really like, I'm living it, and then a couple weeks later they find, well, okay, like we're finding a few more bugs, or maybe merging a bit faster, but nothing's changing after that. Like we we got a small boost, but it's not this exponential, like compounding gains.

SPEAKER_03It's just like a small improvement. Uh, and we're again we're we're stuck again, right? It still feels like code quality is a problem, or like PR review is is still an issue. And yeah, the Tesla philosophy is because you're hitting this wall because you jumped past the, you know, you tried to get to the big muscles without working out, or you tried to uh get healthy without eating your vegetables. Uh, there is a certain amount of you have to write down what good looks like. You have to know like what do I care about?

SPEAKER_03How does quality in the code base manifest? And if you don't do that, if you skip past and you just have a tool that says, like, don't worry, I'll handle it, I'll review the code for you, you will eventually hit a problem where uh two big ones. The first is that, well, now you you'd like to use those standards elsewhere, right? Like code review is good for catching things at the end, but really what you'd like to do, especially to help with costs or to get features developed faster, is while building the feature, you'd like the agent to also see your standards and make sure that it is writing code in the correct way.

SPEAKER_03And you know, not everything is going to get caught by code review, so you'd also like to have regular automations that sweep your code base and say, is there anything that's not conforming to our standards that slipped through or somehow is accrued and we need to fix and pull back? Uh so you need all the, like you want to use it in multiple places. And if you haven't written it down, you can't do that. You can't reuse it. Or say you have a new project, somebody's creating a new app or testing a prototype, it's in a different repo. How do you translate your standards over to them so they can get started right away?

SPEAKER_03Right. And so there is this, one, there's this enormous benefit in writing it down and being able to share it. Uh but also when you write it down, it becomes much easier to reason about and improve it and to even have specialized versions for different parts of the code base, right? So you want to say, in this area, our standards are this. You know, like uh for the front end, we care a lot about ARIA attributes and accessibility and having the right sort of nesting of borders. But when it's a back-end PR, none of that matters, right?

SPEAKER_03Don't waste time, don't waste tokens, focus instead on um, you know, brittleness or sort of reusing certain components. Uh and in fact, this specificity, you know, being able to tune exactly what you're reviewing becomes increasingly important as you move towards factory because really the volume a factory is going to put out, you have to get humans out of the loop for code review. And a general purpose code review is not going to get to 100% quality, right?

SPEAKER_03Like you have to be able to say, for this component, here's all the things you need to check. You have to get really specific. Right. And you just can't do that if you're not writing it down.

SPEAKER_00Hey everyone, my name is Sahil, and I'm one of the engineers here at TESL. I'm gonna be in New York and in Boston on the 16th and 17th of September talking about how we've built our software factory here at TESL and what engineering looks like in an AI native world. I'm gonna be diving into the components that we've built out, why they're there, why they're useful, the mistakes that we've made along the way, the sort of metrics that we focused on early that actually didn't end up helping us, and the ones that actually did. And all the different quirks and features that ended up building out the factory that we have today.

SPEAKER_00This is gonna be a much more practical look into the factory that we have here at TESL and some of the specifics on how it works and why we made some of the decisions that we did. Go to TESL.io slash events and I hope to see you all there.

SPEAKER_04I think uh so really kind of a good articulation of the view that you know you have to kind of capture and define what correct is, what correct looks like, and then you can do that at the granularity that you want. Sometimes it is shared across the different parts of the of the uh of the product stack. You might want it in review and encoding, maybe you want it across repos, and sometimes it's the other way around. It kind of gets more granular as well. In this subfolder, you know, when Drew writes code, I want to make sure uh that you sort of check, you know, that he doesn't again do X, you know, or whatever it is.

SPEAKER_02Exactly.

SPEAKER_04Um so I I think uh I think the sort of core. Is a really good story of it and an example of a loop that you can improve because indeed a factory has to sort of go to uh you have to sort of drive towards higher autonomy, and that has to sort of come with sort of definition of uh of what's correct, hence kind of skill-centric. So I think let's uh let's shift a little bit and and talk about some of the capabilities that uh that we're sort of excited by uh in it. And because we talk about the journey from from skills to loops to to uh to factory, maybe we'll start with a little bit, you know, over the last sort of uh two two to three months on it.

SPEAKER_04What's sort of what's been sort of interesting in the skills part of it of the product? You know, what's what's been launching?

SPEAKER_03How do you capture what's good, what the workflows are, et cetera. So I think last time we checked in, Tesla Agent was a big part of this. We talked about how you could use it to create loops that themselves turn back and help you find new context and things that you want to write down. Since then, we've added a big feature that we've been calling uh skills inventory, context inventory. Uh so what this is, is a uh command line as well as a GitHub app that you can install across your uh your source estate, uh so across all your repos.

SPEAKER_03Um, and we will scan and find all instances of context that's being fed to agents. So the most common form here is skills, but it could also be other like agents MDs or uh plug-in marketplaces, things like this. And then we just help you reason about it. So we give you a bunch of findings around do you have out-of-date context, right? We can detect, oh, you've installed this skill from Anthropic's Marketplace, it's now four versions behind.

SPEAKER_03Do you want to update to the most recent version? Uh we can do the same thing internally, right? Like, hey, we have found this same skill in five different places. It seems like this is actually the source where it's being maintained. Do you want to link all of those downstream versions to this and then get them up to date? But we can also find things like duplicate context where multiple people have been solving the same problem and not realizing it. And we can help you standardize on one so that you can then start improving it for everyone. Uh we also look on stale things where maybe you wrote a skill about how to contribute to the back end, and now the code has drifted away from the skill itself.

SPEAKER_03We can help you identify that and get it up to date. And this is a capability we're going to keep investing in, where we're just continually scanning your code estate, we're finding all of the context that agents are using, we're pulling it out, and we're telling you how can it be improved, how could it be reused, shared. Uh and so that's an area that we'll keep improving, but the sort of duplicate, stale, or uh in general, just finding things where there's opportunities to improve the overall performance and quality of your critical skills uh is a big improvement we've made to the platform.

SPEAKER_04Yeah. Yeah. I think uh it's been interesting to see how how deep the inventory uh capability. We thought it was a quick feature. And it just started quigs like, hey, I will just sort of enumerate files in your repos and I'll find all the ones that are called X. And then okay, we'll sort of run a review on all of those. We already had the skill review, so we'll we'll score them. We'll sort of say, are they any good in doing it? And then from there you get into indeed this like, well, these look like they do the same. It's amazing how uh unimaginative we are when we think about sort of ideas uh for uh for skills on it, like a lot, a lot of duplicates, which makes sense, but then you know that creates problems, you know, it's not just wasteful because the knowledge doesn't accumulate in one spot.

SPEAKER_04It has again creates kind of inconsistency and misalignment in the systems. How do you drive that? Once you created those, there's all this question about well, when people use a skill, how do they discover the right one? How do you do so? We added the whole kind of recommended skills that nudges you to sort of use the right ones. Uh, I guess security has been there for a while, you know, around sort of assessing whether those skills are sort of trustworthy on doing it. And it just sort of goes on and on around actionability, the dynamics between the platform team that has found, you know, maybe done that broad scan and the sort of the people that own the repos.

SPEAKER_03Inventory has kind of become an entry point to all of the tools that you know you've known love for Tesla, right? Like security policies, governance, being able to require, hey, these skills, like our brand voice or our code review, needs to be in every repo. Now we can help you find those skills and set up those policies. So it kind of becomes the integral control plane for somebody who's thinking about how do we manage skills across our organization.

SPEAKER_04Yeah. And I think, you know, I can't avoid the sort of the analogies, the security world I kind of came from, which is, you know, at SNEC, it was very much around, hey, we'll scan all your repos, we'll find all your dependencies. And it was similarly like a very deep kind of uh a tunnel over there to go to sort of say which ones are vulnerable, but you get that sort of instant feasibility. Uh cloud security poster management or CSBM tools like Wiz, you know, are sort of known for that. Uh so I think it's similar. Um and then I think we've also done a bunch of stuff on uh the the quality aspect of the skills.

SPEAKER_04We've started uh building kind of capturing of logs and we built these sort of verifiers. Do you want to maybe say a couple of words on that and then we'll kind of go to loops?

SPEAKER_03Yeah, yeah. Uh and also I'd be remiss if I didn't also mention we've extended support beyond uh just engineering use cases. So now we have the ability, once we've inventoried all your context, you can now version, distribute, deploy uh for your go-to-market function, for sales, for support, for anything like that. So that's been a big boost. You know, we're using it internally as well.

SPEAKER_04And I like the uh I think I'm kind of especially fond right now of the uh the synchronization with uh the marketplace. Yeah. Because I feel people on one hand like the convenience of the cloud marketplace and just sort of pushes it, you know, to to co-work or to others, but then they don't really want to be beholden to too anthropic. You know, what about the you know, the kind of open AI, kind of chat GPT marketplaces? What about other places? Yeah. So managing the sort of the skills centrally on the sort of the Tesla registry and then synchronizing them to the right spots uh has been quite appealing.

SPEAKER_03All the benefits being model agnostic, having an actual versioned life cycle so you can prevent regressions, all of that. Yeah, now comes for your non-devs as well. Uh and in a nice smooth experience where you don't need to teach your uh your sales team Git workflows or SEMVR or anything like that. Um, on top of that, we're starting to explore from our own factor. Like a lot of what we build comes from first, what did we need ourselves to make context-driven factories work inside of Tesla.

SPEAKER_03And one of the big things we found is that skills really are the codification of a workflow of what good looks like. And that's great. That's sort of like the interface where you want to write down this is what's important, this is how agents will look at it. But everybody wants that little bit of extra security, the extra surety, that, like, yes, I wrote it down, but agents don't have to follow Markdown. So did it, did it actually do that? And so verifiers are a thing we've built at TESIL. You can think of it as a way of turning your skills into or creating off of them very focused, high accuracy, very fast, very cheap LM as judge tools where uh to take an example, let's say you have a skill that says all front-end components must have ARIA attributes on them, right?

SPEAKER_03We can generate from that a verifier. This is a very simple case, but we generate a verifier that says for every TSX file or JSX file that's changed, we want to scan all React components and say, do they have ARIA attributes on them? That's like a very simple prompt. Then LMjudge can be very small, very cheap, and get it right effectively all the time. Uh so fast, so cheap that you can actually run it in CI if you'd like. And in fact, that's what we do here at TESIL. And so as a result, it basically gives you the enforcement half of a skill.

SPEAKER_03So you write out what is good look like, what is uh success, and then we generate these verifiers that you can then put up in CI CD so that you know your skills are being followed. And so that's a tool that we are increasingly relying on, where uh I think we're working right now on the ability to say, as soon as you create a skill, we'll create verifiers for it so that you can enforce the skill in a way that feels deterministic, right? It moves us back towards the world we're used to in software development.

SPEAKER_04Yeah, yeah. And I think I I very much see this as the trust but verify, you know, like you run the agent, you give them instructions, you trust them, you have to let go to some extent, you know, for them to do that. But then you build these verifiers. Uh and indeed I like the the speed, and in in general, like verifiers are you know still still forming, you know, but they can check the sort of the agent actions, you know, against the sort of the logs, they can check uh the output of it, of what has it done, they can be deterministic or they can be uh LMS judge. Uh and I think they I think there will be a core fundamental of going alongside in a skill the context of what you want it to do.

SPEAKER_04Verifiers are you know the checks to say, has it actually done it? Um and I think that is very powerful. Um and I think maybe you know we do need to get to loops, but maybe say a few words, which is we've also seen uh a bit of a revived uh um interest in evals. Evals have kind of been a little bit more the uh the purist domain, you know, the people like people reviving it to the extent. And I think what has happened is the world has matured enough and people have come across those cases uh where they where they have a challenge.

SPEAKER_04Um and I think it might feel a little bit scary uh to people as we talk about, hey, you have to sit down and kind of write down all of your paths, but I think one of the key unlocks that we've embraced probably starting about three months ago, uh, is this appreciation that a lot of that correctness can actually be extracted from historical behavior. It just then needs to be codified and captured, uh, and then you need to iterate on it. So verifiers, uh evals are increasingly, you know, even in the Tesla agent example, um, extracted out of a historical activity.

SPEAKER_04So it's not that it's a real burden that you need to sit down and kind of write down, you know, kind of uh painfully uh uh all of those instructions, but it's you know, if there's a thing that you are reviewing, if there's a thing that you own, it really should be these domains of your instructions. And the more you review and trust that sort of instruction set and those verifiers, uh the more uh uh uh autonomy you can give the agents as they operate and the more value you get.

SPEAKER_03Right. And yeah, I think it's just worth pointing out that that the Tesla agent exists to try and make it not feel like going to the gym or eating your vegetables. You know, like to your point, you can point the Tesla agent at your repo, at your PR history, at your tickets, Slack, anything you want. And it's its whole job is to be able to extract this context and capture it for you so that yes, it is a bit of a you're not just clicking a button and getting code review, but you're not sitting down and spending three weeks getting your whole team to it's like look at our code review comments from the last three months, write a document that captures our code review standards, and then we'll just evolve it from there, right?

SPEAKER_03It's very easy to get going and then improve over time, and that's where loops come in.

SPEAKER_01Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

SPEAKER_01Alright, back to the episode.

SPEAKER_04So maybe let me say a few words about loops and we'll dive into specifically code review, which we launched uh as a dedicated loops, an example, one of multiple examples of how to uh build a good loop. Uh, but maybe say a few words about that. So what we've observed is you know, we we work with customers, we help them create very good context, but the reality is that companies kind of evolve their practices. And if you're not if you're not creating loops, if you're not creating a factory, you're not really benefiting from creating this sort of well-crafted piece of context, right?

SPEAKER_04Or these great skills uh because you might not be equipped to to tap the benefit of them. And so we've been uh uh uh building, and I think this was the inception to kind of create the Tesla agent. So it's like, hey, let's let's help you create those loops as well. Uh and the loops will demonstrate that. A code review is an example, we'll dig into, you know, define what correct is, evolve that, improve that as a skill, get better uh code review, but then you know, uh many automations that happen within uh within the system that really should be done recurringly.

SPEAKER_04Uh and we uh have been building sort of two types of maintenance uh loops on it. Uh one is a maintenance loop for your software. Can you find flaky tests and fix them? Can you uh review architecture on a recurring basis? Can you indeed get test coverage of it? Can you uh just handle dependency upgrades? And so all of those are examples of loops that can be skill-driven that the Tesla agent can very easily set up for you, and then you can go on to optimize and improve. Uh and it's not hard, it's not a uh it's not a big lift for an organization to come in.

SPEAKER_04You don't need to be afraid of that sort of factory ward. Exactly. You can kind of increment your way. They're always skill-based, so you always kind of own your sort of heart and your core.

SPEAKER_03And in fact, can start manually at first, make sure it works, run the skill a couple times, then turn it into a loop.

SPEAKER_04Yeah, precisely, and you can pull it back out of it, so you very much are kind of in the in the driver's seat. Um, and uh and you you kind of uh build out of those now this sort of optimization loop. And so the um the the Tesla agent will sort of help you set up the skill and run it. Uh, but what you also need behind the scenes, which is a little bit uh uh uh invisible or like you know it like might not be trivial, is that once you start running, you need a few of these fundamentals. You need to be able to run the skill uh in in a way that captures the logs of what has been executed.

SPEAKER_04Uh you want a central place in which you can modify the model, for instance, to say, okay, now I want to run it with a different model. So if I ran a bunch of evals uh and you know, now I know that I can run it with a cheap model or whatever it is, I can run that. Uh I can I can modify that relatively easily. Uh and and so all of those, you know, the Tesla project, you know, like a bunch of that infrastructure sort of supports creating those loops and running them. Uh and and I I want to stress that we still see the world as skill-centric.

SPEAKER_04Eventually, what the loop does is it hones a skill, right? It creates a very good skill, maybe a harness around it, you know, like again, skill in the broad sense of the word.

SPEAKER_02Right.

SPEAKER_04Uh, but it uh but but it really kind of hones a skill, and then you can use it in many places. You don't just have to use it uh in the context of that specific loop. Um so I think uh feel free to correct me on sort of a bunch of these things that I'm doing. But maybe uh we uh again, this is a very big and kind of confusing world on it. And so one of the things we're excited that we launched is around Tesla code review. Again, we have some dedicated videos around that, which we'll kind of link to in the notes. Uh, but maybe you can say a few words about what Tesla Code Review is and maybe using it as a representation of a loop.

SPEAKER_03So I think the core name of the game is that TESL's code review tool is effectively our stance on context-driven code review as one loop that you're gonna use in a context-driven factory. So we talked about, oh yeah, you get started with code review, it feels great, then you kind of hit a wall and you that happened to TESL. We uh had a code review tool, we still love it, it's still a great tool, but we found that as we scaled further and further into factory world, we were just unable to compound and like get to a place where like we were spending less time reviewing code, which was what we wanted.

SPEAKER_03And so we took a step back and we thought about what's missing. We came to this idea that uh first we needed a way to be able to shift our standards left. So we wanted the same things that are happening in code review to be happening at development time. Uh, and also we needed to have faith that like if things slip through the gap, like slip through the cracks, they will be caught. And so we wanted automated loops that come up after code review time and search for things that got lost. Uh, and so we said we need all of the standards that go into code review to be documented and separate so that we can use them elsewhere.

SPEAKER_03So that's probably the core of TESL's code review is that it is skill powered. Uh we call them lenses just like code review lenses where you can supply any number of skills, it doesn't just have to be one, that codify some standard of good in your code base. You can say which files, which file patterns those lenses apply to. And then uh just like any other code review tool, you'll set it up through a GitHub app. And basically every time a PR goes up, the appropriate lenses will be triggered based on the file patterns and will review your code.

SPEAKER_03But now it's uh it's a set of skills. And so you can also make those available to the agent while it develops or in sort of maintenance agents or automated uh sort of sweeps of your code base. Uh so that's a great piece, right? Like it allows you to reuse it in many places. It's also now a thing that you own. It connects to the rest of Tesla's context platform, so we can help you evaluate your code review skill, optimize it, share it across repos for different teams.

SPEAKER_03Uh and then importantly, it also starts to feed back into the context platform. So that was another thing that we learned is that code review is a very rich source of what's going wrong. Like what skills are we missing, what skills aren't working. Uh, and so as code review is running, it's not just making code review better, it's making the context of your code base better. So it's looking for missing context or uh skills that maybe aren't being triggered or aren't having the effect that you'd like, and suggesting ways to improve them so that they don't have to be caught at code review time later.

SPEAKER_03And that that kind of gets to your point of you really want these things to be integrated with one another. Like you don't want your context for development and your code review standards to be separate. You want them to feed back into each other to create compounding gains. And so that's really what the TESL code review sort of philosophy is.

SPEAKER_04Yeah. And I think uh I think a lot of the analogies to humans, right? Like the the developers on the team do a peer review, right? So like the knowledge that they have available to them when they review is the same knowledge that they have as they build. And the loop is the example of someone having kind of received a comment, right? Having kind of gotten some bit of feedback in the review, uh, and then internalize that and come back and need to do that.

SPEAKER_03So why don't we make a standard for this or document?

SPEAKER_04Exactly. Sort of capture that so the next person coming along, you know, sort of driving it. And so for us, it's not the next person, it's the next agent that's coming along, you know, has that knowledge. Uh and I I really like it. And I, you know, I I I think reviewing is a critical part uh of it, but I guess I think of our code review in comparison to to the sort of the generic, again, powerful code review tools that are out there, is that I think the key difference is uh that that you own it, you know, it's your uh skills, it's your tunings, you know, you can uh therefore kind of refine them to your specific needs, not just what it looks for, but also choices like depth of scanning and choice of how much do you spend on different models on it.

SPEAKER_04And we've done some interesting research around that. So you own it. Uh the second is uh that you know, because of that sort of refinement and doing it, it's actually just better. And we have a bunch of that data, we've sort of seen it in our system. And I don't think it's because you know we have some sort of like brilliant algorithmic improvement over it, but this is probably just sort of the the side effect of the fact that you own it is the result is better. It just sort of finds more issues, it finds them faster, it creates less noise. And so our team very much.

SPEAKER_03Specificity basically is the ruler of all quality. Like if you can have a very specific task that the agent has clear instructions on what to do, it's gonna be better than a generic here's how to look for code smells.

SPEAKER_04Exactly. And then the third is that it is a path to your factory and uh and and you want it, and increasingly like the we say code review and we we imagine a pull request. But in practice, if you're performing uh uh if you're kind of running an agentic workflow, right, or a factory that takes a ticket and and implements it, you don't need to wait for it to have been sort of checked in. And so you just sort of take that skill and run the review as part of your factory uh and you build that out. Um so I guess that sort of leads us. So we we talked about skills and the skill centricity of it, talked about loops that indeed, you know, just sort of improve skills and use them with code review as an example.

SPEAKER_04And there's many others you should sort of uh uh uh you can already do many today, and more and more of these sort of uh cookie-cutter use cases will come out. Um and then maybe like let's conclude with the sort of factory piece. And factory is admittedly like a more nascent piece uh uh for us, right? And and I think it's because we're we're kind of accompanying uh companies, right, as they sort of go through this uh journey. And so their sort of more immediate needs were sort of the skills and then the loops. Now they're getting into that sort of broader factory of it. Um and we've we've sort of learned through both kind of this work with customers and uh through our own sort of factory building, uh uh around kind of the the need for more componentry to be available to kind of make the factory run smooth.

SPEAKER_04Maybe you can say a few words about that.

SPEAKER_03Yeah, so the the core of it, again, we talked about if you have a context-centric uh factory, I'm gonna sound like a broken record on that. You want to start with writing down what the workflow is, what good looks like, what success is. But basically, what we found is that that's important, but you can't stop there. Having a really great skill that you still have to run manually or you have to go paste into all of the various places you want it to be used, you're not going to get the value that comes from having done the work to write it down.

SPEAKER_03And so we found with customers generally, you now need to provide the Rails or the plumbing to deploy that context and to do more and more work on your behalf in an automated fashion. And what we found is basically once you start getting into this world, like factory building is really it's more of a continuum. It's not a final state. You're not going to say, okay, I've made a few loops, now I will have a factory, and in two months I'll be done. It's sort of, you're going to keep working on loops until you have so many loops that most of your work is in maintaining loops and building loops to observe them.

SPEAKER_03And so it's kind of a continuum, whereas you get more and more, I guess to use our term from last time, more and more loopy, right? It's going to just sort of transition seamlessly into factory work. It'll be a sort of a blended approach. And so the first thing that we've built to enable this is an automations platform, which effectively allows you to take the context that you've created and deploy it in a way that runs on schedule. Coming soon will be things like webhooks and more automated triggers.

SPEAKER_03But the idea here is that you've done the work to make the workflow to define a standard. Or give it access to, so a big part of the automations platform is controlling the environment where it runs. So you can say this workflow needs Slack, it needs Notion, and then it needs to run every Friday. You can put all that together.

SPEAKER_03You can pick the cheapest model possible so that you're getting as much, like the best quality, but without overpaying for it. But then importantly, so you know, that's context on the front half of making the automations actually work. But on the back half, the automations platform integrates back into the context platform. So all of those runs are capturing logs. You can then create additional automations that review those logs and make improvements to the skill based on real-world usage. And then, of course, it also integrates into the rest of our context platform where all these automations are multiplayer by default.

SPEAKER_03You can, the same way you can see all of the skills in your repo, you can see all of the automations that your team has created. You can share them, you can create sort of permissions around them.

SPEAKER_04So that's a good kind of description of just these different services that uh that we build to it. I guess the the point I'd emphasize is you know, we think every company should own its factory. In fact, we think there will be a plurality of factories, right? Like they will have factories uh for different repos or business units or sort of segments of it. Uh and we think a factory is something you should own, right? Like it is it is your custom definition of what correct is. It's the new software engineering. And so it doesn't, that does not mean that you should build every aspect of it. Uh and we think that our job as sort of tool providers, right, as enablers of this sort of agentic era uh development uh is to give you a bunch of these tools to make it easy, to make a bunch of these things simple.

SPEAKER_04Uh and so a lot of these kind of components they come out of of portions that we think are reusable, whether they are around observability, whether around, of course, everything to do with context, the capturing of the context, the assessment of the context, et cetera, whether uh it's just just sort of utilities like scheduling, like uh like uh access controls, like dashboards and visuals around you know what makes for kind of an effective dashboard or like an effective factory uh or not, and what should you learn? So our job is to really consolidate a lot of industry learnings and our own learnings and kind of repeat work in various parts of the industry and capture it up.

SPEAKER_04And so uh I would say that our our our kind of factory componentry is still uh relatively light. There's a world that we still need to build over here, uh, and and that is on purpose because again, we sort of advance with our with our customers, uh, and many of them they want to build their own factory. Some of these things should really be custom-built components, yeah. Uh, and we'll sort of help them build the uh the component tree there. Uh, but what we will emphasize maybe once more uh is that we think a factory should really be uh uh skills-centric, context-centric.

SPEAKER_04It should start from your definitions of what uh correct software development is. Uh and what we want to be is just like an SDLC platform that has a sequence of these different components uh to help you apply your development process. This is what Tesla is to you, uh, to to help you kind of build your factory and evolve. Uh and of course, as always, we're sort of there to to kind of help you in the process itself, right? Help you kind of learn and uh and grow. Um so a lot has happened.

SPEAKER_04We've been busy. Uh it's actually been sort of super exciting to uh have these uh these sort of weekly reviews of what's coming this week of it. And it's like, okay, these teams, we have a very empowered team, they've been building great components, working with customers on it. And and just like the next week is like, oh yeah, that sort of big chunk over here that would have been maybe like a quarter's work or something like that in the previous, it's done. Okay, what do we do next? Uh it does feel like you can't blink, you know, which is sort of exciting and exciting and daunting at the same time. Uh so if you found uh all of this conversation interesting, uh, we'd love for you to try the sort of the expanded uh TESL of it.

SPEAKER_04Of course, you can always run still some TESL skill review and the evals and a lot of these components that we've been building for a long while. Uh, we're very excited and we're seeing a lot of sort of uptake around the TESL agent uh and using it to build loops. So we'd love that. Uh and then maybe try out thinking about using Tesla as the platform to kind of build up a broader context-centric, uh, if it wasn't, if we didn't say that enough, context-centric uh uh factory over it. Uh and you know, as hopefully I think is evidence to anyone that has been tracking us, we're very, very community-oriented.

SPEAKER_04So we'd love if you try it out. And we'd also love your perspective on the sort of philosophy that we're espousing here around uh our belief that you should build your factory gradually going from skills to loops to the factory. Uh and with that, thanks for tuning in, and I hope you join us for the next one.

SPEAKER_01The AI Native Dev is brought to you by TESL, the package manager for skills and context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the TESL offices in central London. Visit TESL.io forward slash community to learn more, and I hope to see you there.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Your Software Factory Needs a Bill of Materials | Daniel Meppiel

    Nov 19, 20261 hr 1 minAudio

    Developers are about to write skills instead of code, and Microsoft's Daniel Meppiel argues the software factory that runs them needs versioning, a bill of materials and signed receipts before enterprises will trust it....

    Credit the people on this episode

  2. Paul Stack: I'll Never Write Infrastructure as Code Again

    Nov 17, 20261 hr 2 minAudio

    Paul Stack helped build Terraform. Now he says he'll never write another line of infrastructure as code, because an agent harness runs his team's production infrastructure instead. Watch all the AI DevCon NYC 2026...

    Credit the people on this episode

  3. Edith Harbaugh: Code Is Cheap. The Last Mile Isn't.

    Nov 10, 202649 minAudio

    Automated software delivery is moving faster than the people who used to approve every release. LaunchDarkly co-founder and CEO Edith Harbaugh tells Guy Podjarny why rollback should be autonomous, why one-shot coding...

    Credit the people on this episode

  4. Maggie Appleton: Agentic Coding Has Outgrown the Text Box

    Nov 3, 202652 minAudio

    Agentic coding has outgrown the text box. Maggie Appleton, staff research engineer at GitHub Next, argues that walls of CLI output and AGENTS.md files serve coding agents far better than the humans steering them. She...

    Credit the people on this episode

  5. Caleb Sima: AI Attackers Move at Machine Speed

    Oct 27, 202655 minAudio

    AI attackers now move at machine speed, while most security teams still move at meeting speed. Former Robinhood CSO Caleb Sima has seen an AI attacker up close, and he argues the only way to keep up is to take humans...

    Credit the people on this episode