Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow artwork

Inside the Dark Factory: AI That Ships Code Solo

Jul 28, 202659 minAudio

At Tessl, 95% of the code shipped by their internal "Dark Factory" has never been looked at by a human, and the team still ships hundreds of pull requests a week, including through entire weekends. Rob Willoughby, who leads AI engineering at Tessl, joins Simon to open up the hood on how it actually works: the orchestrator, the verification layers, and the failures that forced the team to rebuild trust from scratch. What we cover: - How Tessl routes 65-70% of its own pull requests through an autonomous "Dark Factory" - Why context in the repo matters more to output quality than which model you use - How natural language "verifiers" turn code review taste into fast, cheap checks agents can pass or fail - The queue bug that took dozens of pull requests to fix, and the from-scratch Elixir rewrite that stress-tested the whole system - How to start building your own software factory, one verification layer at a time Chapters: 00:00:00 - Introduction 00:01:44 - Rob Willoughby joins: Tessl's PR numbers 00:04:01 - Live demo: kicking off two pull requests 00:14:03 - Building the Dark Factory: orchestrator vs. context 00:19:01 - Code review layers: Code Rabbit, Tessl Change Verify, and verifiers 00:29:58 - Earning trust: accountability in an autonomous system 00:33:20 - What broke: the queue bug and the Elixir rewrite experiment 00:39:56 - Onboarding new engineers into the factory 00:42:58 - Advice for teams starting their own software factory 00:52:03 - Back to the demo, and the road to 100% adoption 🌐 Tessl: https://tessl.io 🔔 Subscribe for weekly episodes on AI-native development What would your own verification layer catch, and where would it break? Let us know in the comments.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_01Two weekends ago, um, we had quite a busy Friday, and the Dark Factory ended up shipping 150 PRs for two of us over the course of the weekend. And that was just while we were out hanging about, enjoying London in the heatwave. Last time I looked, I think about only 5% of the Dark Factory code base had ever had a human look at the PR and approve it. I don't think people actually go look at code anymore because they're all just using agents. And so that means, by implication, 95% of our code base has never had a human look at it.

SPEAKER_03The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Pajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. We just wrapped up two amazing days at AI DevCon in London.

SPEAKER_02But the great thing is that we get to do it all over again in New York City this November.

SPEAKER_03You're absolutely right. We're gonna be back in the city that never sleeps on November 3rd and 4th for more amazing sessions, really engaging, hands-on workshops, and much more.

SPEAKER_02Yep, all that great networking, partying, eating and drinking that you've come to expect from AI DevCon. We think we have one of the best hallway tracks in the business, and it's the perfect compliment to our incredible speakers and presenters.

SPEAKER_03We'll both be in person and virtual with live-streamed access to all mainstage keynotes and talks.

SPEAKER_02Sign up right now for our Super Blind Bird ticket for just$100, only available for a limited time.

SPEAKER_03We're really excited to be headed back to the big app, and we hope to see you all there. The week before, we seemed to raise only 608, so only 90 or so more.

SPEAKER_03How do we do this? It's all about the usage of software factories and the dark factory. Joining me is Rob Willoughby, who is the AI engineering lead at Tesla, and Rob's gonna tell us all about how Tesla uses Dart Factories and software factories internally to create such amazing numbers. Rob, welcome. How are you? Thank you, thank you. Doing well. And first time on the AR Native Dev Podcast.

SPEAKER_01It is first time on a podcast.

SPEAKER_03Right, we have to do the uh induction, unfortunately. So um no, no, there's no inductions. Uh I was thinking of making something up then, but yeah, phew. Didn't tell you that. Um so Rob, first of all, um tell us a little bit about what you do at TESL.

SPEAKER_01Yeah, so I run the A engineering team here, which is part of the team that kind of broadly is looking at agents enablement kind of across TESL. And so what our mandate is is really to build and then drive adoption of a dark software factory internally that we use for kind of two main reasons. One is to improve our own engineering velocity. We want to ship faster, ship more, ship higher quality stuff, be able to use agents better. But then also additionally, we use that as a testing ground for our own uh theses and hypotheses about how to effectively run a factory. We want to be able to take the learnings from there and put those into our products that we can then go help other companies bring along on the adoption curve as they start to build um these factories internally for themselves.

SPEAKER_03So let's these numbers are they seem crazy, right? Every every time I take talk to people and say even a few hundred pull requests uh for you know not a huge team, not a small team, but you know, a reasonable sized team, people are surprised. They're like, how do you deal with that? How do we deal with things like uh you know code reviews and the human interaction and things like that? We'll cover a lot of this, and actually during this podcast, we're gonna talk about how uh TESL adopted a Dark Factory, what were the sticking points, what worked, what didn't work, how we overcame them.

SPEAKER_03But for now, let's actually take a look at what a good interaction looks like through the Dark Factory. Let's now go through a full end-to-end test of two pull requests. One which requires human interaction, and the other which will go end-to-end without human interaction, starting from a linear ticket all the way through to a completed pull request.

SPEAKER_01Um so this ticket is one that I prepped beforehand, which has to deal with some uh findings from an audit we did where we can accidentally allow uh data from GitHub repos and uh inject commands into the TESL agent commands that we are showing on the website. That just has to do with string escaping and all that kind of stuff. So relatively simple fix, but security uh important. This is gonna be touching what shows on our website, so it will require a human PR review in order to land. All I've got to do is I've got this ticket prepped saying it's in apps front end, so that's our front end, obviously.

SPEAKER_01Uh and then I'm gonna flip it into to do. Because it's been delegated to the dark factory. That means that the dark factory is gonna pick it up and go do this implementation for me.

SPEAKER_03So that's kicking off the dark factory. Just putting that into a to-do, the dark factory recognizes there's a hand over there, and it kicks off uh a new container where it can kind of like do all that. Exactly.

SPEAKER_01And so now I'm hands-off at this point. The second ticket is one that is targeting our dark factory itself, Hernis slash Kikimura. That's just our own internal name for it. This is another uh one that is around uh expanding out our metrics. We aren't actually capturing which inference geo uh where we're running these queries against, which means that we're not able to properly uh analyze the effects of caching because our requests could go across any different geos, which then affects cache uh rates and also cost overall. So because this is uh within our own dark factory, we have a policy of auto-merging everything.

SPEAKER_01Because if we can't auto-merge it, that means that we don't have the verification in place to have confidence in the process of writing code or building code. So you can see that this auto-merge uh label is down here. This means that as soon as the dark factory is done creating it, it will go through PR cycle process using bots, of course. Uh no human intervention here, and then it will get auto-merged into the repo once all of CI passes and once our review agents uh approve it. So I'm gonna flip that into to do and we'll check back in on this one at the end of the podcast.

SPEAKER_03Amazing. And the fact that that uh is set to auto-merge, if that wasn't there, presumably that then would require a human to go in, maybe review the PR and actually push it themselves.

SPEAKER_01Yeah, exactly. Gotcha. So auto merge is an optional label that just gets the bot to literally select auto merge on the GitHub PR uh so that it will go in once CI is green.

SPEAKER_03Awesome, amazing. So now we've raised uh two tickets, two linear tickets. Uh one which requires a human review, so the Dart Factory will do all the work and then it will say, This is done, here's the pull request, approve it, and it will be uh it will be merged. The second one, because it's uh a PR that goes directly to the Dart Factory, it's a PR for the Dart Factory, uh, that will auto-merge, so as soon as it's done, it will it will squash that down and merge that into the code base. Yep, correct. Amazing. We'll come back to that at the end of the podcast. Uh so why don't we talk about how Tesla uses the Dart Factory numbers-wise, first of all, and then we'll maybe look back as to as to what we were doing before the Dart Factory existed.

SPEAKER_03So when we think numbers-wise, what percentage of Tesla's pull requests today use the Dart Factory?

SPEAKER_01Yeah, so it varies a little bit week over week. Um, but for the last week, we are between 65 and 70%. Uh, mind you, this is including uh pull requests that are building the dark factory using the dark factory. Um, so there is a lot of those, as you can expect. We are heavy users of it internally on our team. Um the last time that I looked for the production part of our code base, so the stuff that is serving the registry, our web UI, um our CLI, all of that kind of stuff, we are at about 40% of production PRs were raised using the Dark Factory.

SPEAKER_03And all of those 40% require a human review to some extent.

SPEAKER_01Because they just our policy is that we only allow auto merge on um stuff that is in our research code base, stuff that is in our dark factory code base, and stuff that um our GTM team is building uh using the Dark Factory to help uh build out their own internal flows too. And how about weekends? Does the Dark Factory work while we sleep? Uh oh yes. Uh we'll happily work weekends and overnight, and as long as there work is there is work in the queue. It feels like a little bit of the shift in my own thinking about this is less about what is the coding work that I think I can do over the course of the week, but what is the coding work that I can be able to effectively feed the queue over a long enough time?

SPEAKER_01Because we respect linear blocking relationships and able to handle kind of subtest and all that kind of stuff, I can actually sequence quite a fair amount of work before I go away for the weekend and come back to not a fundamentally different architecture, but a relatively different architecture over the course of the weekend. I think uh two weekends ago, um, we had quite a busy Friday, and the Dark Factory ended up shipping 150 PRs uh for two of us uh over the course of the weekend. And that was just while we were out hanging about, enjoying London in the heat wave and some of the sun here.

SPEAKER_03And was there any interaction that you had over that weekend during those? No, so those were all auto-merged in. Amazing. So TESL didn't always use a dark factory, of course. Uh what was the uh engineering styles and culture pre-you building the dark factory?

SPEAKER_01So the way I'd framed it was single-player agentic engineering, um, in terms of you had people have their local agent, um, or maybe a cloud agent like Claud Code on the web or cursor agent in the cloud or anything along those lines, but it was still ultimately being driven by an engineer in the interactive human or the interactive driver seat. Um and so you'd figure out with your team what the work you were on to do, you'd then work with an agent to scope that work, do the implementation, test and verify locally, push up and merge the PR.

SPEAKER_01So very similar to kind of the previous flow uh prior to agents, prior to LLMs, just on an accelerated timescale because the implementation was now getting cheap. And so, but it was still that very much so I am a human driving a ticket through to completion with many touch points along the way.

SPEAKER_03Yeah, so it wasn't so much a workflow change, it was more uh and it, like you say, it's an acceleration of that code generation, really. Um and and I guess what what what were the tripping points at that stage? What were the things that would slow us down? Because I guess you know, speeding us up with this with the same workflow is gonna cause problems, it's gonna cause a different bottleneck in it in likely in another place. What were the problems though?

SPEAKER_01Yeah. So I think uh verification and testing was a big problem. Um we uh so we run a monorepo um where we have a bunch of build dependencies, and so in order to test it, effectively test everything, you've got to build it locally. Um, and then you need to walk kind of through a whole bunch of go hit a bunch of endpoints or go click a bunch of buttons in the UI or something along those lines. And so that meant that any change that you were doing, yes, now your implementation is free or functionally free, um, but you still have to hit pay that verification burden in terms of, hey, I've got to go actually check that this did the thing that I wanted it to do, rather than just looking at the code.

SPEAKER_01The other bottleneck is the code review. Suddenly you're shifting this, what an engineer is spending the majority of their day doing from writing code to reviewing code. And there are very few people out there who would have said, I think, in this prior world, that they love code review as the most important part of the job. Um, but it is something that is needed and something where we share best practices, share culture through, and now people were just doing it day in and day out. Because I can churn out 20 PRs in a day using a local agent, but if we have a policy where every PR needs a human review on it, well then my team needs to go review 20 PRs.

SPEAKER_01And yes, then they'll also be churning out a bunch of PRs, and then I have to review their PRs, and we just end up in the cycle of we're actually spending more time reviewing and then PRs get stale and then they have merge conflicts, and it's this fun this um this other bottleneck that I think we'd never had to engage with before because we just weren't making enough code before.

SPEAKER_03So it's very interesting when we compare that previous approach where Tesla was essentially using those coding agents as as an accelerator to writing code. Come the Friday, we're essentially trying to get as much done as we possibly can looking backwards on that week. Whereas today with the Dart Factory, what we're actually doing is it's a bad Friday if we don't prep the weekend. We're actually looking forward to think, can I plan and stack and queue enough for the Dart Factory to have enough work going forward? It's a very different mindset in terms of you're essentially managing the agent work versus you know essentially feeding on a minute to minute basis the agent with tasks to do.

SPEAKER_01Yep, yeah, definitely. And it it's a little bit of that shift in my mind, like you're shifting up the abstraction stack a bit. You're no longer thinking about like what is this ticket that I'm gonna try to get done? You're thinking about what is this feature that I'm gonna I want to do, what is the sequence of steps that someone needs to do it, just like I divvy up to a junior engineer or an intern or something along those lines. And then how do I make sure that the stuff that I care about, either like the correctness or the UI verification or the clicking the buttons or hitting at the endpoints or whatever it is that kind of I is important to me, how do I make sure that's encoded in a way where I trust that it's gonna get done and the the agent is gonna do it?

SPEAKER_01And so I think it's both for me kind of like that stacking up and that forward-looking aspect of it. But the other big shift has been kind of what we now have these agents, these models, where we can literally deterministically load context into their context window to enforce the fact that they have read something. It's very hard for me to get a human to go read a doc on best practices or principles or something along those lines. So I have this additional leverage from being able to say, you know that I want to build the code as uh code base in this way, so you're gonna go build it in that way.

SPEAKER_01Um, and I therefore have a higher level of trust in the code that's gonna be outputted. I can then also start layering in other bits of verification around like coding styles or around testing or around enforcement or stuff along those lines. And this is all about to that feeding the qubit. The only way that I would trust to be able to build a queue that will then sequence and run autonomously, is if I have enough trust that the output of that queue isn't, at a bare minimum, going to break existing functionality, but ideally is going to be something that maybe not proud of writing myself, but I would be okay if I had written it myself.

SPEAKER_01Yeah.

SPEAKER_03Let's uh let's jump through what a dark factory is component by component, and to and and I'd love for you to walk us through how we started on this dark factory journey, what we implemented first, and and Yeah, you know, really understanding that path so that others can kind of follow and understand uh what they should do first. So when we first tried to build this dark factory, what were the initial steps that we took?

SPEAKER_01So I think they're just kind of set the lay of the land a bit, there's a couple important concepts here. One is the orchestrator, which is what's actually picking up tickets, um, and then passing it off to coding agents, doing work, and doing the whole babysitting thing in terms of making sure the PR goes through and all that kind of stuff. But then importantly, and I'll touch on this second, is the actual context and environment within our repo that we give to the coding agents, uh, which I personally think is where the higher amount of leverage comes from. But uh we built the first thing first, uh, the orchestrator first, just because that is the tractable thing, that's kind of what we think of when we think of a dark software factory, is the mechanism to go from linear ticket to actually produce code.

SPEAKER_01I think just one of our big learnings previewing this has been that is dead simple and not that uh hard. It's everything else that surrounds that, where you then because if you don't kind of invest in those other layers, you then end up back in that previous workflow case we were talking about, where you just end up overwhelmed with review, or you end up overwhelmed with code, or you end up doing things that aren't actually driving the whole system forward.

SPEAKER_03Aaron Powell So the orchestrator isn't here an agent orchestrator, it's essentially almost like a workflow orchestrator. Exactly. It's trying it's it's it's connecting, it's an integrator between agents and other tools or other things like Linear as as a good example of.

SPEAKER_01Exactly. And so our journey started with basically a Python queue that connected to Linear and connected to uh GitHub and ran an agent locally in a Docker's handbox. Um we literally had one of our ends not allowed to close our laptop for about a week because it was all running locally on our machine and we had much more adoption on that that uh than we had expected internally there. Um and so kind of the design decisions that we made around that were trying to keep things as simple as possible and relying on other systems as our source of truth where we could, because we didn't want to have to reinvent the wheel there.

SPEAKER_01So we've made an opinionated choice. Linear is our source of truth for work to be done. Everything must flow through linear if it's going to result in a durable artifact uh that is oriented that is a PR in our code base, or a Notion Doc, or any of the other artifacts that are produced by the Dark Factory. That just allows us the audit trail, the debugging, the visibility on that. And it means that we can then rely on all the work that Linear has done in terms of uh blocking relationships, subtasks, tagging for the reporting and the mechanisms that we need there, and we don't need to build it ourselves.

SPEAKER_01Um so we have basically a poll. Uh we're working to shift this over to Webhooks. It just hasn't been at top of the priority list because we haven't broken linear yet with our polling, other than earner tests, which I'll mention later. Um so we have something that pulls linear for new work delegated to our factory and state to do. Picks that up, puts it onto the queue. There's a bunch of prioritization logic uh depending on kind of which project it's coming from, what the priority is, whether it's being blocked, stuff along those lines. Uh that then gets farmed out to a coding agent. Uh as I mentioned originally that was just running in Docker on someone's machine.

SPEAKER_01We just really want an isolated environment because we don't want these agents to kind of be interacting with each other uh because each one is meant to be kind of a one-shot uh implementation. Mind you, they can technically understand what other agents are going on, and we do see this in the logs because we'll often reference other tickets in the description of a ticket. And so an agent will go check in on how that ticket is going or what the PR was associated with that. But that's just kind of an emergent behavior in terms of building up the context needed to be able to do effective implementation. These agents uh currently for us run in a Daytona sandbox.

SPEAKER_01Uh, we're using Daytona just because of the isolation and security guarantees there. Um, this runs in our own internal Daytona environment. Um, and we provision a relatively beefy uh box for that, technically a VM, um, in order to, or technically a container actually, in order to allow it to spin up the entire monorepo testing stack. So basically everything that a local engineer can do on their machine, modulo creds that the local engineer has access to that we don't give to the agent, we want the agent to be able to do. This allows the agent to run our full test suite, including integration tests, uh spin up the monorepo, click around on a UI, click around on basically anything that we could do be doing for validation and testing, all in their own virtual environment.

SPEAKER_01Um that helps us build uh kind of confidence there. One really interesting uh benefit of that is that we've now been able to have the agent post videos and screenshots back onto the PR so that we as humans can go manually verify the work without needing to actually SSH on the box and go click the buttons ourselves. We can see what the flow looks like from that video, from that evidence, which helps then build trust and build confidence in that it implemented the right thing. Um that is kind of the crux of the orchestration flow. Um the other bit of the orchestrator is just basically babysitting a PR.

SPEAKER_01So uh once the coding agent is done, pass a couple pre-commit hooks, pushes up a PR. Uh, we're relatively confident that PR at least like passes lint, builds, passes tests. Like it's a good agentic uh first shot implementation. Um on the PR surface, so this is now we're in GitHub, uh, we run two different coding agents. One is CodeRabbit, one is our own internal code review agent. Our internal code review agents kind of is anchored on skills. So we have written three at the moment, uh, other ones coming soon. Skills that are targeting different bits of kind of the way that we what our code review philosophy is.

SPEAKER_01So one around security, one around legibility, which is really is this code easy to maintain in the long term? Um, and one around kind of catch-all functional correctness. Um, we then uh so we run our own uh code review tool. This is what's exposed via Tesla change uh review. Um, and we are using that as signals on the PR for the agent to go fix. So Code Rabbit and the and uh our internal Zool posts comments back on the PR, uh, and we have failures in CI when some tests don't pass, or um some we do some uh what's it called, mutation testing, uh property-based testing, a whole bunch of that kind of stuff.

SPEAKER_01If those don't pass, those all get posted back as signals on the PR. The orchestrator is watching the PR, and so we'll re-wake the agent when those comments are uh passed back, and the agent then goes fixes them. Uh it will take in the comments, it will reason over them, uh, it then responds in line in those comments where appropriate or as a top-level comment, resolving where appropriate um as needed, and also potentially escalating back to human uh for review. So it has the ability to say, hey, I need context, can pause everything and go post that question back on the linear ticket, which then notifies the human.

SPEAKER_01Um, and so that is kind of the flow in a nutshell. Uh going from a linear ticket to orchestrator to coding agent unboxed to PR to a bunch of cycles on the PR review between automated bots and uh CI checks uh back to the agent until it gets to a state that is good. That's normally when we flip to uh in review and we'll get the human in order to engage with that, uh, because it's kind of like in hopefully the best state that it could possibly be. And so human, the interaction mode there is twofold. One, uh we do have the escape hatch of they can uh pull down the branch locally, make changes themselves, and then push it back up.

SPEAKER_01That's always an option if kind of it's gonna be quicker or they need to do it that way. What we're trying to push for though is doing the interaction mode on that PR surface via comments, uh, because that then allows us to kind of track that and also integrate that back into kind of what a review philosophy is. That makes it surfaceable, actionable, so that we can kind of continue to leverage agents uh or continue to leverage those insights into better review flows so that we don't need to get a human to identify that stuff again in the right.

SPEAKER_03And that's just for those pull requests that do require a human interaction for for a review. Exactly. Awesome. So the orchestrator in this sense actually does sound like kind of what a human would do today in in that kind of flow. So it's uh And and and let's talk a little bit about each step there because you know if if we were to um you know ask an agent to do various things, whether it's coding, whether it's design, whether it's a bit of architecture work or whatever, at the various stages, it's not always going to get it right.

SPEAKER_01Um so twofold. One is we intentionally shift task ownership and task decomposition onto the human at the moment. So we're not saying give the dark factory your Q1 plans and it will go do all of that for you. We're saying give the dark factory a well-scoped linear ticket like you would to a junior or mid-level engineer, um, and it will go implement that for you. And so there is an expectation that there has been a fair amount of design work, design thinking, potentially whiteboarding, potentially uh discussion beforehand, um, which you are then able to encode in the linear ticket, which then kind of guides the agenti implementation after that.

SPEAKER_01And so think of it as your job no longer is take ticket and go implement it. Your job is have the whiteboarding conversation with your colleagues, capture that either as like a granola transcript or a screenshot or whatever, and then work with an agent to then translate that into a set of tickets that the Dark Factory can then go implement for you. Um, so there's a little bit of that shift of prior work up front. In terms of the actual does it follow what you do? There's uh one thing is implementation is now free or functionally free.

SPEAKER_01You have to pay token costs. So kick off six, kick off seven. Like you've got a couple ideas, just go kick them off and don't send them an auto merge and go look at the code. Like I kind of almost view the fact that there is a potential like wide state space of potential solutions as either you don't care about it, and so you should accept whatever satisfies it, or B, you're unsure, and so you should use the agents to then help explore that state space and you can ground your analysis in a real code artifact, like how it looks and how it feels, rather than just in the theoretical abstract of I think this is gonna feel nicer if the interface is designed this way or that way.

SPEAKER_01And so that kind of approach I think is quite powerful because it does it's allowing you to kind of do more experimentation because you do then don't need to run the experiments, Darfactory does it for you, and you're just kind of looking at the outputs of that. Um, on the taste and the architecture specifically, our solution to that has been verifiers, which are basically kind of like natural language single statements that can resolve to yes or no to a Boolean about a specific piece of code. And so they're the mechanism that we're giving engineers to kind of instill principles about the code um into their own specific parts of the code base that will then only trigger when those parts of the code base are modified.

SPEAKER_01Um and so that's a deterministic trigger, or that's it's a deterministic trigger, but it is sent to an LLM as a judge in order for the Boolean evaluation. Um so think along the lines of something like these files should uh only ever import from other library files. Yes, you can technically write a very convoluted regex um in order to do that as a deterministic lint rule, or you can encode that principle of like, hey, we want library to be the single source of truth, and so things should only ever be importing from the library and should never be importing from other modules within the code base in order to uh kind of maintain dependency, clean dependencies.

SPEAKER_01And so that kind of taste is really hard to write deterministic rules for, and we've historically kind of caught it in code review. Agentic code review is uh is expensive and slow. Can we shift kind of these discrete chunks into a human legible form that we can then put next to the code so that the agents can go bang their head against the wall in terms of making them pass um as part of the CI flow? So I think that's a little bit of the shift in framing as well.

SPEAKER_03And it does feel like those like I think you've alluded to that, the verifiers are very, very fast to run. Yes. In terms of the this, you know, the the the task that the agent is trying to do as part of that verifier, um it sounds like the scope is much, much smaller as well. So for example, if that, you know, if that one check was part of a code review, it feels like that's gonna be missed by an LLM because not all the time, but that's gonna be missed by an LLM some of the time because the code review covers a huge number of things. Whereas if it's a small scoped uh piece that an LLM just needs to check on this file or on this set of files, this one thing, it's gonna have a higher chance of actually actually validating whether that's done or not.

SPEAKER_01Correct, yeah. And so like a bit of our philosophy is a genetic code review should be the catch-all for things that we miss. But once we have identified something that we think is important, we should be shifting them to verifiers if we can, because as you mentioned, they're a lot cheaper to run. It's an LLM call over the diff. Um, and so we can uh assess that. And we also run one LLM call per verifier. So that LLM is really just reasoning about this single verification, not about all of the conflicting priorities that it might have in its head. So we're trying to kind of sketch out a boundary around the code, around like what is acceptable code from a bunch of different points rather than trying to build like a smooth, a smooth line, basically, in a skill or in an injector review or something along those lines.

SPEAKER_01The other kind of principle that we have is anything that is a verifier that we can make deterministic, fully deterministic, well then we should make that deterministic. I think historically a lot of people haven't written really complex link rules because they're annoying and finicky and they slow human engineers down and they're hard to write. Well, right now we have no human engineers. Agents are fine with annoying Infiniki, they can very happy to go bang their heads against the wall in order to make something work. Uh and they're not quick, but like an agent can write it. Doesn't matter to me how convoluted it is. And so that then allows us to not even pay the LLM costs, then we're just playing the CPU cost of kind of uh deterministic uh verification in that regard.

SPEAKER_01And so it's like a sequencing almost for engineers to kind of build their the things that they had in their head around quality or around what good code looked like or how good code smelt. We can kind of put that into the code base in that sequence of the very, very quick deterministic verifiers, linting rules, the uh LLM as a judge verifiers that are just over the diff, which are natural language and so probably too fuzzy to express, but easy enough for a good reasoning model to understand, and then the catch-all agentic review. One of the really interesting things about, I think, about that sequence is that we can run automatic promotion.

SPEAKER_01So every night we look at kind of the PR comments that are being posted by agentic review and human review, and we assess whether any of those should be verifiers. And so we can kind of continually shift left in that regard to get faster, to get quicker, to get cheaper, but also importantly as well for me, I think, to get more human legible. If I look at a verifier, it's much easier for me to reason about that than to read a long skill.md and figure out uh is this thing going to be correctly weighted for something that I think is really important against all of the other compounding trade-offs that it might have.

SPEAKER_03Super interesting. Let's go into what worked and what didn't work. Obviously, this unlocked, it sounds like a ton. Yep. Um what were the biggest learnings you had early on with this?

SPEAKER_01Uh coding agents are really good, and if you give them a wide enough scope, they will do some great things. I think there was definitely a bit of a kind of bias internally going from kind of going from a philosophy of I've got to be their babysitting the coding agent to I shoot, I'm I can't even look at the logs for this because it's running on a sandbox in the cloud. Um, I better make sure that my tickets are super well scoped and super small and super tiny, kind of like intern-level tickets. And now we've slowly been, and this is cultural, not technical, slowly been relaxing those constraints.

SPEAKER_01And people have been like, ah, cool. What is actually the biggest thing that I can give to the dark factory and see it produce something that is relatively valid as a result? And so the tickets, was that a trust issue? It was trust it was pure trust issue. I think trust comfortability seeing seeing the results. Yeah. We're asking people to trust that something that is kind of hidden, it's called a dark factory because we're not looking at the logs as it's kind of operating. We are postdoc, but not kind of in the actual operations. Um, they need to be able to trust that that's going to do something that they would be okay with.

SPEAKER_03Yeah.

SPEAKER_01Because ultimately, like we still have an accountability mechanism. The person who filed the ticket is still accountable for the code that got uh produced. They still have to, if it's in the production part of the code, base, they still have to review the PR. Like we're not saying ah, you can abjugate all responsibility because the Dark Factory did it. Yeah. Um and so we're now asking people to assign their own accountability to something that has been autonomously produced. Yeah. And so there definitely is going to be a gradual kind of trust adoption curve. Um, and we will continue to kind of go through that. And I think it's going to be never-ending because there's always going to be kind of an increased scope or increased verification you can do to build that trust.

SPEAKER_01Yeah.

SPEAKER_03I like to think autonomy is earned, not enabled. You can't just turn something on and say, we can now go at full speed. It's about building that cultural trust internally in your organization to get to the next to the next stage.

SPEAKER_01What what problems did you have then? So just uh on that trust bit, I think one of the things that we've been trying to do on the team to build that trust is by uh running as far ahead as we can and running into the walls early and uh mucking up, basically. So we definitely have that. Let's go deep there. So I think um the most interesting there's two kind of classes of ones, one around context and one around um verification. In terms of context, uh I pull data for the number of slack threads uh sent on our team channel uh versus the other team channels kind of every week.

SPEAKER_01Uh and our team, despite being slightly smaller than one of the other teams of the company, uh is sending about between 10 and 15x as many slack messages per week uh on our team channel. We have had to overcommunicate everything that we're doing because we are doing so much more quicker. And also with auto merge on, everything is changing in ways that we don't really understand. Last time I looked, I think about only 5% of the uh Dark Factory code base had ever had a human look at the PR and approve it. And I'm not, I don't think people actually go look at code anymore because they're all just using agents.

SPEAKER_01And so that means, by implication, 95% of our code base has never had a human look at it. Um and so we don't know what's going on there. So we need to build the signals and the context on top of that to give us trust that this is working the way that we think it's gonna be working, but then also we know what everyone else is gonna be doing. And that and that 95% that is of the dark factory. Yes, of the dark factories, basically. Yes, everyone has looked at the project code base. Again, we have human review required there, so that has been looked at, but it's for the dark factory itself. Um, and so we needed to figure out mechanisms in order to make sure that everyone knew what they were doing, make sure that everyone was comfortable with their own work getting thrown away, and make sure that we understood how the code base was changing over time.

SPEAKER_01Um, and so we definitely still haven't figured this out. I still will wake up on some days and be like, uh, I didn't, I don't think that's a good idea. Why did that land? Yeah. But the intention, what we're trying to do there is overshare the context so that we're aware, but then also fix forward. It's not roll back a PR because it wasn't the thing we wanted to do. It is, okay, cool, what we missed on the verification side that allowed this to land, either in a verifier if it's a human concept, or on tests if it's something bad, so that we can continually fix forward and build those guardrails, earn that trust on the autonomy side that you mentioned. In terms of some of the more spectacular failure modes, um as part of this, we changed our core queue a lot uh in the early days because we weren't verifying it strongly enough.

SPEAKER_01And by what I mean by that is we would have a fix, actually, concretely, we had an issue where uh PR comments were being double counted. And so they would end up on the queue twice, and so you would get a agent in two different sandboxes responding to the queue. You could have race conditions where it might produce two different uh code uh commits as part of that. Sometimes the second sandbox would start up after the first one I already committed, so it would recognize, hey, the work's being done, but we just it wasn't very effective.

SPEAKER_03And the queue here, just for context of what the queue is, the queue is the list of tasks that the orchestrator needs to pull to then pass to an agent or do something with or something.

SPEAKER_01Exactly. Okay. Exactly. And so we would fix that, it'd be good for a day, then it would regress, then we'd fix it again, then it would regress, then we'd fix it again. There was a point where I think we had like 60 PRs over two or three weeks, or sorry, two or three days that were all trying to fix this. Um and so this was just causing constant churn and frustration because we'd say we fixed it and then it would regress, and it was just it was a pain. And so the way that we wanted to try to fix that because we were still going to continue to let agents do whatever they wanted, was really lean into the verification side on that.

SPEAKER_01And so what we ended up doing was building a quint formal uh model of our Q behavior, um, specifically the uh independency of um being able to kind of add stuff to the queue so that we could uh verify that on every single PR that we hadn't broken that uh formal model. Um that took I wouldn't have known how to do that in pre-agent world, just flat out. I knew what a formal verification was, wouldn't have known how to actually build it. Uh took me about a day with Fable and then using the Dark Factory to go implement this, and we've had absolutely zero reoccurrences of that after the fact.

SPEAKER_01And so that for us was very painful in the moment, particularly as we were scaling up. Um, but then B, a really important insight of like, no, it's not just the kind of verification that we used before of like test and shared human context and like trust that a human has enough context about everything else going on to know that this is a sensitive part of the code base. No, we actually have to lean in even firmer than we had before because agents don't have that contextual information and we don't want to give it to them because that burns to tokens, which is therefore more expensive and slower. Um so that was kind of one really interesting, annoying, frustrating bit of time.

SPEAKER_01Um, the other one was an experiment we ran during the offsite, um, which I was really excited for and which really did not work the way I wanted it to, unfortunately. Um, and that was uh explicitly about stress testing this verification layer. My personal belief is that if we get our verification layer strong enough and if we are oriented around specifically the behaviors that we care about as an organization, not the actual implementation of the code, I don't care about if you how when I click a button on the UI, the results appear. I care about when I click a button on the UI, the results appear.

SPEAKER_01And so we should be able to, just from that verification layer, be able to rebuild the entire system from scratch in a new language. Um so we tried to do that. Uh we currently are a Python implementation. Uh Elixir is-ish a good fit for this in terms of having uh kind of the actor model. And so we took our verification layer only, so none of our unit tests, nothing um other than kind of our end-to-end tests, our integration tests, this formal uh model that we were talking about, um, and some of the properties that we uh that we verify kind of on the surface boundary rather than on internal dynamics.

SPEAKER_01And we gave that to a coding agent to then go plan out and uh building that in Elixir from scratch. Um and then we went into that. Um that worked, it built something, but we hit kind of two major roadblocks that kind of prevented us from being able to call this a success. One thing I will note, our Q was perfect. Q worked totally fine. So the formal model held up well there. But what it ended up identifying was even after kind of all the investments on verification we thought we had made, we had some pretty big blind spots. We weren't actually verifying formally anything to do with the labels on linear, which are a very important routing mechanism.

SPEAKER_01That was all just built into unit tests, and our end to end tests were just testing the happy path there. And so uh when we ported over the end-to-end test, oh shoot, suddenly our um suddenly the way that we were stacking PR is broke because that wasn't tested in an end-to-end test. That was just in uh unit test. So that's core functionality that wasn't verified externally and so was broken in the Elixir implementation. We weren't able to kind of push out to production, um, even though we run in a sandbox production environment. And so then I had to manually babysit an agent to go do that.

SPEAKER_01And so that's an area where we want to be investing in in terms of getting the dark factory to be able to make non-destructive changes in production environments, um, or at least the sandbox environment first and then eventually production environments to take the human out of the loop there. Um and then finally, there was just some other bits of the functionality. I mentioned the stacking of PRs um in terms of a feature branch, I mentioned the labels, um, I mentioned, or rather, there's also uh some of our GitHub functionality, specifically in terms of not just comments, but on CI batching where we try to take in all of the signals on CI failures and send them as a single as a one-off rather than uh sending them as a sequence that isn't captured by an N10 test.

SPEAKER_01So there was some pretty core functionality and improvements to the system that weren't being captured by this verification layer. And so despite really wanting to turn it on, well, I did turn it on and then it immediately broke and then I fixed it, and then it turned on again, and then it broke some people's PRs, and I had to be like, okay, cool, this didn't work. Um, and so I think the real learning from that is this I think is doable. We got pretty far with it. I think it is an even more ambitious framing though than what Anthropic did with the Rust rewrite, where they had kind of the original Zig source code as an input to that.

SPEAKER_01But I think it actually allows you to write idiomatic code better because our Elixir implementation, I'm not Elixir expert, so this is entirely LLM as a judge, agent vibed. Our Elixir implementation evidently was idiomatic elixir. I don't really know how to assess that, but evidently so, uh, rather than being Python 2 Elixir ported. Um, but we need to make sure that we cover everything. And I don't know yet exactly how we're going to continually lever up on that, other by than by continuing to do experiments like this to be able to identify where the gaps are, because that's the only way that we can build trust and really let agents rip when we're confident that that boundary, which we do require human review on, is super solid to the extent we can rewrite it.

SPEAKER_03So so one of the things I'm getting here from this discussion, from this from this conversation, is that uh you know, in building this Dart Factory, this isn't something that we can n necessarily just turn on and expect everyone to use. A lot of our learnings have come through usage. Yes. And I I'd love to kind of like switch briefly to adoption um and onboarding. Uh so so uh you know, when we would we be able to now with the dark factory that we've got now, if we had a new um team that had never used the dark factory, do you feel like they'll be able to just like on one on you know day zero just switch in directly to a to a scenario where they're using a dark factory, or do you feel like they would need to grow into it like layer by layer, piece by piece?

SPEAKER_01Uh so we had a new starter who on day two was adding verifiers to the dark factory, which as I was just saying is one of the most important parts of the dark factory. Uh so yes, I think you can get new people uh onboarded to it, but it does require a specific mindset too. Um I think it needs to be folks who are recognizing the fact that agents are not just doing the same thing but faster, but are fundamentally new workflow primitive in how we do software engineering. It's this shift in my mind of like what is the role of the engineer?

SPEAKER_01I personally don't think engineers are gonna go away. Um I think there's always gonna be a need for technical expertise in terms of understanding what the systems are. But just as your mechanic doesn't build your actual car but can go debug what happened with your car when it breaks, that I think is gonna be more so the role of the engineer in terms of helping build and create the environment for a factory to run in, and then debugging why it broke and then improving that. So more around the outside. And that I think for kind of a new starter or a new person, it's definitely a mindset shift, but it's still engineering, just engineering at the systems level, which is what you were meant to do as a senior staff, principal level engineer, anyways.

SPEAKER_01We're just now asking junior candidates to start doing that right away.

SPEAKER_03Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

SPEAKER_03Alright, back to the episode. So an individual uh can join an existing team that has all of this set up for their project. What about an organization that is trying to that is, you know, perhaps where Tesla was a little while back and not using uh not using a Dart Factory or a software factory and and perhaps uh using the Clauds and Claude Code and Codecs and things like that to assist them to accelerate that building of software. Is that something they can snap to directly?

SPEAKER_03Like we're gonna turn on dark factory and start building this way, or is that something that should happen gradually in layers?

SPEAKER_01Yeah. So I've been talking a lot about what we're doing with building the dark factory, but as I mentioned, like we intentionally tried to front run a lot of these issues here. What we've been doing with the rest of the production code base is that kind of incremental adoption because we're not gonna come out and say, hey, everyone at Cecil, you must use the Dark Factory. We've gotten to that kind of 65% through or 65% of PRs going through the Dark Factory by convincing people that this is a fun, new, cool way of working that speeds them up. Um, and that is very much so by helping them lever up the parts of the code base they own through all these uh tools and mechanisms that we've identified, specifically around the verification, um, to be able to allow them to build that trust and to build that uh faith and automation.

SPEAKER_01So, concretely, what that looks like is just start with better context in your repo. Like that already improves your local developer flow. Like people, companies, organizations, I think really need to be much more intentional about the context in the repo, um, because that is ultimately what goes into the agent, which ultimately, in my mind and part of my kind of belief of where the value is. Is that we can provide more important than the actual raw underlying model. The context that you give will change model results more than swapping a model will. And so if you feel you're at a good place in that regard, then start layering on the verification bit because that then also helps you trust what agents being driven by local engineers are producing because they are verified two ways that kind of go beyond what you might have done historically.

SPEAKER_01So tests are dead cheap, dead cheap to write, write tests for everything, write end-to-end tests for everything, write integration tests for everything, do behavioral tests for everything, make sure that that kind of layer of the thing that you actually care about, which isn't just what the code does, it's the behavior, it's the product surface, it's the things that is expressed. Make sure that is super solid. Um, because that mechanism will both help build in adoption kind of from your internal engine, because hey, your agent now is producing better work without you needing to take kind of the reins and guide it through that, as well as fully producing higher quality uh outputs over time.

SPEAKER_01I think then you start layering on into the more non-deterministic aspects. So adopting verifiers, adopting these kind of discrete units, which are still very human reviewable, very human understandable, um, but are able to encode kind of fuzzier concepts than you would be able to in a deterministic linting rule. Again, you can then have something deterministic run that, feed that signal back into a local agent or into a dark factory, and use that to kind of encode preferences in the code base, and then into a gentic review and then kind of levering up in that way. And so these pieces that you can adopt, particularly around the verification side, these both do better when you're kind of, or rather allow your engineers to do better uh when running local flows, and give you the building blocks and then f uh move closer and closer to the fully automated factory.

SPEAKER_01The last point on my mind just around the cultural adoption bit, I think the most important thing we've been able to give to the engineers here at TESL is the feeling that they still own the taste and the quality in their code base, and they're not just giving that up to an agent, but that we're trying to help them figure out a way to scale out their taste to all of the agents that will be running on their code base. Previously, let's say you were a staff-level engineer who was responsible for a specific part of the codebase, the way that you would enforce kind of quality there is doing all the uh PR reviews or doing presentations or writing docs on what coding standards look like or writing a style guideline or stuff along those lines.

SPEAKER_01Now you encode that within a set of verifiers. Your agents, all the agents, not just your agents, all of the agents of everyone in the company is going to be running that to uh those verifiers for those bits of the code base, and suddenly your impact is now scaled across everyone. And so that then forces those agents to engage with it and also allows you the ability to kind of take your learnings over however long you've been in the industry and give them to the agents in a way that makes them better. Um and so that I think is where the truly scalable kind of shift in mindset is coming from.

SPEAKER_01It's not just kind of like what can I do to make my local agent better, but then can I commit that into the repo in a form that makes all of the agents better and also then gives me the base to build up towards a dark factory future?

SPEAKER_03Now a lot of people using Tesla today are using that to you know as the registry to store their context, to evaluate and improve their context, distribute their context across organizations in private and public uh uh workspaces. Um but the Tesla agent, of course, while it can be a great interface to a lot of those uh functionalities and capabilities, the Tesla agent can help here as well in doing a lot of what you just said, right?

SPEAKER_01Yeah. So we think of the Tesla agent kind of as like a natural language interface for being able to build the building blocks of for Dark Factory. Um and so it has knowledge of all of the bits within the Tesla CLI that we've kind of touched upon here. So when I was talking about our own internal uh code review, that's Tesla change review. When I was talking about verifiers, that's Tesla Change Verify. We also have Tesla Change Risk, which is our own internal risk implementation to assess whether a PR needs human review or not. Um and so all of these kind of like core primitives are baked into the Tesla agent and the Tesla agent knows how to set them up in your repo.

SPEAKER_01Um we, as I was mentioning earlier, we don't think the orchestration layer is like differentiated. Everyone probably should build their own orchestrator because it's simple and allows you to then integrate against a whole bunch of uh your own internal systems. Like, do you use Jira? Do you have your own internal custom uh ticketing system where your sources of knowledge? Like that is important work, but the differentiated work in my mind comes from the loops uh that you're able to build on top of that and also these primitives that you're able to compose. And so Tesla Agent will then walk you through how do I can set this up in your repo using your context, using your way of doing it.

SPEAKER_01So for kind of open source repos, we'd recommend using GitHub Actions. They're easy, they're cheap, they're everywhere. Um, and so we can then uh Tesla agent can then work with you to be like, cool, I want to do a review of all of the PRs that have landed in this repo every single week and derive verifiers from those. Tesla Agent can do that with you. Um and I'm being key there to say with you, not for you, um, because that verification bit, just to reiterate, is something that I think where it's the human leverage comes from. That's still very much so part of the core engineering job because that's where you're encoding your taste, your preferences, all of that kind of stuff.

SPEAKER_01You're still in control, which is but the actual like go write the GitHub action, do it, yeah, I don't want to do that. Tesla Agent will do that for you.

SPEAKER_03And and and if I wanted to get started, I've got a fresh project that I'm building, I'm currently, it's I I'm in a genetic coder, I'm currently building using various agents, I've got various skills and context in that in that repo as well. I'll start up Tesla Agent. Where do I start? What do I what's the first command?

SPEAKER_01What's the first prompt I should write? Cool. Uh so I guess two commands come immediately to mind. One is do an audit of your code base. Um just ask it to help with that in terms of being able to understand like what context do you have? What where are agents thrashing? Um, we have the ability in there to also go look at some of your local logs. Um, if you are using Clot Code, for example, it knows where to go find your logs for that session. This stays all local to your machine, uh, to be clear, but it will help you be able to identify, like, hey, this is something that the agents have done that is constantly thrashing and constantly kind of causing issues.

SPEAKER_01Do you want to write a skill for that? And then do you want to kind of be publishing and versioning that skill? And so that kind of gives you an initial set of insight into like where current friction points are or where you're already great and you don't really need to do much. And then the second one is on those loops. It's like, what is the first loop? Like get the agent to suggest you a loop. Um, my suggest my um personal bias would be probably something looking at the PR surface or something looking at like docs updates or maintenance. You know, those kind of tasks that you know you should do, but you never want to do, and so you get still docs.

SPEAKER_01Agents and loops are great at that. Um, and so working with the agent to figure out kind of what the highest either highest value loop is uh for you to still then be human in the loop, or highest value that you're comfortable running autonomously. Because it's a really interesting and cool feeling to be able to wake up and see, hey, we call them maintenance agents internally, hey, my maintenance agent ran overnight, identified three issues, filed PRs, they're all in auto merge, and they haven't landed. Yeah. Um and so that just is great.

SPEAKER_03Just seeing work done. Exactly.

SPEAKER_01And it's not work that I had to think about.

SPEAKER_03You didn't even have to initiate it.

SPEAKER_01It's uh I found the signal and I want the signal to go up or go down or do something. Yeah. And then maintenance agent goes and does that.

SPEAKER_03Amazing. So that's available in Tesla Agent right now. So if you have the Tesla CLI installed, just type Tesla Agent and the CLI on the terminal and you'll uh you'll you'll drop into that uh it'll drop into that environment and you can ask those uh ask those prompts straight away. I think it's time. Let's uh so we go back and have a look at uh some of those uh linear tickets that we raised. Let's see if the dark factory's uh done something with them.

SPEAKER_01Definitely. This one, as a reminder, is about adding the metrics for where we did the inference geo within the dark factory. And so we can see that, hey, there's been a PR associated with it, and hey, that PR has been merged. So I'm just gonna click over to the PR here so we can see what happened. Um so we've got the summary from Kikimura, as I mentioned, that's our internal name for our Dark Factory. Uh we have Coderabbit who runs on review. Um, this GitHub action here is our own uh individual review harness. So these are the uh comments that have been responded to by Kikimura and resolved, uh given by our own uh action.

SPEAKER_01Uh Codora Rabbit also then provided some harnesses uh some comments as well. Uh there are some CI failers. Thank you, Kikimura, for fixing that. You can then see that Coderabbit approved it and Kikimura merged it. So this was no human has looked at this up until the point where I'm looking at it now. This was fully autonomous.

SPEAKER_03This joins that 95% of uh code that has been merged with no human in our Dart factory implementation.

SPEAKER_01And I'm not gonna go look at it because I trust that the test pass, and therefore it at least isn't broken. And if it is broken, we will find that out and then we will fix forward.

SPEAKER_03Awesome. So this was the first linear ticket that uh that didn't require the human interaction. Perhaps we jumped to the linear ticket that did require that human interaction, which would go into production.

SPEAKER_01And so this is uh the ticket um where we were looking at fixing the quote issues uh in terms of what was uh landing on our uh web UI. So again, something that was identified in a audit, uh, but probably should be fixed, and something that is fully user visible because this is the code or this is what users see on our website. We can see that uh there has been a PR associated with it, uh, and it is also merged. So clicking over to that, we see that 46 minutes ago, so uh 10, 15 minutes after we kick the ticket off, that's when the initial uh PR was opened by Kikimuro, or internal name for the Dark Factory.

SPEAKER_01We see uh CodeRabbit go kick off as well as uh our own internal code tool. Uh bunch of issues found. Kikimuro then responds to it. Uh Code Rabbit approves it. Then Sahil, who's one of the engineers on our team, as I mentioned, these required human review because they are production touching. He then approved it and merged it. So the only human touch points on this PR to get something from linear ticket to production is that required human review because it is still production code, and someone clicking the merge button.

SPEAKER_03And so that so the fact that Sahil did that was because this was this was flagged or this was marked as something which requires human review because this is a critical system, maybe a production system, or it might not be a production, but it might just be some critical piece of code that we require a human to look at before. Do you feel like there'll be a stage where we'll fully trust this?

SPEAKER_01Yes. There will. Um I think so one of the things that we are looking to invest more in in the near future is how we do risk classification on PRs. Um I would love us to get to a point where we are able to merge straight to production for low-risk PRs according to that classification. And the way that we think about that classification, like a lot of else at TESL, is what is the human visible mechanism for being able to encode that so that we can then reason over it rather than an opaque classifier model that's doing a binary classification?

SPEAKER_01What is the skill we can give to the agent that encodes the things that we think are important so that we can then tune that to that specific risk level? And so something like this uh probably would be auto-merge at some point in the future because it is a user visible, but it is just a documentation change. Doing a full database migration, I still think that one actually might be because that's easy to verify uh mechanicalistically. Uh in terms of switching over what uh infrastructure our pipelines run on, that one might require some human uh in the loop. But I think we're working towards that goal.

SPEAKER_03So during this podcast, we created two linear tickets that have now had PRs created and have been merged, one into production, one into uh the Dart Factory uh codebase. That's two of six to seven hundred pull requests that will be created this uh this week at TESL. I think we said we're about 60-70% of all pull requests at TESL being created through the Dart Factory. What will it take to get that to 100%?

SPEAKER_01Uh building trust, um, allowing shifting left. So actually, stepping back quickly, we think of it in shifting left and shifting right. So from the PR, shifting left is helping engineers create PRs, shifting right is working on the verification of those PRs so that we trust what the output is. Um and we think we have historically focused more on the shifting right to build that initial level of trust of like, hey, the code the agent has done is roughly correct. There's always more that we can do, and there's always more that we will do there to kind of improve the quality of the output.

SPEAKER_01But we think we're in a good enough spot there. Where we really want to be doubling down and investing in is on how do we make this the easiest way to do work at TESIL. Um, not just the fastest or the highest throughput or anything along those lines, but where can we bring the Dark Factory into making that even easier than possible? We have our own uh internal Slack bot um that listens ambiently in channels and can file linear tickets for you and all that kind of stuff. We want to move or we want to move it into actually being able to propose tickets on linear or being able to do decomposition.

SPEAKER_01We want to move it into the planning process to be able to help you go from PRD and be there as a coworker to kind of be able to riff ideas off of or something along those lines. So making it so that the default is the Direct Factory, not because we're enforcing it, but because that's just what people reach to because it's integrated with the tools that they're using every day.

SPEAKER_03Amazing. Rob, this has been truly, truly fascinating and it's an amazing accomplishment to see how much of an impact this has had on the way we build software at TESL today. And for our audience, our listeners, if you wanted to try this out and actually build your own software factory, head to TESL.io for more information about Dart factories and about the TESL agent. Load up the Tessel agent just by typing TESL agent at the command line once you've got the TESL CLI installed and just ask it those questions. Take a look at my environment. What can I do to improve this and how can I add these verifiers, these these you know, the plug-in reviews, the the pieces into the CI and the code reviews.

SPEAKER_03This is absolutely amazing. It's definitely the the the next level of software development. Rob, it's been an absolute pleasure. Thank you so much. Thank you, it's been fun. I hope everyone learned something today and thanks for listening. Tune into the next episode. Bye for now. The AI Native Dev is brought to you by TESL, the package manager for Skills and Context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the TESL offices in central London. Visit TESL.io forward slash community to learn more, and I hope to see you there.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Your Software Factory Needs a Bill of Materials | Daniel Meppiel

    Nov 19, 20261 hr 1 minAudio

    Developers are about to write skills instead of code, and Microsoft's Daniel Meppiel argues the software factory that runs them needs versioning, a bill of materials and signed receipts before enterprises will trust it....

    Credit the people on this episode

  2. Paul Stack: I'll Never Write Infrastructure as Code Again

    Nov 17, 20261 hr 2 minAudio

    Paul Stack helped build Terraform. Now he says he'll never write another line of infrastructure as code, because an agent harness runs his team's production infrastructure instead. Watch all the AI DevCon NYC 2026...

    Credit the people on this episode

  3. Edith Harbaugh: Code Is Cheap. The Last Mile Isn't.

    Nov 10, 202649 minAudio

    Automated software delivery is moving faster than the people who used to approve every release. LaunchDarkly co-founder and CEO Edith Harbaugh tells Guy Podjarny why rollback should be autonomous, why one-shot coding...

    Credit the people on this episode

  4. Maggie Appleton: Agentic Coding Has Outgrown the Text Box

    Nov 3, 202652 minAudio

    Agentic coding has outgrown the text box. Maggie Appleton, staff research engineer at GitHub Next, argues that walls of CLI output and AGENTS.md files serve coding agents far better than the humans steering them. She...

    Credit the people on this episode

  5. Caleb Sima: AI Attackers Move at Machine Speed

    Oct 27, 202655 minAudio

    AI attackers now move at machine speed, while most security teams still move at meeting speed. Former Robinhood CSO Caleb Sima has seen an AI attacker up close, and he argues the only way to keep up is to take humans...

    Credit the people on this episode