Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow artwork

Edith Harbaugh: Code Is Cheap. The Last Mile Isn't.

Nov 10, 202649 minAudio

Automated software delivery is moving faster than the people who used to approve every release. LaunchDarkly co-founder and CEO Edith Harbaugh tells Guy Podjarny why rollback should be autonomous, why one-shot coding agents fail, and why the bottleneck has moved to the last mile. What we cover: - How automated software delivery and autonomous rollback take humans out of the 2 a.m. incident - What makes a good feature flag, and when a homegrown system stops being enough - Why coding agents need an explicit definition of success before they write a line - Where release management fits in an AI software factory - Feature flags for AI agents: switching models and prompts at runtime - Running a company in the agentic coding era, and why teams should take dumb bets Chapters: 00:00:00 - Introduction 00:02:16 - From quarterly releases to 30,000 changes a day 00:03:32 - Autonomous rollback and automated software delivery 00:06:25 - What makes a good feature flag 00:10:46 - Defining success for coding agents 00:16:42 - Agent fleets and engineers as reviewers 00:24:05 - Feature flags in the software factory 00:29:02 - The bottleneck moves to the last mile 00:39:24 - Runtime controls for AI agents 00:44:26 - Running a company in the agentic coding era Build your software factory, one workflow at a time, with Tessl: https://tessl.co/e0p 🔔 Subscribe for weekly episodes on AI-native development Where has the bottleneck moved in your team's automated software delivery? Tell us in the comments.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_01I remember talking to a prospect and I asked them how often they released and they said once a quarter and I asked if they wanted to ship more and they said no. Which now just seems absolutely jaw-dropping. You know, Anthropic just came out with a blog and they talked about how they have moved from 2,500 changes to 30,000 changes in a day. It is just a very different pace that needs a different sort of workflow for production. If a deploy goes wrong at 2 a.m. and it's rolled back within a second, did it really go wrong?

SPEAKER_00The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Fajani, and meet Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. AI DevCon is coming back to London. Our first London event sold out in June. After our second New York event this November, we're heading back to the brewery Moorgate in 2027 with a bigger lineup and two full days of talks from the people actually building with agents.

SPEAKER_00You'll hear honest takes on where the industry's at, real stories from teams adopting AI, the wins as well as the messy bits, and where they think things are heading next. It's on June 7th and 8th. And don't tell anyone, but we've got a really great deal for you. Scan this QR code to grab an early bird ticket for just£100. See you in London.

SPEAKER_03Today we'll talk about what is even a release and how do you kind of manage these uh kind of insane pace of application deliveries that we have. Uh and we have a guest who's probably one of the best equipped uh people to deal with that, talking about sort of delivery and you know, dynamic delivery and such uh in of applications for quite a few years and also kind of a good friend, Edith Harbaugh, uh who is the uh founder and CEO of LaunchDarkly. Uh Edith, thanks for joining us here.

SPEAKER_01Yeah, it's great to be here in the virtual studio.

SPEAKER_03So I guess for for starters, I'd love to kind of get your view a little bit of what does it mean to do this type of feature flag controls, release management, give us a little bit the right terminology uh at you know a pace that seemed or like makes the cloud era pace feel like a snail.

SPEAKER_01Yeah, so it's been a journey. Um so when we started the company in 2014, it was when cloud was still kind of a new idea. You know, a lot of people were actually still shipping on packet software. Uh I remember talking to a prospect and I asked them how often they released, and they said once a quarter. And I asked if they wanted to ship more, and they said no. Uh, which now just seems absolutely jaw-dropping of that they thought once a quarter was more than sufficient.

SPEAKER_02Yep.

SPEAKER_01Um now uh, you know, Anthropic just came out with a blog and they talked about how they have moved from 2,500 changes to 30,000 changes in a day.

SPEAKER_02Yeah.

SPEAKER_01Like in a single day, um, which uh is inspiring and amazing, but it's just a very different pace that needs a different sort of workflow for production.

SPEAKER_03Yeah, yeah, no doubt. But how so so how do you like how do you think about sort of feature flag in this reality or or about kind of managing behavior? Because I feel today there's definitely an ethos of ship it, you can monitor the sort of the reality of it. So if there's a problem, you can roll back and you can iterate. Clearly, all these things require the right infrastructure to be able to do as well. Um but uh but then when you think about you know this type of pace, oftentimes the the decision of whether to roll back and all that is takes longer than the um the the time kind of you you already want to evolve your next feature, right?

SPEAKER_03You already want to build the next one.

SPEAKER_01Yeah, I mean, and that and that's really so that's really what Launch Darkly and feature flags can help with. And let me be clear, uh, you don't need Launch Darkly to use feature flags. Um this is a technique, we're a vendor. Um I would like to think that we make it a lot easier. Um, but everything I'm talking about you can do on your own. Um so that the the concepts you just talked about are a rollback, and to do that effectively, you do need feature flags. You know, so if you're looking at uh even if you're not anthropic scale, but if you're looking at, let's say, 10 or 100 changes a day, um, you do need a way where you can quickly detect if something is going right or wrong and move it back if it's going wrong.

SPEAKER_01And that's where uh one, you could do it on your own if you had feature flags, but too launch darkly could really help you because you can have that as automated. So if uh if it's going bad in the in the field, you could within seconds say, hey, this isn't working, and have it automatically rolled back. And this is what AI is very good at because uh you don't need a human in that loop at all. Uh you don't need somebody to be sitting at a dashboard looking at it, it could just be a very autonomous rollback.

SPEAKER_01And then you multiply that times 10, 100, 1000, and it's a very good fail-safe if you're shipping a lot of code to production and you want not to have to babysit every single change.

SPEAKER_02Yeah.

SPEAKER_03So how so so there's the technical aspect of figuring out uh how to sort of you know do the flag and sort of apply the change on it. Uh, but you know, I hearing this, my my kind of bigger question is how do you decide that you need to roll back? And if you do that agentically as well, what would you sort of say, we're sort of looking right now at sort of state of the art. Maybe we'll talk a little bit about the journey, but if you're really good at it, how do you how do you make those decisions?

SPEAKER_01Um so a lot of the work we're doing right now is to make that very automated, um, so that you can send us whatever telemetry you have, whether it be error rate, whether it be um uh elevated uh operational metrics, whether it be something as fuzzy as just uptick in tickets.

SPEAKER_03It's a good it's a good question though, about like what is it that you should feature flag? Because like some things get you in trouble if you start kind of getting a little bit too dependent on feature flags. You want to roll them out gradually, but you end up creating inconsistent experiences, or when you decide to roll back, you kind of have, I don't know, data structure problems, or you maybe are letting down some user who has now kind of formed some attachment to the to the capability you just launched. I guess what would you sort of systems aside, like uh from from the you know, probably have more exposure to sort of correct and incorrect feature flags than uh than almost anyone.

SPEAKER_03Like what's a how would you define like a good feature flag uh criteria?

SPEAKER_01Um well so leaving aside launch darkly as a vendor, uh I would say a good one is always one that you understand. Um if you have put in one and you don't understand what it does, that's where you get into a lot of trouble. Um so I've seen people who have still flags, a flag that was once very important, but nobody understands what it does anymore, and it actually uh accidentally got triggered, or uh uh the worst case I heard is um one uh one company that had something to shed a lot of load, and one feature flag, if you toggled it, it would actually prevent you from uploading any new files.

SPEAKER_01Uh and it would actually get toggled and nobody knew what it was. So the number one uh I'd say guideline is just make sure you understand that. And then you can still have a human in the loop uh to make those judgment calls.

SPEAKER_03It's it's so interesting to think about um uh judgment, right? I feel uh when I look at how we build a TESL and I look at the pace, alignment becomes this really kind of interesting uh challenge, right? Because you have a bunch of people, they're individually like they're individually empowered and uh you know just sort of so uh equipped, you know, like they have such powerful tools at their disposal, so that each of them operates like I don't know, a directorship, right, of sort of of people. But uh keeping them uh aligned as they make many, many of these decisions is uh is is is hard because you know it takes the same amount of time to talk to their you know colleague and sort of you know describe what it is that they're they're uh they're building, and it's still a pain or like a product discomfort if uh if if they're misaligned.

SPEAKER_03Like how how do you keep like I I'm trying to think about uh sort of deploying capabilities with feature flags, and so you add to that kind of matrix or to the to the already kind of existing risk of uh I'm not sure what my colleague has shipped, right? And we're trying to kind of reconcile this as we as we fly the plane, you add a potentially very long list of feature flags that further customize and sort of tweak behavior, and you do that.

SPEAKER_03I guess if you the when does this go wrong? You know, when uh is there a I don't know if there's magic, like I guess I'm sort of yearning for these sort of magic numbers of like never have more than a hundred feature flags over the eye. Like, I guess what's the what are sort of guidelines to say, you know, until this point it's helpful and from here on it's kind of detrimental?

SPEAKER_01Uh again, I don't want to give too much of a vendor pitch, which means I'm about to give one. Um I I think the guideline is as many as you can safely manage uh in a system. Um so a lot of small startups, I talk to them and they they say, hey, we've rolled our own feature flagging system. I say, wonderful. That's great. Like if you have like 10 to 20 or 40 and they're well understood within your org, you know, you don't you don't really need a vendor. Um it's saying when you get to a scope when you have multiple teams who are perhaps distributed, uh, you start to need more of a framework about managing them.

SPEAKER_01Uh you might also have dependent flags, which is pretty common where you have perhaps a flag that's uh handling five other subsystems where you want them to move in parallel. Uh and you also want to be really dil diligent about cleanup. Uh so where I've seen people get into trouble is if they've just been adding features for every release. And there's always what I say the boring part of doing the dishes and going back and really examining whether a flag can be used or not.

SPEAKER_03So I guess related to that a little bit, um, I think um I guess what I was also kind of wondering is whether the use of feature flags, and especially as you kind of grow them, depends. Uh I mean you sort of said like how much you understand them, but also about how well can you clarify the outcome, right? And and and in that sense, like I've I I really sort of think a little bit about agents. There's this um uh comparable that I often uh kind of run into, which is uh, you know, in DevOps we were talking about all these like great traits of like, you know, you need to instrument your systems and you need to have good lineage and you need to you know manage your data and have kind of you know understandable kind of uh the uh architecture, information architecture.

SPEAKER_03And then if you did all of those, agents can actually help you run faster. And agents can also help you kind of overcome a bunch of uh uh of uh pains that you had and kind of get your house in order. But if you if you had a bunch of that infrastructure, those are actually now multiplied, like the agents really benefit from sort of having that data and that access. And I guess so. I guess sort of similarly, I wonder if with with the feature flags, you know, one of the questions is like what's your judgment? You know, what is the uh the thing that you measure? So I guess how much does that factor in? Like we're in this world where a lot of things are ill-defined, you know, of what correct behavior is.

SPEAKER_03Like, does that get in the way? I'm also kind of wondering about what you're sort of seeing in the wild. There's there's kind of established businesses, but when you're building kind of newer capabilities even within those established businesses, they don't necessarily know what good looks like, right? Like what is it that the feature flag should aspire to achieve?

SPEAKER_01Well, that that was actually, I think, about four or five questions wrapped into one.

SPEAKER_03There's a I've got a tendency of doing that. Apologies.

SPEAKER_01Well, no, it was a really good question. Um it's it's something that I've been thinking a lot about. Uh, you know, so I originally got it, I originally founded Launch Darkly because I'd been an engineer and then a product manager, and I had built things that just did not work. You know, in terms of uh they worked at an engineering level, but then they got out into the field and customers were sometimes just completely apathetic to them. You know, so and I say apathetic deliberately because it wasn't they loved or hated a feature, it was just that they did not care.

SPEAKER_03They didn't care, yeah.

SPEAKER_01They didn't care at all. I didn't do that. Um that's painful. Um the other part of an engineer is that you put out something and it turns out that it breaks in the field, which is also painful. Um, you know, that uh you always either over-engineer or under-engineer, depending on where the minimum viable product spectrum you are. Uh so sometimes you think a lot of people use it and nobody uses it. Sometimes you think you'll have very light workload and suddenly it's through the roof. Um so I think in this world of agents, it forces you to think about that.

SPEAKER_01Um, you know, so I think about a year ago was the heyday of the one-shotter. Um, you know, which we we even tried internally at Launch Sharkly about, like uh, you know, the one-shot of refactor this part of our code base.

SPEAKER_02Yeah.

SPEAKER_01And that phrase works about as well as you think it might. Which is okay. Like to a human, what seems very clear, which you might give to a team of humans of refactor the code base, they go away and refactor it. To an agent, they just have no idea what that means. Refactor for what? Like, what does that mean? Um, so what we found in our own development is that the the dream of the one-shot agent that goes away and does the work is more like us trying to decompose work into smaller and smaller and smaller and smaller um atomic units uh until we got to a point where we could point agents at different points, parts of the code base, and say, okay, here are your tasks.

SPEAKER_01Like make sure that everything is commented out, make sure that an agent understands it, make sure that um we're looking for bloat. Uh so it was being very specific about what we meant by refactor.

SPEAKER_03Yeah. This is like a defining defining what what you actually want. You know, you need to kind of express that. What good what what do you want and what good looks like. So you give the right instruction and you're able to kind of verify it as part of the kind of software building in this case.

SPEAKER_01Yeah, and and and and even add on to that that agents do not know the concepts of good and bad. Like these are abstract words though.

SPEAKER_03Yeah, no, though what what you think good looks like. You know, like what is uh what is a legitimate outcome, right, versus what is a failed outcome.

SPEAKER_01Yeah, so like if you told uh an engineer like make sure your code looks good, there's usually a common understanding about like what it what that word good means, like you know, um, but an agent has no concept of it. So all of this work that we were doing into defining work for agents is actually really good for a feature flag. Because your your original question was, well, how do you define what success looks like? And it's like, well, you're defining it from the beginning for the agent. Like you're saying, hey, this part of code needs to be performant to this standard, um, it needs to, you know, load within this time.

SPEAKER_01And then those are very useful metrics that you can then push into a feature flag for when it's actually out into the world of hey, all that stuff I told you up front about how I wanted this thing to behave, make sure it still behaves that way out of the real world.

SPEAKER_03Yeah. I I think that makes a lot of sense. And I I I feel again, there's like a very common through line, I think, in everything that you deal with sort of delegating any type of work to agents, which is you have to do the hard part, which is describe what what success looks like, what failure looks like, uh, what are the guardrails that you want them to stay within? You know, like what are what are you know what are the lines, like what's disallowed, what's allowed. And so you need to do, you know, what we oftentimes avoid and just saying like we'll know it when we see it, you know, for for humans, you know, now we need to actually codify that uh in the system.

SPEAKER_03Um how I guess kind of pushing that a little bit to the extreme. How how far would you let agents uh evolve? Like, you know, you're you're probably like the most advanced sort of user of uh of feature flags, or like I'm up there. Um how much would you let them just experiment? Like, hey agents, like like look at all my telemetry and how people are using it, listen to all the customer calls that I've had, you know, you know, like read all of my internal documentations, propose changes, roll them out if I'm being bold here, like you know, build them, roll them out under feature flags, see what the result is, uh come up, and you know, I'll be surprised and amazed at the uh at the great enhancements that you've sort of built for my for my product.

SPEAKER_01Um I mean that's that's the dream. I'd say we are not quite there, but we're moving there. Um like uh we have a pretty strong internal blogging culture at Launch Darkly. Um so we have roughly 200 people in RD, and I encourage them to write, and I try to read them. And I was reading one this morning about how um they'd built a fleet of about 20 plus agents to work on a part of the code base.

SPEAKER_01And then the work of the engineer actually turned out to be just reviewing all the PRs. Um so in my mind they had transitioned more to uh what used to be called like the classic architect role.

unknownYeah.

SPEAKER_01Um I'm old enough where like um you used to have the architects that would just do code review. Um so there was still a human in the loop, um, there was still some judgment, uh, but it just went much faster because somebody else was doing actual code writing. Um and then when this stuff is ready to go out to production, it already has guardrails pre-built in. Um, because that's part of the process is of before you change the code, figure out what the guardrails are. So it is much more autonomous.

SPEAKER_02Yeah.

SPEAKER_01Uh we've we've also done projects where uh we pointed an agent at a backlog of bugs. And I think this is just amazing, um, where you can point it at a backlog of bugs, and it could produce PRs, you know, for sometimes months, if not years, of things that never bubble to the top of fixing, and just start releasing them with Gardros. And I love this because I think uh, as I said, I I came out of I started off in engineering and I remember just the tedium of sitting in a bug review.

SPEAKER_01Like, did you ever see a bug review just like, hey, uh it doesn't work on this browser, there's this cosmetic flaw, is this worth fixing or not? And you just spend hours on that, like it's a tedium.

SPEAKER_02Yeah.

SPEAKER_01Uh versus an age that just like goes through all of it.

SPEAKER_03Yeah, yeah.

SPEAKER_01I don't know if you're seeing the same things in your own development.

SPEAKER_03We're we're definitely sort of, you know, definitely many, many, many things are uh are building there. And so but I guess I I want to then sort of push a little bit on like why not sort of drive the autonomy. So like what's getting in the way? Because like my my I guess my my my view is that the feature flags and sort of this type of kind of runtime management should enable you actually to release stuff that is that is bad, you know, like that is uh but it's okay because you're like gradually exposing people to them. And you know, to the extent that you have enough of a user base, uh, and yes, there's probably some constraints around like some areas in which your tolerance for a mistake is very low.

SPEAKER_03But like why why wouldn't you like why do you need the human in the loop if you have the sort of the feature flag uh capabilities with sort of you know tight management uh and are able to gauge like if you have these sort of two that you have these controls and you have like a pretty good understanding of outcomes?

SPEAKER_01Oh, I mean we're we're depending on the feature, we we basically do that. Yeah. Um like if it's a if it's a bug that's coming in uh on UI, we can fix it within minutes. Um it's it's very little uh human there. Um for things with a bigger surface area, there is a lot more human judgment.

SPEAKER_03Yeah. Yeah. Um But I'm curious, I'm still sort of, I guess what would make you unleash this, sort of say, hey, in this area of the product, autonomously improve my product.

SPEAKER_01So I like that you're pushing me. I think the question of autonomously approve is pretty broad. Um so let me let me give you a real example. Like we have SOC uh compliance, we have HIPAA compliance. Yeah. Um so just say autonomously approve is so broad that I don't know if an agent would understand that right now.

SPEAKER_03Yeah. Yeah, it's interesting to So I I think compliance is definitely a legit answer. It's like hopefully a temporary answer, as in, you know, we should learn how to sort of build compliance systems that still have sufficient guardrails that they can evolve autonomously. But I think today that is still a scary proposition. You have to be very confident in the sort of guardrails that you have uh to be able to attest to an auditor uh that that you're doing it in that fashion. Um it just uh I guess I'm sort of pushing on it just because I feel um uh there's like different parts of it, right?

SPEAKER_03There's the judgment of uh of w what is a reasonable change. I think agents are actually already pretty good at that, especially if you're sort of scoping kind of the volume of change for a proposal. They don't need necessarily know that it's great, but they'll have good kind of betting stats over here. Then the implementation, okay, they're getting pretty, pretty good of that, you know, like being able to like build the capability. And then you know, when you release it, there isn't a taste element, right? Or like a curation element. And you know, one way I think about feature flags is just can they replace taste?

SPEAKER_03You know, I don't know, like with with ads, with sort of coupons to supermarkets, with doing it, like I don't know, a lot of taste goes away in favor of data, right?

SPEAKER_01Oh, I I I see the same future. I mean, um, I would like to get there. I I think also though, people like predictability. Um, so a real example is, you know, we have customers who are in education or airlines, and they do not want all of their interfaces to change on a moment-by-moment basis. They're like, you know, the airline clerk, and which who knows might might go away.

SPEAKER_01You know, somebody who's sitting on an interface and is like, I want to push this button, know that the button is in the same place as it was yesterday. They don't want that rate of change. Um, however, there are things on the back end that can be constantly iterating much faster just to improve that front end. Um something we've been talking about internally is uh Launch Darkly, uh the UI is all API driven. Uh about uh making it even easier for people to just have more of a conversational interface with Launch Darkly.

SPEAKER_01Uh where instead of using our uh, for lack of a better word, canned UIs, yeah. Uh that you could be building UIs on the fly with an agent every time you want to talk with us, which I think is great. But I think some people want the familiarity of, I know where everything is. Yeah.

SPEAKER_03This part is like the slow change as a feature. Uh yeah, for sort of some parts of the system is the like stability, really, is the feature. So uh let me kind of shift gears a little bit. So like I I I'm I'm very excited by by the notion of like dynamic behavior uh and you know, how do you how do you control it? I have all sorts of thoughts that probably will sort of suck us down the sort of the wrong rabbit hole over here around personalized software and uh and the likes. Um how how do you think about this type of delivery versus or like as part of the software factory?

SPEAKER_03Is it the same, is it the same thing? Like you build, and maybe I'll I'll I'll stabilize a little bit to say like let's assume the core of the software factory is this like tickets to PR, but then the sort of the breadth of it thinks about the SDLC as a whole. Like, how do you think about Future Flags as a whole, but also maybe like Launch Darkley's place, right, in in that flow?

SPEAKER_01Yeah, so it's it's it's interesting the software factory metaphor has really taken off, I'd say, in the last uh nine months. Um I personally like it, though I know there's a little bit of backlash against it. Um I like it in that a factory seems more predictable. It seems like there are gates and steps and repeatability. Um I know that some engineers fear that it makes them obsolete or just more rote. Uh so I I in my own vision, I think it frees them to be more creative, knowing that there is stuff that they can depend on.

SPEAKER_01Um so that the knowing, for example, that you could go, to extend the factory metaphor, that you could go somewhere and like heat treat your plastic so that it's usable is actually a good thing rather than having to create that heat treater every time. So when I think about Launch Starkly, uh we assume that there's a lot of stuff upstream from us about your how you produce code. Like we don't want to compete with uh a cloud a clot or uh, you know. And then we also assume that you have some sort of code review facility. Uh, we don't want to compete with uh a code rabbit, for example.

SPEAKER_01What we're really good at is that last step of you've built something, you want to release it to the world, and you want to have guardrails around it, um, so that once it's out in the world, uh the things we talked about before, you can either um break up who it goes to. So, for example, if you want to do a phase release where it goes to people in London and then in Scotland and then uh to the rest of the world, you can do that. And also, so if something is suboptimal, you could roll it back. Uh also if you're getting a lot of positive feedback, you can feed that very quickly, perhaps even to a coding agent to enhance a feature.

SPEAKER_01Um, so for example, if you'd built uh a simple workflow and it's getting a lot of use, maybe in the future an agent could say, hey, this is getting a lot of use, let's uh continue iterating on it autonomously without even a human in loop. So that's kind of the vision. Uh what we're doing a lot at LaunchDarkly right now as a vendor, and what I said before is vendor independent, is building up uh agentic controls where you can tune your system about what does and does not need it to be flagged, uh so that you can have a risk tolerance for this is what I want to control, this is what a change that could just go through, but it gets smarter over time, so it's just auto-magic.

SPEAKER_01And you can use that as hey, if any of these measures takes a jump, uh start rolling back. And you could do that in an automated fashion without a human ever knowing something went wrong. Uh we call it internally the the two if a deploy goes wrong at 2 a.m. and it's rolled back within a second, did it really go wrong? Um and then on top of that, what we're adding to Launch Starkling uh as a vendor is a lot of automation around making it a lot easier to know when to feature flag.

SPEAKER_01Um so we've been using our own system for 10 plus years. And there is, I would say, some amount of friction about like you have to wrap a flag, you have to send this telemetry. Um, but in an agentic world, the agent can do all that for you. Like you could say, hey, uh instead of having to look at, hey, I want to wrap this flag, let me think about how what signals to send to it, you just say, agent, do this for me. And it's very automated.

SPEAKER_00Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

SPEAKER_00Alright, back to the episode.

SPEAKER_03So I I feel uh I think there's a lot of kind of value, right, in sort of that portion of it. Uh, but I do I want to sort of geek out a little bit about sort of product scope in this era, because I think the the the the thought about a software factory is just about like how all-encompassing it is, right? And I had uh uh quite a few months ago now, I had Merrill from uh uh Graphite uh on the podcast, and this was before the cursor acquisition, and you know, we we talked about whether code review will be you know a part of the of the factory, right? And and I feel uh I guess I would I would claim today that the answer is yes, code review is a part of the software factory.

SPEAKER_03In fact, if you don't have kind of increasing autonomy, you know, with with sort of growing confidence and competence to be able to uh to merge things more and more you know autonomously, um, then then you're stuck, you know, like that becomes the bottleneck and you're you're just not gonna be able to push through it. Um and also uh in that conversation, we talked about how you know it like at the time they built like their first features to sort of say, hey, we when you found like a bug right in the code review of it, then hey, you can actually sort of fix it over here uh in the path.

SPEAKER_03It's like, well, how how are you different than the coding agent? Uh I don't know, by the way, if the cursor conversation is already happening in the background and that happened, but uh uh you know the um you know how are you fixing there? And then can can't I just run your review as part of that coding agent? Like, why do I need to wait for the PR? Can I can I run it early? And so I guess I'm I'm sort of saying all of this because like how how do you think about does that not extend uh if you think about feature flags and this type of like dynamic insights, you can think of it as like a continuation of quality, right? You can look at things statically, you can look at sort of dynamic tests on it, and then you can look at like actual kind of real world experience uh of this capability that you're releasing it.

SPEAKER_03Like does it make sense eventually? Like what's your what's your um uh uh vision or sort of like a prediction around the world in five years' time in terms of like division of kind of core capability providers, right? You know, in uh in the space?

SPEAKER_01I think there will still be the last mile issue. Like you and a when you're talking about the factory metaphor, I I talked about I was thinking about how there's always a bottleneck somewhere.

SPEAKER_02Yeah.

SPEAKER_01And as soon as you f fix a bottleneck in one place, the bottlenecks move somewhere else. Um you know, so the bottleneck, I'd say, for most of my career has been just code. You know, uh how many engineers can we hire, how fast can they get up to speed, how many and then there's all sorts of artificial metrics of productivity that were gamed, like lines of code. I I don't actually care about lines of code, I care about productive code. Uh now code is incredibly cheap.

SPEAKER_01Like you can ask Claude to make you code, and it will make far more code than you care to. Um and then the bottleneck was code review, and I think there's a lot of very good vendors right now for code review. Um, and then the bottleneck right now for me is there's a lot of code that's been code reviewed, and there's still this last mile of is it actually having a productive impact on the end person? And I say person, it could be uh a system, it doesn't have to be a person. But that gauge of is this doing something that I want?

SPEAKER_03And I think is it kind of uh properly improving whatever is those business metrics that I've sort of defined as uh as good?

SPEAKER_01Yeah, and I and I think it a change from the world in the past, uh, because I talked earlier about the age of package releases, is you used not to be able to get that telemetry. Like if you were shipping, like let's say, uh I'm old enough to remember Windows 95, like it came as this immense 32 disc blob, where it's impossible to say, hey, is this feature better than this feature? You just couldn't know it. But now with the ability to break down uh changes into smaller and smaller bundles, you can actually have a really good gauge of is this working?

SPEAKER_01But you also want to be able to say, hey, all these changes together, is this having a holistic good effect? Right.

SPEAKER_03Yeah. No, I think the management of it is quite complex, and I feel like there's a lot of uh industry learnings. I guess when I think about software factories, I feel on one hand, I feel the software factory foundation will be all-encompassing, as in it it will, it will like eventually you want to aspire to a thing that is the full loop, right? That that goes all the way. And i i I will say that many years ago already, various uh uh kind of uh tech leaders that I spoke to spoke about their RD organization as a software factory, which may or may not have been endearing to their teams.

SPEAKER_03But they they are like, you know, it was still about like a repeatable process of software generation that goes end to end. Um I do think within that there will be sort of worlds of specialization for sort of difficult parts of it, for security, you know, for uh you know, for various aspects of sort of quality. And I think uh I think very much likely that sort of uh feature flags is its own domain. But it's just sort of interesting to think about scope product kind of in the uh in that sort of destination.

SPEAKER_01Yeah, and something I want to emphasize is that um you don't want to lose the the forest for leaves, if that makes sense. Like you can definitely over-optimize for very narrow features and lose track of uh this isn't actually product anymore, and we've over-optimized. Uh uh an example I'll give is um you know, I you can realize that you've backed yourself into a niche product, that you have a very passionate bunch of loyalty, but you have not one in the broader market.

SPEAKER_01So I worry that worry is perhaps too strong, but I I worry that with the rate of change that companies will back themselves into a quarter if they don't have some sort of level of judgment at product management about like, hey, we've overoptimized for you know 50 people in London who love this one feature, but the rest of the world is ambivalent.

SPEAKER_03Yeah. Oh, I love that. So like feature flags are amazing, but they don't replace your judgment. You just they they just sort of enable you to kind of apply it more wisely. Uh you still need to do that. And and I I agree you'll kind of be carry the risk of sort of hitting a local maxima, you know, for for sort of optimizing for it and not really having you have to have some vision. And I guess that comes back to like accountability, uh, because if you're gonna make a bold decision, even a medium bold decision, but one that is not sort of purely in the data, but rather your sort of analysis and sort of uh intuition on it, um there probably needs to be a human signing off on that type of change.

SPEAKER_03Uh maybe if you're like very, very big and very, very confident and have a very, very fast learning cycle, uh, you know, whatever the TikTok algorithm, right, can probably like you know, make more of these wild bits, but uh Yeah.

SPEAKER_01Yeah, I was thinking about it because um I was a product manager at Tripit, uh, which was uh at the time in the 2010s, the number one mobile app on the Apple iOS store. And it was a self-uh fulfilling prophecy. We were the number one app, so people would go to the app store and download us, we were the number one app. Yeah, which was kind of a great place to be. Um but the founders of the company, I wasn't a founder by the way, uh I was product, um, had made this really good decision that they focused on Apple early, and one of their big competitors focused on Blackberry.

SPEAKER_01Yeah. And I don't know if that was something that an AI could ever decide. Make the choice. Yeah. Yeah, they they doubled and tripled down on we want to be the dominant app on Apple, and then Apple expanded versus their competitor who had this shrinking market because people at the time though there there was a time when BlackBerry had more market share, it was it it wasn't sustainable. Yeah, yeah. I think it would actually be difficult to buy a Blackberry right now.

SPEAKER_03Uh I believe I I think I agree. I um I have this sort of uh blog post brewing in my head, which I'm probably never gonna write, that is that the uh greatest advantage of humanity is irrationality. Uh uh it's uh it's sort of this sort of desire uh to or like this willingness to say um no no, this is right, you know, against all the data. And this is like how all great companies, how all great you know, pieces of art have been created, how all uh uh so it's not really about whether an LLM or an agent would be able to conceive of this idea.

SPEAKER_03They'd be able to conceive of all ideas. It's about the sort of like irrational dedication, which typically fails, but then for a subset of people succeeds, uh you know, subset of cases succeeds uh because of that. And and sometimes like actually changes reality, right? Like I think the work you've done with Launch Darkly, I'd like to think the work we've done at Sneak, you know, from a uh DevSec uh perspective, you know, they they don't just roll into the market. It's the sort of persistence that actually helps shape some aspect of the market.

SPEAKER_01Yeah, so when I when I so I I started Launch Darkley in 2014 with my co-founder, uh who'd been at Elassi and I'd been at Tripit, and we both had this strong conviction that we did not want to have um an on-prem version. And it was because I had been the product manager and program manager for supporter platform matrix at vignette, and I was like, this is hell.

SPEAKER_03Yeah, you had the scars.

SPEAKER_01You're sorting out databases, you're sorting out app servers. I was like, don't want to do that again. We're SAS. And my co-founder was the same. And at the time it was very contrarian. Like I remember we were trying to sell to customers, and A, we were trying to sell them on fast releases, which some of them just didn't want at all. And then some of them were like, well, uh, you know, we would we would never use a vendor for this that we cannot install. Um, so but we held firm to that conviction that though we could have gotten more sales, it would have been a quagmire of pain.

SPEAKER_02Yeah.

SPEAKER_01Uh to have an installed version.

SPEAKER_03Yeah. Yeah. We uh I wish we had that much conviction at Snake. We kind of went back and forth, back and forth on it. And it has uh but uh, you know, we we uh uh you know what I'm not even sure what the status is, but I'm pretty sure uh today there's on-prem components, but there isn't an on-prem system. Let me kind of shift us a little bit to uh add a degree of freedom, if you will, on it, uh, which is runtime agents. Uh you know, like you uh up until now we're talking about software that runs in runtime. Yes, it is modifying faster.

SPEAKER_03Uh and you know, by using feature flags, you can try out capabilities within there and you know uh allow it to change and learn even faster. Um is there such a thing as feature flags for agents? Like when you're what as a as a portion of your product becomes instead of software, you deliver a bunch of that value in the form of agents. Like what is the equivalent of feature flags uh in that world?

SPEAKER_01So I feel like you're uh feeding me marketing lines. So um we do have a product called agent control. Um and it is where you can put guardrails around an agent and put in parameters so that if your models drift, uh it can either shut off the agent completely or just dial it down or so that you can have the ability to control an agent in production without a redeploy.

SPEAKER_03And and this will be like how how is that different to just sort of the core management of the agent or like how how people run their agent in like with without LodgeDarkly, right?

SPEAKER_01Uh so we give you a layer where you can easily switch models or even switch the prompts without having to do a redeploy. So you can do it at runtime.

SPEAKER_03And it's again like one of these uh you know, folks in London get one sort of uh prompt versus you know people elsewhere getting another and sort of seeing what happens.

SPEAKER_01Uh correct, or um monitoring for errors. Um so if you have, for example, a chatbot that suddenly is getting a lot of customer complaints, um, you can dial it back or switch Swiss models, or uh if it's suddenly burning a lot of tokens, uh you can switch between models on the fly.

SPEAKER_03Yeah. Who do you think is the primary user for the sort of the runtime control of the agent? And is that is that different? Is it still about sort of like managing product functionality that's available to people, or is it about actually developing the sort of the prompts and the choices of models? Like almost the more kind of implementation, it's the same goal, but it's a different um uh it's a it's a change of the instrumentation that makes it happen.

SPEAKER_01I think people are still figuring out what agents are. Um so I think within LaunchDarkly, we have both very ephemeral agents and long-running agents. Um I'll use an ephemeral agent as an example of you know, we're working on a project to redevelop part of the code base. We spin up uh you know 20, 30 agents to work on it, and they are specific to this specific task. Um we have other, I'd say, kind of long-running agents.

SPEAKER_01Uh, like, so for example, we have an agent called Vega, which is constantly doing flag cleanup on everything that we produce. We, like everybody else, uh didn't always practice what we preached, and we had some stale flags, you know, some dating back, you know, to when we were still a startup at Heavy Bit. Um, in this agentic world, we now have agents that just go around, examine every flag and say, is this still in use, is it still needed, and if not, uh safely delete it.

SPEAKER_01And and that's just really fun. I I um I I think that's a real best practice that AI has made a lot easier. So I think these are actually pretty different use cases, and I would say I think right now I think the industry as a whole is figuring out what the right word what what the right use cases are. Um to use subconsumer use cases, uh uh it's funny because I come from a travel background because I was a trip it. Right now there's a lot of chatter about like uh an agent that will book your next vacation.

SPEAKER_01Okay, that's a one-time use case. Like if I'm trying to figure out how to go to Atlanta to see my aunt, that's a specifically defined project versus a long-running agent of uh, you know, look at New York Times cooking recipes every week and buy the groceries so I can make the most top three popular ones.

SPEAKER_03Yeah, yeah. Interesting. Okay. So even as you think, so I guess in that context, when you do that type of runtime controls, you come back to you need a person, like they need to figure out what is the question they're asking when they're making a modification, uh, because the sort of the mental model of what is it trying to do, what is the context in which it runs, uh, once again. Again, what is what is success, not even for the product as a whole, but for this experiment uh that you're doing here, all of those things need to be pretty clear in your in your mind and then subsequently in the instructions.

SPEAKER_01Yeah, and to go back to the the cooking agent, because I think it's a good example. If the agent went rogue and said instead of the top three recipes, I'm gonna choose the top ten recipe because that's better, it's like, well, that's actually way too much food for me.

SPEAKER_03Yeah. Um so uh I think we're sort of running out of time here. I've got like five more questions on it, but I'll I'll ask one. Or maybe I'll ask one of those questions that is five in one, and we'll sort of find out. Um so you you've built, you know, you you you found it non-starkly even before I founded Sneak, you know, like a year before on it, you know, you've been you've just sort of built in the cloud era. Uh you you had some time away from the CEO seat, you know, for a couple of years, you came back to uh to kind of I don't want to say reinvent, but like uh adapt the company to the uh the AI uh era.

SPEAKER_03Um what would you say is different about running a company in you know a decade ago, you know, and like what is needed to make things successful versus today?

SPEAKER_01Oh gosh, I mean I could talk about that for hours. Um the company ten years ago, I mean, we were a tiny startup. You know, we were you know eight people at HeavyBit, the uh DevTool incubator, and now we're 600 people worldwide. Um and we have thousands of happy customers that I'm proud of. I'd say the big change right now is that a lot of established software principles are up for grabs, but yet still the same.

SPEAKER_01So, like some of the things we've talked about in terms of you know moving the bottleneck within the software development lifecycle is it's very different. Um, but I think the core belief in that our job as a company is to deliver value to our customers and to empower our employees hasn't changed at all.

unknownYeah.

SPEAKER_03Do you think is is there something different you look for in uh in a team in a higher, maybe don't sort of look quite at the sort of the 14 person, but try to sort of imagine a 600-person company a decade ago or like maybe even somewhere along the way versus now? Like are there different aptitudes that you look for in a developer on the team uh now versus uh before?

SPEAKER_01So uh we we just uh republished our values, and a core one that's been constant is the ability to learn and grow. Um so I really encourage the team to always think about how they could be doing things different and better. And I think AI has just made that even more fun. Uh, you know, so we were having a discussion about, you know, should we encourage people to take smart bets? And I said I would like to encourage them to take some dumb bets too.

SPEAKER_01You know, if you're only taking smart bets, you're not taking enough bets. Um if you're taking stuff where you're like, hey, I'm not really sure if this is gonna work out, like that that's the edge where I want people to be, and not in a not in a silly way of like, hey, I know that if I push this button, it's gonna take down production. Please don't do that. Yeah. Um but like hey, this feature with AI is a lot easier to build. And so we can cut off some of the debate about whether or not it's worth building because it's actually faster to just build it than have that debate.

SPEAKER_01Um that that's super fun.

SPEAKER_03Yeah, yeah, I love that. And I agree with that. Like uh adaptability is the name of the game, and uh, I guess a part of it is even how both what you build and how you build uh is all but guaranteed to be different in a year's time, uh, which is not the reality we had before. You know, like uh we had uh we had things that would change over a decade, not over a quarter. Uh but uh you know.

SPEAKER_01Yeah, and I think even about how we're talking about AI in the spring, uh now it's October, and we're we're using it differently. Um the frontier models have come out with different things, we've changed how we approach it, and we know that this will continue to change. And some people find this terrifying. I find it really exhilarating. I'm like, this is cool.

SPEAKER_03Yeah, yeah, you're sort of uh uh outcome and kind of creativity oriented, and it's interesting to think about the people that are more craft-oriented, they feel like someone's taking away their craft, uh, and those who are more like problem-solving or like outcome-oriented, uh, I think get more satisfaction out of it.

SPEAKER_01To go back to the factory model, I I think of it as moving from perhaps a home workshop where you have some limited tools to full makerspace where you have access to all of a sudden all these things that can help you build.

SPEAKER_03Yeah. Yeah, lots of excitement in that. Edith, this has been a great conversation. Thanks for for the time here. Excited to see how you uh continue to sort of build and uh and adapt, and uh, you know, so much more to build here. So thanks for tuning in.

SPEAKER_01Yeah, thank you. It's great to spend some time with you.

SPEAKER_03Cool. And thanks everyone for listening in, and I hope you join us for the next one.

SPEAKER_00The AI Native Dev is brought to you by Tesla, the package manager for skills and context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tesla offices in central London. Visit Tesla.io forward slash community to learn more. I hope to see you there.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Your Software Factory Needs a Bill of Materials | Daniel Meppiel

    Nov 19, 20261 hr 1 minAudio

    Developers are about to write skills instead of code, and Microsoft's Daniel Meppiel argues the software factory that runs them needs versioning, a bill of materials and signed receipts before enterprises will trust it....

    Credit the people on this episode

  2. Paul Stack: I'll Never Write Infrastructure as Code Again

    Nov 17, 20261 hr 2 minAudio

    Paul Stack helped build Terraform. Now he says he'll never write another line of infrastructure as code, because an agent harness runs his team's production infrastructure instead. Watch all the AI DevCon NYC 2026...

    Credit the people on this episode

  3. Maggie Appleton: Agentic Coding Has Outgrown the Text Box

    Nov 3, 202652 minAudio

    Agentic coding has outgrown the text box. Maggie Appleton, staff research engineer at GitHub Next, argues that walls of CLI output and AGENTS.md files serve coding agents far better than the humans steering them. She...

    Credit the people on this episode

  4. Caleb Sima: AI Attackers Move at Machine Speed

    Oct 27, 202655 minAudio

    AI attackers now move at machine speed, while most security teams still move at meeting speed. Former Robinhood CSO Caleb Sima has seen an AI attacker up close, and he argues the only way to keep up is to take humans...

    Credit the people on this episode

  5. Your Coding Agent Should Never Hold a Credential

    Oct 22, 202630 minAudio

    Your coding agent should never hold a credential, and in a software factory that rule is only the start. Recorded live at the AI Security Summit in San Francisco, Simon Maple sits down with Eli Aleyner (Docker), Marcelo...

    Credit the people on this episode