Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow

The AI Native Dev - from Copilot today to AI Native Software Development tomorrow artwork

AWS's Marc Brooker: Specs, Not Code, Are the Hard Part

Sep 16, 202657 minAudio

Spec-driven development is reshaping what software engineers actually do all day, and Marc Brooker, VP and Distinguished Engineer at AWS, has read 3,000 to 4,000 postmortems that convinced him the code was never the hard part. In this episode of The AI Native Dev, Marc explains why testing and specification are now the real engineering work, why metastable failures keep taking down systems that look perfectly healthy, and why he still won't let AI write a single word of his blog. What we cover: - Why spec-driven development is turning testing into the hardest part of software - What metastable failures are, and why they keep taking healthy-looking systems down - How agentic coding tools can learn from postmortems and build their own memory - Why classic authorization breaks down once you're writing agentic policy - Why Marc won't let AI write his blog, but is fully comfortable with AI-generated code - What it takes to bring junior engineers up to speed in an AI-native industry Chapters: 00:00:00 - Introduction 00:02:17 - Marc Brooker: 18 years at AWS building agentic dev tools 00:04:36 - Inside Strands, AWS's open source agent SDK 00:06:44 - Fifteen years on call, and what agents still can't debug 00:11:23 - Metastable failures and the humility of 4,000 postmortems 00:14:32 - Teaching agents to learn from postmortems and build their own memory 00:20:14 - Why agentic policy needs more than classic authorization 00:32:14 - The agentic software development hypothesis: spec-driven development, oracles, and testing 00:39:56 - Why Marc won't let AI write his blog (but will let it write code) 00:46:37 - Advice for leveling up junior engineers in the AI era Build your software factory, one workflow at a time, with Tessl: https://tessl.co/[SLUG] 🔔 Subscribe for weekly episodes on AI-native development Do you trust an agent to run its own on-call rotation yet? Tell us in the comments.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_00I want to talk about 15 years, I think, that you spent on call. That's a lot of 3AMs. Do you feel like agents will very much be able to uh take that role from us?

SPEAKER_01With the basic investigations of on call, the the log scraping, the diving into metrics and so on, you know, if you look at something like AWSS DevOps agent, it's great at doing that kind of automation of, let's say, the more day-to-day aspects of on-call. And then what's left is the really interesting reasoning about why is the system behaving this way? You know, what are we seeing in terms of large-scale failures? What are we seeing in terms of improving performance at big scale and so on?

SPEAKER_01And uh, you know, automation is going to help us there a whole lot, but it doesn't have answers to those questions today.

SPEAKER_00The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI energetic coding. Join your host, Guy Fajani, and meet Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. Yesterday, you were writing code. Today, you're prompting agents. Tomorrow, you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash.

SPEAKER_00This June, AI DevCon London sold out. Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub cut through the hype and get to what actually works. This November, we're bringing it back to New York. Bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.

SPEAKER_00Our early bird pricing ends on September 30th, so make sure you lock in your place before prices increase. As a podcast listener, you can get 15% off your ticket using the promo code POD50. We have so much planned, and we can't wait to have you join us in New York. Hello and welcome to another episode of the AI Native Dev. And joining me today is Mark Brooker. And Mark is a VP and distinguished engineer at Amazon Web Services.

SPEAKER_00Mark, welcome to the podcast. How are you? Well, thank you, Simon. I'm great. It's a pleasure. It's a pleasure to chat. And you're, of course, are you're on the West Coast, right? Are you in San Francisco? I'm in Seattle. In Seattle, a little bit further. Yeah. And and so it must be raining. I assume it's raining there, just like it is in the UK right now.

SPEAKER_01It is threatening terrain. Oh, it's threatening.

SPEAKER_00It's either raining or a threatening terrain, as they say.

SPEAKER_01We've got our two weeks of glorious summer and uh and we're back to Seattle weather.

SPEAKER_00Amazing, amazing. I was I was on a call with someone from Kenya today, and weirdly, it was warmer in London today than it was in Kenya, which was I I doubt there are too many days where you could probably say that, but it's it's nice to boast every once in a while. Um so, Mark, you've had an illustrious career so far. Tell us a little bit about you know what you've been up to and some of the work that you do at AWS.

SPEAKER_01Yeah, so I've been at AWS for a long time, for 18 years now. Um worked on a whole bunch of things, but the last few years has been really focused on agentic AI and Agentic Development Tools and the infrastructure for for agentic AI. And so was deeply involved in the development of AWS Agent Core, this kind of suite of tools for building agents in the cloud. Um but most recently um I've been focused on building a um a team really looking at some of the future of agent building, agentic development.

SPEAKER_01Uh, and um uh so we own uh Strands, which is AWS's agent framework, open source agent framework. We've got some really cool technology coming in the Strands space, uh, and then are building some new development tools, thinking a lot about the future of development uh and doing uh everything ranging from fairly basic research to a lot of code building, a lot of product building. So it's a really fun time to uh you're really kind of be focusing on pushing the frontier of of what it means to build software.

SPEAKER_00Amazing. And actually, we haven't really talked about Strands too much on the podcast. Tell us a little bit about you know what Strands is and who should be interested.

SPEAKER_01Yeah, so Strands is an open source SDK for building agents. Um works with any any model, any cloud provider, local agents, etc. Uh and the goal of Strands is to make it as easy as possible to build great production ready agents. Um so the kind of minimal Strands agent is just a handful of lines of code, but you can build out from there with um tool integrations, uh, you know, all built-in MCP clients, built-in open API clients, um context management, uh all of the things that go into um you know into building a great agent.

SPEAKER_01Um and the strands team is really trying to make that as easy as possible by being, let's say, very opinionated about good agent patterns. And uh and so you can extend strands, you can use it any way you want, but there's a really nice kind of paved path through strands uh that uh has these opinions of of you know what we think right now it means to build a great agent built right in.

SPEAKER_00It's funny because I always think you know developers who have never built an agent before probably think there's actually so much complexity behind the scenes. Yeah. But they can actually be very thin veneers uh around an LLM. And and um I think it's you know I very much encourage people and listeners to actually build that. And it sounds like it sounds like this could be a uh you know a very helpful way of actually doing that from from scratch.

SPEAKER_01Yeah, for sure. And you know, I I really just would encourage people to you know build build your own development agent. I mean, that's not going to be as good as as one of the you know major commercial offerings, but very quickly, even in a couple of hours, you'll get this much better intuition for what's going on under the covers of those agents, how they work, and I think that kind of experience, and maybe we'll touch on this a lot more as we talk about you know how software development is changing, but that experience of knowing on what's about what's going on under the covers, um, I think just makes people better users of uh of even the these great commercial products.

SPEAKER_00Yeah, yeah. Amazing. Um, Mark, I want to talk, uh, I want to go into a little bit of depth now, and I want to talk about something that was very interesting. I think you mentioned in a book as well. Um, and it goes back to the time, 15 years, I think, that you spent on call. That's a lot of 3AMs, first of all, right? That's uh that's a lot to put anyone through. Uh must have been painful. You enjoy on call, you enjoyed the the the those kind of roles?

SPEAKER_01I do, and I I just find them an incredible opportunity for learning how the real world works, like learning how our systems actually run, you know, learning what our customers deeply care about. And you know, this is one of the challenges you come out of university, and you know, I came out of grad school with a ton of what I thought was practical knowledge, but a lot of it was quite theoretical. Um, and then on-call was really this huge catalyst over my career for more deeply understanding how systems actually work, how systems actually fail, how we can stop them from failing, you know, how those things affect our customers and how we can help our customers out.

SPEAKER_00Um it's it's funny, Mark, because I always think a lot of people think they become a senior developer when they've done five or more years in development. Um whereas actually I think it's more these types of roles where you get that real exposure into what it takes to build a real app that's you know reliable and etc. etc. But i i is this still you know, talking about the future of development with AI, do you feel like this is um gonna continue this this type of uh encore is gonna continue, or do you feel like agents will very much be able to uh take that role from us?

SPEAKER_01You know, I think um you know, and this is gonna be a this is just a major theme of of software is we're gonna see a huge amount of automation of increasing automation of the the rote things, right? The oh, you know, there's been you know the this kind of failure. And that's not new, right? Like if you look at the whole history of cloud and and computing, this has been a theme, right? Like we used to get paged for single host failures, but now you know AWS autoscaling 15 years ago has just started dealing with those things for us.

SPEAKER_01And and load balancer health checks started dealing with those things. And so we automated those away. And we're seeing exactly the same thing with with AI, with the basic um the basic investigations of on-call, the the log scraping, the diving into metrics and so on. Um, you know, if you look at something like AWS's DevOps agent, it's great at doing that kind of automation of um, let's say the more day-to-day aspects of on-call. And then what's left is the really interesting reasoning about why is the system behaving this way?

SPEAKER_01Um, you know, what are we seeing in terms of um large-scale failures? What are we seeing in terms of improving performance at at big scale and so on? And uh, you know, automation is gonna help us there a whole lot, but it doesn't have answers to those questions today.

SPEAKER_00Yeah. And and I'd like to think that when I do wake up at 3 a.m. and other early morning times are available, but let's stick with 3 a.m. that an agent has already been through the logs and been through all the information, you know, the gruelling part of the the debugging of, or not the debugging per se, but trying to identify what a root cause could be. And has already provided that as part of you know the on call where I I I I'm woken up and presented with all the information that I need to make a decision.

SPEAKER_00Um I'd love to know as part of the as part of the you know your writing in the past, you've said um around that you've done around 3,000 or you've read 3,000 to 4,000 postmortems. Um I guess what did that teach you um that you wouldn't have really been able to learn without being on call? So as a typical developer trying to build, what what did the postmortems really open up for you?

SPEAKER_01Well, one of them is a deep sense of humility. Uh you know, as as engineers, we uh we tend to be optimistic, which is great. Uh, you know, we tend to think, hey, we can build this, we can solve this problem, which is fantastic. Um, and then you know, you see how things fail and you see when things don't work, and and you get a a dose of humility from that, and a dose of um uh uh sort of closing that loop of okay, well, I'm gonna be optimistic. I'm still gonna believe I can do this, but I'm gonna do it in a better, smarter way this time. Yeah. Um, one very concrete thing, for example, is uh, you know, over the last 10 years or so, there's been this increasing um familiarity with a concept uh that is now called um metastable failures.

SPEAKER_01There's these failures that large systems have where something will happen, whether it's an overload or a component failure, the system will go down, the immediate cause will be repaired, and the system won't recover because it gets stuck in this sort of stable down state. Um and you know, deeply understanding that has been super important to understanding how to avoid whole classes of long and painful outages.

SPEAKER_01And I think a lot of our understanding of that, and certainly personally a lot of my understanding of that, has been come come from looking at real-world incidents, looking at when that happens, and then analyzing the dynamics of the system during those times. Um, and you know, we could have approached that entirely analytically, we could have approached that from the outside, but I think we've learned a huge amount more by looking at how real systems fail, both you know, our own and systems across the industry, and really trying to extract like, what was it that kicked the system into this state?

SPEAKER_01What was it that kept it down? What was it that led to recovery? How can we make systems more resilient against those things? How can we make that recovery faster? And to your earlier question, how can we build automation that can make that recovery automatic when it's when we're in one of those states?

SPEAKER_00Yeah, yeah, super interesting. And I think that learning and feedback is super key. And I think for us as humans, you know, pre-AI being mainstream, it's what we would have learned and what really would have leveled us up to that senior developer. And and I think your point on humility is so valuable. I think you go in with a different mindset when you build code and you think, right, I know this is gonna break, I just don't know how or when. Um and you look at it quite differently. So I I love that answer, I think that's brilliant.

SPEAKER_00Um I'd love to understand we we as humans can learn um from from not just our mistakes but our experiences, uh, our uh uh our view of other uh post-mortems and things. How about agents? Um when we think about uh loop engineering, for example, and we think about how we can take these loops that will automatically um understand how agent c agents could have done better, potentially updating context, updating skills, so that when we write uh and build applications that are similar, we can potentially circumvent uh you know g work around certain problems or at least guard against them.

SPEAKER_00Is that realistic or is that kind of like a pipe dream in terms of the type of bugs that would be raised uh uh to a to a um a postmortem?

SPEAKER_01Oh, that's very realistic, and and and very much that is you know what what is happening today at the at the frontier of these agents. With a a variety of techniques. And so, you know, one of the techniques that you have is to say, hey, I've got all of these postmortems. Um I'm gonna put them in the training set of the model, and so you can improve the model. Another way is to say, I've got all of these postmortems, and I'm going to give my agent a tool to search over those postmortems semantically and say, show me incidents like this one, and then I can understand how I recovered from that.

SPEAKER_01And then at that point, that agent can write its own memories and say, I tried X, it didn't work. I tried Y and it worked. And so next time it will retrieve from those post-mortems, it'll retrieve from its memories and jump ahead in that phase. Increasingly, agents are also seeing, hey, I did this step and this step and this step. I could encode all of those into a piece of code and build myself a tool and put that tool in my memory, this kind of procedural memory, um, and and remember that and just pull that off the shelf when I need it.

SPEAKER_01Um so there is building those libraries of context for agents, uh, there is building great, you know, allowing agents to build and and manage their own memories. Um, there's model customization, uh, and then there's both manual and automatic tool building for agents and a whole variety of other techniques. And each of these things can help us make agents um learn from experience and get better and better at solving uh certain types of problems, both more reliably but also more quickly and more cheaply.

SPEAKER_00And and do you think that loop of learning and understanding, uh particularly around you know, incidents that occur, do you think that's m most valuable during an incident where it can actually learn and understand uh where possible root causes are, or or you know, it's likely that it's going to be something like this because it's the last four were similar or happy I I I saw a similar footprint before. Or do you think that learning can be taken down into the architecture decisions and the development decisions for future building?

SPEAKER_01Yeah, well, I you know, I I I I hope so, and that's that's that's a bet that we're making. I think we haven't seen as much of that yet. Yeah. Um and maybe, you know, to take that back to the human level, one of the things that we've always done at Amazon is our our product teams, the same people who are building our products, are on call for those products. And we do that primarily because it drives exactly that learning loop, right? You uh you get that production experience, you get those customer anecdotes, and you bring them right back into development.

SPEAKER_01Yeah. Whereas separating those things out into separate teams, although it does have its advantages, makes that learning loop much harder to drive and much more abstract. Yeah. And so when we get into the agent world, what we want is that our coding agents, our development agents, our software building agents, are learning from production experience in exactly the same way that uh operations automation agents are. And there's a path to that with memory, with sharing context, with you know, sharing those memories, sharing tools, etc.

SPEAKER_01Even using that development agent as a kind of expert consultant for reviewing a software design. But I would say as an industry, we are earlier on that journey than we are on the optimizing the individual pieces.

SPEAKER_00Nice. I remember when prompt engineering was a thing where you used to be able to write something to say, like, oh, by the way, another agent is going to review this, so make sure you do it well. Um and I love the idea of just thinking, oh, by the way, you're gonna be on call for the code you write, and you you might be woken up at 3 a.m., Claude, or Gemini, or whoever. So so that that could be that could be kind of fun. Um I'd love to know in your experience, what can an agent not what's it not possible for from for an agent to learn from a post-mortem?

SPEAKER_00And I guess this kind of like leans more into where we as humans need to be very uh understanding of where we play a role. So from the postmortem, what's what do you expect an agent just to breeze over?

SPEAKER_01Well, I think the things that agents can solve are the ones that are um have clear definitive outcomes, right? I have a graph of uh orders on Amazon.com, or I have a graph of EC2 instance launches, and it should look like this, it does look like this, figure out why there's that gap and and drive that closed, right? There's a very clear goal. Um the ones that are harder to solve are the ones that are, well, um, you know, some I'm I'm hearing on Twitter that customers are seeing a few more errors than usual, or people are you know just seem less happy with the product, or you know, things feel generally slow when I open it.

SPEAKER_01And so the more ambiguous things. And so, you know, agents can help you dig into those and can help you find um uh find the uh the the metrics that might be driving those perceptions from customers or driving those anecdotes, um, but generally aren't as good at really understanding the end-to-end context of the work. How are customers using the system? What does it mean for it to feel fast? What does it mean for it to feel reliable?

SPEAKER_01Um and you know, as we know, uh encoding those kinds of things in metrics is is often very difficult.

SPEAKER_00Yeah, very, very true. I'd I'd love to jump across now to um to a move that you made actually. After a couple of decades in storage, compute, and database, you moved across to agentic AI, safety and policy. Which is actually very topical given the time which we're which we're recording this episode. Of course, uh uh Dario just mentioned about how we how uh you know Anthropic should slow down its it its development of models and is kind of talking about others to do the same.

SPEAKER_00I know Sam Altman uh agreed, so did Elon Musk. First of all, I'd love to hear your opinion on that. Um and then secondly, I'd love to understand why the move. I'd love to I'd love to go deeper into that uh into that.

SPEAKER_01Yeah, you know, I don't have a opinion to share on model development and the speed of that uh you know, very, very interesting set of trade-offs in that space. Um what I will say is a lot of the you know, my focus here is on what I'm hearing from from customers uh and what I'm hearing from from developers. And one of the very common themes that I hear from customers is um, hey, we built this awesome agent.

SPEAKER_01We we use strands, we we we built this awesome demo, we put it on agent core, it looks great. Uh the board is super excited about the demo. But now we're really worried about how we get it into production, how we make it reliable, how we you know make our it's so our customers can trust that agent to do work on their behalf. And a large part of uh solving that problem is this idea of agentic policy, right? What does it mean for an agent to do the right thing?

SPEAKER_01Um how do We very clearly define what an agent can and can't do? You know, what of that belongs to the environment and the harness and uh and the containment of the agent versus what of that belongs in the prompt, what uh tool descriptions, etc. And what our customers are asking for is ways to think very clearly, very deterministically about how to build a box around agents and say, I'm gonna trust this agent to do this thing, this thing, and this thing, but it is not going to have any access to do you know these things.

SPEAKER_01And a lot of people approach that as a classic authorization problem, as you would with humans, say, hey, I, you know, this human has this authorization to do to take this step. Um, but humans have a level of judgment, they have a level of feedback that you know agents lack, and classic authorization languages lack a lot of the power that customers are looking for for describing what their agents need. And so, for example, one of the things we did is we we recently launched Dogwood, which is this new policy language, happy to go into depth on that, but that is built as a specialized language for describing with mathematical certainty what an agent is allowed to do, what order it's allowed to do it in, whether it can do thing A after thing B, and so on.

SPEAKER_01And so it really all comes out from these customer needs and the fact that you know across the industry, people are saying, hey, agents are super exciting. We're really excited about what we can do with agents. We want to automate more, we want to serve our customers better, we want to spend our humans' time doing uh more important tasks. And a key to doing that is having a great security story and a great policy story, uh a great isolation story, a great identity story. Uh, and so we can build agents that we can trust to run autonomously and do critical things within our companies.

SPEAKER_00And I was gonna mention autonomous actually, because that that I think that's the key. Uh in terms of being able to increase our trust and allow agents to run things more autonomously, we need them to essentially be able to make more decisions, to run longer processes, and they need that understanding uh or rather, they need that level of um those boundaries, those set deterministic boundaries without having to make their own decisions or do anything in a non-deterministic way. Uh they need those boundaries there whereby they can say, yes, you can you have access to this or you can do this, or no, this is you know beyond what we want to provide you with in this container or this space.

SPEAKER_00Um I think that's you know so important, and I think the last month or so has actually shown how important that is in terms of um in terms of that belief and trust. Um and and as we go forward with autonomy and interfactories and long-running processes, how do you see people using this kind of a tool to to run their containers? Do you see like configurations that people will essentially be assigning on various tasks? Um how would someone set this up?

SPEAKER_01Yeah. Well, one of the things that we know from you know decades of research into psychology and operations and operators is that humans are really bad supervisors. Uh you know, just really bad at watching things go by and pushing the red button when something bad happens. Um and you know, there are multiple dynamics behind that. You know, one of them is this classic research from the 80s, there's this great paper called Um The Ironies of Automation about how as an you know a system becomes better and better and better, uh the failure modes become weirder and less frequent and becomes much harder for the human supervisor to understand when they happen, recognize them and react correctly to them.

SPEAKER_01And so I think the only solution to this is that autonomy, right? Pull the human out of the loop, make them not watch the loop go round and push the button when something bad happens. Certainly not make them say yes, yes, yes, after every code generation task, because you know who's gonna do that? Um, but instead pop the human out and say, let's start by describing what good looks like. Let's start by describing the set of things that you would like the system to do. Um, and then once we have that description, we will apply it to the agent without requiring a human to go around and do those things.

SPEAKER_01And so how do you do that? Well, some of that is encoding of it's encoding of common sense, right? You know as you develop a piece of code, um, you know, what kinds of interactions it should have with the system, right? It's gonna do, you know, get pull and get push, but only in the the um the repository it's working in, right? If it goes and tries to do a get push to a different project, well, that's probably bad. And so we're going to encode that little piece of common sense into a policy.

SPEAKER_01Um and part of that is writing that kind of library of those common sense policies. Part of that is this process of natural language to policy where you can say, um, you know, this can read anything on my hard drive except the stuff in my secret documents folder, and and have that automatically and reliably turned into a policy. And then part of that is going out over a company and discovering all of the policies and practices and so on and encoding those existing practices into policy.

SPEAKER_01Um so to answer your direct question, I think the minority case is people writing policy directly. I think there are going to be some people writing dogwood or CEDA or OPA or whatever policy language they're using. But the vast majority aren't going to engage at that level and don't want to because those languages are quite specialized and fairly difficult to learn, um, but instead are going to be engaging at that natural language level of saying, you know, here's what here's what good looks like.

SPEAKER_01There could be me as a developer saying, here's what good looks like for me as a person, or it could be, you know, somebody in the CISO's office saying, you know, here's what good looks like for us as an organization, here's our standard and policy for handling customer content, and our standard and policy for handling our code, and then encoding those. And often all of those things will come together. And so I'll have some things from my CISO's office, I'll probably have some things from Legal, I'll have some things from myself as a developer, and all of those go together and provide this shell around my agent where we can strongly and deterministically make sure that the agent is following all of those policies by using one of these lower level languages.

SPEAKER_00Yeah. I I was gonna ask actually whether whether you see this as something uh in your experience that is an organizational level or a department level, uh from legal, from security. Is it is it majority from those from those teams down to developers, or is it a is it you know closer to a 50-50?

SPEAKER_01Um I think it's closer to a 50-50. Like there is some things that my security teams deeply care about, right? The way that customer content is is handled and where it is allowed to be in a system, for example. And I want every developer in the company's agents to be held to maintaining that standard. And then there's some things that my security teams don't care about, right? Like, can my development agent delete documents from my documents folder? Well, my security teams don't really care about that because there's no, you know, there's no customer impact of that, there's no organizational impact of that.

SPEAKER_01But as an end user, I care about that super deeply. I don't want that stuff deleted. And so I'm going to, it's going to be a mix of these things. Um and, you know, especially as we get down to individual developers, we know that across the industry, and and and certainly, you know, even within teams and within organizations, development practices vary. There's a wide range of what normal looks like. There's a wide range of tools that people use. And so just like you know, most big software organizations don't try and push down a, you will use exactly these tools at this time type software engineering practice, they're not going to succeed at pushing down a complete end-to-end, this is your agent policy, because that needs to be adapted to the kinds of development you're doing.

SPEAKER_01And even day-to-day, right? Like sometimes I'm writing back-end Rust code and I want to set a certain kind of standard, and I want to write things like, hey, you aren't allowed to commit before Clippy is happy with this code. And sometimes I'm writing, you know, a quick and dirty prototype of a, you know, of a of a web demo. And I don't want to set any of those standards. And I'm just like, okay, we'll just hack this thing together with whatever tools make sense, because we're going to show it off for 15 minutes and then discard it. And so I'm going to have different policies.

SPEAKER_01And so those policies do have to come from different places and be dynamic and be contextual.

SPEAKER_00Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

SPEAKER_00Alright, back to the episode. I want to go to another piece that you wrote about, which is the agentic software development hypothesis. And you also mention um uh uh a a number of different kind of like forms of this, which is the weak form, strong form, and strongest form. Um I'd love you to walk us through the agentic software development hypothesis first, then we can kind of cover the rest.

SPEAKER_01Yeah, you might have to remind me exactly what I wrote in that post. Um but it essentially it comes down to this idea that the hard part of software building is um I'm gonna say specification building. And and and what that means is describing what you want. Um and it describing what you want in sufficient detail that it is, let's say, a complete description of what you want.

SPEAKER_01Once you have that specification, once you have that what you want written down, my belief is that we're going to see the building and maintenance of reliable, efficient implementations of that software become over time entirely automated. Um we're gonna see that first, and I think we've already seen that with, let's say, these closed specifications. Hey, I want something that implements this type of file format, and it's very well described.

SPEAKER_01Um, and and and we've already seen that. And we're gonna see that grow to larger and larger systems, systems with more UI, systems with more complex interactions. But I think over time, all of that kind of specification down of software development is going to become automated and driven by an automation loop. Um and you know, the the the kind of stronger claim is that the vast majority of software can be specified crisply enough to enable that kind of automation?

SPEAKER_00Yeah, absolutely. Now, in that same post, you mentioned that very few tasks have complete specifications, and you also talked about oracles that can be deterministic. Um, and you mentioned oracles aren't deterministic. So I wonder, are we relocating the the the tough piece of the problem uh to be in identifying an oracle that's deterministic or being able to create that complete specification?

SPEAKER_00Is that still more of a human issue?

SPEAKER_01Yeah, for sure. I think that is where the you know where where a huge amount of the human the human work comes in is is creating those oracles, creating those specifications. Um and you know, in some sense uh testing uh is is now the most important part of software development. Uh and uh you know some people will say that it always has been, but it's never been the the shiny object for for most developers. Um but now the thinking about how do I build a test that says you know whether this piece of software is good or not is is just really at the frontier of software development.

SPEAKER_01Um and some of that is going to be specification. Um, can I write in a formal enough way to describe that uh in a deterministic way? And again, like if I'm implementing GZIP or something, yeah, I can. I can fully specify those. Um, if I'm implementing a UI for a customer, maybe I can't. And so I have to use other forms of correctness oracles, other forms of evaluation that might be more open-ended.

SPEAKER_01They might be customer in the loop, et cetera, where you know we we get that feedback and and feed it back in. Um and you know, this uh there, if you look at um, you know, the kind of automated reasoning level of specification, there's been this incredible progress in specifying software in languages like lean and a huge growth in just addressing bigger and bigger chunks of software with that.

SPEAKER_01But as we expand those components to entire systems, and those systems have complex interactions and so on, that becomes more difficult. And so I also want to build um build these oracles that can look at a system and say, is this working well? And and that might be looking at those production metrics, it might be looking at customer feedback, it might be um uh comparing a doing a fuzz test between two implementations of a database engine, for example.

SPEAKER_01Um and a wide variety of other techniques.

SPEAKER_00Yeah, amazing. And it's it's it's so cool to see. Uh I remember back in the early days, I think we had um some folks from uh Amazon for we when we talked about Kiro when that was first launched back in the early days. And um yeah, specs are specs are such an important piece, and I think the uh the you know it it it's definitely got a place in agentic development for me in terms of being able to truly describe a function, a feature, an app, and be able to have that as the source of truth going forward.

SPEAKER_00I think I think there's a super compelling piece there. What one thing I always kind of struggled with was delineating almost what is part of the specification and what is part of the test, because very often you can write something in a specification that is essentially describing a test. And you know, similarly, when you when you're talking about a test, writing or describing a test, you're actually talking about how something should function, which is kind of part of the specification. And for me, almost like the the the functional specification as part of as well as the uh as well as the tests almost merged into one, and it's like, well, it could be both, it could be the same thing.

SPEAKER_01Um but but yeah, super super interesting to see how that um Yeah, you know, and testing is a, I would say, informal specification technique. Um and uh and you know, always always has been. Um I love the things the Kiro team has been doing with with Spec driven development. There's been some incredible advancement there. I think looking back on that, um the thing I feel a little bit about unfulfilling about the current uh let's say current moment in in spec driven development is there's still quite a lot of conflation between what are those requirements, those kind of declarative requirements, and the more implementation detail-y stuff, right?

SPEAKER_01Like there's still, oh well, you know, I want to build this in this language or for this kind of UI, or um and ultimately, um I don't want to care about those things as a system builder. I I I want you know great decisions to be made, but great decisions to be made more automatically. Uh and and the specification and the tests and so on should focus on what good looks like for the output of the software, not for the way that it works inside.

SPEAKER_00Yeah, no, super interesting. I wanna now switch a little bit to talk about I think this is uh this is something that you said a while back. I'd love to know if it's still true, that you won't let AI write a word of your blog, but you're very comfortable with it writing AI generated code for you. Um so first of all, is that is that still the case? Yep, that's right. It is still the case. What's the difference between blog content and code?

SPEAKER_01Yeah, uh it's a really great question. And so I think it's it's a difference in intent. And so I think you know, five or so years ago, I would have said with uh strong conviction that the purpose of most code is to communicate with humans. Uh and the execution of that code is important, but not its primary purpose. And so I was always, you know, very big on comments, very big on naming things carefully, very big on writing code in a way that communicates especially the intent of the code.

SPEAKER_01I think that's changed and is very quickly changing across the industry as humans get out of that code level into more of that kind of specification level, as a lot of that coding becomes fully automated, is code is turning more and more into purely an implementation vehicle rather than a communication vehicle. My blog, on the other hand, right, I'm saying this is for people to read and for people to engage with.

SPEAKER_01And if my mind there's a kind of social contract behind that, um, where I'm saying I've spent a bunch of time deeply thinking about these words. I'm gonna share these words with you, and what I'm hoping you're gonna do as a reader is spend some time deeply trying to understand my words. Whereas if I get Claude to write a blog post for me in you know five minutes, and then let's say 10,000 people read it and they each take, you know, each spend 10 minutes reading it, there's this huge asymmetry between the amount of effort that I'm putting in and the amount of effort that my readers are putting in.

SPEAKER_01And I think that is unfair. And then if I take that into work, like you know, Amazon's very doc-driven culture, I write a lot of documents for work about plans, about technology, about the things we're doing. And again, that is a contract between me and the readers, and I am expecting them to deeply read and understand and engage with my words, and they're expecting me to have deeply considered those words and have full ownership for them. Uh and you know, if they're Claude's words, I haven't done that.

SPEAKER_01And then the third part of that is, you know, if you have an idea and want to share it with me and want to share Claude's thoughts on it with me, actually I think that's great. But let's do that. Let's make that interactive. You know, send me the prompt. Let's let's have a have a chat with it. One of the cool things about these agents is that I can go around the loop and ask follow-up questions and so on. And so if that is the mode of communication we want to choose, let's choose that mode of communication.

SPEAKER_01Let's not just land in the one that's in the awkward middle, which is you know, all of the human care taken out without adding in the interactivity and and possibility for follow-up questions and so on, with which go into that agentic interaction.

SPEAKER_00Super interesting. And I think it's, yeah, for me, uh, I really I really love the way you say, you know, you you you care about the fact that it's people's time, and there's a there's a obviously one to very many uh you know people who are going to be reading that and taking the time to read that versus one person's time to actually write it well in your words. Um kind of something that strikes me here is the you've obviously got the the way you say something and then the you know the core of what you're trying to purvey.

SPEAKER_00Are both as important as the other to you there, or if you could get Claude to be able to uh write something in a reasonable way, but the the it you know if it could if it could really capture the message you're trying to share, um would would that be reasonable? I guess the thing I'm saying is what would it take for you to allow AI to be able to write on your blog?

SPEAKER_01Yeah. Um it's a good question. And at some level for my personal blog, it is a philosophical thing where I just, you know, I want to say this is my, you know, these are my thoughts as I wrote them, with my mistakes as I wrote them, and so on. I think for those, you know, kind of work documents, uh, you know, if I could get Claude to write succinctly, you know, just describe the context, pull in the business context and so on for me, I would feel quite um uh you know, feel quite comfortable with that.

SPEAKER_01And so maybe the distinction there is if you're using Claude as an expander, right, saying, here's a prompt, turn it into a six-page document, I think that's a waste of the reader's time. However, if you're using Claude as a context provider, right, like add a paragraph with the business metrics for this week, I think that's fine. Because you're you're you're bringing in new information from the human, and it is almost exactly the same form.

SPEAKER_01I would have done that. I would have done that manually. Yeah. Um But I think for my blog, you know, I I I still I still think there is a level there which I want to say it's a piece of craft.

SPEAKER_00Yeah, I was that's the word I was gonna use actually. It's it's about can you scale craft? And I think sometimes craft is best uh left as a craft, and and that's where people get the joy in both creating it and also consuming it, uh knowing it's not just you know, there's no chance of slop, it's it's come straight from the straight from the hand. Um and I think in terms of you know feedback, I've certainly had a player, and I think it's very, very hard to be able to capture that level of feedback that still you know offers uh in a genuine way the written word in the same way that a an individual would write it.

SPEAKER_00Um so I I I really like that. Um while on the topic of feedback, let's let's jump into the kind of advice or uh being you know being able to suggest what our our listeners can do. In terms of feedback signals um that agents can use, what's the what's the you know one of the key most important things that a team can do today to improve the feedback signals? So to make sure that agents get the clearest signal uh to use to improve on future runs.

SPEAKER_01Yeah. Um so I I think the answer to that varies a lot about sort of where in the agent um agentic development uh maturity hierarchy you are. Uh and so there are a lot of teams still in our industry who are just starting out, right? Who are just getting their their feet wet with agents. And there's just a huge amount of I think the things that a lot of people would sort of roll their eyes at now that are very effective, like that agents.md file and writing you know some directions for the agents about what good looks like into those things.

SPEAKER_01Um you know, definitely not a cutting-edge technique, but something that is proven to be effective. Um and I think a lot of teams early on try and jump to the end and say, oh, I'm gonna do parametric memory or something, when you know you can't ship code until Clippy is happy with it and it passes all the tests, um, is a very effective thing with very little uh very little um overhead.

SPEAKER_01Then, you know, as we go up uh those uh the the maturity spectrum, I think where things get really interesting is making sure that agents have a way to remember team context and remember past conversations. And part of that is just agentic memory, the kind of memory that is built into a lot of the agents.

SPEAKER_01They will, you know, remember things across sessions, and if you say, don't do this again, they'll remember it sometimes too well and sometimes too much, and that can be annoying. But the problem with agentic memory is that it doesn't spread well across um across teams or even across people. Uh and so you need a way to capture those memories and to write those down in a way that the whole organization or the whole team can use them. Uh and this is where I'm seeing a lot of people use these uh agent wiki type patterns, which is are quite popular right now, where essentially I have my team agents um maintain a number of just a small wiki of pages about what good looks like.

SPEAKER_01And that is shared between the agents. And then there are tools like AWS Context that allows agents to build a formal graph of um of the context in its environment, connect that back to memory, connect though to documents, and do that across an organization. Um and part of that is that dynamic discovery and and and learning, and part of that is pulling in uh the right information from my organization where I already have a whole lot of that written down.

SPEAKER_01Yeah. And again, you know, people sort of roll their eyes at RAG and be like, oh, well, that's so 2004 or 2024. Um but the bottom line is that giving uh agent semantic search over the information in my organization is still a super powerful technique. Uh and so there's just a lot of that. I'm gonna plumb that in, I'm going to give give my agent access to that. Uh and what we're seeing with the uh with the commercial agents, especially if you look at something like Kiro or you look at something like Quick from Amazon, um, more and more they're making it very easy to pull in that organizational context and pull in that team context and share those memories through things like AWS context to bake in what these good kind of learning from experience patterns look like across across an organization.

SPEAKER_00Amazing. I'd like to wrap now by another question and around advice, but advice to individuals, advice to um, you know, perhaps juniors. Um and let's let's circle back all the way back to the to the to the front of this, to the top of this podcast episode, where we talked about the three to four thousand um uh post-mortems that that you've read in the past and how that very much helped you and you learnt from that feedback.

SPEAKER_00Um your advice to juniors is own a project, talk to a customer, own a deadline, and going back to the front, also essentially learning from those mistakes that we as teams and individuals have made uh in the past. But historically, these are the these are the tasks that senior developers uh have historically done and often guarded. Um do you think it's reasonable to expect senior developers to now give these types of tasks to junior developers, or do you feel like they're gonna hang on?

SPEAKER_00And I guess this is the the age-old question about how we you know how we bridge that gap between a junior and a senior developer.

SPEAKER_01Yeah. Well, we have to. Uh you know, I I I think the the software industry is gonna grow, you know, just a huge amount in importance and and and economic impact over the next decade. And and the bottom line is we're we're not going to do that growth, and we're not gonna have that growth if we pull up the ladder and and you know the the community of software engineers just gets grayer and grayer. Um you know, we're not gonna build this this fantastic new industry on a you know bunch of folks with with you know gray flecks in their beards.

SPEAKER_01Um and uh you know experience and and seniority and and and you know having having been around still matters a huge amount. Deep expertise still matters a huge amount. But we absolutely need to build this industry on a strong culture of teaching and growing and and bringing new folks in, training and learning. Uh and it's tough to know exactly what that's gonna mean tactically, day-to-day, right? Like what does that look like?

SPEAKER_01But I I have no doubt that that is going to be a critical part of what you know what our success as an industry is gonna look like, and what the success of individual organizations is going to look like, is figuring out what is that path for bringing people in from things like universities who often have you know fantastic and very fresh theoretical knowledge and ramping up on the uh on the practical knowledge, on the day-to-day. Um we've got to figure it out.

SPEAKER_01It's gonna be a big differentiator for organizations going forward, uh, their ability to bring in new people and bring in new ideas. And the the benefit of that is is the folks that I see coming out of college who spent a ton of time learning with AI, learning with agents, um, you know, with without uh you know some of the preconceived notions a lot of us come towards them with. Um just come in with so many fresh ideas and perspectives. And and that's a fantastic in uh you know addition to an organization or a development team, too.

SPEAKER_00Yeah, I totally agree. And I think it's it's like you say, like almost like a joint responsibility from the organization and and and senior engineers to make sure that you know these folks are included. The juniors are included in a lot of the decision making and brought up to speed on this. And I think it it's a it's a very understated thing, uh, under said thing that actually junior engineers they are truly AI native. Like we're you know, from our point of view, we've got years, decades of of of you know development and you know this old way of tapping a keyboard and code comes out.

SPEAKER_00We're trying to learn the new AI ways, the AI native ways, but people who will you know come through um you know the newer courses, the newer curriculums, which are truly very AI focused, this is how they grew up, this is how they learnt it. And I think we can absolutely learn from the from the fresh perspective. So uh very much plus one to that. Um Mark, this has been this has been a wonderful chat. I really enjoyed, I really enjoyed the uh the conversation and in particular uh some of your insights.

SPEAKER_00Particularly, I I love the fact that you've that you read 4,000, 34,000 uh uh postmortems and things like that. And Mark, you've written a number of pieces, blogs, and things like that about all of this. It's actually been it's it you have a very insightful uh uh blog with all of this on. Where can people go to read that?

SPEAKER_01Yeah, you can go to my blog, it's at at at brooker.co.za um and uh I tend to share things on X, on LinkedIn, and a couple of other a couple of other social platforms, but my blog and its RSS feed, uh very old school, uh, is uh is the primary source.

SPEAKER_00Amazing, Mark. Thank you so much for your time and sharing all that wisdom with us today.

SPEAKER_01Well, thank you, Simon. It's been super fun talking to you today.

SPEAKER_00Amazing. And thank you very much for listening. Hope you enjoyed that and tune into the next episode soon. Bye for now. The AI Native Dev is brought to you by Tesla, the package manager for Skills and Context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tesla offices in central London. Visit Tesla.io forward slash community to learn more. I hope to see you there.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Your Software Factory Needs a Bill of Materials | Daniel Meppiel

    Nov 19, 20261 hr 1 minAudio

    Developers are about to write skills instead of code, and Microsoft's Daniel Meppiel argues the software factory that runs them needs versioning, a bill of materials and signed receipts before enterprises will trust it....

    Credit the people on this episode

  2. Paul Stack: I'll Never Write Infrastructure as Code Again

    Nov 17, 20261 hr 2 minAudio

    Paul Stack helped build Terraform. Now he says he'll never write another line of infrastructure as code, because an agent harness runs his team's production infrastructure instead. Watch all the AI DevCon NYC 2026...

    Credit the people on this episode

  3. Edith Harbaugh: Code Is Cheap. The Last Mile Isn't.

    Nov 10, 202649 minAudio

    Automated software delivery is moving faster than the people who used to approve every release. LaunchDarkly co-founder and CEO Edith Harbaugh tells Guy Podjarny why rollback should be autonomous, why one-shot coding...

    Credit the people on this episode

  4. Maggie Appleton: Agentic Coding Has Outgrown the Text Box

    Nov 3, 202652 minAudio

    Agentic coding has outgrown the text box. Maggie Appleton, staff research engineer at GitHub Next, argues that walls of CLI output and AGENTS.md files serve coding agents far better than the humans steering them. She...

    Credit the people on this episode

  5. Caleb Sima: AI Attackers Move at Machine Speed

    Oct 27, 202655 minAudio

    AI attackers now move at machine speed, while most security teams still move at meeting speed. Former Robinhood CSO Caleb Sima has seen an AI attacker up close, and he argues the only way to keep up is to take humans...

    Credit the people on this episode