My friend Mike Rollins joins me for a chat about what it’s like leading a software engineering team that doesn’t read code anymore. Click here to watch a video of this episode. Thanks to SpeechifyAI for sponsoring this podcast. Get started for free. Check the benchmarks for Best Provider Text-to-Speech 😇 My new GitHub Security workshop has launched! A free 2-hour workshop with hands-on labs to harden your repos and your workflows from common supply chain attacks. I'll cover how attackers are getting in, and then we'll lock down a sample repo so you know what needs to be done to protect your code. You'll leave with a deep understanding of risks and mitigations as well as a list of helpful tools to keep your repos safe, including my new "gasa" tool for scanning your repos and orgs. ★Show Links★ https://github.com/rollinsio/delta-v-dev-container-boilerplate https://github.com/rollinsio/beyond-test-coverage (expand your testing strength) https://rollins.io/ https://www.instagram.com/rollins.io Creators & Guests Bret Fisher - Host Beth Fisher - Producer Cristi Cotovan - Editor Mike Rollins - Guest You can also support this podcast by subscribing to my YouTube channel and my weekly newsletter at bret.news! Grab the best coupons for my Docker and Kubernetes courses on Udemy. Join my cloud native DevOps community on Discord. Grab some merch at Bret's Loot Box Homepage bretfisher.com (00:00) - MAIN - Video Podcast (02:14) - Welcome Mike (02:23) - Mike Rollins AI First Journey (06:55) - Speechify.AI (07:57) - Agentic Delivery and Safety Nets (17:50) - PR to Production (35:10) - Testing and PR Chaos (51:58) - Harness Safety and Cost (58:34) - Maturity Milestones and Wrap ★ Support this podcast on Patreon ★
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Michael Rollins: The big milestones for us were we started from, you know, everybody's using AI to code. We moved into, " Hey, "When's the last time any of y'all actually looked at the code?" And all my team was like, like, everybody didn't wanna get caught. I was like, "Oh, 'cause I haven't looked at it in months." Yeah, and they were like, " Oh, God. Okay. Phew. Thank you. Thank you." Bret Fisher: imposter syndrome that jumps out when the junior-est member of the team posts something in the engineering Slack that calls out something the most advanced engineer's done, and you're like, " Oh, God, I'm
not even a real engineer." Like, how… And then you're like, "Oh, wait, you had Claude look at that, didn't you?" And they're like, "Actually, no. It was Codex." You know? Bret: Welcome back to the Agentic DevOps podcast, and I'm your host Bret Fisher. I have my friend Mike Rollins on we went to lunch, and the way he talked about his AI-first engineering or agentic engineering or agentic dev, what- whatever you wanna call this thing, where you're throwing tokens mostly and not writing anything anymore, and you're not even looking at code anymore.
And he runs a team as a CTO, a small team in a startup, and I wanted to talk about what that life is like. 'Cause a lot of us, you know, we- we're in enterprise or we're, I would call, like, legacy teams where, we're toying around with AI. We're certainly using it for code, but we're not necessarily riddling the entire pipeline of the software life cycle with everything AI so that we never see code again and that everything just goes as fast as possible. Like, most of us are somewhere in there in the middle or near the beginning, I think, and certainly with a lot of the people I talk to, where they aren't
necessarily able to throw out the playbook that we've had for the last 20 years on how exactly a team uses Git storage and automated deployments and test-driven development and all the things that maybe, were hot topics before AI, right? So Mike and I go through a pretty lengthy conversation, but I wanted to get as detailed as possible to talk about the culture, the pros and cons. Like, what are the downsides of this? Tell me some horror stories of how this can go wrong in so many ways and what it's like trying to push the envelope, being on the bleeding edge of AI-first development.
So let's get into it Bret Fisher: Mike, welcome to the show. t- tell the audience, who you are and what you do for a living. Michael Rollins: So, uh, my name is Mike Rollins. the majority of my career has been in Silicon Valley. back in '22, Our company was getting bought by Twitter, and then this dude named Elon jumps in, and we were no longer being bought by Twitter. And as a result, I went on this multi-year escapade of, I worked for a healthcare startup, and then after the health- I left the healthcare startup, or I was asked
to leave the healthcare startup, because it turns out if you ask very difficult questions pointedly, people don't like it. I ended up, going o- out on my own, and it was terrifying, scary. I mean, from about 2023 until really November of last year, I was a contractor, and I did a bunch of stuff. I was the founding engineer for multiple projects. I was this… I've been the CTO now of three different companies, though mostly fractional. And I have at some point, the internal clock in my head is always ticking, and
I feel this constant need to go faster. And so as I was building things as typically the only engineer, the clock in my head is just faster, faster, faster. And so when AI became, like, a thing, meaning back in the latter half of 2024 and then especially at the beginning of 2025, myself and a few other folks, a guy named Thanos, who he and I have a podcast together that we're kicking back off called Will AI Snap That, we both just dove in.
And what's funny is that since about really the entry of Claude Code into the market, I have been doing AI first engineering. And what I had noticed is that by the time we hit about May or June of last year, I was no longer writing code. And then right around November of this year, I realized I was no longer reading code And it was this, shocking realization. And so I basically dove headfirst into AI j- and, you know, high risk, high reward.
The risk here is that my family can't eat, right? The reward is that I figure out a way to do engineering that other people don't know, and it's what we did. And so nowadays I'm at this company called, Relefy and then also Family Digital. I'm CTO of both. And I have this team of four other engineers, and really truthfully, only one of them, well, one of them is a designer that we've turned into a front-end engineer, and none of us read code, none of us write code. we ship… We've built this agentic system that's built from the ground
up for sharing, and it is kind of, it's built at right angles to the way everybody else is doing things, and we are moving so fast, it's terrifying. But the clock continues to tick, and I continue to think, "How do we go faster, faster, faster, faster?" But I, dude, never in my life… I got to write mobile. In 2011, I started writing code for iOS and then Android, and mobile felt amazing. It felt like a freaking rocket ship. Everything changed so fast because there was a new phone about every three months.
And now it's dude, didn't we just see a new Frontier model a week and a half ago? 'Cause we did, and it's just unbelievable. So anyway, that's how I ended up here. Bret Fisher: Yeah, that's a great story. And, I was on the hardware side of mobile, not the software side. So, yeah, it has become very apparent to me in the last few years that it's like AI, that's a good analogy of like AI feels like it's kind of replaced mobile as the exciting tech where that was really… I mean, mobile kind of came around a little bit a few years after cloud, but cloud I feel like was pretty well understood within four or five years and
then we had mobile and that was exciting, and then there was like this lull. I kind of feel like 2018 to 2022 was kind of this lull of, we Michael Rollins: Yes, it was. Bret Fisher: yeah, we were all sort of exhibiting the Michael Rollins: was supposed to be crypto. Bret Fisher: Oh, that's true. Yeah, the vacuum of crypto, which, you know, and I don't know anyone that touches it with a 10-foot pole. So, yeah, that was a thing, but not necessarily in my ecosystem. So we all were talking about how isn't it great that Docker and Kubernetes are boring? Isn't it great that like microservices and distributed services and cloud tech are all kind of boring and, we can just focus on the work and, enhancing our craft and getting better at it?
And then, you know, ChatGPT shows up and then we're all copy-pasting. So I wanted to get into some of the nerdy details of… it sounds like it's a relatively small team, so a total of five it sounds like. There's a total of five of you. So does that mean you have five people making PRs into, presumably GitHub? Michael Rollins: Mm-hmm. Bret Fisher: And how do you go about even starting something? Is it like an individual thing where you start an idea? you mentioned this idea of concepts first before ideas or something like that. Bret AI July 2025: Have you been adding text-to-speech in your apps? One of my late-night projects this year was creating a personal
podcast feed so I could listen to the changelog of my favorite open source projects while I do my chores. I needed a text-to-speech API to do that. I also heard from a colleague this week who built an app for his kids to turn their e-books into audiobooks, and he's using the Speechify AI developer platform to do that text-to-speech Speechify AI is a developer platform for real-time text-to-speech, and they make their own voice models. Their Simba 3.2 model ranked number one on the Artificial Analysis Benchmark for conversational voice quality.
They've got a free plan, streaming and REST APIs, TypeScript and Python SDKs, and if you wanna get fancy, you can use their SSML metadata support to add emotion into the speech output Whether you're shipping a weekend project or production voice agents at work, you can start free at speechify.ai. The link is in the show notes. Thanks so much to Speechify AI for sponsoring this episode. Michael Rollins: well, let me start at the end for you Bret Fisher: Okay Michael Rollins: because this is typically the framing… This is the framing I've
been aiming for, and we're almost there. What does it take for me to allow an agent to pick up something, build it, test it, commit it, open the PR, make sure everything runs, listen to all the feedback, merge it to main, and deploy it, and a human is never involved, and as a matter of fact, a human may never know that it happened? What do you have to do to make that real? And risk is really the big asterisk there. And I, you know, as the clock is ticking in my head, I, I make decisions
based on a risk assessment, right? What is the likelihood of failure versus what is the outcome if we fail? And what you find is that even if the likelihood of failure is high, but the risk associated with the actual failure is low, you may choose to take the gamble. Conversely, if the likelihood of failure is very low, but the risk is extraordinarily high, you're pro- you may not take that leap. It's not a gamble. It's almost a guarantee, but you know that if you hit the Enter on this
select statement in the production database, and it goes sideways, like the failure mode could be the database disappears, and then so does the company. And that's one of those things where it's like I've made that decision and had it go bad, right? I've made… I have wiped the company database Bret Fisher: Oh, are you the one that's like on the tip of the spear just so the rest of us can learn from your failures? Michael Rollins: when I was at Outlier, we did machine learning, over time series, and this was in 2018 to 2022. That was the company that was gonna get bought by, Twitter.
Bret Fisher: Yeah Michael Rollins: I was on call for three month- for three year straight, and… I wrote direct queries in production databases constantly, ' cause it was the only way that we could keep the system up and running sometimes. And I would wake up every morning and I would rebuild the system, and there were a couple times that, boy, did I that up. even in a couple, not the current company, but the previous one, there were a couple times I just, I wasn't thinking. I knew I was doing something a little bit risky, and I did it anyways, and
had to go desperately hope that we had a recovery copy stashed away. and we did. And so I have legitimately, I've blown it all up. I'm very risk-tolerant in that aspect, but the reason I'm risk-tolerant is I fixed it And so when I look at… let me give this as an example. we had an incident occur a few weeks ago, and all of a sudden I realized that we could no longer upload files to our system. And now keep in mind, it's an agentic system. You create a bunch of agents.
They can all talk to each other. It's a lot like OpenClaw, except we were doing it first, and we built it to be enterprise safe. And part of that is, you have to upload things, and they get shoved into the RAG database, and they get shoved into the S3 file system, and I tried to move a bunch of things around, and I couldn't. I was like, "What is going on?" Now, I'm at my daughter's dance competition. I don't even have my computer with me. I'm the only engineer that can fix this at this moment. And so I'm actually on my phone, so I swipe over to Claude Code.
I have a session running on my machine at work. I tap into it, and I say, "I want you to you know, clear your context. I want you to pull main. I want you to get to a new working tree, and I want you to help me diagnose what this problem is." Within about four minutes, I knew that the disks that were, serving the ECS, the EC2 or ECS images were full, ' cause we had not been pruning them after we would deploy. And I had Claude go in, 'cause it also has my AWS credentials as part of it, and said, " Go fix it."
It went ahead and fixed it. I didn't even have to restart the system. I had the whole thing back up and running correctly in five minutes. Bret Fisher: Yeah. Michael Rollins: The cost Bret Fisher: basically, or what- whatever, whatever Michael Rollins: yeah, whatever the container service equivalent was. But that was it, right? And well, the risk was actually pretty low, and the likelihood of failure was almost minuscule because I knew that I could do everything from the command line on my machine through the app that was on my phone. And so the risk profiles of everything are just different now. so when I sit down and I think about what does it take to allow an agent
to write code and ship it without me ever knowing, there's an inherent assumption built in that any failure state is something I can recover from in a relatively short amount of time. And now there's an extraordinary number of assumptions that are involved there, and it's not like this is YOLO, right? Our build infrastructure, our GitHub actions, everything that happens post-commit is methodical, it's intense. going back to what was your initial question, we don't do tickets, we do features.
And oftentimes because of the speed with which we move, we… There's not even time to build a UI. Our designer is often catching up to engineering. We have officially broken the entire organization because engineering moves faster than sales. And that is a wild thing to deal with on a business level. like Bret Fisher: that's when people get fired, right? Because there's too many engineers for the sales team and… But that so rarely ever happened 'cause there was always a gap. Michael Rollins: Yeah. I mean, the, typically the lead time from sales to engineering is six months. Bret Fisher: Yeah. What do I tell the customer?
We need them, Michael Rollins: Yeah, what do I tell the customer? Bret Fisher: we need the feature." Yeah. Michael Rollins: yeah. And I'm like, " It's already there. Just sign the contract." they were like, "It wasn't there two days ago." I was like, "I know. We shipped it." Bret Fisher: They're so not used to that. Yeah, especially if they're old school and they're used to the giant lags Michael Rollins: Yeah. Well, and, you know, so the, one thing to say is the mentality of engineers has to be totally different. My engineers have to understand the business, and I don't let my engineers just focus on tickets. As a matter of fact, that's an amazing wayl- way to fail catastrophically at this point.
instead, I say, " Here's a thing, and I want you to understand it and I want you to do it." And the engineers that are working for me at this point, which by the way, none of them have professional experience before they started working with me, these folks are making really strong judgment calls, and that's all I want. I want good judgment, right? The model can, and the harness can figure out how to write the code. Bret Fisher: So is there a large part, is a portion of your training or cross-training in the team helping everyone with judgment? is that become an increased thing?
'cause since you're going so much faster, you, there's less time to talk about things, so people have to be able to make more decisions on their own, which means they need more agency, which, yeah. Yeah. Michael Rollins: There's one amazing way to let people build better judgment, and you let them fail, Bret Fisher: Mm-hmm. Michael Rollins: we shipped something this morning, and, I had actually started work on it and then had to move on to something different. And so Miko, who's a official… He would be classified a junior engineer, right? Miko will have been out of college for approximately a year in the next few days. Miko's making decisions at a senior level, but he shipped something this morning,
and he overlooked something pretty major. Pretty major. Yeah. And I halfway knew it was gonna happen, and I let it ship anyways. And, that is one of the things that I allow my team to fail. I will let the problem hit production, is once again, it's a risk assessment, right? Because you've done it and I've done it. There's nothing quite exhil- as exhilarating as realizing you just shipped a nasty bug, man. everything is being destroyed, and it's your fault.
Bret Fisher: It, that is a permanently ingrained memory that you cannot escape and forget. it becomes a part of who you are, of your identity whenever you do that, Michael Rollins: and so my leadership style actually relies pretty heavily on that. I know that I'm tossing people in over their heads, and I do. And, but I also tell them, " Hey, listen, you're probably gonna fuck this up. Bret Fisher: Yeah Michael Rollins: And when you do, I'll be here. You just come find me. There's no blame. I know you're in over your head. You know you're in over your head, and I happen to know that there's sharks
and they have lasers and you don't yet." Bret Fisher: Yeah. Michael Rollins: so, Bret Fisher: it's challenging. so yeah, you're giving… You're challenging people to grow and rather than coddling them through every mistake that they'll never remember, because I'm sure if you told them, walked by the desk or, the equivalent, the analogy of walking by the desk and going, "Oh, what you working on? Oh yeah, you probably wanna check this so that doesn't fail 'cause that would've totally broke production." They may not even remember that in five months or, you know, two months. So, but basically the boss just told them a rule, but that's, they don't really maybe know the deep why. I'm gonna ask real quick. is the assumption here that you have solid, It sounds like it got past
testing in this scenario, so that means that the testing was lacking for that particular assumption. And then Michael Rollins: it wasn't, Bret Fisher: also assume, by the way, that production can, fail back? do you need… In order to give you the confidence that this isn't going to ruin the company, do you have to have, some other structures to be able to roll back automatically? Or, is there stuff like that in play that you're relying on? Michael Rollins: So in this instance, it actually wasn't a bug. It was a missed part of the feature. so this was a little bit unique. But honestly, I mean, that's okay. Here's the interesting thing. It forced us into the decision of whether or not we wanted to address
it, and we decided not to address it. And yeah, you end up ha- you find yourself in these situations sometimes where because you can't necessarily cover all your bases, you find yourself making decisions you might not have made initially. In this instance, we were basically changing the ownership of a certain type of object within an agent. So previously, it had belonged to the user, and we realized it needed to belong to the agent. And so we fixed that, right? Now all of a sudden these things belong to the agent. But there were a bunch of them, and they previously belonged to the user, so we had
to make the user's core agent own them. But if you were using a bunch of these things in another agent, they were no longer there And it's "Oh, there's no elegant solution for this. So how do we get things from the PR to production, I think is a great place to really hone in. And the first thing you gotta know is that whenever our, in our environment, whenever you issue a PR, we stand up a copy of prod for the whole PR. … it is a somewhat expensive way to run things, and it also catches
so many things immediately. now at the same time that we're standing up that copy of production, and this includes the database. This is everything, right? It is a true copy of production. we have just a boatload of tests and checks and all kinds of things. I think last time I looked, there's 15 or 16 different workflows that get kicked off. we've got multiple, AI, audits. We've got, s- I mean, look, we have tests running, I don't even know what they are.
They just run. Like, Claude was like, "I don't know, we should do this." I was like, "Excellent. Let's make that happen." now it took me about a week and a half to literally tear our old testing infrastructure out and put this new one in. but it's ratcheting. We have rough running to make sure that things continue to get more and more, you know, covered. We have unit tests, we have end-to-end tests, we have Playwright tests. We have, a bunch of, we have linters that run, even though I think they're useless. and so, we catch a lot of things before they even hit, before the
production, before the, that we call them PR environments, before the PR environments are built. We know there are problems often. One of them is, we double-check that we don't end up with a, too many Alembic heads, right? I hate Alembic. It is literally one of the most awful pieces of technology on the planet if you don't use it right. And so we've been forced to use it right. Alembic does our database migrations, by the way. So that's the first line of defense, right? now I will take one step back and say that every single engineer can also run the entire stack on their machine That's-- You have to start there.
If the engineer cannot… Yeah, table stakes. you know, everything in a way, prod-production is a copy of our local environment because even though we don't have the data, like I'm not gonna let you put production environment data on your environment locally, you have, like, all the Docker images that get built on your local also get built with different environment variables in production. And so you have to be able to run everything locally. We also make sure that the model knows, "Hey, you need to make sure that there's unit test coverage for everything you do. You need to make sure that you're
covering, you know, Playwright end-to-end tests." I'm very, very big on… Now, this is because we're primarily a web app, but I've also done it with mobile now. I want the model to run the actual app locally and double-check that everything it built actually works. And so it's not just a set of tests locally, it's also, " Hey, I want you to fire up an, a Playwright MCP session, and I want you to click around and double-check that this actually does what you say it does." hallucinations are real, and the only way to really catch them so that they don't become
catastrophic is to literally make the model prove that it's not a hallucination. so that's our testing infrastructure. it's the best I've ever seen Bret Fisher: what are these, what are the, are these tests, are you using Makefiles? are you g- you got, are you using testing frameworks that are, not frameworks necessarily, but trust testing tooling that is, y- Michael Rollins: we use Jest for TypeScript, and then we use, I don't know, one of the Python… Our back- our entire system is just TypeScript and Python. Bret Fisher: I'm a big GitHub Actions fan. I teach GitHub Actions courses. Michael Rollins: GitHub Actions. Everything's based on the GitHub Actions. Bret Fisher: and GitHub Actions locally is often problematic
and difficult, and I generally don't use it even though we have, Michael Rollins: we don't, I don't run actions locally. Yeah, I, yeah, we don't run actions locally. Bret Fisher: I tend to throw stuff in the agents file, like always run these three linters and then always make sure they're in the, make a make file so that everything you do is through a make file. Because I don't know if this is advice or a, the way that I do it, and I'd love for you to… your opinion on a lot of the things I'm doing. because often, if I go fast enough, it's making so much stuff. I built a Mac app over the last month, a recording app because I'm so particular that the thousand different options for recording screen and video is not good
enough for me, so I had to make my own. Michael Rollins: I agree 100%. Bret Fisher: and as a part of it, it started to have all sorts, like I need to, it needs to do a dev build, and it's got a prod build, and it copies it into my, this place. And so I started to say, "Hey, look, you're just writing a lot of commands for me that I don't even know what you're doing. Could you make sure that all that is always in a make file?" And I made that a rule in my agents file, and it started doing that. And then I said, "Okay, always m- run these linters when you're done at time, at every time you edit, always run these linters, always use this make file." And 'cause I started to realize that I couldn't do anything without it, right? I couldn't sit at the command line and know how to build my app because I'm not a SwiftUI guy.
Michael Rollins: Mm-hmm. Bret Fisher: and so I've started to the point where I, those are almost table stakes enforcements for all of my projects, where I want the agent locally to behave a certain way, like running tests. So the challenge is, because I'm using GitHub Actions, I end up with it just writing, whether it's a compose file or a make file or, you know, whatever. I'm always curious what other teams are doing since I feel like there's this pull to pull the testing into wherever the agent's living. So if you're running a harness locally, you probably need to run the test locally because the agent needs to run them, unless you plan on going to get a new coffee and you just tell the agent, please run all these in GitHub Actions
and let me know when they're done." And then you have to wait 20 minutes because, or 30 minutes or longer depending on your test, harnesses and stuff and, So I was just curious what you do. Michael Rollins: I actually have, it's publicly available on my GitHub, if you wanna go see how I start a project. this is it. It even has a skill that you can use so I call this the DeltaV Launchpad. it has a Claude skill that'll basically set this up for you. it has a Make file. n- I do everything with Make. I'm like you, I got completely just dead tired of Okay, go do this again."
Goes-- Finally, it's just "Go use the Make file." Make files, you know, the Docker configs are all, compose files. it's all pretty standard stuff. this is all-- None of this is rocket science. we all know how this is built. Bret Fisher: Yeah Michael Rollins: And so I just picked something Bret Fisher: yeah, I think a lot of the people in the my guild, the Agentic DevOps Guild, th-there, you know, there's a wide range of skillsets in there. There's people that are, like, very new to this. They're dipping their toes in the Copilot or Cursor, maybe Claude Code. there's people all on the, all the way on the other end that are, you
know, they're using s- they're using plugin marketplaces to manage their skill deployments to their teams and centralizing the skill management as, while at the same time they're l- adding more and more agents to their post-commit workflows to self-improve things, on occasion, which i-is a topic I wanna get into at some point about. the idea that once you've sort of matured yourself to, you know, you've got your agents reviewing PRs. You've got sort of like the t- the table stakes of agents post-commit, where you've got it, you've got agents looking at your PRs. You've got agents, maybe you've, you're lucky enough and you've got
Michael Rollins: Why, but… Bret Fisher: you troubleshoot or something Michael Rollins: Yeah, but here's one of the questions everybody should really ask. Do we have a way to do this in code? ' Cause if we do, we should do it in code. Bret Fisher: Yeah. Michael Rollins: It's funny, I talk to people sometimes, they're like, "I wanna build an app." I'm like, "All right. Tell me about it." And this one guy told me about, he runs his pool business on an Excel spreadsheet. He was like, "Do you think I can turn this into an app?" I was like, "Absolutely." He goes, "Where should I add AI?" And I was like, " Nowhere." it runs on an Excel spreadsheet. It's deterministic. Do you have anywhere in this process where you want a
non-deterministic choice to be made? ' Cause if the answer is no, just run the code. but yeah, I mean, look, I tend to drive certain types of engineers insane because I kind of don't care. You know, you're like, "Well, what are you using for your testing?" I'm like, "I don't know. We're using something. I know there's tests there." Right Who's looked at your tests to make sure that they're valid?" " Nobody." " How do you know your tests are good?" "Well, I have about 80 other things that check." Right? And so, I don't dictate to my team the tools, right?
I do say, "You're gonna use Docker," right? "You're gonna use Docker Compose. You're gonna use this Makefile." But that's more just for, efficiency's sake. you know, one of my engineers, he's way smarter than I am, he uses Codex. I think it's a terrible idea, but he uses Codex, and we fight about whether or not ChatGPT or Claude are better on a daily basis, and I tell him he's still wrong, but whatever. though it is funny, we get to watch the models leapfrog each other. I really don't care what the tools are as long as they get the job done, and that means I also change the tools often.
Bret Fisher: Well, no, but I mean, it brings up a good point that, sometimes the reason I ask these questions is because it, in the, in the question of w- you know, think of some of the things you're using in CI and just rattle off some toolings because we can be vague and hand-wavy about a lot of these concepts that… But I always like to get into the nerd details. But the reason I ask that is because often the answer is, yeah, Docker Compose files and Makefiles and like in other words, there's nothing new in there. There, it is exactly what it always was. I personally, don't, I don't know if this is the general advice you would give, but I personally advise a team to guide their AI to make the
choices on the toolings that they know rather than just let it make… 'Cause in, in theory, it, if you asked an AI before you greenfield something what language it thinks it would use, it might, it probably will prefer Python or TypeScript. It, but it might make a bold choice, and that doesn't mean it's right for your team. And so people I think are sometimes, maybe not so much anymore, but I think they were looking at this at "Is this an opportunity for us to reinvent the wheel in a lot of our tooling?" And my argument to them is n- no, because at any point, look, the model uptimes aren't 100%.
So if at any point you're stuck and you have to read something, it, wouldn't it be nice if it was in a thing you knew in- instead of some completely foreign… Like, I'm not gonna have my AI building a Podman interface to my containers when I'm the Docker guy. So if it starts to do that, I'm gonna guide it back to the places, even though I may never, ever look at that Dockerfile after the first creation of a, the, of the PR. and I don't know if you have standards like that, but I've always been imagining that teams need to actually start, even polyglot teams, they need to kind of settle in and say, "Look, these are the languages we would really prefer
you stick with in AI and not just, don't be COBOLing our production." Michael Rollins: I've got this Instagram channel that seems to be blowing up, and I piss engineers off all the time ' cause I say things like, "You shouldn't be reading code, and you shouldn't be writing code." And s- there's quite a few engineers that are very, very literal. the other, another one I had recently is like, "Nah, just make sure your tests run. It kind of doesn't matter what they are." And, but I had a few people get in, like really get nasty arguing with me, Bret Fisher: Yeah. Messing with their identity, bro. Michael Rollins: No, no, they were right." They were right because my assumption
was different. So I've had a couple people come in and say, "This is insane. How can you do this? It's too risky," da, da, da, da, da. And I've had a lot of people come in and say that. Well, then it turns out that they're building embedded systems. Bret Fisher: Hmm. Michael Rollins: They're in healthcare, they're in banking, they're in some kind of high security field. I'm like, " Yeah, don't listen to me." Bret Fisher: Right Michael Rollins: But if you're building web apps, if you're building mobile that is just an extension of the web app, Bret Fisher: Yeah. Michael Rollins: it's already been done. Just use the standard.
You don't have to reinvent that wheel, Bret Fisher: you have a very valid point about your assumptions because, you know, you both of us op- have been operating… I mean, I've been in cloud native for over a decade. you know, the h- the things that we learned about distributed computing, microservice architecture, you know, API contracts, like all the sort of the things that kind of evolved over the last 15 years, like I've been living and breathing that at the near tip of the spear. So when I meet a team that they have 3% test coverage, they're not using a type safe or memory safe language, they may be YOLOing some old PHP stuff and they've got a very monolithic approach, and then they're like, " How-- We can't do this.
We're having all kinds of production failures." and I look at them and I would evaluate it and say, "Yeah, that's because…" I love this phrase lately, is that AI AI just amplifies the team culture, and your team culture is of 2010. You need to modernize a lot of your culture before you go fast. Like you can't go fast without the safety, and the safety was never there to begin with. And so I do run into that on occasion with teams. And there's surprisingly a number of teams, maybe not in embedded or healthcare, but like a surprising number of teams that have very large, important apps that I get c- to consult with, and I cannot believe the, I
would call it the level of immaturity that the code base and the team has. And 'cause we don't have accreditations in, in, in our business, right? We don't have plumber certifications. So like I can't look at them and assume that they know all these things and that they're doing all these behaviors that I've been doing for so long. Michael Rollins: I have been able to look under the hood at multiple Silicon Valley giants. Bret Fisher: Hmm. Michael Rollins: It's all shit all the way down. Like every single engineering team is "We have the worst stack ever. All of our code is crap." And it's "No, your code's fine." there's really only one measure that matters when it comes to talking about your product.
Does it work? A follow-up question is, does it work all the time? Bret Fisher: Mm-hmm. Michael Rollins: If the answer to those two things is yes, it doesn't matter. Now, there's a third implication now recently, especially with a bunch of the Mythos stuff, which is it secure? And that's always been important, and it's always been relatively easy to exploit sloppy stacks, right? Bret Fisher: By the way, Sloppy Stack is the name of a future podcast. Michael Rollins: Oh my God, yes. That's, that's really… Yeah. I love, I love accidentally, yeah, yeah, break down of your sloppy stack.
Now we're gonna enter it. It doesn't, it also sounds really gross. whether or not your architecture sucks, whether or not your stack sucks, whether or not anything sucks, there's only one measure that you have to apply to it, which is does it work, right? And now that d- decomposes fast into tiny little things. but if it works and it's built on ancient PHP, maybe there's not a reason to fix it, right? Bret Fisher: I have another, follow-on question. So we, we're seeing a lot in the news around, the general degradation of code quality as people onboard AI, and I understand how it's
happening, and I don't understand why it's a mystery on how to fix it. Because it feels, at least in my experience, if I have all these review agents and PRs and I'm using I've got a series of skills, that are auditing, security audits, uh concurrency audits, performance audits, you know, these are things that used to be very hard for engineers to do because it required a lot of manpower, and we would often skip them, especially when budgets were tight and project deadlines were close. and continual documentation updates, I'm just thinking, going down the list of the
things that I can now have agents burn a bunch of tokens on, to me, it doesn't make sense that the code quality would go down. It feels like to me that, with enough gates that I'm triggering AIs in, that the code quality would, in terms of maybe not my impression of how the good code quality is, but at least the way it performs and behaves in production, feels like it should at least stay the same, if not go up. Is that what you're seeing? Do you see that or do you see the opposite? am I wrong? Or Michael Rollins: I don't read the code. Why do I care? Bret Fisher: Yeah, well, but you see the outcome of the code.
So obviously, Michael Rollins: see the outcome of the code. Bret Fisher: So if your production, if you were five nines before and you're four nines now, you would notice that, assuming you're measuring it. But, Michael Rollins: yeah, I mean, listen, we're not five nines. Like I was… We're close. we're much closer now than we were two months ago. you know, it used to be I had to take about 10 to 15 minutes of downtime to d- per any release because of the way our stack is actually built, and because of the way that we release. But I've since fixed that. We don't have downtime. as a matter of fact, I mean, we probably are close to something like five nines.
But the question is like, Well, let me, let me turn this back onto you. Why does code quality matter, and what is it? Bret Fisher: Well, Michael Rollins: Why are you focused Bret Fisher: that's a question. I would say, I- what I'm specifically talking about is failure rates in production as a DORA metric. that's-- and this is the problem. This is my assumption, is when I think of code quality, it isn't whether you tabbed or spaced. It isn't whether you followed the black linter on your, you know, whatever, spec. it is about when you got to production, in an enterprise there will be a security team that does a production analysis of what's going on, and they'll
find more flags than previously, or you'll start to have more outages or customer complaints than before. And so these metrics, according to the mark- the news, the media, you know, these things are going up for teams that are going too fast in AI, and that's, that-- Some are deriving that as the conclusion means humans still have to read the code. I don't necessarily take that as an outcome of that assumption. I take it as, no, you need to put more, you need to put more rigor into the AI process, not back up to the world where we were all reading every line of code.
So I was curious on your take. Michael Rollins: it would be interesting to go back to when the Agile Manifesto was written. Everybody had done Waterfall to that point, and all of a sudden somebody comes out and they say, "This is stupid. We should, change as fast as we can." Right? "We shouldn't plan for a whole freaking year. That's insanity. What we should do instead is we should, take stock of where we are and take stock of where we wanna be in two weeks and get there." I am willing to bet that the performance of Agile at the very beginning was much worse than Waterfall. Nobody had done it. Bret Fisher: Right Michael Rollins: Everybody was trying to figure it out.
This is worse, right? Bret Fisher: This is, this is that on steroids. Michael Rollins: Yeah. I look at the things we are doing now and on my team that we weren't doing two months ago, and I sweat Right? But you know what? My release rate, the amount of code that has landed is hockey sticking, and I've been forced into what I'm doing now. I would be willing to bet that we are shipping more bugs than I wish we were shipping. I don't think we're shipping the same bug twice, at least not one that,
like… Have we had regressions? Absolutely. have we had regressions in this changeover period from the old way we were doing it until the new way we're, we've done it? Yes, 100%. I don't know that we've shipped a regression since we started on whatever this new thing is that we're s- that we've been doing. Now granted, we've only been doing it for three weeks. Bret Fisher: Yeah. Is there like a hot tip on exactly, do you add these to a skill? Do you add just, do you add new tests for every failure? Like Michael Rollins: we have to be absolutely obsessive about tests. Bret Fisher: Yeah. Michael Rollins: Yeah … I hated tests.
I hate tests. I hate tests as an engineer. people that are like, "Test-driven development," I would always be like, "You are out of your… That is the stupidest thing." I could not think of a better way to grind an engineering team to a halt, right? we gotta ship this fast. Bret Fisher: I'm the same way with spec-driven development that I am with test-driven. I'm a shiny person and none of those things give me the dopamine hit of shiny. Which is why no one pays me professionally to develop full-time. Michael Rollins: Yeah, but you know what? I get test-driven development for free now. Bret Fisher: Yeah. Michael Rollins: I don't have to worry Bret Fisher: the pain Michael Rollins: Yeah. let me throw this at you. I was working on 15 different bugs at the same time the other day, okay?
was doing so much that I locked my machine up by filling up the disk, because apparently if you run Node, the, what is it? The npm install and the type, you know, the Python equivalent, that can be a lot of space, and I only have a half a terabyte hard drive on my work machine, 'cause I decided that's a fine thing. Bret Fisher: that code is small and that, you d- Michael Rollins: And I was building Docker images, yeah. and, next thing I know, my machine just stops. not slows down, it just stops, right? But in the process, I had also opened up 15 PRs.
I locked up the entire, stack. I accidentally flooded the GitHub Actions, overloaded the Lambda that runs our, database cloning, where we clone production into the PR environment, and found a bug in the process, which was very nice to find. but it was not the first time that it happened, where I had basically launched this clone storm that locked our entire PR infrastructure up. Everything was still running, but you couldn't do a new thing. I've never tried to land 15 PRs at the same time before.
Have you ever done that? Bret Fisher: Not even Michael Rollins: Has that… Have you ever heard of anybody doing that? Bret Fisher: Yeah, I mean, there's the whole new, what's the PR gating thing where you combine them? Michael Rollins: as, PO it's like merge, merge streams or something. Bret Fisher: yeah, merge stacks or Michael Rollins: yeah, it's basically what GitHub just totally blew up. Oh man, GitHub sloppy stacked this hard. so, but listen, yes, of course it's, it may, yeah, maybe it is degrading, right? but nobody's ever done this before, man. Look, we were doing it before Anthropic and before OpenAI.
We were doing this style of engineering back in July of last year. Yeah. Anthropic and OpenAI did not adopt it. nobody's done this, man. And so, yeah, are we finding things are degrading a little bit? Sure. you know how long it takes me at this point to put a fix into production if I notice it? 20 minutes, 30 minutes, assuming I have to build the images. Bret Fisher: I recorded a very similar episode to this topic yesterday with, someone from GitHub Next and then someone from Microsoft Research, and they're both working on the agentic workflows feature set for GitHub,
which is currently, I believe, in beta, and I'm very excited about it. I thought it was very weird at first, but after spending an hour and a half with them, I'm all on board and I'm very excited about spending more time with it. He talked about that he's also working across 22 projects with multiple PRs in each project, and that he does everything that-- because they're GitHub, they do everything through Copilot, everything in GitHub. And so he spends most of his day as a researcher that's, or as a developer researcher, I should say, prompting through github.com/, what-
whatever the little agentic, page is where you can prompt like you're in Co- Code GUI or Claude Code GUI. and he was saying, you know, he does it, He spends a lot of time on his mobile, doing it, and, sort of talked me through his day to day and I was like, "Man, I feel so antiquated," 'cause I'm still got this habit of if I think of something, I go and write it down on my phone, but I think I'll do that when I get back to my computer because I'm just so not driven to use Dispatch or I'm just not there yet. Yeah. I'm-- I know I have it and I just don't default to it yet as, "Oh, I can kick, I can kick this off right now. I don't need to wait till I'm back at my computer."
Michael Rollins: So this is, this is what I do. I have two machines that I write code on. one is this laptop here, and one is my machine back at work. I typically use my machine back at work for this. I have remote sessions or remote environments or something, turn on automatically in Claude Code, and then I also have Auto Mode turned on by default. Bret Fisher: Right. Safety off. Michael Rollins: no. I leave safety on. Auto mode is not YOLO Bret Fisher: oh, right. Sorry. Auto safety mode. Right. I'm sorry. I'm th- I was thinking of Michael Rollins: it's legitimately called, there's planning, except that it's whatever the default, normal default state is, and now they added Auto Mode.
Auto mode actually has a safety classifier in it, and so there's effectively another agent that's over here running that says, "Is this dangerous? Is this dangerous? Is this dan-?" No longer can you do that, right? Bret Fisher: Right. Michael Rollins: it'll just smack it down. Bret Fisher: Sorry, you don't, real quick, you don't enable, sandboxing in the Claude Code? Michael Rollins: actually Auto mode may be some kind of sandboxing in the background. Bret Fisher: You could call it that, but yeah, 'cause it's gating the tool. it's gating tool access basically, right? Like it's a programmatic deterministic gate, at least as maybe not auto, maybe part of it's a little bit of an, a mini LLM or something in there. Michael Rollins: I'm fairly…
Bret Fisher: deterministic way to do it. Michael Rollins: Yeah, but then there's also, then there's your unsafe mode. Bret Fisher: un- hands-on. Michael Rollins: Okay. So the danger with unsafe mode is actually, auto mode and unsafe mode are very similar, but there are certain things that auto mode will not allow the agent to do, and one of them is just go find a module to use. and that is super-duper important right now. the number of tool chain attacks that are currently in flight are insane. And the security landscape is, it's actually one of the things that,
keeps me up thinking about it. Did we ship something that has a security vulnerability that was intentionally put there so that our stack would be exploitable later? Bret Fisher: it sleeping on my machine? Michael Rollins: And is it sleeping on my machine? Bret Fisher: waiting to take my Michael Rollins: Yeah. but I have these remote sessions running, and I, I'll be sitting… I used to not sit on the couch and watch TV with my wife because I felt like I needed to go be doing something, and I'd have these ideas. I don't love watching TV, but I like being with my wife, and it was like, okay, well let me go sit with her. But then I'd have these ideas, and I'd be like, "I need to go do this."
Well, now I just pull my phone out and I tap a few things on into Claude, and I just put my phone away. And it goes and it tests everything and does the idea, and then it comes back and it's "Okay, yeah, it's ready to ship. Are you ready?" I mean, it's a wild world. Bret Fisher: you can't start a new project from the phone, can you? On Claude Code specifically? Michael Rollins: Yes, you can. Bret Fisher: Okay. Michael Rollins: when you say a new project, what do you mean? Bret Fisher: for example, if I don't have remote enabled on a, on an existing repo that I've cloned, or I wanna start a new code project and I don't have a repo yet or a folder, I can't get it to initiate a local new
session on my Mac remotely, right? I kinda, like you said, you had remote on always, or you have it, I didn't even know you could do this. It's actually a pro tip is, just turn that thing on for everyone, all of them, 'cause I always forget to, to slash enable remote in my sessions. Michael Rollins: All right, so first off, you have to give Claude Code the access to your GitHub repo. If it doesn't have a GitHub repo, you can't use it. Now, if you wanna start a new project, you go create the new repo in the GitHub app, and then you come back to Claude Code, and you give it permission on your phone to just use that, and then it can. And you can start the, you can start it in the, there's like a cloud
version that'll run and make all the Bret Fisher: And you're using, the connections feature that actually Michael Rollins: Yeah, I don't even remember what it's called. Bret Fisher: Yeah, it's an automatic connection to GitHub from the Claude Code web UI. Michael Rollins: Yes, that's correct. And now because of the way our stack works, because I can issue a PR and stand up the copy of production, I actually don't have to have something running on my local, but I can continue it. And so if there's something going on and I'm like, "Huh, I would-- I need to go walk away for two hours, but I wanna make sure this thing progresses,"
I'll just get a notification. I can look at it when I'm able and say, "Okay, go do this, that, and the other." So that is, that is a pro tip. Yeah. you can, if you want, be running through First Landing Park, with 25 pounds of computer equipment on you, and pull your phone out and advance your project again. You know? Bret Fisher: I don't see a lot of, those weight v- exercise weight vests with laptop sleeves in them, but, you know, maybe that's the nerd equivalent. So if there was… we've talked a lot of high-level stuff, and I really, I think this is gonna be really good for people to s- sort of hear a team
that's… I w- I wanna make a point here that I feel like one of your secrets is that you're in tune with the business, which we, you kind of started with that. Your team is aware of the business goals, sales objectives, like in, in terms of what the product needs to be, where we're headed, why does this thing exist, what do our customers look like? Because that allows them more agency and allows them to… Also, there's a metric of judging risk there, because like you said, if you were a healthcare, if you were a fin- fin healthcare company that was, that, you may not have the same posture because you're appropriately dialing up the risk tolerance to the level
of the organization that you're in, and that is different for every organization. So I j- I feel it's like a, feels like it, this bears worth repeating that if you're not… Do not try to shove this round peg in a square hole of, you're in a high safety, you have, you're on a DARPA contract. you probably, yeah, you probably should be doing old school co-pilot code completion with the AI at best, where every line is sweated over and you're just saving keystrokes. Michael Rollins: I've always been the engineer that was connected to the entire organization. It's just my nature.
I like talking to people, and I'm curious about how the business works. However, that is a skill that engineers are gonna have to have in the future, at least to some degree. When we started this, the very first thing we realized was that our infrastructure was the first gate. So we went ahead and we fixed our infrastructure, and we thought, "Great, now we can move forward." But then we broke product. Product could not keep up with us. There were only three engineers at this point. Product could not keep up. Then we broke design. Design is consistently behind in our system, and it's not design's fault.
It's that there's now four other engineers plus our designer who actually commits code, right? Because that's the only way for him to keep s- keep up with us. Then we broke sales, right? I would have to get on sales calls and consistently during the sales call would have one of our sales guys be like, "Yeah, we're gonna have that in, you know, a couple months." And I'll say, "No, it's in the product right now." Right? I have closed deals because it was already in the product and
I was the only one that knew, Bret Fisher: Yeah. Michael Rollins: right? Um, we Bret Fisher: fresh Michael Rollins: Yeah. We, we don't have much marketing, so I don't know if we've broken marketing. I suspect we have. But when you're in a position where the engineering organization is now the fastest organization in the company, the only way for you to move safely as a business is for your engineers to know how the business works, right? So before we ship things, the engineers that work with me know, " Hey, do we have any demos coming up?"
' Cause that's a real question. We're small. We have to ask that question, right? Are there any important things happening today? Because that's something else we need to know. And so these are the questions that engineers typically don't ask, and they're gonna have to. Engineering is different now because the speed at which engineering can progress is effectively putting pressure and force on the rest of the business and breaking it in the process. And I've never seen anything like this, and that's how I know it works. Bret Fisher: Yeah. Y- you're describing this as what I would maybe label as a pro-engineering
culture, where a lot of the messaging we see in the media is around cost optimization of reducing engineering. And what I'm hearing, the the language out of you, which is clearly part of the business culture of your company, isn't looking to save the money by… Because you're so efficient, let's reduce you down to two engineers instead of five, right? Let's get rid of the designer and burden someone else with that workflow. I don't hear that out of you, and that sound- I'm assuming that's pretty deliberate. It's a deliberate strategy to not starve the engineering organization.
Michael Rollins: Well, I mean, to be clear, The idea that what we've built is what we've built with just four or five engineers is absurd. There's no way I could've presided over this thing being built with the number of engineers that we have with the, without these tools. Is AI a cost savings? Only insofar as it is an accelerator to your individual team. AI is not going to be cheap. Boy, have we discovered that recently. Yeah. I'm like, look, man, the minute Claude Max and the $200 equivalent for OpenAI
goes away, small companies like mine are gonna be in a world of hurt. It's too expensive. So, but the option is, but we can't afford to do it any differently either, so Bret Fisher: Yeah. That leads to a question around cost. Do you, um… I have this diagram which I wanna show you real quick. I'm trying to explain in my courses to people that are, you know, not near as, as advanced as you all are yet and, or as experienced. And I'm taking the project management triangle, which was traditionally it was the rule was there's three thing- levers you can pull.
You get to control two, you meaning management, and you don't get to control all three, and that is quality, cost, and time. And so the whole adage was you might be able to, you know, you might be able to reduce the cost and, and resources, but that's gonna cause the time- timeline to stretch. So you move two parts of the triangle, the third one's gonna be affected. And I'm starting to correlate this to what I'm maybe calling like the AI engineer project management triangle, which is a sort of an internal metric that we… Assuming there's not unlimited budget, which some teams are very lucky and they can just token max to the, all day long, that we really kinda get
three new choices we didn't have before. That is model choice, harness choice, and where or when to use AI. Because obviously you could have AI robot, you could put an agent into production and have it Ralph looping every second of the day, constantly looking at stats in production to look for any potential failures, and it will burn thousands of dollars of tokens a day if you let it, right? So not everyone gets to choose max AI in every part of their software life cycle. So if we get to choose model, which effectively not only is a time thing
because Haiku is way faster than Opus, but it's also very much a cost thing. And then which harness choice, which is maybe more around functionality and, ease of use because Claude, I feel like, is so, so far ahead of everyone else that it leads to a lot of new functionality that we don't yet have on some of the other ones. And then when and where to use AI, which some organizations maybe don't even let you put it into production yet for helping with troubleshooting. And then there's these esoteric things of the environmental impact, the privacy impact, and then the political impact of certain things. I have members of my community that are in Turkey, for example, where they're,
relegated to not being able to access all the models, or they have privacy rules where they're not l- allowed to move this data outside their data center, so they have to then be able to somehow house local models that they have to run open weighted or whatever. I was curious if there's anything in here of value that you would think that, my idea has merit in the fact that this is a new paradigm that we've never in the past, I don't recall the last time I had to worry about cost in my daily activities that might blow my budget as an engineering manager. This is a new thing. Like maybe cloud, but that wasn't happening in a day, right?
Michael Rollins: So first off, the most important part of this is the harness. There was a Stanford paper, and the Stanford paper came out and basically said, " You can improve the performance of the model by 6X if the harness is correct." And so harness is really what you need to focus on. And while Claude Code is one of the best harnesses in the world, if not the best, there's still an extraordinary amount of room for optimization. So our models, our harness, so we actually, yes, I code in Claude Code, but in our production environment, we deploy agents.
This is what we sell, is this agent infrastructure. And our agents write code. Now, our agents do not get tool calls. I don't let the LLM touch many tool calls at all. What I do instead is I package, we package the tool calls into an agent SDK, and the only way for the model to touch the tool is through writing Python code. Now, what does that end up doing for us? Well, number one, most models are really, really good at writing Python code.
You can actually deploy small models to write Python code, right? But we realized we have this im- immense cost saving right now that we have not leveraged, and we're going to. Because if our agents are writing code to express themselves constantly, well, I know how to save code. As long as I can find that code again, I can actually go reuse it. I don't have to spend the tokens recreating it, right? And all of this, I've built this Instagram agent, that I've been using for m- my Instagram account, and I actually built my own Instagram integration in Claude
Code, and I moved it into my agent, and I said, "Whenever you do anything with Instagram, go use this code." And so what I found is that it has been using it and improving it daily. Bret Fisher: Mm-hmm. Michael Rollins: And I realized, oh, I think this agent is now using less tokens, ' cause it just goes and runs the code it's already built. And so that's one of those things where it's oh, well, if the harness can be used to make the model better, the harness can also be used to reduce the number of tokens that are necessary to run on a daily basis. And so putting the harness at the center of that thing is actually
the way to look at it, right? The central connector between the cost of the LLM, the time that the LLM takes to run, and whether or not it's safe to run is the harness. And in point of fact, you can even structure your harness such that you can observe the inputs and outputs to the harness without the… ever leaking secrets to the LLM. Bret Fisher: Mm-hmm. Michael Rollins: Because one of the other things we do is we proxy everything in and out of the harness that's running on our remote system.
And since we're proxying it, I can actually inspect the traffic, and I can do that with code. I don't have to do that with an LLM. And- Yeah … it's fast. I know how to do it. I know the things I'm looking for. And so now if I start seeing the harness producing stuff that it should not produce, then I can smack it down. And conversely, I don't have to give the harness all the secrets. I can inspect the traffic on the way out and insert the secrets in flight without ever exposing them to an LLM. So the real the center of that triangle is the harness.
It is the most important thing in what we're doing. Now, are the models, amazing? Yes. But now the models are good enough. Go focus on the harness. Bret Fisher: Yeah. Is that, are you using LiteLLM as your proxy? Michael Rollins: No. Bret Fisher: Do you know? Michael Rollins: No, we rolled our own. Bret Fisher: Oh, okay. Michael Rollins: So in terms of bringing in the LLM, we just bring the LLM in through the, um I don't like LiteLLM. There were at least some significant issues with it previously. We use the LangSmith LangGraph library, and we just make sure that our LLMs conform to that particular structure. Bret Fisher: We've got teams that I'm working with that are trying to figure
out when it comes to local human agent harnesses, that, you know, they're trying to understand usage per person and trying to figure out, give model access as well as maybe inject things at some of the sandboxing tools are also learning how to inject things, in the, on the fly in the HTTPS. so the last question I was gonna see if I could wrap this up with the last question, and I'm gonna show you a graph again 'cause I have another one I spend, I try to spend some of my time… I'm trying to, and I'm really bad at it, so I'm forcing myself to do it. so what I have on screen is something that I'm working on and calling the AI
Engineer Maturity Model, but it comes from a different aspect than every single other one that I see on the internet, and I spend a lot of time looking at what everyone else is talking about, of like, how do I onboard my team to the skill, the human skills they need to really complete the life, the software life cycle of where all the AIs belong, right? It's not just about Claude Code on my desktop. There's lots of other opportunities, and of course, the industry is now starting to figure out that if you speed up the number of PRs you're shipping, but you're not improving the things after that, then you're just gonna have a giant bottleneck in your PR stack, and your production won't actually get faster.
But this screen that I'm looking at is a fictional screen that on it is the idea of over time, as you implement different human workflows or different human skill sets that the team ramps up on, we assume that your overall team productivity improves, and there will be times, like you said, when you first are doing certain activities that team productivity actually declines. Like the day that you implement Claude Code for the whole team and they're all using it, that's a rough week, right? That's a really unique thing you need to give people time to onboard, so there's probably gonna be a lower productivity.
And then there's other points at where I see teams that maybe they've, they fully adopt agent harnesses locally, but they really haven't done much in CI yet. They're barely even doing PR review agents. They maybe not even using that yet. And they realize now that they have this CI backlog, so I would argue that becomes a team productivity dip because now they're spending all their time staring at code that AI built, and they haven't really kind of leapfrogged to the next level. So I was looking at this like imaginary list of things you can do in a team to improve the overall productivity of your software deployments.
Do you see like activities or certain things that sort of rise up as the obvious things besides just using Claude Code on your desktop? Is there activities that you think are a big tell on whether the team is mature or not in their AI, s- software life cycle? Michael Rollins: Um, I hadn't thought much about it. Uh, bec- partially because I haven't encountered teams that are doing it the way we're doing it. Um, I b- I promise you, I've had a lot of people look at me directly and tell me I'm insane. They're like, "This is insane. This isn't real software engineering." And then they come back, they're like,
"Oh." Here's what I can tell you. The end of that graph looked like this gentle line that just went up. Yeah. And there's a point in there where that gentle line just becomes vertical And that's the point where you do have agents fixing things without human input. The big milestones for us were we started from, you know, everybody's using AI to code. We moved into, " Hey, "When's the last time any of y'all actually looked at the code?" And all my team was like, like, everybody didn't wanna get caught.
I was like, "Oh, 'cause I haven't looked at it in months." Yeah, and they were like, " Oh, God. Okay. Phew. Thank you. Thank you." Bret Fisher: They, They didn't know where you were going with it, right? Like, you were, you yeah. yeah. Oh, by the way, imposter syndrome that jumps out when the junior-est member of the team posts something in the engineering Slack that calls out something the most advanced engineer's done, and you're like, " Oh, God, I'm not even a real engineer." Like, how… And then you're like, "Oh, wait, you had Claude look at that, didn't you?" And they're like, "Actually, no. It was Codex." You know? But it is everybody using AI full time?
The next step is when was the last time anybody read code? Not just write it, read it. Do you have full CI/CD yet? And then beyond that, it's as simple as, you know, I'm sure that you could throw some milestones in there, you know, what's your… How fast is it to sh- to fix a bug, right? Because in lieu of a perfect release, how fast can you fix the release if it goes sideways? By the way, you did ask one question: Do you have a single button rollback? We do, but it's only good for 10 minutes. Okay? So we gotta be real careful.
but then finally is can the agents just do your job for you, and under what constraint? And that is when I think you're at full AI maturity, that when it is safe enough to let agents move in your system without getting human interaction, you are now scaling yourself infinitely, because every step change that you add to those autonomous agents is an infinite scaler for the rest of the team. One thing I will point out, though, and we can close on this, is when you make mistakes in this style of engineering, they're big.
I'm not talking about shipping bugs, okay? I'm talking about a misstep in the architecture. I'm talking about a misstep in, holy shit, how did we manage to ship that migration order on the database? When you make a mistake like that in this style of engineering, it's not a self-wound, it is a decapitation. And so every engineering team will hit the point where they're like, " Oh my God, what have we done?" And we've done it twice.
And both times we ended up having to regress the system by two weeks, which doesn't sound like a lot until you think about the fact that we have commits that have been 60 commits behind. We've had PRs 60 commits behind because they were five days old. Michael Rollins: So the cost of the catastrophic failures is higher, and that's the one thing you do have to think through, so Bret Fisher: Do you think about that in terms of are there different tiers of risk in the way you approach AI in different repos or different parts of a repo where you're like, "Well, this one's dealing with the database migration, so
we have extra safety involved with that. Michael Rollins: Yeah, so we do have-- We use an owner file. we need to be using an owner file. I just keep forgetting to put it in. But there are very specific files that if we- Say again? Bret Fisher: Get confession hour. This is confession Michael Rollins: Yeah. Oh, believe me, we're not perfect. Everything I've told you has been relatively true, and maybe I've waved my hands once or twice, right? but the-- Yeah, we need to put the owner file in. If our system prompt changes, I need to know about it. If our list of tools changes, I need to know about it. If our, look, we ship multiple database migrations in almost every single PR.
I mean, I cannot explain how many times we have metamorphized the database, in just a few days. but there are files that if they change, I want to know about it. And by the way, one of those is going to be, like, what are your imports? What node modules are you bringing in? What Python libraries are you bringing in? 'Cause those are important as well. Bret Fisher: Yeah. I can see how that would trigger different, additional safety checks in GitHub Actions where… I mean, I do at times make directory-scoped actions that are… It's also really good just for any action.
This is a pro tip with GitHub Actions is a lot of times your actions that run all the time don't need to run on s- they only need to run on certain types of changes. And GitHub Actions, turns out they c- actually have a lot of filters involved so that you can say, "Hey, don't run the action linter unless the action file's changed," and that's only in this one directory, so there's no even reason to even fire it off unless it hits Michael Rollins: I probably could, if I probably paid you for an afternoon, we I could get so many more efficiencies out of our system, dude. It's insane. Look, I'm not DevOps, I'm not infrastructure, Bret Fisher: Yeah, at this point I'm just a skill.
I'm just a skill. I just have a GitHub Actions workflow skill. My new rule is every new conversation I start with an AI needs a skill. If it's a one-off question, fine. But if I'm planning on this gonna take an hour or two, we're gonna have a conversation about it, I'm looking for a skill to give that agent to enhance its ability, because I personally have noticed on my projects in the last two months that the skill in every case has always improved the outcomes, or at least got me faster to the outcome, whether it's making sure the skill is teaching it the latest version of Go, making sure that it's adhering to the latest Go
patterns, or whether it's the front-end design skill that everybody's using, or whether it's a GitHub Action skill that enforces certain linters and certain security patterns that I behave with. I just feel like there's always a good, good reason to have at least one skill in your, conversations. Pro tip. Michael Rollins: Yeah. Bret Fisher: Hey, man. So what's the one thing you're gonna excited to do next? What have you not done that's, on your to-do list of, I wanna try this out"? Michael Rollins: Turn agents loose. Get them to fix things end to end without me ever knowing. Bret Fisher: Continual self-improvement? Michael Rollins: To be able to have an agent pick up a report from Slack, figure out if it's real, create the issue, fix the issue, commit the code,
build the PR, test everything, merge it to main, and ship it, and I never know. When that starts happening, our engineering team of five will look like an engineering team of 80. Bret Fisher: Check out agentic workflows in GitHub if you haven't seen that yet. it's a-- That's the actual term, agentic workflows, because it writes… it's like a full safety version of putting an AI prompt in a workflow without all the risky behaviors that everybody's using right now. There's like sandboxing, there's all these different extractions. And one of the things that it can do is you can put it on like a, kinda like a Ralph loop. According to what I heard yesterday, was they designed their system so that it's
not all Monday at 9:00 AM, and like they, they, they distribute these things out. It has inherent, rules inside it, so if you ask it to be an issue reviewer, it won't do all the issues right now. Like it's not gonna l- overload you with all of this work, which is what a lot of people I think do, and initially the Michael Rollins: I will Bret Fisher: them a bunch of work. Michael Rollins: I will say this, if Copilot is my only choice, I won't use it. Bret Fisher: Yeah. Well, none of this has to do with Copilot. You Michael Rollins: Okay. good. 'Cause this, Bret Fisher: All of this with Claude Code. Yeah, you pick your own agent. Michael Rollins: Okay. Bret Fisher: framework is agent agnostic, Michael Rollins: Excellent. Then yes, I may try it.
So Bret Fisher: yeah, I think the, the push button features in GitHub are all Copilot, right? You turn on the PR review in your settings. that's, th- these are just, features that happen to use Copilot. This is more of a hobby kit approach, where you're at the command line with this stuff and using YAML files. I'll show it to you someday if you don't ever get a chance. Mike, thanks so much for being here. Can you give us your full name, and title, and company? Michael Rollins: Mike Rollins, CTO of Relify and Family Digital. You can also find me at rollins.io online. Instagram is rollins.io, and uh, I think that's the major points there
Bret Fisher: You talk about AI on Instagram, so if people wanna hear more thoughts from you, I would recommend Instagram, Awesome, man. Well, thank you so much for being here, and we'll, I'm gonna definitely have you back on 'cause we're gonna talk about you pushing the envelope even further, so Michael Rollins: Let's do it. Bret AI July 2025: Thanks for joining us, and I'll see you in the next episode.
The OpenAI Hugging Face hack. The Anthropic testing failures. Listen to the security and ops experts, not lab researchers. I walk you through the production failures that allowed OpenAI’s models to access the internet,...
Luke Hinds, co-founder of nolabs Inc. and the nono project, joins me to dig into AI agent sandboxing with the nono CLI. It's been my go-to sandboxing tool for local agent use, but I know there are more features I should...
A deep dive into GitHub’s Agentic Workflows feature, with Don Syme of GitHub Next & Peli de Halleux of Microsoft Research. Check out the video podcast version here: https://youtu.be/zmM8VISTOwo 😇 My new GitHub Security...
Bret’s joined by the Anyshift.io co-founders, Roxane Fischer and Stephane Jourdan to discuss how an always-on SRE agent can help you proactively find risks and avoid incidents. 😇 My new GitHub Security workshop has...
Sam Alba, co-founder of Mendral, joins the show to discuss their new AI agents that act as “junior devops engineers” against GitHub Actions, security, failed tests, and more. Check out the video podcast version here:...