Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Code with Jason

Code with Jason artwork

327 - Will Carey, CTO of Brand New Box

Sep 5, 202656 minAudio

In this episode I talk with Will Carey, CTO of Brand New Box, about evolving software development practices, the shifting role of AI in coding, and the importance of understanding client needs. We also dive into Will's unique code editor project, Reins, and the importance of observability in development. Links: - Will Carey on Twitter - Brand New Box - Reins on GitHub - Nonsense Monthly

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_01Hey, it's Jason, host of the Code with Jason podcast. You're a developer. You like to listen to podcasts. You're listening to one right now. Maybe you like to read blogs and subscribe to email newsletters and stuff like that. Keep in touch. Um email newsletters are a really nice way to keep on top of what's going on in the programming world. Um, except they're actually not. I don't know about you, but the last thing that I want to do after a long day of staring at the screen is sit there and stare at the screen some more.

SPEAKER_01That's why I started a different kind of newsletter. It's a snail mail programming newsletter. That's right. I send an actual envelope in the mail containing a paper newsletter that you can hold in your hands. You can read it on your living room couch, at your kitchen table, in your bed, or in someone else's bed. And when they say, What are you doing in my bed? You can say, I'm reading Jason's newsletter. What does it look like?

SPEAKER_01You might wonder what you might find in this snail mail programming newsletter. You can read about all kinds of programming topics like object-oriented programming, testing, DevOps, AI. Most of it's pretty technology agnostic. You can also read about other non-programming topics like philosophy, evolutionary theory, business, marketing, economics, psychology, music, cooking, history, geology, language, culture, robotics, and farming.

SPEAKER_01The name of the newsletter is Nonsense Monthly. Here's what some of my readers are saying about it. Helmut Kobler from Los Angeles says, Thanks much for sending the newsletter. I got it about a week ago and read it on my sofa. It was a totally different experience than reading it on my computer or iPad. It felt more relaxed, more meaningful, something special and out of the ordinary.

SPEAKER_01I'm sure that's what you were going for. So just wanted to let you know that you succeeded. Looking forward to more. Drew Bragg from Philadelphia says, Nonsense Monthly is the only newsletter I deliberately set aside time to read. I read a lot of great newsletters, but there's just something about receiving a piece of mail, physically opening it, and sitting down to read it on paper that is just so awesome.

SPEAKER_01Feels like a lost luxury. Chris Sonnier from Dickinson, Texas says, just finished reading my first nonsense monthly snail mail newsletter and truly enjoyed it. Something about holding a physical piece of paper that just feels good. Thank you for this. Can't wait for the next one. Dear listener, if you would like to get letters in the mail from yours truly every month, you can go sign up at nonsense monthly dot com.

SPEAKER_01That's nonsensemonthly.com. I'll say it one more time. Nonsense monthly dot com. And now without further ado, here is today's episode.

SPEAKER_01Hey, today I'm here with Will Carey, CTO of Brand New Box. Will, welcome.

SPEAKER_00Hi, thanks for having me. Good to see you again, Jason.

SPEAKER_01Good to see you too. For those who might not have caught our first episode, tell us a little bit about yourself.

SPEAKER_00So I am uh the CTO, as you said, for Brand New Box. We're a web consulting design development agency. Um and we build products in mostly Ruby on Rails. We also do some mobile app development. And uh we we kind of pride ourselves on partnering with people and helping them figure out the software that they need, uh, which feels like it's becoming more and more relevant as uh people people need more help with that than they need delivering the software.

SPEAKER_00Uh which which we also we're happy to provide that as well. But um it just as as the uh industry is shifting along, we are shifting with it.

SPEAKER_01Yeah, yeah, that's really interesting. So last time you and I talked was like kind of before this this AI wave, or maybe at the start of it, or something like that. We were certainly in a different place. Right. I I remember I first met Nate, and I think you were there too. We we had a three-way conversation together at Rails Conf in Detroit, yeah. And I was really impressed with both you guys, and something I really appreciated about the the approach that you seem to take is this like holistic view of understanding everything, in including like the customer's world and stuff like that.

SPEAKER_01I'm extrapolating a little bit, but no, that's all right.

SPEAKER_00That's all right. Yeah, you're doing a great job of selling us.

SPEAKER_01So well, when I used to do consulting, I was I would always encourage my clients who like were normally people who either ran software businesses or were like CTOs of small software businesses, that kind of thing. I would encourage them to like visit their customers in person, understand their customers' world, uh not just listen to what their customers ask for, but actually gain a deep understanding of again the world they live in.

SPEAKER_00Yeah. I I think one of one of the things that I enjoy the most about brand new boxes, like I don't I don't think we have a client where we have not like in some way shaped the way that they actually do their business. Um and and we and it's literally just from we start by listening to them. Yes, we also have like in-person workshops and and things like that, but um by the end of our projects, I've I feel like there's we usually understand like the process of what goes into it like better than they do.

SPEAKER_00And so like we and we have these long engagements, like sometimes lasting literally 10 years. Um and they by the end of it, like the client will be like asking us like, can we do this type of stuff? Like, is this something that we could do in our business? And like we we are like we have become the subject matter experts on like can they do this or not?

SPEAKER_01What do you mean by that?

SPEAKER_00It's so it's um it's usually things that like combine like a piece of their business and technology. So we did a lot of like physical, uh we do a lot of work in the AEC space, which is like architectural engineering stuff. And our clients would go out and assess buildings. And so by the end of it, they're they're asking stuff like, okay, we we have an app where we can go out and assess a building and figure out what's wrong with it. Now, can we uh like how would it work if we tried to like do 3D mapping in here?

SPEAKER_00And like we we kind of just like sit down and talk through them with A, the technology parts, but more often than not, it's not really the technology that's the problem. It's like how are you going to like what's gonna be your process for QAing this? What's gonna be your process for presenting it back to your clients? What's gonna be your process for uh like what you do with this data? Like we can be we don't want to build something that looks cool but is not usable. Um and so we we tend to like have a lot of conversations in that direction of is this actually going to be something that brings value to your business uh or is it just something that's kind of like flashy and looks cool?

SPEAKER_00And and may maybe even those are like not as bad anymore because we can whip those out so fast with AI. Of like, yeah, we can also make something that looks cool if you want, but we would prefer to build something that like is actually functional and usable for your business.

SPEAKER_01Yeah, I've spent a lot of years working inside of agencies, working with agencies, and doing freelancing and stuff like that. And there's there's kind of a spectrum of sophistication of the vendors. Um, you know, uh on one end there's just like the order takers. Uh like you describe what you want, and we'll try to interpret that, and we'll uh make a list of requirements and and then try to fulfill it, and we hand it to you, and hopefully it is useful.

SPEAKER_00Right.

SPEAKER_01And then at the other end, there's people like you, um, which I think is is a much uh more more savvy from from a business perspective approach because um usually the the order taker style of agency model like it doesn't work out great because there's so much room for things to go wrong. Um you know communicating an idea from one mind to another is way harder than it seems like it should be.

SPEAKER_01Um but it but it is. And then even if the communication happens perfectly, the idea that the client has, even if the client might be a very smart, sophisticated person, it it m probably isn't really the right idea. It probably needs some working through in order to get to a place where it's viable.

SPEAKER_00Yep. Yep. That that is that is what we have found mo more often than not. Um and and to go back, like it again, it does just feel like a very good place to be in. Like I I have told my team, like the value of the code that you write right now is at an all-time high. Like it is going it is only going down from here. Like the value of like individual lines of code is only going down from here. And like what is going to win the day is like the ability to ship the right thing or to deliver the right thing to the client.

SPEAKER_00Um, and and those those decisions that you make in the code and those decisions that you do, but like the actual code that you write is like like we we hit the peak, it's going down.

SPEAKER_01Right. Yeah, that's true. Um, yeah, like if you are the kind of person who is you know just an order taker, um that is is not very valuable. Um there used to be you know, there there's the idea of like hiring somebody on Upwork. I I I've I've come into contact with this style of working where it's basically like you make your specifications in as much detail as you can, um, try to do all the design up front, then throw it over the wall, as they say, to some contractor and have them implement it.

SPEAKER_01Basically, it's like a mechanical Turk version of AI, you know. Um that's like there's like no value in that anymore. Because like, why would you do that? Why would you give that work to a human who costs, you know, maybe it's an offshore person who costs 20 bucks an hour or something like that. Why would you do that and probably get something that only kind of half works when you can give it to an AI almost for free and get something that's probably even better?

SPEAKER_01So there's like a there's a line.

SPEAKER_00It might yeah, it might still only half work, but you probably got it in five minutes instead of ten hours or whatever it was gonna be.

SPEAKER_01Right, yeah, and and the half-working part, like that's that's only getting better. Like something that would only half work six months ago, you can give the same thing to Claud Code now, and it'll like 80% or 90% work. And I I realized something that made me uh well it made me pause. I a few weeks ago or something, I'm like, there are already developers who I would trust less than AI. Like if I had a uh if I knew somebody who was non-technical and they're like, hey, I got a limited budget, I need to get this thing done.

SPEAKER_01Should I like go hire somebody on Upwork or should I use AI? I'd be like, use AI for sure.

SPEAKER_00Yeah, right. And it it's it is so hard because like there are I feel like it's a it's a fairly narrow scope of things the AI is incredibly good at, and one of those things is software development. And like me as a software development I developer, I see every problem as like with that hammer that I can smash with that, uh, or as that nail that I can smash with this hammer, um, which which I think is very helpful, but like I tried I try to get my wife to use it for stuff, and like she she finds some success, but it's like the type of stuff that she's doing isn't as uh like like she she's uh she's a landscaper, and so she's trying to do like garden designs with it.

SPEAKER_00And so she can do like image generation, but as far as like making decisions about the plants to put in a garden, it's it's not as helpful yet.

SPEAKER_01Yeah, interesting. Um yeah, of course the people building the AIs are themselves software developers, so naturally they're gonna be a lot better at making AI good at that than making it good at other things. Um but back to what you said about the value of lines of code being at its peak right now, I I think that's really interesting, and I think that's true. The the value of that is going down.

SPEAKER_00And I've and and I I would say like the corollary I give with that is like we're at a peak. The value of lines of code right now is still high. Like that's what and that's why like we still we still do code reviews. Um I I also tell my engineers, like, when you send me a PR, I don't want to be the first person that has reviewed this code. Like you need to review it yourself, even if it's written with AI, obviously. Um and so like right like at this current moment the value of lines of code is still high, but it is in six months, I think it will be lower.

SPEAKER_00And six months after that, I think it will be lower. Like we're on a downward trend.

SPEAKER_01Well, let's get into that a little bit because I think there's some different layers there. Um so I I for a long time have viewed programming not as writing code exactly. Obviously, code is like the medium that we use. Um, but the the act that we're doing is designing and building software systems. Right. And software systems, even though the material they're made out of is code, they're really made out of like concepts.

SPEAKER_01And uh uh what we're what we're continuing to do now, even when generating lines of code is cheap, is we're like orchestrating at a high level. Um like quick quick example, uh Saturn CI, my CI product that I've been working on. Um I I designed the UI in a certain way, and I I gave the concepts you know, you can slice up the world into in different ways and into different concepts, and I chose to slice it up in in one way instead of another way.

SPEAKER_01Um, and I, for example, there are test suite runs, and there are um inside of test suite runs, you can parallelize the test suite runs into what I call tasks. I could have chosen to model it a different way, different CI products model it a different way. They might call it a build, but a build isn't exactly the same concept as a test suite run. So these are like arbitrary modeling choices that I'm making, and the code's gonna come out different.

SPEAKER_01It's a different system than if I had modeled it a different way. So that part is still like uh I that part is still like we're still doing that work.

SPEAKER_00Yeah, yeah, of course. And and I think that there is a um I think there's an aspect where like clear thinking around a problem kind of leads to like clearer code and clear or or even even the opposite of like clearer code kind of leads to clear thinking around a problem. Like a hundred percent sometimes in the part of figuring out how something should be built is actually sitting down to build it, and that implementation kind of drives your thinking around it. I th I think what we have the advantage of today is you can you can model that as builds on your first pass, and and maybe you realize that's the wrong abstraction, and you can very quickly pivot then to tasks.

SPEAKER_00That's exactly what I did actually. Okay, yeah, there you go. Uh like probably a manner of like one or two prompts or something like that. It's just like uh oh, I'm I'm see I'm already seeing how this is being built out and like the growing pains of this modeling system and how it's leading to like messiness downstream in the actual user experience. And so I need to I need to pivot. Um and so because of that, like I'm it's maybe a a little bit of a ironic thing, but so yeah, I'm like less scared to make a mistake because like I'm I'm like actually more apt to get into an implementation because like I want to see the downstream effects of that.

SPEAKER_00Um so that I can like quickly pivot to something else if I need to. And that's this is actually one of the main parts that I struggle with with our like current iteration of AI products and current like the way that people talk about them is like hey, you you make a plan that encompasses like your entire implementation, and then you you implement that plan step by step. And I'm like, ah, it's usually when I'm implementing that plan step by step, I I find like eight ways that the plan should change.

SPEAKER_00And and of course, like you can kind of find ways to to do that, but it's like I feel like we've kind of settled on this place where we're almost reinventing waterfall development in the way that we build with AI.

SPEAKER_01Exactly. Yeah, I've had that exact same discomfort, and I don't want to like prematurely dismiss it, because like maybe there's something there that I'm just not understanding somehow, but uh at first glance it it sure seems like uh very very waterfally, because where's the feedback, you know? Right. Um, I think feedback is one of the most important uh ideas in programming because there's that saying plans never survive contact with reality.

SPEAKER_01You implement something and then you get feedback on it. It's like, oh yeah, like you you try something on for size and it it doesn't feel quite right, and you're like, huh, what is off about this? Like, I called this a build, but like why is it a build? I'm not like generating uh Docker images for production or something like that. I'm just running some tests, like this isn't really a build, and it took me like living with that for a little bit in order to come to that realization.

SPEAKER_01And if you design the whole thing up front, you don't have that uh that that feedback.

SPEAKER_00And I again I think yeah, like like you said, I'm like I'm also in that phase of like this is how I work with the AI as I try to build it a plan. And I am trying to find ways to like still bring the pain points back to me, because usually in the old world, uh I would as I would code, I think I would feel those pain points, because it's just like ah, I'm this this is like a very leaky abstraction. I'm pulling this code apart and put putting it everywhere, and like that's then a very clear signal to like this being bad.

SPEAKER_00You don't feel that as much when you didn't actually write the code. Um and so you have to like I think try to find other ways to push back against you. Um and that I don't know, I feel like that's like the long journey that I'm on right now. It's like what are the right ways to get to get pushback? Or what are the right ways to like figure out uh when something is not working very well, uh other than just I mean I I still like review my code and like other than just looking at it and just seeing, uh this kind of feels smelly, or this kind of feels not right, or this this breaks whatever whatever set of internal rules that I have at that moment uh because it's before lunch and I feel this way or something like that.

SPEAKER_01To me, the ultimate test of of how good a piece of code is or whatever is when I go and try to change it, um, which which is not really so possible like right after I write it. Um if I write a piece of code and it's just Just like awful. Like, I can tell, and and I can be like, okay, this is just like this should obviously be changed in this certain way, but beyond that, it's not always clear, and then I might wait until you know a few weeks later you need to make some change to it.

SPEAKER_01Uh, and then it's like, oh, okay, this is like really not easy to change, so I'm gonna have to here here's where the obstacle is. I'm gonna have to do something about that in order to make this easy to change. And I found that that uh workflow has kind of stayed about the same with AI. Like I have it generate something, and I pay pretty close attention, like I I I lay eyes on all the code at least.

SPEAKER_01But I certainly like a surprising amount of junk still slips through as careful as I try to be. And then it's it's not until later when I like try to change something where I'm like, what? Like I let this be like this? Okay, well, well that needs to change now. But usually trying to change it is like the the trigger for me. Right.

SPEAKER_00Yeah, have you uh have you worked on many like greenfield projects with AI exclusively?

SPEAKER_01Yeah.

SPEAKER_00Okay, okay. Uh have you hit like the six-week mark yet on anything?

SPEAKER_01Um I couldn't really say that because I haven't worked on something like full-time for six weeks, that that kind of thing.

SPEAKER_00I boy, I thought I was like immune to this. Uh people have talked about it online of just like it's really fun to build out greenfield projects with AI because of like how quickly you can. I mean, it's it's the thing that we've always had from Rails, which is like how quickly you can go from idea to uh something on the paper that you can actually use. And I think AI just accelerates that quite a bit. Um to where you can you can build out a large swath of functionality. And so I I started um boy, I I've got all I've got a whole like side quest that we can go on if you want of a story of uh uh I built my own coding editor, um, except it's actually it's not an editor, uh because that's how I work with AI, is like I don't I don't touch any of the code.

SPEAKER_00Um but anyway, so I start I started on this project, it's called Rains. Um and I was like, I'm I'm I'm a competent software developer. I can I can build this from scratch and like I can follow good practices, I can review all the code all the way through, and like I'm not gonna run into this six-week problem where like suddenly the system just kind of become you tried that's like when you try to change your first big thing and it suddenly becomes unmaintainable. Um and I got to about six weeks in, and it was uh like a week and a half ago when Anthropic cut off third-party harnesses um from using their subscription, which is what my product was.

SPEAKER_00And uh, I mean it wasn't a product, it was a it was a personal tool, which is why I thought I was okay. I was like in this like weird gray area of their terms of service of like you can use your subscription with personal tools, and they they finally like clarified that a bit more and was like, no, you can only use it through this specific way. So, anyways, that was about six weeks into my project, and I suddenly had to uh refactor the entire like agent loop system inside of it. And I that it it almost killed the project for me.

SPEAKER_00Like I had so much excitement for it, and then like suddenly I I mean through external forces, like I was forced to make this change. And I I feel like that could happen from like user pressure or uh just going a different direction, but whatever reason, like you need to make a big change in your code base, exactly like you were saying. Like, this is when you this is when you learned if that code was good or not. And it like it the whole process almost killed me. It was a I landed the PR a couple days ago.

SPEAKER_00It was 9,000 lines of code edition and like 3,000 lines removed. Like it was it was gigantic. And like I I think what I found out is like it some of the stuff did work. Like I just I had explicitly not architected for the system because I didn't think I was gonna have to do it. Um But it was I was like, holy smokes, this is like the big change at the six-week mark that people talk about, where it's like, oh, suddenly you're developing a product that has users, you don't want to break, you can't just like willy-nilly throw things at the wall.

SPEAKER_00Uh and that like that is the hard, I feel like the hard part about agentic engineering right now is like how do you how do you correctly build something in in a way that it won't have that issue, have those growing pains. And and that might just be like still the unsolved problem that we've always had in all of software development. Um, and this maybe just like acutely accelerates that because like you can get to that point faster where you're gonna change something. Um But I don't know.

SPEAKER_00At the same time, I did get through it and I'm like still building that product, so it is in some way a win, I guess.

SPEAKER_01Yeah. Um, you know, the the the the hard part about building software is building something that doesn't collapse under the weight of its own complexity. Yeah. And with AI you can get there a lot quicker. Um I I have had very similar experiences. Um so over the years I've done some experiments with um genetic programming and uh virtual life in various languages.

SPEAKER_01Um historically usually Python, but lately I've switched to Rust because it's a lot faster. Um I don't know Rust, and so I've been doing all the programming through Claude Code. Um and I found it very easy to get something working and then tie myself into knots and not be able to progress any further after that. And so I'm like, alright, I g I gotta just start over. And I had the luxury with these of their throwaway projects that don't matter for anything, and so I can just delete it and start from scratch.

SPEAKER_01Um but I I don't want to just delete it and start from scratch every time, so I um started figuring out how to get the AI better at doing TDD. Um because with the main Rails app that I work on, Saturn CI, it it has a test suite of what I think are pretty good tests, um, and so it can so it it can mimic those tests and kind of extend it and write tests the way that I would, and and I'm happy with that.

SPEAKER_01But with Rust, it it writes tests in the way that it it does, just out of the box, which is unfortunately really really bad. Most of the tests are just like tautological restatements uh like put in five and assert that it's five. It's like alright, well that doesn't accomplish anything. Yeah. Um so I talked about this just one podcast episode ago, but I I introduced mutation testing, which I had no experience with before that.

SPEAKER_01Yeah. And so it's like write a test and then like mess up the code and validate your tests and see if there are like missing cases that you haven't tested yet and stuff like that. And here's how you review your own tests and look for weaknesses. And I had the agent like spawn uh sub-agents so that they could they could be like adversarial with each other, right? Uh because otherwise it's like, okay, let me review my own test.

SPEAKER_01Hey, what do you know? I I did pretty good. Um and and that's not what I want. So once I did all that, things went a bit better. You know, it wasn't like a super duper night and day difference, but it was a lot better, and I got further before I hit that point of everything falling apart.

SPEAKER_00And before before that point, were you doing were you still doing TDD? Like were you still writing your tests first?

SPEAKER_01Yeah, but like it was kind of a joke.

SPEAKER_00Okay, okay.

SPEAKER_01Like, because the tests weren't very good.

SPEAKER_00Okay, I feel like I might actually be at that point where like I kind of feel like my tests are a joke. I write them first, but they might be a joke. Um And this is the this is something I've been like trying to figure out with my team a lot of in order for it to like write good tests, it needs a good mental model. But the problem is like it will try to encode whatever mental model it has into the tests, whether it is right or not. Um and I think the the way that like I see that play out a lot is like it it heavily relies on mocks um or like doubles in testing, and it heavily and it'll just it will basically set up the world according to its mental model, regardless of like what actually is happening in reality.

SPEAKER_00Um and like and and so I've that's what I I've been working on techniques of like can I strip more of that out? Like that and and that feels like the again like the real like software engineering part of like can I get it to exercise the actual code, which means I need to make sure that that code that it's exercising is unit testable and is not just like only accessible via mocks. Um but like that's that's where I'm like running into the most issues is it it puts it puts this mental model into the test and says, yes, they now pass, I'm gonna go do implementation.

SPEAKER_00And then it writes the implementation, it's like, well, actually, you're not really testing the implementation at all because you've mocked out a bunch of stuff. Um I mean I guess actually. In that world, like they they actually are like I'm just trying to do a red-green uh setup. So it's like they they didn't pass to begin with, and then it wrote something and now they pass, but like I don't know, it's asserting like this constant exists or something like that, or this class exists, or or something like that.

SPEAKER_01Yeah, something sad about testing is that I think most developers don't do it, and then most developers who do do it do it in a fairly perfunctory way. They're just doing it to check the box and be able to say, yes, this PR has tests. Maybe there's even a literal checkbox that says this PR has tests. It's like, okay, technically the tests exist, so I can click this checkbox and meet the requirements, so I can get it merged and go on my way.

SPEAKER_01Um and and neither the author of the code nor the person who is reviewing it really understands um what requirements the tests should be meeting, what makes a meaningful test versus a non-meaningful test. Um what I have been fairly successful at teaching the agent is is how to frame the tests. So I I I gave it a TDD skill and I said, here's here's how it works.

SPEAKER_01I'm gonna give you a specification, um, and I tell it, tests are executable specifications. So I'm gonna give you a specification, and then you repeat back to me your understanding of the specification in your own words, and I'll say, yes, you have that right or not.

SPEAKER_00Okay.

SPEAKER_01And then after that, I want you to come up with test scenarios. Like um when okay, here's here's an example that I was that I was writing about yesterday. Um Prime Factorization. So when I here's here's a scenario. When I put in the number 42, what you give me back is uh I think two times three times seven. Um and and then maybe there's another scenario of like when I put in negative five, it gives back like error or something like that, because you can't take the square root of a negative number.

SPEAKER_01Um, and and it says, okay, here's what I think the scenarios are. Does that look good? Um so we're we're starting still at like a high conceptual level. We're not we're not talking about code yet.

SPEAKER_00No lines of code written. Yeah.

SPEAKER_01Yeah. And it's like, okay, yeah, those scenarios make sense. Now encode these scenarios in tests one at a time, and it'll write a test that says, like, okay, when when we put in 42, the output is two times three times seven. Does this test look good? And I'll be like, sure, go ahead. It'll run it, it fails, and then it'll write the code to make it pass, and then move on to the next scenario. So I I I found that to to work pretty well.

SPEAKER_00Okay, that I mean, I I feel like this this is why I came on here. I I needed to like talk to you as an expert in this to understand how my process is wrong. And probably my process is wrong, is like like you said, testing is like a step in my in my plan that I'm going to implement with the AI agent. And I make it the first step, uh, but it's like I don't have as much of a conversation around it as it sounds like you're having. And that's that is probably probably my issue. Um, that's almost certainly my issue with it, uh, and why it's like coming up with bad tests.

SPEAKER_00Um I mean, I I will say I think just like moving them to the front instead of the back end of implementation clears up a lot of implementation details for it. Um and I and it like then it at least has like something to check itself against. Um that's that's like how I'm almost thinking about a lot of this stuff now, is like just what what is the way to provide some sort of ground truth to these systems?

SPEAKER_00Because they like you're saying, like they build you like you're you're sitting there having a conversation trying to make sure that you're on the same page and you guys have the same you guys. Um you you and the AI uh have the same mental model of what you're about to build. Um because yeah, if if it doesn't, like it it will just go and implement whatever its own mental model is. Um and so yeah, I'm I'm I'm trying to like shift my thinking a little bit in that direction of like okay, a conversation is one like what you're saying is like a conversation is one way to establish that ground truth, like shared mental model.

SPEAKER_00Are there um are there other ways to establish that? Um some of the success that I'm getting is like I I was I was integrating like a third-party um API the other day, like connect connecting to a third-party API. And it I I think I had it like do an implementation first just based on API docs, and it was it was weird and it was like flaky in in odd ways. Um and so I was like, okay, we gotta we gotta back up here, and I had it I had it actually just like write a script which like instrumented the way that I was gonna call this API and log every response.

SPEAKER_00And then like that trace became like our shared understanding. That like that trace became the backstop for ever all of its implementation. And like that after that it one-shot it. Like it was amazing. I was like, oh, this uh like okay, I I really need to start thinking about this more of like how do how do I provide that like that ground truth and and get on the same page as the AI agent.

SPEAKER_01Huh. Well why do you think that made the difference?

SPEAKER_00I before that it was like it it could see from the documentation the types of the API, but it was like an event stream type of thing. And it didn't what it didn't have from that API documentation was like the exact order that things would come in. And so it was making all it was like more or less like an event sourced um thing that it was reading from where it was like trying to build up an aggregate over time of like what the data represented, and like at the various checkpoints in its first implementation, it was just like weirdly wrong, or uh some some part of it would be left hanging and never really closed.

SPEAKER_00Like there were just a lot of like weird bugs. Um, and I was I honestly the first one I was like, oh, this is I was so like downtrodden because I was like, I'm gonna have to go in and like trick and figure out like where all these bugs are, and it's just flaky in so many different ways. And and then having the idea of like, oh no, actually, just like run the code the same way you are in the code base, but this time log all the events, and then let's redo the implementation off of those logged events.

SPEAKER_00Um, and I think that that feedback cycle or that ability to really see like not what I think is happening, but what is actually happening here gave it all the tools that it needed to just like uh polish and like fix its own implementation.

SPEAKER_01Um yeah, yeah, I've been thinking more and more like that if I understand what you're saying. Um I and I think this mindset is like good independently of of AI. It's just a a good way to work, I think. Like an example from my day job is there is this program that will ingest uh GitLab pipelines and like aggregate all this data from GitLab pipelines.

SPEAKER_01If anybody's not familiar with GitLab pipelines, think like GitHub Actions, but the GitLab version. Um but there's a problem, like sometimes the statuses were missing. Some of the pipelines would have statuses and some of them wouldn't, and it was really not clear like why the ones that were missing the statuses were missing the statuses. And I also realized that like the at the time you ingest the GitLab pipeline, it might not have a well it might be have like a running status, and then 10 minutes later, by the time you look at it, it might have a status of success or or whatever.

SPEAKER_01And so I I we needed like more clarity around that. So rather than like manually looking into it and and stuff like that, I said let me make a new model um called like GitLab Pipeline snapshot, and every time we call the GitLab API and try to ingest the data from there, let's take a snapshot of that, so that if there's if if there's a a a GitLab pipeline with a missing status, then I can click on that one and the like show page for that pipeline will show me the log of the ingestion attempts and it's like alright, when we first tried to ingest it, this was the status.

SPEAKER_01Then when we hit it again, it had this status, blah blah blah. Not just the status, but the whole payload. Right. So that like visibility into the mechanics of the system is like built into the system so that you don't have to pop open the hood and like tediously dig around there, all that is just built in.

SPEAKER_00Yeah, or instead of like tracing tracing the code and it trying to like establish an understanding of what the response looks like based on the code that's parsing that response, it actually gets to go in and look at that specific response. Like this is what was actually returned.

SPEAKER_01Mm-hmm.

SPEAKER_00Uh yeah. Yeah, I think that's totally totally in the same in the same vein of like what I'm how I'm how I'm shifting, and I think you're right. Like I think it is just a good practice of like I th I think this is probably the type of work that we would have done if we were sitting there in implementation, which is like I might have copied some responses into a document just so that I could reference them. Um but that is maybe like the corners that are getting cut when you do something through AI.

SPEAKER_01Um well I think AI has made me lazy in a way that's not necessarily bad. Um and now like when I have to go out of the mode of just like prompting and watching the code come back for me when there's like an actual problem where it's like, oh, now I have to like actually exert brain power. This is annoying. How can I like not do that? That's what makes me think of these things, like building in those snapshots so that the AI can just look at that itself, or I can look at it more conveniently.

SPEAKER_01I might not have thought of that before because I hadn't got gotten into this lazy mode. Um, and and I would have just it's like, okay, I've been mentally exerting myself all day. This is just more mental exertion, but now it's like I have to climb this hill of mental exertion. And that's what's making me think of these uh lazier, higher leverage, better ways of working.

SPEAKER_00Yeah. And and I think it's it's also the type of stuff that you may not have first thought of before, because it's like, oh, I I can't spend a bunch of time implementing like a snapshot system in order to like capture this data. But if it is something like in the realm of like code is cheap, it it is now something that is one prompt away. Um and so now like I can build that observability in much much more simply.

SPEAKER_00Um one of my favorite techniques is honestly like something is going wrong, like there's clearly something going wrong in an implementation. Maybe I've gone back and forth with the agent a couple times and just like it's not getting it. And I in frustration, or it maybe I'll call it like lazy genius. I'll just say, like, what do you need in order to diagnose this problem better? Um it's like a 50-50 hit. Like sometimes uh it either like comes up with a thing where I'm like, let's do it, man.

SPEAKER_00Like that that is actually the perfect thing that I think will fix this, or it it winds up like getting my gears turning enough to think of the thing that it needs to do because all of the ideas that it came up with were terrible. Um but it it is really fun of just like trying to lean on the AI agent to just be like, hey, what what would make this but like you're the one that's gonna be doing the implementation here, like what would make this easier for you? Um which is which is maybe like one click too much anthropomorphization that I really want to do, but uh it does work in some ways.

SPEAKER_01No, I think I think that's really good. Um I had Dave Thomas on recently, and he mentioned that he asked Claude Code, like, hey, how could I write my code in a way that makes it easier for you to understand? Right. I thought that was brilliant. Um I this sounds to me like basically the same idea. Um I found that like often when there is a bug that's really hard for the AI to diagnose, sometimes that's a sign that my system is designed in a way that's kind of inscrutable.

SPEAKER_01And maybe instead of solving the bug as the next step, uh maybe I change my design so that it's easier to understand. Um, and then maybe it's the case that the design just doesn't leave any room for that bug to even exist, and that's obviously a great situation. I I I don't think I ever fully grasped the the concept that I've heard for many years of like design bugs out.

SPEAKER_01I don't I don't think I really understood what that meant. Um, but now I I feel like I get that a little bit more. Like there are different ways to design things, and you can design things in ways that um well if if I design something in a way that's needlessly complicated, maybe it leaves room for a bug where if I had designed it in a simpler and elegant way, there there's just no place for that bug to happen. Right. And so I think that's maybe what's what's meant by design bugs out.

SPEAKER_00Yeah. Yeah, I th I think it just winds up being like try to surface an error before you see the bug kind of stuff. Where it's uh I I feel I feel like a lot of those boil down to like bad data, and you're like allowing nulls and something that you don't expect to be null. Um sorry, I've been doing too much JavaScript.

SPEAKER_01You're allowing nils uh and something that you um um yeah, and another thing I've been doing a lot of I I learned this from Joel Drapper, I I learned the t term at least of uh runtime assertions. So like if there's some state that the program gets into that shouldn't ever happen because it doesn't make sense, uh halt and raise an error and and fail loudly. I found that to be really helpful.

SPEAKER_00Uh yeah, a hundred a hundred percent. Like those those sorts of things of like I if there if there is a bug in my system, I don't want that to like silently propagate through the various layers. Like the first at the first point that I might have some inclination that that is happening, like I that's when I want to know about it. Uh like that's gonna give me the clearest diagnosis, and that's gonna give me like the clearest uh path to how did this happen. So like if you can if you can do it on like here's where the data enters the system, like validate at the boundaries is obviously like a big part of this.

SPEAKER_00Um if you can do it at the place where the data enters the system, then you have like a much better idea of like oh I something external is providing this bad data versus like something inside of my machine is providing bad data. Yeah. And so I think that those are great. Like fail failing loudly. It I feel like that one always has to be balanced against like a customer need kind of thing, where like some sometimes people don't want it to fail quite as loudly of just like uh maybe I could do the next step even though the this data is bad.

SPEAKER_00Um so I I feel like that's always like a balance to go up against.

SPEAKER_01Yeah, that makes sense. Uh I'll I'll give another example of going back a little bit to the observability concept. Um I've been having well, a while ago I had this challenge with Saturn CI where I migrated from digital ocean cloud where it was elastic resources, I could just order up however many machines I wanted. I I switched to a Hetzner VM where I have exactly one machine, and it's a fairly capacious machine, but there's only one, it has finite uh resources.

SPEAKER_01And I I had uh you know myself and one other like customer with a relatively large test suite uh competing for resources, and so there would arise situations where there were there would be more demand on the system than what it could handle. It's like okay, how do I handle that exactly? Um and I made it so that in instead of every test suite run just trying to execute right away, um it would get queued up if there was not enough resources.

SPEAKER_01And the way that works is um it it it all runs on Kubernetes, and so when a test suite run is queued, um it says, okay, how much resources does this test suite run need? Uh how much headroom does the Kubernetes cluster have? If what's needed is more than what the headroom is, then we don't have enough room. So queue this one up until there is room.

SPEAKER_00Right.

SPEAKER_01That is like conceptually simple, but it it you can imagine how you can run this and observe the behavior and have it be kind of a mystery why such and like why did this one get cued? And why is this not running when I expect it to, and stuff like that. So I did like I I added to the logs, like, okay, this one just got queued up. Here's exactly how much headroom the cluster has right now.

SPEAKER_01Uh-huh. Here's how much I need, and then every few seconds it would check and and it's like, okay, here's the headroom now, here's what I need now, so that anytime uh anytime there was a fork in the road where it could either get it could either stay cued or it could get started up, there was a uh snapshot again of uh of what the state of everything was, so that there was always an explanation, a readily available explanation.

SPEAKER_01I didn't have to dive into logs and stuff like that. It was just right there. And again, this would have been a great idea without AI too, but I think you had a great point with like you can just easily order it up and be like, hey, dump that to the log so I can see.

SPEAKER_00Right, right. Yes, I can yeah, I can get my AI turk to do this. Um is like that's that's what we're able to do now. And and uh that makes it a lot of fun because it is like the only the now the only constraint is ideas. Um and I think like that that makes it like a really fun place to be in as a profession where suddenly like the it's not it's not the ability to like pull off one of these things, it's it's the ability to like think through how how do I scale this up, how do I make this work in a way that can be implemented by an AI, but also like more importantly, like actually like serves the need that I'm that I'm doing.

SPEAKER_00Like you like your thing, like you you don't want people's test suites to go exorbitantly slow or to fail. And so it's like okay, I I only have so much headroom, I can just I can make this Q system. Um and that that's a way like through this problem that you yeah, that it was probably very easy to implement that once it once you had that idea.

SPEAKER_01Yeah, I I wish I could say it was easy. Yeah, it's you know it it it it was probably not easy just because of my lack of like understanding of queuing systems and and all that stuff. It's just complicated stuff, surprisingly complicated. Um okay, we're getting close to time. Um before we go, I want to give you a chance to share anything you want to share, links and stuff like that, uh, whatever you got for us.

SPEAKER_00Yeah, for sure. Uh I am I am not very active on social media, but you can probably most active on Twitter, X, and you can find me at Will T Carry there. Um brand newbox.com is where we uh is my software agency that and we're always looking to help people um solve whatever whatever problems that they have. Um yeah, we we work a lot with people in the non-tech industry, but we we love to work with people in the tech industry as well.

SPEAKER_00Um and then yeah, I've I need to start talking more about like I two in it two months ago I was like, hey, VS Code is not working for me anymore. And so like I started building my own editor, and so that's Rains. It's open source, R-E-I-N S, will t carry slash reigns on GitHub. If anybody wants to check it out and use it, it's it's a lot of fun. It is a code editor with no ability to edit code.

SPEAKER_01Interesting. Well, I will have to check that out. By the way, this weekend I made a program called Yoke Y-O-K-E.

SPEAKER_00Oh, really?

SPEAKER_01Which is like you know, a harness for Ruby projects. So it's that's kind of funny. Um, okay. Well, we will put all that stuff in the show notes. And Will, thanks so much for coming on the show.

SPEAKER_00Yeah, thank you. Uh thanks for you for having me, Jason. It was a ton of fun. Always game to talk about this stuff. Awesome, you know.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. 338 - Code Review, Testing, and Staying Sharp in the AI Era with Joel Hawksley

    Nov 18, 20261 hr 5 minAudio

    In this episode I talk with Joel Hawksley, Staff Software Engineer at GitHub and creator of ViewComponent, about whether AI is making us dumber, why blocking code reviews struggle under a flood of AI-generated code, and...

    Credit the people on this episode

  2. 337 - User Testing, Paper Prototypes, and Escaping Hourly Billing with John Athayde

    Nov 17, 20261 hr 18 minAudio

    In this episode I talk with John Athayde, designer-developer and co-author of The Rails View, about what architecture school taught him about software design. John tells the story of a user test at LivingSocial that...

    Credit the people on this episode

  3. 336 - Riccardo Carlesso, Co-Organizer of Rubycon Italy

    Nov 13, 202646 minAudio

    In this episode I talk with Riccardo Carlesso, Google developer advocate and co-founder of Rubycon, a Ruby conference held in Rimini, Italy in May 2026. Riccardo explains how a joke about PyCon became the conference's...

    Credit the people on this episode

  4. 335 - Blastoff Rails: Starting a Ruby Conference from Scratch with Travis Dockter

    Nov 11, 202654 minAudio

    In this episode I talk with Travis Dockter, organizer of Blastoff Rails in Albuquerque. When Sin City Ruby and RailsConf both ended, Travis decided to start his own conference. We discuss how cheaply you can run a...

    Credit the people on this episode

  5. 334 - Merge Queues and Building Mergify with Julien Danjou

    Nov 10, 202655 minAudio

    In this episode I talk with Julien Danjou, CEO of Mergify, about merge queues: why PRs that pass CI separately can still break main together, and how testing changes in a queue keeps main green. We discuss how Mergify...

    Credit the people on this episode