This is a free preview of a paid episode. To hear more, visit refactoring.fm Listen now:YouTube | Apple | Spotify Today’s episode is brought to you by: Notion Learn more about Notion’s Developer Platform today at https://ntn.so/refactoring Today's guest is Adam Tornhill, founder of CodeScene and author of Your Code as a Crime Scene. With Adam, we talked about how his team refactored the whole Street Fighter 3 codebase, which is more than 300K lines of code, in just a few days and spending a few thousand dollars in tokens, and measured how that led to better and cheaper AI development of new features. And then we discussed how the CodeScene team does agentic development, what the process looks like, what the bottlenecks are, and what are the crucial parts that Adam believes we still need to do as humans. Show notes: * The Street Fighter case study * Controlling the uncertainty machine: do you still need to read the code? * Making legacy code AI-ready 🥇 Interview Summary If you are a 🔒 paid subscriber 🔒 you will find my own summary of the interview below. It’s the 5-minute, handcrafted takeaways of what we talked about, with timestamps to the relevant video moments, for those who don’t have time to sit through the 1-hour chat. Here is the agenda for today: * 🕹️ Refactoring Street Fighter with AI agents * 📖 The refactoring playbook * 🧪 Testing, docs, and the new fundamentals * 🤖 CodeScene’s agentic development process * 🔮 Bottlenecks and the future of the developer role Let’s dive in 👇 1) 🕹️ Refactoring Street Fighter with AI agents Adam’s team at CodeScene, in collaboration with researcher Marcus Borg, set out to explore large-scale refactoring using AI agents. They needed a code base that was non-trivial, legacy, and universally understandable, so they picked a decompiled version of Street Fighter 3, a 25-year-old game with 300,000 lines of C code. When they first analyzed the code base with CodeScene’s own metrics, it scored a 5.4 on a 10-point scale, roughly the industry average for enterprise code and very far from what Adam calls “AI-optimal code” (9.5 or higher). The team had two goals: uplift the code health as much as possible, and preserve the existing behavior of the game. “I’ve never ever seen anything close to this before. A rewrite at that scale would have been a 12-to-18-month project. And now that can be automated for a fraction of the cost at a fraction of the time.” Street Fighter turned out to be a particularly good target because the game already had a clever test approach: computing the hash of each rendered frame and comparing them. This gave the agents a reliable way to verify that refactoring didn’t break anything. The preparation phase took about two weeks (as a side project), with the team exploring different models and patterns. The actual uplift then took just a few days and cost roughly $4,000 in API tokens. They primarily used Claude Code with Opus, because while GPT models were competent at refactoring, they weren’t as good at documenting and extending the playbook. 2) 📖 The refactoring playbook
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
SPEAKER_01Tell me what you've done about Street Fighter first.
SPEAKER_00Yeah, sure. I will do that. So most of the work is done by the Codesyn research team in collaboration with Daniel Webb, a colleague that we work on. And what we were exploring was refactoring at large scale. We're using agents.
SPEAKER_01Yeah.
SPEAKER_00Because we all know that the existing structure and quality of code It's an important part of an agent's context, right? So you won't help the code. And the reason we came to Street Fighter was that we were simply looking at a domain that everyone can understand, that many people are familiar with, and that is non-trivial and large-scale and legacy. So we picked up a decompiled version of Street Fighter. So it's a 25-year-old game, Street Fighter 3.
SPEAKER_00And we looked at refactoring and uplifting the whole codebase using agents.
SPEAKER_01Hey, Luca here. Welcome to a new episode of the Refactoring Podcast, where every two weeks we interview a world-class tech leader. Today's guest is Adam Thornhill, founder of CodeScene, author of Your Code is a Crime Scene, and one of the very few repeat guests of the show. With Adam we talked about how his team refactored the whole Street Fighter 3 codebase, which is more than 300,000 lines of code, in just a few days and spending a few thousand dollars in tokens,
SPEAKER_01and measured how that led to better and cheaper AI development of new features. And then we discussed how the Codescene team does a gentle development, what the process looks like, what the bottlenecks are, and what are the crucial parts that Adam believes we still need to do as humans. So let's dive right into the action. Hey, before we continue, I want to tell you more about the sponsor of this podcast season, which is Notion.
SPEAKER_01If you've been reading Refactoring for a while, you know that I'm a big fan of Notion. I've used it for many years and I've been in touch with the team for a long time. We've compared notes about knowledge management, agents and more. So speaking of agents, with custom agents which came out earlier this year, Notion became a collaborative AI workspace where teams and agents can work side by side. And I love that now they have revamped their entire developer platform to turn that workspace into actual infrastructure that developers can build on.
SPEAKER_01And the new developer platform introduces three new capabilities. First of all, you can sync any data source into Notion, continuously abserting data from tools like Linear or Salesforce or Jira into a Notion database. Second, you can build any tool for your agent, so it can generate assets, query live data, or hit your internal APIs. And finally, you can orchestrate any agent in Notion. You can pick your favorite one, including code, codecs, or the one you built yourself.
SPEAKER_01You chat with them, assign them work, and track their progress right inside the same collaborative workspace where your team already works. So put it all together, Notion becomes an orchestration layer where data, tools, agents, and teams work together. Learn more about Notion's developer platform today at ntn.so slash refactoring. Welcome, Adam, and thank you so much for being here with us today.
SPEAKER_00Thank you so much, Luca. It's our pleasure to be back.
SPEAKER_01Yeah, you're one of the very few repeat guests on the podcast. You are, of course, founder of CodeScene, author of Your Code as a Crime Scene, and I'm a daily user of CodeScene, and we regularly stay in touch to discuss AI coding, code quality, and more. And the occasion for this chat came because you basically told me we need to talk about Street Fighter. And of course, I mean, count me in. So I know you're about to publish or based on when this interview comes out,
SPEAKER_01you may have just published an incredible Refactoring case study that you did. And so I leave it to you first. So tell me what you've done about Street Fighter first.
SPEAKER_00Yeah, sure. I will do that. So most of the work is done by the Codes in Research team in collaboration with Daniel Webb, a colleague that we work on. And what we were exploring was refactoring at large scale. We're using agents.
SPEAKER_01Yeah.
SPEAKER_00Because we all know that the existing structure and quality of code is an important part of an agent's context, right? So you won't help the code. And the reason we came to Street Fighter was that we were simply looking at a domain that everyone can understand, that many people are familiar with, and that is non-trivial and large-scale and legacy. So we picked up a decompiled version of Street Fighter. So it's a 25-year-old game, Street Fighter 3.
SPEAKER_00And we looked at refactoring and uplifting the whole code base using agents.
SPEAKER_01Yeah. Yeah. That, that sounds very interesting already. And we are talking of a very big code base, right? You told me.
SPEAKER_00Yeah. It's fairly big. I would say it's a 300,000 lines of C code.
SPEAKER_01Yeah. Of C code. So when, when you pick that up, what was the shape and this, the health of that code base?
SPEAKER_00So the first thing we did was, of course, to throw Street Fighter into code scene and look at the code health. And what we found was that the average code health came out at 5.4 on a 10 point scale. So this is roughly around the industry average, what you would expect in an enterprise code base. And it's very far from AI optimal code, which means 9.5 or higher, right? Yeah. So this is code which will be high risk to try to accelerate any feature development with AI, I would say.
SPEAKER_01Yeah. Okay. So when you set out to refactor that code base that is massive and it's complex work, what kind of goals did you set for yourself? I mean, Measurements of that the outcome of this work was good for you and now the code base was in good shape. So did you take into account just the score like out of 10 or measures
SPEAKER_01about how easy or hard would it be to develop new things? How did you think about it?
SPEAKER_00So we had two main goals. One was, of course, to make the code as good as we just could using agents. And for that, we needed some kind of objective metrics, some kind of judge that could tell if one version of the code was better than another. This is where we use the code of metric from Codesyn, of course, because we know it correlates with development time and effect risk. So that was that part. The other one was that, of course, we wanted to preserve behavior. And this is the cool thing with Street Fighter.
SPEAKER_00It's a game. We can rebuild it after each iteration. And my colleagues actually did that, right? So they ran. The game after each step and verify that the automated verifications we had in the agentic loop that they had managed to preserve the behavior. Those were the two criteria. Preserve behavior uplift to a place where new changes can be done safely.
SPEAKER_01This is very interesting. When you say preserved behavior, did you have to build like an automated verification suite and testing about the end-to-end behavior of the game that needed to be preserved? Or did you find that already in the code base? Did you have to, you know, improve what was existing to make sure that you could refactor with a high level of confidence? How did you go through it?
SPEAKER_00So this is the cool thing about Street Fighter and perhaps what makes a game slightly different from the traditional enterprise domain. And that is that there was already a fairly interesting test approach. So what those tests do is basically that they compute the hash of each frame in the game and then compare them. So that is something we could use in the agentic loop to verify that the refactoring didn't break an existing behavior.
SPEAKER_01Okay. So games are a particularly good target for this type of work, you're saying?
SPEAKER_00I think that anything that has a visible outcome that is simple to inspect is a fairly good target. And I think I think we're going to talk about that later in the, in the discussion. Right. But, um, I think that's one of the challenges for many enterprise code base today, that, uh, technical depth tends to go in hand in hand with low test coverage and low test code coverage. Right.
SPEAKER_01Yeah. Yeah, absolutely. And so, I mean, we are looking at a massive code base, 300,000 lines of code with plenty of stuff to To improve, to refactor, how do you even approach such work in a way that is maybe split into manageable pieces, starting with the most important things first, you know, segment it into a sequence of things that make step one after the other in
SPEAKER_01the logical order. How do you think through this?
SPEAKER_00In this case, it was relatively straightforward, but that's one important twist to it. So first we had, of course, the tooling set up and the team did a lot of exploration, tried different AI models and tried different patterns to kind of figure out what worked. So that took like roughly two weeks. It was not a full-time job, it was a side project. And then the actual uplift that we're going to talk about soon took just a couple of days. And one of the important things was that Markus Borg,
SPEAKER_00my colleague, he had this idea that he wanted to introduce a refactoring playbook in the process. So the idea is that an LLM should not just solve a specific problem, but also document what worked and under which conditions and constraints. And the idea is that with that playbook, refactoring should become progressively easier, right? As the LLMs can build on existing knowledge in that codebase.
SPEAKER_01Okay.
SPEAKER_00So that was key. So one important outcome of this is that not only did we end up with a super healthy codebase, we also ended up with a large playbook of refactoring patterns.
SPEAKER_01Yeah, that's it. And so we are talking of patterns that are generic, like high level refactoring patterns that work for all kinds of software or, and, or we are talking of patterns that were specific to this instance in particular.
SPEAKER_00Yeah, so I have to admit that when I saw the result, that is where I had my first real wow moment with AI, because it's the first time I see something that looks like a genuine innovation. Because you see that refactoring catalog, there are lots of familiar patterns, stuff that we all know about, extract method, right? Introduce God classes. That's not surprising.
SPEAKER_01Martin Fowler, yes.
SPEAKER_00Yeah, exactly. The generic Fowler refactorings, which are the foundation. But what blew my mind was that AI had innovated a lot of code-based specific refactoring patterns.
SPEAKER_01Okay.
SPEAKER_00And documented them.
SPEAKER_01Okay, very interesting. And so, I mean, of course, it's, I think it's hard to grasp the specifics in a conversation like that without looking at the code base. But so you mean that Figured out patterns that were specific to the code base, but that could be abstracted as more generic refactoring patterns that might be useful even in other circumstances. So it's like invented new refactoring rules.
SPEAKER_00Yeah, you could say so. I can try to give an example.
SPEAKER_01Yeah.
SPEAKER_00One common code smell in Street Fighter and in many other legacy code bases is code duplication, right?
SPEAKER_01Yeah.
SPEAKER_00So you have two or more functions that implement exactly the same process, share the same knowledge, but have slight slight variations. So as an example for Street Fighter, it might be manipulating game objects, right? Then you have basically the same process for manipulating something in In the Y coordinate spaces, in the X coordinate space. That's an example. So I have two duplications of that. So now what the AI did was, of course, that it extracted the commonalities and kept that as a shared function.
SPEAKER_00And then it parameterizes with the behavior that varies. And it came up with the name for this. It called it action parameter. And I thought, what's an action parameter, right?
SPEAKER_01Yeah.
SPEAKER_00Basically, what it does is that it refactors the code to use the design pattern strategy. And that is, of course, nothing new. But what's so unique about this is that these code-based specific refactorings are so narrow and they come with a lot of constraints. And that is what makes it possible for the AI to mechanically translate detailed patterns in the code, these more generic building blocks and foundations for refactoring.
SPEAKER_01Yeah, that's so interesting. And you say that he documented everything into a playbook, basically, where it explained the various patterns that he used, the various actions, the outcomes, documented its own ideas and behavior, basically.
SPEAKER_00Yeah, that's correct. So we have a large playbook, I think they're all like, yeah, 22 prepackaged receipts and they are very, very detailed. They are detailed at the level that no human would ever go to that length.
SPEAKER_01Yeah, I can imagine. And so you said basically there has been like two weeks preparation and just a few days for doing the actual work. So the preparation was actually Figuring out what the approach had to be. And that was the part where the humans were the most involved, I guess, and maybe instructed the AI to look at this or that. And then the AI went off by itself in the later days.
SPEAKER_01Was it like this roughly? You tell me.
SPEAKER_00Yeah, that's roughly the process. And even if we take this conservative estimate that it took three weeks end to end, which it did. And the total cost of that was in API tokens was roughly $4,000. And I mean, I remember from the past, all these projects I worked on, all these large-scale systems, right, where a rewrite at that scale would be like a 12, 18-month project involving, you know, a large staff, high-level expertise.
SPEAKER_00Yeah. And now that can be automated for a fraction of the cost at a fraction of the time. Wow. So to me, this is a big, big step forward.
SPEAKER_01Yeah. And what you said, you tried different models. There were some specifically that you eventually settled with or use different models for different purposes. What did you do?
SPEAKER_00So. There were two people working on this, right? And they had slightly different preferences. But I know that the team ended up using the latest cloud code with Opus simply because the GPT models, you know, 5.6 and all these, they were competent at refactoring. That was not a problem, but they were not particularly good at documenting and extending the playbook. And Opus did the best job at the playbook, right? So that's why we stuck to that one.
SPEAKER_01Yeah.
SPEAKER_00Then I remember that there was a small set of, so Marcus Bore, my colleague, told me this. They ended up with a small set of files that weren't perfectly refactored, right? They were better, but not perfect. So at the end, they brought out a big sledgehammer and took in Fable to do that final uplift. Oh, okay. But it was very minimal. It was like maybe 2% of the work. The rest was done with cheaper LLMs.
SPEAKER_01Yeah. It's interesting because I find it hard to understand many times what are the differences between models. I mean, given that we know they exist, right? As you said, maybe you have two models seem to be perfectly Equally competent, for example, in coding, but then you discover one of them is better at documenting the process and creating the playbook. And this is to me such a narrow skill,
SPEAKER_01you know, that you would not expect to maybe vary a lot from one model to the other. And if anything, probably this is a reminder that it's worth to try out More models because their behavior might be unexpectedly different, at least on different tasks.
SPEAKER_00Yeah, absolutely agree. And whatever knowledge we gain in how LLMs work, it's very transient, right? Because the next generation, they kind of move the needle again on what the relative strengths are of each one of them.
SPEAKER_01Yeah, yeah, totally. And I'm curious when you supplied the LLMs with the code base at the beginning, did you also provide Other kind of information or documentation about the codebase itself or context about maybe Street Fighter as a game or decisions that had been made about the architecture or was most the information just understood by the models from the codebase itself?
SPEAKER_00So I would say that the majority of the information was You know, sourced from the codebase itself. And of course, the LLMs, they are likely to have a lot of training data already and know everything about Street Fighter, the game itself.
SPEAKER_01Probably more than us.
SPEAKER_00Yeah, they are the main experts on that, I'm pretty sure. What's interesting about this too is that there will be a next step. So the origins of this case study was To develop a training course for students. So that's a course that my colleague runs at Lunds University in Sweden. And the idea was to take a legacy codebase, refactor it, and then split students into a group and have one group of students extend the legacy codebase with the eye. And another group, of course, work on the uplifted healthy codebase.
SPEAKER_00So that gives us a way to measure for real. All these predicted benefits are faster and better. Do they materialize in the real world?
SPEAKER_01Okay. Okay. And, and how did it go? So you, you could measure the before and after in terms of, uh, bugs introduced by AI and other measures that can, uh, you know, give an indirect, uh, measure value of the, of the code base and how that impacts, uh, the, the, the further development of new things.
SPEAKER_00Yeah. Yeah. We, we actually know a lot from our research that we, uh, So we can make some pretty good predictions on what the outcome is likely to be, right? And it relates, of course, to development speed and defects. I think that's super important, but it also relates to token spend. Because if a feature can implement that at a fraction of the cost in a healthy codebase compared to an unhealthy, then that's another strong argument for logic refactorings. So we'll see.
SPEAKER_00But I think all of us were Surprised by the outcome because I've never ever seen anything close to this before.
SPEAKER_01Yeah, especially because, I mean, you said only $4,000 and at the end of the day, just a few days of work, you know, it's the elapsed time is maybe a few weeks, but it's a side project run by humans that are also doing something else. And they provided instructions or the first couple of weeks you said, and then Just a few days of intense work by the models and it's done on 300k lines
SPEAKER_01of code. It looks pretty good.
SPEAKER_00It's spectacular. And, you know, I used to joke about that as an alternate retirement plan, I can always go back to maintain legacy C++ code, right? But now that option seems to be gone as well.
SPEAKER_01Yeah, you wouldn't make a lot of money anymore on that, probably. But so this was a specific example right on the Street Fighter code base. If you, I mean, I like this example because I think one problem when we talk about AI in development is that there are a lot of stories and examples that are on greenfield projects, greenfield development, and they do not take into account the reality of multi-year,
SPEAKER_01decade-long code bases with a lot of legacy and turnover by the team. And instead, this example you provided addressed exactly that. If you had to generalize some of the learnings from this project that a team can take when it comes to thinking about refactoring the code base and how to do that with AI, what is something that you learned?
SPEAKER_01How would you approach refactoring a large code base?
SPEAKER_00Yeah, so there are definitely general lessons here. So I think what we have now is a process that can be applied at enterprise scale. And the first thing that's needed is to ensure that you can preserve behavior. And that is not only verifying the functional correctness, but of course also the non-functional aspects, depending on which domain you work in, what type of product. So that calls for a lot of test automation.
SPEAKER_01Yeah.
SPEAKER_00And I think times are a bit different now because I think that test automation has to be more at the system level, right? It's not so important to have the unit test coverage in the way we tended to be obsessed with in the past. But you definitely need to be able to let the agents decide if The refactorings are behavior preserving or not. The other thing to consider is that 300,000 lines of code, it's a lot of code, but many enterprise code bases, there can be 10, 20 million lines of code, right?
SPEAKER_01Yeah.
SPEAKER_00And in that case, it's most likely not the right call to just try to refactor it all.
SPEAKER_01Yeah.
SPEAKER_00And the reason I say this goes all the way back to the stuff from your code as a crime scene with the hotspot analysis, where development work If it's done by humans or agent, it doesn't really matter. It's very unevenly distributed, right? So there are small parts of the codebase where most of the cinch activities. And what I would do is I would drive refactorings and improvements through those hotspots, through those agentic development hotspots, because that's where the return on investment they are.
SPEAKER_00It's just an unnecessary risk to try to refactor code that you haven't touched for years and are unlikely to touch again. And it also narrows down how much of the code you need to cover with new tests. So that would definitely be my recommendation as a starting point.
SPEAKER_01That's very interesting. And I want to go back to something you said before about testing, verifying behavior and the fact that probably As far as I understand by your words, a good verification now has a return on investment that is probably way higher than it used to have, right? Because now we can use it to keep code healthy. You know, against the actual behavior and we can have AI maintain it as opposed
SPEAKER_01to humans keep these tests updated. And maybe does this mean that even like end-to-end tests and behavior tests, many types of tests that before I think were very hard to, um, To develop for humans and keep healthy and not flaky and maybe spending a lot of the product now instead are more important than before because you can, once you have them in place and they are reliable,
SPEAKER_01you can have the AI maintain the code base in a good health because of their presence, because of relying on them.
SPEAKER_00Yeah, that's a very good point because I do agree that The economics of that type of testing has changed, right? So these end-to-end tests, they used to be ridiculously expensive to maintain, right?
SPEAKER_01Exactly.
SPEAKER_00But the economics are different for an AI agent, right? It won't complain that it has to make a predictable change across 200 files, right? They excel at that stuff. And I think we should use that to our advantage. So yes, the value of our Proper test suit is so much higher these days. To me, it's an absolute necessity if we want to reap the benefits of agentic speed. And I also think that it's part of a larger pattern that a lot of fundamentals around system design,
SPEAKER_00software design and architecture are starting to pay off really, really big now. Because it all fits together, right? It's really, really hard to parallelize work with agents if the code base is a messy monolith. It's also really hard to test it if it has super tight dependencies to say third party services and stuff like that. So I think all the fundamentals are being reinforced. That's the way I see it.
SPEAKER_01Yeah. So we talked about testing. Another area that gets a lot of conversation About how it is changing with AI, and I would love to pick your brains about it, is documentation, right? Because there are different camps about it. Because on one hand, we can see that the AI is able to pick We can pick up a lot of how a code base works just by looking at the code. And we may argue that there is no better, you know, documentation than just looking at what exists, right?
SPEAKER_01As opposed to artifacts that are about Other artifacts. But at the same time, docs are also easier now to be written and maintained because AI can do that. And maybe are one of the things that in some parts have a higher return on investment, just like tests. So in your experience, have you found that there are specific types of docs maybe that are particularly useful for the AI or maybe not?
SPEAKER_01Has your opinion changed about that all the time?
SPEAKER_00I think that I'm still learning and exploring, right? But I have come to some conclusions. And the first is that the code itself was never enough as documentation because we all know that we Code can explain mechanically what it does, but we don't know why it does what it does, right? So that type of documentation was always a necessity. And that can include a lot of important context, like,
SPEAKER_00you know, which customer requested this particular feature in the codebase, right?
SPEAKER_01Yeah.
SPEAKER_00So that I think was always important. And that's the way we internally at Codesign, we have always tried to document code that way, right? And I found out with AI, it's simpler than ever, but I have to constantly remind my agents, hey, write this type of documentation, write down the rationale and motivation. The other type of documentation I find really, really important is that helps AI is to document the actual processes I would have gone through as a human, right? So it's a way for me to So it would typically be a traditional agentic skills, right?
SPEAKER_00That I find are super useful to capture processes. So I combine these. So I have processes captured as skills and the rationale and motivation for design, the trade-offs, I capture them either in code or ask the AI to write down a small markdown file once it has completed or change your new feature. And I, you know, commit it together with the code.
SPEAKER_01Okay, so I want to unpack this a little bit. So basically the first part of your answer you said, the most important things to document are the why behind something exists as it exists, as opposed to maybe in the past we used to document a lot what exists, How it is done. Now this is maybe something that the eye can infer by itself by looking at the code. But if you just look at the code, you'll never figure out why something is in the shape that it is,
SPEAKER_01what are the rationale, maybe the alternatives that were considered and discarded as opposed to what was implemented, all the context behind why something is like this. Am I correct?
SPEAKER_00Yeah, exactly. That to me is a critical piece.
SPEAKER_01Yeah. And when you said instead the process that gets like bottled into, into skills, is that the development process? Is that how a developer would think about how to write code? What goes exactly in those skills?
SPEAKER_00So I would say it's the development process and it's also a certain preferences, right? So, uh, we touched upon, uh, tests before. What I found is that in the way we work, we use tests as the boundary, the abstraction level between human inspection and machine generated code that I might not even read, right? So for that purpose, it's really, really important that these tests are optimized for inspection by a human.
SPEAKER_00And that requires certain patterns or certain shape that AI just won't do out of the box. And some of that can be captured as skills. And some of it simply has to be enforced as well via things like domain specific linting rules.
SPEAKER_01Yeah. Okay. Interesting. So you said these tests need to be inspected by humans, which makes me think about like a hot topic today, which is what parts of the code and the work of AI still need to be reviewed by human and inspected as opposed to What parts is that we are just happy to have them verified with automatic testing and our guardrails and sensors like
SPEAKER_01the code scene MCP itself, but we don't need to look at them manually line by line. How is it for your team? So do you still inspect a lot of the code that the AI produces? Or maybe there is still a lot of code that is done by humans. How has been the journey of Codescene itself as a team about agentic development? So maybe we can take it a little bit from behind, expanding the scope.
SPEAKER_00Yeah, yeah. Maybe let's start there with the agentic journey and then we naturally touch upon these other topics. So we went fully agentic, say six months ago.
SPEAKER_01Okay.
SPEAKER_00But one team started a year ago, and that was the team that was developing our code of MCP. Because a core principle at Codesyn has always been that we eat our own dog food, right?
SPEAKER_01Yeah.
SPEAKER_00So if we do an MCP for agentic coding, we need to develop it using agentic.
SPEAKER_01Makes sense.
SPEAKER_00And we obviously had been using AI assisted coding before, but most of our code is in Clojure and earlier iterations on LLMs, they weren't particularly good at Clojure, right? You're like, rather getting even syntax right. But a year ago, something really shifted and experience on that MCP was... It was really rewarding, right? We soon had multiple people develop new feature at a speed that we just couldn't have sustained earlier, right?
SPEAKER_00And we tend to measure our lead times for tasks, and we noticed our 2 to 3x speed up. So that were some of the very tangible benefits. And what we did early on was that we more or less You know, we leaned heavily on the processes we had used when coding manually, right? So that meant strong test automation, relentless refactoring, all that stuff. And I think that the big tipping point internally was that MCP2,
SPEAKER_00because once we had built the first version of it, we brought it out on the market, people start to use it. And after a while, we noticed a really tricky problem. And that was the problem that MCP, it was implemented in Python.
SPEAKER_01Okay.
SPEAKER_00And we had taken a Python code, we had compiled it to binary. The problem is that Python was never meant to be used that way. So the MCP had quite a long startup time, it took a couple of seconds for the MCP itself to program to boot. And the problem was with VS code, the editor in particular, that VS code tended to restart faster than MCP. And back in the day, that meant that if you're really, really quick and you open a, I don't know, a GitHub Copilot chat window, there won't be a connection to MCP.
SPEAKER_00Okay. It's a solved problem now, but it wasn't eight months ago. So we start to think about how can we optimize that speed up? And we didn't really find a good approach until one of my colleagues, as he explained it itself, he put his agents to work in rewriting that MCP to Rust. And then he watched TV for three hours, right? And the fun thing is that it actually worked, right? We had like, we spent a day reviewing and debugging and that was it.
SPEAKER_00We can deploy a new version. Our clients didn't even notice that this was a complete rewrite. And the reason we had pulled this off was because we had invested a ton into proper end-to-end tests and really, really healthy code. So that laid the foundation that gave us these options to solve our business problem. And that was like the tipping point that made us all transition into agentic coding because the power was clearly there and it worked for us.
SPEAKER_01Yeah. I love this story because first of all, it highlights the power of having good testing and a good verification suite that allows you to go faster and And maybe the agent had to do a lot of iteration, but it could do them autonomously, right? Without human intervention, because it could test against the suit. And when you mentioned Rust, I mean, I'm implying you had a good experience I had my experience in with AI and agents and developing in Rust and which matches my own experience as well anyway,
SPEAKER_01because I'm developing Tolalia, which for those who don't know, it's an open source application whose backend is written in Rust. And at the beginning, I was kind of concerned that, you know, Rust is a relevant, It's a recent language, right? So if you think at the training set that the AI models have seen, it's for sure way smaller than Python. Yeah, as you mentioned, for example, or JavaScript. And so I was concerned that performance would not be very good.
SPEAKER_01And instead, it's always been very good. And my suspect is that it's because Rust is a very strict language that provides a lot of telemetry, a lot of information that can steer the eye when they make mistakes and point in the right direction as opposed to other languages. So it also made me think that Maybe languages designed to work well with AI need certain characteristics that are not necessarily all the same as we need as humans, for example.
SPEAKER_01That, for example, we like more loose languages that have less static typing. I don't want to go on a tangent, but yeah, you mentioned Russ and I thought about that.
SPEAKER_00Yeah, it's a super interesting topic, of course. And I think, again, that AI has changed the game. So I always preferred dynamically typed languages because they allow me to iterate really, really fast. Yeah, me too. Yeah, you know, with agents, I can iterate just as fast with a statical type language, at least on the code writing level, right? And I think the There are some research reports out there that show that static types help on LLM do a better job because obviously they constrain the possible solution space and they are also almost executable documentation,
SPEAKER_00right? So it's quite natural, I think.
SPEAKER_01Yeah, yeah, I agree. And I would not go back to, even if I, as a human, I preferred Ruby and a lot of dynamic type languages. But so you said your team now is all in in agentic development. What does it mean? So what does the development process look like I could see in today?
SPEAKER_00So the development process is interesting because we are evolving that as we learn more with agents. That's the challenge, I think. It's not like there are years of expertise that we could just lean up on, right? We're all learning as we go along. The first thing I kind of found out was that there's not really one process anymore. It's multiple processes and they depend on the task, the problem at hand. So in particular, I'm using uncertainty of the task as one of the drivers
SPEAKER_00to how much rigor do we actually need. And as to give you an example, if we take a bug report, right? Bugs tend to be local. They tend to be local and contextual, right? And that makes them excellent candidates for an AI to debug. And the only thing I'm interested in is basically seeing a failing test case that reproduces the bug, right? And then I can use the same test case as evidence that the bug is fixed.
SPEAKER_01Yeah.
SPEAKER_00And I never, ever look at the code. There's no reason for that unless it's some, I don't know, some high-risk area authentication or something like that. Otherwise, we just accept that. And we can contrast that with a more open-ended task, right? Where we might not yet know what the solution is. And in doing the solution, we actually understand the problem better, right? So it's like an iterative process. And those type of tasks, I found that we have to use a much more iterative process.
SPEAKER_00We have stepped away from doing traditional TDD simply because we found that TDD is not aligning well with an agentic process. Because what we can do now in one agentic loop would require, I don't know, tons of TDD iterations, right? And TDD, I mean, it's an excellent methodology for lower level design. But those are not the challenges we have with agents. We know that our CODELF is safeguarded with MCPs and CLIs and all of that.
SPEAKER_00So we're more interested in iterating on a feature as a whole, iterating on a system level.
SPEAKER_01Interesting.
SPEAKER_00That's what we do. And to make it all work, we rely on a lot of deterministic safeguards. I think determinism is really, really key. So I already mentioned CODELF that we automatically check and enforce. And... What we also have is we have super strict gates on
SPEAKER_01code coverage.
SPEAKER_00That's another metric that I was never too fond of in the past, but now with agents, I found it essential for multiple reasons. And then we also have some architectural checks and domain-specific checks where we can automatically check certain dependencies and stuff like that. So it's like a layer upon layer in that safety net that make it work for us.
SPEAKER_01Yeah. So when you say architect, because I think we can all understand the code health gates, test coverage gates, but when you say architectural checks, domain checks, these are specific deterministic gates that are product specific, let's say that you develop for code scene or are things that can be employed by others too.
SPEAKER_00So they are definitely internal and specific to us, and they are local to our codebase. And it could be simple rules like, you know, if you modify that part of the code, you're never, ever allowed to depend on that part of the code. So simple dependency checks to enforce that and give feedback to the agents. To me, the whole architectural space, I think it's the largest remaining challenge to make agentic coding work at scale. There's simply not enough tooling there yet.
SPEAKER_00And we might not even, as a community, we might not even know what we really want to look for. But we see symptoms of it being a problem for sure.
SPEAKER_01Yeah. And so when you have these more open-ended tasks, there might be tasks where Your intuition and your judgment grows together with the iterations and the prototypes or might be tasks that are relevant from an architectural standpoint. You review the actual code that the agents do and then maybe you iterate on the specs. How does the process work on this?
SPEAKER_00So we typically start from a spec, right? So first we iterate on the spec itself. And once we're happy with it, then we, together with the API, together with the AI, and we develop like a rough iterative plan in that spec. And the first thing we do is we ask the AI to generate the end-to-end test. And the end-to-end test in this scenario can mean a different thing. It's not always a UI. Sometimes it ends up a REST API, for example, right? Yeah.
SPEAKER_00Then what I found is that I spent probably 90% of my review effort on those high-level tests, and I tend to iterate on them. So it's crystal clear what the expected outcomes are. Okay. And that, to me, solved a big problem. It solved a validation problem because the problem with the AI is that we never know that it's implementing the right thing. So I'm relying heavily on these higher-level tests to prove that. So once we're satisfied with that, then there's no application code yet.
SPEAKER_00We commit the tests. And I found that essential because the AI always has the tendency to be in a modifier failing test to make it pass, which is not helpful. And once that is done, then we tend to drive it feature by feature until the test suite passes. If it's a smaller feature, then we can let the AI do it in one sweep. If it's a larger and more uncertain task, then I think it pays off to be in the loop as a human. And it's not so much to help the AI do the right thing,
SPEAKER_00it's more for us to allow a human to build up knowledge of the thing we're actually building. So we don't create any intent that, right?
SPEAKER_01Yeah. So it's important to you that your team builds things while also building knowledge About what they're building, right? And I'm asking because I think this is a pretty big topic today. It's easy to go kind of fast without realizing what you're Doing under the hood. So when you talk about knowledge, you're talking about the big tech choices, design choices, the product.
SPEAKER_01So the team needs to go through the pipeline while building, while understanding and building their own knowledge of what's happening.
SPEAKER_00Yeah, I think it's absolutely essential because In the past, you know, we could always rely on one implicit assumption that there was a human programmer in the loop for every single feature. So there was always someone to ask, right? How does this work? Whatever. And that is no longer true, of course. And I had that experience myself way too many times that I asked the agent to do some kind of prototype, right? And I'm looking at that prototype and I realized that I don't know more about the problem now than I did before.
SPEAKER_00Because something gets lost when we outsource, not only the coding, but also the thinking. So that's what I want to separate. I want the thinking to be human. Of course, it can be assisted, but we as humans, we have to learn. Otherwise, we will kill our ability to innovate. And I'm pretty sure we won't be able to maintain and sustain our products and code bases.
SPEAKER_01Yeah. Isn't it hard to enforce, you know, kind of at scale for your team? Because I think there would be situations where it would be kind of counterintuitive because you might feel like you can go a little bit faster if, as a human, you're a little bit less in the loop, right? So how do you make sure people stay in that kind of loop and build that kind of knowledge? It's very tricky to me.
SPEAKER_00It is. I think it's virtually impossible, right? So the only guaranteed way I've found is to have a small team, have a super senior team with very skilled individuals. I don't think there's any way around that.
SPEAKER_01Yeah. I think we are still pretty much learning how to make all of this work. And also one thing I wanted to ask you, you know, now everybody's fond of saying the bottleneck is not this, it's that, it's moved, it's not coding, it's coding. So If you look at your development process at your pipeline, what looks more like the biggest bottleneck candidate for you, the one where humans are more in the loop, where they spend most of the effort,
SPEAKER_01most of their attention that if you were able to, you would try to attack and make it easier?
SPEAKER_00So I think there are two things. One is validation. I think verification we can largely solve via test automation, right? And that we are doing. But validation to ensure that whatever comes out is exactly the right thing, that quickly becomes a bottleneck. And that's simply because in the past, manual coding had a very natural pace to it, right? It simply took time to implement a new feature.
SPEAKER_00Now with agents, if whenever we do a new feature, we have to pull in somewhere from outside the team and ask them to schedule time in the calendar to test out and validate something, then, you know, the wait time for that is going to be three times as long as the actual feature development. So that kind of becomes the new bottleneck, I think. Yeah. And so the solution to me is that I think that we as software developers, we... We have to become better at the domain and we have to become better
SPEAKER_00at making product decisions as well.
SPEAKER_01Okay. So you think the Let's say the future or one of the possible futures is that developers get more scope, they work closer to product and so they can drive some of this validation themselves as opposed to, I don't know if that's what you meant, as opposed to having, for example, product managers in the loop all the time for every single change.
SPEAKER_00Yeah, that's what I'm seeing. I cannot see a fusion of three different roles, right? So, of course, you need to be technical. You need to understand software. And second, you need to have that product perspective. And third, you need to be fairly good at things like software architecture. And, you know, in the past, this used to be three separate roles, each one with a high salary. And now we expect one individual to kind of embody them all. So it will be interesting times for sure, but I think it's right now the only viable way forward that I see.
SPEAKER_01Yeah. And you think, you said before that it's not easy to keep all these things together and basically the only Way right now for you is to have a very senior heavy team, super skilled people that can, um, can go and do all of these. Uh, are you worried or concerned about, uh, junior developers, uh, not being able to have a clear growth path into, into senior, into, into this type of shapes of developers that now we have,
SPEAKER_01but maybe in the future would be harder to produce with new cohorts of people.
SPEAKER_00I'm extremely worried about that. I don't think we as an industry nor academia has a solution to it. And I thought about that a while ago and discussed it with a colleague that, I mean, at least to me, the reason I can do what I can with coding agencies because I have 30 years of professional experience coding by handwriting, building large systems. And maybe you don't need three decades, but I would have needed at least five years to get to a level where I even know what good design is, what good architecture is.
SPEAKER_00And I think it's unreasonable to expect a company to pay someone to code manually for five years to build up that knowledge. It's just not going to happen, I think. So I think we need to find a new way of teaching people how to think about the software, maybe not at the code level, but at the abstraction level and what that looks like. I don't know, but I think it's, it's critical. I'm yeah, I'm definitely concerned.
SPEAKER_01Yeah. Yeah. It's interesting what you said that maybe we cannot expect, you know, new, the new generation of developers to, to, to think about manual coding, just like we did when we grew. But to think at the higher level of abstraction, to grow that kind of seniority, thinking maybe directly at a different level of abstraction. I mean, it feels true to me, even if I agree with you, I would, I still wouldn't know what that look like as a, as a journey.
SPEAKER_00But it, it sounds doable somehow, but, and I think we need to figure that out as a software industry, because I'm pretty sure that we will need More software developers in the future, but it will be software developers with a different skill set. We need to start growing them.
SPEAKER_01Yes, they probably look very different from software developers from today. I agree with you. Adam, thank you so much. This was a great chat. I learned a lot, great case study and I loved learning also about how your team works. So thank you again for coming again on the show.
SPEAKER_00Thank you so much for having me. Thanks.
SPEAKER_01Thank you so much for listening. If you found this chat valuable, you can subscribe to the show on YouTube, Apple Podcasts, Spotify or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the show. You can find all past episodes and learn more about the show at refactoring.fm. See you in the next episode.
This is a free preview of a paid episode. To hear more, visit refactoring.fm Listen now:YouTube | Apple | Spotify - Today’s episode is brought to you by: TRM Labs Criminals are scaling with AI. Build the technology that...
This is a free preview of a paid episode. To hear more, visit refactoring.fm Listen now:YouTube | Apple | Spotify Today’s episode is brought to you by: TRM Labs Criminals are scaling with AI. Build the technology that...
This is a free preview of a paid episode. To hear more, visit refactoring.fm Listen now:YouTube | Apple | Spotify Today’s episode is brought to you by: Notion Learn more about Notion’s Developer Platform today at...
This is a free preview of a paid episode. To hear more, visit refactoring.fm Listen now:YouTube | Apple | Spotify Today’s episode is brought to you by: Notion Learn more about Notion’s Developer Platform today at...
This is a free preview of a paid episode. To hear more, visit refactoring.fm Listen now:YouTube | Apple | Spotify Today’s episode is brought to you by: Notion Learn more about Notion’s Developer Platform today at...