Your Software Factory Needs a Bill of Materials | Daniel Meppiel
1 hr 1 minAudio
Developers are about to write skills instead of code, and Microsoft's Daniel Meppiel argues the software factory that runs them needs versioning, a bill of materials and signed receipts before enterprises will trust it. Watch all the AI DevCon NYC 2026 talks: https://tessl.co/7jw Recorded live at AI DevCon NYC 2026, Simon Maple sits down with Daniel, of Microsoft's Global Black Belt team and creator of APM (Agent Package Manager), to dig into why teams keep rebuilding the same skills, how to version a skill when the change is in its behavior rather than a method signature, and why skills can only ever be guidance while security controls stay deterministic. Then stay for Daniel's full DevCon talk, Version Your Factory, Not Just Your Code, on agent contracts, signed receipts and the evidence a regulated enterprise needs before it auto-merges. What we cover: - Why developers will write skills and prose instead of code - Why teams keep rebuilding the same skills, and the case for a shared skill catalog - Testing and versioning skills with evals: when is a change a major version? - APM, lock files and an agent bill of materials for your software factory - Skills are guidance, security controls are deterministic - Daniel's AI DevCon talk: agent contracts, signed receipts and auto-merging on evidence Chapters: 00:00:00 - Introduction 00:02:53 - Why developers will write skills, not code 00:05:49 - Why teams keep rebuilding the same skills 00:07:52 - Versioning the software factory, not just the code 00:09:48 - Testing and versioning skills with evals 00:14:09 - APM and the agent bill of materials 00:17:44 - Skills are guidance, controls are deterministic 00:21:02 - Skill catalogs, governance and measuring impact 00:24:46 - What developers do once the factory runs 00:28:08 - AI DevCon talk: Version Your Factory, Not Just Your Code Book early for AI DevCon London 2027, 7-8 June at The Brewery, Moorgate: https://tessl.co/wzi 🔔 Subscribe for weekly episodes on AI-native development Would you let your software factory auto-merge on evidence alone? Tell us in the comments.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
SPEAKER_01We are not going to be writing code anymore. So what are we going to write? We are going to write pros. We are going to write skills.
SPEAKER_03How on earth do we go about a good versioning system where we can recognize that this is a major change to the skill.
SPEAKER_01We haven't figured that out yet. The customers I'm talking to still don't see them rigorously and systematically evaluating their skills. And so when bad things happen, you need to have a way to backtrack and say, the problem was here. The problem was in this skill.
SPEAKER_03The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your host, Guy Fajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. AI DevCon is coming back to London. Our first London event sold out in June. After our second New York event this November, we're heading back to the brewery Moorgate in 2027 with a bigger lineup and two full days of talks from the people actually building with agents.
SPEAKER_03You'll hear honest takes on where the industry's at, real stories from teams adopting AI, the wins as well as the messy bits, and where they think things are heading next. It's on June 7th and 8th. And don't tell anyone but we've got a really great deal for you. Scan this QR code to grab an early bird ticket for just£100. See you in London. Hello and welcome to another episode of the AI Native Dev, and I am wearing a very fetching yellow shirt, which can only mean one thing.
SPEAKER_03I am at AI DevCon in New York, and we are here literally live as an existing session is going on across there. I'm kind of keeping my volume down just a touch because I'm worried that we're gonna noise bleed into the other room. But it's literally uh through the glass doors just there. Um, and joining me is one of the speakers at AI DevCon, Daniel Meppiel, um, who is a director of Dev and Agent Platforms at Microsoft in the coolest department I've ever heard, the global black belt team.
SPEAKER_03Tell us a little bit about that, Daniel.
SPEAKER_01And first of all, great pleasure to have you here. All right, thank you. Thank you for bringing me here. So, yeah, the global black belt team at Microsoft acts as a reference solution architecture team in the commercial business of Microsoft on the field. So uh myself I'm focusing on everything Agentic SDLC, AI applied to software engineering, and helping our largest customers around the globe transform by using AI, uh their software practices, right? So everything about uh you know agentic software factories and everything we are discussing uh in the conference.
SPEAKER_03Yeah. Amazing, amazing. And your session that you're giving at the conference is is version your factory, not just your code. And and it talks about, I guess, the prompt skills, workflows behind the agents, um treating them more as software uh rather that we can test or upgrade. Super cool, we're so aligned on that. Um so tell us a little bit about what you what you go through.
SPEAKER_01Yeah, so you know, at the core of that phrase is is the realization that uh we are not going to be writing code anymore. That's something that I think we agree with. So, what are we going to write? Uh we are going to write pros, we are going to write skills, prompts, as a way to try and automate and get some reliability out of agents, right? And that prose that we start writing, we are starting to put it into modules like skills and other types of files, right?
SPEAKER_01And you know, at the end we we can see that that's what allows us to put together what we are calling a factory, which is an automation, automated process. And so it brings up to life a kind of sort of a new programming paradigm itself. So there are many practices that we applied in classic software engineering with the programming languages we are all used to that can be applied directly to this new way of coding, which is about basically creating natural language.
SPEAKER_02Yeah.
SPEAKER_03And it's interesting because I guess from the from the role of a developer there, there's probably two key things that are important for a developer to do. One is decide what are the uh the actual things that are needed to be built. And then the second one is actually focusing on the factory or the the infrastructure itself. So essentially recognizing and understanding how to improve and make the the underlying agents most effective. So there's almost like two pieces that the development teams need to need to focus on there.
SPEAKER_01You know, more and more developers are starting to shift to uh build what allows agents to actually ship code that can be put in production, right? And and that's what uh what we've called skills, and and you know, it it can be understood as all the infrastructure. When you say all the infrastructure, I think it's it goes beyond skills, right? There is there are MCP servers, there are hooks, there are all these what I call agentic primitives that uh we hook onto the harness and allow us to basically get some reliability from agents so that we can get outputs that we trust much better, that they are gonna be doing the job that we want it to do in production, right?
SPEAKER_01So there has to be obviously a lot of rigor that needs to shift towards building that infrastructure or you know those agency primitives in the form of skills, hooks, MCPs, and and all yeah.
SPEAKER_03And and I guess there's uh there's there's two different types, I guess, of types of skills that people will build. The types of skills that are currently today uh parts of the model that is lacking, um, whereby actually this is a useful skill that everyone will want to use. And then there's this type of skill which is more of a flavor and more of a uh a style, where it's like actually I as an organization will prefer to do it this way, others will prefer to do it a different way. So you know, a whole ton of different engineering teams will be building their own skills and and and sharing and using their own skills.
SPEAKER_03Um what would you say today are you know some of the the the biggest problems that people are continuously falling into uh when creating prompts or you know needing skills.
SPEAKER_01Right. Well the main problem is that teams aren't federating efforts, let's say. They are rebuilding the same skills once and once again on their own sites, on their own laptops, on their own teams, right? So think about organizations with thousands of develop thousands of developers, like the ones I I typically work with. It's it's you know, it's it's they are recognizing that putting down their unique knowledge and and processes to get the job done in a skill is really delivering a lot more uh productivity and uh ROI out of agents.
SPEAKER_01But this is being realized in the team in, I don't know, the team who sits in in Europe, the team which sits in in the US, the team which sits in Asia, right? And so the first error, or let's say the first hurdle to overcome is how do we bring all this together in in one place so that we don't rebuild the wheel each time? And so that we can, you know, cease the economy of a scale of that. Right.
SPEAKER_03And and obviously, when we think about factories, a large part of you know building a factory and doing it in in a robust way is probably gonna be skills-driven, like using skills and using uh you know methodologies that are repeatable to do things a similar way. Um your session is titled Version Your Factory, not just your code. If we look under the covers of you know a factory, what are the types of things that today developers, in your opinion, are not versioning?
SPEAKER_01Right. So I I want to I want to step back a little bit from the versioning part because you know um it's it's an opening, right? And um you know, when when you look at a factory and what needs to be versioned, it's obviously we are looking at the skills, but there is there is something else that is not being versioned today, and I think will be versioned very soon, which is you know, uh we we saw in the keynote from from from Guy uh what you are calling loops, agentic loops, right?
SPEAKER_01You get a skill or a bunch of skills, but you also need a trigger, like what's gonna launch this automation, and you also need verification, like what allows us or and the agent to say, okay, I've I've met the bar, I've I've delivered what was expected from me, right? Like the deterministic checks, the guard rates, right? And all these different components today they are spread around, right? We have the checks on one side, the skills on the other, we have the intent on another, uh, all of this, right?
SPEAKER_01So at the end, I think one of the other things that we will need to start versioning are those loops. Yeah. Which, you know, in my talk I'm I'm calling agent contracts. And I think that has the potential to become another standard, standardized unit for harnesses.
SPEAKER_03Yeah. Yeah. Very interesting. Very interesting. And and when I guess when we talk about tests and a test suite for for that, to make sure our loops and our skills and everything are working well. What does a test look like? What does a suite look like?
SPEAKER_01Right. So we we are already in a world where we have two types of tests already, right? The same way we have two ways of programming now with deterministic programming languages, or with pros, which is non-deterministic, we have the two types of tests as well, which is, you know, on one side very important, the uh classic unit integration tests that I think we cannot we cannot take away from an agentic software factory because that's what that's what will give us trust and confidence that we can ship it.
SPEAKER_01And on the other, we also have tests and reviews that are agentic now, right? Uh because they deliver a lot of value. And and for sure we we cannot confer it the same trust as for deterministic checks, but they are really, really helpful. They help detect vulnerabilities where some deterministic engines cannot detect them. They can uh detect, you know, uh things in the code that aren't compliant based on some regulatory frameworks that we basically have no deterministic checks for that.
SPEAKER_01So those are the two families of tests that we need, including when we evaluate the skills, when we evaluate factories.
SPEAKER_03And it's really interesting when our skills, well sorry, when our when our tests essentially become natural language that an LLM as a judge effectively looks over, you obviously now have you know context, which is text, and you have tests that are text. When we start to version these, I give a tricky question now. When we think about code versioning and we do the semantic versioning, it's super easy to identify we need a major uh semantic versioning bump here, or we need a minor bump based on okay, is this a breaking change?
SPEAKER_03Is this a change that's you know the method signatures are changing and it's actually gonna break other things? So we it's it's clear, yeah often abused, but clear when we should jump to a major or when we should jump a um a minor. How on earth do we go about a good versioning system where we can recognize that this is a major change to the skill and what constitutes a major change? Yeah. How do we how on earth do we even think about that?
SPEAKER_01So we haven't figured that out yet as an industry, in my opinion. But I think you know, if if we go to the to the core of the problem, it's about identifying what makes a change that's uh, you know, an incremental change in the same direction versus a change that you know creates a breaking point, that the direction is switched to another one, that the output that you may get is different, or that there are consequences that require users of that module to do something else, right?
SPEAKER_01And so in order for us to have a consistent way of versioning, we need to come to the point where our evals potentially allow us to detect what uh what changes that we are putting into our skill are creating a deviation in the standard behavior or you know the direction of the behavior that we had originally. So a major behavioral change. Exactly. Exactly. But since we aren't still evaluating skill changes at a scale, like uh I still don't see um the the you know the the users, the the the customers I'm talking to, I still don't see them rigorously and systematically evaluating their skills and and and the changes they they put into their skills, uh then we don't have an easy way yet to say, oh, this change I made into the skill uh is completely changing the behavior, and so I should version it in a different way, right?
SPEAKER_01I think as we get to the point where that becomes commonplace, and that whenever we change the skill, there are evals that prove that that change is good because it takes us where we want to go, then we have a more deterministic way and more uh you know uh solid way to say this is version 2.0 and not 1.1. Yeah.
SPEAKER_03So so in your session, you you show APM, which when I first looked at it, I'm like, oh, you've you've been building a whole bunch on APMs. And actually, it's an agent package manager. Talk us through a little bit about the APM.
SPEAKER_01Agent package manager is basically taking the idea of a package manager, like NPM, like node package manager, to the AI native development world. And it's doing that basically not to reinvent the wheel and solve some problems that we have found in A-nated development that we already solved in uh classic software engineering. Yeah. And I'm thinking about first reproducibility of setups. Like if you have some cool skills and you use them to develop our shared project, why should I ask you to send those to me over email?
SPEAKER_01So I can't just basically install them from a manifest and done. But the other could be traceability as well. Like um reproducibility and traceability, it almost come together. Remember, when you install with npm, you get a log file generated, which contains all the um cryptographic signatures, hashes from the dependencies that you installed so that you can basically identify them as real as the ones you asked for. And so here with APM, you get a log file as well.
SPEAKER_03Yeah, and with my kind of like security background hat on, I'm thinking auditability is very interesting as well. So, like, okay, what was used, what versions of what uh context was used when we built this? But also supply chains, when we think about the supply chain problems that, you know, when we when we go back to the dependency world, supply chains become a big problem.
SPEAKER_01Um, so what are the supply chain issues that we should think about when using context from a Yeah, and this is at this is at the core of the trust gap that we need to close for enterprises to adopt uh, you know, uh agentic software factories. And you know, it in terms of supply chain. So, first of all, agency software factories are gonna be built by skills. We we said that before. So if we have a manifest and we have a log file, of what skills are bringing the behavior we want to the agents, to the agents we have working in our software factory, then we have traceability.
SPEAKER_01Right? If we have the log file, we have traceability. We can track from the output that's generated from your factory back to the skills that play that role in building that.
SPEAKER_03And this is essentially like a context bill of materials, right? Do you ever think that will be something that people will care about or need uh as part of audit?
SPEAKER_01Yeah, I'm I'm very bullish on that one, right? So with APM, today you can generate what they call an agent bill of materials, right? An A-bomb instead of S-bomb, which is basically you take the log file and you look at all the dependencies and transitive dependencies, right? In APM, you can have a skill depending on other skills and yada yada, right? You can have this transitive dependency tree. And so from the log file, it's very easy to say here you have, here you have the bill of materials, here you have what went into my agent, which played a critical role in you know everything that happened when that agent ran, whether it was positive or not so great.
SPEAKER_01Yeah. And so when bad things happen, you need to have a way to backtrack and say the problem was here. The problem was in this skill.
SPEAKER_03And when we when we think about skills essentially acting as a guide to an agent, there are certain um times when an agent will do things very well and adhere to those skills, and actually maybe it will know that, but statistically there will be times when it actually will not ignore per se, but maybe do things slightly differently to how the skill was asking. Um how do we when we think about um you know security being a key thing where we have policies, we need to do things in a certain way.
SPEAKER_03How do we uh think about the difference between guidance and a true control that must be followed? Is this down to the validation level versus you know enforcing something at agent time to build in a certain way?
SPEAKER_01Yeah. Alright, so the skills are always guidance, right? Even if you write in caps letters or you know, you repeat it ten times that you cannot do this, right? Um in capitals, very likely. I mean, you you can always have a model, because we say agent at the end at the underneath it's it's the model as well, right? Um doing something that you said in your skill you didn't want to.
SPEAKER_02Yeah.
SPEAKER_01Right, with uh with unwanted consequences. So we need to distinguish from what's absolutely a no-go, and that goes into the controls, the security controls that we put around that agent, from what's the behavior, um, let's say, what is the direction of travel for that agent? Like what is the kind of process and the way and the tools and the taste that we want that agent to have? Yeah.
SPEAKER_01The role that we want that agent to follow. It's a little bit like when you when you hire somebody in the company. When you hire somebody in the company, you are going to interview this person, right? Because they already come with those skills. You don't have to install the skills on them, right? Uh, but at the same time, you'll give them a batch, and the batch allows them to enter the building, right? And that's the security control, right? And you give them a laptop that has some other security measures. So this is the same for agents, right?
SPEAKER_01On one side we have the skills, but on the other side we have the security controls, which are like runtime, you know, runtime governance of checks, of you know, blocking at runtime what the agent is doing, sandboxes and permissions and all of that.
SPEAKER_03And very uh retryable and deterministic in terms of their exact, totally deterministic, totally deterministic. Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button.
SPEAKER_03It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. Alright, back to the episode. One more question and then we'll jump into your talk. Um, for our audience listening, um they most likely have an existing agentic setup. Uh they probably have a bunch of, you know, maybe some skills, but definitely a bunch of prompts that may be shared and that they use probably quite regularly. What's what would you what would you recommend the first change they make next week to improve their general setup?
SPEAKER_03Yeah. And how do you how should they measure it? How would they, how can they identify that it's actually, yes, this has made a tangible improvement to my day-to-day work. Right.
SPEAKER_01Okay, so assuming that we are in a scenario where the team already has some shared skills and that they are using them and so on, I think one step, if it if they haven't done it yet, will be to centralize those in a catalogue. In a shared catalogue where everybody can go, can read them, can uh see them, can work in. Improving them, right, and can source them from there, right? And that creates already this inner sourcing kind of motion around those skills and that practice. The second one after that will be let's put some governance around that.
SPEAKER_01And I'm thinking particularly about large regulated industries. Once you have that, then let's define a policy of what skills need to absolutely be there, depending on the task, on the project, on the team, which ones you don't allow, which ones you allow in, right? So that we can so that they can work towards some level of standardization. Yeah.
SPEAKER_03And who owns that? Would you say maybe that's probably determined by the scope of that skill, whether it's org level, team level, department level. Do you see the platform team maybe stepping in and owning those kind of things more as a hey, we're producing we're providing this as a service to a gentic developers?
SPEAKER_01I totally see the platform team as the one spurheading the effort around a skill catalog, around some level of governance at the enterprise level. But then you are right, then it depends on the department, on the team, even at the project level. Or individually, even or individually, we we I see a world, and in APM you can do this with with uh with the policy engine, you can have a cascading inheritance of policies at uh repository, uh team, uh organizational level, and so on. And and this mimics very well, you know, what we were doing with uh platform engineering uh so far.
SPEAKER_03Yeah, amazing. Um and how does someone recognize whether that change will actually have a positive impact day to day? Obviously, we have a lot of anecdotal, oh, this worked this one time, it's gonna work every time. How do we actually genuinely know?
SPEAKER_01So once you have built that standardization, that first platform, then observation becomes, you know, the the next unlock. You unlock observability as well, right? Because you can start acting on it. So at the moment, that requires ingesting a lot of locks from the coding agents to see, okay, which skills are being used where and what happens when they are being used. So the same as we were measuring uh under Dora before, you know, before the AI era, here we are going the next level, we are measuring the skills that are, we are looking at the skills that are being used, and we need to connect those to the same Dora outcomes or whatever framework that's being used, right?
SPEAKER_01And when we see uh that the the metrics that are important to us, like whether that's uh throughput of PRs merged by by week, by developer, or whatever that is, uh, then we can say, okay, this brought a positive impact.
SPEAKER_02Yeah.
SPEAKER_03Okay, so Daniel, let's project ourselves into the future slightly and say, we've got a team now, they've got their factory, um, it's built, it's working nicely, it's effective, it's it's performant. What does that leave then? What do we do as developers beyond that?
SPEAKER_01Right. So there are there are two things that I see clearly today. One is there is gonna be an operations team taking care of that factory. You know, same as we had DevOps teams taking care of all the DevOps practices and the DevOps infrastructure, we're gonna have that. And that's not per project, that's across an organization, across departments. It can be, you know, I can I can see a place where we may have tens of different factories working at the same time in an organization. Same way we have different DevOps pipelines uh operating per project, right?
SPEAKER_01Yeah. So there are gonna be uh differences there or or or variations. But the the second thing is, okay, so that then where does where does the developer then focus on? And I think we're going to move to a place where we're gonna be focusing on, first of all, intent and decisions. Intent and decisions, right? Which is what should we develop and why? What should we ask our factory to develop and to to ship and why, right? And so the focus is gonna be much more put into you know humans talking to each other, exchanging ideas, or of what should we build and how will this help our users?
SPEAKER_01And I think that's an exciting place. I do agree.
SPEAKER_03Have you not just described a DevOps platform role and a product manager role?
SPEAKER_01Kind of, right? I think I like to stay yet at a place where we deconstruct those labels from the past and wait to see what are the new labels of the future. Yeah. Because when we stay with the core ideas, I think we are freer to invent what's next. Amazing.
SPEAKER_03Now, Daniel, you are good, you haven't given your session yet at AIDEF CON. Um, so I'm gonna ask uh present Daniel here to introduce future Daniel, who's gonna present on stage. Uh what are you what are the topics that you're gonna be talking about? And please introduce your session.
SPEAKER_01Right. So, you know, the session is titled Version Your Factory, not just your code. That's the opening, right? And and the idea behind that is that there is one element that's missing today from the plethora of agent primitives like skills, MCPs, and hooks and so on. And that's something that I call agent contracts. Like, what's the single unit of work that helps you build a factory? And that's what you are calling loops in Tesla, right?
SPEAKER_01But from there, I will go into what is missing for enterprises to close the trust gap and start operating agency software factories in production. And that's gonna be all related to things like you know, the security, the trustability, the integrity, the provenance checks, and the supply chain basically of an agentique software factory.
SPEAKER_03Daniel, massive thank you. Really, really appreciate you joining us. Um, and over to future Daniel.
SPEAKER_01Thank you, Simon, and thank you for bringing me here. Excited to present very soon.
unknownThank you.
SPEAKER_01All right, so hello. So I just wanted to say that from you know uh what my role is, I'm not in engineering or product, I'm in solution sales in technical, technical solution architecture. So my organization is all about advising large enterprise organizations on adopting AI coding agents and transforming towards agentic SDLC, right? And that's been the core of my work uh throughout uh GitHub, now Microsoft, and I'm gonna talk a little bit about you know why we need to version factories and where does that lead us?
SPEAKER_01And that's gonna be coming from experience in the field with large enterprise, often regulated organizations. All right, so let's think that we are in a bank, right? We are developing software for a bank which runs maybe its own uh core banking system, and you know, we are tasked with creating a change in code to change how payments work. So I'm reading here, hey, we need to hold wires of more than 10k dollars for internal approval for whatever reason, right?
SPEAKER_01And let's say our factory generates this pull request, so the question is will you auto-merge it? And I'm pretty sure every bank today will say no. Right? And we can have risk classifiers, we can have verifiers, we can have a lot of these things, but the answer to that question will very likely be no for some time. And what I'm gonna focus on is what do we need so that this can be auto-merged one way, one day, confidently.
SPEAKER_01Alright, so recapping, I think by now we all know what an agentic software factory ideal or concept is. We get the intent in, our agentic software factory runs an automated agentic process to basically transform this into code, code changes. Now, the problem with that is that we create new bottlenecks, and this has been repeated over and over and over yesterday. Uh, one of those bottlenecks is CI, we get so many pull requests, CI runs everything again, and and this becomes a bottleneck, and the other one is us, humans.
SPEAKER_01And we just heard in the keynote that we need to remove all humans, right? And if we don't do this, a genetic software factories cannot scale in large organizations either, right? So that's another piece of the problem that we need to solve. Today we seem to be converging on the idea of a risk classifier, right? Which is basically, hey, uh we identify what the code change is changing and uh then route it to a human only when the risk is uh you know justifying that.
SPEAKER_01But this is, as I said before, this is not enough uh for large regulated enterprise organizations, right? Because even when we auto-merge in this scenario, then there are a lot of questions about which genetic software factory created that change and do we trust it? There is no trust yet, just based on a risk classifier. Every single time I talk with large enterprises going through this journey, there are four elements that come up in conversations all the time about what's needed.
SPEAKER_01The first is verification. How can we be sure that what the Agentive Solar Factory produced as code meets the intent we have and does it well? The second one is about hey, did that code change uh was that code change authored following our standards, our platform engineering standards, our compliance frameworks? Uh did it uh uh did it uh follow our coding styles and so on, right?
SPEAKER_01Which are often already in place? The third one is traceability. Like, you know, if I have a code change that I have merged that came from an agentique software factory that used, I don't know, how many skills, how can I trace that back? If something goes growing in production, how can I trace that back to the skill that I need to improve or the factory that was compromised? And the th and the last one is security. And and it's a little bit related to the to the rest, which is uh, you know, agentic solver factories become a new surface of attack.
SPEAKER_01And it becomes a very lucrative and a very interesting surface of attack because now these agentics over factories are gonna generate all the code, right? They will remove all humans from coding. So the security aspect is super important. Now I want us to think about how we built trust for classic software before the AI era. And how did we build that, technically speaking, right? To me, software runs on a fabric of trust, kind of, if I can use that metaphor.
SPEAKER_01There are several elements that brought us to where we are today, where software runs every uh a lot of mission critical applications. It all starts with versioning. It starts with versioning because versioning is about giving an identity to software artifacts, the fingerprint. Sure we see uh we see version 1.0, 2.0, but behind that we have a digest and we have a fingerprint. Once we have that, we can start creating the rest of the elements that compose this fabric of trust, like inventories, provenance, the evidence, like and we have created a lot of standards so far to standardize this and speak the same language when we check what should we run and what do we trust, what software do we trust to be run in our environments, right?
SPEAKER_01Like Cyclone DX, SLSA, In Toto, and so on. Uh we invented signatures, and finally, ultimately, with all of the above, we could create admission uh admission checks. Basically, a policy that's unique to each enterprise, which says, all right, I'm gonna check the inventory of the bill of materials of this software, I'm gonna check which organization created it and signed it, right? And based on all those things, I'm gonna validate that this software can run in my bank.
SPEAKER_01All right. And so the question is when agentique software factories generate software, where do we leave all of this? Where is the fabric of trust? Right? I don't think we have a lot about that today yet. So we need to build that. And to build that, I like to think about a factory as an agency program, as software. Because we saw yesterday, for example, how uh shout out to Tesla with uh the loops concept and uh some other vendors like ourselves with GitHub uh automations, we call them automations.
SPEAKER_01At the end of the day, it's a bunch of skills, a bunch of prompts running a process that we have written in in natural language, right? And a factory is a collection of those that are chained together. That looks to me like a program, like a software program chaining functions. And so if it's a program, we should version it. Because identity is the first thread of the trust fabric. The other problem we have to solve today is that in the proto factories we're starting to create, we are gluing together a lot of loose parts, right?
SPEAKER_01We saw skills, prompts, scripts, maybe some proxies and and elements on the networking layer, on the sandbox layer, environment layer, infrastructure layer. It's a lot of loose parts, right? So we have all those loose parts with which we can compose a factory and build that ourselves. And the other alternative is just go with a vendor, right, which are starting now to propose a way to create factories, right? In the case of GitHub its automations, in the case of Tesla its loops, in the case of Codex, uh uh no or cursor, it's also automations, right?
SPEAKER_01But that's getting us locked in, right? It's a nice UI, UX and so on. But once we build our factory there, it's very hard to take it somewhere else and run it in another hardness one day, right? So what's missing in between, in my opinion, is a single unique way, standard way to define the smallest element that defines a unit of work, of agent work. What do I mean with this?
SPEAKER_01This metaphor I think is gonna make us uh converge, so I call it agent contracts. Right? I don't call it automations, I don't call it loops, I call it agent contracts. In the same way that when we uh sign up for a job with a company, we sign a contract. I do think that what can help us standardize and make software factories portable are agent contracts. A simple markdown file where we define the assignment or we admit we we basically has an as input, external input, the assignment.
SPEAKER_01The assignment can change. One day, one day the agent can be asked to perform one uh one software change in one piece of software, the next day another one. But what doesn't change is the role, the role in which the agent signs itself for in this contract. The dependencies, what's made available for the agent to perform this job. And the budget, like how much are we happy to pay for, right, for this job to get done. It resembles a lot to kind of work contracts that we sign in real life, right?
SPEAKER_01And what we get back are the results of the checks, the verification, and the deliverables, like software changes or or change diffs of pull requests, right? So once we are able to define agent contracts in a simple markdown file with front matter YAML, or you know, the that's a possible format for it, then we can version it. This contract can link the skills and everything that's required for the agent to do the job.
SPEAKER_01And it becomes a program, an agentic program. It becomes a program that it's version that allows composition, we can start composing contracts between themselves, it allows portability if more harnesses and vendors accept it, and it's identifiable and it allows us to start building the fabric of trust. Alright. So a factory can be thought of as a graph of contracts. Like if we have an agent contract that's for planning, another one for specification, then another one for building and implementing the actual code for documentation, for reviewing, for a lot of different tasks, we chain them together and we have our factory.
SPEAKER_01We give it a version, we get the fingerprint, and we have the identity of the factory. Right? And so if a factory is a program, then once we have that definition, what runs it? Right? And so I want to get my uh I I want to get it to run right now. So let's let's imagine we have basically the um the pull request example that I showed at the beginning, the intent here uh defined in markdown.
SPEAKER_01Whole large wires to new pages, right? This is intent, it can come in the form of uh of an issue, of uh markdown file, of a message from a colleague on a Slack, of whatever it is, right? I have defined here, I have defined it here in a markdown file. So here we have that, right? And this is our core banking uh software, contoso payments, whatever. Okay? Great. And then we have our factory. This is our these are all public repositories.
SPEAKER_01You will get access to that later. And so in our factory we have uh a contracts folder. And these contract folders we have a bunch of contracts, like they're from the graph that you've seen before, and for example, the build contract, which specifies, hey, as input I need the intent, the request, then the specification from the previous contract. I will produce a diff and I will import this specific skill, right? And I have this budget and so on, and this is the the role, the role uh of the uh agent contract.
SPEAKER_01All right. So those are those are our program files. And then we have this APM YAML file, which can be thought of as a package.json file uh in the npm world. Uh this is from agent package manager, and it's defining our dependencies. Like, what are the skills that are required for me to run this factory? And once we uh have this manifest here defining the dependencies and they get installed, then our contracts can link them, import them as modules, as we do in in uh in uh usual uh programming languages, right?
SPEAKER_01And so with that, I'll use APMX, and you can see the analogy here to the NPM world or UV or whatever, right? We have NPM NPX, we have UV, UVX, you can hear see the analogies, and I will use that to run uh the factory. And I will run it, and I hope you can see that in two terminals. One is gonna basically run the factory on Copilot as a harness. I'm gonna launch that right now. And on the right hand side, it's gonna run it on open code.
SPEAKER_01All right. And you can see that the first thing that's happening is that Agent Package Manager, which is akin to NPM, is reading the manifest, installing the dependencies, locking them in a log file, and then the factory can run as a program. And we can see here the chaining of the Ethereum contracts. The first one is called planning being run on the hardness of choice. So this is a demonstration of how we can have a portable open format that can eventually run on any harness.
SPEAKER_01Right? I will leave leave that running, okay? And let's come back to the presentation. Okay, so APM, I have uh introduced that briefly, is a package manager for agents. It's an open source project. I started that that independently, and then uh as I am employed by Microsoft, I transfer it to the Microsoft open source organization. All right. The uh intent here is to move it to a neutral foundation, working on that. And the idea is pretty simple.
SPEAKER_01We have a manifest with the dependencies, we have a log file that's generated when we install, and then we can create a policy in the enterprise which says what are the dependencies we want to allow in and what are the dependencies that we want to deny, and many other cool things. All right. And so APMX is about running these uh modules as programs, the contracts as programs, right? Which depend on scales, MCP, hooks, and whatever it is, right? So So we can run it on any coding agent, and as a result, we are gonna get our pull request, our div, but we are also gonna get a receipt.
SPEAKER_01We are gonna get a receipt, and that's the second important element here. One is the contracts, and the other is the receipt of the run. And so why is the receipt important? Alright, let's go back to uh to how this works. We get intent in, we have the factory as a as a chain of contracts or loops or automations or whatever it is, right? It generates the div, of course, but also the receipt. And the receipt is signed. The receipt is signed by whichever machine which run our factory, which we want we choose to trust as the runner of our factory.
SPEAKER_01All right. Then we have our rules, our policies, which define, hey, which factories do we trust? Which skills do we allow in? Which ones do we deny? Oh, we heard about a skill that was compromised, we're gonna deny it there, right? It's our set of rules. And then we have a gate. And that's typically implemented in CI CD in the pull request targeting main, right? Uh those are the checks that can run in CI C D. And I've chosen to implement that as a demonstrator with a command that's APMX audit.
SPEAKER_01It's just gonna uh read the receipt, read the policy, check whether there is, you know, there is alignment between those and block the uh the pull request or auto merge it. Alright, so let's look at the receipt. The receipt is based on uh open standards that we all know about in the software supply chain world. So if we look at the first block, that's based on SLSA provenance and it gives us traceability. It allows us to understand that the output, the div that was generated, comes from this factory.
SPEAKER_01Because they are chained uh through their fingerprints, through the digests, right? The versions, right? Then we have the results of the checks, the checks that have been defined in each contract and are immutable by the agents. The agents cannot change those checks. Remember that agents cannot be trusted, right? They can change the rules of the game. So we need to also cryptographically um change the the results of the checks and the checks themselves in the receipt, in the evidence, all right?
SPEAKER_01And that's based on in total. And then we have our agent bill of materials. It's acting to the software bill of materials. It's about what's the composition of the factory, right? And finally, we sign it. We sign it with a trusted signer that cannot be the same machine that runs the factory, because we cannot trust the agents, right? So it has to be something else. And that gives us security, right? And ultimately, it's this receipt which allows us to build that fabric of trust, right?
SPEAKER_01So we get the identity as the factory version and fingerprint. We get the inventory as the agent bill of materials out of the APM log file, right? We get the provenance with SLSA. We get the evidence within total, thanks to the check results and the checks themselves, change to the receipt, change to the output, and we get uh you you know we get attested provenance with the signature, and ultimately we can create a gate to auto-merge based on actual evidence, not just risk classification.
SPEAKER_01Right? With a simple command, for example, APMX audit. My goal here is not to position this tool or that tool, but what's needed, what we need as an industry to create trust. So what happens then? All right, so the model is pretty simple. We have the receipt, we have the gate, we have the rules. We have a command, APMX audit, that we can run on pull requests as a check, which is gonna look at the evidence, look at the diff as the output, make sure that there is the integrity, the provenance, the checks passed, that there was no tampering, that we trust the designer, that the policy that we have defined is respected, and then we're gonna have a result, either valid or invalid, right?
SPEAKER_01And that's the verification of you know that fabric of trust that we have chosen uh to to basically accept, right? All right, so I'm gonna move faster here. Okay. What this allows us to do is to basically forgo human review, forego a lot of CI processes that are creating bottlenecks, and auto-merge, but not just based on risk classification and our personal confidence or very smart engineers we have hired who say, Yes, I trust my factory, but actual evidence that we can trace back.
SPEAKER_01And then when an auditor comes in, because we are a regulated organization like a bank or a pharma company, if something goes wrong, we can point exactly at our controls that follow a certain regulatory framework, we can point back to the skill or the factory that we decided to trust, we can point back to what went wrong, and we can take action. And we can place the responsibility where it has to be. Alright, so there are several questions that this approach help us answer, right?
SPEAKER_01So for example, production broke. And so if production broke and everything in production was built by a factory, then the question is which factory is responsible and which step precisely? Well, because we have this receipt that changed the outputs to the evidence, then we can trace back from the source code back to the receipt and back to the factory. Thanks to the versioning, which starts everything because it assigns an identity.
SPEAKER_01Another question we can answer is hey, a skill in the open source world that we are using was compromised. So who's exposed? Just look at the agent bill of materials of every single receipt that you stored, right? And then find which factories are using those and patch it. Revoke it in the rules, the gate will start forbidding any output from those factories from going to production until you fix that. Alright. Okay. So then next question.
SPEAKER_01The checks passed. The checks that we have defined in the contracts, which are exactly our quality bar now, they passed. But did the agent change them, right? We have seen agents gaming the checks uh consistently. So hey, checks live in the contracts. They don't, they they are not expected to be created by the same agent who work in the factory, so they cannot be changed. They are crypto fac cryptographically tied into the receipt. So if somebody tampers those, you are gonna find out.
SPEAKER_01And they are enforced in the gate with the audit command, right? So it's not possible to tamper those. Alright, and so let's imagine, hey, we want to add new checks, because of course the checks at the end become here our bar for auto-merge, for trust, right? So how can we ensure that they become mandatory? Well, if we create new checks, this changes the factory. And so we can basically release a new factory version, right? If we change the checks which are tied in the contracts, then we have a new version of our factory.
SPEAKER_01If we create a new version of our factory, then we can't go to our policy file and say, now we are only going to accept uh divs coming from uh factories above version 2.1, for example, right? And so our APM X audit command, the gate, will forbid any output coming from uh older factory versions. All right. And so, well, you there is another question that we can answer. You know, in the policy, we can also define, hey, what are the skills that the factories can use, are allowed to use.
SPEAKER_01Again, we have that policy in the rules, uh the the dependencies are tied in the manifest in the log file, we can check uh with the audit command and we can gate based on that. Those are just some examples of what this approach can help us with. So now if I go back to the factory runs, I can see on the copilot run that the different contracts were run: planning, specification, build, documentation, review.
SPEAKER_01What allows state to flow through the contracts is basically uh files, right? So it's markdown files, it's diffs, it's whatever files that go into disk. So a contract generates files, another contract will consume those files, right? We see it completed, and we can see the outputs it generated, the receipt that was generated in Intoto uh in an Intoto file, and um I can run the audit command quickly to show, hey, now eventually if we run this audit command, which should run in CICD.
SPEAKER_01Can we check that the integrity is there, that the provenance is the one that we uh that we decided to trust, that the checks run, that they were valid, etc. So we can see that happening there locally. Of course, here we didn't sign the receipt, so the the check doesn't pass. Eventually, what we need to do is to add that uh add that command as a check to a pull request. So I have here a pull request generated by the factory, and as a check, I have a single one, the audit command, eventually we should have also code scanning and stuff like that, which runs the APMX audit command, and the uh basically all the checks pass, the receipt is signed, it was signed by the environment which ran the factory, you know, it can be CICD, it can be a Kubernetes cluster, it can be wherever you decide to run your factories, and that uh becomes a valid statement that allows us to say, okay, can we auto-merge this now, right?
SPEAKER_01And we have traceability and all those properties I spoke about. All right, so I'm gonna close by saying we s we seem to be all converging into something like some sort of definition of the smallest unit of what a factory is. You know, Tesla was speaking yesterday about loops, we have a lot of vendors speaking about automations, I'm speaking about contracts. What I do think is that we need a shared format.
SPEAKER_01If we do want portability of agentic software factories, we need a shared format, a shared standard. And we need not to reinvent the wheel. We created a lot of this fabric of trust in the past for classic software. Let's not reinvent the wheel and use the lessons learned to build the same fabric of trust for agentic software factories. And let's do that in the open, right? So we need to define factories as code in an open format. We need to version them because it all starts with versioning, which is identity, and we need then we can then gate on evidence, on actual evidence, and it and we get the trustability, the provenance, the portability.
SPEAKER_01Uh we skip CI, we skip PR reviews. And I think this is how Agentic Sober Factories can scale to large regulated enterprises. Alright, so this is a proposition for an open format that I have published. Uh it's a draft. I think this is the perfect conference to invite you to participate in the conversation. I hope that uh from there we'll be able one day to say, hey, we're gonna change the payment systems in the bank and we can even auto-merge.
SPEAKER_01All right, thank you very much. With that, I leave you with the QR code for all the content. Thank you. And any questions, I think we have one minute. All right. Thanks.
SPEAKER_02Thank you. I need to talk to you about the content. Thank you, Daniel. That was an amazing session. Um are you gonna be doing questions now or do you want to get them off the stage?
SPEAKER_01Uh if we have time, then I can pick a couple of questions.
SPEAKER_02Oh, there we go. I'll come down.
SPEAKER_00Let me come down. I can do the exercise.
SPEAKER_04Um so as part of the checks that you're defining as part of these kind of these agents. First first off, I love the kind of like OPA design of all of this. I think that makes uh a ton of sense. Um but in regards to the checks, um, are are you kind of uh alluding to kind of them almost being a replacement more for like unit and integration tests um in that, or is it more specific, kind of like evaluated assertions of the specific code that you're producing as part of that individual agent execution?
SPEAKER_01Yeah. Yeah, so the checks have to be deterministic, in my view. The checks that you define in a contract ought to be deterministic. Because that's that's basically the hardness around the model that you can basically trust, that elevates trust. So a check could be hey, what is if we have a contract for writing specifications, then maybe there is a specific format in which you want to make sure that the specifications are written, right? So that could be one check.
SPEAKER_01If you have a contract for performing pull review, uh pull request reviews, then maybe it's about the format. Maybe if you have one for building uh divs, actual code, then maybe it's about, hey, uh you can attach this piece of code, or you have to follow this style, so linting and stuff like that.
SPEAKER_04Yeah, because I think of like in a world where like your organization that may already be following over, where like it's part of your photo check maybe that like your CI has and things like that, but those are searching to staging and so on and so forth. Um I just refer to kind of saying like it helps reduce the load on CI exactly how that kind of intersects with uh uh like uh main types of changes. Like the problem somewhere is that like CI is kind of main changes, and like you kind of like retest on CI or you have retest on merge, or you can see it.
SPEAKER_01The idea is that you skip test uh running the checks again in CI because you have the evidence that they have run. You have trusted evidence sign that they have run and they have passed. So the idea is that you can you you should you don't need to run them again in CI centrally. So shifting left testing as much as possible in a in a trusted way.
SPEAKER_00Time for waterfall. Sure. And hello. Um waterfall was a bad idea for humans. Why would waterfall be a good idea for AI? Sorry, can can you because there is a lot of echo. I didn't hear you well. Uh so waterfall was a bad idea for humans. Yeah. Why would waterfall be a good idea for AI?
SPEAKER_01So, you know, I think waterfall did very well in some situations, still does well in some situations, and then I agree with you that it's not a solution for everything. I think your question comes from the graph that you've seen, which seems to be a waterfall approach. I don't think we are gonna see forms in which that has nothing to do with waterfall. Even when you do uh uh Scrum and all this, you follow a certain approach to something. We are speaking about automation here.
SPEAKER_01So I agree with you that waterfall is not the answer for everything, and this was just one example. I think we can very likely be able to model another sort of process in the form of contracts as well.
SPEAKER_03Great session, Daniel. Thank you so much uh for presenting that at AIDEVCON. And of course, by the time that you're watching this at home, DevCon has officially finished. But every single session has been recorded, and you can click on the QR code on screen or in the show notes if you're listening on audio only, and that will take you directly to all the recordings at AIDEVCon in New York. Thanks for listening. Bye for now. The AI Native Dev is brought to you by Tesla, the package manager for skills and context. Your host is our guy Pajani, and me, Simon Maple.
SPEAKER_03Our producer is Tom Dowler. The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tesla offices in central London. Visit Tesla.io forward slash community to learn more, and I hope to see you there.
Paul Stack helped build Terraform. Now he says he'll never write another line of infrastructure as code, because an agent harness runs his team's production infrastructure instead. Watch all the AI DevCon NYC 2026...
Automated software delivery is moving faster than the people who used to approve every release. LaunchDarkly co-founder and CEO Edith Harbaugh tells Guy Podjarny why rollback should be autonomous, why one-shot coding...
Agentic coding has outgrown the text box. Maggie Appleton, staff research engineer at GitHub Next, argues that walls of CLI output and AGENTS.md files serve coding agents far better than the humans steering them. She...
AI attackers now move at machine speed, while most security teams still move at meeting speed. Former Robinhood CSO Caleb Sima has seen an AI attacker up close, and he argues the only way to keep up is to take humans...
Your coding agent should never hold a credential, and in a software factory that rule is only the start. Recorded live at the AI Security Summit in San Francisco, Simon Maple sits down with Eli Aleyner (Docker), Marcelo...