Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

DevOps and Docker Talk: Cloud Native Interviews and Tooling

DevOps and Docker Talk: Cloud Native Interviews and Tooling artwork

CI/CD via agent tools: Skills, CLIs, MCP, and more with Semaphore

Aug 13, 20261 hr 17 minAudio
  • Apple Podcasts

What does an AI-native CI look like? The Semaphore team joins me to talk about their focus on making your agent harness the gateway to testing, fixing, and deploying your code. Video podcast version here: https://youtu.be/HzXAdtVrW70 ★Show Links★ Semaphore https://semaphore.io/ Semaphore open source https://github.com/semaphoreio/semaphore SLSA Security Checklist https://slsa.dev/ Creators & Guests Marcos Filipe - Guest Cristi Cotovan - Editor Bret Fisher - Host Beth Fisher - Producer Marko Gaćeša - Guest (00:00) - Introduction (00:29) - AI Native CI Vision (04:54) - Bret's Update (09:03) - Semaphore AI Journey (22:39) - Open Source Platform Shift (37:28) - Plugins CLI MCP Skills (43:11) - Work Until CI Is Green (45:42) - Test Boxes and Agentic CI Loop (57:39) - Sandboxing and Least Privilege (01:10:12) - Roadmap SLSA and Getting Started ★ Support this podcast ★

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Bret: This is DevOps and Docker Talk, and I'm your host, Bret Fisher. In this episode, we're gonna get into the weeds with agents and CI. I've got the team from Semaphore on the show, Marcos and Marko, who joined the show to really just talk about their transformation over the last year. Now, if you've been around a minute with this show, I had Semaphore on probably five years ago, four to five years ago, and we talked about sort of typical CI/CD stuff, and this time it was completely different. just in the last year, they have completely transformed their product

and their tooling to be AI first, and we get into what that really means, and it's not… In this case, I don't feel like it's a buzzword. I think it truly is an optimized way for interacting with your CI from your agent harnesses. And two, they open sourced their platform, so you can now run it yourself if you don't wanna pay them to run it. So they offer the typical options of running their open source CI or running it on their platform, paying for runners and having them run their, their runners. They actually, just recently did a benchmark comparing themselves to a lot of the other CIs and saw that they were cheaper and faster than the others.

we dug into exactly what it means to just drive your CI completely from an agent harness and not just sort of in the contrived way that I've been doing it, which is, you know, telling it, "Hey, can you go check on that CI job? Hey, can you make sure that that git push commit thing happened?" And, like, all the… Like, they've sort of rethought it from, I would say, first principles of Why does CI running, or why do tests or workflows that I need to run around my code, like linting, testing, and all the different forms of that testing, security scanning,

like why do I need to wait on a git push in order to kick those things off? What would happen if my agent could know everything about my CI just from starting up, have all the tools and MCPs and skills and command lines, have everything it needs on every project, and then when I start a feature build, it knows that it can spin up testing and all the other various workflows I might have without needing to make a commit or pushing code. And they have multiple features that sort of come together in something they call test boxes. And so we get into a lot of that because that, to me, has been my vision of where

I think CI/CD is going for agents, where they are free to run CI anytime they want. They don't necessarily have to run it through the PR typical workflow just to be able to run tests and run lints and do scans and make sure that their generated code is gonna work, and that feels like what they've just built. I went through their backlog of blogs, and for the last six months or more, they have been posting very regularly about their updates to their open source CLI tooling, to all the dozens of skills that they've made for their CI, everything

from helping it automate the fixing of flaky tests to essentially onboarding yourself to their entire platform by just installing the Claude Code plugin or the Codex plugin and just saying, "Hey, set me up on Semaphore, build me some workflows, and keep working until they're green." Which is a pretty, I don't know. That sounds like a demo type of prompt, but it seems like that's what they've built and that they've designed this whole platform and their skills and their workflows and MCP tooling around just letting the agent cook and not having to worry about whether it's figuring out CI or whether it needs to

go check CI again or whether it needs to fix that thing that it pushed into CI. And it just feels like it, they're streamlining the whole process, and I'm excited to get into it. So let's jump in to Marcos and Marko tell us their journey to building an open source AI native CI/CD platform all right, let's get the titles and locations out of the way. Marcos, what do you do and where are you from? Marcos: Well, I do software, but I'm software engineer in Semaphore, and I'm here from Natal, Brazil. Bret: Nice. Brazil.

Marko, same question Marko: Yeah. So I'm, responsible for the product at Semaphore, and I'm calling in from Slovenia, Europe Bret: nice. and you're now alumni of the DevOps and Docker Talks show, so welcome back. We talked a lot last time around sort of how Semaphore was optimizing CI/CD, and we were f- we were talking a lot about, local testing back then. I mean, this was pre-AI. It was almost like, is there anything relevant in that conversation anymore? Because everything has changed, especially when it comes to automation,

and I'm excited to get into it because you all recently went, well, in the last year, like the rest of us, you went through a journey as a company as well as individual developers and figuring out what agents can do for you. Marko, tell us about the journey of, like, how did you start s- first thinking about CI as something that needed to sort of be reinvented? I don't know if that's the ri- maybe that's a strong word, reinventing. But, I feel like there's so much opportunity for changing the way that automation and CI testing works now that it… how did you come about this?

What's that origin story? Oh, hey Thanks for listening. And I got a couple of quick announcements about stuff that I've created. A lot of this is free stuff you can get right now. Just in case you've not seen some of my other videos or stuff I've been doing on my newsletter, I've created a GitHub security scanner that is an open source project. It's called GASA, GitHub Actions Security Assessment, and it's expanded to even have more rules. I think I'm about at now a dozen rules on it that scans your repos,

including the entire organization, and the focus of it isn't to replace other linters like the GitHub Actions Linter or Sysmor or other scanners like Poutine that are all really great. It's meant to supplement those and solve my own problems as well as my clients' problems, and I just thought I would open source it to give it to everyone, and I've spent a lot of time trying to make it a solid tool. Actually just spent some today, time today in Fable Model building out some better testing frameworks to test it against real organization repos as a part of its PR testing.

But the whole purpose of this is to scan its personal repos or even repos you might wanna use, or mostly your organizational repos all at one time to give you this report on your stance of GitHub Action security. And it looks not just at the workflows, but also the repo settings on the back end. And this is based on my half decade of consulting around GitHub Actions, and there's no agenda there. It's completely free. I've actually been using it to implement and upgrade the security of my own repos to make sure that I'm not as vulnerable to some of the recent supply chain attacks.

So the whole reason of doing it was people were asking me to help them lock down their supply chain against the attacks we've seen over the last year on GitHub Actions, particularly around open source repos, or even using other people's GitHub actions in a more secure way. It will guide you through how you can change and improve the security. It won't actually change all the settings for you, because I'm not quite comfortable with it going in blast radius of your entire org. But go check that out. It's on my GitHub, and the link is in the show notes. Next, I had in July a GitHub Actions security workshop for free

because Chainguard sponsored me. I was actually gonna charge for this workshop 'cause it was hours worth of hands-on learning around how to use your own agent or your own hands to actually lock down your GitHub to make sure that you're not vulnerable. So it's walking through what this GASA tool does, but also why these settings need to be locked down, why you need to care about those particular settings, and what happens when you maybe do one thing in a workflow and another thing in a security setting in your admin settings, and those two things conflict

and cause a security potential risk that GitHub isn't probably warning you about. So it's meant to supplement all the current stuff out there that really just focuses on the workflows, but don't actually also consider the admin settings of your workflows which are behind the scenes, right? People can't see those settings, and they're not usually easy to find. So anyway, you can go look at that workshop also free on my website. Thanks to Chainguard for doing that and making it- Free for everyone. And finally, if you didn't know about my Agentic DevOps Guild, we have been

meeting weekly since March, where we-- I think we're, like, twenty-five weeks of weekly meetings, and we're now over fifty teams of people in there solving problems for their team on how we're gonna manage agents, how we're gonna manage the security and the sandboxing of our AI for platform engineering use, DevOps, anything related to cloud and Kubernetes. If you're somewhere in an ops role, this was the program designed for you. It's not just meetups, but I'm also creating courses. I've got two courses in the works right now where videos drop on a sort

of a random basis, but based on what comes out of our meetings, I turn that into lessons for everyone to adopt. And also that comes with workshops that we do, including this Gaza workshop that was last month. And those workshops are meant for the guild. The guild will always get those for free, even if someone doesn't sponsor the workshop. But go check that out also on my website, and I hope that helps solve some of your AI adoption problems. Back to the episode. Marko: Yeah. So, so, s- with all this automation and, the, the CI is becoming even

more important for the folks, especially that the whole development cycle is, getting shorter and there's a faster turnaround time. So, we started thinking, how can we enable our customers to kind of get on that train even faster and leverage the whole technology that is emerging? And of course, when we started thinking about it, the first thing that we did, what is it? Like la- last quarter of last year, we created an MCP server because that was the thing back then, If, everybody was doing that. Bret: Yeah, it's crazy how early MCP was.

Like, it was pre-skills, I think pre-agent files really. Yeah. Marko: Yeah. Yeah, for sure. But that was the thing back then. It's kind of like everybody was like, you know, "Hey, that's the way how we are going to connect to different tools." And I think I believe it still is, but as we were starting to develop our internal processes around agents and scale our agentic use, we have realized that it's maybe something different, and it's not one or the other thing, but it's actually both things or all things because all the developers, they have a special preferences, and they have their good reasons why

they prefer one, one or the other. But working with our customers, what became very obvious is that, we have this new surface that the developers are using. It's not it's not before you had like, I don't know, five or six windows that you were switching through in order to do your work as a developer. And nowadays it was becoming more, more clear that either you upload code or codecs or, pick a, a, a coding agent, it's becoming your only surface. And you want to be able to, to talk to it and that everything

needs to be done, there. And then we were fine looking into "Hey, what is the way, how can we interface with the coding agent of our customer's choice?" Yeah, in a very efficient way and give it everything that, that the developer needs to do the work, that the developer needs to do. So, in essence, Semaphore is all about developers' productivity. And then for us this was like, "Hey, okay, we need to get in like we need to create an interface that feels native for this new surface coding agent," and that's how the whole Sem AI emerged.

And it's, yeah, Semaphore for AI. The that is all about it. And then, as we were developing our own hands-on experience, we were kind of thinking like, "Hey, how do you connect efficiently with the coding agent?" And then, Marcus was our head engineer on that project. He came up with the concept of "Hey, let's create the CLI." But the CLI is not only commands, but it's a mix of the commands and MCP server that is a local server, so that the developers can choose their preference. But also agents themselves can also like, use whatever they prefer at the same, at

the time, because we also noticed that, for certain tasks, developer, agents have the preferences for some reason. So it's that, that was one of, one of, many, confirmations that it's not one or the other, but it's both. And then we try to make it as, as optimal for the agents as possible. So everything is you know, JSON native and all the commands are very well documented, not for the humans to read that, but for the agents to be able to use them and understand the use case and a lot of examples.

But from the product perspective, my kind of vision for this interface was that, hey, let's enable people to have the natural language in a conversation with the CI and then lower their bar- that barrier of entry. Because in order to deploy a, a kind of you know, put in production a CI/CD tool as you… I can imagine all the viewers know it's a lot of work. You had to learn the platform, develop a com- a, a configuration, make sure to test everything.

Bret: Yeah. Learn how the files work, yeah Marko: Yeah, learn. Marcos: The first try was SemaBot, right? Marko: Exactly, yeah. And yeah, so that, that was kind of like also in the, in that during the journey we had the OpenClaw became a very big thing, and then we did our own version of OpenClaw for CI. But there are a couple of things from the product perspective that, that was very important. First one is that, since we are open source but we are also like serving a lot of enterprises, we are all very big on security. And then if you're in an enterprise environment, you cannot just like, put

in a AI and put your own LLM and say to the customers, "Hey, this is going to be you." That, that will not work. So, so we said, "Hey, let's create a solution where our, where the user, the developer can select the LLM that is, in the enterprise environment, it is approved by the enterprise and they're paying for subscription, all the legal work in is worked out. And then just create an interface that can, that will empower that LLM to do more for developers." So that, that was one thing that, that was like a vision for the project.

The other thing was how can we create, create interface for 100% of functionality of Semaphore so that the developer can complete all of the time stay in that coding agent of choice, let's say, Claude Code. And then never leave and, and do everything through it. And then how can we create experience where you… I don't have to learn anything except high-level things, of course. Yeah, I need to conceptually know what the CI and CD does. But how… I don't have to learn the configuration. And that was described in a way, hey, we know the CI/CD inside out.

We have a lot of engineers on our team that are like, building CI/CD tools for like a decade. Let's put all that inter- expertise, package it somehow so that the other people can leverage it instead of them becoming experts. And that's what we have discovered that the good combination of a right, a CLI with the, as I said, commands plus, MCP and set of skills and lot of examples, are actually doing the thing. And then actually, and getting us to that, that, experience, that developer experience that I can just naturally in natural language say "Hey, this

is what… This is the outcome that I want." And then the, then my coding agent will be able to interact with the, with Semaphore in this case and do everything that needs to be done. Bret: Yeah. It's funny that in a world where we're gonna end up with these ad- advanced AIs that do a lot of the, grunt work for us, that the sweet spot seems right now like to stay at the CLI. And if you'd asked me like- 10 years ago or even five years ago, "Hey, if you can imagine a future with AI, do you think we'll be spending more

time or less time in the terminal?" I would've said probably less, right? We're gonna have AIs do all that, right? But being such a, a terminal fan, like one of… I think actually literally the most popular page on my website is my s- my shell setup page, which I detail, like, all the different things. It's a lot easier nowadays, but 10 years ago, for those of you kiddies out there listening, we used to have to do things like we have to, we'd have to install specific fonts, which, you kind of still have to do. But we'd have to worry about italics, like italicize didn't work by default, so you'd have to customize that stuff in your editor. You'd have to do all sorts of very particular settings and configurations

just to get a, a VIM to work right in a shell with all of the settings you wanted. We had to do a lot more a decade ago than we did do nowadays to set up your shell. But I still get a ton of traffic just when they, they just wanna know the fonts I'm using, the themes I'm using, the, the GUI I used. And then the, now my most popular video this year is my video of talking about leaving Ghostty Terminal, which has been for a couple years the big thing for terminals, leaving that for, Supacode, which is a g- a lib Ghostty. It's actually a Ghostty project, but it adds a bunch of agent management features

so that you can manage your product. So those of you watching, if you're interested in agents in the shell, go look at my YouTube later, but not right now. We're talking about CI right now. But I'm… It's hilarious that, the, the terminal people are getting their, they're getting everything right now. All the best stuff right now I feel like is in the terminal. All the best coding agents. I'm not… I'm trying to use the GUIs, but I don't feel like they're as good. And the fact that I can now… it took, to me, the way I'm looking at this is it took agents or it took AI for us to start looking at platforms like yours in a way of, we, a lot of, we all had CLIs for these tools, but you

had to really memorize the commands. I always feel like I don't know all the commands, and I'm constantly like, "Is there a command for this? Am I searching for a command?" you s- look at me look at the help. Can I even do this in the… and so I think as humans, we have this ability where we learn the command line tool once, let's say Docker. But then Docker adds a bunch of new features five years later, and no one uses them in the CLI because we all already learned it. And when I talk to people about this, they seem to agree that w- we don't feel like we're really experts in the CLI, the full power of the CLI of a lot of these tools.

And now I just feel like that's, that problem has been completely erased. Like y- like the way that I'm looking at your documentation, I'm realizing I, I could pick this up. I could use the CI, and the only thing I have to be good at is understanding CI concepts like the, the basics of CI and automation and how you manage a Git repo with workflows. But I feel like the only thing I really have to understand is, how to prompt my intent to the agent. And I feel like your demos on your website, the, the little ones that

say "Set up my CI for me," they… A lot of websites, they feel contrived. We used to have these sort of made-up demos on the front page, but that's literally probably what I would type to the CI is, "Could you please just set up Semaphore for me?" Because I don't know the shell, I don't know the CLI, and I'm not even familiar yet with the GUI, so I don't know what possible options are for the GUI. And that's just a, it's a cool target to, to approach of like, our onboarding literally needs to be someone who's switching from some other CI and

doesn't know anything about our platform, but the agent can make them extremely productive in the first hour. And is that something that you are constantly testing and building out? Looks like it was, like, a, a major feature for you. Marko: Yeah, absolutely. Absolutely. that barrier of entry, how, b- this is becoming extremely important. Like with all the code being generated by agents, like if you do not have a great CI that can test and make sure that you're like shipping something that, that, has no regressions and, and it's high quality, that's extremely important.

So it's very important that it's very easy for people to get on board with the CI/CD, and then also to give them freedom to easily switch from one to another, whatever is the reason. And then, those barriers were very difficult and hard before. These were like in a really project that DevOps organizations were planning on them, they, like asking for budgets and all like, and so it was very hard. And now it's, it just we have a tool that is like very good at reading documentation, which we as humans were really bad at, writing documentation,

especially reading the documentation. Bret: it's our last Marko: Exactly the last… Bret: hope is documentation. Yeah. How far can I get without documentation? Marko: Yeah. And then nowadays we have s- we have a tool that is very diligent in reading the documentation, especially if you format it in the right way, and then it's, it's "Hey, I have this, please, cop- look at this like Jenkins configuration and make it, in Semaphore." And then it starts to, to run because it, it knows like a lot of details about Jenkins and all the Groovy script and everything, and

it knows exactly how to mimic that in Semaphore and that's awesome. And then you can really quickly s- try it out and see what you can, Marcos: It was even a point that I was pestering Marko, because I wanted to set up a project of mine on Semaphore, only it starts when I started working with the company. But I didn't know how to do that. And now we only-- we just have a comment that you can talk with your agent that can do anything that you want. And you can set up a CI/CD from zero to hero like we have, one series

about that in around ten minutes. So it's very interesting. Bret: Yeah. in fact, I don't think I've thought a lot about the fact that w- what one of the things that agents are letting us do is switch tooling. I Not just switch tooling, but now we're seeing, c- complete projects being rewritten in Rust, Marko: Yeah, that's very popular. Bret: a- and if we can rewrite an application in Rust with $10,000 of tokens, like we can probably switch our CIs much quicker with far fewer tokens. But I haven't thought about, too much about the fact… 'Cause we had, I mean, we have an entire ecosystem, I feel like, of tooling to migrate one workflow

file, you know, migrate everything away from Jenkins, which I approve, and then m- move from, we, a lot of us had to move from Travis years ago when Travis changed a lot of its free… Remember f- a decade ago, Travis was very free, and we were all addicted to the free, and then they weren't so free, and then we had to move to something else, and I, and we have all these tools at the ecosystem, or even some of the, we, you probably, I don't know, maybe made some of these converters yourself. And it never went 100%. It was never, it never really worked on the first try. It was more like it would convert… It was kind of like my Dockerfile m-

making, conversion, converting things that would try to figure out your Dockerfile. It's like they get you to 80%, but then the last 20% actually takes you a l- a really long time and hours of work to troubleshoot. And some of the concepts are just different, but I didn't really ever think about the fact that this is almost a solved problem if you can give all the documentation and all the right skills and all the right MCP endpoints to your agents, like this could be a very smooth, fluid, process. But before, I want to get in that. So we're gonna get into the details of, like, how exactly people can use an agent to access and manage a CI.

There's some really interesting things I wanna talk to you about, because I've been doing a deep dive into your blogs over the last six months, and there's some really interesting ideas there that I like about, that you all are able to… You're basically able to be a little more nimble than some of the cor- the giants out there of CI. It seems like you're able to move quicker and modify the platform to be more agent native, than maybe a lot of the other sort of incumbents and legacy platforms. But first I wanna just talk about you going open source. 'Cause I'm a cloud native ambassador. I go to all the KubeCons, I'm a part of the CNCF and the Linux

Foundation, and now you all are, too. you're members of the Linux Foundation, members of the Cloud Native Computing Foundation. You're actually listed, for those of you out there that you know about the landscape, the CNCF landscape, they're listed on the landscape, which is very cool, 'cause there's a lot of CI options there. But I always thought of Semaphore as this closed source tool that is hosted in Europe and is for teams that are maybe burned by some of the big incumbent players and are looking for something a little more their style. How did exactly, what, how did this come about?

This seems like a really big decision to open source the platform. Can you tell me, can you walk me through how that happened? Marko: Yeah, so, it was a very big lift for the company, in order to open source your code, it kind of like there's a lot of work that you need to do, make sure that everything is buttoned up from the security perspective. So, so it was a big lift, but what we wanted to do is really enable people and to really meet them where they are, If you are only a cloud provider, then there are, there's… You're limited in a way.

Who's your customer? And we also have some, some big customers that are, you know, running the whole, Semaphore ecosystem on-prem. And we thought "Hey, let's open source this, let's enable people to try the Semaphore in the way that suits, suits them the best so that, if you want to run it on your own, you can. That we can open up the platform to call for contributions so that people can… also developers can collaborate with us much, much easier." For the developers it's much easier to submit the PR or add the comment than to hop on a call.

So, so, so it's also to get more feedback, to get more visibility, and to also make it possible for the, all the Jenkins customers. Sorry, sorry, Jenkins. But all the Jenkins customers to try something else, because they are really in, in the corner. It's a very, sticky platform for the, in my opinion, for the wrong reasons. So I think, these are all the, all of, all the, motivations for us to do an open source and we are, like, starting to see benefits of it.

And then we as a company, we believe that, that the, the open source is the way to go. Bret: Yeah. Marcos, were you a part of that effort? Marcos: No, I will-- I came later to that, but I made some AI open source. Bret: Yeah, 'cause that sounds like a lot of work. Marcos: I'm glad to be… didn't have to be part of that work because it was a-- Like, I was reading about the documentation that they had. It was a lot of things that we had to solve before going public with the code Bret: Yeah. Yeah, y- it's, I mean, there's so many factors, and I'm sitting here thinking, as a consultant, I s- I see a lot of different projects, and

a lot of the ones that are closed source, if I was one of the major people on that project, I would be very nervous about opening, opening it up. Not from just a security perspective, but now suddenly all my code is v- it's not … It's one thing when it's a dozen people that are, critiquing my code. It's another thing when it's thousands of people critiquing my code. And and sometimes I'm … When I look back at my old code, I'm often like, "Who wrote that? Oh, that was me. What was I s- what was I thinking? That was dumb." so yeah, that's gotta be … I mean, especially when you have a project that's many years in the making, right?

It's we all develop differently now than we did a decade ago, and … our, our intelligence about a, a code base is better than a decade ago. I mean, everything's different. So, when you have a long sort of a monolith, maybe not monolith, but something, anything older than a decade, I feel like at this point is almost like, legacy code if you don't constantly maintain it, keep it up to date. And I imagine one of the, I'm gonna guess one of the things you wanted to do before you open sourced it was to make sure that it was, like, on the latest libraries and it was using some of the latest paradigms. So all right, so you made it open source.

People can download this and run this themselves, or they can just easy button it with you on the hosted platform. … Tell me about, like, the runners. You provide … Or what do you call those workers or runners or what do you call those, those machines? Marko: Yeah, runners. Yeah, so we provide our own runners, but people can also self-host. And there are a lot of customers choosing for one or the other for their specific reasons. And then the benefit of running on our runners is that we do the, all the maintenance of the runners. The benefit of having your own self-hosted runners is that, you can

create the custom environments that, that are really bespoke for your needs. Maybe they are resembling 100% production environment. So it, it makes sense in different use cases. It makes sense to, to use one or the other. And we have a quite a lot of customers that go hybrid, so some of the runners are self-hosted and some of the runners are, they're using our cloud. So, kind of as I said, we like to, to, to meet our customers where they are, and then, you provide them with the best possible service, that we can offer.

Bret: like I think any CI/CD team at some point they're gonna end up needing to run runners on their, i- whether it's in their data centers or on their VPCs, like they're gonna need something that's closer to staging a production, especially if they're doing CD inside there. And the runner story is one of those things where, until you're a seasoned CI/CD veteran, I feel like you don't… You may not look at a platform to evaluate it and think, let me spend a lot of time playing around with runners, deploying runners, figuring out how to maintain runners, like getting the performance right. Let me…" There is so much to it that, I mean, there are entire

other CI/CD platforms where I d- I don't even recommend they use their runners, that they run their own, or that they use an outsourced service 'cause there's an entire ecosystem of outsourcing your runners. And you had an interesting blog post this year, actually this month, talking about benchmarks, and I thought this was really interesting because a- as you're probably saying here, the speed of runners is actually one of the most critical points to me in terms of evaluating a CI platform because you're gonna spend so much time waiting, and historically that has been infuriating for those of

us that are managing these platforms. Because as someone who manages CI/CD, like, my goal is to get these things to run as fast as possible, because I want my developers to use it more and I am also kind of s- I'm not only the CI/CD custodian a lot of times, I feel like I am the advocate for people automating more. I'm always looking for toil to automate in teams, not just testing their code, but also all of the other maintenance tasks. Let's throw those into CI as well, make it a more universal automation platform. But also I'm also security… I kind of fit in that DevSecOps security

realm where I'm wanting them to do more linting, I want them to do more security evalu- I'm, like, suggesting workflows all the time to them. "Hey, we should add this to the code. Hey, we should add this to code," even when I'm not a developer in the team, because I feel like developers, they spend so much time being told to focus on features and code maintenance of the actual platform that it's one of those, what's the analogy? The, the cobbler's children have no shoes or something like that. I always feel like our CI/CD platforms, if there's not advocates for them, you end up in this space, and Jenkins was the perfect example 15, t- 10 years ago, where it

just became this hodgepodge of fragility, and it ended up being that one person in the team that knew how to maintain it and no one else could touch it. A- and good luck with the backups and restores of that mess, right? Like it was… There was always that precious Jenkins box under someone's desk that could not be restored successfully, so you could not touch it. And I hope for everyone that those years are gone and that we are at least using better maintained and supported platforms. But you do have this- a blog post talking about how much faster, the runners build times, the cost of jobs.

I'm a big advocate of AMD and moving everything you can in your coding platforms to AMD because it's so much a better value on the big clouds, and it's just cheaper. And then, and talking about some of the competitors and how, whether or not they're faster or slower, they also tend to be more expensive. So I… For those listening, do everything you can to avoid running your own runners. Just because it's not a solved problem, and you're probably not better at making them faster than the professionals running these platforms that their

whole job is to make them faster. Marko: Yeah, there's a whole team of full-time employed engineers that are making sure that the, it runs, everything is up to date. I don't know if you have already addressed that in your, with in your podcast before, but in last, I would say three to five months, especially in the last three months, there's like a huge amount of CVEs out there that are popping up every day. Typically, you would do some maintenance every month, and that would be fine. But here, we have every day there are new things popping up.

I'm glad that, probably we will end up with a much more secure software in the end, but at the moment, it's so overwhelming to, to make sure that all the CVEs and that your environment is fully patched. Because in the-- if agents can find security holes, that means that other agents can also exploit your thing. So everything needs to be patched up, and buttoned up really quickly. And having internal teams working on that, I think it's very expensive at, twenty twenty-six. So it's much better to, to do that on scale, as we are doing as example for all of our customers.

Bret: Yeah. We have a question, "Are you guys taking contributors? I'd love to contribute. If yes, how can I get started?" Marko: Yeah, of course. We, we take contributors. It's very easy. you submit the PR and then some of our colleagues is going to review it and, getting the communication with with the person submitting a, a PR. And then if, if everything is looks good, then it passes all the C- CI, then it's merged and it becomes a part of the product. Marcus can maybe talk about when we open… When, we started with Sem

AI, for example, immediately as open source and as we presented it to a couple of our customers, they were starting to submit PRs like week two. So it was very fun to, to watch how people are thinking about it and what were their usage scenarios and their comments. Marcos: Yeah. We are always accepting new things and also improvements that people see that we probably don't see. And you can just go to the repository that we have and follow the contributing documentation, and we will take a look on the pull request.

Bret: Awesome. We got 37 contributors, so pretty cool. This is a question I didn't think to talk about. Do you, are you thinking about contributing to the CNCF? Are you gonna keep it for yourself? I don't know how to ask that question. Are you thinking about contributing it, or are, because it's your platform, I guess it's a little, gonna be a little hard to do that, right? 'Cause this isn't just a, a separate product, right? Marko: Exactly. It's a very big decision to make. So, so yeah, that's still under consideration, I would say. Yeah. Still discussion, yeah. Bret: it's one thing to do open core, but it's another thing when it's the whole

platform and it's something like this. But yeah, we're all kind of, all over the map with CI, and I was gonna talk about earlier that I don't remember the last time… It-- I'd say 50% of the time I walk in as a consultant, I w- especially if, we're talking enterprise here, big companies, hundreds of developers if not thousands, it is very rare for me to see them have one CI. And one of the reasons for that, getting back to our agent story, one of the reasons for that is that everyone's trying to get off Jenkins. For five years or more, everybody's been trying to get off Jenkins.

And the challenge always ends up, it's like an 80/20 thing where they can get that first 80% off in a reasonable amount of months, and then there's this 20% left that are these really complicated, multifaceted jobs that are insanely complex and should have never been created that way in the beginning to begin with. But they end up leaving them there because they just, they burn out on the migration. And so then now you have N plus one problems. And the, one of the biggest projects I was on in the last five years, it was a,

it was an enterprise security company, and they had three different CIs. And they had the old CI, the CI they tried to migrate to, but then it didn't do everything. I think that was Drone. I'll pick on Drone for a second. and this was five years ago or four years ago, so this was pre-AI. And Drone didn't quite do everything they wanted to do, and so then they decided they had to migra- migrate off of that, and then they were still migrating off of Jenkins, and then there was the new platform, and then it was just… talking about the agent migration story, we, I feel like we could do a whole ep- episode and then demo on it, but it would be, it would be amazing for

me to walk in and do a CI/CD migration project in a month and just, end to end, cut it off, and now we're new and improved all in a very short timeline. That, that almost feels like something I need to start selling as a product. I will migrate your CI for you in less than a month. Marko: Absolutely. Yeah. Yeah, that would be a very desirable service out there, I'm quite sure. But it's especially, migrating from the work, such a customizable tool like, like Jenkins, I don't think it's possible to really migrate only

kind of like a one-on-one, because of that level of customization. I was involved in like maybe at least like 10 projects in my career that were all on Jenkins. We were never able to upgrade from Jenkins to Jenkins. Like from one version to another. Even not moving to another tool, it's just keep staying like, you know, up to date and current was not possible because at a certain moment somebody decided to use the plugin rightfully, but then that plugin was not available anymore or not supported and then at that version. So it's, we ended up like, really, sitting down and rethinking some of our flows,

and some of the out- the pipelines. So I think that is necessary, but I also think that's an opportunity and good thing to think through because as you said, what we knew and what we thought it was right five years ago or even three years ago, maybe it's not right now because of like a lot of reasons. And then it's good to see, take some time and say "Huh, okay. If I have a blank slate, how would I like this to work?" And now with agents, if I can describe that, like in, in natural language and create like a very comprehensive like in documentation, again with the hel- help of agent, that

can be configured in I think 10, 15, 20 minutes, with the, with the agent. So, so, yeah, I think it's a great time for engineering. Before we were spending a lot of time in that grind of typing code, learning configuration, clicking buttons. Here we have a lot of time to really think through some of our decisions, take a lot of things in consideration, try out hypothesis. So, so, for our team internally, that's really a blessing, where we can spend more time thinking through and, and discussing and then the, the

groundwork is handed over to the machines Marcos: Actually, one of the test cases of Sem AI was migrating public repositories from Jenkins and GitHub Actions to Semaphore, just like a benchmark for the usefulness of the tool. Bret: Yeah. it's almost like those are like evals to test whether or not the agent can handle all that work, right? Marcos, I'm really, I really wanna dig into, the work that y'all have been doing to sort of make everything agent harness first. But I wanted to first talk about You c- you've prov- you've created a plugin

system for Claude Code and Codex. Can you talk about that real quick, about everything that's included in there and why the decision was made to do things like commands still, even though that feels almost like, w- is that still cool anymore to do commands? I thought we were just vibe- vibing the skills at this point. So, can you walk me through all that? 'Cause it looked like you have a lot of s- rigor there that is more than just someone simply publishing a free skill, for example. seems like a lot more than that. Marcos: Yeah. Initially the SemAI was just to be a CLI, so a simple CLI.

We have the same CLI. It's a old one that we have, and we wanted to develop something that was agent first. So naturally, we started with a tool that was only communicating in JSON, so the agent could run it and pipe it for… to anything that it wanted. But then we discovered that MCP was a thing, was booming, and we wanted to support MCP as well. But one rigor that we took, one choice that we took was we wanted to have the same capabilities of the CLI on the MCP server.

So basically we are using Viper and Cobra on the CLI, and we are basically using the commands that we have for the CLI on the MCP. It's a little bit of a hacky thing, but it's fully supported by the libraries, and you can check it on the repository. But you'll be a- always have the support that you have on the CLI on the MCP. And this is important because as Marko talked about on the start of the podcast, some tasks are easier to do using a CLI and the others are easy to do using MCP.

For example, your agent may decide to pipe the output of SemAI into a Python script to extract something that it wants. This is something that I see daily right now. And for some easy tasks, just asking a question or something for the, to the platform, it normally uses MCP. And we decide to document and make it easy for the agent to do the right thing, not do the wrong thing. So we have included descriptions, examples, and skills. Each one has a, a, a place, right?

So basically the descriptions is basically what the surface level knowledge that the agent should have about that command. Each command has examples, so if the agent is, doesn't know how to do the, the command, it can look through, into the examples and basically figure it out from the examples. And the skills that most people are kind of abandoning, the complex ones at least, is where we have the cheat sheet for the agent. So basically it will be able to know what is all of the step

that it needs to do for one task. So for example, if I want to migrate from Jenkins to Semaphore, it has a com- a skill that it can load, and it will have all of the thi- all of the things that it needs to do, and also will have the input from the developers of SemAI on how to do that, because we encounter a lot, lots of particularities with that flow. And then we are creating skills. So basically when you install the plugin, you will always have the skills available for you, and your agent can choose if it will use the MCP or the S- CLI itself.

And one nice feature of the plugin is that it auto-updates, so you always have the latest version when you install it. Bret: Wow. That's a lot. What about the agent makes it self-healing? I'm looking at this blog post on your journey to making the SemAI. This blog post was what pulled me in a little bit because there was a couple of things that I thought were different. One, yeah, you're doing the slash commands and wha- and it talks about why those matter and basically determinism. Embedding CI/CD best practices into the agents, which, I mean, for other

CIs, I had to build that myself. Like, as a part of my courses I offered because, 'cause it's not really built into a lot of these tools to, to really… Like to me, best practices are an opinion. I used to say better practices because I always felt like, b-a- my best, I don't know what if I have is best. I mean, it's better. And I feel like the more opinionated you can get, like the better that might be. And so sometimes best practices can not always work for everyone. But it looks like you've got all of these different areas that you focused on with like the caching, 'cause that's a thing that like people often will

screw up caching or they don't understand caching, so they're like leaving, they're leaving minutes on the table because they're not getting the most optimum performance out of their CI. And I've always felt like CIs are always trying to optimize for that and like they either go too far to make it too automated with the caching setup, and then it becomes limiting sometimes because it's, you have to do it this way in order to use the cache or it doesn't work, or you have to become like an expert in how their caching works and how your tools would need caching and which tools you have that need caching, and how to avoid cache poisoning.

And like there's just, even just in that world, I feel like there's someone's job dedicated to that, right? And y- to be an expert. But you have this other list of like the, the testing reports, the failure diagnosis, the artifact management, like how do we get these artifacts out of this build and into where they need to live? And so you've got all that built in there into the skills. Then you've got self-healing pipelines. Tell me about… I mean, you literally have this quote in here of, "Tell the agent, 'Work until the pipeline is green.'" Which I will point out is not the thing that I would normally actually type, because that is an

outcome, and I'm often dumb with agents. I need like agent therapy. I'm often dumb, and I'm like giving them the detailed step-by-step because that's what we had to do two years ago, versus I'm, this is an outcome, right? This is like what I would tell the junior engineer who understands CI, but they're like, "What is my job here?" I'm like, "Your job is to work until the pipeline is green." And, and so I'm guessing you test this, right? Like this isn't just a demo. This is something that you verify. Tell me about that.

Marcos: Yeah, basically the comments, the comments, the skills that we put is basically we compile all of the knowledge that we have. We got-- We have lots of customers and we, years of experience on the platform, and we try to make it available for the agent. But the, this, about this quote of just working until it's green is something that I do daily. And yes, I test that because it's basically SemAI has tools, commands that the agent can run that will watch the pipeline until it has a result.

It will extract the errors to the agent, and we will also be-- you also can basically go back in time and see errors as well. So for example, I can start my day just asking my agent how the pipeline, how the CI is going, and can go over all of the, the recent pull requests that we have. It can extract the errors, and this part of extracting the error is the difference between my agent gaming the system and just deleting the test and it actually fixing the test.

So in these skills, we talk about adversarial reviews of the agents, and how to use SemAI to extract the error. And in general, it's a little bit of magic as we, we are having with, agents. So it's, it just works most of the time, ninety-nine percent of the time, because it will only be able to see the error. It will have the error, clearly labeled on its front, like it will just so see the error. And most of the time, it will just look because it tries, it can try on

the CI itself, it can push a commit. But if you want to test it, you can test it locally. And also, if you are not able to test it locally because the test is too big or because it needs something that only the CI has, we have the test box. So it's basically a job that you can spawn, and you can SSH into it, and you can give it that machine because it's ephemeral to the agent to, for it to do anything that it wants there. So it can try lots of different solutions and then just try to get

that to the point that the CI is green Bret: Let's talk about that. let's dig into that 'cause we're, we're running a little long, but I think the Test Box scenario and having the agents doing that work is, to me, that's a really cool feature. I have been envisioning for a while, and I don't make CI/CDs, I just use them constantly. And I just, at the end of the day, I kinda just want my agent to constantly be running CI/CD, whatever that looks like. Whether that's just the testing workflows or maybe just the linting plus testing.

But I kinda want things to constantly be happening without me having to tell it, "Okay, every time you make a change, make a commit and push, make a commit and push, make a commit and push." This just feels like a, That mindset around m- waiting until the git com- commit and push to do automation, it has its place, but it feels like in this world we need this whole new thing, and that thing is before that. And s- a lot of people, I mean, Dagger's been trying to solve this problem for years.

Shout-out to Solomon Hykes. there's all these tools that are trying to make it easy for us to run tests locally and in, in CI, and it's always been a challenge to recreate the CI-like environment on a lo- on a random local machine that's a th- of one of three different operating systems. And that's, I think that's always been challenging for CI. I mean, a lot of CIs do it still, but this idea that my agent can just do this on my behalf, it can do it in the background, it can do it on my CI machines, but it has nothing really to do with me committing code, like, that is pretty cool, and I wanna hear it.

And what, I don't know anything about how you do it, so I wanna dive in for maybe our last topic. Marko: So this is also one of the transformation in the agentic way of working, because previously when you were like, typing the code, then you know, you make a change, you run a couple of tests on your own machine, and then you're, "Aha, okay, I'm happy with it. Let's push this and let's run around the whole CI." But nowadays, that's not exactly… Nowadays, it's not your job anymore. Your job is to orchestrate two, three, five, four, five, how many, like what, how, like depending on the context that you can keep in your

head, agents that are doing things. And then how do you mimic that thing? It's very hard to have a local machine that is powerful enough to run all those things. So here there's like a really nice way to scale it, to use your… A benefit is that you're using a CI box, so that it's exactly the same as the CI box. That's an additional benefit, but it's, it becomes scalable and it keeps you in that same loop as, as before ju- just on a different scale because you, you have like a different you know, automation tool at your disposal

Bret: All right. So you're checking out a Git repo and like I think a lot of us that are getting a little more advanced in agent-based workflows, you have a local clone and then you use either your harness or maybe yourself, you create gert- Git worktrees for, d- th- the each one of the agents has its own safe space essentially locally, right? And then you're, you have this plugin, you have the Semaphore plugin in your Claude Code or in your Codex. I think those are the two you support right now, right? And in that includes, does that, that plugin includes MCP skills, commands,

and a CLI, or I guess it's using the CLI to do some of the stuff? Okay. I wanna make sure I, include everything it is, it has. So it has all this stuff and I guess those skills are essentially like providing it the documentation as well as the best practices and then how do you get it to behave this way? Are you prompting it specifically around Semaphore? Or when you ask it to do a feature, are you just saying, "I'd like you to work on this feature. I need you to implement this thing. Maybe I'm telling you about it or maybe I'm pointing you to my ticket or wherever, where the, where you need to get the information.

And then while you're doing it, I need you to do these CI things." Like how exactly, what does the prompting look like there? I'm thinking about our audio listeners and like they're trying to imagine working through this workflow and how do they get Semaphore to do the coolest stuff here. Marcos: Yeah. For Semaphore doing the cool stuff is just install SemAI because it has already the prompts and everything that you for your agent to do. So for example, if you are using SemAI, SemAI, you can just ask the agent to create a entire organization or project, set up the Git connection

to you so it can do everything there. And there is my part is like I have some documentation on some CLAUDE.md and other documentation on my repository to instruct it on how to do stuff and how to, basically give one machine like one machine to one agent, and it can do anything there. I can be pretty relaxed to, some point with the sandboxing there because I am just letting that sub-agent reuse that machine so my machine is not impacted.

And also, from the side of Semaphore, I can put secrets on the CI and for example, I can create two, two projects and one be production and the other one being just a staging, for example. And I can give each agent access to some project because I can create multiple service accounts. This is the way that Semaphore does the, the things, and I can put different roles and different permissions to each service account. So I, I also had, one month ago, persistent agents that could do things

for me, and they were working with the CI, but because they had specific permissions, they could not go outside of the sandbox that they had on the CI analysis that they were doing for me. So for example, I just ask it for the SemAI to give me the, the status of the project. So I have a 33% pass rate because of IQ. and it's telling me some flaky tests that it detected and what I can do. So for example, I also have two requests that are red. I can ask it to run and fix this stuff.

So while we are talking, please fix the red stuff. And this is it. Bret: So in your initial prompt for the feature, did you just add into the, what presumably was the big prompt on get started on this particular feature, you just said, "And while you're at it, run Semaphore test boxes until everything's green and then commit"? Is that kind of how you would do it? Marcos: Yes, I would do that. And because the plugin is installed, it will probably just use the SemAI. It may be sometimes, it will request for permission, because of the

auto-classifier and stuff like that. But once you get permission to it, it will just use. And also, on the CLI and on the MCP, I don't see many people doing that, but we put some flags that you have to pass so that the auto-classifier will be more likely to prevent that command if you didn't give permission to it. So basically that, "Hey, I want-- I really want to delete this stuff." So normally from the test that I did, the auto-classifier holds the agent back

if I don't really tell it to delete it. So we are trying to improve the tool every day because we use that. Bret: Do you update your Claude file or your agents file in the repo to say something like, every time you make an edit run, run a Semaphore test box to validate so that, that way you don't even… it's just sort of part of the initial prompt initially so that it always defaults to testing. 'Cause that feels, this feels like the kind of thing that I'd always want running for every feature that my app is building.

I would just-- I don't wanna ever have to say, "You also need to remember to run some testing before we make a commit." Marcos: So yes, I have put that on the claude.md, so just use SemAI. This is like for SemAI is the only rule that I have. So use SemAI to test stuff and r- sees the CI. And the other one is use sub-agents to not pollute the main context. Just basically that. Bret: Yeah. I mean, I'm putting this in perspective to what I currently have to do, right? 'Cause I'm not using Semaphore yet, and I'm… I have to do things like I

want it to, I want it to at least lint, or like you said, basic unit testing. And so then I have to make Makefiles or Docker Compose files or something else that's gonna run locally, and then I have to put in my agents file for each project a bunch of… a list of things. "Please always run these linters before," or, "Please determine which are the best linters for this language, and please always run them. And also run this thing to make s- if you edit ever a g- a, a workflow file, go run these linters on this thing." I have to… I mean, I have a list of things that I always want it to do on every change, and those are all really just CI things that I can't run locally,

so I have to give it… I have to make sure those tools are installed. I have to give it permission to install those tools if I think they're not gonna be installed, which means my machine has to be very custom. It's just not… It is not ideal. I feel like I'm kind of working around the limitations of my own CI. And, a- and I was trying to… And I, and this is sort of me asking you all the questions of the things that I really wish my CI did. So, Marcos: Yeah Bret: okay. So we've got this thing r- creating these test boxes. are… I'm just curious, Are you able to… I'm assuming Docker is on there,

so then I can spin up Kubernetes clusters and I can do all those things, right? That's all n- very normal runner stuff. Marko: Absolutely. Yep. Yeah, and then everything that you described, Bret, it's like what this is what we typically see that our customers put in their pipeline. Bret: Yeah Marko: And it's even better than it, when it is part of the pi- pipeline, Because if you tell it to the agent, then it's it will do it, but it's not really algorithmic. When you put it in a pl- pipeline, it has to be done because then that's algorithmic. That's not probabilistic like a, like an LLM. They are great, getting better every day, but still, this

is, for sure it will happen Marcos: we said, most of the s- stuff that is, it's doing on its own is described on the skills. We have the guide on how to debug pipelines, on how to deploy, fixing flake tests, even migrating from GitHub actions to Semaphore. So we have lots of skills of curated procedures that we do. So on Semaphore, we deploy like at least three times a day to production, and we have some spiky days, some bursty days that is more than 20 deploys on that day.

So we have some knowledge on how to do stuff on that. So for example, you can read the skills. I, invite the audience to also go and view what is going on with the skills. We have a list of commands and steps that you should do, for the agent to do and how to get the result that you want. So that's why you can just ask it to do stuff because it already has the documentation, the step-by-step on how to do that Bret: Yeah. And so for those not watching, the, I mean, we're staring at the open source SemAI repo, and there's what looks like dozens of skills there.

I mean, for my GitHub Actions, I only have one giant skill, so now I'm realizing I'm already doing it wrong. Because this is like, these are very specific. They're very detailed, but they're also very specific, and why not? Like, why shove it all into one skill? Why Marcos: Yeah. And we also have the golden goose here, is the sem-ai watch. So this is what enables the loop. basically it's a comb that will check your pipeline. It's simple. It's very simple, but sometimes the simple thing is what we miss. So basically it will watch your pipeline run.

When it's green, it will get back to your agent, so that way he can loop without needing to have the AI harness loop it by itself Bret: So in this scenario, the main agent would probably start a sub-agent with this watch and the command in the background if it's doing it right, and then it-- this would allow my agent to work on consecutive features one after the other, or decide stopping points, I guess, if I gave it some more detailed instruction to stop halfway or whatever and test that particular function, run it through the unit test.

Yeah, that's kinda how I imagine that we would get these agents to do is it's just this automatic thing. It's happening in the background. I'm not fully aware of how many times it's run tests or whether I need to go look at a page and stare at it waiting for the green button. And I am curious, a quick question. You talked about the profiles. This is a subtle little security conversation. you talked about the patches that you're creating for each… What's that? Marcos: Roles, the service accounts basically on the Semaphore. Bret: Yeah, these different accounts essentially, like how does that

work in your harness specifically? Okay, because I feel like this is really important for people to understand that ideally, we wanna isolate our, and sandbox our agents and give them only the keys they need. And right now, one of my biggest frustrations is we didn't… Like the personal access tokens and the, and all the ways we provide our local CLIs and tooling to access ser- services on the internet, like it wasn't really designed for a world where we have disposable, always running agents, and like we need

these things to be ephemeral, and they need-- They might need to be short-lived, or they need to be managed in a way where I can make more than one, right? 'Cause I remember when Docker it was a long time before Docker Hub had anything other than your password, right? And then when `they made it, when they created PATs, they were very limited. They have read or read-write across everything. It's sort of like a giant blast radius. And now, when we're in this agent world, I think what you just described is probably what, what I'm imagining is I really want this agent on this work tree to

really have the smallest access possible. I don't want them to have admin on my repo. I only maybe need them to be able to git and git commit and push, and then run CI. And I only want them to run the CI for that particular repo, and I need to give them that key, but how do I give, how do I do that? Or how do you do that, I guess, specifically, 'cause this is maybe not a Semaphore thing. It's maybe more how you work. How do you do that without it being, like how the other agents on your machine get that? Are you just doing environment variables per shell?

Is that kind of how you're doing it? Or how do you approach that? Marcos: Yeah. What I would say, is that you should, for example, start way more restrict than what I'm doing currently, because this is in scenario, I have lots of tools, tool calls that I have already whitelisted for my agents, right? So you should start small, don't whitelist anything or just do some basis, basic stuff. On Semaphore, you can create service accounts and you can define roles for each

service account on each project, right? I can put it as a reader or as a admin or anything that I want on my specific project. You can basically control more, most of this stuff this way. So if you want a agent that just wants to read the stuff on Semaphore, you can put it. Most of the time you are okay with that because you just want to read the, the pipeline status and then you, if you are using GitOps, you are going to commit another thing on GitHub and it will spawn a-another pipeline.

And you can basically, like the idea here is you can mix and match permissions and each service account will have a token, and that token is used to authenticate on SemAI. what I currently have, right, is basically I have one token that has less privilege than-- it cannot basically hit anything that would damage my account or my Semaphore, organization in this case. because it was like with time and I put more restrict-re-restrictions on it. but basically I just let it run, because it cannot do anything.

It cannot deploy to production. It cannot mess with any production related projects. It can only do stuff on staging. And for example, staging is something that I think is more important in this case because I can let it have more reign over the staging environment. It can do more stuff, but I am always the gate to production when we is talking about deployment. Bret: Yeah. Is this coming in through an environment variable on your machine? Is that how this is in your shell? Marcos: This one, yeah.

This, this whole shell is just one environment variable of the token, basically. Bret: Yeah. I just had a live show last week. We talked about nono, nono.sh, and that's a sandboxing technology, and I had one of the founders on the show. And I'm a big fan of that. So I'm just, for those listening, like if you're trying to think about how to isolate your tokens as well as your agents, and Claude Code has this sandbox thing built in, but the, the one of the cool things that nono can do is you can actually build out profiles that could include these kind of secrets, these

kind of paths for things like Semaphore, as well as egress rules that say like you can only access package managers and GitHub and then Semaphore, and that these are only things you can do or whatever. And then you, when you start up Claude Code, this is kind of the thing I'm wanting to do. I haven't actually set it up yet 'cause I'm using nono today, but to wrap my Claude Code and my OpenCode and my Py, my three favorite harnesses. And I have found that Claude Code's built-in sandboxing, I find it lacking. So, the nono that wraps around it allows me to give it file paths and environment

variables and secrets securely outside of my harness, and then I can just sort of feed all that in so that I don't have to copy and paste things into each shell every time I'm running them. So I don't know if that's a pro tip, but you can use nono in this way to probably make a lot of this CI/CD stuff easier. Like for me, it's like my default GitHub command line has my key, which has every repo that I have access to, and I don't want my agents to do that. Similar for like any of my artifact storage or my CI.

Like I kinda want, especially as a consultant, like I have access to lots of different things, and I don't want the blast radius there. So I get nervous, and I end up using nono to try to basically it removes a lot of those secrets and removes access to those things, so it doesn't have access to my default tokens. Shout out to the Agentic DevOps Guild that I'm running because they're the ones that we kinda worked out a lot of these ideas around how to use these for DevOps specifically. Like, how do I avoi- avoid Terraform, the agent having access to GitHub, Terraform, Docker Hub,

Semaphore, AWS, and every key that I have for the company, how do I avoid it from nuking everything every time I spin up Claude Code? 'Cause I know a lot of us are dangerously skipping permissions. I know I am, and that's bec- that's because we're tired of hitting the Yes key and, the, the, the Enter key for every yes and request it has. So if we're all doing that, like my pro tip, if you haven't heard that episode before this one, is to go check out nono. Add that to Semaphore. Now you're gonna, now you're gonna level up your DevOps. You're gonna… You're now DevSecOps because you're using a sandbox. That's maybe not the entire requirements for being DevSecOps, but that's an

important step that I think a lot of us aren't comfortable with yet. But nono's really easy to use. I feel like I'm doing an ad for them in the middle of this show. And they'll work great with Semaphore for isolating some of these workflows. Very cool. Marcos: You can also do something interesting with Semaphore because you can put the secrets on the project itself. So for example, we can put the GitHub access tokens and other platforms that you are using on the Semaphore. And you can, for example, if you really want to be very, very secure about that, you can create one project per agent or per activity that you are doing.

So you can basically put the secrets on Semaphore. It will only have access to those secrets because you can restrict it to only working on the ephemeral machine and it can only have access to that. And that with the service account will be basically a really good sandbox to keep your agent in check basically. And also for deployments, you can put a lot of s- specific, preconditions on Semaphore. So if you are deploying from Semaphore, you can prevent rogue agents from deploying stuff that it, they shouldn't.

Bret: Very cool. Yeah, CI/CD is turning out to be a really, a really good place for attackers to find weak spots in all of our, in all of our automation. And I feel like, prompt injection, between prompt injection and just general supply chain attacks, I feel like we're, our CI is constantly under threat. So anytime I can level up the security or run, like security audits. Is Yeah, that's cool. I mean, The fact that it's built in is nice. The fact that I don't have to go find a different solution to plug into my CI that makes this kind of possible is probably where everyone

else is gonna go eventually too. I think this is, as, as… I just wanna stay in my harness. I don't wanna have to wait any more than, than the prompts require me to wr- wait. I wanna reduce the number of steps. I feel like in a lot of ways, agents are helping us untangle the last 10 to 15 years of complexity. As someone in the cloud native ecosystem and has been teaching Docker and Kubernetes for a decade, to me, those tools were, especially Kubernetes, was always primarily for operators. It wasn't meant to be a tool for developers.

And also, a lot of the teams I would work with would have a couple people that were well-versed in CI/CD, but not everyone needed to be a CI/CD expert. But something happened over the last decade where, I don't know if we got distracted by the name full stack developer or if the, if somehow people got confused about what DevOps really is, but somehow at some point, the industry decided that developers are responsible for everything, which I feel like is a sad state of things where you can't just focus on being an excellent front-end developer or an excellent architect of applications, but now you

have to know the CI, all the sa- CI/CD paradigms, all of the operations, all of the cloud infrastructure management. You have to know Terraform, you have to know Kubernetes and Docker, and oh yeah, and you also gotta be this great developer, and I always felt like that was completely unrealistic. And now what I'm seeing is, as me and my friends talk about how we talk about a couple of things that aren't necessarily great, like one, when was the last time we actually looked at code besides this? When was the last time I actually wrote some code? It's been a while. And two, when was the last time I actually learned something

new without the agent involved? Like, when was the last time that I learned a new tool just by hand, or learned a new language, or learned a new platform? It's actually, we were talking about it, that we're actually worried that these agents are making us so nimble and it's real- It's, to me, I think it's just a sign that we all love this tech, but it was too much. There was too much we had to know. There was too much we all felt we had to learn. And this is allowing us to do things like be an expert in CI/CD, be an expert in automation, be an expert in Kubernetes without having to

know the Kubernetes command line. And I don't know where this is all going. I don't know if in three years we're gonna wish that we hadn't done all this and, we- we're all Luddites in our understanding of command line tools, but I am enjoying very much not having to remember all the command line nuances of the dozens of command line tools I think I have to use in a given week. It's just, it's gotten crazy. Like 15, 20 years ago we had very few tools. We were all in one editor. On Windows side, we barely even had terminals. So I'm just sort of soaking, basking in the amazement of this moment where

agents are allowing us to implement these things very quickly and to get very productive pretty quickly. I just hope they don't go away because… Or that we don't lose access to them. While we were talking before the show about, you gotta have a backup subscription now. Your agents have to be redundant, so Claude Code isn't… We had a Claude Code outage yesterday. Claude Code can't be your only subscription. You have to have some other backup plan, whether that's OpenRouter or ChatGPT or, yeah, or Kimi or whatever your favorite subscription is. You gotta have something there. Okay. This has been really cool to dive in. I've been really interested in how this all works, and I do think between

your advanced plugin model and the agent, having all those different skills and all the different commands, as well as the fact that it is sort of empowered with the test boxes, I think that's a really cool advantage. 'Cause someone earlier was asking, "What are your differences between you and other CI/CD platforms?" And you're welcome to answer that question. We always get the comparison question in a show about a product. But I will just answer that question myself and say, as someone who operates CI/CDs, those two things, like the plugin model for those harnesses and the test boxes to me with, in just one hour of looking at Semaphore are,

to me, the advantages there, as well as it just being open source and I can run it myself if I feel like it. l- like that, that seems like a, th- the true futuristic agent n- agent native way to operate a CI. So I don't know if you can say it better, but Marko: No, you, no, you did a great job. Thank you so much. Yeah Bret: But yeah, you can hire me anytime. I'll be your DevRel. but yeah, this is r- it's amazing to see how far you all have come in a year since you open sourced. This is just a lot that you've done, and it's pretty impressive. So what's next?

What are you working on? What are the hot tips for the rest of this year? Marko: Yeah. So, so now that we have this interface to the agents, we want to get to the basics. And it's again, focused on, like really doubling down on developers' productivity and what does that mean for us? First of all, if something fails in my workflow, in my pipeline, I want to be able to give as much as possible information, the right information to the agent so that the agent can figure out what it is all about, and hopefully fix it on its own.

So that I don't, as a developer, I don't have to intervene. And there is like a lot of nuances like, what are, what's the format? What's the amount of information that you need to deliver to the agent in order to get this right? So that's the number one thing, because I want to get to green state as Marcus showed as soon as possible. The other thing is that if everything is green and everything is okay, I want to be able to monitor how is my pipeline working over the time and give enough data to my agent so that it can optimize my pipeline, because that's also very important.

We all know that as your test suite grows, the duration of your pipeline grows as well, and it's always healthy to refactor or optimize your pipeline in order to keep it under 10 minutes, under 15 minutes, whatever is your target. And then last but not least is like, enterprises, they need a lot of governance and security. We want to introduce some additional services that can complement the SLSA compliance, that's the word that I'm looking for. So that, that is also, these things can be done with Semaphore right now, but

we want to make them out of the, like working out of the box without additional configuration and thinking through. So it's how can we make developers more productive with the focus on CI/CD. And there are like s- a lot of small things. For example, like a next thing that we are going to roll out is ability to just run, rerun a single job in the whole pipeline. Because a lot of the times, currently you can rerun only the whole block. But for a lot of the times, only one job fails. And you really want to be able to really effectively just rerun that job to s- to

make sure that, hey, maybe it was like a little bit of flakiness or something like that, but you want to be able to push it forward quickly and keep on developing. That's, that's the goal. Bret: Especially if that was the f- the two-minute run versus the, the 20-minute run Marko: Absolutely. Absolutely Bret: You mentioned SLSA. This is what you're talking about, right? S-L-S-A? Marko: Mm-hmm. Yes. Bret: Yeah. So for those of you listening that are not aware, I'm actually a big fan of SLSA. we actually talked about it recently. I've mentioned it more than a few times over the years because I was really

paying attention to it even before version one came out, and this is, this to me is, the supply chain checklist bible. I look at this as if I am not sure around whether my supply chain is up to snuff in terms of its security and me doing all the right things as a DevOps professional, it's not like we have formal education where I come out of school and I know the exact minimum things that I should be doing in an enterprise to keep my software safe. And so SLSA to me is one of the closest things to help me with that.

It will give me very prescriptive step-by-step approaches to make sure that I'm building securely, that I'm providing providence with my code, essentially built into the artifacts, that I'm building the artifacts with all the details that I need in order to understand how and where they were built, and when they were built, and by who, and with all the right, all the different details that I need to put in there. And then making sure that I'm sort of doing the best practices in the industry, like linting code, or I'm not sure if linting is actually on there. But like doing some of the minimum sort of… So the, the stuff that honestly,

quite frankly, if we, if every open source repo in particular was doing that over the last year, we probably wouldn't had near as many attacks on repos that we've seen in the headlines over the last year, because almost all of that is due to a lack of teams properly securing their workflows and their behaviors and approaching things from a DevSecOps mindset. And if they had been following the SLSA, which has levels, so, you can start at a sort of beginner level, and you can sort of promote and graduate as you get better and more strict, 'cause some of these things are easy and some

of these things, the steps by, steps that it makes you do are gonna require your, more than just your own effort. And so I'm always trying to s- on, especially on the re- open source repos where they're public and I care more about them, I'm always trying to, level up my SLSA, from w- I can't remember the levels or whatever. But level one, level two, I'm not sure if they're numbered, but I'm always wanting to try to level those up to get my software a little more hardened, a little more, a little safer essentially. And so go check that out. That's S-L-S-A.dev, and they are part of the OpenSSF, which is also a great thing

for everyone to check out for security, and they have other stuff beside… All, all the software security stuff is over at OpenSSF, so go check that out. That's another part of the cloud native ecosystem, so I feel like I have to plug it ' cause it's, it's cool stuff. awesome. Well, I'm looking forward to it. How can people get started? Okay, so you have the repo. they can go over to, Marko: Just go over to semaphore.ai. Yeah. And yeah, .io, exactly. And, and, there is it's a simple command that, they can run and install it.

it gets installed and then they can say, "Hey, create my account." It will prompt you to create your organization. So the whole onboarding is also part of the SemAI. Bret: Right. yeah, 'cause on your getting started guide, it's funny, this is this is just the indication of the cloud native or the, agent native world that we live in now. it's not download our CLI and create an account and go do this and read our repo for, or read our documentation for how to use a CLI. It's pl- install this plugin into Claude Code or Codex and then ask it to get

started with Semaphore or whatever the com- whatever the prompt is. And, I am so… I feel like this is one of my last resistances against the a- the agents, is I'm so not used to that workflow of just get the tool, install it, create my account, do the things. I don't want it… And tell me when it's all done. I'm just, I d- I s- I still am so programmed to look for the command in Brew, find which Brew package I need to install, read the getting started guide, create my account and all that stuff. So it's cool that you're making-- You're not just making the use

of a CI AI native, you're making the setup and c- account stuff AI native too, so that's really cool. All right, Marcos, Marko, thank you so much for being here. Marko: Yeah. Thank you for having Bret: you around the, we'll see you around the agents, around… We'll be chatting with our agents are just gonna do this from now on. Actually at some point this podcast is just gonna be a bunch of agents talk. Your agents will join my agents. We will, they will live stream and tell everybody about it, and then our jobs will be to sit around the pool and drink margaritas Marko: Yeah. Well, okay. Sounds good. Bret: All right. Sounds good. That's the utopia future. The other future I don't wanna talk about.

Bret (2): Thanks for watching, and I'll see you in the next episode

Open the transcript file

Hosts and Guests

Hosts

  • Bret Fisher Host Feed-stated

Guests

  • Beth Fisher Producer Feed-stated
  • Cristi Cotovan Editor Feed-stated
  • Marcos Filipe Guest Feed-stated
  • Marko Gaćeša Guest Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. A single API for multicloud with Control Plane

    Sep 18, 202654 minAudio

    Control Plane CEO Doron Grinstein joins me to discuss how they’ve created an AI-native cloud API that virtualizes multi-cloud and hybrid cloud to simplify management and ship faster. Watch the video of this episode. 🖥️...

    • Apple Podcasts

    Credit the people on this episode

  2. K8s Maxxing with AI-Native Platform Engineering Stack with OpenChoreo

    Jun 13, 202655 minAudio

    OpenChoreo is a “batteries included”, AI-native Kubernetes platform stack for Platform Engineers that combines GitOps, Observability, AI Agents, and Workflows into a custom K8s distribution “super pack” that is managed...

    • Apple Podcasts

    Credit the people on this episode

  3. Docker AI, what’s new with MCP, Agents, Sandboxes, and more

    Apr 7, 20261 hr 16 minAudio

    Michael Irwin of Docker joins me to run through Gordon AI improvements, Docker Hardened Images and what's now free, Docker Sandboxes for running agents in proper isolation, Model Runner updates including MLX support on...

    • Apple Podcasts

    Credit the people on this episode

  4. Backup S3, Google Drive, iCloud, Notion with Plakar

    Mar 20, 20261 hr 7 minAudio

    Bret is joined by the founders of Plakar - Julien Mangeard and Gilles Chehade - to nerd out over backup engineering. The kind where you're building your own file formats and cryptographic layers, not just wiring up cron...

    • Apple Podcasts

    Credit the people on this episode

  5. Your Images are Out of Date (probably) - The Silent Rebuilds problem

    Mar 4, 202637 minAudio

    Container base images (like Official Docker Hub images) are often updated without new tag versions. I call this Silent Rebuilds. There's no way to know this happens without image digest-checking automation like...

    • Apple Podcasts

    Credit the people on this episode