Modern software is built on open source - but how much do we really know about the code, packages, container images, and CI/CD components entering our production environments? In this episode of DevOps Academy Deep Dive, we explore the growing challenge of software supply chain security and what DevOps teams can do to move from blind trust toward verifiable trust. The conversation looks beyond vulnerability scanning to examine the entire path from source code to production - including dependencies, compromised packages, SBOMs, artifact provenance, signing, container security, CI/CD permissions, and the emerging impact of AI-generated code. 🔑 Key Takeaways Why software dependencies have become a major security challenge How compromised and malicious packages can enter the supply chain What an SBOM tells you - and what it doesn't The difference between SBOMs and software provenance Why artifact signing matters before software reaches production How CI/CD pipelines can become a critical attack surface Why third-party actions and plugins should be treated as dependencies How DevOps can move from blind trust to verifiable trust 🎧 Tune in to explore how engineering teams can continue benefiting from open source without treating every dependency as automatically trustworthy. 💬 How confident are you that your team knows exactly what's inside the software you're shipping?
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Right now, the banking app on your phone is, uh, it's very likely relying on code written by a volunteer in Nebraska whom the bank has never even met. Which is wild to think about. It really is. I mean, we assume the digital world is built by these massive corporations in, like, secure, isolated labs, but the reality, it's all assembled from free parts just found on the internet. Yeah, literally just out there for anyone to grab. Exactly. And that brings us to a fundamentally uncomfortable question for this deep dive. How much of our digital world is built
on code that we just blindly trust? And how on earth can we verify the software that we're bringing into our highly sensitive systems? It's the multi -billion dollar question, right? It really is. OK, let's unpack this. Because if you look at the latest software supply chain security frameworks coming out of the DevOps Academy, the mission is pretty clear. we need to look at the invisible foundations of the tech we rely on. Yeah, we have to look under the hood. Right. And to be clear right up front to everyone listening, open source software itself is not the enemy here.
We love open source. But our assumptions about it, that is a, well, a completely different story. Oh, absolutely. I mean, we have to fundamentally shift how we think about modern coding. Yeah, what do you mean? Well, we've moved away from the concept of writing an application in isolation. like sitting in a dark room typing every single line. We've fully embraced assembling it from pre -existing parts. Like digital Legos. Exactly, digital Legos. And there's a very good reason for that, which is efficiency. You don't want
to reinvent the wheel every time you build an app. Right. That makes total sense. But to understand why there is a massive security problem here, we first have to understand what that assembly process actually looks like in the real world. OK. Lay it out for us. So the modern software supply chain has become incredibly complex, largely because of what we call dependency trees. Dependency trees. Yeah. So say you install one package to handle a simple database connection. Just one thing. Right, just one package. But that package, it relies on 10 other packages just to function.
Oh, I see. And those 10 packages, they depend on another 50 packages. We call these transitive dependencies. So it just spirals out of control immediately. It really does. Suddenly, a single decision to add one simple feature. introduces an entire sprawling tree of dependencies into your application. That is terrifying. And the true scope of the software supply chain, it's staggering. It's not just the code. It encompasses third -party libraries, package registries, container -based images, the build systems themselves,
the CI -CD runners, deployment tools, cloud environments. I mean, it's huge. Hold on. Let me stop you there. CI -CD, you're talking about the automated assembly lines developers use to push out updates, right? like continuous integration and continuous deployment. Spot on. That is the automated factory floor. OK, got it. Every time a developer writes new code, the CI -CD pipeline compiles it, tests it, and deploys it to the servers automatically. Automatically, wow. Yeah. And every single component in that automated process is a potential point
of entry for an attacker. You know, it makes me think of... building a massive house using prefabricated rooms. Oh, that's a good way to look at it. Yeah, like it's incredibly fast, right? You just crane the living room into place. You drop the kitchen next to it and boom, you have a house. Ready to move in. Exactly. But you have absolutely no idea who forged the nails holding the walls together or like who wired the electrical outlets inside those pre -built sections. You are just blindly trusting that the factory did it right and that nobody you
know, tampered with it on the highway. Yes. And if we connect this to the bigger picture, an attacker doesn't need to break your windows to get inside your house anymore. What do you mean? They just bribe the factory, making your prefab kitchen to, say, install a tiny microphone behind the drywall. Oh, wow. Right. And that microphone automatically gets shipped to thousands of houses. Because it's in the blueprint. Exactly. In the software world, if an attacker compromises a widely used open source package, thousands of
organizations will automatically download that malicious version the very next time their automated pipelines build the software. So they don't even know it's happening. Not at all. Trust itself has become the attack surface. It completely changes the economics of cyber attacks. A threat actor, they don't have to breach a thousand different firewalls anymore. They just go to the source. Yep. They'd just compromise one trusted point upstream. But I mean, are developers really just that sloppy though? Because the sources from the DevOps Academy, they mention attacks like
typo -squatting. Oh yeah, typo -squatting is huge. Right, where an attacker publishes a malicious package with a name that's just like one character off from a popular one, like swapping the R and A in standard library or something. Yeah, simple typo. It just seems insane to me that a multi -million dollar enterprise gets breached because a guy typo'd a word on a Friday afternoon. Like, shouldn't they be checking this? Well, it's really not about sloppiness, honestly. It's about the reality of fast -paced, highly automated, modern workflows. How so? Developers type hundreds of
terminal commands a day. Muscle memory just takes over. Or... You know, they are copying and pasting from documentation on random forums. Oh, right. Stack overflow and stuff. Exactly. And the attackers know this. These attacks are insidious specifically because they prey on standard, everyday developer behaviors. That makes sense. But there is an even more devastating exploit that targets the automation itself, something called dependency confusion. Dependency confusion, how does that one work? So many organizations use both public
registries for open source code and private internal registries for their own proprietary code. Okay, so a mix of public and private. Right. Dependency confusion tricks the automated build systems by exploiting the rules of semantic versioning. Wait, versioning? Like version 1 .0? Exactly. An attacker finds out the name of a private internal package that a specific company uses. How do they even find that out? Sometimes it leaks in forum posts, or public repositories, or error logs. It happens more than you'd think. Wow.
Okay. Then the attacker publishes a malicious package with that exact same name to the public open source registry. But they assign it an astronomically high version number. Oh no. Yeah. So say the internal legitimate version is 1 .0. The attacker labels their public malicious version 99 .0. And the machine just blindly assumes that the higher number is the better newer code, like it just grabs it. Yes. The build tools are literally configured by default to automatically pull the latest version to keep everything up to date.
So it's just trying to be helpful. Exactly. The tool queries in the network. sees a version 1 .0 internally and a version 99 .0 publicly, assumes the public one is just a massive new update and pulls it right past the firewall into the internal system. That is absolutely wild. It is. In 2021, a security researcher named Alex Bursan legally hacked dozens of Fortune 500 companies using this exact method. Dozens. Dozens, including Apple, Microsoft, and PayPal. You're kidding. Apple and Microsoft. Yep. He didn't break any
encryption. He just exploited the system's own desire to be helpful and up to date. Wait. So if our apps are just giant Hoovers sucking in code with higher version numbers from the internet, how does a team even know what's running in their own software right now? That is the big problem. They often don't. It feels like we need a map just to figure out what's in the kitchen to use the prefab house analogy and here's where it gets really interesting because the frameworks introduce this concept of the SBOM. Ah, yes.
The Software Bill of Material. The SBOM. Think of it like the ingredient list on a box of cereal. Okay, I like that. An SBOM is a comprehensive, machine -readable inventory of every single software component contained in an application. So it lists everything. Everything. It lists the libraries, the packages, the exact versions, and all those sprawling transitive dependencies we talked about earlier. Right. Remember December 2021? Oh, don't remind me. Log4j. The Log4j vulnerability. A massive internet breaking flaw that just dropped
on a random Friday. It ruined a lot of weekends. It really did. And it was buried inside this tiny ubiquitous logging library that basically everyone used. If a company didn't have an S -bomb on that day, what did their weekend actually look like? Total unmitigated panic. I just chaos absolute chaos without an s -bomb and organizations security and engineering teams had to spend days Just trying to answer the most basic question imaginable Are we even using log4j? Because they
couldn't just look it up. No, they had to manually search through thousands of code bases, check live servers, and basically interrupt every development team to ask what they had installed. While the clock is ticking. Exactly. It's a massive waste of time while attackers are actively scanning the internet to exploit the flaw. But with a comprehensive S -bomb, you just query your database and in seconds, you know exactly which applications use the compromise component. And you can just start patching immediately. Right. Let me ask you this, though. Does generating this S -bomb
actually make the application secure? Like, if I have the ingredient list for the cereal, am I automatically protected from poison? No. Not at all. I figured. Yeah. An S -bomb provides visibility, not security. Visibility, not security. OK. You can have a beautifully formatted, highly detailed S -bomb that clearly lists 10 critically vulnerable dependencies currently running in your production environment. But you're still vulnerable. Exactly. The document itself doesn't fix a single thing. The value of an S -bomb.
only exist if you have automated processes built around it. So you have to actually use it. You have to continuously scan those listed components against databases of newly disclosed vulnerabilities. You have to determine which of your systems are actually affected and prioritize the remediation based on real risk. So the SBOM is really just the inventory. It is just the inventory, nothing more. But, you know, an SBOM just tells me the toxic ingredient is in the kitchen. It doesn't tell me who put it there or like if the delivery guy swapped it on a REIT. That's a great point. It reminds me of a legal chain of custody in
a criminal trial, you know? Oh, absolutely. Like, you can have the physical evidence, which is the code in our case. But if the chain of custody showing who handled it, where it was stored, who had access to it, if all that is broken or undocumented, the evidence is completely useless in court. That is a perfect analogy. In software supply chain security, we call that chain of custody provenance. Provenance. Provenance answers the exact questions that the SB's bomb cannot, it asks. Which specific source repository produced
the software? Right. Which exact commit of the code was used? Was the build runner like the actual machine that compiled the code? Was it secure and trusted? And was the code messed with after it was built, right? Exactly. Was the artifact modified in any way after it left the factory? Your source repository could be perfectly locked down and secure, but if the automated factory building the code is compromised, you have a massive problem. Because it's injecting stuff. Yes. The original source code could look pristine, but during the automated build process, a compromised
runner injects malware into the final binary. And without provenance, you're blind to that. If you don't have provenance, you would literally never know. So how do we actually enforce that chain of custody then? Because, I mean, we can't have a human manually reviewing the legal paperwork for thousands of automated daily deployments. That would take forever. No. Humans are too slow for this. What's fascinating here is the automated solution we use. It's called artifact signing. Artifact signing. Yeah. Artifact signing allows development teams to cryptographically verify
the provenance. It is not just like a digital sticker. It is a hard mathematical proof. Okay, I'm with you. How does it work? When a piece of software is built by a trusted, secure system, that system generates a unique mathematical hash of the code. Sort of like a digital fingerprint. Exactly, like a digital fingerprint. Then it locks that fingerprint using a highly guarded private cryptographic key. Okay. And then the deployment pipeline uses the public key to verify it. Exactly. The deployment server uses the corresponding public key to unlock the fingerprint. Then it
recalculates the hash of the software it just received and compares the two fingerprints. And if they match, you're good. Yes. If even a single zero or one in the code has been altered since it left the factory, the mathematical hashes will not match and the signature is completely invalid. That is so cool. It allows you to implement what we call policy as code. Policy as code. Right. You program your deployment pipeline to automatically reject any software artifact that doesn't have a mathematically valid cryptographic signature. It just blocks it at the door. The
pipeline simply will not allow the deploying to proceed unless the software can definitively prove where it came from and that it hasn't been unexpectedly modified. We're relying really heavily on the automated pipeline here to build the code, check the SBOM, calculate the hashes, verify the signatures. What happens when the pipeline itself, the CICD system, becomes the primary target for attackers? Because that sounds like the golden goose. Oh, it is. It is arguably the most critical target today. CICD systems are incredibly vulnerable, specifically because they
have to be incredibly powerful. Right, they have to touch everything. They hold the keys to the kingdom. To do their job, they need access to source code, they need to read highly sensitive secrets, they need rights to modify cloud infrastructure, create artifacts, and they need admin credentials to push that code into live production. So if I'm a hacker? If an attacker breaches your CICD environment, they have a direct paved highway from the source code right into your live production servers. Wow. How does the factory actually get
breached, though? Usually it's through poor configuration and violation of just like basic security principles. Like what? One massive issue is excessive permissions. Ah, giving people too much access. Yeah, or giving machines too much access. A pipeline that only needs to update a single microservice might be granted broad, sweeping administrative permissions over the entire cloud account. Just because it's easier to configure that way. Exactly. Another mistake is using long -lived credentials stored as static secrets. Like hard -coding a password?
Yes, which could be stolen from a configuration file and used indefinitely by an attacker. The core principle that must be applied here is least privilege. Every single workflow, every pipeline, should be given only the exact access it requires to perform its specific task. And absolutely nothing more. Okay, I get that. But what about third -party CI -CD actions and plugins? Because those plugins make life so much easier for developers. They do. They are very convenient. You want to
deploy to a specific cloud, boom, there's a plugin for that. You want to run a specific security test. There's a plugin. It saves hundreds of engineering hours. Are we really supposed to give up all those one -click cloud deployments just in case? It's a tough balance, but you have to remember those third -party actions and plugins, they are dependencies too. Oh, right. And they are highly privileged dependencies. If your automated pipeline downloads a third -party action from the internet and executes it during every single build, you have effectively invited external code into the most highly privileged part of
your software factory. You're just letting a stranger walk right onto the factory floor. Exactly. Convenience must always be balanced against trust. You need to understand who wrote that plugin. You need to review its permissions, pin its version, and really minimize its access to your secrets. All right. So let's say we get the pipeline locked down. The factory is finally secure. The next challenge is managing the constant unending flow of changes and updates from the outside open source world. It never stops. Right. And the frameworks talk heavily about container security
and the danger of the latest tag. What is that about? Well, modern applications run in containers, which are built from base images. OK. A massive pitfall teams fall into is configuring their build systems to always grab the latest tag for a base image or dependency. Like we talked about with the dependency confusion, it just wants the newest stuff. Right. It seems convenient, but it destroys reproducibility. How so? If your pipeline runs today, and builds the app perfectly
and then runs tomorrow and silently pulls a completely different updated dependency just because of that latest tag, your build is no longer deterministic. Ooh, don't know what changed. Exactly. Running the exact same build process twice gives you two different results. You don't actually know what went into it. To fix this, teams must... Pin versions. Pin versions? Yes. They must specify exact, immutable identifiers, like those cryptographic hashes we discussed, for the code they want. But wait, pinning versions doesn't mean you just
ignore updates though, right? That sounds like a recipe for getting stuck with old, vulnerable code forever. Oh no, definitely not. Pinning is about control, not stagnation. Control. You pin the version to ensure your builds are reproducible. But you use automation to constantly check for updates, run automated tests on the new versions in an isolated environment, and then open pull requests for developers to manually review. So you still update. It's just on your terms. The goal is controlled, deliberate change, not silent, unexpected changes happening on the fly. You
know, reading through the security frameworks for this deep dive, there was one observation that really stood out to me. What was that? We talk so much about how to secure all these dependencies. But sometimes the absolute safest dependency is the one you simply don't add. Oh, I love that. Dependency reduction is perhaps the most underrated security practice in the entire industry. Just don't use it if you don't need it. Exactly. Developers will sometimes import a massive complex library just to use one tiny simple function. Just to
save five minutes of coding. Right. But every single dependency you add is a tax. It is something you now have to monitor, update, investigate when vulnerabilities are announced and eventually replace. It's a liability. It's a massive liability. Before adding anything to the supply chain, teams need to ask if the value it provides actually justifies the security and maintenance burden. And that ties deeply into the human element of open source, too. Absolutely. Take the XZutils back door from early 2024. Oh, that was a nightmare.
For those listening who don't know, you had a critical data compression tool used in basically every Linux system on earth. Everywhere. And it was maintained by one guy just working in a spare time. For free. For free. He got burnt out. his mental health declined, and then a seemingly helpful developer named Gia Tan stepped in to help. Seemingly helpful being the operative word. Right. Because for two years, Gia Tan gained trust, wrote good code, and then quietly slipped a highly sophisticated back door into the global infrastructure. It was pure social engineering
applied to the supply chain. It's terrifying. And this raises a really important question about how we treat open source in general. Yeah. we have critical pieces of global software infrastructure maintained by volunteers with zero enterprise resources. While trillion -dollar companies rely on them. Exactly. For large companies, sponsoring these maintainers, providing engineering resources, or funding the foundations that back them, it isn't just an act of charity. It's self -interest. It is a vital, non -negotiable part of enterprise risk management. If your multi -billion -dollar
business depends on their code, Ensuring their ecosystem is healthy is literally protecting your own bottom line. And as if all of this wasn't complex enough, we have to talk about the rapidly accelerating threat in the room, AI. Oh, AI is changing everything. Because the sheer burden of tracking all these dependencies is driving developers to use AI coding assistance, which, you know, introduces a whole new layer of chaos to the supply chain. AI is absolutely a double -edged sword here. Really? How so? On one hand,
it's great for analyzing code and summarizing security findings. But AI coding assistants generate code based on statistical patterns in their training data, right? Right. They just predict the next work. Exactly. So we are seeing instances where AI will recommend unmaintained packages or even hallucinate entirely fake packages simply because the name looks contextually correct. Wait, fake packages? Yes. A developer instructs the system to pull a dependency that doesn't actually exist. And what happens then? An attacker monitors these common AI hallucinations, registers the fake
package name in a public registry, and fills it with malware. Oh my god. The next time the AI suggests it, the developer downloads a live virus directly into their project. Which means all these controls we've talked about today, like the S -bombs, the provenance, the artifact signing, they become vastly more important now, not less. Oh, infinitely more important. As the volume and speed of AI -generated code increases, human review becomes a massive bottleneck. You can't keep up. So what's the rule, then? The core rule of the modern supply chain must be,
trust the process, not the author. Trust the process, not the author. It doesn't matter if a senior engineer with 20 years of experience or a generative AI assistant wrote the code. The automated pipeline must mathematically verify it. Period. Period. Furthermore, we are seeing the rise of AI agents that can actually modify pipelines and open pull requests automatically. That sounds risky. Very. Those agents must be treated as independent actors within your system. They need strict identities, highly limited permissions, and fully auditable actions. You just cannot
let AI become a shortcut around your security gates. Okay, so what does this all mean? For the team listening right now, how do they actually know if their supply chain defenses will hold up in the real world? You have to run the ultimate test. A test. You cannot just buy a new security tool, plug it in, and assume you're safe. You need to run a supply chain disaster recovery test. Like a fire drill. Exactly like a fire drill. The organization needs to run an exercise where they pretend that a critical, widely used dependency has just been compromised today. OK.
Start a timer. Can the team find it using their S -bombs? Can they understand the business context? Meaning, is this vulnerability sitting in an isolated internal dev tool? Or is it exposed on an internet -facing authentication server? Which changes the priority completely. Right. Can they patch it? Rebuild the application safely through a secure pipeline and deploy it quickly. Because if you can't do that in a low stakes drill, you definitely aren't going to be able to do it when the sky is actually falling and a new log4j is making headlines. Exactly. Security
isn't just about preventing bad things from getting in. It's about your capacity to respond when they inevitably do. Which brings us back to our opening question. The philosophy here isn't to run away from open source, right? Or try to achieve some impossible zero trust environment where we sit in a dark room and write everything ourselves. No, open source is essential. We need it. We do. But the future has to be about verifiable trust. We have to treat every library, every container image, and every pipeline plug -in as a hard security boundary. We have to replace
blind assumptions with hard mathematical evidence. Verifiable trust. Yes. Verifiable trust means you don't just hope the software is safe. You demand cryptographic proof before it ever executes in your environment. So the next time you, the listener, download a new app on your phone or run a system update or pull a library for a weekend side project, I want you to think about that massive, invisible tree of trust you are interacting with. Think about the thousands of interlocking dependencies required just to make a button on your screen turn blue. It really is a modern
marvel, but a fragile one. It is. And I'll leave you with this final, somewhat provocative thought to mull over. We've talked extensively today about cryptographically verifying our software through automated systems, ensuring nobody tampered with the code. But what happens if the cryptographic keys securing the verification system itself are compromised? That is the nightmare scenario. Right. In an entirely automated supply chain where machines are just checking machines, who is watching the watchers? It's like discovering that the inspector who signed off on the wiring
in our prefabricated house, well... was actually hired by the Ursinus. It's a chilling thought. Thanks for joining us on this deep dive. See you next time.
Internal Developer Platforms were built to simplify software development. But as engineering teams begin building AI-powered applications, are traditional developer platforms ready for what comes next? From integrating...
Your engineering team may be using more AI tools than you realize. But do you know which ones have access to your code, credentials, and production systems? From AI coding assistants and locally running models to...
AI agents are moving beyond generating answers. They're connecting to real infrastructure, accessing internal systems, and interacting with production environments. But who controls what they're allowed to do? The Model...
What happens when your AI application is running perfectly, but its answers are wrong? Traditional Site Reliability Engineering focuses on availability, latency, error rates, and infrastructure performance. But...
AI coding assistants are helping developers write code, generate tests, and build applications faster than ever. But is the rest of the software delivery process keeping up? In this episode of DevOps Academy Deep Dive,...