Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

DevOps Academy Podcast

DevOps Academy Podcast artwork

DevOps Academy Deep Dive S1E16: eBPF and the Future of Observability — Can We Monitor Systems Without Instrumenting Everything?

Sep 29, 202620 minAudioSeason 1, Episode 17
ShareXLinkedInFacebookWhatsAppEmail

Modern engineering teams collect more telemetry than ever, yet getting meaningful visibility into production can still require extensive application instrumentation. In this episode of DevOps Academy Deep Dive, we explore how eBPF is changing observability by providing deeper visibility into running systems without requiring developers to modify every application. We discuss where eBPF fits alongside traditional instrumentation and OpenTelemetry, how it can help uncover service dependencies and performance problems, and why automatic observability doesn't mean collecting everything. The conversation also looks at continuous profiling, Kubernetes environments, AI-assisted troubleshooting, security, privacy, and the limits of zero-code instrumentation. 🔑 Key Takeaways What eBPF is and why it matters for DevOps and SRE teams How eBPF provides visibility without modifying application code Where traditional application instrumentation is still necessary Why technical visibility and business context are different How eBPF can help uncover real service dependencies Why it can be especially valuable for legacy and third-party applications How continuous profiling adds deeper performance context Where eBPF and OpenTelemetry complement each other How AI could correlate runtime telemetry and accelerate troubleshooting Why the future of observability is likely a combination of automatic discovery and intentional instrumentation 🎧 Tune in to explore whether the next generation of observability will require developers to instrument less while still understanding production more deeply. 💬 Could your team understand production without manually instrumenting every service?

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

What if I told you that the code you just spent maybe six months writing is entirely blind to how it's actually running in production? Oh, absolutely. Right, because every single line of manual telemetry you added, like every log statement, every metric, it's essentially just a guess. You're asking the application to report on its own health from inside a burning building. And that's assuming it even has the capacity to tell you the roof is collapsing. It really is the ultimate paradox of modern engineering. I mean we deploy these hyper complex distributed

micro service architectures and our primary strategy for understanding them is You know hoping the application remembers to politely inform us when it's failing Right, which is just wild when you think about it The mission of our deep dive today into this DevOps Academy session is to explore a technology that completely flips that dynamic We're looking at eBPF. Yes Exactly. And we are going to try to answer a question that feels like a total pipe dream for anyone out there carrying a pager. Could we eventually understand

our production systems without having to explicitly, you know, manually instrument every single application we write? It's a huge question. OK, let's unpack this because traditional observability relies entirely on a concept that scales just terribly, which is developer cooperation. The whole traditional model hinges on the application doing the heavy lifting. I mean, you need developers to import the right telemetry libraries, initialize them correctly, manually wrap their database calls

in trace bands, and then propagate those context headers downstream. Which never happens perfectly? Never. If someone forgets to pass a context object in Go or, say, an unhandled exception in Python kills the thread before the log flushes, that telemetry just ceases to exist. you are left with a massive blind spot. I want to visualize this for you because the friction here is just massive. Relying on manual instrumentation, it's like throwing a massive party and requiring every single guest to explicitly wear a nametag. That's

a great way to put it. Right? Yep. And they have to explicitly announce every conversation they have to the entire room. If someone forgets their nametag or if, say, a legacy app doesn't have the right telemetry library, they just become a ghost. They're at the party, but you have no idea they're there. That captures the manual effort perfectly. And you know, when the industry tried to fix that, they introduced agents or sidecar proxies. So sticking with your analogy, it's like instead of name tags, you hire security guards who interrogate every single guest before

they can talk to anyone else. Oh, wow. Yeah, that sounds exhausting. It is. You capture the data without asking the guest, but you've just created massive congestion. Every network packet has to be routed out of the application. into the sidecar, processed, and then routed back. It adds latency and just massive memory overhead. So it's a huge bottleneck? Exactly. But eBPF is entirely different. eBPF is more like looking at the floor of the party. It provides a controlled,

verified mechanism to run small programs that observe the system at its most foundational level, without asking the application for permission at all. But how does it actually do that? Because... You know, I hear the floor of the party or like a sensor built into the foundation and I immediately think of the operating system, right? The Linux kernel. You're right on the money. But modifying the kernel sounds like a fantastic way to just panic a production server. Yeah, and you're hitting on the exact reason EBPF took a while to really gain mainstream trust. The foundation is indeed

the Linux kernel. I mean, every application, whether it's a node backend or a Rust service, eventually has to ask the operating system to do the actual work. Like reading a file or opening a network socket? Exactly. Allocating memory, all of it. These all require system calls to the kernel. And EVPF allows us to attach custom programs to those exact events using mechanisms like kprobes for kernel functions or trace points. Wait, wait, wait. You're telling me... we are

injecting custom compiled code on the fly directly into a running production kernel. Basically, yes. That sounds terrifying. If I write a bad while loop in an eBPF program, do I just freeze the entire node, like bring the whole server down? You absolutely would, if the kernel just blindly executed your code. But the real genius of eBPF is the verifier. The verifier. Yeah, so before any eBPF bytecode is allowed to attach to a kernel hook, it has to pass through an incredibly

strict verification engine. The verifier analyzes all possible execution paths of your program. So it checks the homework before it runs? Thoroughly. It ensures there are no infinite loops, it verifies that you aren't accessing uninitialized memory, and it strictly limits how complex the program can be. If your code can't be mathematically proven to be safe and determinate quickly, The kernel outright rejects it. Oh, that is wild. So it mathematically guarantees we won't crash

the machine. We get this sensor in the floor, completely frictionless, reading every system call. But here's where it gets really interesting, though. If eBPF gives me this godlike view of every network packet and every drop TCP connection at the kernel level, why do I even have an observability team anymore? Let's just... Fire them today! Rip out all our manual open telemetry libraries and just let the Colonel tell us everything. Well, I mean, if you want to be completely blind to your actual user journeys, you could certainly

try that. Ouch. Okay. Fair point. It's the trap a lot of platform teams fall into during the hype cycle. You really have to distinguish between technical context and business context. Right. The sensor in the floor is brilliant at telling you that service A talked to service B or that a connection failed or that CPU spiked. But it has absolutely no idea why that matters. I see. So the kernel sees like a socket connection drop or it sees a latency spike of 400 milliseconds

on a specific port? Exactly. EBPF tells you the database query took four seconds, but the kernel doesn't speak shopping cart. It doesn't know that the query was, say, a VIP customer trying to execute an $800 checkout during a holiday rush. It just sees bytes moving across a socket. The application code is the only layer that possesses the business logic. Because it knows the user ID, the inventory count, whether they selected express shipping. Right. Right. If you strip out all application -level telemetry, you know

the infrastructure is struggling, sure, but you have zero idea how it impacts your bottom line. Okay, so the future isn't eBPF replacing everything. It's a hybrid model for your organization. You get this platform that automatically provides the baseline technical visibility via eBPF. You already have visibility, yeah. So developers get to start from a better baseline, and they only spend their time adding custom manual telemetry to track the business -specific stuff. That's the perfect division of labor. Let the kernel tell you how the system is operating, and let

the application tell you why it matters. And the reality is, you absolutely need that underlying kernel truth, because modern infrastructure is practically designed to lie to you at the application layer. Yes, let's talk about those lies. Because one of my favorite takeaways from the source material is how EBPF exposes the sheer hilarity of static architecture diagrams. Oh, they're the best. You know the ones. A lead engineer spends three weeks in a drawing tool connecting perfect little boxes with straight lines claiming, you know, this is exactly how our microservices

communicate. And then the moment you drop an eBPF sensor into the Kubernetes cluster, you realize the diagram is total fiction. It's like flipping on the lights in a dirty kitchen and watching the roaches scatter. You suddenly see a node service talking to like an undocumented external endpoint or some deprecated Redis cache that was supposed to be decommissioned in 2022. That happens way more often than anyone wants to admit. In a dynamic environment like Kubernetes, IP addresses are just ephemeral. Pods spin up, they crash, they get rescheduled on a totally

different node seconds later. Right. It's constantly shifting. Exactly. So, if you rely on application -level logging to understand your architecture, you only see what the application intended to do. eBPF doesn't care about intent. It just sees what actually happened. Right. It sits at the network interface. If a packet leaves a container, eBPF sees it. Which is huge for legacy applications too, right? Like if you inherited a compiled black box Java monolith from an acquired company and the original developers are long gone, you

can't exactly go in and add new telemetry libraries to the source code. No, you can't touch it. But with eBPF, you don't have to touch the code. You just watch the system calls. But hold on. If we are intercepting and inspecting every single network packet leaving a workload to map the architecture, Aren't we crossing a line there? Like, we're not doing observability anymore, we're basically doing security. Well, the boundary between the two is dissolving, and EBPF is definitely the catalyst for that. At the kernel level, an operational failure and a security breach look

remarkably similar. Really? How so? They're both just anomalous system behavior. If a workload suddenly opens a connection to an unknown external IP address in a foreign country, your traditional observability tool might just see, I don't know, a spike in network throughput. But eBPF sees the exact syscall. Yes. It sees the process ID that initiated the call, the binary that executed it, the destination payload. Both disciplines are now just analyzing runtime behavior. In fact,

security teams are using eBPF to enforce policies at runtime, basically terminating unauthorized executions before they even complete the syscall. So what does this all mean? Because we've got this omniscient sensor in the floor now. It sees every network packet. It maps our chaotic Kubernetes architecture. It even intercepts security threats. We are generating an unfathomable amount of raw data. Oh, absolutely. Aren't we just taking the classic DevOps problem of we don't know what's happening and replacing it with we have so much kernel data our engineering team is paralyzed?

Like, it feels like observability overload. If you do it without a strict information strategy, yes. you will build the world's most expensive data swamp. Just because eBPF can observe every single TCB retransmission, it doesn't mean you should stream all of those events to an expensive cloud data warehouse and store them for a year. Right, the storage costs alone would be insane. How do you filter it, then? If the kernel is firing millions of events a second, how does an engineer actually interact with that data meaningfully without just losing their mind?

The best implementation of this, which is highlighted in the source material, is continuous profiling. In the past, profiling an application to see exactly where CPU time was being spent meant attaching a really heavy debugger to the process. Right, and that completely destroyed performance, so you only did it in staging, never in production. If I attach a debugger in production, the app grinds to a halt and customers start getting timeout errors left and right. Exactly the problem. But eBPF... allows for sampling at the kernel

level. So instead of wrapping the application, an EBPF program could just wake up, say, 99 times a second, look at the CPU, record the exact stack trace of whatever function is currently executing, and then go right back to sleep. And that's fast enough not to break anything? The overhead is virtually undetectable. So instead of a generic dashboard metric telling me, hey, the billing service is running at 90 % CPU, I actually get something actionable. You don't just get the symptom, right? You get the actual culprit. The

EBPF profiler aggregates those stack traces and generates a flame graph. Oh, I've seen those. Yeah. So you can look at live production traffic and say, OK, we aren't just using 90 % CPU. A specific regex parsing function, which was introduced in yesterday's deployment, is consuming 60 % of our total compute cycles. It perfectly bridges the gap between infrastructure metrics and a specific line of code. That is incredible for our post -mortem, like being able to pinpoint the exact line of code. But I want to play devil's

advocate here for a second. Go for it. We're still adding another pane of glass. An on -call engineer at 3 a .m. now has to look at their business metrics, their distributed traces, and then they also have to interpret a kernel -level CPU flame graph. The cognitive load is getting absurd. If I'm exhausted, I'm not manually correlating a spike in a trace ID to a specific EBPF stack trace. And you shouldn't have to. The industry spent the last 10 years figuring out how to extract telemetry. The next 10 years are entirely about correlation. And that's really where the integration

of AI models becomes genuinely powerful, rather than just being a gimmick. Okay, I'm a bit skeptical. How does AI actually help here beyond just, you know, summarizing log files? Think about the mechanics of an investigation. A human engineer looks at a failing trace ID. They check the timestamp. Then they pivot to a Metrix dashboard for that same timestamp. They notice a CPU spike. Right, all manual steps. Then they pivot to a profiling tool to see what was running during that spike.

An AI assistant, hooked directly into the observability backend, can do that multi -dimensional correlation instantly. It uses vector embeddings of your logs, your traces, and your EBPF profiles to link causality. So instead of me hunting through five different dashboards, I could just ask the platform, why did checkout latency jump five minutes ago? Exactly. And the AI doesn't just guess. It pulls the specific trace ID, links it to the EBPF CPU profile for that exact millisecond, and tells me, hey, the thread hung on this specific

string parsing function in the node service? That's the vision. It gives you a highly educated starting point for your investigation. But, and this is a really critical caveat, that AI workflow is completely dependent on the underlying telemetry being trustworthy and structured. Which is always the catch. Always. Large language models are fantastic at pattern matching. But if you feed them fragmented, disconnected telemetry where the spans don't match the network topologies, they will confidently hallucinate a completely

wrong diagnosis. Which brings us back down to reality. Because, you know, for you listening, zero code observability doesn't mean zero governance. I want to push you on the actual cost of this magic. You said earlier that eBPF has virtually undetectable overhead when profiling, but low overhead isn't zero overhead. We are running code in the kernel on every single node. What happens when teams start over deploying eBPF agents? This raises an important question, honestly. You hit the wall of physics eventually. Every

context switch, every memory lookup in a BPF map consumes CPU cycles. If you write an eBPF program to trigger on a network packet event that occurs, say, five million times a second, and your program tries to do a complex lookup on every single one of those packets, your node's CPU is going to burn. So the verifier ensures your code is safe from crashing, but it does not guarantee your code is efficient. Bingo. You will choke the machine if you're not careful. And beyond the performance aspect, there is a

massive data governance risk here. Really? Yeah. Because powerful visibility creates powerful access. If an eBPF program sits at the socket layer to measure latency, it has the capability to inspect the payload of that packet before it gets encrypted or right after it gets decrypted. Oh, wow. Wait, so if a customer submits a password or a credit card number, the observability tool could potentially see it in plain text sitting in memory? Depending on where and how the EBPF

hooks are placed, yes. This is a huge deal for platform teams. Your observability agent is now a critical security boundary. That's terrifying. You must treat it with the same strict access controls, auditing, and compliance standards as your primary database. You have to ensure that the EBPF tools are configured to only extract the metadata, like headers, IP addresses, latencies, and explicitly drop the sensitive payloads. And if I have to put that much trust into the underlying agent, I really don't want to be locked into a single vendor's proprietary EBPF black box.

Because if one company controls how I get my kernel data, they can basically charge me whatever they want. Which is exactly why the adoption of OpenTelemetry alongside eBPF is so vital right now. OpenTelemetry provides an open source standardized pipeline for generating and routing observability data. So you use eBPF as the auto instrumentation engine at the bottom. It gathers the kernel events, the network flows, the profiles, but instead of sending that directly to a vendor's proprietary

backend, EBPF formats that data into open telemetry standards. I see. So the data leaves the kernel as a standardized open telemetry metric or trace. Correct. You pipe it into an open telemetry collector, and from there, you can route it to whichever vendor or open source backend you prefer. It fundamentally decouples the collection of the data from the storage and analysis of the data. Which means you maintain ownership of your own runtime intelligence. Exactly. It's the ultimate insurance policy against vendor lock -in. That

makes total sense. So let's synthesize where this leaves us. We started with the sheer agony of manual instrumentation, right? The blind spots, the ghosts in the network. EBPF fundamentally revolutionizes that baseline. We get the sensor in the floor. We automatically see the network topology. We get continuous CPU profiling, and we uncover security anomalies, all without explicitly modifying our application code or crashing the kernel. But you still have to supply the meaning. That doesn't go away. Right. EBPF provides the infrastructure reality, but the developer still

has to inject the business context. Because a technically perfect 200 OK response at the socket level means absolutely nothing if the application logic is silently dropping database rights. You have to know the business impact. If we connect this to the bigger picture, my advice to engineering teams is simply this. Do not adopt EBPF simply to chase a trend. Evaluate your actual pain points. Are you spending weeks trying to troubleshoot legacy microservices? Are you completely blind to your Kubernetes network traffic? Use eBPF

to solve those specific problems. Ultimately, success is not measured by how many terabytes of kernel events you capture. It's measured by how rapidly your team can answer a simple question. Why is the system broken right now? Less effort to get a much better answer. I love that. But that leaves me with one final slightly provocative thought for you to chew on. We've established that eBPF gives us this pristine low -level execution baseline. And we've discussed how AI can ingest

that baseline, automatically correlating, say, a latency spike to a specific inefficient regex function running in the kernel. The ultimate diagnostic loop. Right. But if that vision fully matures, I mean, if the platform becomes perfectly capable of observing its own execution, pinpointing the exact line of failing code and mathematically explaining the inefficiency to us, will the technical art of debugging eventually just disappear? If the infrastructure can automatically explain exactly how and why it's failing at the hardware

level, will the developers of tomorrow simply become business logic designers who never actually need to understand how the Linux kernel executes their code? Maybe in the future, you don't need to know how the machine works. You just need to know how to ask it what hurts. Thanks for taking this deep dive with us.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. DevOps Academy S1E22: Platform Engineering 2.0 — Are Internal Developer Platforms Becoming AI Platforms?

    Nov 16, 202628 minAudio

    Internal Developer Platforms were built to simplify software development. But as engineering teams begin building AI-powered applications, are traditional developer platforms ready for what comes next? From integrating...

    Credit the people on this episode

  2. DevOps Academy S1E21: Shadow AI Is Becoming the New Shadow IT — Do You Know What Agents Are Running in Your Organization?

    Nov 9, 202623 minAudio

    Your engineering team may be using more AI tools than you realize. But do you know which ones have access to your code, credentials, and production systems? From AI coding assistants and locally running models to...

    Credit the people on this episode

  3. DevOps Academy S1E20: MCP in Production — Are AI Agents Creating a New Infrastructure Security Boundary?

    Nov 2, 202622 minAudio

    AI agents are moving beyond generating answers. They're connecting to real infrastructure, accessing internal systems, and interacting with production environments. But who controls what they're allowed to do? The Model...

    Credit the people on this episode

  4. DevOps Academy Deep Dive S1E19: AI Workloads Fail Differently — Is Traditional SRE Enough for Production AI?

    Oct 26, 202623 minAudio

    What happens when your AI application is running perfectly, but its answers are wrong? Traditional Site Reliability Engineering focuses on availability, latency, error rates, and infrastructure performance. But...

    Credit the people on this episode

  5. DevOps Academy Deep Dive S1E18: AI Is Making Developers Faster — Is Infrastructure Becoming the New Bottleneck?

    Oct 19, 202623 minAudio

    AI coding assistants are helping developers write code, generate tests, and build applications faster than ever. But is the rest of the software delivery process keeping up? In this episode of DevOps Academy Deep Dive,...

    Credit the people on this episode