Go 1.26.6 & 1.25.13 pre-release announcement Go SF Meetup, Aug 26 GopherCon Israel CFP extended through Aug 15 Discussion: A more robust proposal process Interview with Kemal Akkoyun Episode 44: Interview with Jon Bodner Datadog APM checklocks ★ Support this podcast on Patreon ★
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Jonathan HallMost people are building things that users see. You're you're not even building the stuff that the engineers see. You're you're building the stuff that feeds the stuff that engineers see. So what excites you about this?
Kemal AkkoyunWe should have definitely talked about this, like, in earlier.
Shay NehmadWe might put this as the hook. Right? Some people think it's boring. That would be a good hook.
Jonathan HallThis is cup of go for some random day in July.
Kemal AkkoyunDon't know.
Shay NehmadIt's August 10.
Jonathan HallOh my gosh. Let me try that again. This is Cup of Go for some random day in August. Keep up to date with the important happenings in the Go community in about fifteen minutes per week. I'm Jonathan Hall.
Shay NehmadI'm Shay Nehmad, and today, we're not gonna do either.
Jonathan HallYeah. We're not doing this is a weird episode.
Shay NehmadEverything's wacky. We're recording on a Monday instead of a Friday. And our editor, Filippo, is on vacation. Hello, Filippo. I hope you're enjoying your vacation. So the edit is gonna take longer, and we don't have a lot of time. So we're not gonna do a normal episode. Things that we will talk about in the future include the new prerelease that's coming up, one twenty six point six was announced. We're gonna talk about CRAP scores. We had an interview about it and actually built it's, like, evolving in our Slack channel, which is really cool. All those meetups or whatever, but we're only going to talk about two things this episode.
Jonathan HallI think it needs to be three things. Yeah. We need to talk about Go For Con Israel CFP also because it's closing soon. Oh, right.
Shay NehmadRight. Right. Okay. So three things. Yeah. Two short ones, and then I'll ask Jonathan how Go For Con US was because he was cool and went, and I was at home and cried. So two short notices. Go SF. Go San Francisco. I'm arranging a meetup. It's coming up real soon, and we would love to have y'all. It's hosted by c one dot a I, Conductor One, and it's in San Francisco, and it's gonna be August 26 coming up real soon. Sign up. And the second thing about Go for Con Israel, the call for papers has been extended. So if you have if you wanted to put an idea and you haven't had time to do it yet, you have time until very soon.
Right?
Jonathan HallYeah. I don't
Shay Nehmadsee Five days from now. Yeah. I don't know when this episode will come out. It might be too late even.
Jonathan HallAugust 15. Hopefully, this comes out before then.
Shay NehmadJust two short notices.
Kemal AkkoyunHow
Shay Nehmadwas Go For Con US?
Jonathan HallI had a blast. It was fun meeting people, a bunch of people I'd never met at all who knew me through the show. Shout out to everybody who listens. Yeah. Cool. Met a bunch of people who've been on the show, but I have not met in person or otherwise have interacted with but not met in person. That was fun. I made some tentative arrangements for a number of interviews that will be coming on the show, details to be announced as they solidify. But we'll probably be talking You went recruiting. Oh, yeah. Oh, yeah. Just a little sneak peek. We'll probably be talking about some bees. We'll probably be talking about TinyGo and AI, and Oh. Yeah.
A number of other things. Some interesting companies using Go that we haven't talked about before. So there's a few things that are in the pipeline. Hopefully, bunch of those will pan out to be good interviews in the future.
Shay NehmadThe weather up in Seattle? It was in Seattle. Right?
Jonathan HallYeah. It was in Seattle. It was beautiful weather, comfortable for walking around Downtown Seattle without a jacket. Shorts and T shirt was perfect. Had some great food, some great drinks.
Shay NehmadDid you have your GO tie on?
Jonathan HallI did not take that, actually. I did wear my Cup of GO shirt, but I didn't
Shay NehmadDid you wear your new shirt?
Jonathan HallI did. I did. Yeah. And Awesome.
Shay NehmadHow is it? I
Jonathan Hallhaven't worn
Shay Nehmadit yet.
Jonathan HallIt was comfortable. Yeah. And a number of people are like, oh, are you related to Cup and Go? I'm like, yeah. What do you mean? So, like, people who knew this show but didn't know my voice, were like, how are you related to Cup of Go? Like, well, I'm one of the co hosts. Oh, I listen to that show all the time.
Shay NehmadFor real? They didn't know your face when
Jonathan Hallyou knew voice? People. A few people did that. Most of the people would recognize my voice. I'd sit down and start chatting, and they go, oh, I recognize your voice from Cup of Go. Well, great. But, yeah, people will do the show, but not me. So Did you go
Shay Nehmadto any of the talks
Jonathan Hallor workshops? I did go to some talks. I didn't go to any workshops. I mostly attended the hallway track, but there were some good talks, and some of those talks led to me inviting people to to hopefully come on the show. One of the talks in particular, I think deserves a fair amount of attention on this show, and that is the one from from Austin who talked
Shay NehmadI about was worried it's going to be like a crime true crime podcast about our podcasts.
Jonathan HallNo. No. So Austin had a proposal that we haven't talked about yet since we haven't been the show for a while. Or it's actually a discussion. It's not a proposal, but it's a discussion about a more robust proposal process.
Shay NehmadOh, I've seen that. I've seen that, and I wanted to talk about it.
Jonathan HallAnd that was the one of the opening talks at the conference was Austin talking about this proposal they've created or this discussion they've created and sort of going through it. And so I'm I'm hopeful that we can get Austin on the show to talk about it too. I haven't reached out to them yet, but I intend to. But we can certainly, when we have time well, I don't know if it's today or another time, but we can go through some of this and talk about it on the show.
Shay NehmadIn high level, his proposal is the backlog is growing too fast, and the priority isn't very clear. So maybe let's create, like, a better process for managing the backlog.
Jonathan HallRight. So it it's not like an overhaul. We've we've talked about the proposal process both directly and indirectly on the show many times. Right? It's not really an overhaul of that as so much as, like, a streamlining and adding abilities to make things get lost less often and some more community feedback.
Shay NehmadI'm I'm down. I I don't know. I feel like my very small contributions to the proposal process have not been very thoughtful on my end, usually. Usually, I I'm I'm more of a watcher. Like, look at these things. You're an actual contributor. You've you've fixed the line of documentation or something.
Jonathan HallYeah. Right. Exactly.
Shay NehmadBut it always feels like, you know, the issue is, like, at some level of quality, if it gets, like, you know, maybe four or five interested people, it goes very well. If it gets more than 20 interested people, just no way it's ever gonna get close until one of the maintainers is like, okay. This is what we're doing. Stop bike shedding about everything. I assume a lot of them, especially ones that we don't see, are just like a proposed, and then like, okay. It's moved to the backlog column and just dies there. I mean, it is an incredible amount of issues. Right? How many issues does the Go project, like, have on GitHub?
Jonathan HallSo issues, I mean, I I don't know how many open ones there are. There are over 80,000.
Shay NehmadThere are more than 5,000.
Jonathan HallYeah. I mean, like
Shay NehmadOh, almost 10,000. That's, like, rough.
Jonathan HallBut I don't know what percentage of those are proposals. I'm sure it's a small percentage, but even a small percentage of 10,000 is a large number. So
Shay NehmadYeah. Anyway, interesting proposal. We may dig into it deeper with Austin or without him, I guess.
Jonathan HallMhmm.
Shay NehmadYep. Other than what is the hallway track? Like, is it
Jonathan Hallthe That's
Shay Nehmadjust talks or whatever?
Jonathan HallNo. That's just the hallway. It's euphemism for hanging out with people. Alright.
Shay NehmadYeah. That's the best place in a conference, not actually the the talks.
Kemal AkkoyunBecause the talks,
Shay Nehmadyou can watch on YouTube later.
Jonathan HallYou can, generally. Much later sometimes.
Shay NehmadWho's who's who are the big sponsors? Like, the big sponsor's name. Just shout shout them out if you had such a good time.
Jonathan HallThey had a bunch of sponsors this year. The big ones were, of course, Google, Microsoft, Capital One, Bayer.
Shay NehmadWhat's Bayer? Capital One's a credit
Jonathan Hallcard. Aspirin company.
Shay NehmadOh, for real?
Jonathan HallYeah. Yeah. They do biotech stuff, biotechs and pharmaceuticals and whatnot.
Shay NehmadAwesome. Maybe they use the
Jonathan HallJetBrains, DNA which is Filippo Valvassori's company, was one of the sponsors. Stable Kernel, it's a company I recently interviewed with. By the way, I'm looking for work, so if you're if you have an opening on a on your team for a Go developer, reach out. Yeah. There were a bunch. I'm not gonna remember them all.
Shay NehmadWell, good on the sponsors for making such a nice event happen. I'm really sad I missed it. Hopefully, next year I'll be able to justify going.
Jonathan HallWord on the street is that next year will almost almost certainly, very likely be in Toronto. Toronto?
Shay NehmadYeah. Interesting. I hope, like, East Coast, because I didn't get a chance to be in the East Coast yet. Maybe New York. Let's see. I don't know if New York is a big, like, go city. Think it's
Jonathan Halla Montreal. That's East Coast ish.
Shay NehmadIt's East Coast ish. I haven't, like, been to The Atlantic.
Jonathan HallI mean, it's, like, almost to New York, but not quite.
Shay NehmadYes. Cool. So I'm glad you had a good time. It, of course, messed with our programming schedule, but that's worth it for you to collect all these interviews and get all the star power and whatever. That's awesome. Yeah. No ad break, you know, as usual. We you can find everything in cupofgo.dev, including the link to our Patreon, which is the best way to support us. Thanks a lot to all our Patreon supporters for helping us maintain this rather expensive hobby. Mhmm. It's getting more expensive now that we're doing video as well. Yeah. Hey, if you're watching on YouTube, let us know if we're doing a good job with the shorts or not. I'm still I'm still working out how to do it. Considering how much time I spent on, like, trying to kill my brain with short form videos on social media, I'm not very good at creating them, I guess.
And stick around because we have a very interesting interview. What's coming up, Jonathan? Let
Jonathan Hallhim know. Yeah. We're gonna be talking with Kamal from Datadog about observability.
Shay NehmadYes. Trace tracing my stolen bike. Let's see how that goes.
Jonathan HallWith with the go run time.
Shay NehmadYeah.
Jonathan HallAlright. Stick around for that. And we'll hopefully be on a regular schedule with regular programming again in about a week.
Shay NehmadIt's the summer, so we'll
Jonathan Hallsee. Yeah.
Shay NehmadWe'll see. Everything's kinda haywire. We will get back to normal programming pretty soon.
Jonathan HallAlright. See you then.
Shay NehmadJonathan.
Jonathan HallHi, Shay.
Shay NehmadWhat's up? I'm I'm into a really serious detective case right now.
Jonathan HallReally? What are you selling?
Shay NehmadSomeone stole my bike in San Francisco. It's really bad. Yeah. If only I could, like, trace it. If only could someone help me trace things. Oh, hi, Kamal. Hello. Maybe the worst maybe the worst the worst jokes we've had. I'm sorry.
Jonathan HallWhere's all this line? We're just trying to see how low we can go.
Kemal AkkoyunNow I've seen how it happens. It's it's fun.
Shay NehmadWell, the joke is, you know, the best jokes are explained, that tracing is something you do with eBPF, and Kamal is here to talk, about that. How about you introduce yourself to our beautiful listeners?
Kemal AkkoyunYep. Yeah. My name is Kemal. Kemal Akkoyun. I'm originally from Turkey, but I'm living in Germany, Berlin now. I work for Datadog. I've been working on observability tooling and goals since, I guess, at this point, 2020 like, 2017, 2018. Right now, I'm mostly focused on instrumentation side of things that includes eBPF, compile time instrumentation, which I'm mostly here because of that. We're gonna I'm I wanna talk about a lot of the cool things that we've been building as part of OpenTelemetry and Go. Basically, yeah, That's what I've been doing.
I yeah. Besides that, yeah. Observability, I am main maintainer of a lot of c c n c f projects. I'm helping to develop Prometheus. I used to work on TANOS, a lot of metrics. Yeah.
Shay NehmadThat's Well, I I use all these things all the time.
Kemal AkkoyunAwesome. Do they do you
Shay Nehmadwork on this stuff as part of your, like, job in Datadog? Are Datadog, like, sponsor sponsors of this? Or was it the other way around? You worked on these things and then joined Datadog?
Kemal AkkoyunThe other way around. I used to work for Red Hat, and I used to work in this team where we collect observability data for all these OpenShift clusters. So Kubernetes, Prometheus, Tanna, Sjager, all those sort of open source projects that I worked for, then I eventually moved to more of a data collection side of things where it's like I worked on profiling, I worked on eBPF, and then went into go internals. Because of that work, I ended up working for Datadog. I'm not doing metrics anymore as you joked.
I am doing a lot of tracing, distributed tracing nowadays. And I that's what I do for Datadog. APM, Application Performance Monitoring, specifically focused on Go. I work for a team called language platform. We build and maintain SDKs on the collection side, whatnot. Actually, I think John Bodner was a guest. He used to work for this team. So I'm trying to fill in his shoes. And yeah. And a lot of things that we talk about, he actually started.
Shay NehmadYeah. And you know our guests. I wouldn't even remember. I think That's amazing. John might have been an interview to Jonathan. Did you did it on your own. Right? I was like sick
Jonathan Hallor something. You weren't able to make a decision for some reason.
Shay NehmadSomehow all the good interviews happen when I'm not here. I don't know.
Kemal AkkoyunOh, is this a bad interview then?
Jonathan HallOh, no.
Shay NehmadLet's see. Let's see. Let's see how it goes. So not sponsored yet. I don't know if Datadog will sponsor us. That'll be freaking awesome. But APM is a really good product. Really, really good product. Like, I don't know how many of our listeners, like, know of these things. I sort of assume it is basic knowledge just because I've been working in distributed systems for so many years. But assuming our listeners don't really know what this stuff even means, just to, you know, get things started, what is, like, this APM distributed tracing product, and what are the, like, challenging parts of it you're solving with Go?
Kemal AkkoyunAPM stands for application performance monitoring. We basically try to collect performance statistics from the application layer. That also that means like in the simple terms, we are collecting a lot of tracing data, distributed tracing data. When we said tracing, tracing is basically you have an application, there's a request, it started here, it ended here, you spend that much time on this function or in this service, then we call the distributed system, and then you can just drill down and see okay, like I'm actually spending a lot of time, let's say 30% of this request in this microservice, and I need to maybe dive deeper, this shouldn't be supposed to be like that. And then you can also associate this data with a lot of other signals, with your logs, with your other metrics, your infrastructure metrics, whatnot.
You can also attach profiling data to this and get the nitty gritty details of your process and check where you can optimize whatnot. So that's the gist of it. And yeah, we also provide basic health metrics, We call them red red metrics for online services, which is like request rate, error rate, and the duration.
Jonathan HallHow does how does what you do differ from or expand on what things like PPROF do sort of out of the box?
Kemal AkkoyunYeah. Pprof is just it is for profiling. And profiling signals is about actually collecting the resource usages of a process. Right? It's not request bound. You are basically how pprof works, it gives signals to go runtime and each let's say that every second, like, collect 10 times of, like, snapshot of what CPU does. Right? It's just the stack traces associated to the CPU usage or you you can associate that to a memory usage allocations or like the lack contention or whatnot. That is profiling, but the tracing is mostly request based. Right?
You got an HTP request, you got a gRPC request, and now we call that this is like transactional, so you have a transaction and within that transaction you are trying to gather like what are you doing. You are trying to get the timings of that data. And the nice part of this, you can also, like, attach a metric profile to this. So while you are handling this request, this was what's going on in your CPU or in memory. And then you can, using that data, find out the bottlenecks of your service, like, detect where to optimize, where not to optimize.
Jonathan HallAlright. So let's trace back to the beginning of this conversation.
Shay NehmadYes. After all the technical difficulties. Yes. We compared we compared p professor to this, like, distributed tracing thing. I'm also interested you mentioned, you know, you had to go into, like, the internals of Go. I mean, Datadog, for listeners who don't know, is, like, a huge company. Right? You have the biggest customers and, like, whatever. Maybe you're the leader in this industry. I don't know. I assume you are. So what sort of, like, Go internal challenges did you have to go into to solve all this stuff?
Kemal AkkoyunYeah. So we are one of the biggest observability companies. Yes. And we use Go a lot. Like, pun intended, and we dog food the software a lot internally. So everything we build, build with Go, and we use, like, distributed tracing profiling a lot. And we run this at scale and we have like large customers and they run this software at scale and then we discover a lot of like problems with the runtime whenever some upgrade happens, whatnot. This is not the team that I'm working on, but we have a dedicated profiling and runtime team. And they're actually contributing back to Go runtime, like crazy. I think we are one of the biggest outside contributors, like, aside Google.
And we are mostly focused on those runtime. So we have, for example, a component for, like, Go runtime metrics. Go runtime itself already, like, exposing these metrics. But sometimes these measurements are not correct, and we find these and report these back and sometimes go and fix them. Or like, we come across this interesting platform combined with, I don't know, some weird Linux version and some hardware, and then that was not tested for Go, but it's technically, you can compile that against Go, and you discover these different things.
And then we reported a lot of things like that, for example, and I think we fixed that as well. But like what we do, in my team, we are focused on the compiled time instrumentation. This is what we try to provide for like an auto instrumentation zero touch observability for Go applications. This is easy for Java, Python, and Ruby because they have Ruby like, run times, and they allow a lot of dark magic. You have a load time. You can rewrite the binary code the byte code whatnot, so a lot of things that you can pull. But Go is simple, and it's really hard to it's really hard. It's not possible to do those things. Right? So our solution was Which
Shay Nehmadwhich, just to clarify, is a good thing, right, in your opinion? Yes. Because on one side you said like, oh, it's easy in Java. On the other side, you said, it's like dark magic. So wait, is it good or not?
Kemal AkkoyunIt is it is bad. It's dark magic. Right? Also, what we are doing is also dark magic. Right? It's not sanctioned at
Jonathan Hallall.
Kemal AkkoyunSo it's when I first, like, encountered this, okay, coming from all these, like, infra work go after years and like, who who whoever actually uses this in production, this is crazy. And then turned out, like, people love this. What we do with compile time, there is actually this, like, flag called tool and flag, which you can, like, pass a script, any binary, and then go tool chain itself would report any build tool chain step to that, like compile and link, whatnot with all the arguments that it's passed to. So the original idea behind this flag was to maybe measure the latency of these steps and like improve basically for good for Go team itself. Right?
We basically piggyback onto this flag and started to rewrite the abstract syntax tree AST during these compilations, and we started to pull any dependencies that we want and like basically introduce the concepts for from like Java, like aspect oriented programming, right? You want to have some security, some observability, but this is independent from your business logic, you need to add those and we just manipulate the AST, add those code in place, and feed that back to the compiler, and feed that to the linker, and everything is instrumented automatically. So that's the dark magic part. And this is the tool that we build and we are injecting our own SDK for that. This tool is also open source, it's called Orchestrion, but then we wanted to like move this to the more neutral governance model.
There were similar tools as well, so we donated this tool to the OpenTelemetry. And there were, as I told you, there was a competing tool from Alibaba, which is a Chinese company and they were doing things a little bit differently, but the same concept, same APIs, they were like changing. And then OpenTelemetry governance said that okay, let's form a special interest group and like you work on these tools. And this is what we've been doing for the past two years. And we built yet another tool.
So this could be like second or third iteration of the same idea for Go. But now it's in a neutral governance model, and like you can just collect these things for OpenTelemetry, and you can send your data directly to an OpenTelemetry collector.
Shay NehmadSo So basically, the options I have, if I understand correctly, is one, maybe the simplest one, but requires code changes, is to go into my code and be like, at these points and these points, send the data to OpenTelemetry. Right? Like register some metrics, register what can I send to OpenTelemetry other than metrics? Traces? Logs? Logs
Kemal Akkoyunas Logs, metrics, and profiles. It's an Yeah. Alpha or better. Yeah. It's the fourth signal that you can send now.
Shay NehmadSo I so I can send all these things just like manually. Right? Write some code in my app that loads an OpenTelemetry client and sends it to whatever configured OpenTelemetry server or whatever. Yes. My second option is to do it with eBPF where I, like, I don't want to change my code. And now what you're saying is you can do it in compile time as well when I don't wanna change my code. Right? Yes. So there's clear there are clear, like, two options. One that requires work for me, which is the manual instrumentation, and then eBPF and compile time.
But they both don't require me to change the code, so why would I pick one over the other? Seems like they're both, like, sorta the same. They require me to change some variables or some things in my build process or whatever, but not change the code. So what's the difference between them?
Kemal AkkoyunYeah. That's a great question. So the the simplest answer is, one is for runtime, the other one is compile time, right? When you think about those advantages, when we say eBPF, it's a subsystem for Linux. So it only works for Linux, right? And you need to have certain events, event hooks, and you need to have an eBPF program that you attach to your kernel, and you run that against that. So the facilities that OpenTelemetry eBPF instrumentation, which is OBI use is Uprobes. Uprobes is the facilities where you can attach any piece of code to function entries or if you're using UreturnPro function exits. And using these things, you can collect, build your traces, build your spans, agreed metrics, and you collect this data. But these are like depend on the application binary interfaces.
Right? Things can change when you change your compile time, when you add another dependency whatnot. So you need to do a lot of like either preprocessing in compile time or runtime preprocessing. Right? So one thing to just like summarize, it's for Linux only. It is, let's say, like, little more brittle than the compile time approach because you need to you make a lot of assumptions on the runtime itself. Right? On the other end, compile time is basically this is just go code. Right? This is abstract syntax stream manipulation.
So this is nearly like source code level. And as you described, like the the manual code that you would have written, we are just we grab that. We just inject that for you. We just don't see that. It ended up in here, so the performance characteristics of that is the same, and it works everywhere that Go works. Either Mac OS, Windows, any architecture that you wanna run this on. So it is more like platform agnostic, let's say. But in the end, these are the same. Like these solve the same problems, right? They have similar characteristics of like you we call them instrumentations, like you let's say that you are using Redis, you are using Postgres, and you are using this third party libraries, and you want to have special labels for these and you need or like these have different, let's say middle layers or like functions that when they call to interact with these external systems and you would like to trace those, right?
You need to add support for these. You need to add specific code for to handle all these third party integrations, either in eBPF or in the AST for Go compile time. So similar things, similar projects, just the biggest difference is runtime versus compile time. If you want to get into nitty gritty details, someone like the eBPF programs needs to work in the kernel space, so if you're running this in a Kubernetes cluster, let's say that you need a daemon side and you need to assign certain capabilities to that daemon side, or if it's running on a host you need to give some privileged access, whatnot. Maybe you don't like that.
Maybe you don't want to give you have a different security posture and you don't want to give those capabilities to this third party agent, then this is a different story. Also, what the compile time instrumentation does, it's like the blast radius is just a single service, like the service that you instrument for Go. So if something goes wrong for whatever reason, because this is the code that you are running, and if you introduce, let's say, some performance overhead, it is just within that service. But if something goes wrong for those eBPF programs, then it you could take down to a whole host. Right?
But these are just the edge cases. Like this is not something you would face your day to day. Yeah. It's really hard to saturate. Yeah. Just to give some examples. Different trade offs in the end. Pros and cons, you should decide what to which one to use.
Shay NehmadI think I don't know. You tell me what you think, Jonathan. But always when I look at these things and it's just like one magic solution at like compile time or eBPF or whatever, and one where I can just add a few lines of code and I sorta know what's happening, I tend to, like, prefer manual instrumentation. Even though it's, like, more work and ostensibly from what you're explaining, not actually better. You might miss some cases or whatever. What do you think?
Kemal AkkoyunOne thing I can add, I think, sort interject, like, I forget to mention that e b p f one works for multiple languages. Right? It's that if you have something more than Go in your, like, mix, if you have heterogeneous services
Shay NehmadWhy would you have something more than Go? This is the company of a podcast. What are you talking about?
Kemal AkkoyunSome people some crazy people, they prefer to use multiple languages in their system. So then
Jonathan HallYou know, that's fine.
Shay NehmadThere. I don't have any problem with it. Some of my best friends
Kemal Akkoyunhere. So
Jonathan HallI I I have mixed feelings on that, honestly. Like, on the one hand, I like I like avoiding the magic, so I like doing it manually. On the other hand, I know that it's really a chore to manually add the telemetry you care about, so I don't know. Maybe the answer is just don't do do observability. Just just skip it all.
Shay NehmadJust don't write bugs. Just write perfect software.
Kemal AkkoyunThat's why we we are trying to make it this, like, happy for the customers. Right? Like, users for auto instrumentation, either eBPF or compile time instrumentation. Instrumentation. For example, like, you you might upgrade your third party Redis library and maybe something has changed. Now you need to change that code and it's breaking whatnot. Like, basically, we are providing all these integrations and you don't need to think about. Like, we apply all the best practices. We make sure that you are propagating your context correctly between your service boundaries, which is, again, if you are, like, talking to multiple services in a microservice environment. We basically also, like, packing with all these integrations in this auto instrumentation scenarios, like, provide the expertise for you.
Shay NehmadMhmm. And in general, you mentioned, you know, all this stuff. You do it as part of Datadog, but you did mention a lot about governance and stuff like that. It's happening as part of OpenTelemetry. So this is some open source, like, a governance thing that I don't fully understand just because the open source projects I've worked on are either very small scale, you know, just like a library I whipped up and put out, or something that's backed by a company. Like, you I worked at a company and they had that open source project, and it was theirs. OpenTelemetry, CNCF, these are, like, not really companies, but they are they do have money. They do have people working there. And you work at Datadog, which is not that company, but your project is part of that CNCF thing or OpenTelemetry thing. Do they give you money back?
Do you sponsor them? Like, how does it work? Do you are you part of that relationship, or do you just, like, write the code and let the lawyers deal with the other stuff?
Kemal AkkoyunI wish they have given me money. I've been doing this for years and nobody paid me anything. So I think this is best of the both worlds. Right? You have certain vendors for CNCF, say that for Kubernetes. Like, you have Google, Reddit, Microsoft, and Kubernetes is, like, a huge beast, and a lot of people needs to maintain that. And these vendor companies basically giving these developers, letting them work on these projects, or they sponsor the events, they directly contribute financial contributions to the CNCF and they make sure these projects are healthy, right? But in these equations, they don't pay any of the contributors whatnot. Usually companies let their engineers to work on these projects, right? Same for the OpenTelemetry.
The nice part of this, now you are like vendor neutral, right? You don't need to think that, okay, like, I put the Datadog SDK in my process and now I wanna use some other vendor. Like, what I'm going to do about this? Like, you don't need to think about that. OpenTelemetry is already neutral. You are putting the OpenTelemetry SDK in there and then you can easily switch to whatever vendor you would like to go. So this is more democratizing for the users and it's the benefit of the users are what isn't it for the vendors? It's like it's again, that flexibility sometimes help to that vendor, sometimes to other vendor, and they we we also would like to this community to thrive and these tools to be basically grow. Because nobody makes money out of instrumentation. Let's be, like, clear about it.
So services and people pay for the services. Nobody pays for the instrumentation or the data collection, but it's the hard engineering part and someone needs to do that. So so I'm, like, super happy that, like, OpenTelemetry and CNCF exists, and my company allow me to work on these open source projects.
Shay NehmadAnd I think there's a question that always comes up with open source projects. Right, Jonathan? Whenever we mention one, people are like, how do we get how do we join? How do we get started? Or whatever. At at least if there are more junior engineers listening to the show right now. So with other open source projects we present, you know, I think last week we had Peter Downs, right, with his PG test thing. So it's just like a guy, then like, oh, just talk to the the dude and get, like, get started. With CNCF Project, it's a it's a bit more, I think, of a process or a more official thing, but it actually should be easier for juniors to to get in on that. Right?
Kemal AkkoyunYes. It is easier. It could be intimidating, but it's easier. Everything is documented. For OpenTelemetry, there is an OpenTelemetry community report, for example. In that, you can find all the SIG instrumentation links, the calendars, like when the meetings happen, where are the Slack channels, what are the project that depends on that special interest groups. Same goes for Kubernetes and all other CNCF projects, right? These are all public knowledge. You can just go into the meeting, say, hi, I'm a new contributor. I'm looking for new issues, like, I would like to contribute back.
What is the best way to do that? For the easy stuff, usually there is GitHub issues tagged with, like, good first issue, whatnot. But for more complicated issues, maybe you should have a conversation. But in any case, like, being in the room, like, listening to the meetings, you learn a lot. There are a lot of, like, open source veterans out there, like, contributing to special interest groups. Basically, this is free education. As an extension to that, LFX Linux Software Foundation, which is the parent foundation of, like, a CNCF Cloud Native Foundation, there are mentorship programs. And Google Summer of Code, for example, is a joint effort with CNCF and Google, I guess. There's LFX mentorship programs. And if you are, like, fresh out of college or or, like, if you are already in college, you can apply for these programs.
And for three months, like, there are mentors or depends on the SIGs and the projects, they help you out to cut your teeth in the open source and all these projects. My In SIG context
Shay Nehmadin this context, you would be like a Google open source, summer of code, whatever, mentor. Right?
Kemal AkkoyunYes. Exactly. I did it several times. I did this for LFX several times. I'm already doing it, currently I'm doing it for this Go Compile Time instrumentation, we have an LFX mentee, and like for three months over the summer we are working on a big project together, like they learn everything to writing, RFCs request for, like, comments, project proposals, like how to engage with the community, how to contribute back, everything.
That's also another way to contribute back. Maybe you are not contributing already a SIG, maybe you are already, like, experienced, but you know how to do mentorship, for example, you can join these SIGs and, like like, basically donate your time to help others.
Shay NehmadDid you ever get a chance to mess around with these open source programs, Jonathan?
Jonathan HallNot not like that. No. No, I haven't.
Shay NehmadI had a Google Summer of Code, like, mentorship position when I was working on this thing called Infection Monkey. This was, like, 2019 or something like that. It was very interesting because, like, they pay the same for all the interns, so it sort of biases itself because it's not a a lot of money, it biases itself towards countries where the same amount is a lot of money. Mhmm. And people, like, would try really hard, and you would get, like, incredibly bright engineers.
So it's it's a pretty funny program in that regard. I don't know if they changed it because this was, like, seven years ago, but you get this, like, the best engineers from, like, rural India and then, like, very mediocre engineers from The United States. You know? Like, of course, I'd rather and they're all of them are, like, juniors just starting fresh out of college. Like, well, of I know who I'm gonna pick. I'm gonna pick the really bright, like, hardworking ones. And for them, it's like, you know, like a full salary. But, yeah, I used to do that. It was it was really it was a really good program. I didn't know that CNCF, like, these sorts of projects work for it as well.
Kemal AkkoyunI don't know about the recent changes of, like Google Summer of Code, but for LFX, for example, the stippens amount changed recently and it's like adjusted to the inflation of the country whatnot. So it's not like a flat amount anymore. Oh, So I guess it's more like democratic right now. But then we have a lot of we get a lot of applications from certain countries compared to others, I guess. It's more competitive and people see this as an opportunity to, like, be better in the job market, whatnot.
Shay NehmadYeah. No. It's a great it's a great opportunity. But even, I think, if you're not part of these programs, that, Jonathan, you can for sure say because you've done a ton of open source. Starting to hack on these projects, like, all on your own can be intimidating, but now if you have, like, a little bit of a community, like a Discord or a Slack or a whatever, it can make it a lot easier. You can just hop in and be like, hey. I'm thinking about getting started. Maybe try to use it, the whole thing. I don't know.
Jonathan HallSo have a question, a very different question. I probably should have asked this question at the beginning of the interview, but I didn't think of it till now, because I think this is the kind of question that I'm imagining that our audience falls into two general camps. Those who think observability is boring and probably aren't listening anymore, and those who think observability is really, really interesting or or it's it's an enabler. It's something you know, that's sort of the camp I'm in. Like, as a consumer of observability tools to enable me to be more effective.
But you're one level deeper than that. You're building these observability tools, and I'm curious to hear what excites you about that, because you clearly are passionate about this topic. What is it about that stuff that gets you excited? Because it's like three levels removed from what most people work on, if that makes sense. Right? Most people are building things that users see. You're you're not even building the stuff that the engineers see. You're you're building the stuff that feeds the stuff that engineers see. So what excites you about this?
Kemal AkkoyunYeah. We should have definitely talked about this, like, in earlier. For for example, I
Shay Nehmadam We might we might put this as the hook. Right? We're we're trying a new format where we put a hook. So it's like, some people think it's boring. That would be
Kemal Akkoyuna good hook. For example, like, as you described, I am one of the worst engineers to ask about, like, product questions about, like, Datadog because I rarely open open the product and poke around. Right? Like, I'm really deep and working with the run times and maybe the hard hardware to collect these data. Exactly those aspects are, like, challenging for me to understand how hardware works, find the performance bottlenecks, like, learn about this, like, as a threaded Linux APIs, learn about, like, how Go runtime works, how even these, like, data structures are implemented and integrated details on how we can actually improve the performance.
When you change something, all of a sudden, like, your program is slower, but like, what the heck? I just like added a single line, like what's going on? Like all these things are like super interesting to me. Since the college days, I was interested in these like details of the runtime, the system programming, low level bits, compilers, whatnot. I guess it's hard, but it's also fun. And like, this is a funnel, There are like so few engineers also working on this area, and like, it could also pay a lot of dividends, right? Like you are not doing the same web application that a lot of other people does, but then it's a double edged sword, right? But right now you have a competitive job market because you can maybe find a lot of employers to basically hire you because not everyone actually needs these skills. So yeah, it's a lot of trade offs, but like there's a more general question in there, like why do we need observability in the first place, right? This is my journey started from like the service level, right?
I was a back end engineer, then I was a reliability engineer, and it when you run the service and be on call, would like to know about like is the service healthy and what is going on? Are, like, the users are actually using that? What they are experiencing? Is are there an outage, whatnot? All these questions can only be answered by the observability understanding what you deployed in production and know what's going on. Right? I can't imagine that you are running a service without any observability. Sometimes people come and they tell that, oh, no. We don't have any metrics. Like, there are some logs or, like, what is distributed tracing?
That is really interesting to me. And, like, you cannot optimize what you can measure. That's but, again, it's the secondary effect, it's just the performance. But like, you need to know like what is going on if your service is up, if it's like fast enough, if your like database is saturated, whatnot. So you can only answer these questions by, like, having observability in your stack.
Jonathan HallRight. Yep. Very good. Well, I'm glad you're doing the work because You say you
Shay Nehmadsay a lot lot of not a lot a lot of people work on it, but there are a lot of people in the CNCF working on this stuff. Right?
Kemal AkkoyunYeah. But CNCF is bigger than the observability. Right? That that's like a lot of infrastructure work. Observability, yes, it's big. There are a lot of observability companies and vendors, but that's just a small part of what CNCF does, right? That's the whole cloud infrastructure and all these projects solving some other aspect of the cloud infrastructure problems, security, runtime, a lot of things. Because of all the agentic work nowadays, sandboxing is a huge topic, for example, Kubernetes and CNCF. And there are a lot of projects and a lot of movement around that, to how to serve models, how to basically provide sandboxes for these constantly running agents, whatnot. CNCF provides a solution for all of that.
Shay NehmadWell, we're coming up to close here, and we have a very classic, like, end of interview. First of all, what do you wanna plug? So you you mentioned all these projects, all these links, maybe about yourself, maybe your LinkedIn, maybe, I don't know, the best Turkish coffee in Germany. I don't know. What's the what do you wanna plug? We really appreciate your time, you know, coming to talk to all the people. So where should they find you and your work?
Kemal AkkoyunFirst of all, thanks for the opportunity. Yeah. I think best Turkish coffee is only in Turkey, so you need to visit there, unfortunately.
Shay NehmadI don't know if I specifically should go
Kemal Akkoyunvisit there, my friend. Yeah.
Shay NehmadThat Maybe in a few years.
Kemal AkkoyunMaybe in a few years, yes. Maybe it's not the best
Jonathan Hallfor you.
Shay NehmadGo for Con Turkey. I would be down to do Go for Con Turkey. That would be great.
Kemal AkkoyunThere was a Go for Con Turkey once. I think they did it twice twice in a row, but it's been a while. So what I would like to plug is we have these special interest groups in OpenTelemetry, right? OBI, eBPF based instrumentation, we have Go Compile Time instrumentation, we have profiling, it's also based on eBPF, and these are hard work but these projects need to sustain and for sustainability they always need more help. So if you're interested, try to engage with the community, help us.
Like this is mutually beneficial, we can teach you how to get into these nitty gritty details and you can contribute back. That's one thing. I guess the other one is like I'm trying to write more blog posts, so I have a blog. I will be writing more about Go, so I would appreciate if you can follow me.
Jonathan HallFor sure.
Kemal AkkoyunI guess you can provide the links.
Shay NehmadYeah. Yeah. We'll put the links in the in the show notes, of course. The special interest group is in Slack. Right? I'm seeing, like, a CNCF cloudnativedot, Slack, hotel go compile and instrumentation.
Kemal AkkoyunThere there are there is, like, Cloud native Slack, and in that Slack, each special interest group or or each CNCF project has a has multiple maybe Slack channels you can engage with. Yeah.
Shay NehmadI I found that link.
Kemal AkkoyunYeah. These are all, like, documented in GitHub OpenTelemetry slash community, and you can also, like, find all the information from that page. Cool.
Jonathan HallAlright. One last question prompted you before the show started, you were be prepared. We asked this to all of our guests lately, which is what is your favorite third party tool or library related to Go?
Kemal AkkoyunYeah. I worked on a lot of compile time stuff. That's also I mean, like, I like a lot of, like, static checkers and whatnot. I would like to pluck one of these static checkers. It's called Checklocks. It's kind of niche one. It's part of, like, gVisor project. It's part of their tooling. It is basically you are adding these annotations. Basically, you are adding your expectations around the locks and the fields of a struct, then this lock should be locked or not, and should this field incremented atomically or not.
And the static checker tool using the SSA step output of the Go build tool chain and checks if these assertions hold and like catch deadlocks, whatnot, like or the lock ordering problems. This is not something that you reach out for or day to day, I guess. But if you are using a lot of logs, if you have a comp like, complex code base, and you if you wanna provide some static assertions, this is an amazing tool.
Jonathan HallVery cool.
Shay NehmadCool. Wouldn't, like, the race detector do that? Oh, that's on test time. You're saying it's compiled time. Yes.
Kemal AkkoyunThis is static. It's
Jonathan Hallon Cool. The
Shay NehmadI didn't know about this library at all. What
Kemal Akkoyunthe hell? Check it out. This is yeah. This is very, like, unknown. I guess this is also, like, written by a core Go team member that used to work for gWiser. So it's maybe it should be part of GoVet. I don't know. Maybe they they probably they have a reason for this, but maybe it will someday.
Jonathan HallI will check this out. Interesting.
Shay NehmadGVisor. Cool.
Kemal AkkoyunYeah. It's part of gVisor under tools, check locks. They also have other, like, esoteric static checkers, whatnot, like, against Golink names, whatnot. Like, check check the the tools directory. It's it's it's kind of a gold mine.
Jonathan HallWow. There's, that's a cool Two or three dozen Yeah.
Shay NehmadHaven't I haven't tried it.
Jonathan HallYeah. Well, Kamal, thank you so much for coming on and helping us trace Shay's bicycle.
Shay NehmadUnfortunately, mystery didn't solve it off. No. Of course, I didn't find it. It's San Francisco. I went ahead and walked to the train station and bought a new one the next day.
Kemal AkkoyunIt sounds like Yarden as well.
Shay NehmadIf if we wanna if we really wanna close the book on that, what happened was I went I walked to the train station because I live in San Jose. So I I took the train back to San Jose, like, two hours late than I actually wanted to. And I did file a police report and everything. And because it was so late, I asked my my wife to pick me up from the train station, and she came with my daughter. She was already, like, ready for sleep, but she didn't fall asleep because she heard my bike was stolen. She got excited. And I was like, I was super pissed off. Like, I was so angry. And then I got in the car, and I I talked to them about it, and my my daughter was like, well, you know what? Did did someone steal your bike, dad?
And she's, like, five for context. Yeah. She's very small and cute. And I was like, yeah. And she was like, well, you know what? Maybe he needs it more than you. I was like
Kemal AkkoyunOh, so cute.
Shay NehmadYeah. I was like, okay. I'm not gonna explain the moral place of is it okay to steal if you need it? I was like, oh, that's such a cute thought, and I've been okay with that theft since then. Just imagined, like
Jonathan HallAlright.
Shay NehmadSomeone carrying water up a mountain with it or something.
Jonathan HallAlright. Well, I Alright. Yeah. Thanks a lot. I think that's a wrap.
Shay NehmadOh, do you wanna do the outro? You mentioned you listened to
Jonathan Hallthe show
Shay Nehmadfor a long time. You can do the program exited part.
Thank you to https://www.assembled.com/ for hosting Cup o' Go! Go Sets proposal: go test should run analyzers specified by the workspace Vercel Go Support Lightning Round: JSON v2 Phuong Le's YouTube about Go ★ Support...
🗺️ Accepted proposal: add maps.Grow 🇪🇪 IVXV online voting system Reddit thread 🐧 Protecting against CVE-2026-98432 🌩️ Lightning round agent-manager tinygo 0.44.0 released ★ Support this podcast on Patreon ★
Do the 2026 Go Developer Survey, and tell them Cup o' Go sent you! Go 1.27.3 and Go 1.26.10 are released due to eMudhra Intermediate CA Compromise Interview with Ainsley Clark AinsleyClark.com templ ★ Support this...
Proposals Accepted: support for acme profiles New: crypto/tls: Profiles Blog: Always Active: Faking Microsoft Teams Green Status in Go Interview with Daniela Petruzalek LinkedIn Blog: danicat.dev Ebitengine Golab, Nov...