Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

ADSP: Algorithms + Data Structures = Programs

ADSP: Algorithms + Data Structures = Programs artwork

Episode 193: Kevlin Henneys with Kevlin Henney

Aug 2, 202423 minAudio

In this episode, Bryce chats with Kevlin Henney about Kevlin Henneys. Link to Episode 193 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Twitter ADSP: The Podcast Conor Hoekstra Bryce Adelstein Lelbach About the Guest Kevlin Henney is an independent consultant, speaker, writer and trainer. His software development interests are in programming, practice and people. He has been a columnist for various magazines and websites. He is the co-author of A Pattern Language for Distributed Computing and On Patterns and Pattern Languages, two volumes in the Pattern-Oriented Software Architecture series, and editor of 97 Things Every Programmer Should Know and co-editor of 97 Things Every Java Programmer Should Know. Show Notes Date Recorded: 2024-07-11 Date Released: 2024-08-02 HPX Intro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons - Attribution 3.0 Unported - CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Kevlin HenneyYou there will be failures. They will need to be reported. How are you gonna deal with that?

ConorWelcome to ADSP the podcast, episode 193, recorded on July 11th, 2024. My name is Connor, and today with my co-host Bryce, we chat with Kevlin Henney about Kevlin Henys.

BryceSo so one one thing switching topics a little bit, but one thing we haven't talked about is I I've realized the the original way that I know of Mr. Kevlin Henney is not from his talks or not from his books, but it's from the term Kevlin Henney, which I can tell from the look of his face is something that he is well known for, but perhaps not the thing that he would like to be remembered for. But but can you explain what a Kevlin Henny is?

Kevlin HenneyYeah, I think I'm gonna be remembered for it because you know it's uh yeah, my name is now associated with failure. Yeah. So um literally. So um back in the day, um, I started taking screenshots of you know, if an application started crashing, I'd take a screenshot of it. You know, it's just like, oh, okay, let's say, and I'd sometimes integrate it into a tool. Uh, and then mobile phones with cameras, that was a thing. Um, and it's just like, oh, okay, I can start taking you know, we we've got software everywhere in the world.

Kevlin HenneyBut you know, it's not all perfect, and we see a number of failures. And sometimes it's the software we write, sometimes it's the something else about the system. But uh, from the public's point of view, it's computer, it's failing. That's that's it. And I I took pictures of these, I would include them in my talks, I'd include my workshops. Um, sometimes just to make a point: guess what? Software runs the planet. Here's the thing that doesn't work. Sometimes, you know, one of my favorite ones um from about 15-20 years ago was a screenshot.

Kevlin HenneyI took a PowerPoint crashing. Um, and you you know, and and one of the things is a dialogue box saying, Oh, you know, there's a uh you've you've called a you know, basically memory error um uh in the particular DLL and pure virtual function called. And I said, right, what can I learn from this? Uh you know, I sort of said, I said, here is an application that offers you, we were talking about illusions. The whole point about software is that is that is the business of software, it's creating an illusion. Okay, whatever you're doing, you're creating an illusion.

Kevlin HenneyUm and uh and for as long as anything works, that illusion is maintained. Um, the minute something breaks, there's a crack, uh, and you start to see how the thing was built, how it was constructed. It's like a magic trick going wrong.

ConorYeah.

Kevlin HenneyUm, you know, it's like a stage set falling apart. You go, oh, that's how they do that. Oh, okay, I can see behind the scenes. And so here was PowerPoint. Okay, sure, I knew Microsoft used C, but suddenly I'm getting information about their DLLs. In fact, if I remember correctly, it was an Internet Explorer DLL that was failing. And I thought, here I am in PowerPoint, but an Internet Explorer DLL failing is causing PowerPoint to crash. That's interesting. Um, and it's written in C and there's a pure virtual function call. Now, how can you call a pure virtual function? Well, there's a couple of cases where that might happen.

Kevlin HenneyUm, you know, perhaps there's a path in the code that actually leads to calling a pure virtual function. Somebody mistakenly called a pure virtual function in a constructor or destructure of the class in which it was declared. Uh that one that happens every now and then. It's statically detectable, so perhaps Microsoft are not using static analysis on this or code reviews, uh, or alternatively, it's a memory trampling thing, something got zeroed, and you know, so therefore we've got a uh use after free issue. Uh you know, in other words, I'm speculating, I don't have the answer, but I'm already learning something about the way the system was constructed by the way that it is failing.

Kevlin HenneyAnd you can see this in a number of public screens. So I would use these kind of instructions kind of humorously, and sometimes people would then send me examples, photos they had taken. You know, hey, look at my printer. You know, here's the device, you know, here's the device driver version on my printer, this kind of stuff, because my printer's crashed and they've got all this information. And then we hit the era of social media, and people don't start emailing me, they are ating me, they are including me. And then I start basically resharing it. And that particularly visible on Twitter, I started resharing these screens because I thought this is kind of interesting.

Kevlin HenneyUm, I think it's amusing, but it's also educational, but it's also humbling. You know, you know, if we're software developers, we kind of make the world go round. Um and and so, therefore, our software is in everybody's face in one way or another, whatever level you are in the stack, there's there's there's an error message for you. And then people started calling it. It was around 2016, somebody started calling it, oh, I saw a Kevlin Henney screen at the airport, and that kind of that moniker stuck. Um, and so yeah, it's kind of ended up being a thing.

Kevlin HenneyAnd uh, yeah, I've actually, you know, it was it was uh it has got to that point that my you know my wife started a new job last year, and uh it's just like, oh yeah, what's your you know, what's your husband known for, or what is it? I said, Oh well, he's he's got he's got you know, failure screens are named after him in some circles. It's just like, oh okay, yeah, that's kind of different. But yeah, so that that is actually the way that works, and that is it is kind of interesting that that's become uh popular, but I think it is, you know, I don't mind I don't mind that association with my name.

Kevlin HenneyUm uh it's normally if it normally is it's a case of it's really just software developers and a few people who know software developers. Yeah, but also it's kind of an interesting thing, that idea that failure screens are a thing they uh we can learn from them. I also like to think of it as a public service, you know. I I I that the ones that I'm gonna say are a public service is when people are at um you know uh train stations and stuff like that, um, sometimes they will at in the rail company, the relevant rail company.

BryceYeah, yeah.

Kevlin HenneyAnd and normally they will respond, oh, we're really sorry about this. Which station was this? We'll try and get this fixed. So in other words, it's kind of like, yeah, it's like, yep, happy to help, happy to be associated with this and to have people report it.

BryceAnd so so my sort of succinct definition of what a Kevlin Henney screen is, is specifically it's a containment failure. It's it's not only a failure screen, but it's a failure screen that exposes a diagnostic that was intended for consumption by programmers or the people who designed the system, not by humans. Um like like uh I when I was checking in to the um to the Hilton hotel at the St. Louis committee meeting, Hilton was experiencing like a worldwide outage with their entire check-in system.

BryceAnd uh all of the hotels had to just it was just the honor system. They were just like, oh well, what what were your check-in dates? Like, you know, you know, what we'll just figure it out when the system comes back up. But I would I got this error on the app, which it's a little hard to see here, but I'll I'll read it out, where it said like minus 7,000 colon RCI check-in failed, open DB equals minus 7,000 application and database versions are out of sync. So that is a Kevlin Henny in my book. But if it had simply said uh we're sorry, we're experiencing a system you know error, we can't check you in right now, that I don't think I would call a Kevlin Henny, because that is, I think, the proper way for an error to be surfaced.

BryceAnd in particular, if I see an error message like this, where it tells me application and database versions are out of sync, what I'm getting from that is that this has failed in some way that has not been predicted by the people who developed the system. And also, if I'm seeing this error, it's likely that it's not been reported back to whoever runs that system. Because if it was, they would probably like log it and send it back to them and and then just tell me like, we're so sorry this failed.

BryceBut um it's it's like it's some some edge case where the uh the unexpected has happened.

Kevlin HenneyIt's that idea, it was in some way unexpected, and uh, and I'd agree with you because it's what what you're learning is typically for from my perspective, the ones that I, regardless of how other people might use it, I regard that, I definitely regard that as a Ketlin Hetty in the sense that that's for me is of it of interest because it reveals something about the way the system was built in a way that was clearly not intended, it's not part of the illusion. If something says, um, I'm sorry, we're experiencing problems right now, that's that's just and I'm sorry, we you know that this was anticipated, it's planned for, this is this is actually business as usual in that sense, because we it was anticipated.

Kevlin HenneyBut if in failing it surprises both the both the reader and a developer, you know, if you said this to the developer, it's just like, oh, that's surprising, we weren't expecting that. In other words, it surprises everybody and it shows us something about what was assumed and what was uh about how something was built. Yeah, you know, and this can be integer overflow errors, um, you know, every now and then you get a piece of software that sort of says that gives you um you know maximum the int max for a 32-bit number, and you sort of suddenly, well, yeah, that probably shouldn't be in the screen.

Kevlin HenneyUm uh and you suddenly see various bits and pieces. One of the most common ones people experience is NAN in web pages. You know, that was a popular one. Um, you know, I I got a few of those. Uh we just had a general election in this country, and uh there was a point where I think it was the BBC's coverage, um, they were talking about the various percentages because that's what you do on election night. And apparently, um, you know, nan out of nan seats had been uh won by. It's just like, okay, the parliamentary swing is nan.

Kevlin HenneyOkay, well that that's probably you know not what you wanted to say. And that kind of thing is is is the thing that is surprising. It it shows that something was not anticipated, it shows you that something is broken in a way that surprises all people concerned, but it also reveals how the thing was built. It reveals something internal.

BryceThe diagnostics that I like best from user-facing things are uh a sort of a happy middle ground between the two. And I think um Hulu and the Microsoft Blue Screen of Death perhaps do this best. So with in Hulu, if there's some sort of error, um Hulu will usually say, like, we're sorry, there was an error, like, you know, uh, we couldn't play this video, um, you know, please re- and it'll give you like instructions like please try logging out or something, but it also has uh an error code, uh, a code that says and it's not an error code that tells you, the user, anything.

BryceThat's not necessarily what I'm looking for. If I get an app that just tells me we're sorry, we're experiencing a system error, and then I go to report that to the people who created the app, because maybe maybe the issue's on my end, maybe it's something that hasn't been reported yet. If I want to be helpful and report it, if if all I have is sorry, it's a system error, I can send a screenshot, but that doesn't necessarily have the information they need. But if it's something like a Hulu error where they they've got some code, um one, I can give them the code, and two, I can Google the code.

BryceAnd in some cases, like I do learn some information about that because maybe some people online have figured out like what these codes mean. Um and I and I think that that's like the best model for things that have um uh like a user-facing interface is like like mask, don't don't expose programmer stuff up to civilians, but have some reference code or something so that if somebody reports it, you you can get the useful information out of it.

Kevlin HenneyYeah, it offers something distinct, it but it shows both, yeah. We something's gone wrong, but we we've already it it's in basically it shows a kind of a um a deliberate approach. There's an intent here. We have designed for this, we have accounted for the things that cannot be accounted for. It's part of our architecture. I think a lot of developers regard that as either not that, you know, it it sometimes it falls between the posts. Sometimes you have an organization where you know UI designers, yeah, UI designers kept separate, you know, work different separately from developers.

Kevlin HenneyAnd the UI designers go, oh, user experience is all about this. That's errors is not part of the user experience that we care about because that's not shiny and interesting. But actually, errors is part of the user experience. It is. And then developers are sitting there going, well, that's kind of error stuff. That's not really what I'm here to do. And maybe you know, it it fails in a way they're thinking from a programmer point of view, but not from a user perspective. So it falls between these two posts, and you often end up with either something, you know, something's inadequate either way, either you end up sort of anticipating but not offering useful feedback or information somebody can do something with.

Kevlin HenneyIn other words, you just say, we're sorry, there is a problem. Well, that's astoundingly vague. Um, I hate those ones where, you know, like you, it's sometimes you get it's just like an error has occurred. It's like, yes, but exactly. I can't Google for this, and I also can't report it meaningfully. Um I've I can't give them the thing that they would give me. You know, here is the little treasure item that you can give us, and we will unlock this. You know, it's it's just like this is the key for us. They don't give me that information, I can't do anything with it. There is an error, is is not false, but it's so vague that the truth is not useful.

Kevlin HenneyAnd then at the other end of the scale, scale is like, here's your stack trace. Um, you know, here here's this arbitrary message, and that is kind of like, what does that mean to me as a user? You've you know, I can't do anything with that either. Um, but it also shows me you didn't care or hadn't really thought that failure was an option.

BryceAnd that this all this this whole um topic also uh is clickable at the the level of you know libraries and like our interactions with other programmers and how our libraries report errors. Well, I used to start in my career, I worked on HPX, which was a distributed system, and um we had this distributed exception mechanism where you know if something failed in another node, it would, you know, that exception would get propagated, and eventually, like asynchronously, like you would receive the failure as the system got shut down.

BryceUm and even within a single node, if you had something that failed in another thread, like then it would be reported as an exception to while the other threads sh shut down. And um one problem we would have is um people would send us error reports where they would just be like, I got this exception, like I like I don't know what it means, I don't know where it came from. And it would be very hard for them to um uh on their own get a stack trace in a debugger um uh because the the the asynchronous event that that caused the failure has already occurred and is gone.

BryceAnd this is just the reporting of that thing. Um and so we built in to our exception catching uh throwing mechanism something that would um take a stack trace. Um and we also would capture the environment variables. And one of the reasons we did that is distributed system. We would want to know like, hmm, what's what's the like what's the specific setup of this node where the thing failed? Um and you see actually a lot of programming languages and a lot of libraries have their have a stack trace that'll print out by default in their error reporting.

BryceLike Python does this. Um I segfault to the Python interpreter doing some um wild stuff the other day, and I didn't get a stack trace. And I was like, like, what do I do now? Like I don't have I don't have a stack trace, I don't and I don't know how to go about you know getting getting uh GDB running on the Python interpreter in the way I want. And like LLVM does this as well. A lot of compilers will will dump out a stack trace of the compiler internals. Um and it and some compilers will even say, like, hey, we've put into a temp into a file in slash temp, we've put all the information that you need to file a bug report.

BryceJust like go copy this file and then like send it as a bug report. Um and uh I I think that like if you're developing a system, uh anything that's gonna be to play at scale, you really want to build in some some you want to think about this. So like how are people gonna report bugs to you? Uh because otherwise you're gonna get you're gonna get people opening issues, which is just like, I I I got you know, file not found exception. What does that mean?

Kevlin HenneyYeah, and um yeah, you're gonna end up with a lot of uh noise. I mean that's that's the thing. Is either you end up with information that's not useful because it's not really information, or you end up with just like a load of noise, and you've got to cut through that to try and find um something. And I and I think that that again it goes back to the uh this this kind of issue of of constraints and how sometimes things the shape of uh development changes over time. Um you will you know uh you there will be failures, they will need to be reported.

Kevlin HenneyUm, how are you gonna deal with that? And that is something that historically people have always said, going back to I I just highlighted it as the UX people versus the developers before, but actually I'm gonna say it's all role, so let's take a let's take a step further back. It's just like this is in the architecture. Sometimes a developer will say, Oh, I'm gonna solve it like this, and they think of it as a programming problem. It's actually not a pro it's architectural, it's it's about the you what's the user experience? What about the customer relations? So it's even the customer experience at that level. And so although a developer sometimes sees you know you're close to the code, and though so you see it as a localized problem.

Kevlin HenneyOh, a thing might bad might happen. Oh, I'll just you know, uh either we report this or I just log it or something simple, or I th I throw an exception, or I just rely on the magic of undefined behavior to figure out whatever this platform's gonna do. Um, and maybe that is going to be a uh uh a safold with a core dump, or maybe it's gonna be a Windows exception or whatever, and somebody else or you know, somebody else's problem. Okay, Douglas Adams, somebody having having an SCP field around something. It's somebody else's problem, makes it invisible. And each group thinks that the architect says, oh, that's just error handling, that's a programmer problem.

Kevlin HenneyUh the user, the user experience person goes, like, well, you know what, this isn't really what this is about. Even the product owner is going to start saying, Oh, well, that's not really what we're in this for. That's not the value of this product. So it's kind of everybody's problem, which is why sometimes it ends up being nobody's problem, but it becomes much more visible. We see these failures, they are frustrating, but how do people act on them? How um enabled do they feel? And then going back to where you say, you know, if I've got it, if I've got a searchable term, then maybe I find something online, and I as a user I can understand, oh, okay, uh, other people have this.

Kevlin HenneyIt's not just me. Or this is how I report it. Um, or they've thought out a process, just send us this file, and that's fine, you know, for the more developer-centric thing. But what you're doing is you're actually enabling people to say, oh, okay, you know, you're not ignoring me, it's nothing worse. It failed, and you're on your own kid. Oh, wow. You know, you're stuck in the rain, it's it's it's dark, it's miserable, and you can't get your work done because whatever has failed. Uh here, you're at least offering people like, okay, you had a problem, but don't worry, we've got this. You know, there's a thing that you can do. Um, and you know, we apologize for that, and so on.

Kevlin HenneySo, and that's not something that happens by an individual developer having an afterthought. Um, it's it's very much uh kind of uh a whole team and a whole um a whole architecture view rather than uh a bolt-on and a detail.

BryceYep. Yeah, you you have to you have to design for you have to design for failure and you have to design for for debugging. Um and and you know, if you do that, like I think I think the languages that that stack trace by default on error, um, it's far like it's far simpler for me to fix a problem because I'm gonna say 70, 80% of the time, just from the stack trace, just knowing like where in the code, like which function did the did the bad thing happen from uh is is enough to for me to know.

BryceAnd and in C I don't get that by default. And if I open up the if I open up the debugger in C, I by default can don't compile my code with debugging information on I compile it with O3. Um and I have a lot of problems that only show up at O3. And like by default, I open up the debugger and it's it doesn't give me the stack trace. Like I don't have a way to get that stack trace and figure out exactly where I am.

Kevlin HenneyI was thinking just you were talking about uh Hilton Honours, and I I've got a screenshot from a uh website. Uh I can't remember what I was doing, but it was Hilton, and it's a Hilton Honours one, and it gives me this perfect stack trace. It's a Java stack trace, and you can go through it and you can tell everything about the technology stack that they're using. Um leads us straight back into the discussion of security because you can kind of look at this, it's a it's like a manifest of like, oh, you're building it like that. Oh, okay. You know, it's just like um and if you and if you have any kind of like guesses like, oh okay, you're using this version, which I happen to know has a vulnerability, then you could you know, uh if you if you're a bad actor, then you can you can get in on that.

BryceOh yeah, seeing seeing like ver like language or stack version information in like a uh you know a publicly exposed error from anything that's on the internet is like a little bit scary. It says something about the security posture of the of the system. I wonder how many vulnerabilities have um have grown out of information that's been found through from a Kevlin Henney screen.

Kevlin HenneyOh, that's a really interesting thought, isn't it? Actually, yeah, yeah, yeah.

BryceBecause sometimes you don't even need to know anything about the stack, it's just like if you get enough information about what type of error has occurred, like and you can re-trigger the error again, then you might you might be able to figure out how to use that to do something something of evil.

Kevlin HenneyYeah, yeah, something malicious. Oh, I like that. I mean, I don't, of course I don't. No, yes, but no, no, no, I'm not I'm not on record for saying that. Dan, let's let's say no, but uh that yeah, that that's a point that it is uh and again you are you're you by showing the fracture lines you know in in this broken illusion, uh it allows somebody else to potentially it it's it's interesting from an academic perspective, it's interesting from a support perspective, but for somebody who is a bad actor, it's kind of like okay, I know something here that I can take advantage of.

Kevlin HenneyUm, and that's the the caution that needs to be done. And again, why it reveals again why this is an architectural question. It's got to be in the architecture. Um, you've got to be able to say, this is how we fail, this is how we fail gracefully, but with sufficient, you know, with sufficient grace and sufficient information, but without showing the whole, you know, without dropping our whole hand on the table and saying, here's what I'm playing. Um you've got, you know, there's got to be this balance, but it has to be intentional. It doesn't happen by accident.

ConorBe sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions. Thanks for listening. We hope you enjoyed and have a great day.

BryceLow quality, high quality, that is the tagline of our podcast.

ConorThat's not the tagline. Our tagline is chaos with sprinkles of information.

Open the transcript file

Hosts and Guests

Guests

  • Kevlin Henney Guest Inferred

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Episode 313: C++26 Reflection vs CRTP, AI, Kids & More with Zach Laine

    Nov 20, 202647 minAudio

    In this episode, Conor and Bryce chat with Zach Laine about C++26 Reflection vs CRTP, AI, kids & more. Link to Episode 313 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials ADSP:...

    • Apple Podcasts

    Credit the people on this episode

  2. Episode 312: The C++26 Contracts Fiasco

    Nov 13, 202652 minAudio

    In this episode, Conor and Bryce chat with Zach Laine about the C++26 contracts fiasco. Link to Episode 312 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials ADSP: The Podcast:...

    • Apple Podcasts

    Credit the people on this episode

  3. Episode 311: Career Changes, GPUs & More with Zach Laine

    Nov 6, 202628 minAudio

    In this episode, Conor and Bryce talk with Zach Laine about his career and the career changes he has made. Link to Episode 311 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials...

    • Apple Podcasts

    Credit the people on this episode

  4. Episode 310: Leaving NVIDIA for... ?

    Oct 30, 202624 minAudio

    In this episode, Conor and Bryce talk about Bryce's next chapter and where he is leaving NVIDIA for. Link to Episode 310 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials ADSP: The...

    • Apple Podcasts

    Credit the people on this episode

  5. Episode 309: Default Construction & CppCon 2026 with Jason Turner

    Oct 23, 202629 minAudio

    In this episode, Conor and Ben chat with Jason Turner about default construction in C++. Link to Episode 309 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials ADSP: The Podcast:...

    • Apple Podcasts

    Credit the people on this episode