Ben asks what happens when you curl google.com, and Matt peels back HTTP until the rabbit turns out to have been in the hat all along. Then a USB hub stages a dramatic intervention.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Matt GodboltHey, Ben.
Ben RadyHey, Matt.
Matt GodboltI can't believe we're laughing already. Ben, very, very, I don't know what the right word would be, but like, you've been auspiciously auspicious.
Ben RadyAuspiciously?
Matt GodboltThat's not even, though that's not the right word, but you took your microphone and kind of bent it out of the way so we could have a nice talk, and your headset microphone isn't in the way.
Ben RadyYeah, right.
Matt GodboltOh gosh, it's too early in the week to be recording a podcast, and yet here we are.
Ben RadyYou mean Friday? It's...
Matt GodboltYeah, I guess. I don't know, now everyone knows when we record these.
Ben RadyThere's not a lot of time left, my man.
Matt GodboltThat's true. Somehow we're in May as well, which, this podcast will go out in May as well, because it has to.
Ben RadyYeah. Yeah, right, right.
Matt GodboltOtherwise, we're in trouble. We've not got any in the bank.
Ben RadyOkay.
Matt GodboltYeah, so what are we talking about today, Ben?
Ben RadyI thought we could do at least one podcast, and maybe we'll do this again, on kind of like how computers actually work. And I think a way in which to cover the extremely broad topic of how computers actually work is to look at some of the maybe non-intuitive behaviors that they have.
Matt GodboltI love it. Oh.
Ben RadyThings that, you know, people kind of take for granted as sort of like an abstraction that you rely on. And, you know, peel back the covers a little bit and explore how it works. And maybe even, if we get to it, some like non-obvious ways in which it breaks. Right. And like when it doesn't work, what is actually happening? Because I think that sort of understanding the failure modes of these things is a great way to understand what's actually going on. So that's my general idea.
Matt GodboltFor certain, yeah. OK, so we've got like this sort of general idea of the theme of how computers really work, which obviously is very close to my heart as well. So this sounds brilliant. And then we were discussing some things before we started, and you know, shooting around ideas. And I sort of suggested the, there used to be like an old Google interview question, which was, what happens, you know, you type google.com into a browser, you hit enter, what happens? And you know, I always liked that because there's almost any number of ways that you can go, but how about we simplify that bit?
Matt GodboltWhat happens if I have curl rather than a whole web browser there, and I curl some simple URL, right?
Ben RadyRight.
Matt GodboltNot, well, not JavaScript or anything. Well, I guess it doesn't matter, but yeah.
Ben RadyI mean, could be JavaScript. A document.
Matt GodboltI guess so.
Ben RadyYou're fetching a document with curl.
Matt GodboltIt doesn't even matter at that point, because it'll just print it out in front of you.
Ben RadyThat's right.
Matt GodboltYeah. So, so yeah. When should we try something like that? What happens if we curled google.com? In fact, I did this only earlier today as I was playing around with sandboxes.
Ben RadyYeah. Oh, perfect.
Matt GodboltAnd I curled exactly HTTPS google.com. And I know exactly what it returns.
Ben RadyCan I, can I offer a guess before you tell me the answer?
Matt GodboltYou can offer away.
Ben RadyYes. Does it give you a redirect? Something in the HTTP 300 range would be my guess.
Matt GodboltIt does. It tells me the document has moved, and it tells me the document has moved to www.google.com, which was like, oh, really?
Ben RadyYes, okay.
Matt GodboltYeah, I guess because I was very lazy.
Ben RadyYeah, so how does it tell you that?
Matt GodboltWell, in this instance, the curl told me just a document. It printed out a document in front of me that said 300 or whatever it was, moved, document moved. And then it had a tiny little bit of HTML that came to me, because I didn't have, you know, dash D dash or whatever the thing is to print out all the headers and stuff like that. But no.
Ben RadyOkay. Okay. So you didn't see, so this was going to be my question. My understanding of this is, whenever you request a document over HTTP or HTTPS, you're getting two things back: you're getting the headers and you're getting the document, the document body.
Matt GodboltAha. Yeah.
Ben RadyRight. And so when it was redirecting you, all you were seeing there was the document body, which was some HTML. Is that correct?
Matt GodboltCorrect. A very small piece of HTML, which you would normally never see. You know, a browser would render that, really because the real meat of it was in the headers, right?
Ben RadyMm-hmm. Mm-hmm. It would just redirect you. Yeah. Yeah. Okay. So then when you do that, and I'm just kind of going through, I'm trying to like answer the interview question here a little bit.
Matt GodboltThat's right, yeah.
Ben RadySo when you, or maybe I'm interviewing you. I don't even know how this is supposed to work.
Matt GodboltI don't know which way around this is anymore, because I thought it was going to go the other way and now we seem to have flipped it. Yeah, so carry on.
Ben RadyI don't know who's, who's, I mean, you know, it's a collaboration.
Matt GodboltWhat even job is this?
Ben RadyWe're collaborating here. No one knows.
Matt GodboltWhat are the benefits anymore?
Ben RadyWe don't have jobs anymore. They're all taken by robots.
Matt GodboltOh, yeah, that's true.
Ben RadySo when you make this request, you're going to get back headers and you're going to get back the body. Like, how is the request delivered and how exactly is the body returned? Like, what protocol is it using to do that?
Matt GodboltRight, right. Well, I mean, I know that the particular command I was using is absolutely HTTP slash 1.1.
Ben RadyMm-hmm.
Matt GodboltAnd I mean, this is one of those cool things. I know that maybe this is showing our age a little bit, or certainly my age. I think I've got a few years on you. But at university in the late 90s, in the mid 90s, darn, you know, learning that you could Telnet to a port on a machine and just type stuff in. And get them to do things. So like, you know, the most happy day was to discover how to send emails from other people. I don't recommend that folks try this anymore. I don't think it works anymore, but you used to be able to Telnet to the mail port and give it just raw commands.
Ben RadyYeah.
Matt GodboltAnd it would be like, very trustingly, say, sure.
Ben RadyRight,
Matt Godboltthat you're god@universe.com, fine, send your emails please, right? So you kind of got this idea that everything was a text-based thing. And I remember this early web thing, and this was HTTP one, so you would just literally Telnet to port 80 of some machine and type get space slash and hit enter, and then you'd get the document back. No headers, no nothing. It was just, that was the whole thing.
Ben RadyRight. Right.
Matt GodboltBut nowadays, HTTP 1.1, we have to tell it more information. So you are going to connect to a TCP port on the machine, usually port 80. Sorry, 443 nowadays. Everything's HTTPS. So for the purposes of talking about the protocol, at least this level of the protocol, HTTP, let's ignore SSL for now.
Ben RadyYes. For HTTPS it's 443.
Matt GodboltLet's see if we can get to it in time.
Ben RadyRight.
Matt GodboltBut I don't know.
Ben RadyRight. Like Tom said in his most recent video.
Matt GodboltCorrect. And I'm so sorry, this is my family coming home in the background, and we thought we'd beat them, but no. So my dog is announcing their arrival. Much as you announce your arrival at the HTTP server by asking it...
Ben RadyYeah.
Matt GodboltWhoa, segue there.
Ben RadyMeta.
Matt GodboltSo get, it's still a verb, like get or put or post. And most of these things would be like get. Then the URL, rather, the document locator on that machine itself, and I forget the exact terminology for it, but that's the bit after the host name,
Ben RadyYeah.
Matt Godboltbut before the hash, if there's one in your URL. So like in Google, it will probably just be slash, forward slash, if it was your Google, because that's the only thing you've got.
Ben RadyMm-hmm.
Matt GodboltIf you're going to google.com, then you put a space and then you tell it that it's HTTP slash 1.1, which tells it, hey, I'm talking to you in the new way, the new new way.
Ben RadyMm-hmm. Mm-hmm.
Matt GodboltI mean, there's QUIC and there's HTTP 2 and there's all other things, but like, for simplicity, let's stick with this. Then you're expected to send a bunch of headers. Each header is a bunch of ASCII bytes followed by a colon, followed by a space, I think. Maybe the space is optional. And then some other stuff and the new line.
Ben RadyYeah.
Matt GodboltSo it's pretty human readable, as these things go. And the one thing that you will be most required, most required, I don't know if it can be more or less required, but...
Ben RadyThe utmost requirement.
Matt GodboltThe highest requirement is to put host colon and say, actually, what host (again, my dog in the background, you'll have to apologize), but what host I'm actually requesting from. Because this was like the big unlocker, I think, in like the early aughts, whenever this thing came out, which allowed you to have more than one website per IP address.
Ben RadyRight, right.
Matt GodboltBecause if you just do get slash, you're like, well, which website did you want? I don't know, right? It's whatever.
Ben RadyMm-hmm. Mm-hmm.
Matt GodboltBut that was the way to solve it. You know, maybe it would have made more sense to do get HTTP colon blah and just give it the whole URL and say, you work it out. But anyway.
Ben RadyMaybe.
Matt GodboltBut host colon google.com is what we would have said. And then, I mean, from my own personal experience of Telnetting to google.com to see how it works, I know that all you need to do to get it to work at all is to do get slash HTTP 1.1, host colon google.com return, and then a blank line that says, I'm done with the headers now.
Ben RadyMm-hmm.
Matt GodboltAnd then you get the document that says, no, you meant, surely you meant www.google.com. You're like, ah,
Ben RadyRight, right.
Matt Godboltbut nevertheless, connectivity is there. And then it will close the connection, I believe. So that's what I think would happen if I'm doing it manually myself with Netcat or Telnet or something of that nature. What am I missing? What's next?
Ben RadyNo, that's good. And I mean, I love doing that exercise when kind of explaining HTTP to people, because it takes the magic out of it. Now, you introduce a whole lot more magic when you add in like TLS and a bunch of WebSockets and other things like that.
Matt GodboltThere's a fantastic quote by one of our friends. It was to do with optimization, as it happens, but, you mentioned magic and I wanted to sort of bend it to this, which is that he said, you know, like, HTTP is like magic. And you're like, well, it isn't. It's just like a bunch of ASCII being sent over it.
Ben RadyIt's just text.
Matt GodboltYeah. And then...
Ben RadyIt's words.
Matt GodboltAnd that is like magic, because if you actually show people how it's done, it's kind of disappointing afterwards. You're like, oh, it's just, oh, I see. Yeah, there's just two of them. Like, oh, yeah.
Ben RadyThe rabbit was already in the hat.
Matt GodboltYeah.
Ben RadyIt was there the whole time.
Matt GodboltThat's right. It was there the whole time.
Ben RadyIt's not special.
Matt GodboltOh.
Ben RadyIt's not special. It's not a special hat.
Matt GodboltIt's not special.
Ben RadyIt's not a special rabbit. It's just, the rabbit was in there the whole time.
Matt GodboltIt was there the whole time. Exactly. But I love that feeling of, like, magic is that. That's what magic means, really.
Ben RadyYeah, yeah, yeah.
Matt GodboltIt's disappointing when you know how it's done, actually.
Ben RadyRight, right.
Matt GodboltBut anyway, not disappointing, because now you know how it's done. You can go, oh cool, this is something I can build on and I understand. I can debug. And so on.
Ben RadyMm-hmm.
Matt GodboltBut this obviously is not curl.
Ben RadyMm-hmm.
Matt GodboltThis is just me doing it manually. And you do it manually.
Ben RadyRight. So when curl is doing this, in the base case, when it just works as expected, it is, yes,
Matt GodboltBass. Not, not that kind of bass.
Ben RadyIt is opening a TCP connection to whatever IP address was resolved from the host name that you gave it.
Matt GodboltNo treble. Yes.
Ben RadyIt is sending those, that initial request with the HTTP verb, as it's sometimes called, the get, the post, the put, whatever it is. It's sending all the headers that it might send. And curl sends a lot of headers on your behalf, right? You don't, you can tell it to send specific headers, but you can also not. I just want this document, please. And it will add in whatever headers it thinks are appropriate, including a user agent.
Matt GodboltFor example, like the user agent, yeah, hey, I'm curl this version and stuff like that.
Ben RadyYep, yep.
Matt GodboltAnd what type of documents they'll accept, I think, is a list of things.
Ben RadyMm-hmm.
Matt GodboltAnd then potentially something to do with the connection, in terms of closing the connection or keeping the connection open for further requests, which, yeah.
Ben RadyMm-hmm. Okay. One, side note here, one of the interesting ways that you can detect which client is actually connecting to you... because the user agent is supposed to say, but obviously that's like kind of voluntary.
Matt GodboltYou can change that, yeah.
Ben RadyYou can put whatever you want in there. But one of the interesting things that you can do is you can look at the order of the headers, because the order of the headers is not specified in the protocol. So it can be whatever you want it to be.
Matt GodboltOh, but it's a sort of, it's a fingerprint for the client code that connected.
Ben RadyYes, certain clients tend to put the headers in certain orders, and depending on what client you're using, that can be an indication of what it actually is.
Matt GodboltOh my, that's sneaky.
Ben RadyYes.
Matt GodboltBen, you are a sneaky man.
Ben RadyRight.
Matt GodboltWhat is, is the sort of like fingerprinting based on the fact that, yeah, well, you know, curl always sends them in this order, and Chrome's fetcher does this and Firefox does this?
Ben RadyRight.
Matt GodboltSo it doesn't matter how you configure it. It's, you know, to say like, lie, user agent liar, is like, yeah, but we know you're Chrome really.
Ben RadyYeah, yeah. Similarly with like, you know, as you said, like spaces in between the headers and the values, again, optional, and you can do it either way.
Matt GodboltRight.
Ben RadyAnd that creates, also the casing on the values is another thing that can create sort of a fingerprint here. So curl is choosing to do it the way that curl does it. And even if you change the user agent to be like, oh, no, actually I'm Firefox, it's possible that somebody else might be able to figure that out. But the header order is not actually really, and I might be lying here, but I think it is specifically not important. Like, the order is completely unspecified.
Matt GodboltI think it is unspecified which sequence they're in. Yeah, I think so. But yeah, reader, dear reader, listener, sorry, not reader, gosh. Listen, but again, it's, it's been a week.
Ben RadyYeah.
Matt GodboltYou know, check what we're saying here, but this is just two old farts comparing notes on what they remember about how this works.
Ben RadyRight, right, right. And, well, and as you were saying before, oftentimes web servers are very tolerant of any kind of, you know, sort of garbage they may receive. And I think you said that you literally were just doing get slash on Google earlier today with no protocol.
Matt GodboltNo, this one had a, yeah, no, this one did have a get slash.
Ben RadyOkay.
Matt GodboltNo, this was still using curl. I didn't actually Telnet today.
Ben RadyOkay.
Matt GodboltBut like, I think nowadays, if you do get space slash, it will tell you to go away, because that's HTTP 0.9 or whatever, the one that didn't even have the version in it, pre-versioning.
Ben RadyYeah. Yeah, the unversioned version. Yeah, right.
Matt GodboltYeah, something like that.
Ben RadyExactly. So yeah, so those headers, you're sending those headers whether you know curl is doing it for you or not. You can obviously be explicit about it. But if you don't do anything, it'll do it for you. And then, depending on the type of verb you choose, the request itself may also have a body.
Matt GodboltOh, yes, that's a good point.
Ben RadyAnd this is particularly important for put and post and probably patch, but you can also do it with get.
Matt GodboltAnd patch, I think, is another one I'm thinking of. Oh.
Ben RadyYou can, I'm pretty sure that you can include a request body with get.
Matt GodboltI think one of the headers, one of the header fields, is content length. Is that right? Is that the thing that gates this?
Ben RadyOoh.
Matt GodboltAnd then you tell it how much you're going to be sending it or receiving. You know, certainly the response has a content length that says, this is how long it's going to be, or it can do.
Ben RadyYeah, yes.
Matt GodboltBut if you're going to do anything that requires you uploading, you need to tell it where the end of your thing is, your data.
Ben RadyI think you're right. I think you're right. Which is why, by the way, it is particularly difficult, if not impossible, to stream the body of a request.
Matt GodboltUh-huh.
Ben RadyLike, if you have some input stream that you're reading from and you're like, I don't know how many bytes are in this stream, I'm just, you know, reading it, by the time you're writing the body, you should have already written the header that says the length.
Matt GodboltYes.
Ben RadySo you have to consume the whole stream in order to figure out how many bytes you're going to send.
Matt GodboltWhich is, I mean, it's fine if you're serving static content off of a disk, where you're like streaming it from disk and you can go, how long is this file?
Ben RadySo you can put that in the header. Right.
Matt GodboltAnd it's immutable. It's not going to change on me. Okay. It's two terabytes. Cool. Content length, two terabytes. Here it comes. But yeah, if it's being streamed by a process for some reason, it's like outputting, you know, you're catting a file or whatever, something ungodly. Don't do this in the web.
Ben RadyOr another socket that you're reading from, right? Like you're trying to act as a proxy perhaps.
Matt GodboltRight, then you don't know ahead of time how long the thing is going to be to write it into the header, because you've already sent it. And I mean, anyone who's written web server software has almost certainly had some kind of exception thrown when they've tried to do something silly in the body handler, where it's like, no, you can't do that because the header has already been sent.
Ben RadyMm-hmm.
Matt GodboltLike, you know, this is, forget, I've definitely hit it with, oh, crikey, I can't think of the name. But in some cases where you're trying to do something, it's like, no, look, I've already sent the header. I can't now mutate this. I can't like turn it into a streaming request. I can't turn it into a different size request because we've committed. We're done now. You just have to send text now.
Ben RadyMm-hmm. Mm-hmm.
Matt GodboltSorry about that.
Ben RadyMm-hmm.
Matt GodboltYeah, there is multi-part though, which is a pretty crazy hack.
Ben RadyRight, right, multi-part uploads. If you've ever used like an object store that supports HTTP like Amazon S3 and other ones like this, you wind up sometimes doing multi-part uploads, because otherwise, if you have a lot of data to upload and something goes wrong on the 99th percentile byte of your five terabyte file, you've got to do it all over again, which is very inconvenient. I think there might also be, for S3, now that I say it out loud, there might also be another portion of the S3 protocol that allows you to do that.
Matt GodboltI was going to say, I think those are separable, because certainly S3 lets you configure, like, where do the slices of a multi-part upload go and how long do they live before I delete them away.
Ben RadyYeah. Yeah, yeah, yeah. How long they live. Right. Right.
Matt GodboltBut then there's multi-part, which is more like, you say, it's almost like a streaming protocol where you say, hey, I'm going to keep sending you data. Here is the delimiter, and you come up with some string that you hope to God doesn't appear in your actual data.
Ben RadyRight.
Matt GodboltAnd then you say, this is the thing that tells us that we've done this chunk, going to give you some more headers, I think.
Ben RadyRight.
Matt GodboltI think that's how that works. Now, I'm very much in dodgy territory here.
Ben RadyYep.
Matt GodboltI've looked at it with WebSockets before. There's some kind of thing that you have to do to get it to, during the handshake, at least WebSockets as they were in the 2010s. So 15 years ago.
Ben RadyYeah, yeah. Okay, so exploring other areas where you might do this, and we've explained this simple model of the world, we're sending text, we're receiving text, everything's fine.
Matt GodboltOh, God. Right. Right.
Ben RadyAnd you sit down with Telnet and you're like, I'm gonna do what Ben and Matt said and gonna request the thing.
Matt GodboltOh, we didn't really, we didn't see, so, yeah, the curl, for example, or if you Telnet directly, you will see the headers of the response, but curl was hiding them from me because I just did curl, the end.
Ben RadyYes.
Matt GodboltSo typically you'll see something along the line of, first of all, something that says 200 OK, the response thing that says, like, your get verb and whatever, you get a response document that starts with the...
Ben RadyYep, yeah, there you go, yep.
Matt GodboltWhat was the code? And you know, we all have seen the codes. The two XX ones are like, everything's fine, and they have slightly different flavors, but usually 200 OK is the one that you'll see.
Ben RadyRight.
Matt GodboltI think there's a, one of the two XX ones is like, OK but no data, is like, just, yes, this is fine, but there's nothing to come, which you see only very occasionally.
Ben RadyYeah. Yes.
Matt GodboltThree XX ones are like exceptional case type stuff. Is that right? What is the definition of three XXs?
Ben RadyI think the...
Matt GodboltCause they're like moved and stuff like that, isn't there?
Ben RadyYeah, like there's temporarily moved, permanently moved.
Matt GodboltYeah, you know, like...
Ben RadyI mean, it's most of the, I think all the redirects are in 300. There might be other things in 300.
Matt GodboltThat's the kind of edge casey thing where there might be. And then certainly 400 XXs are errors of sorts, and like, the world famous 404.
Ben RadyYeah, the world famous 404, which you probably know even if you're not a programmer. But there's the one that Twitter proposed and I think actually got introduced to the standard, 420 Enhance Your Calm, for rate limiting responses, which is a reference to the movie Demolition Man.
Matt GodboltYeah. Oh. Oh, no, I'd not seen that. Oh, that's cool. I know there's "I'm a teapot," which is one of the things, which is an error code that a coffee, a tea machine may return if it is asked to serve coffee or something like that.
Ben RadyYep. Yep, exactly.
Matt GodboltAnd then obviously if something breaks in the server itself and the server wants to let you know that something hideous went wrong internally, it'll give you a five-XX. What are the 1XX things? Are those like continuations, I think, for stuff like multi-part?
Ben RadyI think so.
Matt GodboltOh, no. See, I'm trying to do this without Googling because I have the loudest mechanical keyboard.
Ben RadyI know.
Matt GodboltSo everything you're hearing, dear listener, is coming off the top of our heads, and that, possibly, other parts of our anatomy, depending on how accurate they are.
Ben RadyThat's right.
Matt Godbolt[laughing] So anyway, and then after that, you get to see a sequence, sorry, of headers that look like the ones that you sent up.
Ben RadyRight. That's exactly right.
Matt GodboltSo it's, you know, something colon something else. And then there'll be two new lines, and then off you go with the data, the response.
Ben RadyMm-hmm.
Matt GodboltSo that's, it's pretty cool.
Ben RadyMm-hmm.
Matt GodboltIt's, yeah.
Ben RadyIt is. It is. And so this is perfect, actually.
Matt GodboltYeah.
Ben RadyThis is perfect. So coming back to what I was saying before about, like, okay, I'm going to do what Ben and Matt said, I'm going to sit down and I'm going to like type these things out in Telnet, and I'm going to get back some response. And when I do that, all I see is a bunch of gobbledygook. I don't see any text. I don't see any HTML.
Matt GodboltYes.
Ben RadyWhat is going on? One of the things that might be going on is that the server has compressed the response. And it will tell you that it has done that, and it will tell you which compression algorithm it has used to do that in the headers. So you will see a compression header, and then you will see...
Matt GodboltYes. So the header, the text-based headers are still there, just to be clear, right?
Ben RadyYes. Yes.
Matt GodboltAnd then there'll be a new line, and then you'll see garbage, which, although a well-configured server ought not to do that unless you said I would accept a compressed thing. But nevertheless, this is exactly the kind of thing, to your point at the beginning of this, is like, when something goes wrong, you know, I'm getting an endpoint that just assumes every client can support a compression algorithm, which I believe you're about to go and talk about.
Ben RadyThat's true. Right. Right, right.
Matt GodboltSo I don't want to keep, I don't know.
Ben RadyWell, yeah, no, and that's another thing that curl is, again, you know, peeling back the abstraction here, that's another thing that curl is doing for you, right? It's reading that header, it's applying the correct compression. I think you might need to tell it. I can't remember if, like, if you just, but can you disable it? Can you tell it not to do that?
Matt GodboltI don't remember either. I know I've had problems before because, and this is going to lead to one of my favorite observations, or rather a mutual friend's observation, you know, whether it's transparently decompressing because it says it's deflated, and so curl is then going to say, well, I'll undeflate it then, because that's what it says that I should do. Or whether, you know, that just means the fact that the compression, whether the compression is there or not, is now opaque to you. You don't know if it's happening or not, which is the kind of dichotomy of transparency: it means that now it's impossible to see whether it's actually happening or not, you know?
Ben RadyRight. Yes.
Matt GodboltSo our friend coined the term, which I introduced to somebody at work yesterday, transpaque, which is like that kind of combination of, like, well, it's like you're meant to not know it's there,
Ben RadyYeah, yeah, yeah. Right.
Matt Godboltbut if it is there, now you can't tell if it's there or not, and maybe it's broken, and now you're just being gaslit by your system.
Ben RadyRight.
Matt GodboltYou know, I think we had this with at least one of the things at work where a vendor was sending gzipped data, but whatever... accept header we were getting was then un-gzipping it for us, even though we'd asked, you know, like, get bob.gz or get, you know, this.gz. And then the library was like, oh, cool, I'll un-gzip that for you. And then it was storing it as, and we were writing it to disk as, you know, bob.gz. And you're like, why is everything so huge? Oh, wait, everything's called .gz because that's what we asked for, but something transparently, again, transpaquely...
Ben RadyTranspaquely.
Matt Godboltun-gzipped it for us, being helpful, thinking it was being helpful, and it wasn't, right? Anyway, so, but yes, it could be deflate compressed, I think. Or what are the other ones that are out there? Zstandard, I think, is that standardized? I don't know, but deflate's the one that, you know, everyone uses, which is to say gzip. Pretty much, to a first approximation, deflate is the algorithm that powers gzip, and then there's a little tiny bit of a header associated with an actual gzip file, which I think is slightly different, and a CRC.
Ben RadyOkay.
Matt GodboltSo that's definitely something that can go wrong.
Ben RadyYeah. Yeah. Do we want to try to get into TLS and compression and HTTPS? Is that a bridge too far for this conversation?
Matt GodboltWell... that might be a bridge too far. Certainly, I don't know enough about it. So what we've been talking about so far, let's just graze the topic and see if we feel comfortable with this.
Ben RadyYeah, right.
Matt GodboltAnd then our listener can roll their eyes at how wrong we are. But what we've been talking about is like the HTTP protocol. And in HTTPS, there's another layer, another sort of wrapping of the protocol where the, a certain amount of, well, a certain amount, some, encryption is applied, and authentication and certificate exchange, all that kind of stuff. But once you've peeled that back, you're back to HTTP again. So like, effectively, from the point of view of the web server itself and from the web client, you can almost, almost ignore the fact that the tunnel that connects them is...
Ben RadyMm-hmm.
Matt Godboltan encrypted tunnel, as opposed to just a regular Telnet. So obviously you can't Telnet to port 443 and type anything and expect it to happen, which is unfortunate. What you can expect, what will happen is, if you're very lucky, you might see a lot of garbage bytes being sent back to you, and then the connection is closed. That's probably what you're gonna see if you do anything at all, if you Telnet to port 443. Something that surprised me the other day is, I naively assumed that you connect to port 443, some kind of negotiation happens with a certificate and key exchange and the whole Diffie-Hellman thing to, like, you know, get a unique key on both sides, and then, and then, and only then, you start saying, hi, I'd like to get this, you know,
Ben RadyRight. I was going to say this exact same thing.
Matt GodboltBut, yeah, yeah. But it turns out, for all that, for the same reason that we added the host colon to the sort of, like, the message body rather than just relying on the Telnet, you know, the IP address that you're connecting to being unique for that website. We lose that if we encrypt it, which means that, like, you know, the web server you're talking to, if it's, you know, google.com is co-located with yahoo.com, unlikely. But whose certificate should the server give you when you connect to it?
Ben RadyRight.
Matt GodboltYou're like, ah, good question. I don't know. Which means that you, the client, need to tell it first.
Ben RadyYes.
Matt GodboltBut how do you encrypt that so that no one can tell which site you're actually browsing? And the surprising answer is, you don't.
Ben RadyYep.
Matt GodboltWhich was like, mind blown.
Ben RadyYep. Yep.
Matt GodboltOh.
Ben RadyNot as secure as you thought. All that information is being exposed to the interwebs.
Matt GodboltSo... does this mean that all these adverts that I keep fast-forwarding through about VPNs... And, dear listener, do not worry, this is not about to segue into an advert for one of those. This is, it's not a way of announcing that finally Surfshark or whatever is...
Ben RadyWe've taken sponsorship. Yeah.
Matt GodboltNo, no sponsorship.
Ben RadyUh huh.
Matt GodboltThis is just my fun.
Ben RadyRight.
Matt GodboltWe're not turning into anything else. But no, that is the one argument I can see for that, because I just assume, you know, like, fine, HTTPS everything, and it's, you know, no one need know what I'm doing. But it's, you know, you are actually, if you cracked out Wireshark and visited a website (which maybe we should do live sometime, we need to work out a way of screen recording this or something).
Ben RadyYeah. Yeah, yeah, yeah.
Matt GodboltBecause, yeah, that's, I think the first time you show somebody Wireshark and then you're browsing or doing anything trivial on the network is the first time the scales fall.
Ben RadyMm-hmm.
Matt GodboltCertainly was for me. Like, oh my gosh, look how much stuff's going on on my network or on my computer.
Ben RadyMm-hmm.
Matt GodboltOh, golly.
Ben RadyMm-hmm.
Matt GodboltBut anyway, yeah, so, popping back, you were going to say the same thing about this. Surprising, was it SNI or something, was the name of it?
Ben RadyYeah, you should be very careful about assuming that the data that is in your request, the URL, including the query parameters, the headers that you are either adding yourself or being added on your behalf by the tools that you're using, like curl, are actually secure, because they're almost certainly not, in many, many cases.
Matt GodboltHmm.
Ben RadySo when you make that request,
Matt GodboltI was thinking it was only the host that was the thing that was leaked, just purely so the correct certificate could be served back to you. But I may be wrong.
Ben RadySo I think that, so my understanding of this is that you're doing this sort of, like, escalation request, you're like, I'd like to escalate to this other protocol. And I would not be very confident in telling anybody that I knew for certain that, in all cases, it could only be these things, right?
Matt GodboltNo, that's fair.
Ben RadyIt's, yeah, it's one of those things where it's like, if it's important to you that that information is encrypted, then you should check, probably using Wireshark or something else like that.
Matt GodboltI've just fired it up. I couldn't help myself. Oh, I need to sudo make me a sandwich, of course, because, why, I shall. Yeah, okay, given that I have to escalate, I'm not going to do that now, otherwise, you know, you are going to hear my long password rattling out. But no, that's definitely an exercise for the reader. And I'm sure, hopefully people will comment and tell us where we're going wrong here. But again, this is from the top of our heads. So, yeah, no. So. Okay, I think that's probably the most we should do on the encryption, because it's clear that we've reached the limit of our knowledge.
Ben RadyYeah, yeah. We've hit the layer of abstraction that we can recall sitting here.
Matt GodboltI mean, there is definitely some things to do with certificates and things, and, you know, the little lock that you see.
Ben RadyYeah.
Matt GodboltAnd a little bit about that.
Ben RadyNot going to pretend to know nearly enough about that to explain it to the world.
Matt GodboltWell, I mean, there is like a chain of trust, isn't there? We can probably talk a little about that, where...
Ben RadyMm-hmm. Yeah, that's true. Certificate authorities.
Matt GodboltYeah, so your browser and curl or whatever, and your operating system, may provide a list of certificate authorities, which are like trusted third parties that have said, we will sign other people's certificates to say it's definitely them. Trust me, bro. And then, as long as you trust the person who signed the certificate, you can trust. Google.com is in fact Google.com because it's signed by VeriSign or whoever the heck actually signs Google. Google probably are a CA, aren't they? Probably, and that's fine.
Ben RadyYeah, probably at this point.
Matt GodboltAnd that's kind of how you know that you are actually looking to the correct site, the authentication part of HTTPS. And "know" has got air quotes here, because obviously it really does depend how much you trust the people who signed these certificates to say it really is who they say that it is.
Ben RadyMm-hmm.
Matt GodboltAnd if you look at the number of CAs that there are inside your browser, if you go and look at the list, and there's like a bewildering number of them, including some from, to me, things that would raise an eyebrow, and I'm like, oh, apparently we trust this company.
Ben RadyMm-hmm.
Matt GodboltI don't know who they are, and it doesn't look very trustworthy to me.
Ben RadyRight, right, right, right.
Matt GodboltSimilarly, in a corporate environment, you might discover, reasonably, your own company's root CA is in that list, because then you can sign, first of all, you can sign all of your internal applications with your own certificate that doesn't have to go out to the outside world, doesn't have to be signed by VeriSign. It means that you can go to, you know, benscoolapp.internal.com, and you can get a certificate for that, and your browser won't go, wait a second, this is just self-signed. That's no good. You know, it can be signed by, you know, benscoolcompany.com, which is then in there.
Ben RadyMm-hmm.
Matt GodboltBut it also allows potential for, like, security scanning software to decrypt, by man-in-the-middling you in your own, you know, on your own network, to be able to check that things aren't going outside, which is really important in, certainly in a large corporate environment. I don't know that it happens in any, like, the companies we're in, but.
Ben RadyNo, no, I mean, that's definitely a thing. I mean, again, like, you know, the theme of this being, like, you know, breaking people's expectations around abstractions and things like that.
Matt GodboltRight.
Ben RadyLike, it's probably been told to you in indirect ways by people at your company if they're doing this, but maybe not in as direct ways as, we're reading your personal email if you check your personal email at work, right?
Matt GodboltRight.
Ben RadyLike, the little security lock in the browser that you're running, right?
Matt GodboltIs not as strong as you think it is, because if you're using your company, I mean, obviously, if you're on your personal device on the guest Wi-Fi of your company or whatever, fine.
Ben RadyYeah. Sure. Right.
Matt GodboltAnd that's a very good reason to do that, right? But yes, if you are, most companies have at least a clause that says something like, there's no expectation of privacy, which is reasonable, right? You're at work, you're doing work things.
Ben RadyYeah.
Matt GodboltI mean, reasonable people could have disagreements about this.
Ben RadyYou're on a work computer doing work things.
Matt GodboltI get it. I get it. I get it. But like, that is the way in which they can potentially look at the encrypted data, is that they can say, well, any website you go and get, if you go to secure.com, I will say, yes, sure, I'm secure.com, and I can prove I am because I'm signed by this certificate, which, by the way, your company-given security system says I can sign anything I damn well like, and that's fine.
Ben RadyYeah. Yeah, right.
Matt GodboltAnd that's, yeah. So yeah, just knowing that that exists is all you need to do. And it's an exercise to the listener to go and see whether that's something that's happening to them or not. And I mean, it's a fairly standard thing. Anyway, popping the stack all the way back, right? So we said, what about curl, right? You're right, with curling this endpoint. What else? Is there anything we've missed? Let's just think about this.
Ben RadyWell, I got, so I got a good one.
Matt GodboltOK.
Ben RadySo let's say that, what you see, when you do this request, and maybe you're trying to drop down a level here and you're gonna use, like, the Telnet approach, is that you connect to the site, you type in your headers, you get back your response headers, and then you get some of the body, and then it stops.
Matt GodboltAh, yeah,
Ben RadyWhat happened? What is happening? Like, how do you even know that it's not done? What is the mechanism for that, if I recall correctly?
Matt GodboltYeah.
Ben RadyIt's like, two new lines is the fancy technology that they use to tell you that the body of the document is complete. It's that we're gonna do a new line and then we're gonna do another new line, and that's how you know that it's done.
Matt GodboltI think that's the hardcore thing.
Ben RadyBut let's say that you don't see those two new lines and it just stops.
Matt GodboltRight.
Ben RadyWhat is curl doing in that situation?
Matt GodboltThat's an excellent question. I don't actually know the answer to that. Yeah, I don't know, because, like, my instinct, if this was an interview question and you, Ben Rady, were asking me what's going on here, I would be like, oh, this sounds like the remote end has decided to keep the connection open. But it has, in fact, finished the document. And I don't know if it puts something at the end of that. I don't think it does. Maybe that's one of those 100, 100 continue is like one of those things, as it puts it in. I don't know. But, totally separately, and just pause there, I did, in fact, just Telnet to google.com and do get space slash enter.
Ben RadyI think you could get something like that, yeah.
Matt GodboltAnd it doesn't even tell me that I'm at the wrong address, because it doesn't know that I came to the wrong address. It doesn't try and redirect. It just gives me the whole... you know, minimized google.com thing pukes out. So it still works with even, you know, HTTP 0.9.
Ben RadyInteresting.
Matt GodboltOkay. What happened then, Ben? Tell me the answer. What was going on with my hung connection, or what might be happening?
Ben RadySo this could be, this could be a number of things, but one scenario in which you would see this is that there was, like, an error, like a programming error on the server side, where it like read a document,
Matt GodboltOh.
Ben Radyand encountered an error and for whatever reason didn't close the connection. Right, maybe it's waiting for, itself is waiting for data to show up that just isn't showing up.
Matt GodboltRight. It's not going to come, or it's deadlocked in itself trying to do something.
Ben RadyRight. It's, like, in the midst, stuck on some database query or streaming some data from another source or whatever it might be, and it's just stopped.
Matt GodboltYeah. Yeah.
Ben Radycurl has a number of different timeouts that you can configure. And part of the reason for those timeouts is situations like this, because you don't want some process to just be stuck forever because it made some request that is now stuck forever. It has a connection timeout, which is more of a TCP level thing. It is a TCP level thing. It's like, hey, I want to connect to this IP address and send it some text so I can get my documents. If it never establishes the TCP connection, or it might take some time to establish a TCP connection, there's a timeout that you can control for that. I believe there is also a response timeout, to say, like, if I start getting data and then I stop getting data, just whacked my microphone.
Matt GodboltThat's fine, I'm sure that will be edited away. It won't be edited away!
Ben RadyYeah. And you stop getting data, and then you continue to not get any data, how long should I wait before I just give up? And it's important to know, like, with all of these protocols, we use it often enough, you almost think of them as, like... You almost think of it as being, like, this synchronous thing, right? Like, almost like an atomic thing, where it's like, I'm going to do the request thing and we get the response, and it all works.
Matt GodboltRight, right, right, right.
Ben RadyAnd it is entirely possible that you get some, but not all, of the response. It is entirely possible that you get halfway through sending the request and then the connection just dies. It's entirely possible that you start sending the request and, depending on exactly what is happening on your network, it may not be clear how much of it was actually sent. Like, did you, I'm doing this, like, put request or delete request, right?
Matt GodboltOh.
Ben RadyLike, did you actually delete it on the other side?
Matt GodboltRight.
Ben RadyI'm not sure.
Matt GodboltThat is, yeah, the sort of, like, what's the name of it? The Byzantine generals problem or whatever, where you've got, like, the two, you're like, I sent a thing to you that says, did you delete it?
Ben RadyYeah.
Matt GodboltAnd you go, I got it, but I didn't hear your response. You're like, well, did you then? I don't know. I don't know, because I didn't, and like, there's any number of acks on both sides that you, and you can never be a hundred percent sure. And yeah.
Ben RadyExactly, exactly.
Matt GodboltYeah.
Ben RadySo that is definitely another place where this sort of abstraction can break down, where it's like, well, I did the delete request and it failed.
Matt GodboltI've...
Ben RadySo clearly it's not deleted.
Matt GodboltThere's at least five different timeouts.
Ben RadyWell,
Matt GodboltI've just been looking through the man page of curl, five different termouts, termouts, timeouts for different conditions.
Ben RadyTermouts.
Matt GodboltYou know, like, how often, how long do you wait before connection? As you said, overall timeout for the entire request, the first response, and then, you know, subsequent chunks of data, that kind of thing.
Ben RadyMm-hmm.
Matt GodboltAnd then some other things to do with IPv4 and IPv6, if you're trying to do both at the same time and get the first one to respond, some really clever things like that. But yeah, a lot of those things are sort of, yeah, as you say, we think of these things as atomic. We think about, you know, like, everyone talks about RESTful APIs and you're like, okay, it's like a magic RPC that I just do, and it happens or it doesn't happen and I'm done. And you're like, you could see an error on your side that says the connection never made, you know, the request never went through, but it did in fact make it through.
Matt GodboltIt was just the response didn't get to you and the TCP was torn down, but it did in fact take effect on the remote end, and now you're stuck. You didn't get back, you know, the response. So it's a tricky world out there, but yeah.
Ben RadyAbsolutely. Yeah.
Matt GodboltNo, there's many timeouts. Oh my gosh, yeah.
Ben RadyYou said there's five timeouts?
Matt GodboltI think so. I've got connect timeout, expect 100 timeout. "Maximum time in seconds you'll wait for curl for a 100 continue." Oh, good. I wasn't completely making that up.
Ben RadyThat's good.
Matt GodboltSee, you also, connect timeout. Happy eyeballs timeout. That's hilarious to me.
Ben RadyWhat? Happy eyeballs timeout?
Matt GodboltYeah, I'm going to read this to you because it's joyous. "Happy Eyeballs", with a capital H and a capital E, "is an algorithm that attempts to connect to both IPv4 and IPv6 addresses for dual-stack hosts, giving IPv6 a head start of the specified number of milliseconds." So it's like a way of, you know, preferring IPv6, but not necessarily...
Ben RadyOh man, that's good.
Matt GodboltSo yeah, we've got, see also, a max timeout, retry max time. So there's an overall max time, and then there's a retry maximum time, which is like the number of times it will, including, yeah, retry time is reset before the first... It's complicated, but I mean... And then we're going to keep talking.
Ben RadyYeah, and we're back.
Matt GodboltWhat happened there was a terrible break in the abstraction layer, because my USB hub crashed and put us into a very bad state. And we've just about recovered it. And, can't remember what on earth we were talking
Ben RadyNo idea. No idea at all. It's like it was a different dimension.
Matt Godboltabout. And if only, I mean, if only we had some kind of permanent record of what we'd said. But sadly, neither you nor, you said you could download the MP3s okay, and it was working. So at least we've got that on our side.
Ben RadyI did do that, yes.
Matt GodboltOkay, right. So we've got the previous part. So, dear listener, we apologize. Whatever we were talking about before has slipped our minds. But we didn't want to leave you in the lurch, or, more importantly, we didn't want to leave editing Matt with a job of, like, how do you finish an episode where...
Ben RadyRight.
Matt GodboltClearly something went wrong. But I think there's another abstraction layer we can talk about another time. It's like, how on earth do our devices talk to each other in a sort of day-to-day environment?
Ben RadyYeah, USB actually would be kind of a cool one.
Matt GodboltUSB, USB-C and DVI and digital video and all those kinds of magic.
Ben RadyMaybe a bit ambitious, but I think I like that one.
Matt GodboltI mean, it's a miracle that it works as well as it does, frankly.
Ben RadyYeah. Yeah. Yeah.
Matt GodboltBut yeah, we were looking at curl, and I think we were just about looking through all the timeouts. That's what I'd recall us talking about.
Ben RadyYeah. Oh, the happy eyeballs.
Matt GodboltHappy eyeballs for IPv6, IPv4, which is crazy. I saw someone put an IPv8 proposal out there, which I think is somewhat tongue-in-cheek, but also I read through it and I was like, this isn't as stupid as it looks.
Ben RadyWhat?
Matt GodboltMaybe this is real. You know, I can't tell.
Ben RadyAnd they skipped seven because we skipped five. That's how, yeah.
Matt GodboltI guess. I think the, it uses, like, a 64-bit IP address rather than the full crazy IPv6, I don't know, 256-bit, 128, whatever bit size that is, where it's like, hey, all this information about the host, you're like, I don't want to leak that outside my network. Thank you very much.
Ben RadyYeah, yeah, yeah.
Matt GodboltRight, so it was, and it's sort of backwards compatible with IPv4 because it's essentially, all existing IPv4s are 0.0.0.0.0.0, the rest. You're like, oh, the rest, you're like, oh, that makes sense.
Ben RadyInteresting.
Matt GodboltMaybe that's got legs. Anyway, that's really off topic, and I have, that's about as much as I know about it. My friend, I don't know if there's anything that we can recover from where we were going. So I think we should probably just call it at this point here.
Ben RadyNo, yes.
Matt GodboltAnd having introduced the world to transpacity, I feel like, you know, we've at least achieved something.
Ben RadyTranspacity?
Matt GodboltYeah.
Ben RadyTranspacity. Okay.
Matt GodboltI guess it would be, wouldn't it?
Ben RadyYes.
Matt GodboltYeah.
Ben RadyYes, it is.
Matt GodboltTranspacity. Opacity, transpacity.
Ben RadyThat's exactly, transpacity.
Matt GodboltWe'll have to ask our friend and see if maybe he'll join us on the podcast sometime to talk about it. But it's definitely a thing.
Ben RadyYeah, there you go.
Matt GodboltAll right, mate. Well, it's been a journey today.
Ben brings a 1976 letter from Bill Gates, and Matt brings the after-effects of last night's pub. The hat gets out of the rabbit, a squirrel is briefly involved, and nobody downloads a computer.
Ben's friend is locked out of their AWS account and Matt's AI has decided he's a hacker. The escape route runs through a cheap computer, a mesh network, and Neal Stephenson.
Matt records from the wrong continent and explains how he taught his computer to be a 1950s television. Ben guesses his way to the secret of the colo(u?)r burst and wins today's podcast.