Episode 66: The BTCD Bug That Brought Down LND Nodes
32 minAudio
In this episode of Bitcoin, Explained, hosts Aaron van Wirdum and Sjors Provoost discuss a recent bug in the btcd Bitcoin implementation that affected a large part of the Lightning network, as it disconnected lnd Lightning nodes from the Bitcoin blockchain. In the episode, Aaron and Sjors explain that a developer going by the name Burak on Twitter created a 998-of-999 multisig transaction by leveraging Taproot. Although this was a valid transaction, btcd and lnd nodes rejected it, and therefore rejected the block that included the transaction and all blocks that came after it. Specifically, Sjors explains, btcd rejected the transaction because it has a maximum limit on how much witness data a Segwit transaction can include. Although other Bitcoin implementations do enforce this limit on Segwit version 0 transactions, Segwit version 1 (that is, Taproot) transactions have no such limit. Still, it is a bit unclear why this bug in btcd seemingly also affected many lnd Lightning nodes which use Bitcoin Core rather than btcd to validate blocks. In the second half of the episode, Sjors speculates how the two may be connected. Finally, Aaron and Sjors explain how the Lightning Network is affected when Lightning nodes reject the Bitcoin blockchain. Chapters 00:00:00 Bitcoin Amsterdam and opening updates 00:01:27 The LND outage and Burak’s transaction 00:04:26 Taproot multisig and witness size limits 00:11:49 The btcd validation bug 00:15:14 How stalled blocks affect Lightning channels 00:17:11 Why LND nodes using Bitcoin Core were affected 00:22:42 Fork Monitor and a bug predating Taproot 00:24:43 User risks: channel timeouts and cheating 00:29:49 Routing failures and open questions
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Aaron van WirdumLive from Utrecht, this is Bitcoin.
Sjors ProvoostExplained.
Aaron van WirdumSjors, I have horrible news.
Sjors ProvoostOh my god, tell me.
Aaron van WirdumNo one, zero people, a great nil, a great total of nil people used the promo code Sjorsnado. From what I understand.
Sjors ProvoostThat's unsurprising.
Aaron van WirdumI don't know what that means. It either means we only have hardcore fans that already bought their ticket before we even gave out the code. Or it means no one can remember your name.
Sjors ProvoostOr it means they used the other promo code at the end of the show that was spliced in by the editor.
Aaron van WirdumOh, was there another one spliced in?
Sjors ProvoostYeah.
Aaron van WirdumYou listen to our shows. I sometimes do, but I didn't notice that. Oh, that wasn't easier. That one was easier to remember?
Sjors ProvoostProbably.
Aaron van WirdumOh, okay. Okay. That's a bit sad. Or just no one listens to our podcast. That's option three, I guess.
Sjors ProvoostNo, I think some people listen because some people at the conference talk to me as well that they listened and that they read the book.
Aaron van WirdumWhich book is this, Sjors? Do you want to shill it?
Sjors ProvoostYeah, it's a book I wrote called Bitcoin, a work in progress.
Aaron van WirdumVery well. Sjors, I have more horrible news. The Lightning Network crashed last week. I know we were both at Bitcoin Amsterdam, or at least maybe it was a couple of days before that. We were in full preparation of Bitcoin Amsterdam. Our mind was occupied, but the Lightning Network crashed. Did you hear about this?
Sjors ProvoostYes, today.
Aaron van WirdumIn preparation for this podcast.
Sjors ProvoostExactly.
Aaron van WirdumGreat. I'm gonna ask you all of the details so I know you've studied hard the past couple of hours to learn everything about this crash. And you're going to inform our dear listeners. Sjors, what happened?
Sjors ProvoostYeah, and I have to say, we were both surprised about how little has been written more thoroughly. Maybe we couldn't find it, but we were expecting to find very large blog posts explaining everything that went wrong and all the implications of that. But so far, we only found a couple of tweets and a few fixes, though I think it's more important than people think it is.
Aaron van WirdumYeah, either a blog post, some kind of post-mortem, or even sort of detailed... conversation somewhere or but yeah the only thing we could really find were some tweets so either these discussions are happening somewhere where we're not aware of it or maybe uh I don't know…
Sjors ProvoostOr maybe we can get them started.
Aaron van WirdumLet's see so Sjors, what happens? Uh
Sjors ProvoostBasically a lot of Lightning nodes just stopped working uh at a certain block height and at the same time there was somebody on twitter uttering the words oh my god I made this really cool transaction right and then you know he got a lot of replies saying oh my god you broke uh LND.
Aaron van WirdumCan we can we mention do you have the link in front of you who was this?
Sjors ProvoostYeah he's called Burak at @brqgoo with two o's
Aaron van WirdumOkay so what did he do what did Burak do what did you do Burak?
Sjors ProvoostI just did a 998 of 999 Tapscript multisig, and it only costs 490 in transaction fees.
Aaron van WirdumOkay, let's start there.
Sjors ProvoostThis is about as close as... Oops, I accidentally killed it. As we have ever gotten on Bitcoin, even though he didn't kill Bitcoin.
Aaron van WirdumWhat was that again? Because I remember the term. What happened there?
Sjors ProvoostOops, I accidentally killed it. It was about the destroyed smart contract that contained, I don't know, a couple hundred million dollars worth of reserves.
Aaron van WirdumThat was an Ethereum, right?
Sjors ProvoostYeah, and apparently somebody accidentally changed that smart contract into a state where nobody could touch it again.
Aaron van WirdumRight. So an Ethereum that happened.
Sjors ProvoostOne of those like, I wonder what this button does. I really hope this button doesn't do what I think it does. No, they must have prevented that from happening. Click. Oops.
Aaron van WirdumYeah. And then everyone who had their money, their ETH in a smart contract basically lost their ETH, right? That's what happened there. Yeah. ETH were frozen. Okay. So it's not that bad in this case, luckily, although, you know, it's still a problem. So we'll get into that. But first of all, so what did he actually do? You mentioned a 908.
Sjors Provoost998 out of 999. So it turns out with Bitcoin, you can do a multisig address, which means your multiple keys have to sign, and you can set your own threshold. So traditionally, you might say, I need two signatures out of these three public keys. But there was a limit to that based on various constraints, mostly that the way Bitcoin Core implemented, or the way Satoshi originally implemented the multisig check was extremely inefficient. And that means that if you did anything more than, say, 15 or 16, it would just
Sjors Provoostbecome too slow to validate any of the blocks.
Aaron van WirdumBut you could do it technically?
Sjors ProvoostNo, there was a limit on it.
Aaron van WirdumDo you know what the limit was?
Sjors ProvoostI think it was 15 or 16.
Aaron van WirdumOkay.
Sjors ProvoostSo a maximum of 15 signatures.
Aaron van WirdumSo then you could do a 14 out of 15 or 2 out of 15?
Sjors ProvoostSomething like that. And that might have even been like, yes, you can do 15, but you can only do 6 signatures out of that. Some constraint like that. It's not about the block size that it's using, but about the validation time going up quadratically, I think, instead of linearly.
Aaron van WirdumRight. Okay. So that's sort of the original multisig in Bitcoin, but that was improved upon. Is that right?
Sjors ProvoostExactly. It was improved upon in Taproot. By making it much simpler, basically, instead of having one opcode called the OP_CHECKMULTISIG, it has something called OP_CHECKKEY or something like that. I forgot the name. But basically, you just tell it to check one key to check the next key, etc. And then it just adds up the number. And so what he did was basically push that to the maximum and have these 998 keys in it.
Aaron van WirdumOkay, so that is the maximum? 999?
Sjors ProvoostI'm not sure.
Aaron van WirdumBut x out of 999 I'm not sure.
Sjors ProvoostOkay but it was big it was much bigger than anything we've previously seen before on the blockchain but it's perfectly valid under Taproot rules so when you're spending such a thing that means you need to put 998 signatures on the blockchain
Aaron van WirdumI guess there are two options right either: 999 people have to all agree on how to spend the coins, in which case it looks like a very regular one sig, because that's what Taproot does, right?
Sjors ProvoostIf you were to use MuSig, yes. So if you use MuSig to create a sort of one key...
Aaron van WirdumNo, that's not what I'm referring to, Sjors. With Taproot, you always have the option that if everyone in the contract agrees, then you should be able to just spend the coin, the UTXO as if it's a very regular transaction, right?
Sjors ProvoostYou have that option.
Aaron van WirdumSo you don't have to reveal what the actual conditions were under the hood.
Sjors ProvoostIdeally, yes. But that's not... You still have to do that yourself. You have to use software to create that joint signature. And that requires something like MuSig. So it's not something... Taproot doesn't solve it for you. Taproot just says you can have one signature, which is just one public key as far as the Bitcoin Core nodes are... As long as validation is concerned, it's just one public key and one signature. But you could have made it using a million keys or whatever you want. But in order to do that, you need something like MuSig or some other algorithm to make that combined signature.
Aaron van WirdumAt this point, I'm confused. Are we talking about the same thing? We might be. We might be. Because I thought MuSig was something else. But yeah, go on.
Sjors ProvoostSo in Taproot, you have two ways to spend coins. One is to provide a signature for basically the key path, as it's called. And that's just a public key. As far as the blockchain is concerned.
Aaron van WirdumYes.
Sjors ProvoostAnd the other way is to satisfy a script. Now, in order to make that key, that public key, you have to make that public key yourself. And you can use all sorts of ways to combine different public keys into one. One of them is MuSig. So MuSig is just a way to combine signatures, regardless of how you use it. You don't have to use it in Taproot.
Aaron van WirdumOkay I'll believe you then I thought these were two different things sorry to our listeners who already knew this that we had to go over this again but that's fine. Thanks for clarifying Sjors, yeah okay so but in this case the option that was used was not MuSig and by the way when I see MuSig I don't mean a song. We mean, how do you spell it?
Sjors ProvoostYeah, with a G sound, not with a C sound.
Aaron van WirdumAnd it stands for multi-sig, I guess, right? Just short, or what does it actually stand for?
Sjors ProvoostI have no idea.
Aaron van WirdumWe're getting off the track anyways. So he didn't go for the MuSig option. He went for revealing individual signatures.
Sjors ProvoostNow, of course, any of these individual signatures could be a 100,000... multisig, right? It could be. Through MuSig. We don't know that. The blockchain doesn't care. It just sees 998 signatures. Each signature is 64 bytes. So this was a very large transaction. This was, I think, a megabyte or something. Something bigly. And so this is no problem because you can make these Taproot scripts can be as big as you want to. Now, it turns out that with SegWit 0, the old version of
Sjors ProvoostSegWit, they could not be as long as you want to. There was a limit of, I think, 10,000 bytes. And this probably had something to do with the way multisig was implemented in all the versions of Bitcoin, but I'm not sure.
Aaron van WirdumOkay, hang on. So with SegWit, there was... What were the original SegWit, the first SegWit version?
Sjors ProvoostYeah, I mean, Taproot is technically SegWit 1, but to make it easier, yeah, with SegWit, which is SegWit 0.
Aaron van WirdumRight, yeah.
Sjors ProvoostThere was a limit.
Aaron van WirdumThere was a limit on how big transactions could be.
Sjors ProvoostHow big the witness could be.
Aaron van WirdumHow big the witness could be.
Sjors ProvoostOkay. So those signatures, you could only fit a certain number of signatures into the witness, namely 10,000 divided by 64, which is some number less than what he was doing today.
Aaron van WirdumRight. So this would only be revealed when you're spending the coins, right?
Sjors ProvoostYes.
Aaron van WirdumOkay.
Sjors ProvoostExactly. So...
Aaron van WirdumSo then technically, just to finish that thought, the reason I ask is so that would mean that it would be possible to receive coins that you can then never spend. I mean, you shouldn't. You determine under what conditions you receive coins.
Sjors ProvoostOf course, in SegWit, you couldn't make such a multisig anyway. So that wouldn't be an issue. But yeah, it's possible. If you could send somebody coins using a set of conditions that the only way to satisfy that is to put too much stuff in the transaction, namely more than 10 kilobytes.
Aaron van WirdumRight, yeah.
Sjors ProvoostAnd so maybe what you can do is, you know, one multisig can only do 16, but I think you can just do one multisig and then... do another multisig, and another multisig, and another multisig, and say all of these multisigs have to be satisfied. And then you can have as many transactions as you want, and then you can't satisfy it.
Aaron van WirdumYeah.
Sjors ProvoostYeah.
Aaron van WirdumAnyways, we keep being distracted by some side notes. Let's try to... This is my responsibility. We're going to stay on the track. So SegWit, zero. The original SegWit had a transaction size limit. Taproot or...
Aaron van WirdumOh, yeah. Sorry. Taproot does not. Taproot transactions do not have this limit, right?
Sjors ProvoostYeah, either it doesn't have a limit at all or it has a very large limit in the order of megabytes.
Aaron van WirdumRight, so now it's possible to create a big transaction like this one, like the 998 signatures out of 999.
Sjors ProvoostYeah, so you're making a very big input of a transaction. And now this is not a problem for Bitcoin Core nodes. It's also not a problem according to the rules of Taproot. However, btcd had an implementation mistake, so we should explain what btcd is.
Aaron van WirdumYes.
Sjors Provoostbtcd is another implementation of Bitcoin, essentially. There are a couple of implementations. There's Libbitcoin, there's Bitcoin Core, there is Bitcoin Rust, although I don't know if it's a full implementation. There is...
Aaron van WirdumBitcoin, I don't know.
Sjors ProvoostYeah, Bitcoin is in JavaScript, I think so, yeah. There's a bunch of them, and some of them implement Taproot, others don't. And basically, all these nodes implement consensus code from scratch. That's always a risk. Though I think Libbitcoin will actually use the Bitcoin part of the consensus code. But in any case, they've all implemented themselves, so they all have to be very careful to make sure that they copy everything in the spec or every bug that's in the Bitcoin Core source code. Otherwise, problems happen. And in this case, in btcd, was... incorrectly checking
Sjors Provoostthat this thing was less than 10 kilobyte, because that was a rule for the old version of SegWit and not for the new versions of SegWit. And so it would simply reject the whole block because it thought the transaction was invalid. So it would just freeze at that cool block.
Aaron van WirdumRight. So btcd was applying a rule that really only applied to SegWit. Let's keep it simple. It applied to SegWit, and it was applying it to Taproot. I'm simplifying, but that's basically what happened, right?
Sjors ProvoostAnd slightly under the hood, btcd has at least two places where it checks certain rules. One is when it receives a block over the wire, and we'll kind of get into later what receiving over the wire means. But basically when it receives a block from another peer, it does some preliminary checks on it. And then once those checks are passed, it goes to a full validation check. Now, that full validation will happen even if you feed the block manually, if you copy-paste it or something like that. But these preliminary checks should only
Sjors Provoosthappen when they receive the block over the network. And it was those checks that were enforcing this 10-kilobyte limit on all versions of SegWit, essentially, not just on SegWit 0. Because the way Bitcoin SegWit has been designed is there's multiple versions of SegWit. And if the version number is more than zero, then in the beginning, you would always approve the transaction. It would always be spendable. You would simply not check any rules, including rules about size limits.
Aaron van WirdumYeah. So I would propose from now on, let's just say SegWit and Taproot, even though technically it's SegWit zero and SegWit one, right?
Sjors ProvoostYeah, in this case, because it's about the witness. And every version of SegWit will have a witness, but the rules for that witness will change for every version of SegWit. So with regular SegWit, there was a limit of 10,000, but there was no limit defined for any other version of SegWit.
Aaron van WirdumYeah.
Sjors ProvoostSo it turns out this...
Aaron van WirdumOkay, so btcd, they did the same check kind of twice. And you mentioned that the first check happens when they receive it over the wire. And then you said, we'll get into that. So let's get into that. What do you mean by that?
Sjors ProvoostYeah, and presumably they check it again in the code that checks the consensus. And there they were probably doing it correctly. So in the code that really, you know, once they've received the block and put it in their own database, it might say, okay, is this block actually obeying SegWit? And then it would do it correctly. It would say, what's the version of SegWit? Okay, version number zero. Okay, that's the limit. Or if it was Taproot, it would check some other limit.
Aaron van WirdumOkay. btcd is an alternative Bitcoin implementation. However, we were talking about Lightning. So how does this tie in with Lightning then?
Sjors ProvoostYeah. So first I'll describe sort of an imaginary scenario because I don't think this actually exists in the wild or at least not very often. Let's say you're running LND, which is the Lightning implementation built by the same people who also built btcd.
Aaron van WirdumSo that's Lightning Labs, right?
Sjors ProvoostYeah. Now you can use Lightning LND using different node software. If you were using it with btcd, then this bug would very clearly occur because your btcd node is talking to other nodes in the network might be Bitcoin Core nodes it sees this block it receives it over the wire and it says this is wrong and then it stops syncing
Aaron van WirdumYou still haven't explained what over the wire is, by the way.
Sjors ProvoostYeah, just over the internet, something like that. So it basically rejects it that way. And then how would this impact Lightning? Your Lightning node wants to see all the new blocks in order to make sure that its channels are still okay, nobody's trying to cheat on it, etc. And so if your Lightning node is not seeing any new transactions anymore, it's going to get increasingly unhappy. At some point, it may even be talking to its peers on the Lightning Network, and its peers are saying, no, no, no, there are much newer blocks than what you're talking about. Why don't you follow the newest blocks? And maybe the peers will
Sjors Provoostsay, hey, the fees have gone up. Why don't you increase the fees on the penalty transactions? Because Lightning nodes will constantly renegotiate what the penalty transactions should be, etc. But you're noticing, no, no, no, the blockchain hasn't moved. Why are you talking about these higher fees? And at some point saying, fuck you. I'm going to disconnect from you. I'm going to close all the channels now. And so some people on the internet were reporting that they were seeing their channels getting closed by the other side because of some miscommunication because their node was basically stuck. And that can cascade, of course, if lots of nodes start closing
Sjors Provoostthe channels at the same time.
Aaron van WirdumOkay, yeah. So we were explaining or you were explaining that if LND uses btcd to read the blockchain, to get the blockchain, then the problem is obvious. And then you hinted at some problems.
Sjors ProvoostYeah.
Aaron van WirdumSo do all LND nodes use btcd? Is this a requirement for LND?
Sjors ProvoostI think they... They do under some circumstances, and they don't under others. So generally, most people, I think, run Bitcoin Core as the node. However, if you're running a mobile app, then you may want to not download all the blocks from your mobile app. You want to use these filters, BIP158, Neutrino filters that we talked about in some other episode. So those filters basically give you a quick summary of the block, and then you only request the blocks that you want. And that also, under the hood, is btcd. So what's possible on the mobile app
Sjors Provoostis that your btcd node on the mobile app will get stuck because of this. Because the mobile app will fetch a block, it will fetch the magic bad block, because you happen to have asked for that one, and then your mobile app will stop too. But that does not really explain what we saw people remark. Because people were complaining that, first of all, they were complaining that their node stopped, their Lightning node stopped. And I don't think a mobile app would necessarily stop, but maybe it would. But they were also complaining about channels getting closed. And mobile apps tend to be very patient in this stuff.
Sjors ProvoostLike you can have your, I don't know, your Phoenix wallet or Breez wallet or whatever. You can ignore it for weeks and it'll start annoying you, warn you, but it generally doesn't start closing channels on you. So that doesn't seem to be what's going on. When people have forced closes of their channels, I think they're actually running their own node. They might be using something like Umbrel or something. And at the same time, today I saw on Twitter, on October 10th, Umbrel said, hey guys, there's a new update. Please install this hotfix
Sjors Provoostbecause there's a problem with Lightning. So then my question is, why would Umbrel nodes have a problem with this? And this is where I would have loved to have read a good postmortem because now I have to guess because I don't know exactly how Umbrel works. But as far as I know, software like Umbrel uses Bitcoin Core to download all the blocks. So when people say over the wire... I would assume that the wire is Bitcoin Core receiving blocks from other nodes out there. And Bitcoin Core doesn't have this bug. So then the question is, why are we still
Sjors Provoostrunning into this bug, even though we're using Bitcoin Core to validate the blocks? You know, the Lightning software shouldn't be doing extra consensus checks. So this is where I think what's going on is that on your little node, you have Bitcoin Core running and you have LND running. But LND, under the hood, uses bits and pieces of btcd. It just uses some of the building blocks from btcd. And one of those building blocks...
Aaron van WirdumFor example, for Neutrino.
Sjors ProvoostYeah, for Neutrino. And then, again, there's two sort of paths that I would imagine could have gone wrong. One path is that the node, the LND software, is talking to your full node, not directly in the sense it is talking using these filters. So it might be asking your node, so it's on the same device, but your LND software is asking the node for the filters on the same device, and then given the filters, it asks the node for the blocks. And those blocks are still going over a wire. Even though the wire is inside your own computer, it's still a
Sjors Provoostwire from the point of view of the software. And so then it would also run into this bug where the block briefly goes on the wire from your Bitcoin Core node to this little piece of btcd, really. And so that's a problem because you don't want to have these extra valid... You don't want to be reliant on some other implementation than Bitcoin Core for your consensus checks because there could be other bugs in there. That's one possibility. The other possibility is that it's even different there. Maybe LND talks directly to Bitcoin Core and
Sjors Provoostjust asks for every block, which would make sense. I mean, Lightning nodes, if it's on the same machine anyway, it might as well just ask every block from Bitcoin Core, just in raw form. And that could also still be interpreted as if it was going over the wire. So maybe it was still running those same checks, even though there was nothing going over a wire. In any case, somehow some of the btcd consensus code is, you know, network code, but network is kind of, you know, important for consensus too. Somehow it's getting run, even though you're not running btcd as the node, which
Sjors Provoostis kind of worrying from an architectural point of view. Because again, in general, other implementations than Bitcoin Core can cause problems when they have even the slightest different interpretation of what the rules are.
Aaron van WirdumRight okay so we so to summarize that we know what the bug in btcd was what we're unsure about and we've tried to look for it is why it affected the Lightning network or to the extent that it did why it affects LND nodes.
Sjors ProvoostWhy why it affected nodes even though you'd think they were not running btcd at least not the full version of btcd right so that's that's an open question I think we've we've postulated some possibilities I would also observe…
Aaron van WirdumSomeone has the answer.
Sjors ProvoostLet us know.
Aaron van WirdumLet us know.
Sjors ProvoostWe'll mention maybe by the time this airs it's already been explained by somebody and they're like oh you guys are wrong you didn't read this post it's because you wrote it after we recorded it so one more thing is that I think some people framed this as a Taproot-related bug, but it really wasn't, because I work on this other project called Fork Monitor, which is by BitMEX Research, and Fork Monitor checks different nodes to see if they have any problems compared to each other. And because I was also busy with the conference, I completely ignored what Fork Monitor was doing, but it actually turned out that
Sjors ProvoostFork Monitor, the btcd node on Fork Monitor, was also stuck at this magical block, or one block before the magical block. But just nobody noticed it. Or maybe people did notice it and told me about it, and I ignored it anyway because I didn't know there was this drama going on.
Aaron van WirdumBusy partying in Amsterdam.
Sjors ProvoostExactly. That has to happen. But basically, this particular node and fork monitor has not been updated for a while because I was being extremely lazy. It was running a very old version. Actually, it was running a version that was from before they had Taproot support. So the node that stopped did not support Taproot, which means this bug is not related to Taproot at all. And in fact, somebody could have triggered this bug before Taproot was even created. Depending on when the bug was introduced, probably when SegWit was introduced, I'm guessing this check was written when SegWit was introduced. So this bug has been sitting
Sjors Provoostthere for like five years.
Aaron van WirdumBut then someone had to make a SegWit version one transaction before there was a SegWit version one.
Sjors ProvoostYes, but that's possible. And we've talked about this in the earlier episode about accidental hard forks or what is a hard fork or what isn't a hard fork. You can use features that don't exist yet in Bitcoin Core. And the thing is, you're giving away your money, but you can give away your money in a way that, you know, causes other nodes to crash or causes half the Lightning network to crash in this case. Or maybe not half the network. I don't know how big the explosion was. This is why I'd like to see a postmortem. Like how many nodes actually went down because of this?
Aaron van WirdumRight. Yes, me too. And so are we done with this part? Are we done with the speculation part?
Sjors ProvoostYep, I think so.
Aaron van WirdumOkay, so then I guess to sort of close this off. And you did allude to it already a little bit but what is actually the problem or the harm or how does this affect users so to say because I would assume that at least initially a Lightning node would actually continue to operate like normal even if it's not on the latest block because it doesn't require blockchain confirmations to make transactions or to forward payments, right?
Sjors ProvoostYeah, but a lot of it depends on the internal details. I think one of the tweets I read was saying that nodes couldn't open new channels or close them, which you think should be possible, right? Because you don't need to wait for a new block to close a channel. But there may be some implementation detail that causes those two things to depend on each other. Even though that's not intentional. So that's not really related, I guess. I mean, that's more of a matter of robustness. Like what does a node do when it stops syncing? What does a Lightning node do when the underlying node stops? That needs to be thought through a bit. Because one of the
Sjors Provoostthings that could go wrong is, let's say somebody's trying to cheat on you. And then it depends on all your settings. If you set things that it waits for a month and you can punish the cheater for a whole month, then as long as you figure out there's a problem within a couple of days and upgrade, you're fine. But let's say you've really tightened all the screws to maximize your fee revenue or whatever. And you've set it so that if you don't respond within 12 hours, let's say. Now, let's say 24 hours that's a bit of a better example let's say you don't respond
Sjors Provoostwithin 24 hours if somebody can cheat on you that means you meet you need to make sure your your Lightning node is not down for 24 hours
Aaron van WirdumYeah so to just to clarify this real quick anyone who knows a bit about Lightning knows this but maybe some of our listeners don't so the idea with Lightning is you're updating the balance between each other people have balances with each other and what's stopping people from out from broadcasting older channel states where they get more money than the latest channel state is that you can respond by basically taking all of the coins, right?
Sjors ProvoostI think so, at least a fair chunk of it. Yeah, so it's a deterrent.
Aaron van WirdumYes, it's a deterrent. However, in order to see if someone is broadcasting an old transaction or if someone is trying to steal from you, you kind of need to know what's going on on the blockchain. So if you can't see that, you can't respond and then someone can steal from you. That's a problem you're explaining here, right?
Sjors ProvoostExactly. Now, to add some nuance to that, it doesn't have to be in the blockchain. You can look at the mempool. We've explained the mempool in earlier episodes. Typically, your mempool is set to 300 megabytes. That's the default on Bitcoin Core.
Aaron van WirdumHang on, you say you can look at the mempool, but...
Sjors ProvoostYou can see if somebody's cheating by looking at the cheat transaction inside the mempool.
Aaron van WirdumYeah, but you can't see it with your eyes, right? So I would assume that software is written in a certain way. Do they look at the mempool?
Sjors ProvoostYeah, so that's another question I don't know the answer to. I think it would be wise if your Lightning software checks what's happening in the mempool. I don't know if that means it should immediately respond to it. Maybe that's overzealous, you know, because maybe it was a mistake and the other side corrects it with, you know, bumps it into a non-[unclear]. I don't know what the game theory is there. But if we run through this example, let's say, so let's say your node is no longer catching up with blocks because some exploit caused it to stop. It caused it to stop syncing new blocks. Then that means you will see transactions appear
Sjors Provoostin the mempool. And your mempool will get very big very quickly. If it's two megabytes per block within 150 blocks, you have a full mempool, and any new transaction that comes in is probably going to be ignored. It kind of overflows. Depends, maybe it's paying a high fee, maybe not. That means that's about a day, 150 blocks. So that means if your timeouts are set to about a day, you could be in a situation where somebody's cheating on you. That transaction is in other people's mempool, but not in your mempool because your mempool is already
Sjors Provoostoverflowed through the bug.
Aaron van WirdumYeah, and because there's no new blocks coming in from your perspective. So every transaction just stays in the mempool.
Sjors ProvoostSo unless you have a giant alarm system, you might not even know that this is going on and you've already been cheated. But let's say there is a big alarm system, then you have another problem because you still have this one-day deadline. So even though you know there's a problem, you can't act because the node is broken and you don't know how to fix your own node because you'd have to be able to write consensus code. So if there's not a patch within that 24 hours, you're still in trouble. So that's kind of one of the things I would be worried about here. Now, that's not a problem if you set the timeouts as something more reasonable. But, yeah, if you're reliant on somebody making a hot fix within 24
Sjors Provoosthours while you're on a plane, it's not good. So, you know, maybe this is completely exaggerated, but I'm just trying to illustrate why these things could be bad and why I think it's good to have a good post-mortem on thinking, in general, thinking through all these scenarios. And maybe the Lightning developers are thinking through all those scenarios.
Aaron van WirdumYeah, I was going to say, I mean, it might be a hard attack to pull off or a niche risk, but you've got to think about it, right?
Sjors ProvoostYeah, I think so.
Aaron van WirdumOkay. What about routing? Would routing be affected? I think routing was affected. I'm not sure how, but I think there were less routes available, I guess, because nodes were going down.
Sjors ProvoostIt could be the same reason. It could be that a node says, hey, I'm too far behind the real world, or I'm not seeing any new blocks. So generally, nodes have a heuristic. At least Bitcoin Core has a heuristic. I think that if you don't see a new block within two hours, some alarm bells go off internally. It tries to make more connections. Perhaps Lightning software has similar things that after a few hours of no update, it says, you know what? I'm not sure what's going on in the world. I'm going to stop forwarding payments now. So maybe it just refuses to forward any new payments until it's up to date again. Things like that could happen.
Aaron van WirdumRight.
Sjors ProvoostSo that would definitely cause some systemic effects where you would certainly find that all your routes are not working or a lot of them because some of them are running as a piece of software that's down.
Aaron van WirdumYep. Okay Sjors, is that it?
Sjors ProvoostI think so.
Aaron van WirdumYeah I think so. So clearly this was a little bit of a guessing episode or speculation informed speculation how you want to call it we're trying to figure out what went wrong and we weren't exactly sure and we've we weren't really finding the answer we're looking for so again if anyone knows more than us Let us know. Maybe we'll cover it in the next episode.
Sjors ProvoostWho knows?
Aaron van WirdumWe'll definitely mention it in the next episode. I don't know if we'll do a whole other episode about it. But please let us know if you know anything more. Sjors, is Lightning still reckless?
Sjors ProvoostIt depends on what you're doing. I think generally it's fine.
Aaron van WirdumSuch a nuanced answer. Just say yes. Reckless. Lightning is reckless. Everyone's reckless. That's what we want to hear.
Sjors ProvoostIt's not too bad. I mean, Bitcoin is still an experiment, right? So I guess that's all. Then thank you for listening to Bitcoin.
Aaron and Sjors discuss Bitcoin Core 32.1 and its updated security disclosure policy in this live conference recording, followed by audience questions. Links Bitcoin Core 32.1 release announcement and release notes...
Aaron and Sjors discuss the COLDCARD hack: the bug itself, how it could have happened, and some reflections on having had Coinkite as a sponsor. Note: If you’re using a COLDCARD and still haven’t moved your funds to a...
The COLDCARD hardware wallet has a critical vulnerability. If you are using the COLDCARD to store bitcoin, move them to a secure location as soon as you can. More information:...
Aaron and Sjors discuss what’s new in Bitcoin Core 31. Chapters 00:00:00 Episode 100 and Core 31 00:02:41 Cluster Mempool 00:10:45 Package Relay and Fee Selection 00:18:59 Private Broadcast over Tor 00:24:03 ASMAP and...