0:18Aaron van WirdumLive from Utrecht, this is Bitcoin.
0:21Sjors ProvoostExplained.
0:22Aaron van WirdumHey, Sjors.
0:23Sjors ProvoostHey, Aaron. It's been a while.
0:25Aaron van WirdumYes. Welcome back on our... Hey, this is our 100th episode, Sjors.
0:30Sjors ProvoostThat is correct. Episode 100.
0:32Aaron van WirdumI did not think we would make it with our biannual podcast. But we're back.
0:37Sjors ProvoostWe've exponentially approached it. Exponentially slowing down approached it. But we made it. Well, at least we made it to the start.
0:45Aaron van WirdumYeah, we're sticking to our semi-official rule to only record in person, so that's why it took a while. I was not in the Netherlands, but now I'm back in beautiful sunny Utrecht for a beautiful, fresh Bitcoin Explained episode.
1:00Sjors ProvoostExactly, and we're going to pick a very fresh topic for a node that was released only three months ago.
1:06Aaron van WirdumYes, we do make an episode. I think we've made an episode for every new Bitcoin Core release since we started the podcast. So we're at least keeping that up. But yeah, so this is Bitcoin Core 31, which came out when?
1:21Sjors ProvoostIn April 19th.
1:23Aaron van WirdumFantastic.
1:24Sjors ProvoostAnd in fact, there is now a 31.1 release with some bug fixes.
1:28Aaron van WirdumRight, we're not even... Well, I guess these are just bug fixes. So when did this come out?
1:33Sjors ProvoostBut the bug fix is related to one of the features we're going to cover. So it's good that we had to wait for this bug fix.
1:39Aaron van WirdumOkay. Well, oh, is CoinKite still sponsoring us, Sjors?
1:45Sjors ProvoostI think so.
1:47Aaron van WirdumShould I?
1:47Sjors ProvoostWe're optimistically going to assume that they still are.
1:50Aaron van WirdumShould I endorse or advocate for their projects just in case?
1:55Sjors ProvoostWe should at least thank them.
1:56Aaron van WirdumDear listener, check out CoinKite with your favorite hardware wallet vendor with the COLDCARD, the OPENDIME, the SATSCARD, the TAPSIGNER. What else do they have, Sjors?
2:10Sjors ProvoostI have no idea.
2:12Aaron van WirdumDefinitely these things. CoinKite.com. Check it out I assume that's their website.
2:17Sjors ProvoostYes.
2:18Aaron van WirdumAll right. Or just Google CoinKite and see where it takes you.
2:22Sjors ProvoostI'm pretty sure they have every conceivable domain.
2:24Aaron van WirdumI would think so. Okay, Sjors. So Bitcoin Core 31 released a couple of months ago, and you made a special selection for our dear listeners of some of the highlights in the new Bitcoin Core release. Is that right?
2:40Sjors ProvoostA highly curated list.
2:41Aaron van WirdumYeah. Okay, so the first one you picked, which we'll discuss, is there's been... We wrote down Cluster mempool. Is cluster mempool... First question, is this a completely new thing? So the term cluster mempool, because I feel like I heard it before, is that just because we were discussing that it was coming? So my question is... Has cluster mempool been updated, or is cluster mempool a new thing in Bitcoin Core 31?
3:11Sjors ProvoostI believe that 31 is the first version that has cluster mempool in it. The concept has been around for a few years, so it's been mentioned on different podcasts, maybe even ours, but I'm not sure.
3:22Aaron van WirdumOkay, so this was a long time coming, and now, three months ago, in Bitcoin Core 31, there's now cluster mempool.
3:30Sjors ProvoostYes, and you will probably not notice a difference.
3:33Aaron van WirdumAs a regular user?
3:35Sjors ProvoostAs a user, yeah.
3:35Aaron van WirdumOkay, so what is cluster mempool? Just take it from the top.
3:40Sjors ProvoostWell, I don't completely understand it, so maybe I'm not the right person to ask.
3:43Aaron van WirdumThat's a great start.
3:44Sjors ProvoostBut it is just a better or sane...
3:48Aaron van WirdumThat's why people listen to our podcast, to get sort of half-baked, they think this is how it works type of explanations.
3:54Sjors ProvoostWell, it depends on the subject, but I mean, the bottom line is that it is a sane version of the mempool. The original mempool code was kind of awkward, difficult to reason about. One of the weirdest aspects of it was that... So the mempool is there to decide, or it should reflect what miners are going to mine basically first. And so ideally, if your mempool is full and you need to kick something out, then you're going to kick the most worthless transaction out and not the most valuable one.
4:24Aaron van WirdumAnd by that you mean the transaction that pays the lowest fee?
4:28Sjors ProvoostWell, not necessarily, because it's more complicated than that, right? Because it's not just about the fee that your transaction pays, but it's also how much your parents pay or how much your uncles... and nephews and et cetera pay.
4:38Aaron van WirdumAnd I guess that's the point.
4:40Sjors ProvoostAnd that was very complicated to reason about. So there were some strange scenarios where if the mempool was full, you would actually kick out the transaction that if you were a miner would be the first one you'd want to mine because of this intricate relationships between transactions.
4:56Aaron van WirdumOkay, but the simple example of this is child pays for parents, right? So there's a transaction that pays a very low fee, but from that transaction are coins spent again in a new transaction that pays a high fee. So as a miner... You want to mine both of those, including the very low fee transaction, because then you also get the high fee from the next transaction.
5:20Sjors ProvoostYeah, and as far as I know, that worked. But once you make the scenarios more complicated, things do not behave very well.
5:26Aaron van WirdumYeah, but so that's the point I'm making. That already worked, right? Bitcoin Core already had at least the basic child pays for parent logic. Also in the mempool, I think, right?
5:39Sjors ProvoostYeah, as far as I, my limited understanding is that it was able to reason basically from the ancestors down to the children or from the children up to the ancestors. Like you can go down the family graph up or down.
5:52Aaron van WirdumLike one step.
5:52Sjors ProvoostBut not sideways. And I think now it can take into account like, oh, you know, this parent and child are worthless, but they have a rich cousin that's paying a lot of fees. And so we should include that.
6:04Aaron van WirdumSo they're connected through, I don't know, like the grandfather in this analogy. You could just draw more.
6:10Sjors ProvoostYeah, so you have a parent, a child.
6:12Aaron van WirdumThe term cluster kind of says it. You can take a bigger cluster of transactions into account. when it comes to which transactions you want to keep in the mempool or kick out of the mempool.
6:26Sjors ProvoostYeah and where this becomes important is we have talked about this previously I think in equally vague terms things like pinning attacks where you're trying to escape from a Lightning channel so you broadcast a transaction and then somebody else makes a child of that transaction that makes it so that you will never get confirmed because the total fee is too low and so you want to bump that fee but you can't because you hit some sort of limit and there's a lot of these annoying complicated cases.
6:55Aaron van WirdumReal quick, I don't know if you know the answer to this question, but is it the case that the more sophisticated miners and mining pools were practically already doing this, but Bitcoin Core was just not keeping up and therefore mempools weren't really reflecting what was actually being mined? Or is this also really an innovation that miners are going to use?
7:16Sjors ProvoostThere's a couple of different things happening here. As far as people that have looked at what blocks look like, the quirks in the blocks suggest that they use Bitcoin Core to decide what goes in the block. However, they might have tweaked things like the minimum fee rate. So last summer we saw that... suddenly miners started including 0.1 sats per vbyte transactions. And that is a case of the miners are ahead of what Bitcoin Core was doing by default. And then
7:47Sjors ProvoostBitcoin Core tends to follow what the miners are doing because of this rule that we want to basically anticipate what's going to be mined. So I wouldn't say that they use some sort of sophisticated algorithm as far as I know. In fact, it's kind of the opposite. If the mempool is not as sophisticated as it can be, There is an incentive for somebody to say, hey, I can make better block templates than Bitcoin Core does. Why don't you send the transactions to me as sort of an accelerator? And then you get centralization pressure. So Bitcoin
8:19Sjors ProvoostCore kind of needs to be as good as possible while not wasting too many resources of the node. Because there's a computational time there, right? Every time a transaction comes in, you need to do some calculation to see if it's good or bad.
8:32Aaron van WirdumYeah, as good as possible from the... perspective of a miner that wants to maximize fee revenue, right?
8:40Sjors ProvoostYes.
8:42Aaron van WirdumWas this just something that was kind of always on Bitcoin Core's to -do list, but it's hard and it's complicated and that's why it took a while? Or is there a more specific reason it was implemented? I think you told me that it has something to do with Ark maybe also.
9:01Sjors ProvoostNo, it predates it. So the idea of cluster mempool predates Ark, I think by several years.
9:07Aaron van WirdumOkay.
9:08Sjors ProvoostI mean, over the past 10 years or so, I guess since Lightning especially, there's been a lot of discussion about pinning attacks that I just described before. And then you have things like RBF, which has a bunch of different rules. Like it has some rules in the BIP and it has some rules, actual rules that are not in the BIP. And, you know, it got quite convoluted. It wasn't very elegant. It was kind of a hot, sort of a hodgepodge of, well, if we make this rule and this exception, and then, you know, we have a carve out here. It was kind of a messy set of rules
9:41Sjors Provoostthat was just good enough, but then people kept finding ways to still pin you as a Lightning operator. So there were stakes. And this was kind of a redesign of it, which people came up with several years ago, thinking of how can we design the mempool logic from scratch. the sorting logic from scratch in a way that gets rid of all these weird exceptions too. So I think a couple of rules have been dropped in how RBFs are processed and exceptions have been removed. So it's more elegant and it seems to be more robust.
10:10Aaron van WirdumOkay. And this is a very, I mean, we're talking about the mempool as a kind of a very individual thing in the sense of you have your own mempool and, you know, every node has that. But this, I assume automatically, or I mean, you can answer it automatically. This also applies to what you share with your peers.
10:29Sjors ProvoostAs far as I know, you share with your peers the moment you add it into your own mempool.
10:33Aaron van WirdumRight, exactly.
10:34Sjors ProvoostSo that doesn't really change. I mean, if for some reason a transaction now goes into your mempool because of these different heuristics, give a slightly different result, then yeah, you would be sharing it with your peers.
10:45Aaron van WirdumYeah, but that was like a whole project, right? That Gloria in particular was working on, I think, with the package relay. So is this related to that? Is this kind of the same project in that sense?
10:56Sjors ProvoostSo my understanding is that this will make package relay... So what we discussed in a previous episode, the one parent, one child, special case of package relay. So it's a package of two. And this is quite useful because if you need to get out of a Lightning channel, that's usually just two transactions. One is the penalty transaction that you want to apply or the escape transaction. And the other is the fees that you want to add because when you make the channel, you agreed on either very low fees or in the case of ephemeral dust, no fees at all. And so the special case of two was possible
11:31Sjors Provoosteven without cluster mempool. But the hope is that with cluster mempool, you can reason about a much larger set of transactions. And then my optimistic take, and could be wrong, is that things like Ark would now become possible. Because with Ark, you have, say, 25 transactions to get out. And don't ask me how Arks work. You might have a package of 25 because you have a tree of transactions that are needed to escape. But right now, the nodes tend to process transactions one at a time. Whereas in such a scenario, maybe all the transactions don't pay enough
12:05Sjors Provoostfees except the very last one, which pays all the fees. And the node needs to understand that it needs to consider all 25 of them and that the last one is paying all the fees. And I think that's possible with package relay. And then hopefully cluster mempool makes it so that the node can reason about these big packages. It doesn't right now. It just has the better architecture under it.
12:28Aaron van WirdumSorry, that last sentence caught me off guard. It doesn't have it right now?
12:32Sjors ProvoostNo. We have cluster mempool, so it's a better architecture that might allow packages larger than two, but we don't do that yet.
12:40Aaron van WirdumOh, okay, but that gets back to how do you even get the transactions in the first place and if other pairs don't get the whole cluster to you?
12:49Sjors ProvoostWell, okay, so the cluster is a bunch of transactions that might have been given to you sequentially. So it builds up over time.
12:58Aaron van WirdumRight. But you need to keep them in order to know if the child or the parents are also valid, right? You need to not eject them, which is the point.
13:05Sjors ProvoostYeah. When you start out, you will accept anything that's over one sat per V-byte because your mempool is empty. You've got 300 megabytes.
13:11Aaron van WirdumYeah, or 0.1 sat per V-byte, I think.
13:13Sjors ProvoostYes, exactly. So you build up this mempool, including the clusters that are in it of related transactions. And then at some point, your mempool starts overflowing. And so you now have to decide which one are you going to throw out. And that is when you look at these clusters and you find the worst cluster and then each cluster is divided into chunks. So you take any given set of transactions that are related as a cluster and you kind of squash it in a line in a way that the best part of the cluster is at the top and the worst part of the cluster is at the bottom. And so you take the worst part of the cluster and you throw
13:48Sjors Provoostthat out. And that's a very fast calculation now.
13:52Aaron van WirdumOkay, but you throw that out in a way that the rest of the cluster is still valid.
13:56Sjors ProvoostYes, so the way you squash the cluster, like you flatten it, you flatten it in a way that one side of it doesn't need the other side of it. There's a certain order to it. So you want to order transactions by the most fees first, whenever possible. But if that's not possible because there's a child for parents, you might have the first transaction is a cheap parent, the second transaction is a... cheap child, and then the third transaction is a grandchild that pays a lot. And that's the first chunk. And then the next chunk
14:29Sjors Provoostmight be some other uncle transaction that pays a little bit less than the super rich child, etc. And that means that if you cut the tail off from the cluster, you're throwing away the least valuable part of it, but you're keeping it. And you can cut it at multiple places. But not at all places, right? Because like you said, some transactions depend on each other. And so it's sorted in a way that you can cut it in a number of places without breaking those dependencies.
14:58Aaron van WirdumRight.
14:58Sjors ProvoostAnd then you cut the worst part. And for mining, you do the opposite. So for mining, you look at the best cluster and the best part of the cluster.
15:07Aaron van WirdumMaybe I'm going a bit too much into the weeds right now, but in this hypothetical scenario that you describe, you're starting with an empty mempool, basically. But in this scenario that you describe, you're starting with an empty mempool, basically.
15:23Sjors ProvoostYes.
15:23Aaron van WirdumBut in reality, most of the time, you don't start with an empty mempool. Your mempool is kind of always full by default, right?
15:29Sjors ProvoostWell, not today, but let's say it is full, and a new transaction comes in. So then what you have to do, as far as I understand it, is you look at your mempool and see which is this relevant to any of the clusters I have, or is it a completely new cluster because it's unrelated to anything I've seen. If it is a new cluster, then you just see, okay, is this new cluster better than any of my existing clusters?
15:51Aaron van WirdumBut then you need the other transactions in the cluster as well, right? Otherwise, how can you tell?
15:56Sjors ProvoostSo a new transaction can be a cluster of one. It's not related to anything else in the mempool. It's basically just spending from something that's confirmed.
16:03Aaron van WirdumYes, but what if that is a very low fee-paying transaction and you don't see the rest of the cluster yet, then you would normally throw it out, even though maybe the rest of it...
16:10Sjors ProvoostYeah, so if you receive a new transaction...
16:12Aaron van WirdumSo for that, you need package relay, ultimately, right? Or some form of it that is so compatible with...
16:16Sjors ProvoostYeah, so if you see a new transaction that is not related to any of the clusters you're aware of, but it is part of a cluster, apparently... then yeah, you might just reject it. And you'd need something like package relay, especially big package relay to say, okay, here's somebody escaping from their Ark, for example, 25 transactions. I see the first one of that 25 transactions. I'm like, well, this is a new cluster, but it pays nothing. So I'm going to ignore it. And the second one, I'm also going to ignore it because it's paying from a parent that I, you know, spending from a parent that I don't know about. So that's where this
16:50Sjors Provoostcomes in. But in the case where it's just a single transaction and you need to evaluate it on its own, either it's better than any of your clusters or parts of the clusters, and then you know where to sort it and you put it in the mempool, or it's not and you ignore it. And that's a fast calculation, so you can do that when you get lots of new transactions every second.
17:09Aaron van WirdumOkay, so then I think this will be my last question on this topic. Ideally, we'd want package relay for clusters, right?
17:19Sjors ProvoostI think so, but some people might say no. I'm not sure.
17:24Aaron van WirdumBut in any case, that's not here yet. That's not here yet.
17:27Sjors ProvoostNo, the only package relay for clusters is a cluster of two.
17:30Aaron van WirdumOkay, right. Okay, well, that answers my question. That was my question. So right now, cluster mempool works for your mempool, and you make more complicated, complex, advanced. calculations for that, but it's not necessarily all related as a package yet.
17:45Sjors ProvoostExactly.
17:46Aaron van WirdumAnd maybe you say there's arguments against doing that also.
17:50Sjors ProvoostMaybe. I mean, if you look at the one child, one parent example in Lightning, the TRUC transactions or the ephemeral dust transactions, the question there is what is the maximum cost that an attacker can impose on you? So right now I think that's capped at like 10,000. sats in total, so that's like $5 now, the worst that the attacker can do to you. But if you say, okay, we're allowing a cluster package of size million transactions, then, I don't know, maybe the attacker can do more damage to you. But maybe that's not too bad. I don't know.
18:22Aaron van WirdumOkay, in any case, short summary, cluster mempool, your node, your mempool, will make more advanced calculations to determine what to keep and what to throw out. which benefits miners that would use this to create blocks and individual nodes because then their mempools better reflect the blocks that are actually mined, which benefits the estimation and stuff like that and relay. Correct?
18:48Sjors ProvoostYeah, I think some of this will or is being used for fee estimation too.
18:52Aaron van WirdumRight. Does that conclude our topic?
18:57Sjors ProvoostYeah, I think so. We can move on.
18:59Aaron van WirdumOkay. So the next upgrade in Bitcoin Core 31, there's now a function called private broadcast.
19:09Sjors ProvoostA startup option called private broadcast.
19:11Aaron van WirdumA startup option called private broadcast. Okay.
19:14Sjors ProvoostAnd it's been fixed in version 31.1. So if you want to use it, you should use 31 .1 to be on the safe side. What it does is, so right now, if you make a transaction and you send it, it just goes to a bunch of your peers. And that's not great if one of your peers happens to be Chainalysis and they can sort of figure out, oh, well, I saw it from you first and not from others, so maybe it's yours, right? So you're kind of leaking, potentially leaking the origin of your own transaction. In the past, there were proposals like Dandelion.
19:45Sjors ProvoostI don't know if we've ever covered that.
19:47Aaron van WirdumOh, that's years ago.
19:48Sjors ProvoostYes.
19:48Aaron van WirdumThat's like a 10-year-old proposal. I don't think we ever covered that.
19:52Sjors ProvoostOkay. Yeah, it's quite an old proposal. It's quite cool, but it basically said, okay, we're going to send a transaction, one hop, another hop, another hop, like just as a stem basically, and then it explodes at some random node in the distance.
20:04Aaron van WirdumRight.
20:05Sjors ProvoostThat turns out to be extremely complicated to implement for all sorts of reasons. But there's a sort of a poor man's dandelion, I guess, where it's this. So instead of connecting to your peers, when you have this new transaction, you open a new Tor connection.
20:23Aaron van WirdumJust good old Tor?
20:25Sjors ProvoostYes. Completely new Tor circuit, I think. And then you find a peer randomly on the internet. You connect to it. You say, hi, here's the transaction. Bye. And then you wait. And you see if any of your peers, normal peers, announced this transaction.
20:40Aaron van WirdumTo you, basically.
20:41Sjors ProvoostTo you, because then you know it kind of went through the network. If not, then you just keep trying another Tor peer, etc.
20:48Aaron van WirdumRight.
20:48Sjors ProvoostSo this is still a bit cumbersome to use, but it's there.
20:52Aaron van WirdumRight, right, yeah. Okay, so yeah, the problem, I mean, yeah, one of the ways to de-anonymize Bitcoin users is basically for, you know, the companies like Chainalysis. Am I pronouncing it?
21:07Sjors ProvoostThere are different people who pronounce it in different ways for different reasons, but it's Chainalysis.
21:11Aaron van WirdumBut there's different companies, right? They actually monitor the Bitcoin network itself to see where transactions originate, which is not foolproof, but at least it's supposedly used, right?
21:26Sjors ProvoostYeah, so the first peer that you, if you're a spy and you're connecting to lots of different nodes, and the first node you hear about for a given transaction is, you know, more likely to be the original one than the second node you hear it from.
21:38Aaron van WirdumYeah, exactly. Okay, so the, and then the solution now is to use, to just use, sounds simple enough. Why is this new? I would say, I mean, I didn't propose it 10 years, so it's easy to say now, but you would think someone could have just come up with this 10 years ago. Is there something difficult about this?
22:00Sjors ProvoostNo, I suspect the idea itself actually might already be at least five years old, if not more. It's possible that somebody proposed it, then came up with Dandelion. That sounds really cool. And then you try that and it doesn't work. It took another five years to get this one-shot Tor thing through. I think the idea is not super new. The downside, I think, is relying on Tor. Because it's kind of like this black box trust assumption. Like, okay, we hope Tor actually works and doesn't. Because if this doesn't work, let's say Tor was fake. Like, it doesn't actually protect
22:32Sjors Provoostyour privacy. Then you are creating a connection to the spy agency and telling them, this is my transaction.
22:39Aaron van WirdumAnd I want to keep it private.
22:40Sjors ProvoostBecause you're only using this connection to send your transactions. There's no decoys.
22:46Aaron van WirdumYeah, it's a dependency, right? Ideally, you would not want to...
22:51Sjors ProvoostAnd there's some practical issues. How do you configure your node to use Tor, etc.? Now, a lot of these node-in-the-box solutions will do that for you.
22:59Aaron van WirdumOkay, so this is new in Bitcoin Core 31, because it's not by default, right? You have to actually opt in somehow?
23:08Sjors ProvoostYeah, you have to turn on a setting, and then you have to do something special when you make a transaction to use this, I think.
23:14Aaron van WirdumOkay, and if I understand correctly, this is not implemented in the wallet right now. As a Bitcoin Core wallet user, you can't use this. You would have to... What do you actually have to do? You have to construct your own transaction and then...
23:27Sjors ProvoostYeah, I think you create the transaction as normal, but you don't send it, you don't broadcast it, you just copy it and then send it over, copy it to an RPC command.
23:35Aaron van WirdumOkay.
23:36Sjors ProvoostSo it's a little tedious, but...
23:37Aaron van WirdumIt's a little bit manual labor.
23:38Sjors ProvoostYeah, but I can imagine that you could add this to the wallet. It's not rocket science, it's just one step at a time.
23:46Aaron van WirdumYeah. Okay, so that's a new option, that's a new feature in Bitcoin Core 31. I think that basically covers it, right? It's kind of straightforward, or is there more to say about this?
23:56Sjors ProvoostYeah, no, that covers it.
23:57Aaron van WirdumOkay, great. More privacy is great. I like it. Okay. ASMAP.
24:04Sjors ProvoostOr ASMAP, depending on your mood. We had an episode about this, so just listen to it. Back then, we...
24:13Aaron van WirdumDid we do a whole episode on only ASMAP?
24:17Sjors ProvoostI think so. We had a whole bunch of episodes about eclipse attacks, and one of them was about ASMAP.
24:22Aaron van WirdumOkay, but to recoup that a little bit.
24:27Sjors ProvoostThe quick summary is that we don't like eclipse attacks.
24:30Aaron van WirdumWhat is an eclipse attack again?
24:31Sjors ProvoostIt's where you are connected to only one.
24:36Aaron van WirdumPretending to be several.
24:37Sjors ProvoostPretending to be several. And then they can feed you maybe an alternative blockchain or pretend there's no update or all sorts of shenanigans you don't want.
24:45Aaron van WirdumIf every pair you're connecting to is actually the same attacker, then they can give you blocks that are... double spending or whatever. It's not necessarily the largest chain you're seeing then.
24:57Sjors ProvoostAnd if you are a miner, then they can make you produce a block that's not actually on the most work chain, and then they can abuse you to double spend somebody else. So it's not just you as a normal node, but it's also you as a miner node where you do not want to be eclipsed.
25:16Aaron van WirdumYeah. Bottom line, you want to be connected to the actual network. You want to see actual blocks that the rest of the network sees. So an eclipse attack stops that. So this is a solution, or this is a measure against eclipse attacks.
25:31Sjors ProvoostYeah, so over the years, lots of different measures have been taken to prevent eclipse attacks. One of the measures that's been in there for quite long, I think from the beginning, is that when we decide who to connect to, we use this thing called buckets. We have a whole list of peers, or a whole list of... nodes in our database that we think are nodes, so just a bunch of IP addresses, essentially. And we say, okay, let's take all the IP addresses that start with a 1, and we make only one connection to
26:01Sjors Provoostthose, and then we look at all the IP addresses that start with a 5, and we only make one connection to those, and that's how we diversify the people that we connect to. And that's just based on what the IP addresses look like, which is not great because there's companies like Amazon that have lots of different IP addresses that start with different first digits, basically. So ASMAP is a project to actually look at these owners and group those and try to diversify across actual owners of the network.
26:36Sjors ProvoostSo there's a bunch of university research projects out there that are monitoring. I think it's basically the gossip between different internet providers. So AS is an autonomous system. It's a group of IP addresses. It could be your ISP. It could be Amazon. It could be Google. Like sort of one entity that controls a group of IP addresses. And this stuff is gossiped around the internet because it's needed in order for you to find a route from... A to B goes through all these autonomous systems. And so there are these giant databases of exactly
27:10Sjors Provoostwhich IP address belongs to which autonomous system. And what happens is every month or so or every two months, a group of volunteers, Bitcoin Core-related volunteers, come together and at the same time try to download these giant maps from those research institutions, a couple of different ones in the world. And then they do that at exactly the same time because this map changes every second or more often. And then if they all have the exact same view, some math is performed on it and it becomes like a one megabyte file that's added
27:42Sjors Provoostto Bitcoin Core now. And this one megabyte file just means that for given any IP address, tell me that it's like, what group does it belong to? It doesn't really give you the name. It just says, okay, this IP address is a different group than this other IP address. Because the only thing you care about is that you don't connect to the same group twice, the same autonomous system twice.
28:02Aaron van WirdumRight, okay, the very, I think it's a pretty hypothetical attack, right? But the potential attack here would be that, say, Amazon, whoever controls Amazon or whoever hacked into Amazon. they control a whole bunch of IP addresses now, and these IP addresses don't even look the same because Amazon is so big. They have so many IP addresses under their control. So to prevent an Eclipse attack from such an attacker, there's a more accurate
28:34Aaron van Wirdumway to map which IP addresses belong to which entity or behind which ISP or something like that.
28:41Sjors ProvoostYeah, exactly.
28:42Aaron van WirdumOkay, and from there, these buckets are then used to... for a node to say, okay, connect to a diverse range of it.
28:52Sjors ProvoostYeah, exactly.
28:52Aaron van WirdumBut we're not just connected to Amazon, we're connected to all kinds of different ISPs.
28:57Sjors ProvoostYes, and so in practice, this probably means that some obscure internet provider in Zambia will now get more connections because they're sort of picked as the not standard option and Amazon might get fewer connections.
29:11Aaron van WirdumRight, okay. And these maps, S-maps, these are actually... These are files. You actually download these on your Bitcoin Core node. These are now on your hard drive.
29:22Sjors ProvoostYeah, so what's already been available is, basically since we did that episode in 2022, is this ability to download this ASMAP file, put it in your node, and then enter it in your config, and it would use it. What's new now is that, by default, Bitcoin Core comes with one of those maps.
29:40Aaron van WirdumOkay, and uses them. Or is that actually?
29:43Sjors ProvoostNo, you still have to turn that on.
29:45Aaron van WirdumOkay.
29:45Sjors ProvoostBut now you just set asmap=1. And it works. Rather than downloading a file, putting it somewhere, etc. And then every time Bitcoin Core updates, the new version will have a new map. If you've turned this feature on, it will use the new map.
30:00Aaron van WirdumRight, yeah. It's not automatically updated or gossiped around. It's just whenever there's a new Bitcoin Core release, you get the newest ASMAPs.
30:08Sjors ProvoostYes. And you can still download your own or make your own, but that's kind of difficult. And the reason it took so long is because... We kind of don't like shipping a blob that is hard to reproduce, is hard to verify. So that's why there's this ritual of like multiple people in different places of the world try to make the same AS map file at the exact same time. And then if like three of them or so have the same result, then there's at least some auditability that this process was fair. And then you can look at the code and
30:39Sjors Provoostsee how it works. It's not ideal because the map of the internet, there's no official map of the internet. these basically maps that people are downloading exist for one second and then they change. Just like Google Maps, I guess, you know, every time there's a different street somewhere in the world, the whole map changes.
30:58Aaron van WirdumYeah, there's no official map, but this is publicly available data. And the Bitcoin Core developers themselves actually use that to turn it into maps. This is a Bitcoin Core-specific project.
31:09Sjors ProvoostYeah, I mean, I'm sure other projects could use it if they have the same need to avoid eclipse attacks.
31:13Aaron van WirdumRight. Okay. So that's also new in Bitcoin Core 31?
31:19Sjors ProvoostMm-hmm.
31:19Aaron van WirdumDid that cover it?
31:21Sjors ProvoostYeah, I think so.
31:22Aaron van WirdumIs there anything else to say about that?
31:23Sjors ProvoostNo.
31:24Aaron van WirdumOkay. Smaller risk of eclipse attacks. Also sounds good.
31:29Sjors ProvoostYeah.
31:30Aaron van WirdumI like privacy. I like smaller risks of eclipse attacks as well. All good so far.
31:35Sjors ProvoostGreat.
31:36Aaron van WirdumWhat was the first thing again? Oh, cluster mempool. I like more fees and more accurate mempools as well, Sjors. All good so far.
31:44Sjors ProvoostI'm glad you're happy.
31:45Aaron van WirdumVery. Last one. Oh, no, I was to ask. Wait, was this this topic or the last one? No, this was the last topic on the private broadcast. I do want to ask. So you mentioned there was a bug and it's fixed. What was the bug, actually?
32:00Sjors ProvoostYeah, if I remember, it's related to another episode we probably have done, which is about the V2 transport, the encrypted peer-to-peer network. What happens is if you try to connect to a node that has the encrypted... So if you try to connect to another node and you want to use a new encrypted communication protocol, if that doesn't work because they don't actually speak the encrypted protocol, you kind of disconnect and reconnect and then try the unencrypted protocol. Now, for Tor, it doesn't matter whether you use encrypted or unencrypted, if both sides are on Tor anyway, because Tor itself encrypts too,
32:34Sjors Provoostso it's going to double, but maybe you're connecting via Tor to somebody who's not on Tor through the exit nodes, right? So you need the encryption, but the... The problem was, I think that if the V2 connection didn't work, it would disconnect and reconnect, but the reconnection would not go over Tor.
32:53Aaron van WirdumRight.
32:54Sjors ProvoostSo that's very bad.
32:55Aaron van WirdumRight, so you're trying to get privacy, but then when it fails, you just give up on the privacy.
33:00Sjors ProvoostYeah, and this is exactly what I said. One of the reasons why these features like this are scary is because you are trusting the black box Tor, and in this case, if you make a mistake, because you're only using this connection, for this thing that you say it's yours, it's extra dangerous when something goes wrong, like this.
33:19Aaron van WirdumRight, and so it did go wrong at the initial release.
33:22Sjors ProvoostYeah, I'm sure it was somewhat of an edge case, but it still makes it clear that you want to be careful with features like this, because they can be a foot gun if they go wrong.
33:30Aaron van WirdumRight, and in any case, it was fixed in 31.1.
33:33Sjors ProvoostYeah, and that's why we waited with this episode, right?
33:36Aaron van WirdumExactly. Okay, last topic. I think that's the last topic for us, right?
33:41Sjors ProvoostAlready?
33:41Aaron van WirdumYeah. I wrote down IPC interface, and I already forgot what it... IPC interface, Sjors? Tell me.
33:51Sjors ProvoostOh, yeah, that's the mining-related stuff.
33:53Aaron van WirdumOh, right, yeah.
33:53Sjors ProvoostStratum V2-related.
33:55Aaron van WirdumRight. So this is... Okay, just take it. Go for it.
33:59Sjors ProvoostYeah, I don't know how much we've covered that Stratum V2 in earlier discussions, but...
34:03Aaron van WirdumI think we had a whole episode on that also at some point. But that's maybe, like... three years ago by now. I think that's been a while.
34:10Sjors ProvoostIn any case, the Stratum V2 system allows you to make your own block templates, among other things.
34:17Aaron van WirdumIf you're a miner, yeah.
34:18Sjors ProvoostExactly. Well, you can make them even if you're not mining.
34:22Aaron van WirdumIt's kind of useless.
34:25Sjors ProvoostSo the initial idea was to have Bitcoin Core actually speak the Stratum V2 protocol and provide block templates. but it was a bit controversial to add even more complexity to Bitcoin Core because it's already a big project. So the idea then became to say, okay, we're not going to implement Stratum V2 itself, but we're going to just offer all the information that Stratum V2 needs. So you install Bitcoin Core, and then you install a separate piece of software called a template provider. That template provider speaks to your own node on one
34:58Sjors Provoostside, and it speaks to Stratum v2, the rest of the Stratum V2 ecosystem on the other side.
35:03Aaron van WirdumIt's kind of like a translator.
35:05Sjors ProvoostThe mining pool or yet another piece of software, because Stratum V2 can be multiple pieces of software.
35:10Aaron van WirdumOkay.
35:11Sjors ProvoostSo in the scenario where you are mining your own block templates, you have Bitcoin Core that talks to the template provider. The template provider then talks to the job declarator server, sorry, client. And the job declarator client talks to the mining pool. to declare the job so it doesn't really matter like in practice nowadays I think there's wizards and so you just say do you want to solo mine or do you want to use a pool do you want to make your own templates yes no and it just installs whatever you need so it hopefully gets to a simplicity level that's comparable to
35:44Sjors Provoostwhat DATUM is doing where you just download like one thing but basically Bitcoin Core does not speak does not understand any of that all it does is just create new block templates for you And the way the Stratum software talks to the node is using this thing called IPC, inter-process communication. And it's just a more efficient way than RPC, what we had before. And that was all part of Bitcoin Core v30, so we might have even covered it two
36:16Sjors Provoostor three episodes ago.
36:20Aaron van WirdumWhat's new? Because I imagine you could already solo mine, for example, with Bitcoin Core. You could already construct a block, right? You could already... say I want to mine or if I were to mine.
36:31Sjors ProvoostYeah, Stratum V1 or just Stratum has been around forever and that was using the RPC and you would call a method called getblocktemplate and it would create a very large JSON blob with all the transactions you need, etc. The problem with that was that it was kind of poll-based. So you as the mining software or the miner, you need to constantly tell your node, okay, can you make a new block for me? Can you make a new block for me? Etc. And there was a way to say, okay, make me a block and let me know when there's a better block
37:04Sjors Provoostavailable. But the only thing it did was check if anything changed in the mempool. And maybe there was a long time ago that that was actually interesting, but nowadays the mempool changes all the time, so you would just get a new block every fraction of a second, which is not useful.
37:19Aaron van WirdumThat's not useful?
37:21Sjors ProvoostNo, you don't want to have... You don't want to have a new block if it's not any better. So something changed in the mempool could be like the bottom one gigabyte of your mempool changed and you get a new template. That's not useful.
37:32Aaron van WirdumIt sounds like this maybe ties into the first thing with the cluster mempool.
37:38Sjors ProvoostSo what the Stratum V2 stuff lets you do, the IPC interface lets you do, is... First, you ask for a block template. That's kind of similar to how it worked before.
37:46Aaron van WirdumIt's what goes into a block.
37:47Sjors ProvoostYeah, a little bit more efficient in terms of the encoding. But for this discussion, it doesn't matter. And then you can subscribe to updates. And in particular, you can say, give me a new block if the fees have gone up by at least one sat per vbyte. So not any change in the mempool, but the actual block itself is better.
38:06Aaron van WirdumYeah, okay.
38:07Sjors ProvoostAnd you can make it a thousand or whatever number there is because... you have some mining infrastructure, some ASICs that maybe you don't want to give them a new template like 100 times a second because it takes them time to update.
38:19Aaron van WirdumRight.
38:20Sjors ProvoostAnd updating is... When it's updating, it's not mining, so you're losing some revenue, so maybe you only want a new template if it's 100 sats or whatever number makes sense.
38:31Aaron van WirdumRight. Okay, so if I understand correctly, this is not... technically a Stratum V2 upgrade but it is very compatible with Stratum V2 so Bitcoin Core 31 released something that's particularly compatible with Stratum V2 and and specifically when it comes to creating your own blocks you know as a miner deciding what transactions you want to include in a block
38:56Sjors Provoostyeah this was in V30
38:57Aaron van Wirdumyou still need some other pieces of software that's the actual Stratum V2 software that takes that data and mines or connects to a pool and or whatever else is needed. We don't have to get into that.
39:10Sjors ProvoostYeah, exactly. So you still need to install some extra software, but Bitcoin Core has methods that are tailor-made for that software, but could be used by some other mining protocol too.
39:22Aaron van WirdumRight.
39:22Sjors ProvoostIt could even be used for Stratum V1-style mining, right? If you run some pool software, it could use the IPC too.
39:29Aaron van WirdumRight, and you mentioned DATUM, so is this also compatible with DATUM?
39:33Sjors ProvoostIf they want to. They're not currently using it, but they could. As far as I know, there's nothing magic that they're doing that doesn't work with this, and otherwise they should open a GitHub issue.
39:42Aaron van WirdumDo you know why they don't use it?
39:44Sjors ProvoostI think partly because they were building this before the IPC interface was even finished. So it's a matter of just sequencing.
39:51Aaron van WirdumBecause Bitcoin core is woke source. That's what I heard.
39:53Sjors ProvoostThat's possible. Maybe there's some evil feature in there. I remember making a GitHub issue on the Bitcoin.
39:59Aaron van WirdumWe were just discussing ASMAP. That's at the very least a bit queer, don't you think?
40:04Sjors ProvoostSo I was making a GitHub issue. on the Bitcoin Core repo, asking the DATUM people, like, hey, is there something you need in the IPC interface? And I think Luke probably replied with something like, well, you guys are fascists or dictators or something like that. So, like, okay, I guess not. But they can still ask for things if they're missing something.
40:23Aaron van WirdumRight. Okay. In any case, this is very compatible with Stratum V2.
40:28Sjors ProvoostAnd so this was already in V30, but V31 just made it better. It fixed some bugs, added some whistles, some bells.
40:36Aaron van WirdumRight.
40:36Sjors ProvoostSo it's great.
40:39Aaron van WirdumOkay.
40:41Sjors ProvoostI think that's all I've got. So anything else?
40:44Aaron van WirdumYeah, I think so as well. I mean, no, I think this was our 100th episode then, Sjors.
40:51Sjors ProvoostExcellent. Well, thank you again then for listening to Bitcoin.
40:54Aaron van WirdumExplained.