Progress pill
Bitcoin Gouvernance

Upgrading

  • Vocabulary
  • Historic upgrades
  • Risks
  • Conclusion about Upgrading
Upgrading Bitcoin in a safe way can be extremely difficult. Some changes take several years to roll out. In this chapter, we learn about the common vocabulary around upgrading Bitcoin, and explore some examples of historic upgrades to its protocol as well as the insights that we gained from them. Finally, we talk about chain splits and the risks and costs related to them.
To get in tune for this chapter, you should read David Harding's piece on harmony and discord:
Bitcoin experts talk often of consensus, whose meaning is abstract and hard to pin down. But the word consensus evolved from the Latin word concentus, "a singing together harmony" so let us talk not of Bitcoin consensus but of Bitcoin harmony. Harmony is what makes Bitcoin work. Thousands of full nodes each work independently to verify the transactions they receive are valid, producing a harmonious agreement about the state of the Bitcoin ledger without any node operator needing to trust anyone else. It's similar to a chorus where each member sings the same song at the same time to produce something far more beautiful than any of them could produce alone. The result of Bitcoin harmony is a system where bitcoins are safe not just from petty thieves (provided you keep your keys secure) but also from endless inflation, mass or targeted confiscation, or simply the bureaucratic morass that is the legacy financial system.
This chapter discusses how Bitcoin can be upgraded without causing discord. Staying in harmony, i.e. maintaining consensus, is indeed one of the biggest challenges in Bitcoin development. There are lots of nuances to upgrade mechanisms, which might be best understood by studying actual cases of previous upgrades. For this reason, the chapter puts much focus on historic examples, and it starts by setting the stage with some useful vocabulary.

Vocabulary

According to Wikipedia, forward compatibility refers to the condition in which an old software can process data created by newer softwares, ignoring the parts it doesn't understand:
A standard supports forward compatibility if a product that complies with earlier versions can "gracefully" process input designed for later versions of the standard, ignoring new parts which it does not understand.
Vice versa, backward compatibility refers to when data from an old software is usable on newer softwares. A change is said to be fully compatible if it's both forward and backward compatible.
A change to the Bitcoin consensus rules is said to be a soft fork if it is fully compatible. This is the most common way to upgrade Bitcoin, for a number of reasons that we'll discuss further in this chapter. If a change to the Bitcoin consensus rules is backward compatible but not forward compatible, it is called a hard fork.
For a technical overview of soft forks and hard forks, please read chapter 11 of Grokking Bitcoin. It explains these terms and also dives into the upgrade mechanisms. It's recommended, although not strictly necessary, to get a grip on this before you continue reading.

Historic upgrades

Bitcoin is not the same today as it was when the genesis block was created. Several upgrades have been made throughout the years. In 2018, Eric Lombrozo spoke at the Breaking Bitcoin conference about Bitcoin's different upgrading mechanisms, pointing out how much they have evolved over time. He even explained how Satoshi Nakamoto once upgraded Bitcoin through a hard fork:
There was actually a hard-fork in bitcoin that Satoshi did that we would never do it this way- it’s a pretty bad way to do it. If you look at the git commit description here [757f076], he says something about reverted makefile.unix wx-config version 0.3.6. Right. That’s all it says. It has no indication that it has a breaking change at all. He was basically hiding it in there. He also posted to bitcointalk and said, please upgrade to 0.3.6 ASAP. We fixed an implementation bug where it is possible that bogus transactions can be displayed as accepted. Do not accept bitcoin payments until you upgrade to 0.3.6. If you can’t upgrade right away, then it would be best to shutdown your bitcoin node until you do. And then on top of that, I don’t know why he decided to do this as well, he decided to add some optimizations in the same code. Fix a bug and add some optimizations.
He points out that, be it intentionally or not, this hard fork created opportunities for future soft forks, namely the Script operators (opcodes) OP_NOP1-OP_NOP10. We'll look more into this code change in cve-2010-5141. These opcodes have been used for two soft forks so far:
  • BIP65 (OP_CHECKLOCKTIMEVERIFY)
  • BIP113 (OP_SEQUENCEVERIFY).
Lombrozo also provides an overview of the way upgrade mechanisms have evolved throughout the years, up until 2017. Since then, only one other major upgrade, Taproot, has been deployed. The long and somewhat chaotic process that led to its activation has helped us gain further insights on upgrading mechanisms in Bitcoin.

Segwit upgrade

While all the upgrades preceding Segwit had been more or less painless, this one was different. When Segwit activation code was released, in October 2016, there seemed to be overwhelming support for it among Bitcoin users, but for some reason miners didn't signal support for this upgrade, which stalled the activation with no resolution in sight.
Aaron van Wirdum describes this winding road in his Bitcoin Magazine article The Long Road To Segwit. He starts by explaining what Segwit is and how that taps into the block size debate. Van Wirdum then outlines the turn of events that led to its final activation. At the center of this process was an upgrade mechanism called user activated soft fork, or UASF for short, that was proposed by user Shaolinfry:
Shaolinfry proposed an alternative: a user activated soft fork (UASF). Instead of hash power activation, a user activated soft fork would have a “‘flag day activation’ where nodes begin enforcement at a predetermined time in the future.” As long as such a UASF is enforced by an economic majority, this should compel a majority of miners to follow (or activate) the soft fork.
Among other things, he cites Shaolinfry's email to the Bitcoin-dev mailing list. In that occasion Shaolinfry argued against miner activated soft forks, listing a number of problems with them:
Firstly, it requires trusting the hash power will validate after activation. The BIP66 soft fork was a case where 95% of the hashrate was signaling readiness but in reality about half was not actually validating the upgraded rules and mined upon an invalid block by mistake. Secondly, miner signalling has a natural veto which allows a small percentage of hashrate to veto node activation of the upgrade for everyone. To date, soft forks have taken advantage of the relatively centralised mining landscape where there are relatively few mining pools building valid blocks; as we move towards more hashrate decentralization, it's likely that we will suffer more and more from "upgrade inertia" which will veto most upgrades.
Shaolinfry also drew attention to a common misinterpretation of miner signaling: people generally thought that it was a means by which miners could decide upon protocol upgrades, rather than an action that helped coordinate upgrades. Due to this misunderstanding, miners might have also felt obliged to proclaim in public their views on a certain soft fork, as if that gave weight to the proposal.
The UASF proposal is, in a nutshell, a "flag day" on which nodes start enforcing specific new rules. That way, miners don't have to make a collective effort to coordinate the upgrade, but can trigger activation earlier than the flag day if enough blocks signal support:
My suggestion is to have the best of both worlds. Since a user activated soft fork needs a relatively long lead time before activation, we can combine with BIP9 to give the option of a faster hash power coordinated activation or activation by flag day, whichever is the sooner. In both cases, we can leverage the warning systems in BIP9. The change is relatively simple, adding an activation-time parameter which will transition the BIP9 state to LOCKED_IN before the end of the BIP9 deployment timeout.
This idea caught a lot of interest, but didn't seem to reach near unanimous support, which caused concern for a potential chain split. The article by Aaron van Wirdum explains how this finally got resolved thanks to BIP91, authored by James Hilliard:
Hilliard proposed a slightly complex but clever solution that would make everything compatible: Segregated Witness activation as proposed by the Bitcoin Core development team, the BIP148 UASF and the New York Agreement activation mechanism. His BIP91 could keep Bitcoin whole — at least throughout SegWit activation.
There were some more complicating factors involved (e.g. the so-called "New York Agreement"), that this BIP had to take into consideration. We encourage you to read Van Wirdum's full article to learn about the many interesting details in this story.

Post-Segwit discussion

After the Segwit deployment, a discussion about deployment mechanisms emerged. As noted by Eric Lombrozo in his talk at the Breaking Bitcoin conference and by Shaolinfry, a miner activated soft fork isn't the ideal upgrade mechanism:
At some point we’re probably going to want to add more features to the bitcoin protocol. This is a big philosophical question we’re asking ourselves. Do we do a UASF for the next one? What about a hybrid approach? Miner activated by itself has been ruled out. bip9 we’re not going to use again.
In January 2020, Matt Corallo sent an email to the Bitcoin-dev mailing list that started a discussion on future soft fork deployment mechanisms. He listed five goals that he thought were essential in an upgrade. David Harding summarizes them in a Bitcoin Optech newsletter as:
The ability to abort if a serious objection to the proposed consensus rules changes is encountered . The allocation of enough time after the release of updated software to ensure that most economic nodes are upgraded to enforce those rules . The expectation that the network hash rate will be roughly the same before and after the change, as well as during any transition . The prevention, as much as possible, of the creation of blocks that are invalid under the new rules, which could lead to false confirmations in non-upgraded nodes and SPV clients . The assurance that the abort mechanisms can’t be misused by griefers or partisans to withhold a widely desired upgrade with no known problems
What Corallo proposes is a combination of a miner activated soft fork and a user activated soft fork:
Thus, as something a bit more concrete, I think an activation method which sets the right precedent and appropriately considers the above goals, would be:
  1. a standard BIP 9 deployment with a one-year time horizon for activation with 95% miner readiness, +
  2. in the case that no activation occurs within a year, a six month quieting period during which the community can analyze and discussion the reasons for no activation and, +
  3. in the case that it makes sense, a simple command-line/bitcoin.conf parameter which was supported since the original deployment release would enable users to opt into a BIP 8 deployment with a 24-month time-horizon for flag-day activation (as well as a new Bitcoin Core release enabling the flag universally).
This provides a very long time horizon for more standard activation, while still ensuring the goals in #5 are met, even if, in those cases, the time horizon needs to be significantly extended to meet the goals of #3. Developing Bitcoin is not a race. If we have to, waiting 42 months ensures we're not setting a negative precedent that we'll come to regret as Bitcoin continues to grow.

Taproot upgrade - Speedy Trial

When Taproot was ready for deployment in October 2020, meaning all the technical details around its consensus rules had been implemented and had reached broad approval within the community, discussions on how to actually deploy it started to heat up. These discussions had been pretty low key up until that point.
Lots of proposals for activation mechanisms started floating around, and David Harding summarized them on the Bitcoin Wiki. In his article he explained some properties of BIP8, which at that time had some recent changes made in order to make it more flexible.
At the time this document is being written, BIP8 has been drafted based on lessons learned in 2017. One notable change following BIPs 9+148 is that forced activation is now based on block height rather than median time past; a second notable change is that forced activation is a boolean parameter chosen when a soft fork’s activation parameters are set either for the initial deployment or updated in a later deployment.
BIP8 without forced activation is very similar to BIP9 version bits with timeout and delay, with the only significant difference being BIP8’s use of block heights compared to BIP9’s use of median time past. This setting allows the attempt to fail (but it can be retried later).
BIP8 with forced activation concludes with a mandatory signaling period where all blocks produced in compliance with its rules must signal readiness for the soft fork in a way that will trigger activation in an earlier deployment of the same soft fork with non-mandatory activation. In other words, if node version x is released without forced activation and, later, version y is released that successfully forces miners to begin signaling readiness within the same time period, both versions will begin enforcing the new consensus rules at the same time.
This flexibility of the revised BIP8 proposal makes it possible to express some other ideas in terms of what they would look like using BIP8. This provides a common factor to use for categorizing many different proposals.
From this point forward the discussions became very heated, especially around whether lockinontimeout should be true (as in a user activated soft fork, referred to as "BIP8 with forced activation" by Harding) or false (as in a miner activated soft fork, referred to as "BIP8 without forced activation" by Harding).
Among the proposals listed, one of them was titled "Let’s see what happens". For some reason, this proposal didn't get much traction until seven months later.
During those seven months, the discussion went on and it seemed like there was no way to reach broad consensus over which deployment mechanism to use. There were mainly two camps: one that preferred lockinontimeout=true (the UASF crowd) and the other one that preferred lockinontimeout=false (the "try and if it fails rethink" crowd). Since there was no overwhelming support for any of these options, the debate went in circles with seemingly no way forward. Some of these discussions were held on IRC, in a channel called ##taproot-activation, but on March 5th 2021, something changed:
06:42 < harding> roconnor: is somebody proposing BIP8(3m, false)? I mentioned that the other day but I didn't see any responses. [...] 06:43 < willcl_ark_> Amusingly, I was just thinking to myself that, vs this, the SegWit activation was actually pretty straightforward: simply a LOT=false and if it fails a UASF. 06:43 < maybehuman> it's funny, "let's see what happens" (i.e. false, 3m) was a poular choice right at the beginning of this channel iirc 06:44 < roconnor> harding: I think I am. I don't know how much that is worth. Mostly I think it would be a widely acceptable configuration based on my understanding of everyone's concerns. 06:44 < willcl_ark_> maybehuman: becuase everybody actually wants this, even miners reckoned they could upgrade in about two weeks (or at least f2pool said that) 06:44 < roconnor> harding: BIP8(3m,false) with an extended lockin-period. 06:45 < harding> roconnor: oh, good. It's been my favorite option since I first summarized the options on the wiki like seven months ago. 06:45 <@michaelfolkson> UASF wouldn't release (true,3m) but yeah Core could release (false, 3m) 06:45 < willcl_ark_> harding: It certainly seems like a good approach to me. _if_ that fails, then you can try an understand why, without wasting too much time
The "let's see what happens" approach finally seemed to click in peoples`' minds. This process would later be labeled as "Speedy Trial" due to its short signaling period. David Harding explains this idea to the broader community in an email to the Bitcoin-dev mailing list:
The earlier version of this proposal was documented over 200 days ago and taproot's underlying code was merged into Bitcoin Core over 140 days ago.If we had started Speedy Trial at the time taproot was merged (which is a bit unrealistic), we would've either be less than two months away from having taproot or we would have moved on to the next activation attempt over a month ago. Instead, we've debated at length and don't appear to be any closer to what I think is a widely acceptable solution than when the mailing list began discussing post-segwit activation schemes over a year ago I think Speedy Trial is a way to generate fast progress that will either end the debate (for now, if activation is successful) or give us some actual data upon which to base future taproot activation proposals.
This deployment mechanism was refined over the course of two months and then released in Bitcoin Core version 0.21.1. The miners quickly started signaling for this upgrade moving the deployment state to LOCKED_IN, and after a grace period the Taproot rules were activated mid-November 2021 in block 709632.

Future deployment mechanisms

Given the problems with the recent soft forks, Segwit and Taproot, it's not clear how the next upgrade will be deployed. Speedy Trial was used to deploy Taproot, but it was used to bridge the chasm between the UASF and the MASF crowds, not because it has emerged as the best known deployment mechanism.

Risks

During the activation of any fork, be it hard or soft, miner activated or user activated, there's the risk of a long-lasting chain split. A split that lingers for more than a few blocks can cause severe damage to the sentiment around Bitcoin as well as to its price. But above all, it would cause great confusion over what Bitcoin is. Is Bitcoin this chain or that chain?
The risk with a user activated soft fork is that the new rules get activated even if the majority of the hash power doesn't support them. This scenario would result in a long-lasting chain split, which would persist until the majority of the hash power adopts the new rules. It could be especially hard to incentivize miners to switch to the new chain if they had already mined blocks after the split on the old chain, because by switching branch they would be abandoning their own block rewards. However, it's worth mentioning a remarkable episode: in March 2013 a long-lasting split, occurred due to an unintentional hard fork and, contrary to this incentive, two major mining pools made the decision to abandon their branch of the split in order to restore consensus.
On the other hand, the risk with a miner activated soft fork is a consequence of the fact that miners can engage in false signaling, which means that the actual share of the hash power that supports the change could be smaller than it looks. If the actual support doesn't comprise a majority of the hash power, we'd probably see a long-lasting chain split similar to the one described in the previous paragraph. This, or at least a similar issue, has happened in reality when BIP66 was deployed, but it got resolved within 6 blocks or so.

Costs of a split

Jimmy Song spoke about the costs associated with hard forks at Breaking Bitcoin in Paris, but much of what he said applies to a chain split due to a failed soft fork as well. He spoke about negative externalities, and defined them as the price someone else has to pay for your own actions:
The classic example of a negative externality is a factory. Maybe they are producing– maybe it’s an oil refinery and they produce a good that is good for the economy but they also produce something that is a negative externality, like pollution. It’s not just something that everyone has to pay for, to clean up, or suffer from. But it’s also 2nd and 3rd order effects, like more traffic going towards the factory as a result of more workers that need to go there. You might also have- you might endanger some wildlife around there. It’s not that everyone has to pay for the negative externalities, it might be specific people, like people who were previously using that road or animals that were near that factory, and they are also paying for the cost of that factory.
In the context of Bitcoin, he exemplifies negative externalities using Bitcoin Cash (bcash), which is a hard fork of Bitcoin created shortly prior to that conference in 2017. He categorizes the negative externalities of a hard fork into one-time costs and permanent costs.
Among the many examples of one-time costs, he mentions the ones incurred by exchanges:
So we have a bunch of exchanges and they had a lot of one-time costs that they had to pay. The first thing that happened is that deposits and withdrawals had to be halted for a day or two for these exchanges because they didn’t know what would happen. Many of these exchanges had to dip into cold storage because their users were demanding bcash. It’s part of their fidicuiary duty, they have to do that. You also have to audit the new software. This is something that we had to do at itbit. We want to spend bcash- how do we do it? We have to download electron cash? Does it have malware? We have to go and audit it. We had like 10 days to figure out if this was okay or not. And then you have to decide, are we going to just allow a one-time withdrawal, or are we going to list this new coin? For an exchange to lis ta new coin, it’s not easy- there’s all sorts of new procedures for cold storage, signing, deposits, withdrawals. Or you could just have this one-off event where you give them their bcash at some point and then you never think about it again. But that has its problems too. And finally, and whatever way you do it, withdrawals or listing– you are going to need new infrastructure to work with this token in some way, even if it’s a one-time withdrawal. You need some way to give these tokens to your users. Again, short-notice. Right? No time to do this, has to be done quickly.
He also lists the one-time costs incurred by merchants, payment processors, wallets, miners, and users, as well as some of the permanent costs, for example privacy loss and a higher risk of reorgs.
Indeed, when a split happens and the chain with the most general rules becomes stronger than the chain with the stricter rules, a reorg will occur. This will have a severe impact on all transactions carried out in the wiped-out branch. For these reasons it's really important to try avoiding chain splits at all times.

Conclusion about Upgrading

Bitcoin grows and evolves with time. Different upgrade mechanisms have been used over the years and the learning curve is steep. More and more sophisticated and robust methods keep being invented, as we learn more about how the network reacts.
To keep Bitcoin in harmony, soft forks have proven to be the way forward, but the big question is still not fully answered: how do we safely deploy soft forks without causing discord?
Quiz
Quiz1/5
What does the concept of Bitcoin harmony, as described through David Harding's analogy, primarily emphasize about the network's operation?