The Long Road to SegWit: How Bitcoin’s Biggest Protocol Upgrade Became Reality

Segregated Witness (SegWit) has activated on Bitcoin. As of today, all SegWit-ready nodes on the Bitcoin network are enforcing the new rules, marking Bitcoin’s biggest protocol upgrade to date.

But activation did not come easy, and it did not come fast.

This is a look back at the long road to SegWit.

The Problem

Bitcoin transactions consist of two main parts. One part is “base transaction data.” That covers which bitcoins are being moved and where they are being moved to, as well as some other data. The second part is called the “witness.” This contains a bit of code with cryptographic signature data, which proves that the owner of a bitcoin really did want to spend the bitcoin.

It’s this signature data that brings a slight complication with it. In what is referred to as the “malleability bug,” Bitcoin signatures can be slightly altered by anyone, even after these signatures are created and without invalidating the signatures. This in turn means that the appearance of the whole transaction, and more specifically the transaction identifier, can be altered by those relaying transactions over the Bitcoin network or by miners that include transactions in blocks.

Statistics from the 2015 malleability attack on Bitcoin. The red lines roughly represent malleated transactions on the network
This doesn’t need to be a big problem in itself. Transactions are still valid and will move the bitcoins from the same place to the same place, under all the same conditions. However, it does complicate creating newer transactions depending on unconfirmed transactions: New transactions need to know the transaction identifier they rely on. This, in turn, makes it significantly harder to build certain second-layer protocols on top of Bitcoin, like bi-directional payment channels.

The Idea

The general idea to solve the malleability bug by “separating” signature data from other transaction data stems back several years.

As far back as 2012, the likes of Bitcoin Core contributors Russell O’Connor, Matt Corallo, Luke Dashjr and Gregory Maxwell, as well as Bitcointalk moderator “Theymos,” discussed the issue on IRC Bitcoin development channels — but at that time they didn’t see a tenable way of pulling it off on the Bitcoin network.

Russell O’Connor, Gregory Maxwell, Luke Dashjr and Theymos discuss the malleability bug on IRC back in 2012
A year later, in August 2013, the issue resurfaced, as Bitcoin Core contributors Peter Todd and Gregory Maxwell were having similar discussions on IRC. But now, the two were making progress with their ideas to counter malleability. “I’m talking about making the [entirety] of the scriptsig largely [separate],” Maxwell wrote. “I’d even suggest using as [transaction ID] the transaction without the scriptsigs.”

Another month later, Maxwell and, this time, well-known cryptographer Dr. Adam Back were discussing the malleability issue on IRC once again. Now, Back suggested computing the transaction ID by omitting the signature. Though, Maxwell commented, “getting the sig out of the txid could help but that would be a very deep hardforking change … and it’s actually tricky to make secure.”

The Sidechain

Blockstream’s initial proposal for sidechain extensions for Bitcoin’s blockchain
In August 2014, blockchain technology company Blockstream was founded by the same Adam Back and Gregory Maxwell, as well as entrepreneur and investor Austin Hill and several Bitcoin Core developers, including Dr. Pieter Wuille. The company was set to focus on sidechains: alternative blockchains that can effectively be pegged to Bitcoin.

By early 2015, Blockstream engineers decided to implement a new feature in the company’s prototype sidechain Elements, which was publicly announced in June of that year. This feature would conclusively solve the malleability issue on the sidechain — by separating base transaction data from witness data into different data structures.

The name of this new feature was, of course, Segregated Witness.

The Block Size Dispute

It had been looming for some time, technically since October 2010, more concretely since February 2013and finally publicly, bursting onto the scene by the spring of 2015: the block size limit dispute.

Former Bitcoin Core lead developer Gavin Andresen and Bitcoinj lead developer Mike Hearn, in particular, believed that Bitcoin’s 1 megabyte block size limit should be increased with a hard fork, an incompatible protocol change that would require almost the entire Bitcoin ecosystem to upgrade. No easy task — even more so because there was no community-wide consensus for this change.

Regardless, by the summer of 2015, Andresen and Hearn announced that they would move forward with their plans, using the alternative Bitcoin XT software client. The controversial nature of the effort put the Bitcoin development community and industry in somewhat of a state of emergency.

In an attempt to resolve the divide and potentially help figure out a resolution to the block size dispute, two conferences (or workshops) were quickly organized in the latter half of 2015: Scaling Bitcoin Montreal andScaling Bitcoin Hong Kong.

One of the most promising scaling proposals presented in Montreal was the lightning network, a sophisticated second-layer scaling solution that was detailed in a white paper published by Joseph Poon and Thaddeus Dryja only months earlier. The only problem: this solution would require a malleability fix.

The Soft Fork

Eric Lombrozo (CodeShark), Wladimir van der Laan (wumpus), Luke Dashjr (luke-jr) and Dr. Pieter Wuille (sipa) discuss SegWit as a soft fork on IRC

At this point in time, developers were still not sure if and how the malleability bug could be fixed. Most still thought Segregated Witness could not be implemented on Bitcoin’s main chain without a hard fork.

But not Bitcoin Core contributor (and Bitcoin Knots maintainer) Luke Dashjr.

In October 2015, right between the two Scaling Bitcoin conferences, Bitcoin Core contributors Eric Lombrozo, Pieter Wuille, Wladimir van der Laan and Luke Dashjr discussed a potential new model for soft forks on IRC. During this chat, Dashjr pointed out that the proposed mechanism wouldn’t work for all potential soft forks, like a SegWit soft fork.

Interestingly, what Dashjr considered obvious — the option to deploy SegWit as a soft fork — had not even been considered by others at all. And even Dashjr didn’t seem to realize the implications of this possibility at first.

To deploy SegWit as a soft fork, witness data had to be placed in a new part of a Bitcoin block. And the “anchor” for all of this witness data (the “Merkle root”) had to be moved to a somewhat unconventional part of a Bitcoin block: the coinbase transaction that rewards miners new coins.

While unconventional, the Bitcoin Core contributors would, over the days and weeks that followed, also come to realize that this method opened up an interesting “bonus.” By creating a new part of a Bitcoin block for the witness data, Bitcoin’s block size could be increased in such a way that non-upgraded nodes wouldn’t notice. This could actually increase Bitcoin’s block size without increasing Bitcoin’s existing block size limit.

Mere weeks before the second Scaling Bitcoin workshop, several Bitcoin Core contributors thought they may finally have found at least a temporary solution for the block size limit dispute. Segregated Witness would effectively increase the limit in a backward-compatible manner, while at the same time fixing the long-standing malleability bug, thereby enabling more advanced scaling solutions like the lightning network.

A win-win-win solution — or so they thought.

The Presentation

Segregated Witness — as a soft fork — was first presented by Pieter Wuille in December 2015, at the second edition of the Scaling Bitcoin workshops in Hong Kong. Many first heard about the proposal there, and it initially seemed to be welcomed with enthusiasm.

Shortly after this second edition of Scaling Bitcoin had ended, Gregory Maxwell proposed what has become known as the scaling roadmap, which featured SegWit as a centerpiece. This roadmap was quicklyendorsed by the Bitcoin Core development team, as well as other developers and users in the broader Bitcoin ecosystem.

The Critique

But despite initial excitement, Segregated Witness had its critics, too.

Concerns about the proposed protocol upgrade varied. Jeff Garzik, the former Bitcoin Core contributor — who would soon after found his own development company Bloq — did not consider SegWit a sufficient short-term scaling solution. Bitcoin XT lead developer Mike Hearn, meanwhile, was not convinced by the proposal at all: He dismissed the solution as an “accounting trick” and completely quit Bitcoin development shortly after.

Jonathan Toomim, developer for alternative software client Bitcoin Classic, argued that the proposal was “ugly and awkward,” suggesting it would be better implemented as a hard fork. Even Bitcoin Core contributor Peter Todd had his concerns, in particular related to mining.

Most of these issues were deemed either solvable, unconvincing or worth the trade-off by the Bitcoin Core development team at large, however. Development of the soft-fork upgrade commenced.

The Development

Even though a version of Segregated Witness was already implemented on Elements, the code for the Bitcoin main chain version mostly had yet to be written, not only because it needed to be implemented as a soft fork, but also because SegWit for Bitcoin would enjoy a range of new features not present in Elements: for example, the “witness discount” necessary to increase the block size, new compatibility for the peer-to-peer network and more.

The concrete Bitcoin Improvement Proposal for SegWit, BIP141, was authored by Pieter Wuille, Ciphrex CEO Eric Lombrozo and independent Bitcoin Core contributor Dr. Johnson Lau. By early January 2016, in the midst of a heated scaling debate, these and other Bitcoin Core contributors launched an initial dedicated test network for the protocol upgrade, which was dubbed SegNet. Another two weeks later, this testnet was taken public for the wider Bitcoin development community to experiment with. And by March, SegNet was upgraded to support test versions of the lightning network.

Development continued for the months to come, taking in feedback from Bitcoin’s development community, fixing bugs, improving the codebase accordingly and launching several more iterations of SegNet(s).

The SegWit GitHub page, where development and other issues are publicly visible for anyone to keep track of and contribute to
Meanwhile, Bitcoin Core contributors also reached out to the broader Bitcoin industry, which over time led to a consistently growing list of companies and projects committing to supporting Segregated Witness.

By June, the Segregated Witness code counted 4,743 lines of code (including test code) and proposed removing or modifying 554 existing lines of Bitcoin Core code. After more review from several contributors, Bitcoin Core lead maintainer, Wladimir van der Laan, merged it into Bitcoin Core’s “master branch” by the end of that month.

The Meetings

At the same time that SegWit was being developed, block size tensions in the Bitcoin community were once again heating up. This time spearheaded by Bitcoin Classic, a number of Bitcoin companies and miners appeared determined to hard fork in order to increase the block size limit to 2 megabytes.

In what is perhaps best described as an emergency meeting, once again in Hong Kong, several Bitcoin Core contributors, mining pool operators and other Bitcoin industry members met to discuss the scaling issue.

The meeting led to an agreement that came to be known as the “Bitcoin Roundtable Consensus” (or the “Hong Kong Agreement”). The Bitcoin Core contributors present at the meeting agreed to work on a block size limit increase hard fork to be proposed to the Bitcoin Core development team and the wider Bitcoin community. The miners, in turn, agreed to run a SegWit release in production by the time such a hard fork would be released in a version of Bitcoin Core. The crisis seemed to have been averted — even though it quickly became clear that not everyone was happy about the agreement.

Several months later, an even bigger group of Bitcoin Core contributors and mining pool operators met in California. The Bitcoin Core contributors present at this meeting left convinced that Segregated Witness would be activated by the miners.

The Release

About six months behind on the initial schedule — the release was originally set for April — Segregated Witness was officially introduced October of 2016, in Bitcoin Core version 0.13.1. The protocol upgrade was also implemented in several other Bitcoin implementations, like Bitcoin Knots and Bcoin.

Using an activation method called “VersionBits” (BIP9), designed to minimize network disruption, 95 percent of miners (by hash power) had to signal support for SegWit to activate on the Bitcoin network. This miner signaling was to start on November 15th. Meanwhile, users were encouraged to upgrade their clients, which over time, it seemed, many did.

As of August 2017, the vast majority of the Bitcoin network consists of SegWit-ready nodes (source: luke.dashjr.org)
Based on the meetings with mining pool operators, as well as a general conviction that SegWit would be a boon for Bitcoin, many expected that the soft fork would be activated rather quickly.

The Politics

But that’s not what happened. As it turned out, several attendees of the Hong Kong Roundtable Consensus disagreed over what they had actually signed onto.

Bitmain co-CEO Jihan Wu, in particular, indicated he would only be willing to activate SegWit if the Bitcoin Core development team also implemented a hard fork to increase the block size limit in their codebase. Other mining pools, including F2Pool, HaoBTC and bitcoin.com didn’t signal support for the soft fork either.

Bitmain (and subsidiary AntPool) demand a hard fork block size limit increase in return for SegWit activation.
Moreover, a new Chinese mining pool emerged: ViaBTC. With close ties to Bitmain, ViaBTC alone garnered enough hash power to single-handedly block SegWit activation. And its operator, Haipo Yang, positioned himself as a staunch critic of the proposed protocol upgrade.

SegWit activation seemed far away.

The UASF

The avatar of pseudonymous Bitcoin and Litecoin developer Shaolinfry
In February 2017, a little over three months after the official release of SegWit, a new opportunity presented itself.

The pseudonymous developer “Shaolinfry,” who had previously contributed to Litecoin, dropped a new proposal in the Bitcoin development mailing list and the popular bitcointalk.org forum: a “user activated soft fork” or “UASF.”

Shaolinfry argued in his email that the hash power activation mechanism that had become the standard for soft forks was never intended to be a “vote.” “[T]he signaling methodology is widely misinterpreted to mean the hash power is voting on a proposal and it seems difficult to correct this misunderstanding in the wider community,” he wrote.

Shaolinfry proposed an alternative: a user activated soft fork (UASF). Instead of hash power activation, a user activated soft fork would have a “‘flag day activation’ where nodes begin enforcement at a predetermined time in the future.” As long as such a UASF is enforced by an economic majority, this should compel a majority of miners to follow (or activate) the soft fork.

The idea immediately generated buzz throughout Bitcoin forums and social media. And when former BTCC COO and outspoken SegWit proponent Samson Mow set up a bounty fund for the development of a UASF software implementation, it seemed like the proposal could become a reality.

The Patented Technology

In the first week of April 2017, Gregory Maxwell dropped what was widely considered a bombshell revelation on the Bitcoin development mailing list.

Maxwell claimed to have reverse-engineered a specialized ASIC-mining chip and found that it included patented AsicBoost technology. What’s more, Maxwell revealed that covert use of the technology would be incompatible with a soft-forked version of SegWit. “An incompatibility would go a long way to explain some of the more inexplicable behavior from some parties in the mining ecosystem,” he noted.

While no specific ASIC-manufacturer was mentioned in Maxwell’s email, Bitmain acknowledged that it had implemented the patented technology in their mining chips — though it denied having used it on Bitcoin’s mainnet.

Either way, for some users the revelation added to the desire to have the Segregated Witness soft fork activated on the Bitcoin network. And, as hash power activation seemed even less likely now, a user activated soft fork was increasingly looking like the solution to accomplish that.

The BIP148 Proposal

Shortly after proposing the general idea of a UASF, Shaolinfry and about a dozen other members of the Bitcoin community opened a UASF channel on the Bitcoin Core Community Slack.

The channel became a central point of discussion and organization for the initiative. A flag date was picked, initially for October 1, then later moved to August 1 to better account for potentially low hash power support. Shaolinfry authored a concrete Bitcoin Improvement Proposal: BIP148. Open Dime founder Rodolfo Novak also established an informational website to promote the idea.

The initial plan was to get exchanges and other businesses on board with the UASF. If these companies would support the proposal and enforce the soft fork, it would go a long way in realizing a desired economic majority.

But the UASF did not gain the level of traction some of its proponents hoped for. While a number of companies and some developers seemed onboard with BIP148, no major exchanges or other businesses declared support and some even spoke out against the initiative.

And, by mid-April, Gregory Maxwell on the Bitcoin development mailing list stated that he believed BIP148 to be a bad idea as well. Coming from one of the most respected and influential Bitcoin Core contributors, his rejection of the initiative had an impact: This version of a UASF appeared to be losing all momentum.

Instead, some began to work on an alternative UASF: BIP149.

The Altcoins

Many altcoins are based on Bitcoin’s codebase. This means that the SegWit code, while developed for Bitcoin, is largely compatible with these alternative cryptocurrencies. Unsurprisingly, therefore, several altcoins decided to implement SegWit. The first to activate Segregated Witness was Groestlcoin as early as January 2017.

But other coins were struggling. Litecoin, Vertcoin and Viacoin all seemed to have been caught in Bitcoin’s political game. These coins relied on the same miners as Bitcoin, to a large extent, and most were not signaling support for the upgrade.

This was allegedly due to technical issues or other stated reasons, but, as Viacoin lead developer Romano noted, “It seems more likely they want to refrain from activating Segregated Witness on altcoins because that would give them even less reason to hold off activation on Bitcoin.”

By April of 2017, this attitude led Litecoin creator Charlie Lee to advocate for a user activated soft fork on “his” coin. His initiative was eagerly picked up among Litecoin users; it didn’t take long for Litecoin miners, Lee, and other members of the Litecoin ecosystem to arrange an online meeting, the result of which was the Litecoin Global Roundtable Resolution. In exchange for some commitments by Lee, miners agreed to activate SegWit. ShaolinFry and others considered the UASF effort a success.

Within a week after SegWit activation on Litecoin, an unknown person made a bold move. He (or she) sent $1 million worth of the cryptocurrency to a SegWit-protected address, challenging anyone to steal the funds if they could. To this date, the bounty remains untouched, further strengthening confidence in the technology.

The New York Agreement

Meanwhile, the block size debate raged on. Another software client to increase Bitcoin’s block size limit per hard fork, Bitcoin Unlimited gained traction among Bitcoin’s mining community. Endorsed by Bitmain’s Wu in particular, the project appeared to be heading toward a potential (and controversial) hard fork.

This looming threat, and the possibility of a “split” in Bitcoin’s blockchain, was reason for DCG founder and CEO Barry Silbert to organize a meeting ahead of the Consensus 2017 conference in New York. Initially announced on a private email list for Bitcoin entrepreneurs and other prominent industry members, the meeting would bring together a significant chunk of the Bitcoin industry, including miners — though, notably, no Bitcoin Core contributors.

The outcome of that meeting is typically referred to as the “New York Agreement.” Participants agreed on what they deemed to be a compromise between those who wanted to increase Bitcoin’s block size with a hard fork and those who preferred SegWit. Based on an idea originally proposed by RSK founder Sergio Demian Lerner, SegWit would be activated under specific conditions, while there would also be a hard fork to double Bitcoin’s “base block size limit.”

The New York Agreement and its two concrete points of action
But while it sufficed to say not everyone in the Bitcoin ecosystem supported the agreement, one specific problem stood out in particular. The conditions for SegWit activation were largely incompatible with those proposed by the Bitcoin Core development team, for which the code was already widely adopted by Bitcoin users.

The Intolerant Minority

Imagery by Samson Mow in support of the BIP148 UASF
While the BIP148 UASF seemed to have lost a lot of steam in favor of BIP149, not everyone had given up on this first UASF proposal completely.

Shaolinfry had proposed the concept under the assumption that it would be backed by an economic majority and thought it should be aborted before the flag day otherwise. But a group of users on the UASF Slack channel had a different idea. Some of them — including Bitcoin Core and Bitcoin Knots developer Luke Dashjr — were contemplating activating the soft fork regardless of what the rest of the Bitcoin ecosystem would do. Even if they were a minority, and even if they’d effectively spin themselves off into a new altcoin, they would move forward with the upgrade.

Around mid-May, Alphonse Pace linked this determination to a game-theoretical concept described by statistician and author Nassim Nicholas Taleb: the “intolerant minority.” In short, this idea presupposes that even an economic minority should be able to compel miners to activate the Segregated Witness soft fork. They would unnecessarily lose a chunk of their “customer base” (Bitcoin users) otherwise.

Seemingly fuelled by the AsicBoost scandal, the SegWit activation on Litecoin and discontent regarding the New York Agreement — and this time backed by game theory — BIP148 support started to snowball into somewhat of a viral phenomenon on social media and message boards once again.

Several more articles discussed the growing potential of the UASF and much debate on social media, YouTube channels other discussion platforms followed. Meanwhile, Eric Lombrozo also threw his weight behind the effort, and UASF hats distributed by Samson Mow became the rage. Inspired by the codename for an upcoming Electrum Wallet release, August 1 was dubbed “Bitcoin Independence Day.”

The only problem: activation methods for BIP148 and the New York Agreement were as incompatible as the New York Agreement was with the activation methods proposed by the Bitcoin Core development team.

The Kludge

It was Bitmain Warranty engineer James Hilliard who came to the rescue. Hilliard proposed a slightly complex but clever solution that would make everything compatible: Segregated Witness activation as proposed by the Bitcoin Core development team, the BIP148 UASF and the New York Agreement activation mechanism. His BIP91 could keep Bitcoin whole — at least throughout SegWit activation.

As long as a majority of miners would activate BIP91 before August 1, all Bitcoin nodes should remain part of the same network. It was a relatively small time window, since the solution was only proposed by late May, but Jeff Garzik, the main developer attached to the New York Agreement, adopted the proposal and planned to release the software client resulting from that agreement weeks before August 1. It was doable.

The Activation

Informational website XBT.eu at the time of BIP91 lock-in
By mid-July, Bitcoin miners had missed their window to activate SegWit through the method proposed by the Bitcoin Core development team in time to be compatible with BIP148. As a result, markets seemed to get nervous about a potential “split” between a BIP148 chain and a non-BIP148 chain. In the span of only a week, bitcoin’s exchange rate tumbled from around $2500 to $1900: the lowest it had been in well over a month.

Possibly startled by these market movements, Bitcoin’s mining community started to rapidly signal support for BIP91, even ahead of the schedule set forth by the New York Agreement. And on July 20, ten days before BIP148’s August 1 flag day for activation, BIP91 locked in. It activated a little over two days later.

With BIP91 locked in, it was only a matter of time before Segregated Witness itself would lock in. This ultimately happened on August 9 — the point of no return having been reached on August 8.

Bitcoin would “officially” get SegWit after another two-week grace period.

The Adoption

Segregated Witness logo designed by Albert Dros
The final step for Segregated Witness is, of course, actual user adoption. Since SegWit has only just activated at the time of publication of this article, it’s impossible to know how quickly and how much the upgrade will actually be used. Some critics, perhaps most notably Garzik, predict that widespread adoption could take up to a year or even longer. Others, including a number of wallet and library developers, think they can utilize the feature within weeks, or they are prepared already. And other technologies that depend on the upgrade, such as the Lightning Network, but also Merkelized Abstract Syntax Trees (MAST), atomic swaps, faster transaction signing for hardware wallets, the more efficient Schnorr signature algorithm, and TumbleBit in payment processor mode, are in various stages of development as well.

It’s been a long road, but anyone who wants to use Segregated Witness should now be able to do so, starting today.