Abstract:ForkMonitor has now implemented unexpected inflation detection and warning systems for Bitcoin. The block reward is currently 12.5 bitcoin, which means that no more than 12.5 new bitcoin should be created each block. Some of the ForkMonitor nodes now calculate the total coin supply each block, using the gettxoutsetinfoRPC call. If the total coin supply increases by more than 12.5 bitcoin, warnings systems are initiated. This service potentially provides additional assurances to network participants about the supply of Bitcoin at any given time.
ForkMonitor has recently added a new feature, unexpected inflation detection. This feature has been added for Bitcoin and Testnet Bitcoin. The system periodically checks the total coin supply by summing up all the UTXO values. If the value is unexpectedly large, warnings are activated. Bitcoin nodes are already supposed to check the coin supply, however this occurs by only checking that no unauthorised coins are created in each individual transaction and there is no macro total supply check. Therefore the ForkMonitor service could provide an additional layer of security and assurance for Bitcoin users, as well as an early warning system which could encourage people to run these checks on their own nodes if an issue is detected.
If the inflation is in line with expectations, a green tick is displayed on the website. However, if unexpected inflation occurs, a red cross is displayed alongside other warnings.
Illustration of unexpected inflation detected by Bitcoin Core 0.18.1
Please subscribe to the feeds, to be altered in the event of unexpected Bitcoin inflation.
Coin Supply Checking Mechanisms
The systems plan to check the inflation using the following methods:
Coin supply change from the previous block – After each Bitcoin block is produced, the system checks the total coin supply and stores the figure in a database. As each new block is produced, the summation is repeated and the total coin supply is subtracted from the previous figure. If the change is higher than the allowed block reward (12.5 bitcoins today, 6.25 bitcoins from around May 2020, etc.), then the warnings are initiated.
Consistency across multiple node versions – In addition, the system will also check that the total bitcoin supply is consistent at each block height for all the nodes participating in the inflation check (which is illustrated on the ForkMonitor website).
One of the main challenges we faced when implementing this inflation check feature was that it took considerable time for Bitcoin Core to run the gettxoutsetinfocall, typically around 2 minutes. This created several implementation challenges for ForkMonitor, such as what to display in this two minute period or what happens when a block is found while the calculation is occurring. For example, the maximum rate at which the inflation check can move forwards is one block every two minutes; if many blocks are found in a row, with smaller than two minute intervals between them, our check can be ineffective for a while.
Gettxoutsetinfo RPC call – Bitcoin’s supply of approximately 18 million is illustrated
(Source: Output from Bitcoin Core 0.18.0 “Gettxoutsetinfo” call)
Others are aware of these issues, as Bitcoin developer Fabian Jahr recently put it:
[Thegettxoutsetinfocall] does not have a sufficient user experience, you call it and it actually takes several minutes to respond and there is no feedback
In 2017 Bitcoin developer Pieter Wuille posted to the Bitcoin development mailing list, a potential improvement, which he says could make this Remote Procedure Call (RPC) call faster.
Replacement for Bitcoin Core’s gettxoutsetinfo RPC’s hash computation. This currently requires minutes of I/O and CPU, as it serializes and hashes the entire UTXO set. A rolling set hash would make this instant, making the whole RPC much more usable for sanity checking
Based on the above idea, Fabian recently indicated he may work on implementing this potential fix, in an attempt to improve this RPC call. If this improvement is implemented, it would certainly be helpful for ForkMonitor.
Bitcoin’s 2018 Inflation Bug (CVE-2018-17144)
ForkMonitor was very much inspired by the events of September 2018, when it emerged that Bitcoin Core had a bug which would enable miners to create coins out of nowhere in addition to the normal block reward. This bug affected versions of Bitcoin Core spanning from 0.14.0 to 0.16.2, before the fixes were released. (0.14.X nodes merely crashed while later nodes would have accepted the blocks with the unexpected inflation).
A successful exploitation of this bug could have had catastrophic consequences for the network, for example Bitcoin’s supply could have inflated above 21 million or a large rollback may have occurred, undermining the security many users and businesses depend on.
ForkMonitor was launched to mitigate these risks. If such a bug existed today, our systems should be able to detect it in three ways:
ForkMonitor runs multiple versions of Bitcoin Core, spanning many years of development. If a newly-introduced bug results in unexpected inflation or an unauthorized spend, the older nodes should detect this and mark the block as invalid, triggering the warning systems.
The website also runs independent implementations of Bitcoin, such as bcoin, btcd and Libbitcoin. If Bitcoin Core has a bug which allows unexpected inflation or an unauthorized spend, as long as the same bug wasn’t independently implemented, these other clients should mark the block as invalid, triggering the warning systems.
As of October 2019, ForkMonitor also directly checks the total coin supply of each block. In the event of unexpected inflation, even in the unlikely scenario that all our nodes mark the block as valid, the warning systems will still be triggered. The inflation checking system is also helpful even if nodes do mark the block as invalid, as it can help users determine why this was the case in a timely manner.
As we explained in our October 2018 piece, Competing with Bitcoin Core, there are advantages and disadvantages of competing implementations and in particular independent implementations. One key advantage of independent implementations that we mentioned is that there could be a bug in Bitcoin Core or the reference implementation which is not present in the independent implementations.
For the above reason, we are keen to add one of the three independent implementations (bcoin, btcd and Libbitcoin) into the total coin supply inflation checking system as soon as possible. The method of calculating total coin supply used by these implementations may be independent from that used by Bitcoin Core, which should provide extra reassurance that the number is correct.
This new service may not solve all potential problems with regards to detecting unexpected inflation. For example there could be a bug in the gettxoutsetinfo check. In addition to this, the various mechanisms to check for unexpected inflation and block validity may not be truly independent from each other. Even the independent Bitcoin implementations may have inadvertently copied a bug or erroneous concept from Bitcoin Core. However, we believe this macro inflation checking service is potentially a useful addition to network security.
As a reminder, the ForkMonitor website is open source, therefore please feel free to contribute, fork the project or reproduce the website.
Abstract: In this piece we look back at the history of Bitcoin, focusing in on “The Bitcoin Foundation”, once one of the most prominent organisations in the ecosystem. We look at Foundation’s origins and then examine its failings with respect to its governance, transparency and finances, which ultimately led to a total loss of legitimacy within the Bitcoin community. We conclude that an all-encompassing Foundation was never likely to have been a good idea given the high governance and transparency standards of some in the community, and that a constant stream of scandals damaged the Foundation’s brand to such an extent that its duties had to be carried out by other organisations.
(Screenshot of the Bitcoin Foundation’s website and logo in 2013)
The Foundation’s Origins
Following on from our July 2018 piece, which took us back to shenanigans and incompetence at MtGox in 2011, this second look at Bitcoin’s scandal-rich history takes us back to July 2012, when The Bitcoin Foundation was founded. The Foundation had seven founding members, or six if you exclude Satoshi, who was oddly included as a founding member.
Bitcoin Foundation Founding Members
Gavin Andresen, Bitcoin Developer
Peter Vessenes, CEO of CoinLab
Charles Shrem, CEO of BitInstant
Roger Ver, CEO of MemoryDealers
Patrick Murck, Principal at Engage Legal
Mark Karpeles, CEO of MtGox.com
Satoshi Nakamoto, author of the white paper “Bitcoin: A Peer-to-Peer Electronic Cash System”
The objective of the Foundation was never completely clear, with the original bylaws stating the following:
The Corporation shall promote and protect both the decentralized, distributed and private nature of the Bitcoin distributed-digital currency and transaction system as well as individual choice, participation and financial privacy when using such systems. The Corporation shall further require that any distributed-digital currency falling within the ambit of the Corporation’s purpose be decentralized, distributed and private and that it support individual choice, participation and financial privacy.
The Foundation was funded by membership fees – the initial membership fee schedule is provided below. However, the Bitcoin-denominated prices did start to decline in 2013 as the Bitcoin price appreciated.
It was believed by many that due to the membership subscription fees, the Foundation had considerable financial resources to spend on its mission.
Approximate lower bound of member contributions in April 2013 (Assuming initial fee rates)
2 Platinum Industry Members * 10,000 BTC = 20,000 BTC
7 Silver Industry Members * 500BTC = 3,500 BTC
175 Lifetime Members * 25BTC = 4,375 BTC
Total Resources = 27,873 BTC
(Source: BitMEX Research)
As we see later on in this report, the Foundation only had around 8,000 BTC at the end of 2012, still a nice warchest, but a lower balance than many had expected. It is possible our estimate above could be an overestimate, as the timing of member subscriptions is unclear.
The Foundation Board
The governance structure of the Foundation was quite complex and arcane. There were three classes of members:
The board initially consisted of five members, one nominated by the founders, two nominated by individuals and two nominated by corporate members. The term of each appointee was expected to be 3 years. At the start of the Foundation, all five board members were appointed by the founders and all board members were founders, with the exception of Jon Matonis.
Bitcoin Foundation Board Members (2012 to 2019)
(Source: Bitcoin Foundation Website, BitMEX Research)
Critics can point to the fact that the governance structure gave too much power to the initial founders and that new members of the organisation should have been able to join as equals to the founders.
The first board elections took place in 2013, with Meyer Malka winning the Industry seat and Elizabeth Ploshay winning the vote amongst individual members.
Board Election – Industry Seat (2013) – Winners: Meyer Malka
At the start of 2014, the holders of the two founding industry seats resigned. Charles Shrem resigned on 28 January 2014, two days after his arrest at JFK airport for money laundering and unlicensed money transmitter related offences. Charlie was eventually convicted and sentenced to two years in prison in December 2014. The main substance of Mr Shrem’s felony appears to be that he continued to provide customer support to a user of his BitInstant Bitcoin purchasing service, despite him allegedly knowing the customer wanted Bitcoin for the purposes of purchasing illegal drugs on the Silk Road e-commerce platform (Or that the customer wanted to supply the Bitcoin to somebody else, who wanted to purchase illegal drugs, one extra layer removed). Mark Karpeles, the holder of the other industry seat, resigned on 24 February 2014, following the failure and insolvency of the MtGox Bitcoin exchange, where Mark was CEO.
Brock Pierce and Bobby Lee were then elected as the two replacement industry appointed board members.
Board Election – Industry Seats (2014) – Winners: Bobby Lee & Brock Pierce
The appointment of Brock Pierce to the board proved controversial, with some claiming the Foundation should have done more vetting before allowing Mr Pierce to stand. The allegation against the former child actor, who featured in the “Mighty Ducks” and Disney’s “First Kid” was related to his alleged involvement in the sexual exploitation of children in the late 1990s. Although only a teenager at the time, Mr Pierce was an executive and co-founder at the internet video startup, Digital Entertainment Network (DEN), which was accused of hosting several parties where sexual abuse may have taken place. The allegations resulted in co-founder and CEO Marc Collins-Rector, along with Mr Pierce, resigning from DEN and supposedly fleeing to Spain. Mr Collins-Rector eventually plead guilty to child abuse related offences and according to Reuters, court record show that Mr Pierce paid US$21,000 to settle a related civil suit, while other claims were dropped, the article also states that Mr Pierce denies the allegations.
Towards the end of 2014, in the face of considerable pressure, the Foundation made the following improvements to its governance:
Board member terms were reduced to 2 from 3 years
The founder board seat was eliminated
The founder member class was removed
The Foundation’s Finances
The below table provides a basic analysis of the Foundation’s finances, in the period where most of the member dues were depleted (2012 to 2014). The data is based on the organisation’s IRS990 forms. With respect to the pay of the board, the disclosure seems reasonably strong. Most board members received no remuneration other than those acting as executives. Paying Gavin was one of the main aims of the organisation and Gavin’s pay appears to be disclosed in a reasonably clear and appropriate manner.
Jon Matonis (Contractor)
Other pay costs
Total pay costs
Disclosed Bitcoin figures
Bitcoin (US$ value at year end)
BTC sales proceeds
Realised Bitcoin gains/(losses)
Unrealised Bitcoin gains/(losses)
(Source: IRS 990 Forms, BitMEX Research)
The main criticisms related to the Foundation’s finances at the time appear twofold:
There was a sharp increase in spend in 2014, depleting the organisation’s reserves to near zero
There was a lack of transparency with regards to the Foundation’s Bitcoin balance
As for the first criticism, concerns did seem somewhat justified. In 2014 pay costs increased by 81%, the 2014 conference made a significant net loss and other costs increased significantly. As for the $1.3m in other costs, we have provided a breakdown below, therefore readers can judge the extent of the excesses. Compared to the excesses of the ICO bubble in 2017/18, where the total sum of the costs below perhaps represent a fraction of just one marketing party for the most egregious ICOs, the spend is moderate. However, some Foundation members clearly expected their funds to be spent more prudently. The main issue appears to be that expectations were not clearly set out in advance. Whatever your view, the fact is that by the start of 2015, the Foundation had almost run out of financial reserves and to that extent, its finances were mismanaged.
2014 breakdown of other spend
Other professional services
Professional event expenses
Payments to affiliates
Total other spend
(Source: Bitcoin Foundation IRS 990 form)
The lack of transparency with respect to the Foundation’s Bitcoin balance is another area of concern. At the end of each year the IRS990 form disclosed the USD value of the Bitcoin holding, the realised Bitcoin gains and the unrealised Bitcoin gains. Based on this information we calculated the following:
BitMEX Research BTC calculations
Bitcoin price at year end
Implied BTC balance at year end
Change in BTC balance
Implied sales price
Realised Bitcoin gains/(losses)
Unrealised BTC gains/(losses)
Lowest Bitcoin price figures
Lowest Bitcoin price in the year
Implied BTC sales proceeds
Realised Bitcoin gains/(losses)
(Source: IRS 990 Forms, BitMEX Research)
The disclosures in the IRS990 forms lead us to the following apparent Bitcoin related discrepancies:
The Foundations closing bitcoin balance in 2012 seems reasonably low given the volume of Bitcoin donations (See the c.28,000 BTC figure earlier in this report)
The Foundation disclosed an unrealised Bitcoin gain in 2013 of $5.2m, however based on the annual price movement and the calculated year end balance, we calculated an unrealised gain of only $4.4m
The Foundation disclosed an unrealised Bitcoin loss in 2014 of $2.0m, however based on the annual price movement and the calculated year end balance, we calculated an unrealised loss of only $0.6m
The Foundation disclosed Bitcoin sales proceeds of $569,728 in 2014, while even assuming all Bitcoin were sold at the lowest traded price in the year, given the large reduction in the Bitcoin balance of 4,600 coins, sales proceeds should have been $1.2m
Although there were accusations of embezzlement, we do not consider these disclosures to indicate any such crime. The Foundation was probably receiving Bitcoin and spending Bitcoin throughout the period, therefore clear financial record of Bitcoin sales are not likely to be available. At the same time, rules related to the reporting of realised and unrealised gains with respect to financial assets are not strict for this type of organisation and the Foundation does have a degree of discretion with respect to the calculation methodology. Therefore, the filings themselves do not indicate wrongdoing in our view. However, what we can say is that filings do not clearly explain what happened to the Bitcoin balance and an explanation from the board could be helpful.
Some members clearly expected greater transparency and wanted to question the board about the funds, but they were never given such an opportunity. The following quote from Bitcoin commentator Andreas Antonpoulous (who at the time was a Foundation committee chairman), reflected the views of many in the community at the time.
You say they are funded. Where are those funds? Who controls those funds? When were the last audited? Are they actually solvent? Or have all of those funds disappeared into a big black hole? Just remember who was in the leadership until recently, who is in the leadership today and what their track record of ethics has been and I would suggest that I would not be surprised at all if the Foundation implodes in a giant embezzlement problem sometime down the line or funds get stolen, within quotes or without quotes, or something like that. It’s bound to happen because these things don’t happen due to technical failures of bad actors, they happen due to failures of leadership The Foundation is the very definition of a failure of leadership.
To make matters worse, there were also accounations of the Foundation’s entanglement in the MtGox insolvency:
The MtGoX CEO, Mark Karpeles, was a founder and founding board member of the Foundation, while the company itself was a platinum member of the Foundation
Founding member, Roger Ver, famously assured MtGox customers of the solvency of the platform shortly before the exchange failed
The Foundation’s founding chairman, Peter Vessenes (who may have believed he was entitled to some MtGox equity), has been involved in various legal disputes with MtGox dating back to 2013 as a result of a failed business partnership. Peter’s company Coinlab sued MtGox for US$75m in 2013. As of August 2019, Peter now claims a remarkable total of US$16bn (Y1.6 trillion) from MtGox, an amount large enough to effectively block distributions to MtGox clients, and a large source of frustration to creditors to this day.
Andreas compared the Foundation’s situation to MtGox as follows:
Its problems go directly back to a complete failure of leadership. A completely closed, insular, arrogant, sheltered, uncommunicative leadership. Part of which was Karpeles himself, but there are another couple of relics left on that board, who pursue the exact same approach with their leadership. The Foundation is the Gox of Foundations. I am surprised it didn’t blow up in the wake of the Gox scandal, because there were a lot of significant conflicts within that environment.
However, perhaps it is unfair to make much of the association between MtGox and the Foundation, afterall, the ecosystem was small and MtGox was the dominant exchange, therefore a degree of association was inevitable to some extent.
The Amsterdam Conference (May 2014)
In May 2014 the Bitcoin Foundation arranged what was, up until this point, the largest conference in the space. It was the first conference (at least one which we attended), with characteristics familiar to many in the 2017/18 era. Unabated enthusiasm, unrealistic expectations about the underlying technology, expensive catering and countless booths representing new businesses with plans that appeared to make little commercial sense. As the figures above indicate, despite the expensive ticket prices of up to $800, the conference appears to have generated a net loss of around US$250,000.
The conference was split into two sections, a commercial section in the main exhibition hall, and the Bitcoin Foundation annual meeting (or technical track), which was down the hallway in a hotel conference room, entry to which was free for Foundation members. The technical discussions were followed by the Foundation members’ meeting
Journalist Ryan Selkis (now founder & CEO of Messari), was one of the key lifetime members at the event trying to hold the Foundation to account. At the annual meeting he asked several challenging questions to the Foundation board members, asking for greater transparency. Up until this point much of the debate and complaints had taken place on online web forums and this real world interaction marked a significant and novel change. In response to his challenges, one board member said the following:
We can spend a lot of our time trying to be transparent as much as we can and higher resources can be transparent or we can spend a lot of time in the board level making sure that we [have the] resources to make bitcoin bigger. It’s possible but right now, honestly, we’re in an environment where bitcoin is not well perceived. You asked for priorities at least from my side as a board member, it’s more about [making bitcoin bigger]
It was clear from this response that, for whatever reason, some board members had chosen not to tackle the transparency and governance concerns, leaving some members feeling frustrated and more convinced of wrongdoing on the part of the board.
The Blockchain Election (February 2015)
Given the issues that the Foundation had faced and the concerns in the community about transparency, governance and the purpose of the Foundation, this was a relatively important set of elections. There was a large number of candidates and a reasonably good quality debate among the candidates, for example a dedicated Let’s Talk Bitcoin podcast on the election.
The Foundation decided to conduct the 2015 individual board seat elections on the blockchain. As the chair of the election committee, Brain Goss said:
I believe in the concept of using the block chain for storage of compact proofs/hashes (as the market dictates), and I’m a big believer in transparent voting that any one can verify
However, the blockchain voting process did not run smoothly and the following issues arose:
The first round of voting took place using the Helios voting system. However, no candidate achieved more than 50% of the vote, as required by the by-laws, therefore a second round was required. The Foundation then made the odd decision to switch the voting platform to Swarm between the voting rounds, a decision met with widespread opposition. Despite initially starting the final round voting process on Swarm, during voting the Foundation then decided to switch back to Helios, invalidating the Swarm votes
The decision to reduce the number of candidates to four after the first round of voting appeared arbitrary
Registering to vote was widely regarded as a cumbersome and complex process and some candidates complained
(Source: Email received as part of the Swarm voting process)
Board Election – Individual Seats First Round (2015)
(Source: Helios voting system records)
Board Election – Individual Seats Final Round (2015) – Winners: Oliver Janssens & Jim Harper
After the voting controversy, Patrick Murk told Bitcoin Magazine:
This clearly struck a nerve with folks that think blockchain technology should only be used for transferring Bitcoin and not other [applications] like voting. [It] sparked a debate on how people use the blockchain
Removal of Directors & The End Of Board Elections (December 2015)
In December 2015, the two newly elected board members, Oliver and Jim, were removed by the other board members, due to a disagreement over the best way forward for the Foundation. Oliver and Jim had recently succeeded in competitive elections from individual members, giving them a considerable democratic mandate. At the same time the two year election terms of Elizabeth and Meyer had already expired, while Brock and Bobby had been elected by the industry and not individuals. Therefore, from the point of view of the individual members, Oliver and Jim were the only two board members with a significant mandate and they had been removed. In a violation of the by-laws, the Foundation then decided not to conduct any further board elections. As the executive director Bruce Fenton put it:
I used to believe that public, open elections were a great thing. I’m not as convinced now…. We unfortunately don’t have the time or resources for more process.
In our view, this logic seemed difficult to justify, given many of the problems were caused by the boards apparent lack of accountability to individual members, with Elizabeth Ploshey being the only board member elected by individual members who served on the board for any meaningful amount of time. If the Foundation did want to revive itself, it could have reinstated Oliver and Jim and allowed further elections to replace the other board members who could have left. Instead, the Foundation decided to distance itself even further from members, avoiding the challenges this accountability would have imposed, and consequently the Foundation appeared to lose any remaining legitimacy it had left.
After this point, between 2015 and 2019, four new board members were appointed from the pool of candidates that were defeated in the previous elections, except this time appointments were made by the board rather than members.
The Foundation still exists today, with Brock as Chairman and Bobby as Vice chairman, although their elected terms have long since expired and no more elections are in sight. The Foundation has no significant financial resources and is largely irrelevant. The activities the Foundation used to conduct are now carried out by others, for example Coin Centre does some regulator lobbying, and Bitcoin development is funded by other organisations such as Chaincode Labs, Blockstream, MIT’s DCI and other industry players. In many ways the conclusion to this piece writes itself. Bitcoin never needed a Foundation, it is stronger without one, and any all-encompassing Foundation like this was always doomed to fail.
The outrage at the lack of transparency at the Foundation exposes some of the key divergences in expectations and culture between members of the Bitcoin (now cryptocurrency) community. Some Bitcoiners, especially those involved since the early days of the Foundation, were often highly conspiratorial, paranoid and expected radically high levels of transparency, accountability and financial prudence. The Foundation seems to have misjudged these expectations, lost the backing of the community and ultimately failed. However, compared to the excesses of the coin offering era, which picked up from around 2014 onwards, peaking at the start of 2018, the financial accountability and transparency of the Bitcoin Foundation was almost impeccable, relatively speaking. Some members of the cryptocurrency community (not all newer ones), had radically different expectations, focusing more on what they perceived as game-changing technology, changing the world and getting super rich, rather than governance. Even in this new climate, irreparable damage to the Foundation’s brand had been done and it never again found its place.
UPDATE – 23 September 2019
After the publication of this piece, several prominent Bitcoin developers, whose names were displayed on the Foundation’s website, indicated to us (in some cases with proof) that they were given membership status for free (rather than by paying 25BTC). This may indicate that:
for the foundation may not have been as widespread as we initially thought
bitcoin balance in 2012 may never have been as large as we initially thought
Abstract: BitMEX Research has upgraded its lightning nodes to include watchtower functionality. The watchtower functionality is a mechanism for connecting to another friendly node, which monitors your lightning channels for you and prevents a dishonest counterparty from stealing your funds, even when you are offline. We successfully conducted an experiment, proving the watchtower concept actually works, at least in our case. It is encouraging that the watchtower concept, which has been around for years in theory, now actually works in practise.
On 29 June 2019, LND 0.7.0 (Go implementation of lightning) was released and this included the watchtower functionality. A watchtower is a third party lightning node, that can detect if a dishonest party attempts to steal funds and then broadcast a justice transaction, sending the funds back to the honest party, even when the honest node is offline.
There two modes of watchtower functionality
The client connects to a watchtower server. Whenever the lighting channel states change, data is sent over to the watchtower server with the latest channel state. In the event of a channel breach, the watchtower can broadcast a justice transaction, sending the funds to the honest node’s onchain wallet.
The watchtower server does not need to have any lighting channels or make any payments. The server connects to a lightning client and monitors the client’s lightning channels for them, on their behalf.
To connect the node to a watchtower server, one needs to add the following line to the lightning configuration file:
Where the public key and IP address is provided by the watchtower server.
To activate a watchtower server, one needs to add the following line to the lightning configuration file:
After this, one can run the command:
> lncli tower info
The watchtower server should then display the watchtower public key (different from the lightning node public key). This key is needed by the watchtower client. Due to potential denial of service threats, it is currently not advisable to publish the watchtower public key.
One can check if the watchtower is working by viewing the logs.
It is possible for a node to be both a watchtower server and client at the same time. If you run two nodes, each node can be the watchtower server of the other. BitMEX Research currently has three operating lightning nodes and the nodes all watch over each other in a loop configuration.
Successful test of the watchtower
On 30th July 2019, BitMEX Research successfully tested the watchtower system. Much like our previous piece on justice transactions, we tried to cheat ourselves, but this time used a watchtower. In an encouraging sign, the watchtower functionality correctly worked and the would-be thief was punished.
In order to do this test, we needed to run three nodes:
The dishonest node – BitMEXThief
The node using the watchtower service – BitMEXTowerClient (the user of the watchtower service)
The watchtower itself – BitMEXResearch
Manually constructing a watchtower justice transaction
(Source: BitMEX Research)
The eventual justice transaction, broadcast by our watchtower can be seen here.
All BitMEX Research lightning nodes are now protected by watchtowers. While a watchtower is a large improvement in security, in our view, a greater problem than dishonest channel breaches, is the risk of a lightning node’s memory becoming accidentally lost or destroyed – under such circumstances the node could lose the latest channel states. A watchtower does not fix that problem, although there have been improvements in this area, with Static Channel Backups (SCBs). Using SCBs, as long as no new channels were created post backup, all the funds should be safe.
A successful test of the watchtower does provide us with a greater degree of assurance about the robustness of the lightning network. It is encouraging that ideas such as watchtowers, which have been theoretically discussed for years, finally exist. However, when it comes to improving the robustness and reliability of the lightning network, there is still a long way to go.
Abstract: In our third look at the lightning network, we examine lightning channel closure scenarios and the incentives to punish dishonest parties and prevent them from stealing funds. This punishment mechanism is called a “Justice Transaction”. We explain how to arbitrarily construct a “Justice” scenario and present data on the prevalence of this type of transaction on the Bitcoin network. We have potentially identified 241 Justice transactions, representing 2.22 Bitcoin in value, since the lightning network launched at the end of 2017.
Following on from our January 2018 discussion of the motivation behind the lightning network and our March 2019 analysis of lightning network routing fee economics, this third piece on the lightning network looks at channel closures and the incentives designed to prevent dishonest lightning nodes from stealing funds, by broadcasting an earlier channel state.
It should be noted that, by design, when a thief attempts to steal funds on the lightning network, if caught, they do not only lose the money they tried to steal, they lose all the funds in the relevant channel. This “punishment” is expected to act as a deterrent and is sometimes called “justice”.
The four lightning channel closure scenarios
Opening lightning channels is, generally speaking, more simple than closing them, there is only one way to open a lightning channel, cooperatively with interactive communication between the parties. On the other hand, when evaluating channel closures, one needs to consider four different scenarios, as outlined in the decision tree below (See figure 1).
A non-cooperative non-breach closure occurs when an honest node initiates the closure, without directly communicating with the node on the other side of the channel.
Funds are distributed to each party’s onchain wallet based on the latest channel state.
These two different economic scenarios, are represented by one technical onchain scenario.
This scenario requires two onchain transactions.
Firstly the funds are redeemed using a 2 of 2 multi-signature witness and sent to two outputs. The node which did not initiate the closure is allocated funds based on what the channel closing party says is attributable to them, while another pot of funds is sent to an output which can be redeemed by using either an OP_IF or an OP_ELSE script.
In a second transaction, the funds sent to the OP_IF script, are claimed by the party that initiated the channel closure, using the OP_ELSE branch of Bitcoin script.
A non-cooperative breach non-justice closure occurs when a dishonest node initiates the channel closure, by broadcasting an earlier channel state, attempting to steal funds from the node on the other side of the channel.
The non closing node does not check the network within the locktime period, normally 24 hours and does not broadcast a justice transaction. Therefore the theft is successful.
Funds are distributed to each party’s wallet based on an earlier channel state, such that the non closing party losses funds and the dishonest channel closing party successfully steals funds.
A non-cooperative breach justice closure occurs when a dishonest node initiates the channel closure, without directly communicating with the node on the other side of the channel.
The non closing node does check the network within the locktime period, and creates a justice transaction, such that the attempted theft fails.
The would-be thief is punished and all the funds go to the honest non closing party.
In the justice scenario, two onchain transactions are also required.
In the first transaction, the funds are redeemed using a 2 of 2 multi-signature witness and sent to two outputs. The node which did not initiate the closure is allocated funds based on what the channel closing party says is attributable to them, while another pot of funds is sent to an output which can be redeemed by using either an OP_IF or an OP_ELSE script.
In a second transaction, the honest node, that did not initiate the closure claims all the funds sent to the OP_IF script, using the OP_IF branch.
This is the most revealing of the three channel closure types and provides the lowest level of privacy.
In the below arbitrary scenario, we manually created a justice transaction, using the following steps:
1. Spin up a new lightning network node (LND), with the alias “BitMEXThief” and open a channel, worth US$50 (400,000 Satoshis) with the BitMEXResearch lightning node 2. Switch off the BitMEXThief node and back up the .lnd directory 3. Restart the BitMEXThief node and make a lightning payment of US$25 (200,000 satoshis) to BitMEXResearch. The channel is now balanced, US$25 in both directions 4. Switch off the BitMEXThief node again 5. Switch off the BitMEXResearch lightning node (to prevent it broadcasting the latest channel state to the thief node) 6. Restore the BitMEXThief node back to its state prior to the channel re-balancing, the state in step 2 7. On the restored BitMEXThief node, attempt to close the channel from its earlier state and claim the full US$50 (400,000 satoshis) to the BitMEXThief node’s onchain wallet 8. Restart the BitMEXResearch node. The node then automatically detects the attempted theft and broadcasts the “justice transaction”, sending the full US$50 (less fees) to its onchain wallet. The would be thief was punished, by losing all the funds inside the channel. Note that the thief attempted to steal US$25, but ended up losing the full US$50.
The above experiment occurred successfully, providing some assurance that Lightning does actually work and if you try to steal, you will be punished.
Network Justice transaction data
After conducting our own justice transaction, we looked at the characteristics of this transaction (Inputs redeemed using the OP_IF branch) and searched for other justice transactions on the Bitcoin blockchain. We identified 241 transactions, which appear to be justice channel closures, dating back as far as December 2017. Mr. Alex Bosworth, from Lightning Labs, has created a tool to identify justice transactions, which may be more robust than our more basic search methodology.
Figure 3 – Number of justice transactions – monthly
(Source: BitMEX Research)
(Note: There is a possibility the data includes false positives)
Figure 4 – Value redeemed in justice transactions – monthly (BTC)
(Source: BitMEX Research)
(Note: There is a possibility the data includes false positives)
The justice transactions we identified had transaction inputs totaling 2.22 BTC, with the monthly total peaking at around 0.67BTC in February 2019, as figure 4 above illustrates. This does not necessarily mean thieves tried and failed to steal 2.22 BTC, as the dis-honest nodes may have punished thieves by a amount larger than the value they tried to steal (we do not know the latest channel state). The 2.22 BTC represents the total funds claimed by honest non channel closing nodes, part of this value is funds originally owned by the dis-honest nodes and part of the value will be the value they tried to steal.
It is also possible that many of the 241 justice transactions do not indicate genuine dishonestly, for instance it could be users testing the system, where the same user owns both lightning nodes in question. For example BitMEX Research is responsible for 5 of the 241 justice transactions, when there was no victim, as BitMEX owned all the nodes and funds.
241 justice transactions, with a value of just over 2 BTC is reasonably small relative to the size of the lightning network. The lightning network statistics website 1ml.com, indicates that there are currently 940 BTC locked up in 32,951 channels. The total number of justice transactions in the last 18 months is therefore only 0.7% of the current number of lightning channels.
In order for the lightning network to succeed as a robust, reliable and scalable payment system, the justice mechanism needs to be effective in deterring and preventing theft. As for the optimal justice rate, this is hard to determine, if it is too high and it shows that successful thefts may be too prevalent and the threat of justice may not be sufficient. If it is too low, it may mean nobody is attempting theft, thereby increasing the risk that users do not monitor their channels. This may lead to increases in the risk of large systemic channel thefts in the future.
For now, at least according to the data we have analysed, there appears to be a reasonable degree of justice on the burgeoning lightning network.
Abstract: In a bold move, social networking giant Facebook, has challenged the traditional finance and ETF industry, with its “Libra coin”, or as we call it the “Libra ETF”. We note that there are many unanswered questions about Libra, which may lack transparency, when compared to traditional ETFs. Another key disadvantage of Libra is that unlike with legacy ETFs, investment income is not distributed to unit holders. We conclude that although Libra has significant disadvantages when compared to traditional ETF products, Facebook’s wide consumer reach with platforms such as Whatsapp and Instagram could give Libra a key commercial advantage.
(Facebook vs Blackrock – The battle for the ETFs)
The structure of Libra is analogous to the popular Exchange Traded Fund (ETF) model, where unit holders are entitled to the financial returns of a basket of financial assets. The units are tradable on exchanges and a select group of authorised participants are able to create and redeem units using the underlying assets.
As we pointed out in our February 2019 piece, the ETF industry has enjoyed considerable growth in the last decade or so, in particular in the area of fixed income (See figure 1 below). In June 2019, in a bombshell moment for the ETF industry and challenge for the established players such as Blackrock and Vanguard, social media and internet conglomerate Facebook, entered the game. In a direct challenge to Blackrocks’s “iShares Core U.S. Aggregate Bond ETF” (AGG), Facebook announced plans to launch a new ETF, the “Libra ETF”, also focused on fixed income and government bonds.
Figure 1 – Size of the Top Bond ETFs Targeting US Investors – US$ Billion
(Source: BitMEX Research, Bloomberg)
(Note: The chart represents the sum of the market capitalisations of the following bond ETFs: iShares Core U.S. Aggregate Bond ETF, Vanguard Total Bond Market ETF, iShares iBoxx $ Investment Grade Corporate Bond ETF, Vanguard Short-Term Corporate Bond ETF, Vanguard Short-Term Bond ETF, Vanguard Intermediate-Term Corporate Bond ETF, iShares J.P. Morgan USD Emerging Markets Bond ETF, Vanguard Total International Bond ETF, iShares MBS Bond ETF, iShares iBoxx $ High Yield Corporate Bond ETF, PIMCO Enhanced Short Maturity Strategy Fund, Vanguard Intermediate-Term Bond ETF, iShares Short-Term Corporate Bond ETF, SPDR Barclays High Yield Bond ETF, iShares Short Maturity Bond ETF)
Comparing the new ETF structure with the traditional space
In figure 2 below, we have analysed and compared the new innovative Libra ETF to a traditional ETF, Blackrock’s iShares Core US Aggregate Bond ETF (AGG). Our analysis shows that, although the Libra product is new, much of the relevant information, such as transparency of the holdings and frequency of the publication of the NAV, has not yet been disclosed.
The analysis also highlights that Libra may suffer from unnecessary complexity with respect to portfolio management. The fund appears to be managed by the Libra Association, which consists of many entities in multiple industries across the globe. These same entities are responsible for issuing the ETF and the list of companies is set to expand further. At the same time, the investment mandate is unclear. In contrast Blackrock’s fixed income ETF product has a clear investment mandate, to track the Bloomberg Barclays U.S. Aggregate Bond Index, which is managed independently of the ETF issuer.
Perhaps the most significant disadvantage of the Libra product, is that unit holders do not appear to be entitled to receive the investment income. This contrasts unfavourably with Blackrock’s product, which focuses on an almost identical asset class and has an investment yield of around 2.6%. Defenders of Libra could point out that the expenses need to be covered from somewhere and that the Libra’s expense fee is not yet disclosed. However, the ETF industry is already highly competitive, with Blackrock charging an expense fee of just 0.05%. This expense fee is far lower than the expected investment yield of the product, at around 2.6% and therefore the Libra ETF may not be price competitive, a key potential disadvantage for potential investors.
Figure 2 – Libra ETF vs iShares Core U.S. Aggregate Bond ETF (AGG) – Detailed Comparison
iShares Core U.S. Aggregate Bond ETF (AGG)
The Libra Association/Facebook
Bank deposits and government securities in currencies from stable and reputable central banks
Fixed income – Investment grade government and corporate bonds
Bloomberg Barclays U.S. Aggregate Bond Index
The Libra Association, based in Switzerland will manage the reserve. The investment mandate is not currently disclosed. The current members are as follows:
PayU (Naspers’ fintech arm)
Union Square Ventures
Creative Destruction Lab,
Women’s World Banking
James Mauro and Scott Radell, with a clear constrained mandate to track the index
Use of investment income
Unit holders are not entitled to investment incomeInvestment income will:
first go to support the operating expenses of the association — to fund investments in the growth and development of the ecosystem, grants to nonprofit and multilateral organizations, engineering research, etc. Once that is covered, part of the remaining returns will go to pay dividends to early investors in the Libra Investment Token for their initial contribution
Attributable to ETF unit holders
The Libra Association
will encourage the listing of Libra on multiple regulated electronic exchanges throughout the world
Creation/redemption basket size
Authorized Participants (entities able to create and redeem units)
Authorized resellers, not currently disclosed
Information about holdings and Net Asset value (NAV)
We have also analysed the two alternatives from a technical perspective. As figure 3 below indicates, the key difference is that control of Libra tokens may in part be managed by digital signatures. As long as no whitelist of addresses is implemented, this may provide some advantages:
A limited amount of censorship resistance
Relatively easy integration with cryptocurrency exchanges
However, as we mentioned in our Tether report in February 2018, history has shown that these characteristics can cause platforms to ultimately face a choice between implementing KYC or face being shut down by the authorities. Facebook has already censored politically controversial figures on its main platform, therefore it may appear likely the extent to which Libra ETF units are managed by public private key cryptography is significantly constrained or eventually becomes phased out.
Figure 3 – Technical and cryptographic considerations
iShares Core U.S. Aggregate Bond ETF (AGG)
Not applicable (An ETF does not require a consensus system)
Not relevant (Grouping records of ETF transactions into a chain of blocks linked together by hashing, is inconsequential for ETFs)
Control of units based on digital signature
The Libra Blockchain is pseudonymous and allows users to hold one or more addresses that are not linked to their real-world identity
Despite the key disadvantage, namely that Libra unit holders are not entitled to the investment income, many industry analysts are carefully examining the impact Libra could have on the traditional ETF industry and existing electronic payment systems.
While our comparison to ETFs is a bit tongue and cheek, it does highlight that the structure of the product has similar attributes to existing financial products. We therefore think it is an appropriate comparison, and if Libra wants to be competitive, it should emulate some of the governance and fee characteristics of traditional ETFs.
However, Libra could attract clients due to integration with platforms such as Facebook, Whatsapp and Instagram. If Libra does retain the property of allowing coins to be controlled by private keys, this is an interesting development and the coin is likely to gain share from tokens such as Tether. However, in our view, in the long run, it is likely Libra either disables this feature or makes it technically difficult, such that only a tiny minority of users have these “non-custodial” wallets. If that happens, Libra is nothing more than a high fee ETF.
We are delighted to announce HDR Global Trading Limited’s support of the MIT Digital Currency Initiative, which conducts research into the development and betterment of the global cryptocurrency ecosystem.
Sam Reed, CTO of HDR Global Trading and co-founder of the BitMEX trading platform, announced the sponsorship:
Our company has always been energized by the potential of cryptocurrency. Our donation into research and development is about ensuring that the network is more robust. A stronger Bitcoin network will be beneficial to all, and we are very excited to be able to aid in its progress.
HDR Global Trading owns and operates BitMEX, the world’s largest cryptocurrency trading platform by volume. HDR Global Trading is proud to support Bitcoin research and engineering that will make Bitcoin stronger, improving Bitcoin’s robustness, scalability and privacy.
In particular, HDR is keen to help support the work of Bitcoin Core developers Wladimir van der Laan and Cory Fields. Their roles have important implications on different parts of the Bitcoin protocol.
The donation is provided unconditionally and without restrictions.
Abstract: The 15 May 2019 Bitcoin Cash hardfork appears to have suffered from three significant interrelated problems. A weakness exploited by an “attack transaction”, which caused miners to produce empty blocks. The uncertainty surrounding the empty blocks may have caused concern among some miners, who may have tried to mine on the original non-hardfork chain, causing a consensus chainsplit. There appears to have been a plan by developers and miners to recover funds accidentally sent to SegWit addresses and the above weakness may have scuppered this plan. This failure may have resulted in a deliberate and coordinated 2 block chain re-organisation. Based on our calculations, around 3,392 BCH may have been successfully double spent in an orchestrated transaction reversal. However, the only victim with respect to these double spent coins could have been the original “thief”.
Illustration of the Bitcoin Cash network splits on 15 May 2019
(Source: BitMEX Research) (Notes: Graphical illustration of the split)
The three Bitcoin Cash issues
Bitcoin Cash’s May 2019 hard fork upgrade was plagued by three significant issues, two of which may have been indirectly caused by a bug which resulted in empty blocks. The below image shows the potential relationships between these three incidents.
The relationships between the three issues faced by Bitcoin Cash during the hardfork upgrade
(Source: BitMEX Research)
The empty block problem
Bitcoin ABC, an important software implementation for Bitcoin Cash, appears to have had a bug, where the validity conditions for transactions to enter the memory pool may have been less onerous than the consensus validity conditions. This is the opposite to how Bitcoin (and presumably Bitcoin Cash) are expected to operate, consensus validity rules are supposed to be looser than memory pool ones. This is actually quite an important characteristic, since it prevents a malicious spender from creating a transaction which satisfies the conditions to be relayed across the network and get into a merchants memory pools, but fails the conditions necessary to get into valid blocks. This would make 0-confirmation double spend attacks relatively easy to pull off, without one needing to hope their original payment doesn’t make it into the blockchain. In these circumstances, an attacker can be reasonably certain that the maliciously constructed transaction never makes it into the blockchain.
An attacker appears to have spotted this bug in Bitcoin Cash ABC and then exploited it, just after the hardfork, perhaps in an attempt to cause chaos and confusion. This attack could have been executed at any time. The attacker merely had to broadcast transactions which met the mempool validity conditions but failed the consensus checks. When miners then attempted to produce blocks with these transactions, they failed. Rather than not making any blocks at all, as a fail safe, miners appear to have made empty blocks, at least in most of the cases.
Bitcoin Cash – Number of transactions per block – orange line is the hardfork
(Source: BitMEX Research)
The asymmetric chainspilt
At the height of the uncertainty surrounding the empty blocks, our pre-hardfork Bitcoin ABC 0.18.2 node received a new block, 582,680. At the time, many were concerned about the empty blocks and it is possible that some miners may have reverted back to a pre-hardfork client, thinking that the longer chain was in trouble and may revert back to before the hardfork. However, this is merely speculation on our part and the empty block bug may have had nothing to do with the chainsplit, which could have just been caused by a miner who was too slow to upgrade.
Bitcoin Cash consensus chainsplit
(Source: BitMEX Research)
The chainsplit did highlight an issue to us with respect to the structure of the hardfork. We tested whether our post hardfork client, ABC 0.19.0, would consider the non-hardfork side of the split as valid. In order for the break to be “clean”, each side of the split should consider the other as invalid.
In order to test the validity of the shorter pre-hardfork chain, from the perspective of the Bitcoin ABC 0.19.0 node, we had to invalidate the first hardfork block since the split. We then observed to see whether the node would follow the chainsplit or remain stuck at the hardfork point. To our surprise, as the below screenshot indicates, the node followed the other side of the split. Therefore the split was not clean, it was asymmetric, potentially providing further opportunities for attackers.
Screenshot of the command line from our Bitcoin ABC 0.19.0 node
(Source: BitMEX Research)
The coordinated two block re-organisation
A few blocks after the hardfork, on the hardfork side of the split, there was a block chain re-organisation of length 2. At the time, we thought this was caused by normal block propagation issues and did not think much of it. For example, Bitcoin SV experienced a re-organisation a few weeks prior to this, of 6 blocks in the length. When Bitcoin SV re-organised, all transactions in the orphaned chain eventually made it into the main winning chain (except the Coinbase transactions), based on our analysis. However, in this Bitcoin Cash re-organisation, we discovered that this what not the case.
The orphaned block, 582,698, contained 137 transactions (including the Coinbase), only 111 of which made it into the winning chain. Therefore a successful 2 block double spend appears to have occurred with respect to 25 transactions. The output value of these 25 transactions summed up to over 3,300 BCH, as the below table indicates.
List of transactions in the orphaned block (582,698) which did not make it into the main chain
As the above table shows, the total output value of these 25 double spent transactions is 3,391.7 BCH, an economically significant sum. Therefore, one may conclude that the re-organisation was an orchestrated event, rather than it having occurred by accident. If it occurred by accident, it is possible there would be no mismatch between the transactions on each side of the split. However, assuming coordination and a deliberate re-org is speculation on our part.
We have provided two examples of outputs which were double spent below:
Example of one of the double spent UTXOs – “0014”
(Source: BitMEX Research)
The above table illustrates what happened to a 5 BCH output during the re-organisation. The 5 BCH was first sent to address qzyj4lzdjjq0unuka59776tv4e6up23uhyk4tr2anm in block 582,698. This chain was orphaned and the same output was eventually sent to a different address, qq4whmrz4xm6ey6sgsj4umvptrpfkmd2rvk36dw97y, 7 block later.
Second example of one of the double spent UTXOs – “0020”
(Source: BitMEX Research)
What happened to the above outputs shares characteristics with almost all the funds in the 25 double spent transactions. Most of the outputs appear to have been double spent around block 582,705 on the main chain, around 7 blocks after the orphaned block.
The SigScript, used to redeem the transaction inputs, starts with “0020” or “0014”, highlighted in the above examples. These may relate to Segregated Witness. According to the specification in Segregated Witness, “0014” is pushed in P2WPKH (Pay to witness public key hash) and “0020” is pushed in P2WSH (Pay to witness script hash). Therefore the redemption of these inputs may have something to do with Segregated Witness, a Bitcoin upgrade, only part of which was adopted on Bitcoin Cash.
Indeed, based on our analysis, every single input in the 25 transactions in the orphaned block 582,698 was redeemed with a Sigscript starting “0014” or “0020”. Therefore it is possible that nobody lost funds related to this chain re-organisation, other than the “attacker” or “thief” who redeemed these SegWit outputs, which may have accidentally been sent to these outputs in the first place.
As part of the Bitcoin Cash May 2019 hardfork, there was a change to allow coins which were accidentally sent to a SegWit address, to be recovered. Therefore, this may have occurred in the incident.
Allow Segwit recovery
In the last upgrade, coins accidentally sent to Segwit P2SH addresses were made unspendable by the CLEANSTACK rule. This upgrade will make an exemption for these coins and return them to the previous situation, where they are spendable. This means that once the P2SH redeem script pre-image is revealed (for example by spending coins from the corresponding BTC address), any miner can take the coins.
It is possible that this 2 block re-organisation is unrelated to the empty block bug. However, the split appears to have occurred just one block after the resolution of the bug, therefore it may be related. Perhaps the “honest” miners were attempting to coordinate the spend of these outputs directly after the split, perhaps to return them to the original owners and the empty block bug messed up their timing, allowing the attacker to benefit and sweep the funds.
On the other hand, the attack is quite complex, therefore the attacker is likely to have a high degree of sophistication and needed to engage in extensive planning. Therefore, it is also possible this attack may have been effective even without the empty block bug.
There are many lessons to learn from the events surrounding the Bitcoin Cash hardfork upgrade. A hardfork appears to provide an opportunity for malicious actors to attack and create uncertainty and therefore careful planning and coordination of a hardfork is important. On the other hand, this empty block bug, which may be the root cause of the other 2 incidents, could have occurred at any time and trying to prevent bugs like this is critical whether one is attempting to harfork or not.
Another key lesson from these events is the need for transparency. During the incidents it was difficult to know what developers were planning, the nature of the bugs, or which chain the miners were supporting. Open communication in public channels about these issues could have been more helpful. In particular, many were unaware of an apparent plan developers and miners had to coordinate and recover lost funds sent to SegWit addresses. It may have been helpful if this plan was debated and discussed in the community more beforehand, as well as during the apparent deliberate and coordinated re-organisation. Assuming of course if there was time to disclose the latter. It may also be helpful if those involved disclose the details about these events after the fact.
The largest concern from all of this, in our view, is the deliberate and coordinated re-organisation. From one side of the argument, the funds were stolen, therefore the actions were justified in returning the funds to their “rightful owners”, even if it caused some short term disruption. However, the cash like transaction finality is seen by many, or perhaps by some, as the only unique characteristic of these blockchain systems. The ability to reverse transactions, and in this case economically significant transactions, undermines the whole premise of the system. Such behavior can remove incentives to appropriately secure funds and set a precedent or change expectations, making further reversals more likely.
For all those in the Bitcoin community who dislike Bitcoin Cash, this could be seen as an opportunity to laugh at the coin. However, although Bitcoin Cash has a much lower hashrate than Bitcoin, making this reversal easier, the success of this economically significant orchestrated transaction reversal on Bitcoin Cash is not positive news for Bitcoin in our view. In some ways, these incidents contribute to setting a dangerous precedent. It shows that it may be possible in Bitcoin. Alternatively, this could just illustrate the risks Bitcoin Cash faces while being the minority chain.
Abstract: In this piece we present data on a relatively new phenomenon, Initial Exchange Offerings (IEOs). The ICO market is down around 97% in Q1 2019 (YoY), based on the amount of capital raised. In this relatively challenging climate to raise funds, some projects have changed the “C” in ICO to an “E”, perhaps in an attempt to assist with raising capital. At least for now, to some extent, this appears to be working, with almost $40m having been raised so far this year. However, we remain sceptical about the prospects for long term investors.
We consider an Initial Exchange Offering (IEO) as the issuance and sale of a token based on public-private key cryptography, where participation in the issuance occurs exclusively through one trading platform or exchange. This piece provides a basic overview of the largest IEOs and tracks various IEO token metrics, including investment performance.
First we briefly look at the ICO market. As the following chart indicates, the market has dried up following a massive boom in 2017 & 2018.
Funds raised by ICOs – US$M
Source: BitMEX Research, icodata.io Notes: Data as at 25 April 2019
As the below chart illustrates, the investment returns of the 2018 ICOs has been poor, many of the projects are down around 80% from the ICO price, if the coin even trades at all. Peak to trough, project token prices typically declined much further than this.
Top ten ICOs by funds raised in 2018 – Investment performance data
Funds raised – US$m
Return based on average ICO price
Coin not listed
Coin not listed
Coin not listed
Returned capital to investors
Source: BitMEX Research, tokendata.io Notes: Data as at 25 April 2019
Changing a “C” into an “E” – The IEO market
Perhaps in an attempt to address some of the concerns about the poor investment returns and the lower levels of enthusiasm for ICOs, IEOs appear to have gained in popularity. Below is a list of the major IEOs and the main exchange platforms involved.
List of IEO token sales
IEO issue amount vs total coin supply
Return vs first exchange trade price
Return vs IEO price
US$m raised in IEO
Source: BitMEX Research, IEO Launchpad websites, Coinmarketcap Notes: Data as at 25 April 2019
The number of IEOs taking place has intensified in recent months, as the model is proving somewhat successful. Smaller exchange platforms are attempting to replicate the model, as the long list of IEOs below illustrates.
Other IEOs with limited data available
KIZUNA GLOBAL TOKEN
Link by BlockMason
GTEX Gaming Platform
Source: BitMEX Research, IEO Launchpad websites
With respect to all but one of the tokens, investors have earned strong positive returns based on the IEO price. However, after the tokens begin trading, the investment returns have typically been poor. This is illustrated by the below chart, which rebases the token price to the IEO issuance price.
IEO Investment performance since launch (IEOs in 2019)
Source: BitMEX Research, IEO Launchpad websites, Coinmarketcap Notes: Data as at 25 April 2019
US$38.9m has been raised so far by IEOs in 2019 (up to 25th April). Binance has been the most prolific IEO platform by a considerable margin.
Top exchange platforms by IEO funds raised – US$m
Source: BitMEX Research, IEO Launchpad websites, Coinmarketcap Notes: Data as at 25 April 2019
The proceeds from IEOs can be relatively small, however on average only 4.4% of the total token supply is made available in the sale. Therefore, there are opportunities for project teams to make considerable profits from selling coins they granted to themselves. The 2019 IEOs were priced at a level which implies a total market capitalisation of US$907.7m, based on the disclosed total token supply.
Top exchange platforms by IEO token market capitalisation at IEO price – US$m
Source: BitMEX Research, IEO Launchpad websites, Coinmarketcap Notes: Data as at 25 April 2019
While exchanges, traders & subscribers may have done very well from IEOs thus far, we are less confident on the outlook for long term investors. However, this is simply a high level analysis – we have not looked into any of the individual projects in detail.
Any views expressed on BitMEX Research reports are the personal views of the authors. BitMEX (or any affiliated entity) has not been involved in producing this report and the views contained in the report may differ from the views or opinions of BitMEX.
The information and data herein have been obtained from sources we believe to be reliable. Such information has not been verified and we make no representation or warranty as to its accuracy, completeness or correctness. Any opinions or estimates herein reflect the judgment of the authors of the report at the date of this communication and are subject to change at any time without notice. BitMEX will not be liable whatsoever for any direct or consequential loss arising from the use of this publication/communication or its contents.
If we have made any errors in relation to particular projects, we apologise and are happy to correct the data as soon as possible.
Abstract: We summarise and provide context for a recent Bitcoin softfork upgrade proposal, which includes a new digital signature scheme (Schnorr), as well as a complementary upgrade called Taproot, which adds new capabilities that extend Bitcoin’s smart contracting capability. The upgrades are structured to ensure that they simultaneously improve both scalability and privacy. Other than increased complexity, there are no significant downsides to the proposal, and the most controversial aspect of it is likely to be the lack of other anticipated features. We conclude that although many will be enthusiastic about the upgrade and keen to see it rolled out, patience will be important.
On 6th May 2019, Bitcoin protocol developer Pieter Wuille posted a softfork upgrade proposal to the Bitcoin developer mailing list, called “Taproot”. If this proposal is accepted, it is likely to complement the Schnorr signature softfork upgrade, which Pieter posted in July 2018. The benefits of these proposals are related to both scalability (efficiency) and privacy. Scalability and privacy enhancements now appear somewhat interrelated and inseparable. Removing details about transactions, ensures both that transactions are smaller (improving scalability) and that they reveal less information and are therefore potentially indistinguishable from transactions of different types, thereby improving privacy.
The Schnorr signature scheme was patented in 1991 by Claus Schnorr and the patent expired in 2008. Although the Schnorr scheme is said to be stronger, a variant of it, the Digital Signature Algorithm (DSA) scheme was more widely adopted, as the patent for this scheme was made available worldwide royalty free. However, Dr Schnorr himself always maintained that DSA should be covered under his patent.
When Bitcoin was launched, in 2009, it therefore used a variant of DSA, Elliptic Curve Digital Signature Algorithm (ECDSA) for its digital signature scheme, due to its widespread adoption. However, the original Schnorr signature scheme was always more simple and efficient than DSA, with less burdensome security assumptions. After 10 years of experience of Bitcoin usage, it is becoming more apparent that these efficiency advantages could be important. Therefore it seems sensible that Bitcoin should migrate over to the Schnorr signature scheme.
The main benefit with Schnorr signatures, is that multi-signature transactions appear onchain as a normal single signature transaction. Using Schnorr signatures, multiple signers can produce a joint public key and then jointly sign with one signature, rather than publishing each public key and each signature separately on the blockchain. This is a significant scalability and privacy enhancement. This implies that Schnorr signatures result in significant space savings and savings to verification times, with the comparative benefits getting larger as the number of signatories on a traditional multi-signature transaction increase.
Schnorr signature space saving estimates
We have tried to calculate the potential Bitcoin network capacity increase this aggregation feature of Schnorr multisig can provide. However, due to the large number of assumptions involved, our 13.1% capacity increase figure below should be considered as a very approximate estimate.
Savings estimates based on UTXO count
Estimated current multi-signature usage by UTXO count
(Source: BitMEX Research calculations and estimates, p2sh.info)
(Notes: The estimates ignore the impact of Schnorr’s smaller signature size and only include the benefits of joining the public keys and signatures. The capacity increase was estimated by using p2sh.info related to multi-signature usage and applying a savings multiple to each multi-signature type (ranging from 50% to 85%). A network wide capacity increase was estimated by assuming the UTXO usage proportion was typical of blockchain usage and applying a higher weight to larger multi-signature transactions. Unspent P2SH outputs were allocated to multi-signature types in proportion to the spent outputs. This figure should only be considered as a very approximate estimate. Data as at 07 May 2019 )
The above estimated capacity increase can be considered as small, however one should consider the following:
Economic usage of multi-signature technology is far more prevalent than by merely looking at the UTXO count. Around 21.5% of all Bitcoin is stored in multi-signature wallets, a far higher figure than the 5.9% adoption by UTXO count
Multi-signature adoption is growing rapidly, as the below chart indicates. While at the same time new systems like the lightning network require multi-signature adoption and with Schnorr signature making multi-signature systems more powerful, adoption is likely to increase
Bitcoin stored by P2SH address type – chart shows strong growth of multi-signature technology
Therefore, although based on the current usage of the network, according to our basic calculation, even 100% Schnorr adoption only results in a 13.1% network capacity increase, in the long term the potential space savings and network capacity gains are likely to be far higher than this.
Merkelized Abstract Syntax Tree (MAST)
MAST was an idea worked on by Bitcoin protocol developer Dr Johnson Lau in 2016. Dr Lau has written for BitMEX Research in the past, in his February 2018 piece entitled The art of making softforks: Protection by policy rule. The MAST idea is that transactions can contain multiple spending conditions, for example a 2 of 2 multi-signature condition, in addition to a time lock condition. In order to avoid putting all these conditions and scripts into the blockchain, the spending scripts can be structured inside a Merkle tree, such that they only need to be revealed if they are used, along with the necessary Merkle branch hashes.
Graphical illustration of MAST spending conditions
(Source: BitMEX Research) (Notes: The diagram is trying to illustrate a transaction structure assuming MAST was used in conjunction with Schnorr. In the above construction funds can be redeemed the cooperative way if both Bob and Alice sign, or in an uncooperative way after a timelock. The above is supposed to illustrate the type of structure which could be required when opening and closing lightning network channels)
Based on the above design, it can be assumed that only one spending condition will need to be revealed. For example, to spend the output, all the signers need to do is provide one Schnorr multi-signature and the hash at the top of the right hand side of the Merkle tree (Hash (1 & 2)). Therefore despite the existence of a Merkle tree, in the majority of cases, where everything goes as planned, only a single signature and 32-byte hash is required. More concisely, in order to verify a script, you need to prove that it is part of the Merkle tree by revealing other branch hashes.
However, the disadvantage of this structure is that even in normal optimal circumstances, when the single key and script on the top left of the Merkle tree is provided, one still needs to publish another hash to the blockchain (Hash (1 & 2) in the above diagram), using up 32 bytes of data. This weakness also reduces privacy, since third parties can always determine if more complex spending conditions exist, as the top branch of the Merkle tree is always visible.
As far as we can tell, the origins of the Taproot idea are from an email from Bitcoin developer Gregory Maxwell in January 2018. Taproot is similar in construction to MAST, except at the top of the Merkle tree. In the case of Taproot, in the cooperative or normal scenario, there is an option for only a single public key and single signature to be published, without the need to publish evidence of the existence of a Merkle tree. An illustration of the Taproot transaction structure is provided below.
Graphical illustration of Taproot spending conditions
(Source: BitMEX Research)
(Notes: The diagram attempts to illustrate the same spending criteria as the MAST diagram above)
The tweaked public key on the left (or address) can be calculated from the original public key and the Merkel root hash. In the event of a normal or cooperative payment, on redemption, the original public key is not required to be onchain and the existence of the Merkle tree is not revealed, all that needs to be published is a single signature. In the event of a lack of cooperation or abnormal redemption, the original public key is revealed along with information about the Merkle tree.
The benefits of Taproot compared to the original MAST structure are clear, in the cooperative case, one is no longer required to include an extra 32-byte hash in the blockchain or the script itself, improving efficiency. In addition to this, the transactions looks “normal”, just a payment with a public key and signature, the existence of the other spending conditions do not need to be revealed. This is a large privacy benefit, for example when opening a lightning channel or even doing a cooperative lightning channel closure, to an external third party observer, the transaction would look exactly like a regular spend of Bitcoin. The transaction could be structured such that only in an uncooperative lightning channel closure would the existence of the Merkle tree need to be revealed. The more different types of transactions look the same, the better it is for privacy, as third parties may be less able to determine which types of transactions are occurring and establish the flow of funds. A long term objective from some of the Bitcoin developers may be to ensure that, no matter what type of transaction is occurring, at least in the so-called cooperative cases, all transactions look the same.
The confusion over Signature aggregation
The potential scalability benefits of reducing the number of signatures needed on the blockchain are large and therefore the concept tends to generate a lot of excitement. Schnorr signatures do provide the capability to aggregate signatures in multi-signature transactions, which should be a significant benefit to Bitcoin. However, the inclusion of this and the existence of other signature aggregation related ideas, has lead to some unrealistic expectations about the potential benefits, at least with respect to this upgrade proposal. As far as we can tell, for this particular upgrade proposal, the only aggregation benefits are in the form of joining signatures in multi-signature schemes, not for multiple inputs or multiple transactions.
Summary table of signature aggregation ideas
Included in softfork proposal
Combined public key and signatures in multi-signature transactions – Included as part of Schnorr
Joint signature for multiple inputs in a transaction
Joint signature for multiple inputs in multiple transactions (Grin coin has some capabilities in this area, using Mimblewimble)
(Source: BitMEX Research)
In our view, the benefits associated with this softfork are not likely to be controversial. This softfork appears to be a win-win-win for capability, scalability and privacy. The largest area of contention is likely to be the absence of the inclusion of other ideas or arguments over why to do it this particular way.
That being said, many are likely to be excited about the potential benefits of these upgrades and keen to see these activated on the network as fast as possible. However, when it comes to Bitcoin, and in particular changes to consensus rules, the need for patience cannot be overstated.
Abstract: On 18th April 2019, the BitMEX Research Bitcoin Cash SV node experienced 2 block re-organisations. First a 3 block re-organisation, followed by a 6 block re-organisation. In this brief piece, we provide data and graphics related to the temporary chainsplit. The chainsplit appears to be caused by large blocks which took too long to propagate, rather than consensus related issues. Our analysis shows there were no double spends related to the split.
Chainsplit diagram – 18 April 2019
Source: BitMEX Research Notes: The above image indicates there were two valid competing chains and a non-consensus split occurred at block 578,639. Our node followed the chain on the left until block 578,642, then it jumped over to the right. About an hour later, it jumped back over to the left hand side. The chain on the left continued, while the chain on the right was eventually abandoned.
Chainsplit transaction data
Number of transactions
Main chain (within 6 blocks)
Overlap (within 6 blocks)
Eventual double spends
Source: BitMEX Research
Based on our analysis of the transactions, all the TXIDs from the forked chain (on the right), eventually made it back into the main chain, with the obvious exception of the coinbase transactions. Therefore, it is our belief that no double spends occurred in relation to this incident.
Timestamps of the blocks related to the split – 18 April 2019
If one is interested, we have provided the above table which discloses all the relevant details of the blocks related to the chainsplit, including:
The block timestamps
The local clock timestamps
The block hashes
The block sizes
The total accumulated PoW up to each block
With the above details one can follow what occurred in relation to the chainsplit and create a timeline.
Conclusion Our primary motivation for providing this information and analysis is not driven by an interest in Bitcoin Cash SV, but instead a desire to develop systems to analyse and detect these type of events on the Bitcoin network. Systems are being developed on our website, https://forkmonitor.info, to help detect chainsplits, caused either by poor block propagation or consensus related issues. This event on Bitcoin Cash SV is good practice for us.
As for Bitcoin Cash SV, the block sizes were particularly large during the period of the re-organisations. On the forked chain, the last two blocks were 128MB and 107MB respectively. On the main chain many of the blocks were over 50MB. Therefore, in our view, it is likely the large sizes of the blocks were the root cause of the re-organisations, as miners couldn’t propagate and verify these large blocks fast enough, before other blocks on different chains were found.
As for the implications this has on Bitcoin Cash SV, we have no comment. We will leave that to others.
Abstract: BitMEX Research examines the market dynamics of Lightning network routing fees and the financial incentives for Lightning node operators to provide liquidity. We identify the interrelationship and balance between Lightning routing fees and investment returns for channel liquidity providers, as a major challenge for the network, rather than the computer science aspects of the routing problem. We conclude that if the Lightning network scales, at least in theory, conditions in wider financial markets, such as changing interest rates and investor sentiment may impact the market for Lightning network fees. However, regardless of the prevailing economic conditions, we are of the view that in the long term, competition will be the key driver of prices. Low barriers to entry into the market could mean the balance favours users and low fees, rather than investment returns for liquidity providers.
We first wrote about the Lightning network back in January 2018, when it was mostly theoretical. Today, as the Lightning network transitions from abstract to experimental, we felt it was time to take another look. The primary focus of this report is to analyse the Lightning network from a financial and investment perspective, notably with respect to fees and the incentives for Lightning network providers. We will not examine other aspects of the technology.
The routing problem
Critics of the Lightning network often point to routing as a major problem, typically making claims like its “an unsolved problem in computer science”. In general, we do not really agree with this characterization of the routing problem and do not see the computer science of routing to be a major challenge, finding paths between channels to make payments may be relatively straightforward and similar to other P2P networks, such as Bitcoin.
However, what we do think its a major challenge is the interaction or balance between the financial and economic aspects of liquidity provision and payment routing. Lightning network node operators need to be incentivised by routing fees to provide sufficient liquidity, such that payments can be made smoothly. Liquidity needs to be allocated specifically to the channels where there is demand and identifying these channels may be challenging, especially when new merchants enter the network. This balance between ensuring the network has low fees for users, while also ensuring fees are high enough to incentivise liquidity providers, is likely to be a significant issue. As we explain further in this article, the magnitude of this problem and the fee rates at which the market clears, may depend on economic conditions.
Lightning fee market dynamics
For onchain Bitcoin transactions, users (or their wallets) specify the fee for each transaction when making a payment and then miners attempt to produce blocks by selecting higher fee transactions per unit block weight, in order to maximise fee revenue. In contrast, Lightning currently appears to work the other way around, routing node operators set the fee and then users select a path for their payment, selecting channels in order to minimise fees. With Lightning, suppliers initially set fees rather than users. Lightning may therefore offer a superior fee architecture, as suppliers are providing a specialised service and it is more suitable that suppliers compete with each other over fee rates, rather than ordinary users, where the priority should be on simplicity.
In Lightning there are two types of routing fees node operators must specify, a base fee and a fee rate.
Two types of Lightning network fees
A fixed fee charged each time a payment is routed through the channel
This is expressed in thousandths of a Satoshi.
For example a base fee of 1,000 is 1 satoshi per transaction.
A percentage fee charged on the value of the payment
This is expressed in millionths of a Satoshi transferred.
For example a fee rate of 1,000 is, 1,000/1,000,000, which is 0.1% of the value transferred through the channel. Equivalent to 10bps.
In order to provide liquidity for routing payments and to earn fee income, Lightning node operators need to lock up capital (Bitcoin) inside payment channels.
Two types of channel capacity
Inbound liquidity, are funds inside the node’s payment channels which can be used to receive incoming payments.
These funds are owned by other participants in the Lightning network.
If the payment channels are closed, these funds will not return to the node operator.
An inbound balance is created in one of two ways:
* When another network participant opens a payment channel with the node
* When the node operator makes a payment via an existing channel
Outbound liquidity, are funds inside the node’s payment channels which can be used to make outbound payments.
These funds are owned by the node operator and part of their investment capital. The node operator may consider the opportunity costs of other investments, while considering the total outbound balance.
If the payment channels are closed, these funds will return to the node operator.
An outbound balance is created in one of three ways:
* When the node operator opens a payment channel with another network node
* When the node operator receives a payment via an existing channel
* When payments are routed through the node and fees are received
Graphical illustration of a channel’s inbound and outbound capacity
(Source: Bitcoin Lightning Wallet) (Note: The orange balance is the inbound capacity, while the blue balance is the outbound capacity)
The operation of the Lightning fee market
Becoming a successful routing node is harder than one may think. At the time of writing, according to 1ml.com, there are 7,615 public Lightning nodes. However, it is likely that only a few hundred of these nodes are doing a good job providing liquidity, by managing the node, rebalancing channels and setting fees in an appropriate manner.
Node operators may need to:
Adjust both fee rates and the base fee, monitor the impact of the adjustments and calibrate for the optimal income maximising settings
Analyse the network and look for poorly connected Lightning nodes with high payment demand, such as a new merchant
Analyse the fee market, not just for the network as a whole, but the high demand low capacity routes you are targeting
Constantly monitor and rebalance ones’ channels, to ensure there is sufficient two way liquidity
Implement a custom backup solution for the latest channel states, to protect funds in the event that the node machine crashes
Currently, there are no automated systems capable of doing the above functions. If this does not change, specialist businesses may need to be setup to provide liquidity for the Lightning network. However, just as with liquidity, the challenges in overcoming these technical issues do not necessarily mean payments will become difficult or expensive. These technical challenges may simply adjust the equilibrium market fee rate. The more difficult these problems are to overcome, the higher the potential investment returns will be to channel operators and the greater the incentive will be to fix the problems. It will be demand that drives Lightning’s success, not the challenges for node operators.
In order for Lightning fee markets to work, node operators may need to adjust fees based on the competitive landscape, this could be based on algorithms or be a manual process, aimed at maximising fee income. In an attempt to emulate what may eventually become standard practise, BitMEX Research experimented with modifying the fee rate on one of our nodes over a three month period, as the below section reveals.
Fee rate experimentation
BitMEX Research decided to conduct a basic experiment to try and evaluate the state of the fee market, even in the Lightning network’s current nascent state. We set up a Lightning node and regularly changed the fee rate to attempt to determine which rates would maximise fee revenue, just as node operators may eventually be expected to do as the network scales.
Our basic non-scientific analysis from one node is illustrated in the scatter chart below. It appears to indicate that fee rates do currently have an impact on a lighting node’s fee income. The daily fee income appears to quickly accelerate as one increases the fee rate from 0 till around 0.1 bps. Once the fee is increased above this rate, average daily fee income appears to gradually decline. Therefore, based on this experiment, it appears as if the revenue maximising fee rate is around 0.1 bps, which is certainly very low when compared to other payment systems. However, of course, this is only the fee for one hop, a payment may have multiple hops. At the same time, the current Lightning fee market barely exists, indeed BitMEX Research may be one in only a handful of Lightning nodes that has significantly experimented with economic revenue maximising behaviour by changing fees. Once the network scales and other parties try to maximise revenue, fee market conditions are likely to be very different. This exercise should therefore only be considered as an illustrative experiment, rather than anything particularly revealing about lighting fee markets.
Lightning node daily fee income versus the feerate
(Source: BitMEX Research) (Lightning fee income data charts – notes and caveats: * Daily data from 31st December 2018 to 24th March 2019 * Data from one Lightning node * The base fee was 0 across the period * The investment return data excludes onchain Bitcoin transaction fees, when including the impact of fees all but the most optimal fee rate buckets would show a negative investment return * The data includes both weekdays and weekends, in general Lightning network traffic is significantly lower at weekends * The fee rate was changed every day at around 21:00 UTC. The fee rate was reduced each day and then jumped up to the top of the fee rate range after several days of declines, to begin the next fee rate downwards cycle. The reason for this was that some wallets (e.g. mobile wallets) did not always query the fee rate each time it attempted to route a payment through the node, therefore when increasing the fee rate, many payments would fail. For example, when opening a channel from a mobile wallet to the Lightning node, then increasing the fee rate and immediately attempting to make a payment, the payment often failed as the wallet attempted to pay with a fee which was too low. In our view, in order to Lightning network fee markets to work, node operators may need to regularly change fees and therefore wallets may need to query fee rates more often * Channel rebalancing occurred manually, once every two weeks. Approximately 30 minutes was spent on each occasion * The Lightning node was running LND and the software was updated to the master every two weeks * Approximately 30% of the channels (by value) were opened using the autopilot, the other 70% were opened manually * The investment return was calculated by taking the outbound channel capacity of the network each day, annualising the investment return based on the daily fee income and then calculating a simple average based on all the days with a fee rate inside the particular range * The data is based on one node only and its particular set of channels, the experience for other node operators may be very different * We tried to use our public node for this experiment, however the fee income was too sporadic, with some network participants regularly paying well above the advertised fee rates by considerable amounts, making the data unreliable * Unfortunately we needed to use a log scale for both axis. With respect to the fee rate we were unsure of which rates to charge, even which order of magnitude to set, therefore we tried a wide range of fees, from 0.0001% to 0.5% and a log scale was appropriate. At the same time, the daily fee income was highly volatile, ranging from 0 satoshis to over 3,000 satoshis. Therefore a log scale was deemed most appropriate. As the network develops and fee market intelligence improves, a linear scale may be more appropriate)
Fee incomes and investment returns
In addition to daily fee income, one can also consider the annualized investment return associated with running a lighting node and the various fee rates. This is calculated by annualising the daily fee income and dividing this number by the daily outbound liquidity.
The highest annualised investment investment return achieved in the experiment was 2.75%, whilst the highest fee bucket investment return was almost 1%. This seems like a reasonably attractive return for what should in theory be a relatively low risk investment, at least once the ability to backup lighting channels in real time becomes implemented. Existing Bitcoin investors could be tempted by these returns and provide liquidity to the Lightning network, or alternatively US dollar based investors could buy Bitcoin, hedge the Bitcoin price exposure using leverage and then attempt to earn Lightning network fee income.
Lightning node annualised investment return by fee bucket
(Source: BitMEX Research)
Of course, liquidity providers in the current Lightning network are not likely to be motivated by investment returns. Current node operators are likely hobbyists, with the overwhelming majority of node operators making losses when considering the onchain fees required to open and rebalance Lightning channels. Although this hobbyist based liquidity probably can sustain the network for a while, in order to meet the ambitious scale many have for the Lightning network, investors will need to be attracted by the potential investment returns.
Lightning network fees and economic conditions
A 1% investment yield may seem attractive in the current low yield environment, however the Lightning network may initially have difficulty attracting the right commercial liquidity providers. Investors in this space are typically looking for a high risk high return investment, which appears to be the opposite end of the spectrum for the relatively low risk low return investment on offer for Lightning liquidity providers. Therefore a new type of investor, one that fits this profile, may be needed.
If the Lightning network reaches a large scale, it is possible that the highly liquid investment product, with stable low risk returns, is sensitive to economic conditions.
Consider the following scenario:
The federal reserve base rate is 1.0%
Lightning node operators are typically earning an annualised investment yield of 1.5% on their outbound balance
Due to robust economic conditions and inflationary pressure, the federal reserve open market committee increase interest rates from 1% to 3%.
Due to the more attractive investment returns, Lightning network node operators withdraw capital from the Lightning network and purchase government bonds
Due to the lower levels of liquidity in the Lightning network, users are required to pay higher fees to route payments and the Lightning network becomes more expensive
However, if Lightning network liquidity is large enough for the above logic to apply, Lightning would have already been a tremendous success anyway.
The risk free rate of return
In some ways, if the Lightning network matures, one can even think of the investment returns from running a Lightning node as Bitcoin’s risk free rate of return, or at least a rate of return free from credit risk. In traditional finance this is often the rate investors earn by holding government bonds, where the government has a legal obligation to pay the principal and coupon and a means to create new money to pay the holders of the bonds, such that the risks are near zero. In theory, all other investment projects or loans in the economy should have a higher return than this risk free rate. The same could apply to Bitcoin, with Lightning node liquidity providers return rates being considered as the base rate within the Bitcoin ecosystem.
In the future, if most of the technical challenges involved in running nodes have been overcome and there are competitive fee setting algorithms, this Lightning network risk free rate could ultimately be determined by:
Conditions in wider financial markets – higher interest rates could mean a higher Lightning network risk free rate
The demand for Lighting network transactions – more demand or a higher velocity of money, should increase the Lightning network risk free rate
Whether specialist hedge funds and venture capital investors will have the same enthusiasm about becoming Lightning network liquidity providers, as they did for the “staking as a service” business model for proof of stake based systems in mid 2018 remains to be seen. While the investment returns for Lightning network liquidity providers do not yet look compelling, with the network in its formative stages, we do see potential merit in this business model.
In our view, the Lightning network can easily scale to many multiples of Bitcoin’s current onchain transaction volume without encountering any economic fee market cycles or issues, all based purely on hobbyist liquidity providers. However, if the network is to reach the scale many Lightning advocates hope, it will need to attract liquidity from yield hungry investors seeking to maximise risk adjusted investment returns. Should that occur, unfortunately the network may experience significant changes in fee market conditions as the investment climate changes over time.
However, it is relatively easy to set up a node, provide liquidity and try to earn fee income by undercutting your peers. Where the balance is ultimately struck between the operational channels of running nodes, the extent of liquidity provision and the investment returns, we obviously do not know. However, if we are forced to guess, based on the architecture and design of the Lightning network, we would say the system is somewhat rigged towards users and low fees, rather than liquidity providers.
Abstract: BitMEX Research is delighted to announce the launch of a new website to monitor the Ethereum network, Nodestats.org. The website connects to five different Ethereum nodes and collects data every five seconds. The main focus of the website is providing metrics related to the computational resources each Ethereum node requires. While analysing some of the metrics, we may have identified issues with respect to the integrity of the data reported by the nodes, which may be of concern to some Ethereum users. Nodestats.org was produced in collaboration with TokenAnalyst, who are BitMEX Research’s Ethereum network data and analysis partner.
(Screenshot of website as at 12 March 2019)
Nodestats.org compares the statistics of the two largest Ethereum node client implementations by overall adoption – Geth and Parity. Within these client implementations, Nodestats.org compares the performance of different node configurations – fast, full, and archive nodes.
To provide metrics comparing the computational efficiency of the different Ethereum implementations. For instance by comparing requirements related to:
To compare the resource requirements between running Ethereum node software and that of other coins, such as Bitcoin
To evaluate the strength of the Ethereum P2P network and transaction processing speed, by looking at metrics related to whether the nodes have processed blocks fast enough to be at the chain tip or whether poor block propagation results in nodes being out of sync for a significant proportion of the time
Nodestats.org began collecting data at the start of March 2019 and it is too early to draw any firm conclusions. However, we are saving the data and hope to analyse the long term trends at a later point. The Nodestats.org data is produced by querying our five Ethereum nodes or machines running the nodes, every five seconds (720 times per hour) and then storing the results in a database. Various rolling averages and other metrics produced from this data, are displayed on the Nodestats.org website.
Description of the Nodestats metrics
% of time in sync
This represents the percentage of time the node has verified and downloaded all the block data, up what the P2P network is informing the node is the chain tip.
The hourly metric is calculated by determining if the node is at the tip every 5 seconds, which should be 720 queries per hour. The proportion of these queries where the node says it is at the tip is the reported metric.
This field is based on the web3 “isSyncing” field, which we believe uses the highest block the node has seen, the “highestBlock” field, to determine if the node is behind what its peers regard as the highest block ever seen.
Nodes typically report they are at the tip around 99.8% of the time, which means that in only around 1 of the 720 hourly queries are the nodes not at the chain tip.
The only exception here is the Ethereum Parity full node, which we talk about later in this report.
We believe the data integrity of this metric is poor, for instance in the case of the Parity full node the integrity of the information provided is weak, as we explain later in this report. Going forwards we aim to establish a more effective way of calculating this metric.
% of time on conflicting chain
This represents the percentage of time the node is following a different or conflicting chain to the node opposite it on the website.
This is determined by storing all the block hashes in our database, if the nodes have a different block hash at the same height, they are considered to be on different chains.
Typically Nodestats.org is not able to identify times when the clients are following different chains. As such this metric is normally 0%. (i.e. 0 times out of 720 in a one hour period)
This represents the average percentage utilization of the machine’s CPU resources.
All the machines Nodestats.org are using have the “Xeon(R) CPU E5-2686 @ 2.30GHz” processing unit with two cores. The exception to this is the archive node, which has 16 cores.
All the nodes are using the AWS “i3.large” machines, with the exception of the archive node, which is running “i3.4xlarge”.
Generally speaking, CPU usage tends to be between 0.01% and 1.0%. Parity tends to be towards the 1% level, while Geth appears to use less CPU power.
Geth’s CPU usage appears less stable than Parity’s, with Geth’s CPU demand occasionally spiking to around the 1% level.
Nodestats.org takes a reading from the machines every 5 seconds, related to how much memory is being utilized by the Ethereum client.
All the machines Nodestats.org are using have 14GB of Ram, with the exception of the archive node, which is a 120GB of Ram machine.
Generally speaking, however much RAM is available, the nodes use up the overwhelming majority of it (e.g. over 95%).
The memory demands of the clients appear to be reasonably stable.
The node provides Nodestats.org with the number of network peers, every 5 seconds.
Parity tends to have around 450 peers, while Geth only has around 8.
Geth’s peer count is more volatile than Parity, as it appears to occasionally fall to around 6.
Nodestats.org takes a reading from the machine every 5 seconds, related to the total network upstream bandwidth of the server.
Parity, which has more peers, tends to use over 100KB/s of bandwidth (in each direction). In contrast Geth tends to only use around 4KB/s of bandwidth.
Geth’s bandwidth demand tends to be more volatile than Parity, with occasional spikes to around 60KB/s.
Nodestats.org takes a reading from the machine every 5 seconds, related to the total network downstream bandwidth of the server.
Chain data size
This metric represents the total data utilized by all the directories dedicated to the client.
Unlike the other metrics, the disclosed figure is the absolute value, not a rolling 1 hour average.
Currently, Parity requires around 180GB, Geth uses just under 200GB, and the full archive node uses up 2.36TB of data.
The Parity full node is still syncing
The Parity full node was started on 1 March 2019, at the time of writing (12 March 2019) it has still not fully synced with the Ethereum chain. The client is around 450,000 blocks behind, and based on its current trajectory, it should catch up with the main chain tip in a few days. Due to the slow initial sync, the “% of time in sync” metric is shown as near 0%, as the client is never in sync.
The Ethereum Parity Full node machine has the following specifications:
Dual Core 2.3GHz
14GB of RAM
10 Gb/s internet connection
The fact that a machine with the above specification takes over 12 days to sync may indicate that it is the initial sync issues could be a greater concern for the Ethereum network than post sync issues, such as block propagation. While the slow initial sync is a potential problem, at least for this system setup, Ethereum has not yet reached a point where the node cannot catch up, as the sync is faster than the rate of blockchain growth.
Data integrity issues
The Parity full node also sometimes reports that it is in sync, despite being several hundred thousand blocks behind the chain tip. For instance in the screenshot at the start of this piece, the website reports that the node is fully synced 0.02% of the time, indicating the node falsely thought it was at the tip for some periods of time.
As the chart generated from the Parity full node logs below illustrates, the highest block seen on the network figure, in blue, appears potentially incorrect. The highest block number seen on the network figure, sometimes falls in value as time progresses and has remained consistently well behind the actual chain tip (shown in green). On occasion this potentially buggy figure fell towards the height of the verified chain (orange) and our website incorrectly reports the node as in sync. This may be of concern to some Ethereum users, since the Parity full node has many connections to the network, therefore this may be a bug.
Ethereum Parity Full Node Block Height Data – 11 and 12 March 2019 (UTC)
(Source: Ethereum Parity full node logs)
This potential bug could undermine this whole metric for our website, even for the other nodes, as the highest tip seen field may not function appropriately and our figures may be inaccurate. However, we continue to include this metric, since the Nodestats.org website displays the data reported by the nodes, regardless of our view on the integrity of the data. We may look to implement our own improved metric in the future.
One could argue the impact of this potential bug could be severe in some limited circumstances, if exploited by an attacker in the right way. For example a user could accept an incoming payment or smart contract execution as verified, while their node claims to be at the network chain tip. However, the client may not really be at the chain tip and an attacker could exploit this to trick the recipient into delivering a good or service. The attacker would need to double spend at a height the vulnerable node wrongly thought was the chain tip, which could have a lower proof of work requirement than the main chain tip. Although successful execution of this attack is highly unlikely and users are not likely to be using the highest seen block feature anyway.
Like its sister website, Forkmonitor.info, Nodestats.org is very much a work in progress. Along with TokenAnalyst, over the coming months and years, we plan to add more features, such as:
Improving the integrity of the data, by being less reliant on what the nodes report and developing our own calculation methodologies
Charts & tools for analysing longer term trends
Improved granularity of the data
Fork detection systems
Data related to other peers
For now, Nodestats.org provides a useful tool to assess the approximate system requirements for running Ethereum nodes. At at a very basic level, it also provides mechanisms to assess the reliability of the Ethereum network and its various software implementations. However, we accept that the “% of time in sync” metric may not be reliable, but it does highlight a potential issue.