Abstract: We look at the recent elevated level of hashrate volatility on the Bitcoin Cash network. We note that the apparent cyclical nature of the oscillations may indicate a form of manipulation, although we found no direct evidence of such behavior. We conclude that there are no easy solutions to Bitcoin Cash’s hashrate volatility issue, on the other hand the negative impact on the usability of the coin has been marginal so far. The best course of action in the short term may be to be patient and research possible solutions, unless a particular cause becomes more apparent.
Bitcoin Cash Hashrate Estimate- (8 hour rolling average) – PH/s
(Source: BitMEX Research) (Notes: Eight hour periods were chosen to maximise the visual impact of the apparent hashrate oscillations. Hashrate data calculated by using the difficulty and block timestamps)
Bitcoin Cash Hashrate Volatility Concerns
Recently some have expressed concerns about the apparent unusually high level of volatility in the Bitcoin Cash hashrate, causing larger than expected variances in block times. As the above chart indicates, the volatility in hashrate appears to have picked up at around the start of October 2019. It is important to consider that we calculated the hashrate based on block times, which is a random and volatile process anyway, therefore determining if short term changes in the hashrate are random or caused by actual changes can be challenging. However, in our view the data is reasonably compelling. This volatility may have made the network marginally less reliable for payments in some periods, although its not a major issue, in our view.
For the month of October 2019 we conducted a basic analysis of Bitcoin Cash’s difficulty, block timestamps and the times in which our node received the Bitcoin Cash blocks.
As the below chart illustrates, Bitcoin Cash does indeed appear to have more volatile time gaps between blocks when compared to Bitcoin.
Time interval between blocks – 50 block rolling average (Minutes) – October 2019
(Source: BitMEX Research) (Notes: The x-axis is block heights for Bitcoin Cash, while Bitcoin is added for the same 29 days ending 29th October 2019)
Not only does the Bitcoin Cash time gap between blocks appear more volatile than Bitcoin, with higher peaks and lower troughs, but it also appears to be slightly less random, with more regular peaks and troughs. There may be some form of cyclicality in the data, which might imply manipulation, although we found no direct evidence of this. On the other hand, the random nature of the data means that the chart may look cyclical, but it could be an illusion. Please note, the Bitcoin Cash difficulty is adjusted on a per block basis and therefore the adjustment algorithm should not cause such cyclicality. Please also note that a 50 block rolling period was chosen to maximize the size of the peaks and troughs, therefore the chart may exaggerate the issue slightly.
The Inconclusive Hunt For Evidence Of Causation
In the table below, we analysed the average difficulty of each block for each mining pool, by calculating the difficulty at each block height. We were trying to determine if one mining pool had been successful in some strategy aimed at achieving a lower average difficulty than its peers. The analysis was inconclusive and the unknown miner(s) achieved an average difficulty reasonably close to the total average. However, it is possible a more detailed analysis of the data could find something more interesting.
Bitcoin Cash – October 2019 Mining Pool Statistics
Number of Blocks Mined
Average Difficulty When Each Block Was Mined
Average Time Gap Between Timestamp & Our Local Clock (Minutes)
(Source: BitMEX Research)
We also analysed the time gaps between the block timestamps and the time our local system clock says it received the block, again looking for discrepancies between pools for evidence of manipulation. Perhaps exploiting the potential 8.3% vulnerability we mentioned in yesterday’s piece. Again, no such direct evidence was found, with the unknown pool bang on the average, with the timestamp 0.48 minutes before our system clock.
The following charts continue to look at the timestamp gap to our local clock, and compare Bitcoin and Bitcoin Cash, in an attempt to visually identify any irregularities. What they appear to indicate is that Bitcoin timestamps are on average more consistent with our local clock and the time gaps are less volatile. This may merely suggest that Bitcoin has a stronger peer-to-peer network than Bitcoin Cash, with faster block propagation, rather than any nefarious manipulation of the timestamps.
Bitcoin Cash – Average Time Gap Between Timestamp & Our Local Node Clock (Minutes) – October 2019 – (Y-axis cut to compare with Bitcoin)
(Source: BitMEX Research)
(Notes: Orange line is 50 block moving average)
Bitcoin – Average Time Gap Between Timestamp & Our Local Node Clock (Minutes) – October 2019
(Source: BitMEX Research)
(Notes: Orange line is 50 block moving average. The x-axis is the month of October 2019)
We were unable to find any evidence of timestamp manipulation or other nefarious mining strategies. Bitcoin Cash is a minority hashrate coin and therefore this is likely to make the hashrate more volatile to some extent. Perhaps the apparent cyclicality is caused by lags in automated systems designed to mine the most profitable coins or some other more benign factor.
Fixing the potential problems with Bitcoin Cash’s hashrate volatility may require a hardfork and the coin is already scheduled to have a hardfork upgrade in a few days time, which does not include a fix for the above issue. Any fix is likely to require considerable development work and analysis/discussion before it is rolled out. Therefore a fix is unlikely in the short term. However, the issue may not be urgent.
As for how Bitcoin Cash may consider fixing the problem, we suggest consideration of the following proposals:
Merged Mining – Enabling merge mining with Bitcoin, as we explained back in November 2017, could add stability to Bitcoin Cash mining, however some in the Bitcoin Cash the community may still be unwilling to adopt this due to the somewhat unfriendly attitude towards Bitcoin. In our view, this anger will dissipate over time.
Adopt Bitcoin’s Two Week Adjustment Period – Bitcoin Cash could return to Bitcoin’s fixed two week window difficulty adjustment system. This is perhaps one of the more simple solutions, although it will not completely solve the problem.
Reduce block times – Bitcoin Cash could re-enable the 1MB blocksize limit and reduce the block time to around 1 minute instead, although this may not fix hashrate volatility, due to the lower block time gaps, blocks would appear reasonably regularly anyway. This strategy more directly aligns with the Bitcoin Cash community’s objective of higher onchain throughput and improved usability without waiting so long for confirmations, than blocksize limit increases would, in our view.
As of now, the elevated hashrate volatility issue does not appear urgent and has only persisted for one month. The apparent sudden increase in volatility in October 2019 remains a mystery, at least to us. However, if this continues or a particular cause becomes apparent, in our view, a fix may be necessary.
Abstract: We examine two of Bitcoin’s little-known rules, designed to stop nefarious miners from manipulating the block timestamps and achieving unfairly high mining rewards. We discuss why constants such as the two-hour MAX_FUTURE_BLOCK_TIME value may have been chosen and how this value may have particular implications for Bitcoin Cash. We conclude that Bitcoin’s time protection rules appear reasonably effective, an impressive feat, considering the lack of a functioning network when the rules were implemented.
One might think that time is not an important consideration for the Bitcoin network, since the blocks already have a sequential order, as each block references the hash of the previous block. Bitcoin blocks also contain transactions (inputs, outputs and values), a Merkle tree leading to the block header and the block hash itself, used for proof of work. On the surface this may seem sufficient for a transaction and consensus system. However, there is the matter of the difficulty adjustment – if too many miners join the network, the block times could become too fast, or if too many miners leave, the block times could become too slow, making the network unreliable. To resolve this problem, every two weeks the mining difficulty adjusts to aim for a target time between blocks of ten minutes. Unfortunately, in order to calculate the two-week period, the concept of time needs to be introduced to the blockchain and be part of the consensus system. Therefore blocks are required to contain a timestamp, and one can think of Bitcoin as the world’s first distributed electronic clock.
Block Timestamp Security Rules
When a Bitcoin block is produced there are essentially two times involved:
The timestamp in the block header, put there by the miner
The actual time the block was produced.
Of course, these two times should be pretty much the same. After all, surely the miners have reasonably accurate clocks and why would they lie about the time?
As it happens, there are some incentives for miners to lie about the time. For instance, nefarious miners could add a timestamp which is in the future. For example, if a block took 10 minutes to produce, miners could claim it took them 15 minutes, by adding a timestamp 5 minutes into the future. If this pattern of adding 5 minutes is continued throughout a two week difficulty adjustment period, it would look like the average block time was 15 minutes, when in reality it was shorter than this. The difficulty could then adjust downwards in the next period, increasing mining revenue due to faster block times. Of course, the problem with this approach is that the Bitcoin clock continues to move further and further out of line with the real time.
To resolve or mitigate the above issue, Bitcoin has two mechanisms to protect against miners manipulating the timestamp:
Median Past Time (MPT) Rule – The timestamp must be further forwards than the median of the last eleven blocks. The median of eleven blocks implies six blocks could be re-organised and time would still not move backwards, which one can argue is reasonably consistent with the example, provided in Meni Rosenfeld’s 2012 paper, that six confirmations are necessary to decrease the probability of success below 0.1%, for an attacker with 10% of the network hashrate.
Future Block Time Rule – The timestamp cannot be more than 2 hours in the future based on the MAX_FUTURE_BLOCK_TIME constant, relative to the median time from the node’s peers. The maximum allowed gap between the time provided by the nodes and the local system clock is 90 minutes, another safeguard. It should be noted that unlike the MPT rule above, this is not a full consensus rule. Blocks with a timestamp too far in the future are not invalid, they can become valid as time moves forwards.
Rule number one ensures that the blockchain continues to move forwards in time and rule number two ensures that the chain does not move too far forwards. These time protection rules are not perfect, for example miners could still move the timestamps forward by producing timestamps in the future, within a two week period, however the impact of this would be limited.
As the above ratio illustrates, since two hours is only a small fraction of two weeks, the impact this manipulation has on network reliability and mining profitability may be limited. This is the equivalent of a reduction in the time between blocks from 10 minutes to 9 minutes and 54 seconds, in the two weeks after the difficulty adjustment. In addition to this, it is only a one-off change, as once the two-hour time shift has occurred, it cannot occur again, without first going backwards. At the same time, the miner may want to include a margin of safety before shifting forwards two hours, to reduce the risk of the block being rejected by the network.
These rules have proven reasonably effective in preventing miners from manipulating Bitcoin’s timestamps in nefarious ways, as far as we can tell.
Bitcoin Cash’s Theoretical Block Time Problems
As we first mentioned back in September 2017, Bitcoin Cash is an alternative coin which split off from Bitcoin in August 2017, the primary objective of the coin was to increase the blocksize limit. One of the concerns the Bitcoin Cash developers had at the time was that not many miners would mine Bitcoin Cash, and therefore the time gap between blocks could be too large. As a result something called the Emergency Difficulty Adjustment (EDA) was implemented to alleviate this concern. We will not go into the details here, but suffice to say this mechanism was highly complex and turned out to be fundamentally flawed. This algorithm meant that if a certain number of blocks were not found within a certain time period, the difficulty would reduce. The policy was particularly aggressive, as it meant that the longer the time gaps between blocks, the larger the downward difficulty adjustment. Miners manipulated the network by leaving large time gaps on purpose, resulting in large difficulty changes, followed by low-difficulty periods with blocks being produced at a very high frequency. The network then became unreliable.
As a result of this flaw, more Bitcoin Cash blocks were produced than expected and miner revenue increased during this period. Bitcoin Cash built a c.5,000 block lead over Bitcoin, a lead it still maintains to this day. A couple of months later, in November 2017, a fix was eventually rolled out. The EDA was removed and replaced with a new difficulty adjustment system, a more simple rolling 24-hour system. However, this is still different to Bitcoin’s two-week window system. Bitcoin Cash’s system is more dynamic and faster to adjust. While this means Bitcoin Cash may have a more volatile difficulty in the short term, the coin is faster to adjust to any changes, while any block time discrepancies in Bitcoin may take longer to correct.
Overview of Bitcoin and Bitcoin Cash’s Difficulty Adjustment Systems
Every 2 weeks
Less likely to suffer from block time discrepancies
Any discrepancies take longer to resolve
More likely to suffer from block time discrepancies
Any discrepancies are resolved faster
(Source: BitMEX Research)
One thing that many may have overlooked in Bitcoin Cash’s new difficulty adjustment algorithm is its interrelationship with the two-hour time protection rule. As far as we are aware, Bitcoin Cash has retained the 2 hour constant.
A two-hour period is now 8.3% of the calculation period. This is the equivalent of a reduction in the time between blocks from 10 minutes to 9 minutes and 10 seconds. This does appear to be potentially significant and could result in changes to miner profitability, if exploited. Bitcoin Cash may therefore be somewhat vulnerable to miners manipulating timestamps, or at least more vulnerable than Bitcoin.
On the other hand, although Bitcoin Cash may be more vulnerable to miner timestamp manipulation attacks than Bitcoin, any issues will be resolved faster.
The apparent vulnerability with Bitcoin Cash’s time protection rule, perhaps unexploited, illustrates how well thought out Bitcoin’s time protection rules were. As far as we know, these time protection rules existed since Bitcoin launched in 2009. When designing the system, Satoshi had to innovate at least three layers deep:
Proof of work system → Difficulty adjustment system → Robust time protection rules
While this may not seem especially ingenious to us now, we have had almost 11 years’ experience with such systems. That Satoshi had thought all this through before any such network existed is quite remarkable, in our view.
Abstract:ForkMonitor has now implemented unexpected inflation detection and warning systems for Bitcoin. The block reward is currently 12.5 bitcoin, which means that no more than 12.5 new bitcoin should be created each block. Some of the ForkMonitor nodes now calculate the total coin supply each block, using the gettxoutsetinfoRPC call. If the total coin supply increases by more than 12.5 bitcoin, warnings systems are initiated. This service potentially provides additional assurances to network participants about the supply of Bitcoin at any given time.
ForkMonitor has recently added a new feature, unexpected inflation detection. This feature has been added for Bitcoin and Testnet Bitcoin. The system periodically checks the total coin supply by summing up all the UTXO values. If the value is unexpectedly large, warnings are activated. Bitcoin nodes are already supposed to check the coin supply, however this occurs by only checking that no unauthorised coins are created in each individual transaction and there is no macro total supply check. Therefore the ForkMonitor service could provide an additional layer of security and assurance for Bitcoin users, as well as an early warning system which could encourage people to run these checks on their own nodes if an issue is detected.
If the inflation is in line with expectations, a green tick is displayed on the website. However, if unexpected inflation occurs, a red cross is displayed alongside other warnings.
Illustration of unexpected inflation detected by Bitcoin Core 0.18.1
Please subscribe to the feeds, to be altered in the event of unexpected Bitcoin inflation.
Coin Supply Checking Mechanisms
The systems plan to check the inflation using the following methods:
Coin supply change from the previous block – After each Bitcoin block is produced, the system checks the total coin supply and stores the figure in a database. As each new block is produced, the summation is repeated and the total coin supply is subtracted from the previous figure. If the change is higher than the allowed block reward (12.5 bitcoins today, 6.25 bitcoins from around May 2020, etc.), then the warnings are initiated.
Consistency across multiple node versions – In addition, the system will also check that the total bitcoin supply is consistent at each block height for all the nodes participating in the inflation check (which is illustrated on the ForkMonitor website).
One of the main challenges we faced when implementing this inflation check feature was that it took considerable time for Bitcoin Core to run the gettxoutsetinfocall, typically around 2 minutes. This created several implementation challenges for ForkMonitor, such as what to display in this two minute period or what happens when a block is found while the calculation is occurring. For example, the maximum rate at which the inflation check can move forwards is one block every two minutes; if many blocks are found in a row, with smaller than two minute intervals between them, our check can be ineffective for a while.
Gettxoutsetinfo RPC call – Bitcoin’s supply of approximately 18 million is illustrated
(Source: Output from Bitcoin Core 0.18.0 “Gettxoutsetinfo” call)
Others are aware of these issues, as Bitcoin developer Fabian Jahr recently put it:
[Thegettxoutsetinfocall] does not have a sufficient user experience, you call it and it actually takes several minutes to respond and there is no feedback
In 2017 Bitcoin developer Pieter Wuille posted to the Bitcoin development mailing list, a potential improvement, which he says could make this Remote Procedure Call (RPC) call faster.
Replacement for Bitcoin Core’s gettxoutsetinfo RPC’s hash computation. This currently requires minutes of I/O and CPU, as it serializes and hashes the entire UTXO set. A rolling set hash would make this instant, making the whole RPC much more usable for sanity checking
Based on the above idea, Fabian recently indicated he may work on implementing this potential fix, in an attempt to improve this RPC call. If this improvement is implemented, it would certainly be helpful for ForkMonitor.
Bitcoin’s 2018 Inflation Bug (CVE-2018-17144)
ForkMonitor was very much inspired by the events of September 2018, when it emerged that Bitcoin Core had a bug which would enable miners to create coins out of nowhere in addition to the normal block reward. This bug affected versions of Bitcoin Core spanning from 0.14.0 to 0.16.2, before the fixes were released. (0.14.X nodes merely crashed while later nodes would have accepted the blocks with the unexpected inflation).
A successful exploitation of this bug could have had catastrophic consequences for the network, for example Bitcoin’s supply could have inflated above 21 million or a large rollback may have occurred, undermining the security many users and businesses depend on.
ForkMonitor was launched to mitigate these risks. If such a bug existed today, our systems should be able to detect it in three ways:
ForkMonitor runs multiple versions of Bitcoin Core, spanning many years of development. If a newly-introduced bug results in unexpected inflation or an unauthorized spend, the older nodes should detect this and mark the block as invalid, triggering the warning systems.
The website also runs independent implementations of Bitcoin, such as bcoin, btcd and Libbitcoin. If Bitcoin Core has a bug which allows unexpected inflation or an unauthorized spend, as long as the same bug wasn’t independently implemented, these other clients should mark the block as invalid, triggering the warning systems.
As of October 2019, ForkMonitor also directly checks the total coin supply of each block. In the event of unexpected inflation, even in the unlikely scenario that all our nodes mark the block as valid, the warning systems will still be triggered. The inflation checking system is also helpful even if nodes do mark the block as invalid, as it can help users determine why this was the case in a timely manner.
As we explained in our October 2018 piece, Competing with Bitcoin Core, there are advantages and disadvantages of competing implementations and in particular independent implementations. One key advantage of independent implementations that we mentioned is that there could be a bug in Bitcoin Core or the reference implementation which is not present in the independent implementations.
For the above reason, we are keen to add one of the three independent implementations (bcoin, btcd and Libbitcoin) into the total coin supply inflation checking system as soon as possible. The method of calculating total coin supply used by these implementations may be independent from that used by Bitcoin Core, which should provide extra reassurance that the number is correct.
This new service may not solve all potential problems with regards to detecting unexpected inflation. For example there could be a bug in the gettxoutsetinfo check. In addition to this, the various mechanisms to check for unexpected inflation and block validity may not be truly independent from each other. Even the independent Bitcoin implementations may have inadvertently copied a bug or erroneous concept from Bitcoin Core. However, we believe this macro inflation checking service is potentially a useful addition to network security.
As a reminder, the ForkMonitor website is open source, therefore please feel free to contribute, fork the project or reproduce the website.
2015年12月，新当选的两名董事会成员 Oliver 和 Jim 被其他董事会成员免职，理由是他们对基金会未来的最佳发展方式存在分歧。Oliver 和 Jim 近期在个人成员的竞争性选举中获得成功，积累了相当大的民主授权。与此同时，Elizabeth 和 Meyer 的两年任期已经届满，而 Brock 和 Bobby 是由行业而非个人选出的。因此，从个人成员的角度来看，Oliver 和 Jim 是仅有的两名负有重大使命的董事会成员，但他们却被免职。随后基金会违反章程，决定不再进行任何董事会选举。正如执行董事 Bruce Fenton 所说：
我们认为这种逻辑很难自圆其说，因为很多问题是由于董事会对个人成员明显缺乏责任感所导致的。Elizabeth Ploshey 是唯一一位由在董事会有效工作过一段时间的个人成员所选出的董事会成员。如果该基金会真的想重整旗鼓，本可以让 Oliver 和 Jim 复职，并允许进一步的选举来替换其他本应当离开的董事会成员。但是恰恰相反，基金会甚至决定与成员拉开距离，避免这种问责可能带来的挑战，结果也就失去了它剩余的全部的合法性。
Abstract: In this piece we look back at the history of Bitcoin, focusing in on “The Bitcoin Foundation”, once one of the most prominent organisations in the ecosystem. We look at Foundation’s origins and then examine its failings with respect to its governance, transparency and finances, which ultimately led to a total loss of legitimacy within the Bitcoin community. We conclude that an all-encompassing Foundation was never likely to have been a good idea given the high governance and transparency standards of some in the community, and that a constant stream of scandals damaged the Foundation’s brand to such an extent that its duties had to be carried out by other organisations.
(Screenshot of the Bitcoin Foundation’s website and logo in 2013)
The Foundation’s Origins
Following on from our July 2018 piece, which took us back to shenanigans and incompetence at MtGox in 2011, this second look at Bitcoin’s scandal-rich history takes us back to July 2012, when The Bitcoin Foundation was founded. The Foundation had seven founding members, or six if you exclude Satoshi, who was oddly included as a founding member.
Bitcoin Foundation Founding Members
Gavin Andresen, Bitcoin Developer
Peter Vessenes, CEO of CoinLab
Charles Shrem, CEO of BitInstant
Roger Ver, CEO of MemoryDealers
Patrick Murck, Principal at Engage Legal
Mark Karpeles, CEO of MtGox.com
Satoshi Nakamoto, author of the white paper “Bitcoin: A Peer-to-Peer Electronic Cash System”
The objective of the Foundation was never completely clear, with the original bylaws stating the following:
The Corporation shall promote and protect both the decentralized, distributed and private nature of the Bitcoin distributed-digital currency and transaction system as well as individual choice, participation and financial privacy when using such systems. The Corporation shall further require that any distributed-digital currency falling within the ambit of the Corporation’s purpose be decentralized, distributed and private and that it support individual choice, participation and financial privacy.
The Foundation was funded by membership fees – the initial membership fee schedule is provided below. However, the Bitcoin-denominated prices did start to decline in 2013 as the Bitcoin price appreciated.
It was believed by many that due to the membership subscription fees, the Foundation had considerable financial resources to spend on its mission.
Approximate lower bound of member contributions in April 2013 (Assuming initial fee rates)
2 Platinum Industry Members * 10,000 BTC = 20,000 BTC
7 Silver Industry Members * 500BTC = 3,500 BTC
175 Lifetime Members * 25BTC = 4,375 BTC
Total Resources = 27,873 BTC
(Source: BitMEX Research)
As we see later on in this report, the Foundation only had around 8,000 BTC at the end of 2012, still a nice warchest, but a lower balance than many had expected. It is possible our estimate above could be an overestimate, as the timing of member subscriptions is unclear.
The Foundation Board
The governance structure of the Foundation was quite complex and arcane. There were three classes of members:
The board initially consisted of five members, one nominated by the founders, two nominated by individuals and two nominated by corporate members. The term of each appointee was expected to be 3 years. At the start of the Foundation, all five board members were appointed by the founders and all board members were founders, with the exception of Jon Matonis.
Bitcoin Foundation Board Members (2012 to 2019)
(Source: Bitcoin Foundation Website, BitMEX Research)
Critics can point to the fact that the governance structure gave too much power to the initial founders and that new members of the organisation should have been able to join as equals to the founders.
The first board elections took place in 2013, with Meyer Malka winning the Industry seat and Elizabeth Ploshay winning the vote amongst individual members.
Board Election – Industry Seat (2013) – Winners: Meyer Malka
At the start of 2014, the holders of the two founding industry seats resigned. Charles Shrem resigned on 28 January 2014, two days after his arrest at JFK airport for money laundering and unlicensed money transmitter related offences. Charlie was eventually convicted and sentenced to two years in prison in December 2014. The main substance of Mr Shrem’s felony appears to be that he continued to provide customer support to a user of his BitInstant Bitcoin purchasing service, despite him allegedly knowing the customer wanted Bitcoin for the purposes of purchasing illegal drugs on the Silk Road e-commerce platform (Or that the customer wanted to supply the Bitcoin to somebody else, who wanted to purchase illegal drugs, one extra layer removed). Mark Karpeles, the holder of the other industry seat, resigned on 24 February 2014, following the failure and insolvency of the MtGox Bitcoin exchange, where Mark was CEO.
Brock Pierce and Bobby Lee were then elected as the two replacement industry appointed board members.
Board Election – Industry Seats (2014) – Winners: Bobby Lee & Brock Pierce
The appointment of Brock Pierce to the board proved controversial, with some claiming the Foundation should have done more vetting before allowing Mr Pierce to stand. The allegation against the former child actor, who featured in the “Mighty Ducks” and Disney’s “First Kid” was related to his alleged involvement in the sexual exploitation of children in the late 1990s. Although only a teenager at the time, Mr Pierce was an executive and co-founder at the internet video startup, Digital Entertainment Network (DEN), which was accused of hosting several parties where sexual abuse may have taken place. The allegations resulted in co-founder and CEO Marc Collins-Rector, along with Mr Pierce, resigning from DEN and supposedly fleeing to Spain. Mr Collins-Rector eventually plead guilty to child abuse related offences and according to Reuters, court record show that Mr Pierce paid US$21,000 to settle a related civil suit, while other claims were dropped, the article also states that Mr Pierce denies the allegations.
Towards the end of 2014, in the face of considerable pressure, the Foundation made the following improvements to its governance:
Board member terms were reduced to 2 from 3 years
The founder board seat was eliminated
The founder member class was removed
The Foundation’s Finances
The below table provides a basic analysis of the Foundation’s finances, in the period where most of the member dues were depleted (2012 to 2014). The data is based on the organisation’s IRS990 forms. With respect to the pay of the board, the disclosure seems reasonably strong. Most board members received no remuneration other than those acting as executives. Paying Gavin was one of the main aims of the organisation and Gavin’s pay appears to be disclosed in a reasonably clear and appropriate manner.
Jon Matonis (Contractor)
Other pay costs
Total pay costs
Disclosed Bitcoin figures
Bitcoin (US$ value at year end)
BTC sales proceeds
Realised Bitcoin gains/(losses)
Unrealised Bitcoin gains/(losses)
(Source: IRS 990 Forms, BitMEX Research)
The main criticisms related to the Foundation’s finances at the time appear twofold:
There was a sharp increase in spend in 2014, depleting the organisation’s reserves to near zero
There was a lack of transparency with regards to the Foundation’s Bitcoin balance
As for the first criticism, concerns did seem somewhat justified. In 2014 pay costs increased by 81%, the 2014 conference made a significant net loss and other costs increased significantly. As for the $1.3m in other costs, we have provided a breakdown below, therefore readers can judge the extent of the excesses. Compared to the excesses of the ICO bubble in 2017/18, where the total sum of the costs below perhaps represent a fraction of just one marketing party for the most egregious ICOs, the spend is moderate. However, some Foundation members clearly expected their funds to be spent more prudently. The main issue appears to be that expectations were not clearly set out in advance. Whatever your view, the fact is that by the start of 2015, the Foundation had almost run out of financial reserves and to that extent, its finances were mismanaged.
2014 breakdown of other spend
Other professional services
Professional event expenses
Payments to affiliates
Total other spend
(Source: Bitcoin Foundation IRS 990 form)
The lack of transparency with respect to the Foundation’s Bitcoin balance is another area of concern. At the end of each year the IRS990 form disclosed the USD value of the Bitcoin holding, the realised Bitcoin gains and the unrealised Bitcoin gains. Based on this information we calculated the following:
BitMEX Research BTC calculations
Bitcoin price at year end
Implied BTC balance at year end
Change in BTC balance
Implied sales price
Realised Bitcoin gains/(losses)
Unrealised BTC gains/(losses)
Lowest Bitcoin price figures
Lowest Bitcoin price in the year
Implied BTC sales proceeds
Realised Bitcoin gains/(losses)
(Source: IRS 990 Forms, BitMEX Research)
The disclosures in the IRS990 forms lead us to the following apparent Bitcoin related discrepancies:
The Foundations closing bitcoin balance in 2012 seems reasonably low given the volume of Bitcoin donations (See the c.28,000 BTC figure earlier in this report)
The Foundation disclosed an unrealised Bitcoin gain in 2013 of $5.2m, however based on the annual price movement and the calculated year end balance, we calculated an unrealised gain of only $4.4m
The Foundation disclosed an unrealised Bitcoin loss in 2014 of $2.0m, however based on the annual price movement and the calculated year end balance, we calculated an unrealised loss of only $0.6m
The Foundation disclosed Bitcoin sales proceeds of $569,728 in 2014, while even assuming all Bitcoin were sold at the lowest traded price in the year, given the large reduction in the Bitcoin balance of 4,600 coins, sales proceeds should have been $1.2m
Although there were accusations of embezzlement, we do not consider these disclosures to indicate any such crime. The Foundation was probably receiving Bitcoin and spending Bitcoin throughout the period, therefore clear financial record of Bitcoin sales are not likely to be available. At the same time, rules related to the reporting of realised and unrealised gains with respect to financial assets are not strict for this type of organisation and the Foundation does have a degree of discretion with respect to the calculation methodology. Therefore, the filings themselves do not indicate wrongdoing in our view. However, what we can say is that filings do not clearly explain what happened to the Bitcoin balance and an explanation from the board could be helpful.
Some members clearly expected greater transparency and wanted to question the board about the funds, but they were never given such an opportunity. The following quote from Bitcoin commentator Andreas Antonpoulous (who at the time was a Foundation committee chairman), reflected the views of many in the community at the time.
You say they are funded. Where are those funds? Who controls those funds? When were the last audited? Are they actually solvent? Or have all of those funds disappeared into a big black hole? Just remember who was in the leadership until recently, who is in the leadership today and what their track record of ethics has been and I would suggest that I would not be surprised at all if the Foundation implodes in a giant embezzlement problem sometime down the line or funds get stolen, within quotes or without quotes, or something like that. It’s bound to happen because these things don’t happen due to technical failures of bad actors, they happen due to failures of leadership The Foundation is the very definition of a failure of leadership.
To make matters worse, there were also accusations of the Foundation’s entanglement in the MtGox insolvency:
The MtGoX CEO, Mark Karpeles, was a founder and founding board member of the Foundation, while the company itself was a platinum member of the Foundation
Founding member, Roger Ver, famously assured MtGox customers of the solvency of the platform shortly before the exchange failed
The Foundation’s founding chairman, Peter Vessenes (who may have believed he was entitled to some MtGox equity), has been involved in various legal disputes with MtGox dating back to 2013 as a result of a failed business partnership. Peter’s company Coinlab sued MtGox for US$75m in 2013. As of August 2019, Peter now claims a remarkable total of US$16bn (Y1.6 trillion) from MtGox, an amount large enough to effectively block distributions to MtGox clients, and a large source of frustration to creditors to this day.
Andreas compared the Foundation’s situation to MtGox as follows:
Its problems go directly back to a complete failure of leadership. A completely closed, insular, arrogant, sheltered, uncommunicative leadership. Part of which was Karpeles himself, but there are another couple of relics left on that board, who pursue the exact same approach with their leadership. The Foundation is the Gox of Foundations. I am surprised it didn’t blow up in the wake of the Gox scandal, because there were a lot of significant conflicts within that environment.
However, perhaps it is unfair to make much of the association between MtGox and the Foundation, afterall, the ecosystem was small and MtGox was the dominant exchange, therefore a degree of association was inevitable to some extent.
The Amsterdam Conference (May 2014)
In May 2014 the Bitcoin Foundation arranged what was, up until this point, the largest conference in the space. It was the first conference (at least one which we attended), with characteristics familiar to many in the 2017/18 era. Unabated enthusiasm, unrealistic expectations about the underlying technology, expensive catering and countless booths representing new businesses with plans that appeared to make little commercial sense. As the figures above indicate, despite the expensive ticket prices of up to $800, the conference appears to have generated a net loss of around US$250,000.
The conference was split into two sections, a commercial section in the main exhibition hall, and the Bitcoin Foundation annual meeting (or technical track), which was down the hallway in a hotel conference room, entry to which was free for Foundation members. The technical discussions were followed by the Foundation members’ meeting
Journalist Ryan Selkis (now founder & CEO of Messari), was one of the key lifetime members at the event trying to hold the Foundation to account. At the annual meeting he asked several challenging questions to the Foundation board members, asking for greater transparency. Up until this point much of the debate and complaints had taken place on online web forums and this real world interaction marked a significant and novel change. In response to his challenges, one board member said the following:
We can spend a lot of our time trying to be transparent as much as we can and higher resources can be transparent or we can spend a lot of time in the board level making sure that we [have the] resources to make bitcoin bigger. It’s possible but right now, honestly, we’re in an environment where bitcoin is not well perceived. You asked for priorities at least from my side as a board member, it’s more about [making bitcoin bigger]
It was clear from this response that, for whatever reason, some board members had chosen not to tackle the transparency and governance concerns, leaving some members feeling frustrated and more convinced of wrongdoing on the part of the board.
The Blockchain Election (February 2015)
Given the issues that the Foundation had faced and the concerns in the community about transparency, governance and the purpose of the Foundation, this was a relatively important set of elections. There was a large number of candidates and a reasonably good quality debate among the candidates, for example a dedicated Let’s Talk Bitcoin podcast on the election.
The Foundation decided to conduct the 2015 individual board seat elections on the blockchain. As the chair of the election committee, Brain Goss said:
I believe in the concept of using the block chain for storage of compact proofs/hashes (as the market dictates), and I’m a big believer in transparent voting that any one can verify
However, the blockchain voting process did not run smoothly and the following issues arose:
The first round of voting took place using the Helios voting system. However, no candidate achieved more than 50% of the vote, as required by the by-laws, therefore a second round was required. The Foundation then made the odd decision to switch the voting platform to Swarm between the voting rounds, a decision met with widespread opposition. Despite initially starting the final round voting process on Swarm, during voting the Foundation then decided to switch back to Helios, invalidating the Swarm votes
The decision to reduce the number of candidates to four after the first round of voting appeared arbitrary
Registering to vote was widely regarded as a cumbersome and complex process and some candidates complained
(Source: Email received as part of the Swarm voting process)
Board Election – Individual Seats First Round (2015)
(Source: Helios voting system records)
Board Election – Individual Seats Final Round (2015) – Winners: Oliver Janssens & Jim Harper
After the voting controversy, Patrick Murk told Bitcoin Magazine:
This clearly struck a nerve with folks that think blockchain technology should only be used for transferring Bitcoin and not other [applications] like voting. [It] sparked a debate on how people use the blockchain
Removal of Directors & The End Of Board Elections (December 2015)
In December 2015, the two newly elected board members, Oliver and Jim, were removed by the other board members, due to a disagreement over the best way forward for the Foundation. Oliver and Jim had recently succeeded in competitive elections from individual members, giving them a considerable democratic mandate. At the same time the two year election terms of Elizabeth and Meyer had already expired, while Brock and Bobby had been elected by the industry and not individuals. Therefore, from the point of view of the individual members, Oliver and Jim were the only two board members with a significant mandate and they had been removed. In a violation of the by-laws, the Foundation then decided not to conduct any further board elections. As the executive director Bruce Fenton put it:
I used to believe that public, open elections were a great thing. I’m not as convinced now…. We unfortunately don’t have the time or resources for more process.
In our view, this logic seemed difficult to justify, given many of the problems were caused by the boards apparent lack of accountability to individual members, with Elizabeth Ploshey being the only board member elected by individual members who served on the board for any meaningful amount of time. If the Foundation did want to revive itself, it could have reinstated Oliver and Jim and allowed further elections to replace the other board members who could have left. Instead, the Foundation decided to distance itself even further from members, avoiding the challenges this accountability would have imposed, and consequently the Foundation appeared to lose any remaining legitimacy it had left.
After this point, between 2015 and 2019, four new board members were appointed from the pool of candidates that were defeated in the previous elections, except this time appointments were made by the board rather than members.
The Foundation still exists today, with Brock as Chairman and Bobby as Vice chairman, although their elected terms have long since expired and no more elections are in sight. The Foundation has no significant financial resources and is largely irrelevant. The activities the Foundation used to conduct are now carried out by others, for example Coin Centre does some regulator lobbying, and Bitcoin development is funded by other organisations such as Chaincode Labs, Blockstream, MIT’s DCI and other industry players. In many ways the conclusion to this piece writes itself. Bitcoin never needed a Foundation, it is stronger without one, and any all-encompassing Foundation like this was always doomed to fail.
The outrage at the lack of transparency at the Foundation exposes some of the key divergences in expectations and culture between members of the Bitcoin (now cryptocurrency) community. Some Bitcoiners, especially those involved since the early days of the Foundation, were often highly conspiratorial, paranoid and expected radically high levels of transparency, accountability and financial prudence. The Foundation seems to have misjudged these expectations, lost the backing of the community and ultimately failed. However, compared to the excesses of the coin offering era, which picked up from around 2014 onwards, peaking at the start of 2018, the financial accountability and transparency of the Bitcoin Foundation was almost impeccable, relatively speaking. Some members of the cryptocurrency community (not all newer ones), had radically different expectations, focusing more on what they perceived as game-changing technology, changing the world and getting super rich, rather than governance. Even in this new climate, irreparable damage to the Foundation’s brand had been done and it never again found its place.
UPDATE – 23 September 2019
After the publication of this piece, several prominent Bitcoin developers, whose names were displayed on the Foundation’s website, indicated to us (in some cases with proof) that they were given membership status for free (rather than by paying 25BTC). This may indicate that:
for the foundation may not have been as widespread as we initially thought
bitcoin balance in 2012 may never have been as large as we initially thought
Abstract: BitMEX Research has upgraded its lightning nodes to include watchtower functionality. The watchtower functionality is a mechanism for connecting to another friendly node, which monitors your lightning channels for you and prevents a dishonest counterparty from stealing your funds, even when you are offline. We successfully conducted an experiment, proving the watchtower concept actually works, at least in our case. It is encouraging that the watchtower concept, which has been around for years in theory, now actually works in practise.
On 29 June 2019, LND 0.7.0 (Go implementation of lightning) was released and this included the watchtower functionality. A watchtower is a third party lightning node, that can detect if a dishonest party attempts to steal funds and then broadcast a justice transaction, sending the funds back to the honest party, even when the honest node is offline.
There two modes of watchtower functionality
The client connects to a watchtower server. Whenever the lighting channel states change, data is sent over to the watchtower server with the latest channel state. In the event of a channel breach, the watchtower can broadcast a justice transaction, sending the funds to the honest node’s onchain wallet.
The watchtower server does not need to have any lighting channels or make any payments. The server connects to a lightning client and monitors the client’s lightning channels for them, on their behalf.
To connect the node to a watchtower server, one needs to add the following line to the lightning configuration file:
Where the public key and IP address is provided by the watchtower server.
To activate a watchtower server, one needs to add the following line to the lightning configuration file:
After this, one can run the command:
> lncli tower info
The watchtower server should then display the watchtower public key (different from the lightning node public key). This key is needed by the watchtower client. Due to potential denial of service threats, it is currently not advisable to publish the watchtower public key.
One can check if the watchtower is working by viewing the logs.
It is possible for a node to be both a watchtower server and client at the same time. If you run two nodes, each node can be the watchtower server of the other. BitMEX Research currently has three operating lightning nodes and the nodes all watch over each other in a loop configuration.
Successful test of the watchtower
On 30th July 2019, BitMEX Research successfully tested the watchtower system. Much like our previous piece on justice transactions, we tried to cheat ourselves, but this time used a watchtower. In an encouraging sign, the watchtower functionality correctly worked and the would-be thief was punished.
In order to do this test, we needed to run three nodes:
The dishonest node – BitMEXThief
The node using the watchtower service – BitMEXTowerClient (the user of the watchtower service)
The watchtower itself – BitMEXResearch
Manually constructing a watchtower justice transaction
(Source: BitMEX Research)
The eventual justice transaction, broadcast by our watchtower can be seen here.
All BitMEX Research lightning nodes are now protected by watchtowers. While a watchtower is a large improvement in security, in our view, a greater problem than dishonest channel breaches, is the risk of a lightning node’s memory becoming accidentally lost or destroyed – under such circumstances the node could lose the latest channel states. A watchtower does not fix that problem, although there have been improvements in this area, with Static Channel Backups (SCBs). Using SCBs, as long as no new channels were created post backup, all the funds should be safe.
A successful test of the watchtower does provide us with a greater degree of assurance about the robustness of the lightning network. It is encouraging that ideas such as watchtowers, which have been theoretically discussed for years, finally exist. However, when it comes to improving the robustness and reliability of the lightning network, there is still a long way to go.
Abstract: In our third look at the lightning network, we examine lightning channel closure scenarios and the incentives to punish dishonest parties and prevent them from stealing funds. This punishment mechanism is called a “Justice Transaction”. We explain how to arbitrarily construct a “Justice” scenario and present data on the prevalence of this type of transaction on the Bitcoin network. We have potentially identified 241 Justice transactions, representing 2.22 Bitcoin in value, since the lightning network launched at the end of 2017.
Following on from our January 2018 discussion of the motivation behind the lightning network and our March 2019 analysis of lightning network routing fee economics, this third piece on the lightning network looks at channel closures and the incentives designed to prevent dishonest lightning nodes from stealing funds, by broadcasting an earlier channel state.
It should be noted that, by design, when a thief attempts to steal funds on the lightning network, if caught, they do not only lose the money they tried to steal, they lose all the funds in the relevant channel. This “punishment” is expected to act as a deterrent and is sometimes called “justice”.
The four lightning channel closure scenarios
Opening lightning channels is, generally speaking, more simple than closing them, there is only one way to open a lightning channel, cooperatively with interactive communication between the parties. On the other hand, when evaluating channel closures, one needs to consider four different scenarios, as outlined in the decision tree below (See figure 1).
A non-cooperative non-breach closure occurs when an honest node initiates the closure, without directly communicating with the node on the other side of the channel.
Funds are distributed to each party’s onchain wallet based on the latest channel state.
These two different economic scenarios, are represented by one technical onchain scenario.
This scenario requires two onchain transactions.
Firstly the funds are redeemed using a 2 of 2 multi-signature witness and sent to two outputs. The node which did not initiate the closure is allocated funds based on what the channel closing party says is attributable to them, while another pot of funds is sent to an output which can be redeemed by using either an OP_IF or an OP_ELSE script.
In a second transaction, the funds sent to the OP_IF script, are claimed by the party that initiated the channel closure, using the OP_ELSE branch of Bitcoin script.
A non-cooperative breach non-justice closure occurs when a dishonest node initiates the channel closure, by broadcasting an earlier channel state, attempting to steal funds from the node on the other side of the channel.
The non closing node does not check the network within the locktime period, normally 24 hours and does not broadcast a justice transaction. Therefore the theft is successful.
Funds are distributed to each party’s wallet based on an earlier channel state, such that the non closing party losses funds and the dishonest channel closing party successfully steals funds.
A non-cooperative breach justice closure occurs when a dishonest node initiates the channel closure, without directly communicating with the node on the other side of the channel.
The non closing node does check the network within the locktime period, and creates a justice transaction, such that the attempted theft fails.
The would-be thief is punished and all the funds go to the honest non closing party.
In the justice scenario, two onchain transactions are also required.
In the first transaction, the funds are redeemed using a 2 of 2 multi-signature witness and sent to two outputs. The node which did not initiate the closure is allocated funds based on what the channel closing party says is attributable to them, while another pot of funds is sent to an output which can be redeemed by using either an OP_IF or an OP_ELSE script.
In a second transaction, the honest node, that did not initiate the closure claims all the funds sent to the OP_IF script, using the OP_IF branch.
This is the most revealing of the three channel closure types and provides the lowest level of privacy.
In the below arbitrary scenario, we manually created a justice transaction, using the following steps:
1. Spin up a new lightning network node (LND), with the alias “BitMEXThief” and open a channel, worth US$50 (400,000 Satoshis) with the BitMEXResearch lightning node 2. Switch off the BitMEXThief node and back up the .lnd directory 3. Restart the BitMEXThief node and make a lightning payment of US$25 (200,000 satoshis) to BitMEXResearch. The channel is now balanced, US$25 in both directions 4. Switch off the BitMEXThief node again 5. Switch off the BitMEXResearch lightning node (to prevent it broadcasting the latest channel state to the thief node) 6. Restore the BitMEXThief node back to its state prior to the channel re-balancing, the state in step 2 7. On the restored BitMEXThief node, attempt to close the channel from its earlier state and claim the full US$50 (400,000 satoshis) to the BitMEXThief node’s onchain wallet 8. Restart the BitMEXResearch node. The node then automatically detects the attempted theft and broadcasts the “justice transaction”, sending the full US$50 (less fees) to its onchain wallet. The would be thief was punished, by losing all the funds inside the channel. Note that the thief attempted to steal US$25, but ended up losing the full US$50.
The above experiment occurred successfully, providing some assurance that Lightning does actually work and if you try to steal, you will be punished.
Network Justice transaction data
After conducting our own justice transaction, we looked at the characteristics of this transaction (Inputs redeemed using the OP_IF branch) and searched for other justice transactions on the Bitcoin blockchain. We identified 241 transactions, which appear to be justice channel closures, dating back as far as December 2017. Mr. Alex Bosworth, from Lightning Labs, has created a tool to identify justice transactions, which may be more robust than our more basic search methodology.
Figure 3 – Number of justice transactions – monthly
(Source: BitMEX Research)
(Note: There is a possibility the data includes false positives)
Figure 4 – Value redeemed in justice transactions – monthly (BTC)
(Source: BitMEX Research)
(Note: There is a possibility the data includes false positives)
The justice transactions we identified had transaction inputs totaling 2.22 BTC, with the monthly total peaking at around 0.67BTC in February 2019, as figure 4 above illustrates. This does not necessarily mean thieves tried and failed to steal 2.22 BTC, as the dis-honest nodes may have punished thieves by a amount larger than the value they tried to steal (we do not know the latest channel state). The 2.22 BTC represents the total funds claimed by honest non channel closing nodes, part of this value is funds originally owned by the dis-honest nodes and part of the value will be the value they tried to steal.
It is also possible that many of the 241 justice transactions do not indicate genuine dishonestly, for instance it could be users testing the system, where the same user owns both lightning nodes in question. For example BitMEX Research is responsible for 5 of the 241 justice transactions, when there was no victim, as BitMEX owned all the nodes and funds.
241 justice transactions, with a value of just over 2 BTC is reasonably small relative to the size of the lightning network. The lightning network statistics website 1ml.com, indicates that there are currently 940 BTC locked up in 32,951 channels. The total number of justice transactions in the last 18 months is therefore only 0.7% of the current number of lightning channels.
In order for the lightning network to succeed as a robust, reliable and scalable payment system, the justice mechanism needs to be effective in deterring and preventing theft. As for the optimal justice rate, this is hard to determine, if it is too high and it shows that successful thefts may be too prevalent and the threat of justice may not be sufficient. If it is too low, it may mean nobody is attempting theft, thereby increasing the risk that users do not monitor their channels. This may lead to increases in the risk of large systemic channel thefts in the future.
For now, at least according to the data we have analysed, there appears to be a reasonable degree of justice on the burgeoning lightning network.