Abstract: This piece is written by Bitcoin Core contributor and BitMEX Research guest writer, Michael Ford. Michael is the recipient of the HDR Global Trading Limited Bitcoin development grant of US$60,000 per annum. In this report, Michael explains recent Bitcoin Core build system improvements and how he has been involved in removing third party software dependencies, such as OpenSSL. The number of packages built in Bitcoin Core 0.19.99 is down 44% since Bitcoin Core 0.13.2 and the build time has fallen 42% since the peak, to 135 seconds, according to Michael’s tests. This work has improved the security of the software, by reducing the attack surface and improved software performance.
Bitcoin Core Dependencies – Build Time
(Source: Michael Ford’s analysis)
(Notes: Only requires packages, excludes downloading. Make -C depends -j8 NO_QT=1 NO_UPNP=1 etc, Conducted on MacOS)
Bitcoin Core Binary Size (MB)
(Source: Michael Ford’s analysis)
One of the less glamorous to work on, yet important components of the Bitcoin Core codebase is the build system. It controls how the code is configured and compiled, how one interacts with dependencies and it is what helps us build completely reproducible binaries, which many bitcoin users run.
For some background on builds systems, security, and why it’s particularly important for a project like Bitcoin Core, I would recommend watching the below talks from Cory Fields and Carl Dong, both long-term Bitcoin Core contributors.
In this post I will quickly discuss some recent build system changes I have been involved with, most of which will be part of the 0.20.0 release of Bitcoin Core.
Removing BIP70 support from Bitcoin Core has been on the cards for a long time. There are many write-ups summarising its problems, therefore I won’t elaborate here, but will just say that they included security vulnerabilities, privacy issues and incompatibilities between implementations. BIP70 was also one of the last portions of the codebase that required OpenSSL, thus blocking its removal. A good summary of the above is available in this GitHub comment.
Removing or deprecating outward facing functionality from any project usually requires multiple phases, and removing features from Bitcoin Core is no different. In October 2018, pull request (PR) #14451, which was based off of code proposed nearly a year prior in #11622, made it possible to disable BIP70 while building Bitcoin Core. However, it remained in the binaries release as part of 0.18.0.
Shortly after, in #15584, BIP70 support was disabled by default, meaning anyone building Bitcoin Core would have to opt in using `–enable-bip70` when compiling. BIP70 being disabled by default was part of the recent 0.19.0 release, along with GUI warnings and a suggestion to switch to BIP21 URLs where possible.
Removing BIP70 from Bitcoin Core was proposed in #17165. The PR removed our Protobuf dependency, X509 related OpenSSL usage, the need to link OpenSSL into the GUI and ~2,000 lines of code overall. In a follow up pull request, #17730, it was also possible to remove nearly all of Qt’s internal networking features from Bitcoin Core; however issues with the Windows build prevented removing them entirely. BIP70 being removed was one of the last steps required before removing OpenSSL became possible.
Bitcoin Core maintains an internal RNG (random number generator), which is fed entropy from many different sources, including hardware, the operating system, and historically, OpenSSLs RNG. Prior to removing OpenSSL, it was decided to augment Bitcoin Core’s RNG to ensure that it would be fed additional entropy from the environment (time, performance data, system configuration etc), which would also include some of the sources previously used by OpenSSL. This was done in #17270. Two additional PRs removed some of OpenSSLs RNG seeding by dropping calls to RAND_screen() #17191 and RAND_event() #17151.
At this point it was possible to remove all remaining OpenSSL usage from the project, and this was done in #17265 and #17515; these changes will be part of the Bitcoin Core 0.20.0 release. OpenSSL has long been a source of bugs, emergency releases and performance issues. Its consensus critical usage (signature validation) has long since been replaced by the secp256k1 library, developed by Bitcoin Core developers in response to OpenSSLs shortcomings, as part of the 0.12.0 release of Bitcoin Core.
macOS Dependency Reductions
Some other build system related work I’ve been involved with has been more macOS specific, and includes reducing the number of dynamic libraries our macOS binaries link against. Compared to 0.19.0, macOS binaries built using the latest codebase (which will be in the upcoming 0.20.0 version), link against six less dynamic libraries. The libraries we no longer link against include DiskArbitration, Security, SystemConfiguration, OpenGL, AGL and CFNetwork.
Some of these libraries were already unused, but still linked against as part of the build process. After the introduction of the -dead_strip_dylibs linker flag in #17663, they were dropped. Others, like OpenGL or D-Bus, were being pulled in by Qt, but were still ultimately unused, and could be dropped by changing our Qt build configuration (see #17521 and #17676 for those changes).
macOS Security Improvements
After working to decrease the number of macOS dependencies, it seemed worthwhile to ensure that we didn’t reintroduce any of those dependencies unnecessarily. #17863 has been opened to add macOS dynamic library checking (already done for Linux) as part of the build process. #17787 added PIE (Position Independent Executable) and NOUNDEFS (no undefined references) to our security-check.py script.
This was a very short, high-level overview of some of the recent build systems work I’ve been involved with. The result of which has been less (non-bitcoin) code in the Bitcoin Core repository, less build dependencies, for both bitcoind and bitcoin-qt, smaller executable sizes and improvements in security, if just by reducing the attack surface.
The majority of the work here has only been possible due to countless hours of prior work, as well as review, from many other Bitcoin Core contributors. I’d especially like to thank Cory Fields, for his help and guidance with much of my build systems work.
Michael Ford, Bitcoin Core Developer
CC BY-SA 4.0