On Friday 8 June 12:00 UTC we will be delisting the 110% strike UP contract, XBT7D_U110 and 90% strike DOWN contract, XBT7D_D90. These will be replaced with two new contracts: a 105% strike UP contract, XBT7D_U105 and a 95% strike DOWN contract, XBT7D_D95.

Today, May 17, 2018, the BitMEX trading engine encountered several separate and heretofore unpredictable problems, causing feed latency and downtime in spurts throughout the day.

Disks mounted to the main trading engine hardware degraded sharply in performance at roughly 10:00 UTC. This degradation caused feed latency during scheduled archive and reindex jobs, which caused significant backpressure. Disk I/O operations were running at roughly 1/20 of their expected rate.

BitMEX runs redundant drives, but in this case, both drives were simultaneously exhibiting this degraded behavior. We had no choice but to schedule a maintenance downtime to replace them. Unfortunately, backpressure reached critical levels faster than we expected and we moved up our timetable.

At no point was data integrity compromised by this problem, but restoring the machine to a functional state with nominal disk performance took longer than expected to execute and verify.

After this action was complete, we restarted trading. Unfortunately, another problem was uncovered during the next archive, where a reindex job combined with a previously rare request pattern led to unexpected index regeneration and symbol revalidation on specific tables. This led to another backpressure scenario, with similar symptoms.

We have identified and fixed multiple contributing factors to the above behavior. The trading engine team will be closely monitoring engine performance throughout the day while continuing root cause analysis for the slowdowns.

BitMEX is proud to launch its first optionality products: UPs and DOWNs. This marks a very significant milestone in the product development history of the platform. With futures, swaps, and now options, BitMEX is inching closer to the goal of offering all manner of derivative products for the crypto-coin industry.

Why UPs and DOWNs?

UPs or Upside Profit Contracts, and DOWNs or Downside Profit Contracts are similar to call and put options. One of our biggest strengths at BitMEX is in engaging with the community and listening to our customers, and we have heard the roar for such products as the level of sophistication grows in this industry.

Why Now?

The liquidity profile of Bitcoin derivatives trading has changed dramatically over the past 12 months. The BitMEX XBTUSD Perpetual Swap is now the most heavily traded instrument in the entire crypto trading industry. Average daily trading volumes are in the billions of USD notional.

Before non-linear products like options are viable, linear products (Perpetual Swaps and Futures) must be sufficiently liquid. Given the liquidity profile of XBTUSD and the quarterly Bitcoin / USD futures contracts, we now believe there is sufficient liquidity in order to launch a successful options product.

What’s the Use Case?

Imagine Bitcoin is currently trading at $10,000 and you believe that by the end of the week, it will move 10% higher to $11,000. However, you don’t want any exposure to the price unless it hits your target. Also, you do not want your position to be liquidated before your target is reached, irregardless of the intra-week spot movements. For example, if the price drops to $5,000 but recovers to $12,000 by the settlement date, you will still profit and will not be liquidated.

Hence, you want the ability to participate above your target of $11,000 on the long side. The UPs product allows you to express this view, however this “optionality” comes at a cost which is the premium you pay to the seller of the option.

Why Can You Only Buy?

Selling naked (i.e. unhedged) options is one of the fastest and easiest ways to financial ruin given the potential for unlimited losses. On BitMEX, traders are limited to the margin they deposit on the platform, hence if the seller of the option cannot make good on potential losses, then socialised loss systems will need to be put in place which we want to avoid. As a result, we require sellers of the options to post the full notional value of the UP or DOWN contract.

Because no leverage is offered to sellers, it is very expensive from a capital perspective to make a market. In order to guarantee tight spreads at sufficient size, the BitMEX affiliated anchor market maker will be the only entity allowed to sell options initially.

Many of you may have concerns that the BitMEX affiliated entity is the sole market maker, however here are some points to consider:

1. As further discussed below, one cause of the engine overload issue is that we have many market makers constantly updating quotes on currently listed products. This consumes a vast amount of precious engine capacity. Until our engine performance is fixed, we refrain from listing any new product that exacerbates the issue. (E.g. this is one of the reasons why we delisted a number of our quarterly altcoin futures contracts, since the volumes they generated did not justify the engine resources consumed.) Hence, if only one market maker quotes on the UPs and DOWNs product, then the impact will not be meaningful on the engine.
   
2. The UPs and DOWNs products need to be fully margined. That is, buyers must pay the premium in full and sellers must post the full notional of the option in margin. That means that irregardless of where the price settles, neither buyers nor sellers will ever be liquidated. If the contract settles in the money, buyers are assured they will always receive their profit. Furthermore, this means that the anchor market maker cannot manipulate the UP or DOWN market in any way to liquidate any customer. 
3. The anchor market maker is tasked with keeping a tight market so that buyers can enter and exit trades as they wish throughout the contract’s length. We want to increase liquidity, having wide markets or an empty order book is not in BitMEX’s interest.
4. As we respond to customers’ feedback about the products, changes will be made to the UPs and DOWNs contracts. The anchor market maker will be able to adjust to the new product structure faster than any third party. That means that we can fail fast, and relaunch the product quickly with guaranteed liquidity.

What Are We Doing About Engine Performance?

At BitMEX, our top priority is improving the performance of our engine. In a detailed blog post, BitMEX Technology Scaling: Part 1, our CTO Samuel Reed explains in detail the issues we face and what we are doing to resolve these issues. However I will reiterate some points here.

The solution is not as simple as adding more servers or more engineers. The engine has a maximum throughput that is constrained by risk checks and calculations which are performed on each order, position, trade or price change so that we can maintain mathematical consistency on a platform that allows for 100x leverage. We have a two-pronged strategy to solve for this unique problem that BitMEX experiences:

1. Optimise as many existing functions as possible to obtain efficiency gains. We have been rolling out improvements weekly; however, that extra capacity is consumed very quickly as the demand increases to match engine performance improvements.
2. Re-architect the engine from the ground up so that the aforementioned issues can be scaled horizontally which will allow for more products and more users without overload issues. This work is ongoing, it won’t be solved overnight, but we are working towards this goal.

I want to emphasise that we will not list any products that worsen engine performance until we increase capacity sufficiently.

Additionally, we are scrutinizing which API users cost us the most in resources. Further API rate limits on traders with non-optimal Quote / Trade ratios are forthcoming. I remember how hard I fought as a CEO to convince traders to provide liquidity on our platform. This action pains me deeply and further sharpens my focus on finding a solution to this problem so that anyone who desires to provide liquidity may do so.

Launching the UPs and DOWNs products does not mean that BitMEX has forgotten or ignored the overload issue. Rather, we must continue to launch and test new products within reason so that in a year’s time we have another wildly successful product like XBTUSD.

If you are a talented engineer who believes he or she has a solution to this problem, we want to hear from you / hire you. Please reach out to us via the careers page or email, and a senior member of staff will review your qualifications or suggestions.

– Arthur Hayes, CEO and co-founder

We have recently updated our Terms of Service to explicitly clarify the relationship between BitMEX the trading platform, and an affiliated entity that engages in market making.

ToS Update

BitMEX has a for-profit trading business that, among other things, transacts in products traded on the BitMEX platform.  The trading business primarily trades as a market maker. The trading business is organised to be separate and distinct from the platform business. Specifically, no front office personnel are shared between the trading business and the platform, the trading business operates from a separate physical location, and the trading business does not have access to any platform order flow, execution, customer or other information on terms that are not otherwise available to any other platform user. In addition, unless otherwise set forth in the terms of a specific BitMEX product, the trading business receives access and trading privileges only on the same terms as are available to any other user.

Why Market Make?

Early on we discovered that market makers are very fickle. They only want to invest the time connecting to a trading platform that already has flow. What they don’t want to do is try out a new exchange where they must expend resources connecting only to have no takers. Given trades must be collateralised, this reduces their returns.

In order to entice others to provide liquidity, we funded an entity that would quote as soon as a new product listed. As the product became more liquid, this entity would scale back it’s quotes and focus on another product with lower liquidity on the BitMEX platform.

Right now the activity of this affiliated entity is concentrated on the altcoin contracts. XBTUSD and the quarterly Bitcoin / USD futures contracts have plenty of liquidity, and new market makers join every day to beef up those orderbooks. Mission accomplished … for now.

Being able to immediately support a new and illiquid product allows us to experiment with products that other platforms without an anchor market maker cannot. It also speeds up the process to obtaining other 3rd party liquidity providers.

How Do We Align Incentives?

The trading entity is a for-profit operation. However, their earnings are comprised of a service fee paid by the business, that is the BitMEX trading platform. In terms of trading PNL, the market making desk’s goal is to be breakeven.

If the desk is making too much trading PNL, the business will instruct them to tighten spreads and increase size. As a franchise, BitMEX succeeds because of greater trading volumes, not because of the market making desk’s trading PNL.

The market making desk earns the most if the exchange earns the most. That also means that dishonest and manipulative behaviour on the part of the market making desk is not tolerated. As you saw mentioned, the desk sits in a separate physical location. They also have no better information or access than any other trader on BitMEX. If traders feel that the platform is not fair, they will leave, and no one will get paid.

Our lead outside counsel is fully aware of the operation and advises us on best practices to ensure that we place the interests of BitMEX customers first.

What Activities Does The Desk Engage In?

The primary trading activity is providing two-sided liquidity on selected BitMEX products. The desk’s current focus is on increasing the liquidity on the altcoin contracts. The desk will also be the anchor market maker for the UPs and DOWNs products.

The desk also trades OTC with various counterparties globally.

As mentioned earlier, the desk does not engage in manipulative behaviour. The desk does not front-run customers. The desk does not manipulate either the market on BitMEX or the underlying exchanges for the purposes of stop loss hunting, or causing cascading margin calls.

None of this behaviour has occurred in the past, and if such behaviour is discovered, those responsible will be terminated immediately for cause.

Who Runs The Desk?

The head trader is Nick Andrianov. He is a former Deutsche Bank equity flow and exotics options trader. Nick and I have known each other for over ten years. His integrity is unquestionable.

Nick receives the business objectives from various senior members of BitMEX. The business and the market making desk work closely with the express goal to make every single BitMEX product as liquid as possible.

Financial Risks

Trading losses incurred by the market making desk will not affect the solvency of the BitMEX trading platform.

As stated above, the market making desk sits within a separate entity. Their goal is to provide liquidity to BitMEX and the wider crypto capital markets.

– Arthur Hayes, CEO and co-founder

Hi there – I’m Samuel Reed, CTO of BitMEX.

It’s been an incredible journey over the last four years building BitMEX. When we started, I don’t think any of us could have imagined the success this platform would achieve or how it would come to dominate Bitcoin/USD trading in 2018.

From 2014 to today, the BitMEX platform has grown from zero to an average of $3B of trading volume per day. Our flagship product, XBTUSD, trades more than any crypto product in the world. We serve customers all over the world, in five languages, and have become the premier platform for Bitcoin price discovery and liquidity.

The BitMEX team has been hard at work improving capacity, building a solid mobile offering, and creating a tech team that is truly best-in-class. We are not resting on our laurels, enjoying this success for the sake of it. Quite the opposite: we’ve been busier than ever.

We’d like to let the community in on how we formed and how we’re moving forward. As was wisely said: “In order to defeat the bug, we must understand the bug.”¹

---

Origins

I’ll begin with a true story.

In 2014, I was speaking at a web development panel in Hong Kong for General Assembly, a coding bootcamp. They wanted to give their soon-to-graduate students a taste of what it was like to work professionally. I took the opportunity to talk about my history: a career made of positions in several small businesses, startups, and government – with an emphasis on how incredibly in-demand software engineers are.

A rather loud personality in the back asked a question: “How do cash-poor startups looking for a CTO make a case? How do you attract great talent in such a competitive atmosphere?”

“Well, that’s a good question, and a tough answer,” I said. “Without funding, you have the challenge of a serious risk versus a sure thing. Why should any experienced developer forgo $200,000 or more at a large tech company, in a comfortable, resource-rich environment, to work 80 or more hours a week? You essentially have to find some bozo” – I really said this – “who believes in your idea so much he’s willing to take the risk despite so many better options.” I wished him good luck and we continued the panel.

He came up to me after the panel and told me he wanted to do a Bitcoin derivatives exchange. I knew then: I was that bozo, and Arthur Hayes and I were to become business partners.

Without any major funding, we brought an alpha online within six months and started with the BitMEX Trading Challenge, an no-rules trading competition where we put the exchange through its paces. And it really was no-rules (aside from multiple accounts) – hacking the site would win you the prize. We paid out a few Bitcoin in bug bounties in those days but we didn’t have any major failures.

Much to the annoyance of my wife, we launched BitMEX on during our honeymoon in Croatia, on November 24, 2014. Ben and Arthur celebrated separately, in Hong Kong. Notice the original trading interface in both photos. You can still read the original Trollbox messages from that day.

 

---

Building BitMEX, 2014

All projects are a product of the time in which they are built. In early 2014, the crypto ecosystem was reeling from the vacuum Mt.Gox left behind. The focus at the time was not “proof of work” vs. “proof of stake”, as it is today, but a forgotten term called “proof of reserves” – just Google it and look at the timestamps of all the popular posts. In fact, a question about this was the top-voted comment on our Reddit launch announcement.

The first rule of running a Bitcoin exchange is, and always has been, “Don’t lose the Bitcoin.”

This rule pervades everything we do at BitMEX. It permeates our policy, even today: we still use a 100% cold wallet where every transaction is multisig. Look up a 3BMEX transaction on the blockchain, and you’ll see it. For 1,250 straight days (!), at least two out of the three of us have gotten up, read the day’s withdrawals, done our risk checks, and signed, to be passed onto the next partner for signing and eventual broadcast.

At the time, I thought users would resist this. Yes, Bitcoin is better in so many ways than any monetary system that has come before it. But it is weaker too. Custodianship is an unsolved problem that requires constant vigilance. I think our customers know this and appreciate it. In our early days, we received a large number of complaints about withdrawal times. Today, where we are the largest exchange by volume in the world, we receive barely any. People get it – caring for your deposits this way is not easy. We do it not because it is convenient, but because it is safe.

The atmosphere in 2014 influenced how we built BitMEX. My frontend experience lead me to adopt ReactJS for the frontend. BitMEX was the first exchange to launch with it, a choice that has paid dividends well into 2018.

We were also the first – and likely still the only – exchange to build our matching and margining engine on kdb+/q, a technology traditionally used for querying of large-scale time-series data. It’s a natural fit. It’s fast (bear with me), using SIMD instructions to greatly boost throughput, it’s flexible, and it’s accurate. Kdb+’s flexibility and speed allowed us to pivot our product offerings twice: from low-leverage inverse and quanto futures to high-leverage ones, and from high-leverage futures to our flagship product, the XBTUSD Perpetual. We also pivoted loss-recovery mechanisms twice, from guaranteed settlement, to Dynamic Profit Equalization, to ADL.

BitMEX is a company known for listening to its customers and adapting. This required flexibility, innovation, and a lot of sweat equity from everyone on the team, and we’re so proud of how far it’s come.

---

Now, it wouldn’t be fair to come this far without addressing the title of this post. BitMEX now trades as much as US$6.5 billion per day. Our most recent 1-minute record was US$35 million, a number that is higher than the entire month of April 2016.

The following charts show monthly turnover in increasingly large timescales, to highlight detail completely lost in the overall view:

To understand why BitMEX is experiencing slowdowns, despite using a solid technology like kdb+, it’s important to understand what BitMEX does differently than other exchanges.

100x is a number that elicits a large number of reactions, ranging from “are you crazy?” to “how is this possible?” It is only possible due to incredible financial engineering from our co-founder and CSO Ben Delo. Ben is a diligent and brilliant mathematician. He built a perfect mathematical model for trading, a constantly-coherent system that continuously audits all trades and always sums to zero. Transactions don’t get lost in the BitMEX engine. A user’s balance never goes negative. There are entire classes of bugs that are common on other platforms that never occur on BitMEX, and it is that attention to detail that makes all the difference. Mark/Fair Pricing, the weighted ADL system, perpetual contract funding rates, and live isolated/cross remargining are all new, novel concepts that did not exist before BitMEX.

This consistent coherency inside the BitMEX engine makes 100x possible. Kdb+ has historically been fast enough that we can continuously remargin all positions upon each and every price change. This provides the safety and speed necessary to not only survive within the razor-thin requirements of 0.5% maintenance margin, but thrive. The BitMEX Insurance Fund, a fund that guarantees settlement of BitMEX contracts, contains (at the time of writing) an incredible 6,149 XBT, over US$50M. Competing firms have insurance funds in the single digits of Bitcoin, despite offering as low as only 20x leverage.

BitMEX won’t sacrifice safety for speed. The security of our users’ funds and confidence in their trades is paramount. But we hear all of you: you want to trade faster, you want freedom from “System Overload” messages, and we will give that to you.

Since late 2017, the BitMEX team has refocused on engine performance as our highest priority. We have built, and are continuing to build, a team full of the top professionals in the space. This team works hard, building capacity for the next 100x increase in trading volume.

---

In the second part of this series, I’ll explain in-depth:

• How the BitMEX engine processes orders and remargining
• How real-time messages flow through our system to your browser
• How BitMEX uses API-first design to provide the most powerful API in the business
• Performance charts showing hot-spots, peak versus baseline load, and corner-cases
• A breakdown of the dreaded “System Overload” message, and how it is generated

In the third part, I’ll also explain:

• Performance numbers showing how capacity has increased since 2017
  • We have made large strides in the past months – but demand has increased to match
• Roadmaps and pending work for Q2
• BitMEX’s vision for the future of online derivatives trading

Thank you to all of you for being a part of BitMEX’s success. Ben, Arthur and I feel fortunate than to be a part of such a great company: our customers, team, and market opportunity are simply best-in-class.

Reach out to me directly on Twitter at @STRML_ and on Telegram at STRML. I also occasionally talk with traders on the Whalepool TeamSpeak, a fun community of traders that have given great feedback and encouragement to BitMEX for years.

---

¹ – Starship Troopers was ahead of its time with its views on software development.

At 02:22 UTC on April 15, 2018, the ETHBTC on Poloniex crashed approximately 18% from 0.063 to 0.052. The Poloniex price is used as 100% of the index price (.ETHXBT30M) for the ETHM18 contract.

Crypto trading is very risky and underlying prices are highly volatile. While this type of movement is unexpected and undesirable, the price on BitMEX accurately reflected the price on the underlying spot market throughout this crash and recovery. Due to BitMEX systems functioning as expected and according to specification, there will be no refunds of losses sustained due to liquidations.

Index stability is important to us. BitMEX will continue to evaluate if adjustments are needed to existing index compositions. The liquidity of crypto spot markets is constantly in flux and can change significantly over the lifetime of a quarterly contract. Poloniex hosts one of the most liquid ETHBTC markets (note, not ETHUSD) in the world, but this unexpected action has triggered an internal re-review.

The Index Price is used as the central calculation for marking BitMEX futures. This Mark Price is used for margin calculations, which trigger liquidations. More information is available here.

On 30 March 2018, we are making the following changes to altcoin products:

• The 0.25% settlement fee will be removed on futures products. We hope that will encourage greater liquidity by removing barriers to entry and exit.
• We will be removing the DASH, ETC, NEO, XMR, XLM, and ZEC pairs after they expire, for the time being. This is to free up trading-engine capacity on our more popular contracts. After we make certain optimisations, we may re-list these.
• The ADA, BCH, ETH, LTC, and XRP contracts will be re-listed for another quarterly today:
  • BitMEX Cardano / Bitcoin 29 June 2018 futures contract (ADAM18)
  • BitMEX Bitcoin Cash / Bitcoin 29 June 2018 futures contract (BCHM18)
  • BitMEX Ether / Bitcoin 29 June 2018 futures contract (ETHM18)
  • BitMEX Litecoin / Bitcoin 29 June 2018 futures contract (LTCM18)
  • BitMEX Ripple / Bitcoin 29 June 2018 futures contract (XRPM18)

Thanks to all of the participants for helping to make another BitMEX giveaway a success, and would like to congratulate all of the winners! We were very impressed with the performance of the top two traders, and as a result decided to name them both grand-prize winners. Special congratulations to both of you.

Winners, please see the email you received for prize details.

To see the giveaway details, click here.

BitMEX is happy to announce a new $100,000 giveaway for NEO (NEOG18) contracts!  

Start: Wednesday, 7 February 2018 at 00:00 UTC
End: Saturday, 17 February 2018 at 00:00 UTC

Prizes:

1 Grand Prize of 50,000 USD: The trader who continuously quotes the largest two-sided volume within a 0.5% spread gets the grand prize.*

1 Second Prize of 25,000 USD: The trader who continuously quotes the second largest two-sided volume within a 0.5% spread gets the second prize.*

1 Third Prize of 20,000 USD: The trader who has the largest profit (in XBT) from trading the NEO (NEOG18) contract gets the third prize.

1 Lucky Prize of 5,000 USD: Any trader who trades at least one NEO (NEOG18) contract automatically enters a random draw to win this prize.

*BitMEX will take snapshots of the orderbook at frequent intervals throughout the competition. A trader will add to their quoted volume in each snapshot by taking the minimum of their quoted bids and quoted asks within 0.25% either side of the midpoint. BitMEX will then sum that volume in each snapshot, and the trader with the largest cumulative volume at the end of the competition will win.

It’s simple to participate:

• First, sign up for a BitMEX account.
• 
• Next, trade the NEO (NEOG18) contract during the contest period.

Cheers,
The BitMEX Team

Terms & Conditions:

1. BitMEX reserves the right to cancel or amend the giveaway or giveaway rules at our sole discretion.
2. Users who engage in market manipulation will be excluded from the contest. This determination will be made at the sole discretion of BitMEX.
3. Profit is defined as realized profit (in XBT terms) of all trades where the trade was opened and closed during the contest period window.
4. No user can win more than one prize.
5. Winners will be notified via email on 22 February 2018.
6. All awards are paid out in Bitcoin at the prevailing price of the .BXBT index at 17 February 2018 at 00:00 UTC.

Thanks to all of the participants for helping to make the giveaway a success, and congratulations to all the winners! Please see the email you received for prize details.

A total of 1,993 participants entered the giveaway with a total of 4,403 tickets earned.

All 15 lucky winners were selected randomly from the 4,403 tickets. Any trader, big or small, can win a lucky draw, so make sure your name is in the hat for our upcoming giveaways.

To see the giveaway details, click here.

*Trader has agreed to show his/her real username in the winner announcement.

Wishing you a prosperous 2018!

Cheers!
The BitMEX Team