WebSocket API Feed Interruption, 24 July 2019

Between 10:44:13 and 10:52:29 UTC on 24 July 2019, the realtime data on the following websocket API feeds was interrupted:

  • account, affiliate, execution, funds, instrument, margin, order, position, trade, transact, wallet

Users of the BitMEX website may have noticed that some data was not updating during this period; e.g. in the Recent Trades panel, Open Orders panel, Fills panel, and Position panel.

The following public feeds were unaffected during this period:

  • funding, insurance, liquidation, settlement, impactQuote, impactQuoteBin1m, quote, quoteBin1m, quoteBin5m, quoteBin1h, quoteBin1d, tradeBin1m, tradeBin5m, tradeBin1h, tradeBin1d, orderBookL2_25, orderBook10, orderBookL2

The REST API was also unaffected and client-side data loss due to missed feed updates during this period can be resolved using the REST API.

During this period we continued to process all order instructions and the trading engine was unaffected. The outage was due to a slow consuming internal market data service which became degraded, putting pressure on our market data infrastructure. This internal market data service has been suspended whilst the root cause is being investigated. Additionally, several safety mechanisms are currently being deployed to prevent a potential recurrence from impacting user-facing feeds.

We apologise for any inconvenience this interruption may have caused. If you have any further questions please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

Update to BitMEX Indices, 15 July 2019

Effective 15 July 2019 at 15:15 UTC, BitMEX will reintroduce Kraken into its indices following the resumption of trading on Kraken. The updated indices are detailed in the table below.

All traders should be aware that these indices may fluctuate substantially, and should exercise caution when trading contracts that reference these indices.

If you have any further questions, please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

Affected Index

Index Constituents

.BXBT

Bitstamp, Coinbase Pro, Kraken

.BETH

Bitstamp, Coinbase Pro, Kraken

.BETHXBT

Binance, Poloniex, Kraken

.BBCHXBT

Binance, Poloniex, Kraken

.BXRPXBT

Binance, Poloniex, Kraken

.BLTCXBT

Binance, Poloniex, Kraken

.BEOSXBT

Binance, Poloniex, Kraken

Temporary Change to BitMEX Indices, 14 July 2019

Effective 14 July 2019 at 21:00 UTC, BitMEX will temporarily remove Kraken from its indices in response to Kraken’s scheduled downtime. This scheduled downtime is expected to last for 3-8 hours and will affect the 7 BitMEX indices as detailed in the table below.  Kraken will be reintroduced once trading has resumed. We will announce their reintroduction 12 hours in advance.

All traders should be aware that the price of these indices may fluctuate substantially, and should exercise caution when trading these indices.

If you have any further questions, please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

Affected index

Current constituents

Temporary constituents

.BXBT

Bitstamp, Coinbase Pro, Kraken

Bitstamp, Coinbase Pro

.BETH

Bitstamp, Coinbase Pro, Kraken

Bitstamp, Coinbase Pro

.BETHXBT

Binance, Poloniex, Kraken

Binance, Poloniex

.BBCHXBT

Binance, Poloniex, Kraken

Binance, Poloniex

.BXRPXBT

Binance, Poloniex, Kraken

Binance, Poloniex

.BLTCXBT

Binance, Poloniex, Kraken

Binance, Poloniex

.BEOSXBT

Binance, Poloniex, Kraken

Binance, Poloniex

WebSocket API Feed Interruption, 27 June 2019

Between 07:50 and 07:58 UTC on 27 June 2019, the following websocket API feeds were interrupted due to a complication during a planned upgrade of our market data distribution services:

  • Account, affiliate, execution, funds, instrument, margin, order, position, trade, transact, wallet

Users of the BitMEX website may have noticed some data not updating during this period e.g. in the Recent Trades panel, Open Orders panel, Fills panel, and Position panel.

The following public feeds were unaffected during this period:

  • Funding, insurance, liquidation, settlement, impactQuote, impactQuoteBin1m, quote, quoteBin1m, quoteBin5m, quoteBin1h, quoteBin1d, tradeBin1m, tradeBin5m, tradeBin1h, tradeBin1d, orderBookL2_25, orderBook10, orderBookL2

During this period we continued to process order instructions and the trading engine was unaffected.

Due to this issue, data in a subset of data mirrors which service user REST API requests was left in an incomplete state. A side-effect of this was that some users observed stale open orders on the BitMEX website for orders which were already cancelled for a period of 90 minutes whilst data was being restored. Any API users that may be missing updates for this period can now backfill data via the REST API.

If you are experiencing order cancellation issues via the website, please refresh your web browser.  We apologise for any inconvenience this interruption may have caused. If you have any further questions please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

WebSocket Rate Limiting Issue, 25 June 2019

At 21:09:00 UTC 25 June, we released an update to our API layer that inadvertently started to count WebSocket subscriptions to certain tables against the request rate limit that had otherwise been exempt. This update may have impacted customers who heavily utilise the WebSocket API. Once the issue was identified at 00:19 UTC 26 June, we immediately rolled back the update to bring systems back to normal.

We apologise for any inconvenience this may have caused. To read more about which subscriptions are exempt from the request rate limiter, see our previous blog post for details. 

ETHUSD Orderbook Feed Issues, 24 June 2019

Between 09:25:54 UTC and 09:44:30 UTC 24 June 2019 the orderBookL2, orderBookL2_25, orderBook10, and quote realtime websocket feeds for ETHUSD were in a degraded state. During this period, the state of the ETHUSD orderbook on these feeds was incorrect.

We were able to identify and resolve the root cause of the issue within a minute of detection. The issue was caused by a rare sequence of order events that triggered a bug in an optimisation of the orderBookL2 calculation which had been deployed to the production environment several hours earlier. This change has since been reverted.

There was no impact to orders in the trading engine itself – just the presentation of the calculated orderbook for ETHUSD downstream of the trading engine.

We have deployed additional automated feed validators to detect potential similar issues in the future and to alert us earlier.

We apologise for the inconvenience this may have caused. If you have any further questions, please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

Scheduled System Update, 04 June 2019

Please be advised that we will be performing a scheduled system update to our database service starting 01:00 UTC 04 June 2019 and it is expected to last 3 – 5 hours. Trading, logins, and other key API features will remain operational, however please note that the following features will be disabled during the update period:

  • New account signup
  • Email verification
  • Enable TFA
  • Disable TFA
  • Password reset
  • Withdrawal
  • Update preferences
  • Mute accounts on the Trollbox
  • Create API Key
  • Disable API Key
  • Enable API Key
  • Delete API Key

Once we have completed the system update we will make a further announcement.

We apologise for any inconvenience this may cause. Feel free to contact our Support with any concerns you may have about the scheduled update. You may reach us via our contact form: https://www.bitmex.com/app/support/contact

Websocket Latency, 30 May 2019

Between 16:00 and 17:00 UTC 30 May 2019 the websocket API experienced periods of substantial lag due to spikes of traffic generated by the trading engine during large market moves. During this period some websocket connections also experienced dropped market data updates as memory limits on an internal messaging layer were hit, forcing reconnections.

Our engineers are accelerating the development effort in an already-planned strategic upgrade of our market data distribution architecture to vastly increase its capacity and lower the overall latency of the websocket feed. This capacity upgrade is scheduled for Testnet release this week and we will update users once this has been released to the main platform.

If you have any further questions, please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

Websocket API downtime, September 3rd 2018

From 06:53 to 07:03 UTC today, September 3rd, 2018, the service of our websocket API was impaired, which impacted the live data updates on the bitmex.com website as well as clients connected directly to our websocket API.

As part of a scheduled software release on the trading engine, our market data distribution component became saturated with a cascade of updates during the restart process which temporarily prevented downstream data distribution on the websocket API.

We apologise for the disruption.  The root cause has been identified and mitigated in the production environment.  BitMEX engineers have already deployed a fix for the underlying issue to prevent a re-occurrence.  We are also improving a number of alerting mechanisms to enable us to recover from potential issues in a more timely manner.