Websocket Latency, 26 November 2019

Between 21:30 UTC 25 November 2019 and 07:05 UTC 26 November 2019 the Websocket API was running with degraded capacity. This resulted in slightly more latent feeds during this time and some isolated occurrences of substantial lag during traffic spikes correlated with large market moves.

The issue was identified when latency thresholds were breached in our automated monitoring systems around 06:55 UTC. From 07:05 the issue was resolved and full capacity was restored. We continue to monitor the impacted services closely.

The issue was caused by incorrect CPU pinning following a release of a market data distribution component at 21:30 UTC 25 November 2019. The impact of this was only observed during large traffic spikes which occurred several hours later and so was not identified during the post release checks at the time. The configuration of this service has been corrected and the deployment tested to prevent this from happening again.

We apologise for any inconvenience this may have caused. If you have any further questions, please contact Support via our contact form.

WebSocket API Feed Interruption, 24 July 2019

Between 10:44:13 and 10:52:29 UTC on 24 July 2019, the realtime data on the following websocket API feeds was interrupted:

  • account, affiliate, execution, funds, instrument, margin, order, position, trade, transact, wallet

Users of the BitMEX website may have noticed that some data was not updating during this period; e.g. in the Recent Trades panel, Open Orders panel, Fills panel, and Position panel.

The following public feeds were unaffected during this period:

  • funding, insurance, liquidation, settlement, impactQuote, impactQuoteBin1m, quote, quoteBin1m, quoteBin5m, quoteBin1h, quoteBin1d, tradeBin1m, tradeBin5m, tradeBin1h, tradeBin1d, orderBookL2_25, orderBook10, orderBookL2

The REST API was also unaffected and client-side data loss due to missed feed updates during this period can be resolved using the REST API.

During this period we continued to process all order instructions and the trading engine was unaffected. The outage was due to a slow consuming internal market data service which became degraded, putting pressure on our market data infrastructure. This internal market data service has been suspended whilst the root cause is being investigated. Additionally, several safety mechanisms are currently being deployed to prevent a potential recurrence from impacting user-facing feeds.

We apologise for any inconvenience this interruption may have caused. If you have any further questions please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

WebSocket API Feed Interruption, 27 June 2019

Between 07:50 and 07:58 UTC on 27 June 2019, the following websocket API feeds were interrupted due to a complication during a planned upgrade of our market data distribution services:

  • Account, affiliate, execution, funds, instrument, margin, order, position, trade, transact, wallet

Users of the BitMEX website may have noticed some data not updating during this period e.g. in the Recent Trades panel, Open Orders panel, Fills panel, and Position panel.

The following public feeds were unaffected during this period:

  • Funding, insurance, liquidation, settlement, impactQuote, impactQuoteBin1m, quote, quoteBin1m, quoteBin5m, quoteBin1h, quoteBin1d, tradeBin1m, tradeBin5m, tradeBin1h, tradeBin1d, orderBookL2_25, orderBook10, orderBookL2

During this period we continued to process order instructions and the trading engine was unaffected.

Due to this issue, data in a subset of data mirrors which service user REST API requests was left in an incomplete state. A side-effect of this was that some users observed stale open orders on the BitMEX website for orders which were already cancelled for a period of 90 minutes whilst data was being restored. Any API users that may be missing updates for this period can now backfill data via the REST API.

If you are experiencing order cancellation issues via the website, please refresh your web browser.  We apologise for any inconvenience this interruption may have caused. If you have any further questions please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

WebSocket Rate Limiting Issue, 25 June 2019

At 21:09:00 UTC 25 June, we released an update to our API layer that inadvertently started to count WebSocket subscriptions to certain tables against the request rate limit that had otherwise been exempt. This update may have impacted customers who heavily utilise the WebSocket API. Once the issue was identified at 00:19 UTC 26 June, we immediately rolled back the update to bring systems back to normal.

We apologise for any inconvenience this may have caused. To read more about which subscriptions are exempt from the request rate limiter, see our previous blog post for details. 

ETHUSD Orderbook Feed Issues, 24 June 2019

Between 09:25:54 UTC and 09:44:30 UTC 24 June 2019 the orderBookL2, orderBookL2_25, orderBook10, and quote realtime websocket feeds for ETHUSD were in a degraded state. During this period, the state of the ETHUSD orderbook on these feeds was incorrect.

We were able to identify and resolve the root cause of the issue within a minute of detection. The issue was caused by a rare sequence of order events that triggered a bug in an optimisation of the orderBookL2 calculation which had been deployed to the production environment several hours earlier. This change has since been reverted.

There was no impact to orders in the trading engine itself – just the presentation of the calculated orderbook for ETHUSD downstream of the trading engine.

We have deployed additional automated feed validators to detect potential similar issues in the future and to alert us earlier.

We apologise for the inconvenience this may have caused. If you have any further questions, please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

Websocket Latency, 30 May 2019

Between 16:00 and 17:00 UTC 30 May 2019 the websocket API experienced periods of substantial lag due to spikes of traffic generated by the trading engine during large market moves. During this period some websocket connections also experienced dropped market data updates as memory limits on an internal messaging layer were hit, forcing reconnections.

Our engineers are accelerating the development effort in an already-planned strategic upgrade of our market data distribution architecture to vastly increase its capacity and lower the overall latency of the websocket feed. This capacity upgrade is scheduled for Testnet release this week and we will update users once this has been released to the main platform.

If you have any further questions, please contact Support via our contact form: https://www.bitmex.com/app/support/contact.

Websocket API downtime, September 3rd 2018

From 06:53 to 07:03 UTC today, September 3rd, 2018, the service of our websocket API was impaired, which impacted the live data updates on the bitmex.com website as well as clients connected directly to our websocket API.

As part of a scheduled software release on the trading engine, our market data distribution component became saturated with a cascade of updates during the restart process which temporarily prevented downstream data distribution on the websocket API.

We apologise for the disruption.  The root cause has been identified and mitigated in the production environment.  BitMEX engineers have already deployed a fix for the underlying issue to prevent a re-occurrence.  We are also improving a number of alerting mechanisms to enable us to recover from potential issues in a more timely manner.