At 21:09:00 UTC 25 June, we released an update to our API layer that inadvertently started to count WebSocket subscriptions to certain tables against the request rate limit that had otherwise been exempt. This update may have impacted customers who heavily utilise the WebSocket API. Once the issue was identified at 00:19 UTC 26 June, we immediately rolled back the update to bring systems back to normal.
We apologise for any inconvenience this may have caused. To read more about which subscriptions are exempt from the request rate limiter, see our previous blog post for details.
Between 09:25:54 UTC and 09:44:30 UTC 24 June 2019 the orderBookL2, orderBookL2_25, orderBook10, and quote realtime websocket feeds for ETHUSD were in a degraded state. During this period, the state of the ETHUSD orderbook on these feeds was incorrect.
We were able to identify and resolve the root cause of the issue within a minute of detection. The issue was caused by a rare sequence of order events that triggered a bug in an optimisation of the orderBookL2 calculation which had been deployed to the production environment several hours earlier. This change has since been reverted.
There was no impact to orders in the trading engine itself – just the presentation of the calculated orderbook for ETHUSD downstream of the trading engine.
We have deployed additional automated feed validators to detect potential similar issues in the future and to alert us earlier.
We apologise for the inconvenience this may have caused. If you have any further questions, please contact Support via our contact form: https://www.bitmex.com/app/support/contact.
Between 16:00 and 17:00 UTC 30 May 2019 the websocket API experienced periods of substantial lag due to spikes of traffic generated by the trading engine during large market moves. During this period some websocket connections also experienced dropped market data updates as memory limits on an internal messaging layer were hit, forcing reconnections.
Our engineers are accelerating the development effort in an already-planned strategic upgrade of our market data distribution architecture to vastly increase its capacity and lower the overall latency of the websocket feed. This capacity upgrade is scheduled for Testnet release this week and we will update users once this has been released to the main platform.
If you have any further questions, please contact Support via our contact form: https://www.bitmex.com/app/support/contact.
From 06:53 to 07:03 UTC today, September 3rd, 2018, the service of our websocket API was impaired, which impacted the live data updates on the bitmex.com website as well as clients connected directly to our websocket API.
As part of a scheduled software release on the trading engine, our market data distribution component became saturated with a cascade of updates during the restart process which temporarily prevented downstream data distribution on the websocket API.
We apologise for the disruption. The root cause has been identified and mitigated in the production environment. BitMEX engineers have already deployed a fix for the underlying issue to prevent a re-occurrence. We are also improving a number of alerting mechanisms to enable us to recover from potential issues in a more timely manner.