On 9 January BitMEX experienced two unrelated minor outages.
At 02:44:10 UTC the WebSocket API saw a degradation in performance for a minute where 7% of commands sent by clients failed. Connections continued undergoing a 1% failure rate of commands until servers recovered at 02:47:00 UTC.
Clients may have also seen an increase in response times for some market data REST endpoints up during this period. This was due to a rolling restart of the API servers that occurred in too tight of a timeframe.
In addition, at 05:48:10 UTC and 06:10:10 UTC, BitMEX experienced minor outages for approximately 30 seconds whereby requests to the trading engine were load-shed as the engine was busy. During these times, clients would have observed a lack of updates over the WebSocket API for the same reason. The outages were due to data replay complications during a regularly scheduled market data distribution component restart.
There was no data loss during these events and an additional safety mechanism to prevent a similar situation from impacting the trading engine has already been deployed. The root causes have also been identified and we are currently working on permanent fixes to prevent a recurrence. Updates will follow in the future.
We apologise for the inconvenience. If you have any questions, please contact customer support.