About 13:00 UTC today, the BitMEX platform sustained a record amount of traffic, both in orders and in number of realtime subscriptions. This traffic caused our proxies’ network traffic to be throttled on our cloud host, which caused data streams to back up. Those data streams eventually backed up enough that they caused a cascade of website users to request new data images, which was a volume of traffic large enough to render the site unusable.
We recovered the platform by temporarily disabling trading, reprovisioning the effected services, and restarting trading. Performance remained sluggish for a few hours afterward, but we have successfully scaled one of the two main systems responsible for the slowdown. Our readings show socket latency well below the 24hr average at the time of writing.
The second system will be migrated shortly and will mitigate this problem going forward. We are still experiencing slowdowns on the engine layer that will result in the dreaded “Overload” messages for many users despite the above scaling. We are very aware of the issue and have a team dedicated to solving this problem as quickly as possible. The solution in progress involves rearchitecting certain systems integral to BitMEX operation, so extensive planning and testing is in progress. We will update you once this work is complete.
—
In other news, a number of major updates to the frontend have hit, including a new popout chat box. Some users may have stuck caches on the chart due to an upgrade. If you are seeing this, please full-screen the chart, then close the fullscreen window. If that doesn’t work, please clear your browser cache.