We understand many of you are concerned about the email disclosure which happened over this weekend and no doubt have many questions.
Our teams across the world have been working around the clock to protect your account security and make sure we are back on course. Our support team has already assisted many of our users and we are continuing to establish contact with everyone. This is a staggered process, to ensure that the proper processes are all followed, the delivery is logistically smooth and that all underlying security concerns are appropriately covered. If you have not yet heard from us already, you will do very soon.
We would like to apologise unreservedly for the concern this has caused. Below contains further information about what happened, how we can assist you and some steps that you can take to improve your protection.
What happened?
On Friday, November 1 at 06:00 UTC, many of our users received an email which contained the email addresses of other users in the “To:” field. This was a general email update to our users about upcoming changes to the weighting of our indices. As a result, many BitMEX user email addresses, including a large number of inactive addresses, were disclosed to other users in small batches. No other information was disclosed.
BitMEX is a global business that sends emails to many different email providers. Email deliverability itself is a multi-layered problem, involving decades of work in building sender reputation systems and automatic spam filters. Unfortunately, this makes the job of large services such as BitMEX difficult at times: we only send mass emails to all users on rare occasions. We intend to keep a high signal-to-noise ratio, and only send emails when absolutely necessary.
The index change we published on 1 Nov was of sufficient importance – it will impact pricing of all of our products – that we felt it necessary to inform all our users about it. However, bulk mail sends such as this are a difficult and complex undertaking when it’s on a global scale, to all recipients. Some mail servers, especially the global arms of large brands like Yahoo and 163, have very tight controls that are often triggered when we send large amounts of mail. For system notifications such as withdrawals, password resets, and liquidations, it is imperative that the customer receives mail dependably.
To remedy this, we built an in-house system to handle the necessary rendering, translation, staging, and piecemeal (as not to trigger rate limits) sending of important email. BitMEX has not sent an email to every customer at once since 2017, and much has changed since then. When we initiated the send, it became clear that it would take upwards of 10 hours to complete, and there was a desire on the team to ensure users received the same material information on a more reasonable timescale.
To handle this, the tool was quickly rewritten to send single SendGrid API calls in batches of 1,000 addresses. Unfortunately, due to the time constraints, this was not put through our normal QA process. It was not immediately understood that the API call would create a literal concatenated “To:” field, leaking customer email addresses. As soon as we became aware, we immediately prevented further emails from being sent and have addressed the root cause. Since then we have been aiding all who have been affected as best we can and mitigating the damage to contain the leak.
BitMEX is a company that takes engineering seriously, and we are disappointed that this lapse in care has resulted in unwanted disclosure for our customers. We believe that processes, not engineers, are to blame for these failures. Our processes failed here. We are working around-the-clock to revamp them and to ensure that even the simplest-looking code changes are put under strict review.
Additionally, and unrelated to this action, the BitMEX Twitter account was accessed by an external individual. The account was back under BitMEX control within 6 minutes and re-secured, and the event is under security review.
Beyond email addresses, no personal or account information has been disclosed. At no point were any of our core systems at risk.
Who was affected?
Most BitMEX users were affected by this action. You can self-diagnose your exposure with the following steps:
- If you received an email about the index change, and your email was the only one listed in the “To:” field, you were not affected.
- If you received the index change email, and you saw multiple addresses in the To: field, you were affected.
- If you did not receive an index change email, you may have been affected and we still recommend that you follow steps below to improve your protection online. While the system was cut-off before it completed entirely, many recipients began marking BitMEX emails as spam, understandably out of hope that it would stop further emails. This caused deliverability issues at some hosts, causing mail not to be delivered. Unfortunately, someone else in your batch may have received the email, exposing your email address.
- The deliverability issues caused by the spam reporting caused some follow-up password resets to be delayed for several hours. Our operation teams remedied this by 06:00 UTC on Nov 2.
What are we doing to help?
After the discovery of the disclosure, BitMEX employees have since worked through the nights and days to reduce risk for users. We are aware that many users reuse email addresses across services. This, combined with a very human tendency to reuse passwords, meant that many of our users may have been at risk due to password hash dumps on other platforms, even ones unrelated to crypto.
For this reason, we took the following steps after we notified our users of the disclosure:
- Our Security and Support teams began enhanced monitoring of access patterns to flag accounts with suspicious activity after the disclosure. This led to several account password resets and human review with Support.
- At 13:00 UTC on the day of the email, we conducted additional checks during our usual human review of withdrawals. We identified criteria that could be indicative of a compromise given the circumstances. We cancelled requests from accounts that (i) did not have two-factor authentication, (ii) were withdrawing to a previously unseen Bitcoin address, (iii) were submitted with previously unseen new IP address, and (iv) were made after the email address disclosure had occurred. All other withdrawal requests were unaffected. These actions were taken in the interest of protecting our users and those affected have already been contacted.
- As it became clear that several groups were working to collate BitMEX email addresses in order to attempt to compromise them, BitMEX engineers forced a password reset for all users with balances and without Two-Factor devices. Affected users were notified via email (after a thorough QA review and retrospective on the original bug).
- BitMEX Support (contact here) is working shifts with extra agents, continuing to handle customer requests to change email addresses, answer questions, and provide security assessment and advice.
If you are concerned about your personal exposure, on BitMEX or on any other platform, the best thing you can do is to enable Two-Factor Authentication on all critical services. Start with your email address first. We have published advice on this topic, as have others, including this very helpful guide by Paul Stamatiou.
BitMEX engineering teams are working on new features to increase the number of security keys supported by the platform, to improve the signal of account notifications, and to give users more tools to avoid and contain account takeovers.
Do I need to do anything?
Although no-one’s personal information or account details beyond their email address were disclosed, as best practice, we recommend that you:
- Please be vigilant against phishing attempts. Emails from BitMEX are sent from “support@bitmex.com” and “noreply@bitmex.com”. We recommend adding these addresses to your contacts list. We will never ask for your password.
- Note that BitMEX will never ask you to transfer any funds. The only way to fund your BitMEX account is to send Bitcoin to your unique BitMEX deposit address. Your unique BitMEX deposit address will begin with “3BMEX” or “3BitMEX” and can be found on the deposit page of your BitMEX account.
- Please take note of our official BitMEX communications channels. Only instructions provided via these avenues should be observed.
- Protect your account by using strong and unique passwords; enabling Two-Factor Authentication (2FA) for all of your accounts (both BitMEX and personal); and to use a password manager.
We want to reassure you that beyond email addresses, no personal or account information has been disclosed. At no point during this issue were any of our systems at risk, and they remain secure, as we continue to take measures to enhance our security. Your privacy and security remain our top priority.
In the meantime, if you need any immediate assistance, please contact Support via our contact form.
Vivien Khoo,
Deputy Chief Operating Officer
