Important Security Advisory
Tl;dr: A botnet is attempting known email/password combinations from a large data leak on Bitcoin sites. Use Two-Factor Auth (2FA) and don’t reuse passwords. BitMEX services have not been compromised.
About four weeks ago, I was rudely awakened in the early morning by our uptime alarms clanging that the website was going up and down. Dozens of emails flooded my inbox: page loads were sometimes taking 5s+, or not loading at all.
Nobody likes this; I jumped out of bed and logged in. The rest of the team informed me that the site had been underperforming for a few minutes, but it had just gotten worse. Dramatically worse.
I opened up the logfiles to see tens of thousands of lines of this:
Jun 07 20:30:57 113.xxx.xxx.xxx - "POST /login {"email":"xxx@vp.pl"}" 401 79b
Jun 07 20:30:57 113.xxx.xxx.xxx - "POST /login {"email":"xxx@ntlworld.com"}" 401 79b
Jun 07 20:30:57 113.xxx.xxx.xxx - "POST /login {"email":"xxx@126.com"}" 401 79b
Jun 07 20:30:57 112.xxx.xxx.xxx - "POST /login {"email":"xxx@o2.pl"}" 401 79b
Jun 07 20:30:57 112.xxx.xxx.xxx - "POST /login {"email":"xxx@ntlworld.com"}" 401 79b
Jun 07 20:30:57 112.xxx.xxx.xxx - "POST /login {"email":"xxx@gmx.co.uk"}" 401 79b
Jun 07 20:30:57 39.xxx.xxx.xxx - "POST /login {"email":"xxx@btinternet.com"}" 401 79b
Jun 07 20:30:57 46.xxx.xxx.xxx - "POST /login {"email":"xxx@ntlworld.com"}" 401 79b
Jun 07 20:30:57 180.xxx.xxx.xxx - "POST /login {"email":"xxx@onet.eu"}" 401 79b
Jun 07 20:30:57 124.xxx.xxx.xxx - "POST /login {"email":"xxx@yahoo.fr"}" 401 79b
Jun 07 20:30:57 14.xxx.xxx.xxx - "POST /login {"email":"xxx@wanadoo.fr"}" 401 79b
Jun 07 20:30:57 180.xxx.xxx.xxx - "POST /login {"email":"xxx@uwclub.net"}" 401 79b
Jun 07 20:30:57 49.xxx.xxx.xxx - "POST /login {"email":"xxx@o2.pl"}" 401 79b
Jun 07 20:30:57 39.xxx.xxx.xxx - "POST /login {"email":"xxx@op.pl"}" 401 79b
A botnet.
They were hitting us hard, but these didn’t correspond to any of our registered accounts. It was spray & pray. We were seeing tens of thousands of these requests every minute, coming from all over the world. There was little common pattern between them, aside from a common Chrome User-Agent (which was too common to block outright) and a propensity to just log in, over and over and over again.
Staying Online
The first order of business was to get the site stable again. While trading was continuing unhampered, and users who were already in were fine, the login page and initial dashboard were up and down. Thankfully, we built for this situation and could simply scale out more instances. I spun up a few large instances and added them to the rotation, and within 5 minutes we were rock-solid again.
While we were prepared for some types of abuse, others were unfortunately still vulnerable. I spent the better part of that day building and deploying a strategy to control this traffic. By just after lunchtime a process was in place. Watching our cluster’s CPU load, I scaled down the extra instances and felt good about that day’s work.
Origins
Where was this list coming from? I emailed a few other exchanges we’re friendly with. Not everyone I asked was seeing it, but the general rumor was that this could have been from the recent LinkedIn hack, which had a number of unsalted hashes. Lots of motivated parties have the resources to crack the lion’s share of those passwords. There are likely to have been other sources as well. We looked up a few dozen emails on HaveIBeenPwned, which aggregates identities compromised by many recent hacks.
It is human nature to reuse credentials, and attackers take advantage of this. Once an email/password combination is stolen, it is tried on as many sites as possible. A Bitcoin exchange is an obvious target, as are email providers.
With the traffic under control, the attempts slowed down to a trickle, essentially indistinguishable from legitimate traffic.
Users Hit
I received a reply email to one of our login notifications. The user claimed he hadn’t logged into the account in months.
Looking at the logs it was evident: they actually hit one. The account didn’t have any funds, but I immediately reset the password. The login was successful, but the attacker behind the botnet didn’t do anything with it. Maybe there wasn’t really anyone on the other side.
I started typing this blog post when another user piped up. He had received a login notification, then his positions closed. He then received an email asking for withdrawal confirmation… then an email stating his withdrawal had been confirmed. There was someone on the other end waiting this time.
They had control of the user’s email, and they knew our site well enough to execute these steps quickly. There is a real threat: and if they’re hitting BitMEX, they are likely hitting dozens of other Bitcoin-accepting sites.
A Sidenote: Manual Review
This is a prime example of why it is A Good Thing to involve manual review in Bitcoin withdrawals. We were able to lock the account and cancel the withdrawal well before it had any chance of going out and the funds being lost forever. The user quickly changed his email password, reset his BitMEX password, and set up 2FA.
Thwarting this particular attack was a combination of caution and luck, but don’t rely on services you use being able to catch this kind of thing every time.
Protecting Your Accounts
Take your account security seriously. If you have Bitcoin on any website, use a unique password and use 2FA.
Email notifications of account actions are unreliable. On many sites, they can be turned off. Even if they can’t, if an attacker gains access to your email account, it is trivial to set up an automatic filter that will mark new messages from a service as read or delete them automatically.
If you reuse passwords, your accounts could be drained without any notice.
Use Two-Factor Auth. We are continuing to monitor for this behavior and have sent out an email to all active users without 2FA. As time goes on, it is all but guaranteed we will see more of these attacks.
BitMEX supports Two-Factor Auth via the following providers:
Support for U2F and BitID is in the works.
As always, if you see any unusual activity on your account, email us immediately by replying to any BitMEX email or at support@bitmex.com.