SSD Cloud Hosting provider, Linode, has forced all customer passwords to be reset after discovering two Linode user credentials on an external machine. It made the announcement on it status page, and in a blog post. All customers will be prompted to set a new password when they next log in.
The user credentials on the external machine were discovered after a security investigation into the unauthorized access of three separate accounts. This implies that there was a security breach at some point that allowed the details to be read from a Linode database. The information found consisted of a user table containing usernames, emails, hashed passwords as well as encrypted two-factor seeds.
You can see an extract of the announcement below:
The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We’ve retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings.
Linode could not determine whether the same group or persons was behind the two malicious acts, and they have not been contacted by anyone claiming to be responsible.
The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services.
It is unclear whether this is related to the DDOS attack they have been suffering since Christmas Day, but they did state on the 31st December 2015 that:
It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business.
They have been proactive in hardening themselves against the majority of the forms of the attack, but they confirm that the vectors of the attack have changed to adapt to their actions. Linode confirmed that they would be sharing a technical post-Mortem about the incident, but at present, none has been made. It is unclear whether this attack is still ongoing, although the status post is still marked as active.
Having your servers compromised is never ideal, but we must give credit to Linode to proactively contacting their customers, as well as taking steps quickly to ensure that login credentials were reset. They have shared a lot of information about the breach and been very upfront with their customers, which is always a good thing.
Update 11th January 2016
We reached out to Linode with some further questions about the breach but are still awaiting a response.
In the meantime, however, further information has come to light that indicates that the scale of this breach is much more serious than we initially thought. It appears the breach has been in place since July or earlier (we will clarify this further in a moment, with reference to sources).
Furthermore, with Linode acting as the Data Center partner for some reputable hosting providers, such as WP Engine, the scale of the breach could be significant.
Connection to PagerDuty Breach — July 2015
In a post made on Hacker News on the 5th January 2015, a PagerDuty employee (and ex-Linode Employee) spoke out about a data breach they suffered in July via the Linode Manager. In this case, the hackers used access from the Linode Manager to reset the root passwords on several systems. Fortunately, for PagerDuty, they were alerted to the breach within 60 minutes after the first compromise and deactivated the relevant user account. After investigating the breach, this is what they found:
In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side.
We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise.
Ultimately, PagerDuty migrated away from Linode due to the breach.
WP Engine — 9th December 2015
We are writing today to let you know that we learned of an exposure involving some of our customers’ credentials. Out of an abundance of caution, we are proactively taking security measures across our entire customer base.
While we have no evidence that the information was used inappropriately, as a precaution, we are invalidating the following five passwords associated with your WP Engine account. This means you will need to reset each of them. Instructions for how to reset these passwords are at the bottom of this email.
In their final update on the matter on the 31st December 2015, they confirmed that the point of entry was through one of their cloud infrastructure partners. Knowing that Linode is one of their partners, and it coinciding broadly within the same time scale as the problems announced by Linode it is possible that the two are related, although we have no independent confirmation of this. It is, of course, possible these are completely unrelated.
Update 12th January 2015
You can read Linode's response to us here.