Microsoft has been eviscerated for a “cascade of security failures” that gave Chinese hackers unbridled access to any Exchange email in the world – they downloaded 60,000 from the State Department alone.
The incident, covered by The Stack here, occurred after the group, known as “Storm-0558”, was found in 2023 to have stolen an all-powerful Microsoft cryptographic key that was inadequately protected in “legacy” infrastructure and which had not been “rotated” for seven years.
CSRD report roasts Microsoft's security culture
The US’s independent Cyber Safety Review Board (CSRD) has a direct line to the Secretary of Homeland Security and the President. Its members include CIOs and CISOs at the highest levels of the US government.
In an utterly damning report published on April 2, after an extensive investigation, the CSRD found that: “Microsoft operational and strategic decisions… collectively point to a corporate culture that deprioritized both enterprise security investments and rigorous risk management”.
The CSRD also found that:
- “Microsoft’s security culture was inadequate” – and it had also “not sufficiently prioritized rearchitecting its legacy infrastructure to address the current threat landscape.”
- It was still manually rotating consumer security keys – but “stopped key rotation entirely in 2021, following a major cloud outage linked to the manual rotation process." (The stolen key had not been rotated since 2016 and was inadequately protected.)
- Its theft "permitted Storm-0558 to gain full access to essentially any Exchange Online account anywhere in the world."
- A September 6, 2023 post-mortem by Microsoft on the breach was misleading and speculative – but it took six months to update it, after coming under pressure from the CSRD which described itself as "concerned with Microsoft’s public communications after the incident."
See also: Microsoft’s “top notch” China hack post-mortem was "troubling" speculation
- It STILL doesn’t know how the key was stolen and is investigating 46 hypotheses: It has “NO evidence or logs showing the stolen key’s presence in or exfiltration from a crash dump” as it had earlier suggested in the post-mortem.
- Microsoft’s “CEO and Board should develop, and share publicly, a plan with specific timelines to make fundamental, security-focused reforms across the company and its full suite of products, and then hold leaders at all levels of the company accountable for its implementation.”
See the detailed CSRD report here (pdf).
Saying that other cloud providers had significantly more robust security, the Board urged Redmond’s leadership bluntly to “deprioritize feature developments across the company’s cloud infrastructure and product suite until substantial security improvements have been made."
Among the other victims were the US Commerce Department and numerous "high-profile individuals" in the UK as well as several organisations that the report does not name. The CSRD also criticised Microsoft's victim outreach, giving the example of Congressman Don Bacon, "a prominent congressional voice on national security matters."
Microsoft "first noticed outreach to Congressman Bacon about the intrusion was an email prompting him to change his password...
"Congressman Bacon thought the password change email looked strange and was potentially fraudulent, so he changed his password directly rather than using the link provided in the notification instructions... Microsoft did not advise Congressman Bacon to take any action to protect his account beyond the one email recommending a password change."
Microsoft's inadequate "core digital identity systems".
Highlighting a string of security failures, the CSRD took particular aim at Microsoft's approach to authentication, noting that:
Stateful tokens: Microsoft’s authentication system accepted a token that it had not issued. By storing records in a database when tokens are issued and validating against that database at access time, CSPs may enforce that only tokens issued by the CSP can access customer data. Note: this approach is not possible for use with third-party services reliant on an IDP maintained by a cloud provider.
Automated frequent key rotation: Microsoft paused manual key rotations for its MSA system in 2021 but did not remove the 2016 MSA key. By rotating encryption keys frequently (e.g., monthly) and in an automated manner with monitoring of rotation systems, CSPs can ensure that the blast radius of a compromised key is limited in duration.
Per customer keys (key scope): Microsoft had a single key that signed tokens for all consumer, and due to the validation flaw, enterprise customers. Tying encryption keys to customer tenancy would limit the scope
of key compromise.
Bound tokens: Microsoft’s identity system used bearer tokens that did not require any proof of possession, thus making the tokens more vulnerable to replay attacks. By digitally binding tokens to specific requests or network sessions, token theft and token replay attacks can be eliminated. While this incidentdemonstrates the risks of key compromise, some victims and responders spent significant time investigating bearer token replay attacks to which not all CSPs are vulnerable.
Common authentication libraries: Microsoft used a variety of different client libraries to verify tokens across different systems. This diversity complicated implementing uniform, and correct, validation behavior, as well as made the remediation efforts much more complex and time sensitive. By ensuring that all CSP services use the same authentication libraries, CSPs can more effectively enforce consistent token validation behavior and authorization policy.
Secure key storage: While Microsoft separated the organization and production environments, this incident illustrated that Microsoft insufficiently protected MSA system key material. By storing key material
in isolated systems and leveraging, where feasible, technologies such as dedicated HSMs, the risk of key compromise can be reduced. The Board recognizes that in some situations and levels of scale, traditional HSM technology may not be viable but believes that the core idea of isolated key storage with minimal key release is appropriate.
Linkable tokens: The relationship between the tokens used in this incident was not exposed in logs made available to customers, making them difficult to track. By linking all tokens derived from a single root authentication event together and exposing this linking to their customers in logs, CSPs and customers can better track and discover identity-related attacks and respond, including in an automated way.
Proprietary data use in token generation algorithm: Some CSPs inject proprietary data into their generated authentication tokens, which they can use to differentiate between tokens that their own systems generated and those generated by malicious third parties. While one cannot rely on the fact that the adversary would not detect and reproduce such behavior, it can nevertheless prove potentially helpful as a “canary in the coal mine” alert that the CSP is observing tokens that had not been generated by its own code," the CSRD highlighted.
Microsoft has since promised a sweeping security overhaul, led by Amazon veteran Charlie Bell under its "Secure Future Initiative." (The Stack has contacted Redmond for comment on the CSRD report.) But a year later it went attacked again. The CSRD concluded that that it is "convinced that Microsoft should address its security culture" and needs a "a security-focused corporate culture of accountability, which starts with the CEO."
Roger Cressey, a former senior national security official of the Clinton and Bush administrations, told The Stack: "Make no mistake, this is Microsoft’s Boeing moment; there must be real cultural and leadership changes.
"The U.S. government needs to reconsider its relationship with the company that dominates the public sector IT market but continually fails to fulfill its security obligations. At a minimum, the CSRB report presents a clear case for putting a hold on any new contract awards to Microsoft until it demonstrates that it can be a dependable partner to the federal government.
"The Administration deserves kudos for its blunt assessment of Microsoft’s security failures; the next step is to use the government's procurement power to demand accountability and change from Microsoft.”
A Microsoft spokesperson told The Stack: "We appreciate the work of the CSRB to investigate the impact of well-resourced nation state threat actors who operate continuously and without meaningful deterrence. As we announced in our Secure Future Initiative, recent events have demonstrated a need to adopt a new culture of engineering security in our own networks. While no organization is immune to cyberattack from well-resourced adversaries, we have mobilized our engineering teams to identify and mitigate legacy infrastructure, improve processes, and enforce security benchmarks. Our security engineers continue to harden all our systems against attack and implement even more robust sensors and logs to help us detect and repel the cyber-armies of our adversaries. We will also review the final report for additional recommendations."
