What actually failed in a breach?

What failed was a credential that still worked

In three 2023 incidents at Microsoft, Okta and Cloudflare, what failed was a credential that still worked when its owner didn’t expect it to. Cloudflare says it set out to rotate every credential exposed in Okta’s breach and missed four out of thousands, and that those four let an intruder in [6].

On 29 November 2023, Okta changed the story of its own breach for the second time. Its security team had been rebuilding, by hand, the reports the intruder had run. One report the intruder downloaded proved much larger than the copy from Okta’s first investigation. The intruder had run it with its filters removed, so the file held the names and email addresses of every user of Okta’s support system [5].

We think a breach write-up should start with the credentials that could still act. Below we read each case that way, and keep apart what each source saw, what it concluded and what it left open.

Storm-0558: Microsoft revoked the key without knowing how it leaked

Microsoft’s case shows that a team can shut a known path before it learns how the path opened. In May and June 2023, the espionage group Storm-0558, assessed to be linked to China, broke into Exchange Online. The US Cyber Safety Review Board reported that it reached the mailboxes of 22 organizations and more than 500 individuals [1].

Microsoft’s July 2023 account had said Storm-0558 accessed email at approximately 25 organizations [2]. The two figures may differ in timing or scope, and a careful account keeps both, each with its date.

The alarm came from the US Department of State. It had bought Microsoft’s premium audit logging, without which the MailItemsAccessed log wasn’t available, and built its own alert rule on that log. The Board heard that few organizations analyzed that log in detail [1].

The group forged its way in with sign-in tokens signed by a consumer signing key that Microsoft had made in 2016. That key should only have been able to sign tokens for consumer email. A bug in Microsoft’s token checks let consumer tokens work for business accounts too. The Board concluded that the combination gave the actor full access to essentially any Exchange Online account in the world [1].

The key should have been dead by then. The key was due to be retired about two years before the intrusion, but Microsoft delayed its removal [1]. Microsoft’s July 2023 account called it an inactive key [2].

Microsoft, the Board found, had stopped rotating consumer signing keys entirely after a major cloud outage linked to the manual rotation process. Nor did it have an alert for the age of active keys. Had Microsoft kept rotating, automated the process or flagged aging keys, the Board wrote, the 2016 key would not have been valid in 2023. In the Board’s words, the intrusion “was preventable and should never have occurred” [1].

As of the Board’s report, Microsoft did not know how or when Storm-0558 obtained the key. In September 2023, Microsoft had published a blog post naming a crash dump as the most probable route. The Board found that Microsoft had no evidence or logs showing the key in a crash dump, or leaving one [1].

The correction came late. Microsoft acknowledged to the Board in November 2023 that the post was inaccurate, yet it corrected the post only in March 2024, after the Board’s repeated questions. The Board listed that delay among the grounds for its verdict on Microsoft’s security culture [1].

Microsoft didn’t need to know how the key leaked to find where it had been used. Its identity system was no longer signing tokens with the key. So once Microsoft had found and revoked it, every signature made with it was the attacker’s. That let Microsoft pick out the actor’s activity with high confidence [1].

We’d tell anyone cleaning up a breach to list every key that can still sign before trying to learn how one leaked. The Board notes this was not the first time Storm-0558 had shown interest in stealing authentication keys. Industry researchers link it to the RSA SecurID breach, in which the attacker stole the secret keys used to generate SecurID login codes [1].

Okta’s account widened twice after its first notice

Okta’s case shows that the scope of a breach depends on where you look. Its first search missed the intruder’s downloads, because they left a different kind of log entry from the one investigators searched for [4].

Okta’s first public notice came on 20 October. It said someone had used a stolen credential to reach its customer support case-management system, which Okta described as separate from the production Okta service. It also warned that the HAR files customers upload for troubleshooting can contain cookies and session tokens that let an attacker pose as a real user [3]. Calling the support system separate didn’t make every file inside it harmless.

The second notice, on 3 November, said the intruder had used a service account stored in the support system itself. An employee had signed in to a personal Google profile in Chrome on an Okta-managed laptop and saved the account’s username and password there. Okta judged that the most likely route to exposure was a hack of that personal account or device [4].

The same notice dated the intruder’s access from 28 September to 17 October. The intruder reached files associated with 134 customers, fewer than one percent of Okta’s customers. Tokens in some HAR files had been used to hijack the real Okta sessions of five customers. The notice’s timeline shows Okta had identified the 134 customers by 17 October, and had told customers with registered security contacts whether they were affected before its first public notice [4].

Okta’s early investigation focused on access to support cases. Opening a file attached to a case writes one kind of log event. Going straight to the support system’s Files tab writes a different event, with a different record ID. So for two weeks of active investigation, Okta didn’t find the suspicious downloads in its logs [4].

By Okta’s 29 November account, the downloaded report covered users at all Workforce Identity Cloud and Customer Identity Solution customers. It left out customers in Okta’s FedRAMP High and DoD IL4 environments, which use a separate support system the intruder didn’t reach [5].

Many users of the support system, Okta noted, are Okta administrators. For 99.6% of users, the only contact details recorded were full name and email address. Okta said it had no direct evidence that the information was being exploited. But it assessed a higher risk of phishing and social engineering aimed at those users [5].

The two statements measure different things. The 134-customer figure covers accessed files [4], and the later report covers support users [5].

Okta’s timeline names Cloudflare as the adversary’s fifth and final target [4]. Cloudflare’s case shows what a missed credential can do a month later.

Cloudflare says four missed credentials out of thousands let an intruder in

Cloudflare’s case shows that recovery is an inventory of every credential that might still work. On Thanksgiving Day 2023, Cloudflare found a threat actor on its self-hosted Atlassian server. The intruder had come in with one access token and three service-account credentials taken in the Okta breach the month before [6].

Cloudflare says it hadn’t rotated those four because it wrongly believed they were unused. Each gave different access. A Moveworks service token gave remote access to Cloudflare’s Atlassian system, and a Smartsheet service account had administrator access to its Jira ticket system. A Bitbucket service account reached the system that holds its source code [6].

The fourth opened an AWS environment that powers the Cloudflare Apps marketplace. Cloudflare says that environment was walled off from its global network and held no customer or sensitive data [6].

The intruder set up lasting access to the Atlassian server and reached source-code repositories, some of which Cloudflare treats as stolen. It also tried to reach a console server in a new São Paulo data center that wasn’t yet in production, and failed. Cloudflare says no customer data or systems were affected, and that all of the actor’s access ended on 24 November [6].

Cloudflare’s response went far wider than the access it saw. It called the effort Code Red and moved a large part of its technical staff onto it [6].

• It rotated every production credential, more than 5,000 of them.
• It ran forensic triage on 4,893 systems.
• It reimaged and rebooted every machine in its global network.
• It sent the São Paulo equipment back to its manufacturers.

Cloudflare’s stated reason was to keep the attacker from using technical information about how its network operates to get back in. It reports that CrowdStrike’s independent assessment backed its own investigation and turned up nothing it had missed, an assessment we know only through Cloudflare [6].

In all three cases, a working credential got past the login check

The credentials came in four kinds: a forged token, a service account’s stored password, a session token from a support file and a credential left active after a supplier’s breach. Each let someone sign in, or seem to. So the useful question is what each still-working credential could do from there.

That weakens the familiar claim that MFA would have stopped it. MFA guards the front door, and Okta and Cloudflare both say it still matters. Okta’s advice after its incident was that all customers use MFA, at least for administrator access [5]. Cloudflare credits its access controls, firewall rules and hard security keys, enforced by its own Zero Trust tools, with limiting how far the intruder could move [6].

The front door can fail too. In multiple engagements, Mandiant saw a group it calls UNC3944 talk service desks into resetting passwords and bypassing MFA for privileged accounts. Mandiant’s 2024 account doesn’t attribute any of our three cases to the group [7].

Mandiant also saw the group, once inside, give a hacked account access to every application in a victim’s single sign-on platform, and copy data out of SaaS applications with cloud sync tools. It says firewalls and network flow sensors are ineffective at detecting such transfers [7].

Owning a credential is a weak reason to trust it. NIST’s zero-trust architecture makes a related point. It grants no implicit trust to assets or user accounts based solely on their network location, or to an asset because the enterprise owns it [8]. Cloudflare owned its four credentials and believed them unused, and they still let an intruder in [6].

Each row starts from what could still act and ends with a question a cleanup team can act on this week.

Check this week which exposed credentials still work

If a supplier has told you its systems were breached, check this week which of the credentials it could have exposed still work. At Cloudflare, four missed credentials were enough [6].

Then start the write-up with whatever could still act after the first containment step, and say what it could reach. Stitching sources into one smooth story can create a claim that neither source makes, so keep three kinds of statement apart.

• A reported observation records what a named source says it saw or did.
• A source assessment records a source’s conclusion from its evidence.
• An unresolved question marks what the public record can’t answer.

Decide how far to go in recovery with the unresolved questions in view, and expect the record to change after you act. Okta’s account widened on 3 November [4] and again on 29 November [5]. The Board wrote that a cloud provider’s customers reasonably expect timely updates as an investigation evolves, including corrections to information that later proves wrong [1]. A team that wrote down what it didn’t know can see what the new facts change.

We haven’t tested our own method, the claim record, which labels each statement and keeps each revision. The test would give reviewers the same public evidence, frozen at the moment of a decision, some as a versioned claim record and others as an annotated timeline.

We’d drop the claim record and keep the timeline if it didn’t make containment choices easier to trace to the evidence, or if it slowed containment without cutting unsupported claims. We set out that comparison in full with our other proposed studies.