Sometimes when a complex story takes us by surprise or knocks us back on our heels, it pays to revisit the events in a somewhat linear fashion. Here’s a brief timeline of what we know leading up to last week’s mass-hack, when hundreds of thousands of Microsoft Exchange Server systems got compromised and seeded with a powerful backdoor Trojan horse program.

When did Microsoft find out about attacks on previously unknown vulnerabilities in Exchange?

Pressed for a date when it first became aware of the problem, Microsoft told KrebsOnSecurity it was initially notified “in early January.” So far the earliest known report came on Jan. 5, from a principal security researcher for security testing firm DEVCORE who goes by the handle “Orange Tsai.” DEVCORE is credited with reporting two of the four Exchange flaws that Microsoft patched on Mar. 2.

Reston, Va.-based Volexity first identified attacks on the flaws on Jan. 6, and officially informed Microsoft about it on Feb. 2. Volexity now says it can see attack traffic going back to Jan. 3. Microsoft credits Volexity with reporting the same two Exchange flaws as DEVCORE.

Danish security firm Dubex says it first saw clients hit on Jan. 18, and reported their incident response findings to Microsoft on Jan. 27.

In a blog post on their discovery, Please Leave an Exploit After the Beep, Dubex said the victims it investigated in January had a “web shell” backdoor installed via the “unifying messaging” module, a component of Exchange that allows an organization to store voicemail and faxes along with emails, calendars, and contacts in users’ mailboxes.

“A unified messaging server also allows users access to voicemail features via smartphones, Microsoft Outlook and Outlook Web App,” Dubex wrote. “Most users and IT departments manage their voicemail separately from their email, and voicemail and email exist as separate inboxes hosted on separate servers. Unified Messaging offers an integrated store for all messages and access to content through the computer and the telephone.”

Dubex says Microsoft “escalated” their issue on Feb. 8, but never confirmed the zero-day with Dubex prior to the emergency patch plea on Mar. 2. “We never got a ‘real’ confirmation of the zero-day before the patch was released,” said Dubex’s Chief Technology Officer Jacob Herbst.

How long have the vulnerabilities exploited here been around?

On Mar. 2, Microsoft patched four flaws in Exchange Server 2013 through 2019. Exchange Server 2010 is no longer supported, but the software giant made a “defense in depth” exception and gave Server 2010 users a freebie patch, too. That means the vulnerabilities the attackers exploited have been in the Microsoft Exchange Server code base for more than ten years.

The timeline also means Microsoft had almost two months to push out the patch it ultimately shipped Mar. 2, or else help hundreds of thousands of Exchange customers mitigate the threat from this flaw before attackers started exploiting it indiscriminately.

Here’s a rough timeline as we know it so far:

• Jan. 5: DEVCORE alerts Microsoft of its findings.
• Jan. 6: Volexity spots attacks that use unknown vulnerabilities in Exchange.
• Jan. 8: DEVCORE reports Microsoft had reproduced the problems and verified their findings.
• Jan. 11: DEVCORE snags proxylogon.com, a domain now used to explain its vulnerability discovery process.
• Jan. 27: Dubex alerts Microsoft about attacks on a new Exchange flaw.
• Jan. 29: Trend Micro publishes a blog post about “Chopper” web shells being dropped via Exchange flaws (but attributes cause as Exchange bug Microsoft patched in 2020)
• Feb. 2: Volexity warns Microsoft about active attacks on previously unknown Exchange vulnerabilities.
• Feb. 8: Microsoft tells Dubex it has “escalated” its report internally.
• Feb. 18: Microsoft confirms with DEVCORE a target date of Mar. 9 (tomorrow) for publishing security updates for the Exchange flaws. That is the second Tuesday of the month — a.k.a. “Patch Tuesday,” when Microsoft releases monthly security updates (and yes that means check back here tomorrow for the always riveting Patch Tuesday roundup).
• Feb. 26-27: Targeted exploitation gradually turns into a global mass-scan; attackers start rapidly backdooring vulnerable servers.
• Mar. 2: A week earlier than previously planned, Microsoft releases updates to plug 4 zero-day flaws.
• Mar. 2: DEVCORE researcher Orange Tsai (noted for finding and reporting some fairly scary bugs in the past) jokes that nobody guessed Exchange as the source of his Jan. 5 tweet about “probably the most serious [remotely exploitable bug] I have ever reported.”
• Mar. 3: Tens of thousands of Exchange servers compromised worldwide, with thousands more servers getting freshly hacked each hour.
• Mar. 4: White House National Security Advisor Jake Sullivan tweets about importance of patching Exchange flaws, and how to detect if systems are already compromised.
• Mar. 5, 1:26 p.m. ET: In live briefing, White House press secretary Jen Paski expresses concern over the size of the attack.
• Mar. 5, 4:07 p.m. ET: KrebsOnSecurity breaks the news that at least 30,000 organizations in the U.S. — and hundreds of thousands worldwide — now have backdoors installed.
• Mar. 5, 6:56 p.m. ET: Wired.com confirms the reported number of victims.
• Mar. 5, 8:04 p.m. ET: Former CISA head Chris Krebs tweets the real victim numbers “dwarf” what’s been reported publicly.
• Mar. 6: CISA says it is aware of “widespread domestic and international exploitation of Microsoft Exchange Server flaws.”
• Mar. 7-Present: Security experts continue effort to notify victims, coordinate remediation, and remain vigilant for “Stage 2” of this attack (further exploitation of already-compromised servers).

Update, 12:11 p.m. ET: Correct link to Dubex site (it’s Dubex.dk). Also clarified timing of White House press statement expressing concern over the number of the Exchange Server compromises. Corrected date of Orange Tsai tweet.

Tags: Chopper web shell, DEVCORE, Dubex, Orange Tsai, proxylogon.com, Volexity

This entry was posted on Monday, March 8th, 2021 at 11:05 am and is filed under A Little Sunshine. You can follow any comments to this entry through the RSS 2.0 feed. You can skip to the end and leave a comment. Pinging is currently not allowed.