Episode 62: Exchange Hell

After talking about a hack that was caused by Microsoft’s cloud email service last week, we now look at the next infosec disaster in recent months: How Microsoft stood by as hundreds of thousands of their customers' on-premise Exchange mail servers got breached and totally owned.

After talking about the SolarWinds attack in the previous episode, on this week’s The Private Citizen, we’re looking at the recent mass hacking of Microsoft Exchange servers.

Errata #

I made a pretty significant mistake when talking about the Oxford/AstraZeneca SARS-CoV-2 vaccine (AZD1222), simply because I did not know better. This vaccine does not work at all like the well-known influenza vaccines. In fact, it works in ways much closer to the competing mRNA vaccine, in that it uses a viral vector. It’s also DNA-based, which, as far as I can tell, has several downsides that an mRNA vaccine has not.

If anyone knows why recipients of this vaccine are not classed as GMOs, please contact and enlighten me.

It’s 2021 and Exchange Is Still Shit #

At the beginning of this year, hundreds of thousands of Exchange mail servers were hacked. The first sign of an exploit was reported to Microsoft on 5 January, with the first data breach being observed a day later. Even though all hell soon started to break lose, it took Microsoft until 2 March to publicly acknowledge the attacks and release patches. Wikipedia has an overview.

A global wave of cyberattacks and data breaches began in January 2021 after four zero-day exploits were discovered in on-premises Microsoft Exchange Servers, giving attackers full access to user emails and passwords on affected servers, administrator privileges on the server, and access to connected devices on the same network. Attackers typically install a backdoor that allows the attacker full access to impacted servers even if the server is later updated to no longer be vulnerable to the original exploits. As of 9 March 2021, it was estimated that 250,000 servers fell victim to the attacks, including servers belonging to around 30,000 organisations in the United States, 7,000 servers in the United Kingdom, as well as the European Banking Authority, the Norwegian Parliament, and Chile’s Commission for the Financial Market (CMF).

On 2 March 2021, Microsoft released updates for Microsoft Exchange Server 2010, 2013, 2016 and 2019 to patch the exploit; this does not retroactively undo damage or remove any backdoors installed by attackers. On 12 March 2021, Microsoft announced the discovery of “a new family of ransomware” being deployed to servers initially infected, encrypting all files, making the server inoperable and demanding payment to reverse the damage.

We touched in the previous two episodes (61 & 60) on different kinds of security vulnerabilities and attacks and why email servers are an especially high value target – an attack on Microsoft’s cloud email service having been the catalyst that got the SolarWinds ball rolling.

How the Attack Was Discovered #

A quick timeline of this attack:

On 5 January 2021, security testing company DEVCORE made the earliest known report of the vulnerability to Microsoft, which Microsoft verified on 8 January. The first breach of a Microsoft Exchange Server instance was observed by cybersecurity company Volexity on 6 January 2021. By the end of January, cybersecurity company Volexity had observed a breach allowing attackers to spy on two of their customers, and alerted Microsoft to the vulnerability. After Microsoft was alerted of the breach, Volexity noted the hackers became less stealthy in anticipation of a patch.

And this is how it worked:

Hackers took advantage of four separate zero-day vulnerabilities to compromise Microsoft Exchange servers' Outlook Web Access (OWA), giving them access to victims' entire servers and networks as well as to emails and calendar invitations, only at first requiring the address of the server, which can be directly targeted or obtained by mass-scanning for vulnerable servers; the attacker then uses two exploits, the first allowing an attacker to connect to the server and falsely authenticate as a standard user. With that, a second vulnerability can then be exploited, escalating that user access to administrator privileges.

The final two exploits allow attackers to upload code to the server in any location they wish, that automatically runs with these administrator privileges. Attackers then typically use this to install a web shell, providing a backdoor to the compromised server, which gives hackers continued access to the server as long as both the web shell remains active and the Exchange server remains on.

Through the web shell installed by attackers, commands can be run remotely. Among the actions observed are the downloading of all emails from servers, downloading the passwords and email addresses of users as Microsoft Exchange stores these unencrypted in memory, adding users, adding further backdoors to affected systems, accessing other systems in the network that are unsusceptible to the original exploit, and installing ransomware. As patching the Exchange server against the exploit does not retroactively remove installed backdoors, attackers continue to have access to the server until the web shell, other backdoors and user accounts added by attackers are removed. After the patch was announced, the tactics changed when using the same chain of vulnerabilities.

Microsoft Exchange Server versions of 2010, 2013, 2016 and 2019 were confirmed to be susceptible, although vulnerable editions are yet to be fully determined.

On 2 March 2021, the Microsoft Security Response Center (MSRC) publicly posted an out-of-band Common Vulnerabilities and Exposures (CVE) release, urging its clients to patch their Exchange servers to address a number of critical vulnerabilities. On 15 March, Microsoft released a one-click PowerShell tool, The Exchange On-Premises Mitigation Tool, which installs the specific updates protecting against the threat, runs a malware scan which also detects installed web shells, and removes threats that were detected; this is recommended as a temporary mitigation measure, as it does not install other available updates.

The Original Attacker #

Surprisingly, this time, it’s supposedly not The Russians™.

Microsoft said that the attack was initially perpetrated by the Hafnium, a Chinese state-sponsored hacking group that operates out of China. Hafnium is known to install the web shell China Chopper.

No, not this Chopper (Source: Renée Brack, Hard Copy, 1992)

Microsoft identified Hafnium as “a highly skilled and sophisticated actor” that historically has mostly targeted “entities in the United States for the purpose of exfiltrating information from a number of industry sectors, including infectious disease researchers, law firms, higher education institutions, defense contractors, policy think tanks and NGOs.” Announcing the hack, Microsoft stated that this was “the eighth time in the past 12 months that Microsoft has publicly disclosed nation-state groups targeting institutions critical to civil society.”

I really don’t get what difference it makes who the attacker was and if it was state-sponsored or not. It certainly makes no difference to the victims of the attacks. Like with the SolarWinds attack, these insistent shouts of “state-sponsored!!!” seem to only serve to deflect blame from Microsoft, who has fostered an atrociously lax atmosphere around its product and completely fucked up deploying a patch in time. Once again.

After the Patch #

Of course, it didn’t stop with the Chinese.

On 2 March 2021, another cybersecurity company, ESET, wrote that they were observing multiple attackers besides Hafnium exploiting the vulnerabilities. Wired reported on 10 March that now that the vulnerability had been patched, many more attackers were going to reverse engineer the fix to exploit still-vulnerable servers. Analysts at two security firms reported they had begun to see evidence that attackers were preparing to run cryptomining software on the servers.

On 10 March 2021, security researcher Nguyen Jang posted proof-of-concept code to Microsoft-owned GitHub on how the exploit works, totaling 169 lines of code; the program was intentionally written with errors so that while security researchers could understand how the exploit works, malicious actors would not be able to use the code to access servers. Later that day, GitHub removed the code as it “contains proof of concept code for a recently disclosed vulnerability that is being actively exploited”. On 13 March, another group independently published exploit code, with this code instead requiring minimal modification to work; the CERT Coordination Center’s Will Dormann said the “exploit is completely out of the bag by now” in response.

Security company ESET identified “at least 10” advanced persistent threat groups compromising IT, cybersecurity, energy, software development, public utility, real estate, telecommunications and engineering businesses, as well as Middle Eastern and South American governmental agencies.

The attacks came shortly after the 2020 United States federal government data breach, which also saw the compromising of Microsoft’s Outlook web app and supply chain. Microsoft said there was no connection between the two incidents.

Yeah, except Microsoft’s stupidity.

The Hell That Is a Typical Exchange Server #

Why was the attack so successful?

Automatic updates are typically disabled by server administrators to avoid disruption from downtime and problems in software, and are by convention installed manually by server administrators after these updates are tested with the existing software and server-setup; as smaller organizations often operate under a smaller budget to do this in-house or otherwise outsource this to local IT providers without expertise in cybersecurity, this is often not done until it becomes a necessity, if ever. This means small and medium businesses, and local institutions such as schools and local governments are known to be the primary victims of the attack as they are more likely to not have received updates to patch the exploit. Rural victims are noted to be “largely on their own”, as they are typically without access to IT service providers.

The attack was discovered after attackers were discovered downloading all emails belonging to specific users on separate corporate Exchange servers. An undisclosed Washington think tank reported attackers sending convincing emails to contacts in a social engineering attack that encouraged recipients to click on a link.

But not only the small guys got hit.

On 18 March 2021, an affiliate of ransomware cybergang REvil claimed they had stolen unencrypted data from Taiwanese hardware and electronics corporation Acer, including an undisclosed number of devices being encrypted, with cybersecurity firm Advanced Intel linking this data breach and ransomware attack to the Microsoft Exchange exploits. Advanced Intel detected one of Acer’s Microsoft Exchange servers first being targeted on 5 March 2021. REvil has demanded a $50 million U.S. dollar ransom, claiming if this is paid they would “provide a decryptor, a vulnerability report, and the deletion of stolen files”, and stating that the ransom would double to $100 million U.S. dollars if not paid on 28 March 2021.

On 11 March 2021, Check Point Research revealed that in the prior 24 hours “the number of exploitation attempts on organizations it tracks tripled every two to three hours.”

Check Point Research has observed the United States as being the most attacked country with 17% of all exploit attempts, followed by Germany with 6%, the United Kingdom and the Netherlands both at 5%, and Russia with 4% of all exploits; government/military is the most targeted sector with 23% of exploit attempts, followed by manufacturing at 15%, banking and financial services at 14%, software vendors with 7% and healthcare at 6%.

And to round it all out with an additional laugh:

On 7 March 2021, CNN reported that the Biden administration was expected to form a task force to address the breach; the Biden administration has invited private-sector organizations to participate in the task force and will provide them with classified information as deemed necessary.

Politicians. Making things worse when it’s already too late to help.

Producer Feedback #

Evgeny Kuznetsov chimes in from Moscow:

I wanted to chip in re vaccines, actually. In Russia, as well as in some other post-Soviet countries (I know for sure about Ukraine and Uzbekistan, perhaps several others, too) it is a legal requirement for a kid wishing to attend a state-financed kindergarten or a state-financed school that the said kid is vaccinated against several diseases, or that the kid has a “proper reason” not to be vaccinated. “Proper reasons” include medical counterindications to vaccination, but exclude parents' refusal. I can’t remember the whole list of the required vaccines right away, but these are things like polyo, TBC and measles, basically.

It is not illegal for a parent to refuse to have their child vaccinated, but then they can not attend a state-sponsored school. Seeing as how state-sponsored schools are free of charge to attend, private schools with all the required certificates are still uncommon (especially outside of Moscow and St.-Petersburg) and expensive, and that giving your children a state-recognized basic education (a 9-year-school level) is a legal obligation for every parent in Russia, being an anti-vaccer is not that easy.

The argument is basically that vaccinating every kid that can safely be vaccinated helps reduce the chance of a non-vaccinated kid to contact the disease (since everybody around that kid is vaccinated and less likely to spread the disease for a long period of time even if they got it somewhere) and have serious problems or even die (which they are more likely to if they get sick, because of the medical issues that prevented them getting vacced in the first place). You’ll quote me being born in USSR and having a broken mindset as the result again, but I must confess I think the argument is valid. I’m not an anti-vaccer anyway, me and the wife just had our newborn vacced against several things, including non-mandatory ones.

However, I’ll be postponing my anti-COVID vaccination for as long as I realistically can (which will likely be not very long, since the university I work at is increasing the pressure). I don’t think my chances of getting a severe case of COVID-19 with really bad long-term effects are high (I’m 38 in two weeks, and relatively healthy, even if overweight and enjoying good ale a bit more frequent than I perhaps should), I don’t like the data on vaccines (or, rather, lack thereof), and I seriously don’t trust the vaccines I’m offered (I do have contacts in the institutions that developed those, and what I hear from them makes me even less enthusiastic).

A piece of trivia re transmittable diseases and immunities in Russia (also re regulations and enforcement); just for your amusement.

To be allowed to be present at my daughter’s birth (in the room) I was required to have 3 documents. One - obviously - a negative test result for COVID-19 (or a doctor’s note that I had already had it), but that’s the times we live in. The other two were a recent chest scan to prove I had no TBC, and a recent lab result for measles antibodies (yes, antibodies; lack of measles doesn’t count, you have to be immune), and these two have been a requirement all along, since USSR.

Here comes the funny part. My wife ended up having a C-section, and I ended up in the OR with her. I had the three papers on me, in my jeans back pocket. I was never asked to produce them. Go figure.

If you have any thoughts on the things discussed in this or previous episodes, please feel free to contact me. In addition to the information listed there, we also have an experimental Matrix room for feedback. Try it out if you have an account on a Matrix server. Any Matrix server will do.

Toss a Coin to Your Podcaster #

I am a freelance journalist and writer, volunteering my free time because I love digging into stories and because I love podcasting. If you want to help keep The Private Citizen on the air, consider becoming one of my Patreon supporters.

You can also support the show by sending money to via PayPal, if you prefer.

This is entirely optional. This show operates under the value-for-value model, meaning I want you to give back only what you feel this show is worth to you. If that comes down to nothing, that’s OK with me. But if you help out, it’s more likely that I’ll be able to keep doing this indefinitely.

Thanks and Credits #

I like to credit everyone who’s helped with any aspect of this production and thus became a part of the show. This is why I am thankful to the following people, who have supported this episode through Patreon and PayPal and thus keep this show on the air: Georges, Butterbeans, Niall Donegan, Michael Mullan-Jensen, Jonathan M. Hethey, Dave, Steve Hoos, Mark Holland, Shelby Cruver, Vlad, Jackie Plage, 1i11g, Philip Klostermann, Jaroslav Lichtblau, Kai Siers, ikn, Fadi Mansour, Dirk Dede, Michael Small, Joe Poser, Matt Jelliman, Bennett Piater, David Potter, Mika, Martin, Larry Glock, Dave Umrysh, RikyM, drivezero, MrAmish, tobias, avis, Jonathan Edwards, Barry Williams, m0dese7en, Neil, Captain Egghead, Sandman616, funkyduck and D.

Many thanks to my Twitch subscribers: Mike_TheDane, Galteran, Sandman616, indiegameiacs, Andyp4nts, redeemerf, m0dese7en_is_unavailable, Halefa and l_terrestris_jim.

I am also thankful to Bytemark, who are providing the hosting for this episode’s audio file.

Podcast Music #

The show’s theme song is Acoustic Routes by Raúl Cabezalí. It is licensed via Jamendo Music. Other music and some sound effects are licensed via Epidemic Sound. This episode’s ending song is Backstage of My Heart by Thyra.