Episode 61: The Most Sophisticated Attack

Analysing the SolarWinds hacker attack, which has been called the largest data breach the world has ever seen. Was it actually that bad? I’m trying to put it in perspective and discuss some aspects that have been neglected by much of the mainstream coverage.

I’ve been asked to talk about this multiple times, so here we go: Today’s episode of The Private Citizen deals with the SolarWinds hack that was reported last December.

Overview of the SolarWinds Attack #

According to Microsoft, the SolarWinds data breach was “the largest and most sophisticated attack the world has ever seen.”

Wikipedia sums it up as follows:

In 2020, a major cyberattack by a group backed by a foreign government penetrated thousands of organizations globally including multiple parts of the United States federal government, leading to a series of data breaches. The cyberattack and data breach were reported to be among the worst cyber-espionage incidents ever suffered by the U.S., due to the sensitivity and high profile of the targets and the long duration (eight to nine months) in which the hackers had access.

Within days of its discovery, at least 200 organizations around the world had been reported to be affected by the attack, and some of these may also have suffered data breaches. Affected organizations worldwide included NATO, the U.K. government, the European Parliament, Microsoft and others.

The attack, which had gone undetected for months, was first publicly reported on December 13, 2020, and was initially only known to have affected the U.S. Treasury Department and the National Telecommunications and Information Administration (NTIA), part of the U.S. Department of Commerce. In the following days, more departments and private organizations reported breaches.

Naturally, like anything that occurred in the last four years, you’d have to somehow make it about Trump. What a joke.

The global data breach occurred over the course of at least 8 or 9 months during the final year of the presidency of Donald Trump. Throughout this time, the White House lacked a cybersecurity coordinator, Trump having eliminated the post itself in 2018. When the breach was discovered, the U.S. also lacked a Senate-confirmed Director of CISA, the nation’s top cybersecurity official, responsible for coordinating incident response. The incumbent, Chris Krebs, had been fired by Trump on November 18, 2020. Also at that time, the DHS, which manages CISA, lacked a Senate-confirmed Secretary, Deputy Secretary, General Counsel, Undersecretary for Intelligence and Analysis, and Undersecretary for Management; and Trump had recently forced out the Deputy Director of CISA. Numerous federal cybersecurity recommendations made by the Government Accountability Office and others had not been implemented.

The Exploits #

The attack was initially a hack of Office 365 email accounts perpetrated via Microsoft’s cloud platform:

The attackers exploited flaws in Microsoft products, services, and software distribution infrastructure. At least one reseller of Microsoft cloud services was compromised by the attackers, constituting a supply chain attack that allowed the attackers to access Microsoft cloud services used by the reseller’s customers. Alongside this, “Zerologon”, a vulnerability in the Microsoft authentication protocol NetLogon, allowed attackers to access all valid usernames and passwords in each Microsoft network that they breached. This allowed them to access additional credentials necessary to assume the privileges of any legitimate user of the network, which in turn allowed them to compromise Microsoft Office 365 email accounts. Additionally, a flaw in Microsoft’s Outlook Web App may have allowed attackers to bypass multi-factor authentication.

Attackers were found to have broken into Microsoft Office 365 in a way that allowed them to monitor NTIA and Treasury staff emails for several months. This attack apparently used counterfeit identity tokens of some kind, allowing the attackers to trick Microsoft’s authentication systems. The presence of single sign-on infrastructure increased the viability of the attack.

One of the uses of this massive email hack was to gain access to supply chain of SolarWinds, a network management company, which was very badly protected.

The attackers accessed the build system belonging to the software company SolarWinds, possibly via SolarWinds’s Microsoft Office 365 account, which had also been compromised at some point. The attackers established a foothold in SolarWinds’s software publishing infrastructure no later than September 2019. In the build system, the attackers surreptitiously modified software updates provided by SolarWinds to users of its network monitoring software Orion.

In March 2020, the attackers began to plant remote access tool malware into Orion updates, thereby trojaning them. If a user installed the update, this would execute the malware payload, which would stay dormant for 12–14 days before attempting to communicate with one or more of several command-and-control servers. The communications were designed to mimic legitimate SolarWinds traffic. If able to contact one of those servers, this would alert the attackers of a successful malware deployment and offer the attackers a back door that the attackers could choose to utilise if they wished to exploit the system further. The malware started to contact command-and-control servers in April 2020, initially from North America and Europe and subsequently from other continents too.

Apparently, VMware was also compromised.

Vulnerabilities in VMware Access and VMware Identity Manager, allowing existing network intruders to pivot and gain persistence, were utilized in 2020 by Russian state-sponsored attackers. As of December 18, 2020, while it was definitively known that the SUNBURST trojan would have provided suitable access to exploit the VMware bugs, it was not yet definitively known whether attackers had in fact chained those two exploits in the wild.

The whole thing seems to have been a relatively standard intelligence operation: get into a network, gain persistence, bury as deep as you can, observe as much of the data flying by as you can and then get it out.

The attackers appear to have utilized only a small fraction of the successful malware deployments: ones located within computer networks belonging to high-value targets. Once inside the target networks, the attackers pivoted, installing exploitation tools such as Cobalt strike components, and seeking additional access. Because Orion was connected to customers' Office 365 accounts as a trusted 3rd-party application, the attackers were able to access emails and other confidential documents. This access apparently helped them to hunt for certificates that would let them sign SAML tokens, allowing them to masquerade as legitimate users to additional on-premises services and to cloud services like Microsoft Azure Active Directory.

Once these additional footholds had been obtained, disabling the compromised Orion software would no longer be sufficient to sever the attackers' access to the target network. Having accessed data of interest, they encrypted and exfiltrated it.

Long live the cloud! Wait …wasn’t the cloud supposed to be more secure? Shit.

The attackers hosted their command-and-control servers on commercial cloud services from Amazon, Microsoft, GoDaddy and others. By using command-and-control IP addresses based in the U.S., and because much of the malware involved was new, the attackers were able to evade detection by Einstein, a national cybersecurity system operated by the Department of Homeland Security (DHS).

As The New York Times claims, Einstein missed the attack “because the Russian hackers brilliantly designed their attack to avoid setting it off.” It is now apparently brilliant to hide exfil data from a network management breach in network management cloud traffic.

Security at SolarWinds also seems to have been extremely shoddy. Especially for a company with such high value customers.

In November 2019, a security researcher notified SolarWinds that their FTP server had a weak password of “solarwinds123”, warning that “any hacker could upload malicious [files]” that would then be distributed to SolarWinds customers. The New York Times reported SolarWinds did not employ a chief information security officer and that employee passwords had been posted on GitHub in 2019.

They even told their customers to turn off AV software to “avoid problems”.

Interestingly, there might have been more than one attacker.

FBI investigators recently found that a separate flaw in software made by SolarWinds Corp was used by hackers tied to another foreign government to help break into U.S. government computers.

How the Attacks Were Discovered #

A lot of people discovered signs of the attack independent from each other. It just took ages until everyone started talking to each other.

During 2019 and 2020, cybersecurity firm Volexity discovered an attacker making suspicious usage of Microsoft products within the network of a think tank whose identity has not publicly been revealed. The attacker exploited a vulnerability in the organization’s Microsoft Exchange Control Panel, and used a novel method to bypass multi-factor authentication. Later, in June and July 2020, Volexity observed the attacker utilising the SolarWinds Orion trojan; i.e. the attacker used Microsoft vulnerabilities (initially) and SolarWinds supply chain attacks (later on) to achieve their goals. Volexity said it was not able to identify the attacker.

Also in 2020, Microsoft detected attackers using Microsoft Azure infrastructure in an attempt to access emails belonging to CrowdStrike. That attack failed because – for security reasons – CrowdStrike does not use Office 365 for email.

Separately, in or shortly before October 2020, Microsoft Threat Intelligence Center reported that an apparently state-sponsored attacker had been observed exploiting zerologon, a vulnerability in Microsoft’s NetLogon protocol. This was reported to CISA, who issued an alert on October 22, 2020, specifically warning state, local, territorial and tribal governments to search for indicators of compromise, and instructing them to rebuild their networks from scratch if compromised.

On December 8, 2020, the cybersecurity firm FireEye announced that red team tools had been stolen from it by what it believed to be a state-sponsored attacker. FireEye was believed to be a target of the SVR, Russia’s Foreign Intelligence Service. FireEye says that it discovered the SolarWinds supply chain attack in the course of investigating FireEye’s own breach and tool theft.

After discovering that attack, FireEye reported it to the U.S. National Security Agency (NSA), a federal agency responsible for helping to defend the U.S. from cyberattacks. The NSA is not known to have been aware of the attack before being notified by FireEye. The NSA uses SolarWinds software itself. Some days later, on December 13, when breaches at the Treasury and Department of Commerce breaches were publicly confirmed to exist, sources said that the FireEye breach was related. On December 15, FireEye confirmed that the vector used to attack the Treasury and other government departments was the same one that had been used to attack FireEye: a trojaned software update for SolarWinds Orion. The security community shifted its attention to Orion. The infected versions were found to be 2019.4 through 2020.2.1 HF1, released between March 2020 and June 2020.

And the rest is history.

Assessment of the Attack & the Public Response #

So, what is the actual harm that was done? A shitload of emails from government agencies and companies were stolen. And probably other information that was circulating in their networks.

I am guessing the US government has some non-forensic intelligence that tells them the Russians were the culprits – compare my discussion about forensic and non-forensic attribution in the previous episode of the podcast. It is certainly plausible from looking at the list of attacked entities. It could have been the Chinese just as much, though. Or North Korea.

I think everyone in the US is whining how bad this is to distract from their own failures. Microsoft, SolarWind, the government agencies …all of them were unprepared for an attack that everyone who’s seen NotPetya absolutely knew was coming. The more they can make it sound like the worst attack ever, basically an act from god, the more they can distract from their own abject failures. And stop people asking questions like: Why the hell did your whole government and its intelligence agencies use software from a company that was this shit at security?

It’s a bit like the COVID-19 thing and what governments are doing there: Painting it as an unavoidable act of a higher power to not have to answer questions about why they destroyed our health systems for decades or why their response measures still don’t make sense a year later.

I find it also very interesting that everyone seems to trace this back to SolarWind, when the actual origin of this whole thing was Microsoft’s Office 365. A good PR effort by Microsoft, who managed to connect SolarWind’s name to the attack and turned its own involvement into a mere side note. Another data point that shows how they are trying to hide how they fucked up.

Not to mention the appalling hypocrisy that seems to be integral to anything the US government does since Vietnam. As an ex-CIA operative points out in the Times, the United States are doing the exact same shit themselves. If this was, as the US government said, a declaration of war, what do you think Stuxnet was?

There is indignant howling over what is surely Russia’s role in infiltrating, again, the networks of the U.S. government and corporations – this time through a tainted software update by the company SolarWinds. Politicians of both parties have called it a virtual act of war. “America must retaliate, and not just with sanctions,” Senator Marco Rubio said.

This recalls Shakespeare’s line in “Hamlet” about the lady protesting too much.

The United States is, of course, engaged in the same type of operations at an even grander scale. We are active participants in an ambient cyberconflict that rages, largely unseen and unacknowledged, across the digital globe. This is a struggle that we can’t avoid, and there is no need to play the victim. Just as we use cybertools to defend our national interests, others will use cyberweapons against us.

The National Security Agency and Central Intelligence Agency exist to break into foreign information systems and steal secrets, and they are damn good at it. They, along with the Defense Department, regularly use cybertools to purloin intelligence from servers across the world and to place foreign information systems and industrial infrastructure at risk.

The Pentagon’s cyberwar force, known as Cyber Command, overtly acknowledges, through its “defend forward” doctrine, that the government will target foreign entities and information systems to fight cyberattacks. In November 2018, Cyber Command reportedly disrupted the internet access of the computers of Russia’s Internet Research Agency, the organization responsible for the disinformation campaign during the 2016 U.S. midterm elections. In 2019, in response to Russian cyberincursions into the U.S. energy grid, Cyber Command reportedly placed malware tools on Russia systems that could enable the United States to turn out the lights in Moscow should a conflict between the two nations arise.

As solid as the U.S. cyberoffense is, the defense leaves much to be desired, richly demonstrated by the litany of digital disasters, including the hacks of SolarWinds, the Office of Personnel Management, Equifax and Sony. The reality is that the U.S. government and private companies both underinvest in cybersecurity. Effective defense is a collective effort, but agencies and companies are often clueless and defenseless when it comes to countering the intrusions of countries like Russia, China or Iran.

Producer Feedback #

Barry Williams replies to my comments on the SARS-CoV-2 vaccine from the previous episode.

I agree with you that the vaccine should be completely optional, although my reasoning for that is actually I think forcing something on the population can backfire. However I do think the vaccine aids in reducing spread of COVID; although it likely does not stop the spread.

First principals tells us if I am vaccinated and I contract the virus my antibodies fight the virus and reduce my viral load. It is likely this makes me less contagious than someone unvaccinated. I did hear on another podcast there are preliminary studies that suggest this.

If you have any thoughts on the things discussed in this or previous episodes, please feel free to contact me. In addition to the information listed there, we also have an experimental Matrix room for feedback. Try it out if you have an account on a Matrix server. Any Matrix server will do.

Toss a Coin to Your Podcaster #

I am a freelance journalist and writer, volunteering my free time because I love digging into stories and because I love podcasting. If you want to help keep The Private Citizen on the air, consider becoming one of my Patreon supporters.

You can also support the show by sending money to via PayPal, if you prefer.

This is entirely optional. This show operates under the value-for-value model, meaning I want you to give back only what you feel this show is worth to you. If that comes down to nothing, that’s OK with me. But if you help out, it’s more likely that I’ll be able to keep doing this indefinitely.

Thanks and Credits #

I like to credit everyone who’s helped with any aspect of this production and thus became a part of the show. This is why I am thankful to the following people, who have supported this episode through Patreon and PayPal and thus keep this show on the air: Georges, Butterbeans, Niall Donegan, Michael Mullan-Jensen, Jonathan M. Hethey, Dave, Steve Hoos, Mark Holland, Shelby Cruver, Vlad, Jackie Plage, 1i11g, Philip Klostermann, Jaroslav Lichtblau, Kai Siers, ikn, Fadi Mansour, Dirk Dede, Michael Small, Joe Poser, Matt Jelliman, Bennett Piater, David Potter, Mika, Martin, Larry Glock, Dave Umrysh, RikyM, drivezero, MrAmish, tobias, avis, Jonathan Edwards, Barry Williams, m0dese7en, Neil, Captain Egghead, Sandman616, funkyduck and D.

Many thanks to my Twitch subscribers: Mike_TheDane, Galteran, Sandman616, indiegameiacs, Andyp4nts, redeemerf, m0dese7en_is_unavailable, Halefa and l_terrestris_jim.

I am also thankful to Bytemark, who are providing the hosting for this episode’s audio file.

Podcast Music #

The show’s theme song is Acoustic Routes by Raúl Cabezalí. It is licensed via Jamendo Music. Other music and some sound effects are licensed via Epidemic Sound. This episode’s ending song is Sunstorm by ELFL.