SolarWinds Hack Defenses: Protecting Against ‘Solorigate’ TTPs

eSecurity Planet content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

A March 2020 software update of the SolarWinds Orion management platform gave malicious actors unhindered access to key government and enterprise networks. Microsoft has dubbed the infamous supply chain compromise of SolarWinds as “Solorigate.” In December, eSecurity Planet detailed FireEye’s initial findings, implications for the industry, and how to mitigate similar attacks.

Since then, much has been learned about the tactics, techniques, and procedures (TTPs) deployed and what steps organizations are taking to harden their network and application security.

This update touches on the newly detected malware, attack vectors to guard against, and why the targeting of security vendors is a critical development in cybersecurity.

Also Read: FireEye, SolarWinds Breaches: Implications and Protections

Brief timeline of findings

The extent of the most recent attacks is still being unraveled. Before jumping into the technical details regarding each new malware detected and proper safeguards, here is a brief look at the events to date:

Second Orion attack vector detected

On February 2, 2021, Reuters reported that a second advanced persistent threat (APT), connected to China, also exploited SolarWinds software, in addition to the previously identified Russia-connected actors. FBI investigators revealed that a federal payroll agency, the National Finance Center (NFC), was breached but has yet to confirm additional exposure to U.S. federal agencies. Former Department of Homeland Security (DHS) officials noted “this could be an extremely serious breach of security.” NFC records include social security numbers, phone numbers, banking information, and personal email addresses for thousands of federal employees.

SolarWinds has added that one of their clients was also exposed to the same APT “in a way that was unrelated to SolarWinds.” Notably, in late January U.S. Cybersecurity and Infrastructure Security (CISA) director Brandon Wales told the Wall Street Journal that “approximately 30% of both the private-sector and government victims linked to the campaign had no direct connection to SolarWinds.” It’s becoming clear that the attack went much further than just breaching SolarWinds.

Interestingly, findings show the Chinese APT exposed a separate vulnerability in the Orion software from that of Solorigate.

The Solorigate malware family

When the SolarWinds news first broke, a malicious backdoor dubbed Sunburst was the primary culprit for the vast compromise. In actuality, the compromised DLL and backdoor that caught FireEye’s attention had a few friends. Welcome to the unfortunate gathering: Teardrop, Sunspot, and Raindrop.

Also Read: Types of Malware & Best Malware Protection Practices

Teardrop, Malicious Memory Dropper

In an update to their Sunburst findings, FireEye touched on the part of an additional malware strain. Teardrop was deployed thanks to the Sunburst backdoor and is distinctly new, with no code overlap with previously detected malware. Presenting itself as a JPG file named “gracious_truth.jpg,” Teardrop is a memory-only dropper built to enter a network seamlessly and replace the embedded payload. Teardrop can then execute a customized Cobalt Strike Beacon, emulating various malware and other advanced threat tactics on the network.

Sunspot, Sunburst-Enabler

Now we know how Sunburst initially made its way into SolarWinds. In early January, CrowdStrike published its findings on Sunspot. This malware infiltrated SolarWinds in September 2019 with the expert insertion of code to avoid detection. As a malware program, Sunspot would monitor the Orion product’s running process, and when the time was right, replace the source file to include another backdoor.

Almost six months after Sunspot deployed at SolarWinds, the malware inserted the Sunburst back door into the Orion platform’s build.

Raindrop, Loader and Spreader

In mid-January, Broadcom’s Symantec division uncovered the third additional strain in the case of Solorigate. Like Teardrop, Raindrop is a loader that can also enable a Cobalt Strike, but the Sunburst backdoor didn’t deploy it. In Symantec’s analysis, they noted three examples of how Raindrop behaved:

1. Enabled the malware to access network computers via the management software, and later extract a copy of the Directory Services Internals. With access to DSInternals, the malware could query the AD servers and steal data, passwords, and keys.
2. Executed Microsoft PowerShell commands to create more instances of Raindrop on network computers.
3. Executed Cobalt Strike extracted data shows configuration for a network pipe over server message block (SMB), unlike numerous recent attacks that learn towards using HTTP-based command and control (C&C) servers.

Read Also: The IoT Cybersecurity Act of 2020: Implications for Devices

Orion Vulnerabilities Keep Emerging

Though SolarWinds wasn’t the only target, the vendor’s vulnerabilities are being brought to light. On February 3, 2021, threat detection and response vendor Trustwave released three additional findings on SolarWinds vulnerabilities.

The first involves SolarWinds Collector Service employing the Microsoft Message Queue (MSMQ) – a program not included in modern Windows systems. Because there are no permissions on private queues, an attacker can send trusted Collector Service messages. Once those messages are processed, the attacker can move to a remote execution code (RCE) attack as LocalSystem.

Martin Rakhmanov of Trustwave’s SpiderLabs added that after the patch, a digital signature validation step on arrived messages ensures that messages not bearing a signature are not processed. This doesn’t resolve the unauthenticated MSMQ allowing anyone to send trusted messages.

The second identified vulnerability is a case of unprivileged access on the web. The Orion backend credentials were discovered on a world readable file. Attackers with access to the filesystem can key-log Orion database login details and use the decrypted information to gain database owner access. With admin-level access, the malicious actor can modify authentication data stored.

Unlike the first two vulnerabilities that were specific to the Orion User Device Tracker, the third vulnerability lies in the SolarWinds’ Serv-U FTP for Windows. Trustwave found any authenticated Windows user could log in and drop files that define new users. The Serv-U FTP program logs this automatically. The attacker can then define an admin account, setting the home directory to the root of C:\ drive. With login access via FTP, any file on the C:\ can be read or replaced.

While we are just learning of these vulnerabilities, Trustwave disclosed the vulnerability to SolarWinds in late December and a patch was released in late January. At this time, researchers are unsure of the extent to which attackers exploited each in relation to Solorigate and similar attacks.

Attacker methodology

In the handful of weeks since Solorigate, we’ve learned plenty about the TTPs (tactics, techniques, and procedures) used by attackers. In a report detailing Sunburst, Teardrop, and Raindrop’s second-stage activation, Microsoft dove into the OpSec methods used. These included:

• Avoiding any share indicators for each compromised host
• Disguising locations inside folders mimicking existing files
• Disabling and re-enabling event logging at their pleasure
• Amending firewall rules to allow sensitive, outgoing protocols
• Moving laterally with caution, only when security services could be disabled
• Timestomping, wiping, and DLL-implant obfuscation

Microsoft said both the complex attack chain and length of the operation mean organizations need comprehensive real-time visibility and access to months of historical data for related investigations. A primary software target of Solorigate was Microsoft 365. For malware used against Microsoft programs, FireEye disclosed the attack behavior tactics as:

• Accessing the AD FS token-signing certificate to forge additional tokens
• Changing Azure AD trusted domains to create attacker-control IdPs
• Stealing high-privilege user credentials synchronized with Microsoft 365
• Adding a backdoor to existing Microsoft 365 infrastructure for remote access

In a follow-up to their first Solorigate report, CISA also published their research on the threat actor’s TTPs. Like FireEye’s findings, CISA touched on the bypass of federated identity solutions, forged authentication tokens, and privileged access to install persistent API-based access.

Also Read: The Software Supply Chain: Where Security Starts

Compromised certificates, forged tokens

When considering attack vectors, the first concern for code signing is the prospect of stolen digital certificates. However, in the case of the Solorigate breach, attackers targeted the build process. Why steal certificates when you can control the token generator’s ultimate access to an organization’s network? As the primary attack vector for Solorigate, we look at how attackers strengthened their attack by manipulating digital tokens.

Also Read: How to Secure Digital Signatures

The Rise of Golden SAML Attacks

In 2017, CyberArk published findings on a new attack vector related to certificate signing. TheSAML 2.0 protocol serves as the authentication mechanism between an identity provider and service provider for cloud computing. With the migration from on-premises to cloud services, it’s no surprise the SAML protocol could be a growing target.

In the case of Solorigate, attackers gained initial network access and then misused X.509 certificates and keys to forging SAML tokens.

CyberArk identified this tactic as a “golden SAML.” While activating the malware requires domain admin access, threat actors get the keys to creating trusted SAML authentication objects if successful. The result: unmitigated access to your network for attackers.

Microsoft Software Targeted

In many cases, the Solorigate compromise led attackers to the host’s Microsoft Office 365 email services and Microsoft Azure cloud infrastructure. Like previously mentioned TTPs, the malware involved would manipulate Microsoft 365 and Azure so that its presence would go undetected while it could access and monitor host data. With user account credentials, attackers had a suite of email, documents, and data at their fingertips.

By manipulating the trusted SAML token signing certificate, attackers could create new accounts, escalate privileges, and access sensitive data across the MS software. Cases showed malware added X.509 keys or password credentials to legitimate OAuth applications to offer protracted authorized access.

Also Read: Top CASB Security Vendors

Hardening the build environment

Many analysts point to the severity of this supply chain compromise with calls for more robust build environments. The build process is on trial, from vulnerabilities in the supply chain for IoT devices to the Solorigate breach. The longest-tenured malware, Sunspot, monitored the build server and seamlessly replaced the source code files in the Orion software with Sunburst-loading files.

David Wheeler, Director of Open Source Supply Chain Security for the Linux Foundation, notes, “Unfortunately, a lot of conventional security advice cannot counter this kind of attack.”

The Orion update was signed and presented as the latest update, and malware taking root went undetected. Similarly, reviewing the source code when attackers have control of the build process would help little.

A critical fault of SolarWinds was their lack of attention to the build environment. SaveBreach reported SolarWinds was “using [an] unencrypted plain FTP server for their Downloads server in the age of global CDN technologies.” More frightening, the FTP credentials were available on a mib-importer GitHub repo for well over a year.

Protections and considerations

Organizations continue to wonder how they can come out of this moment with stronger security. Given the sophisticated nature of the attacks and what experts report, we offer a few significant considerations for guarding against similar TTPs.

Mitigating Digital Certificate and Token Compromise

With concern surrounding certificate forging and similar attacks to come, the following DevSecOps recommendations are critical:

• Enhance visibility into token signing certificates and define strict configurations
• Evaluate network resources and privileges to shut down out-dated or unnecessary permissions
• Ensure token activity is being monitored and tracked to mitigate misuse

As the inherent authentication tool for cloud services, SAML is not going away as an attack vector. Its successful use in Solorigate tells us we can expect similar attacks.

Noted in our first update but worth restating are some of Keyfactor’s best practices to mitigate key and certificate vulnerabilities:

• Code-signing keys should be kept in a FIPS 140-2 validated HSM
• Create a system of accountability by segregating roles for authorizing, approving, and monitoring code signatures
• Monitor all active certificates by location, user, device, and traffic
• Enforce a strict certificate issuance policy to ensure all certificates are trusted
• Prepare for rapid response by testing issuance and revocation capabilities for certificates

Are Verified Reproducible Builds the Future?

In the effort to establish secure builds for the future, reproducible builds are certainly one solution. The idea is simple, organizational software builds should create symmetric input-output results, and technicians should thoroughly define the build process. By developing reproducible build environments, analysts have greater visibility into the existence of vulnerabilities or malware.

Marketwide adoption of verified reproducible builds is a ways off. While plenty of software is reproducible, most organizations aren’t ready to invest in the project. Reproducible builds will be easier on open-source software (OSS) developers, but proprietary software will benefit as well, considering 99% of all commercial software includes at least one open-source component.

Wheeler of the Linux Foundation noted that the industry could prioritize more critical software initially, but there’s no doubt that this is where we’re heading.

Also Read: IoT Security: It’s All About the Process

Software Bill of Materials (SBOM) for Greater Security

Devices constructed with re-used or out-of-date software can pose an unnecessary risk to your network security. The problem: software can be mighty complex, made up of components, development frameworks, operating system features, libraries, and more. The National Telecommunications and Information Administration (NTIA) offers the concept of a Software Bill of Materials (SBOM) to address this problem.

While offering a SBOM could mean more time spent on technical details, the stress-free full visibility into your software’s build will be worth it. Both your organization and client base can be confident in the software they’re employing. As more organizations request the list of ingredients in their software, time will tell how quickly the industry adopts the format.

On the contrary, Rob Graham of AT&T Cybersecurity makes an argument for why a SBOM isn’t everything. Mandating vendors to publish SBOM could weed out unethical companies and cement transparency. But in its current design, a SBOM isn’t granular enough to enhance vulnerability management.

Tools for Detecting Solorigate Vulnerabilities

To minimize the Solorigate breach’s impact, several organizations have published detection tools for organizational use. These tools only go so far in identifying potential compromises but can identify similar authentication-based vulnerabilities and Solorigate behavior.

• CrowdStrike Reporting Tool for Azure (CRT) (GitHub)
• CISA Cloud Forensics’ Sparrow (GitHub)
• FireEye’s Mandiant-Azure-AD-Investigator (GitHub)

Cybersecurity vendor targets and vigilantes

Almost two months after the news broke, organizations are still disclosing the impact of the breach. In addition to hitting key government agencies, Bloomberg reports on how Solorigate targeted cybersecurity firms. The attackers’ goals were to:

• Breach cybersecurity vendors responsible for protecting network security
• Craft more robust malware to target the vendor’s client network

Vendor compromise has severe implications for clients. Attackers can steal source code, detection tools, and penetration testing technologies built to fend off the best malicious threats in the world. If successful in infiltrating certain vendor network privileges, an attacker’s remote access to a client’s network security is a scary possibility.

In the last two weeks of January, cybersecurity vendors Mimecast, Fidelis, Qualys, and Malwarebytes confirmed breaches. Organizations like Palo Alto Networks and CrowdStrike reported being targeted but withstood Solorigate attack efforts.

Also Read: Best Penetration Testing Software

Breached Organizations

Earlier in January, UK-based email security vendor Mimecast disclosed to customers that a sophisticated threat actor abused a stolen digital certificate to access 10% of their Office 365 accounts. Mimecast has already informed affected clients, asking organizations to re-establish a new digital certificate. This breach was later confirmed as a result of a trojan SolarWinds Orion app installed on its network.

Fidelis Cybersecurity also confirmed downloading the trojan app. Upon tracing the download, it remained inside a test system machine isolated enough that attackers couldn’t move further in-network. Auditing and vulnerability management vendor Qualys confirmed their breach but downplayed any impact on their production environment or exfiltrated data.

Despite never using the Orion software, Malwarebytes reported being targeted by the Solorigate threat actors. Their incident response team found attackers “leveraged a dormant email protection product within our Office 365 tenant that allowed access to a limited subset of internal company emails.” As for TTPs, the attacker deployed a golden SAML attack that enabled them to make API calls to request emails via MSGraph.

Also Read: Common IT Security Vulnerabilities and How to Defend Against Them

Withstanding Solorigate

How did organizations fend off Solorigate? The attack was rooted in the Orion software, but targets were not limited to SolarWinds clients. With bigger goals, attackers deployed similar TTPs against other organizations in 2020 but weren’t always successful. Two that have come forward are Palo Alto Networks and CrowdStrike. Both companies have detailed the TTPs used, but didn’t reveal everything about how they successfully defended their networks when other organizations faltered.

In September and October 2020, attackers attempted to breach Palo Alto Networks. In both instances, the in-house SOC team isolated the targeted server and stopped any malware from taking root, with help from the company’s XDR platform. What wasn’t detected then was the larger issue: using the supply chain for the attack vector. Upon further review, since Solorigate news broke, Palo Alto Networks has found no shared indicators with the malicious attack.

For CrowdStrike, months after the attempted hack, they were first alerted about attempts to read email and abnormal communication between their MS Office licenses and MS cloud APIs. CrowdStrike does not use Office 365 email but did conduct a thorough review of any infrastructure shared with Microsoft, including their Azure environment. Their audit found no evidence of impact.

Specific to Solorigate, CrowdStrike said the following network security solutions can be critical in boosting your defenses:

• Endpoint Detection and Response (EDR)
• Cloud Access Security Broker (CASB)
• Zero Trust Network Access (ZTNA)
• Identity and Access Management (IAM)
• Encryption
• Secure email gateway
• Mail DNS controls
• Cybersecurity awareness training and campaigns

Also Read: Top Endpoint Detection & Response (EDR) Solutions

Vendor Catch-22

SolarWinds and their clients weren’t the only targets, in fact, they made up less than a third of reported breaches. As researchers try to unravel the additional vulnerabilities and targeted vendors, Wired asks how organizations can trust third-party software providers. The truth is that business today requires access to a  suite of SaaS products like Slack, AWS, and Google Analytics, to name a few. Widely used software like Microsoft Windows, SolarWinds Orion, and others are targets simply because of their global popularity.

Malwarebytes CEO Marcin Kleczynski noted, “It’s a catch-22. Rely on one vendor and you’re screwed if they get hit. Rely on multiple and all it takes is one. Rely on the big brands and deal with the consequences that they’re most targeted. Rely on the small brands and deal with the consequences they’re not yet investing in security.”

Demands for greater visibility into the software supply chain are clear. But one glaring concern for many organizations in light of Solorigate is insufficient knowledge of their software catalogue. Organizations must be vigilant in monitoring third-party software and inspecting the possibility of existing vulnerabilities.

Also Read: Almost Half of All Third-Party Software Components Are Outdated, Insecure

Building comprehensive network security

Solorigate may well be the most significant supply chain compromise in modern history. Due to the sophisticated nature of the attacks, defending against them isn’t easy, but there are some lessons and steps organizations can take. While it’s tempting to dismiss the attack as largely behind us, the TTPs used were widely successful and likely aren’t going away. Learning from Solorigate means recognizing that advanced persistent threats and malware are constantly evolving to exploit victims better. A straightforward solution is building comprehensive network and application security that can guard your network no matter the danger – a challenge easier said than met. Constant vigilance is required against an ever-evolving foe, and that means keeping up on the latest defenses and vulnerabilities and taking action to protect against them.

Also Read: SASE: Securing the Network Edge