SolarWinds: Illuminating the Hidden Patterns That Advance the Story
activereach partner with RiskIQ, the leader in Attack Surface Management (ASM)
Below we share an article by Team Atlas updating us on the SolarWinds story and the vital role of RiskIQ.
Though the Russian espionage campaign that compromised the SolarWinds supply chain is progressing, public-facing research into the campaign seems to have stopped. The last significant public-facing research into the SolarWinds campaign from the private industry came in March of 2021, more than a month before this publication. Since then, our collective understanding of the campaign has atrophied due primarily to the adversary’s steps to thwart forensic analysis. These impediments to analysis impacted both the tactical and strategic responses to the campaign.
This gap in the analysis happened mainly because piecing together what has happened so far is exceptionally challenging. The threat actor,identified by the U.S. Government as APT29 but tracked in the private industry as UNC2452 (Nobelium, StellarParticle, Dark Halo), went to great lengths to avoid creating the type of patterns that make tracking them simple. For months, the Russians successfully compromised or blinded the very security companies and government agencies most likely to pursue them.
However, RiskIQ’s unique vantage point and collection of internet telemetry helped RiskIQ’s Team Atlas, our new threat intelligence analysis team, illuminate a more significant portion of the infrastructure used in this campaign. Their findings indicate that the SolarWinds espionage campaign’s network infrastructure footprint is significantly larger than previously identified in U.S. government and private industry reporting.
RiskIQ’s Team Atlas detected an additional 18 servers with high confidence that likely communicated with the targeted, secondary Cobalt Strike payloads delivered via the TEARDROP and RAINDROP malware. These servers represent a 56% increase in the size of the adversary’s known command-and-control footprint and will likely lead to newly identified targets after further analysis.
RiskIQ’s Team Atlas’ discoveries suggested that efforts to advance the investigation into the SolarWinds espionage campaign beyond the second, more targeted, stage were inhibited by two main factors:
- The use of U.S.-based infrastructure in the first stage, which effectively blocked or limited pursuit by the NSA; and
- Highly skilled measures to avoid the creation of patterns typically identified and tracked by threat hunters.
RiskIQ’s findings provide tactical intelligence to advance the community’s understanding of this campaign and enable better incident response. RiskIQ has notified US-CERT of these findings prior to publication.
SolarWinds: The Story So Far
Initial reporting of the SolarWinds hack indicated that no later than September of 2019, the threat actor compromised SolarWinds through unknown means and later introduced a backdoor called SUNBURST into an Orion software update. Analysis of the first stage used in the campaign confirmed that despite initial alarm over the unusually broad and seemingly indiscriminate reach of the backdoored software update, the campaign was, in fact, highly targeted.
Arguably, the most significant strategic impact has been the security community’s inability to definitively attribute the attack, causing the U.S. government to stand alone in attributing the campaign to the Russians. In a statement released in April of 2021, the White House formally named APT29, also known as Cozy Bear and The Dukes, as the perpetrator. In the private sector, attribution has been more tentative because the tactics, techniques, and procedures (TTP) employed by the threat actor intentionally did not match known Russian attacker profiles as currently defined, including APT29. As a result, our colleagues attributed SolarWinds to a separate threat group, without associating it with Russia, and referred to it as UNC2452 (FireEye), Nobelium (Microsoft), StellarParticle (Crowdstrike), and Dark Halo (Volexity).
Volexity is the only organization to publicly acknowledge and describe ‘Dark Halo’ breaches that precede the SolarWinds campaign. In a December 2020 blog, Volexity detailed a series of incident response engagements where Dark Halo gained access to an American think tank.
In one of the breaches, Volexity noticed crossover with domains cited by FireEye as used in the second stage of SolarWinds. Volexity attempted to fingerprint those servers based on unique features in the HTTP server responses’ header, only to find that whatever patterns Dark Halo may have established were quickly changed on discovery. In doing so, they stumbled on a critical tactical countermeasure that significantly impeded analysis: the threat actor’s strict adherence to pattern avoidance.
Hide Patterns, Cover Your Tracks
RiskIQ’s Team Atlas found that pattern avoidance was a tactic used in all aspects of the SolarWinds campaign. In Congressional testimony, Crowdstrike’s CEO said that the threat actor “leveraged unique Internet Protocol (IP) addresses for command and control infrastructure for each of its victims,” an action that makes correlating threat activity more difficult.
The researchers found that this infrastructure was registered under varying names and at different times over several years to avoid establishing a traceable pattern. The team assess that APT29 likely purchased the domains from resellers or at auction, which were valuable because legitimate organizations previously owned them.
APT29 was also careful about where it hosted its campaign infrastructure, avoiding patterns there, too. RiskIQ’s Team Atlas noted that the group’s first-stage infrastructure was hosted entirely in the U.S., a move likely meant to avoid raising suspicion, as domestic network traffic is more plausible.
Hosting traffic locally could have also been an attempt to avoid the NSA’s prying eyes because the NSA cannot legally take action except in foreign countries. The second-stage campaign infrastructure was primarily hosted in the U.S. By the third stage, the group hosted its infrastructure almost entirely in foreign countries. In this way, they avoided creating discernable patterns that could be traced while simultaneously making it harder for the U.S. government to investigate.
APT29 also avoided patterns in its use of malware. They designed the first-stage implant to beacon to its command-and-control servers with random jitter after two weeks, likely to outlive the typical lifespan of event logging on most host-based EDR products.
APT29 used a modified version of Cobalt Strike in the second stage, which is significant because Cobalt Strike, a pentest tool, is available publicly. The group designed its third-stage malware to look completely different from the second-stage malware, designed to look nothing like the first-stage malware. This measure ensured that analysts that found only one stage’s malware would have difficulty following it into the other stages.
Taken together, the threat actors implemented their TTPs in this campaign to avoid resemblance to prior patterns associated with APT29 or any of the other known Russian APT groups. Researchers or products attuned to detecting known APT29 or other Russian APT activity would fail to recognize the campaign as it was happening. And they would have an equally hard time following the trail of the campaign once it was discovered.
Internet Telemetry Busts APT29 Pattern Avoidance
The typical method of identifying a threat actor’s attack infrastructure footprint involves correlating IPs and domains with known campaigns and detecting patterns in domain registration, the aging of infrastructure before use, activity periods, etc. As discussed above, this threat actor made apparent attempts to avoid creating such patterns.
Despite their best efforts, the threat actors did not account for RiskIQ’s unique network telemetry. Combined with the known indicators of compromise, this telemetry resulted in RiskIQ’s Team Atlas successfully fingerprinting command-and-control servers APT29 used in its more insidious later stages of the SolarWinds campaign.
The team began by reviewing all of the previously published indicators of compromise and noticed two patterns emerge through an analysis of the SSL certificates used:
- The majority of the SSL certificates used by the group were issued by Sectigo (Formerly Comodo CA). Additionally, they were all of a particular class called “PositiveSSL,” which costs around $11 a year per domain.
- The issue date of the certificates, otherwise known as “Not Before” in x509 terminology, was often more than a week before when the certificate itself was deployed in the wild. Or in several cases, more than 40 days later.
Searching for all SSL certificates issued by Sectigo Limited after February 1, 2020, matching the above criteria yielded ~ 334,053 results, too many to be useful.
Next, RiskIQ’s Team Atlas built upon Volexity’s previous research in discerning patterns from HTTP banner responses from previously identified domains and IP addresses and succeeded in identifying several patterns.
They then combined the two data sets, correlating the domains and IPs that returned the specific banner response patterns with their relevant SSL certificates. They noted the corresponding periods of activity and hosting locations, which resulted in a more complete and context-rich view of the previously identified command-and-control infrastructure.
After a bit of trial and error, the team discovered a new pattern they could leverage across the entire second stage of the attack to identify additional infrastructure: modified Cobalt Strike Beacon servers. While this pattern was not in and of itself unique (it returned about 3,000 results on the internet at any given time), it became highly unique when correlated with the SSL patterns mentioned above. The result was RiskIQ’s Team Atlas identifying a significant number of additional malicious servers.
The team identified numerous other domains matching the original constraints, including the server responses and SSL Certificate features. After further investigation, they could not rule them out from being associated with the actor. However, they typically served place-holding content or were hosted with previously unobserved providers.
If you’d like to learn more about Attack Surface Management and RiskIQ please call activereach on 0845 625 9025 or contact us here.