Part1 Preface
Hello everyone, my name is ABC_123. The official account has been officially renamed "Xitan Laboratory", please stay tuned . In this issue, we share an APT case that is the most influential and harmful supply chain attack in history - the Solarwinds supply chain attack incident. SolarWinds has tens of thousands of customer companies, including Fortune 500 companies and all the top ten companies in the United States. In the end, approximately 18,000 companies, including major telecommunications companies, all five major branches of the US military, the US State Department, the National Security Agency, the Office of the US President, and many top US network security companies, installed the Orion network management software update package containing backdoors, APT The organization selected more than 50 major company units for horizontal penetration of the intranet.
Note: The attacker's intrusion into the U.S. company Solarwinds falls into the category of upstream software supply chain attacks , while the attacker's intrusion into U.S. critical infrastructure OEM manufacturers through Solarwinds falls into the category of industrial supply chain attacks . Therefore, this APT attack has far-reaching consequences and is extremely harmful.
This APT supply chain attack incident is extremely complex. ABC_123 has reorganized domestic and foreign English reports, combined with its own practical network security experience, and repeatedly verified various analysis articles word for word, striving to restore the true and complete picture of the entire intrusion incident and provide the vast network with Provide a reference for security practitioners to better contribute to the network security cause of the motherland .
Part2 Introduction to prerequisite knowledge
APT flow chart and attack process introduction
First, let’s release a flowchart drawn by ABC_123 about the Solarwinds supply chain attack on critical infrastructure in the United States. It is summarized from a large number of domestic and foreign reports on this attack. Next, based on this flowchart, the entire intrusion will be explained in detail. process.
Attack process: The attacker first invaded the intranet of the American Solarwinds network management software company, then obtained permissions for all source code compilation servers, and monitored the update package of its Orion network management software through the written Sunspot tool (a backdoor delivery and source code modification tool) During the source code compilation process, the Sunburst backdoor program is implanted from the source code level. After compilation , it comes with Solarwinds legal digital signature Solarwinds Worldwide, LLC . This method has a built-in whitelist effect and perfectly bypasses various protective measures. After Solarwinds officially released the software update package, the intranets of tens of thousands of its Orion customer companies were compromised.
Introduction to attack weapons and backdoors
Next, ABC_123 will introduce the various attack weapons used by the attackers during this APT supply chain attack, so that everyone will not be confused by the English names of these tools when reading the article.
1 First stage weapons:
Distribution tool: This tool is not specifically named and is used to deliver the Sunspot attack tool to more than 100 compiled virtual machines in batches.
Sunspot: Used to monitor the Orion source code compilation process and implant the Sunburst backdoor code in the Orion update package source code through a series of complex operations.
Sunburst: This is an extremely complex and hidden backdoor. When the user runs the Orion installation package, this backdoor will wait for 12 to 14 days to be triggered , and then use the DGA domain name communication method to initialize the domain name, computer name, etc. of the controlled server. The information is sent to the C2 server.
2 Second stage weapons:
Teardrop: Loader program, provided by SUNBUSRT, extracts malicious shellcode from steganographic images, loads and executes it, and is finally launched on Cobalt Strike.
Raindrop: Loader program. Unlike Teardrop, it is mainly used for horizontal propagation within the intranet. In order to hide the attack behavior, the attacker compiled Raindrop into a 7-zip.dll file by modifying the 7-zip source code for subsequent loading and execution. Define CS shellcode.
3 Third stage weapons:
Through the second-stage Loader program, backdoor programs suitable for different scenarios such as Cobaltstrike , GoldMax , SiBot , and Goldfinder are implanted.
According to the analysis of foreign security companies, the attacker deliberately isolated the two-stage backdoors of Sunburst and CobaltStrike in order to facilitate the victim to still protect the existence of the Sunburst backdoor and maintain permissions after discovering and emergency removal of the Cobalt Strike backdoor.
Part3 The whole process of APT supply chain attack
Obtain external network server permissions
After evaluation by foreign security companies, organizing such a large-scale supply chain attack operation involves formulating plans, simulation exercises, preparing attack infrastructure, early reconnaissance, customized development of weapons and command and control platforms, obtaining initial permissions to the target network, and later penetration expansion, etc. A series of tasks required the support of more than 1,000 engineers . Later, in order to investigate and trace the source of this APT attack, Microsoft deployed more than 500 security engineers to investigate .
Since the APT attack was exposed on December 12, 2020, there has been no convincing investigation result on how the APT organization obtained solarwinds external network permissions. Currently, it is generally believed that there are the following possibilities:
1 Github password leaked . In 2019, a security officer discovered the leaked server weak password solarwinds123 on github. After investigation, an intern at Solarwinds posted his password solarwinds123 on his personal github in 2017. Use this password to log in to the outside world. The update server of solarwinds.com.
2 Violent guessing of weak passwords . solarwinds123 is a weak password, and the attacker may have successfully guessed it through brute force cracking. The weak password solarwinds123 has been in use since at least 2017.
3 Office365 service vulnerabilities . In February 2019, it was discovered that an attacker had stolen the Office 365 accounts of Solarwinds employees, and was suspected to have obtained relevant permissions through a vulnerability in Microsoft's Office 365 service, but Microsoft officials firmly denied it.
Implanting test backdoor code during the source code compilation process of Orion network management software
After obtaining access to the external network, the APT organization conducted a very covert horizontal penetration of the internal network. It spent a long time and energy to collect sufficient information on the internal network topology and source code compilation server, and finally obtained the Solarwinds source code construction server. all permissions.
After several months of evaluation and analysis, the APT organization formulated a crazy plan: to implant backdoor code in the source code of Solarwinds network management software Orion , so that the generated update package could be Bypassing various anti-software protections, when Solarwinds releases the Orion software update package to customers, it can obtain all the permissions of tens of thousands of its customer companies. This is the huge power of upstream supply chain attacks.
The plan sounds cool, but it is very difficult to implement. In order to successfully implement the plan, the APT organization spent more than a year preparing . Next, ABC_123 lists several key time points to give everyone an intuitive understanding of the APT organization’s rigorous work plan.
On January 30, 2019, the APT organization logged into an employee's VPN account, presumably to find and analyze the SVN source code server.
On January 31, 2019, which was the second day, the attacker logged into the VPN again, downloaded the source code of more than 129 sets of Solarwinds products, and obtained a large amount of user information, presumably to analyze which customers of Solarwinds used which products. products to prepare for subsequent supply chain attacks. After that, the attacker disappeared for 2 months. During this period, the APT organization conducted an in-depth analysis of the source codes and the usage of different Solarwinds software products by customers, striving to maximize the use of these source codes. In the end, they found that Orion software, owned by Solarwinds, was the perfect choice to carry out supply chain attacks because this software has a large user base, occupies a privileged position in the customer network, and connects and communicates with many other servers, making it very convenient for lateral penetration.
On March 12, 2019, the attackers regained access to the software build environment and then disappeared for another six months. The build environment for Orion software is so complex that it would take a newly recruited programmer several months to master it. In the following six months, the attacker created a copy of the Orion source code build environment , and then wrote the backdoor code. The encoding method and naming rules of the backdoor code deliberately imitated the habits of the programmers of the original Solarwinds company, so that many security personnel were involved in the later period. After several inspections, no abnormal code was found. Then we continued to try to insert backdoor code during the source code compilation process, and constantly tested to solve various possible unexpected situations, so as to achieve first-time success in the production environment .
On September 4, 2019, that is, 6 months later, the attacker visited the source code construction environment again. Judging from the recorded operation logs at this time, the attacker 's operations have shown professional skills in source code , indicating that they spent Huge time and energy are spent on in-depth research and testing of the source code.
On September 12, 2019, the attacker implanted harmless test code in the source code build of the Orion network management software. It is speculated that he was conducting a "non-destructive" test, that is, to test whether the source code replacement operation can be completed in the production environment. Unnoticed, they then quietly wait for Orion software updates. During this period, they used the email accounts of dozens of key executives and security personnel to secretly track and monitor Solarwinds to see if they discovered the intrusion .
On October 26, 2019, the attacker made minor modifications to the source code of the Orion update package, added empty class code, but did not contain backdoor code, tested whether the server could be compiled through the Orion source code, and was successfully released. Finally, Solarwinds released the Orion software version 2019.4.5200.8890, the first modified version by the attacker. The attacker found that the harmless Orion source code modified by him was successfully compiled, indicating that everything was ready.
On November 4, 2019, the attacker officially completed all testing work. Almost a year had passed by this time, which shows the attacker's full preparation and determination to fight a protracted war.
Use Sunspot to implant backdoors in production environments
Once the Sunspot tool is run, it will increase its debugging rights. By adding a scheduled task and executing it every 11 minutes, it will monitor the compilation process of the Orion software compiler MSBuild.exe , and obtain the Orion code package by obtaining the command line parameters of MSBuild.exe. Enter the absolute path of the core.sh solution file, further find the path of the InventoryManager.cs source code, and back up this source code file as InventoryManager.bk.
Sunspot then decrypted itself a copy of the InventoryManager source code file modified by the APT organization containing the SunBurst backdoor, and replaced the original InventoryManager.cs code in the Orion source code that will be officially released. The backdoor code will eventually be compiled in SolarWinds.Orion. Core.BusinessLayer.dll file , and comes with Solarwinds digital signature, and was finally packaged in the update patch installation file CORE-2019.4.5220.20574-SolarWinds-Core-v2019.4.5220Hotfix5.msp .
When users of Orion network management software install the patch package, once SolarWinds.BusinessLayerHost.exe or SolarWinds.BusinessLayerHostx64.exe is run, the Sunburst backdoor code in SolarWinds.Orion.Core.BusinessLayer.dll will be called, causing users who install the update package to The back door was planted.
Difficulties encountered during the process of implanting backdoors
On February 20, 2020, the attacker implanted the Sunspot tool into the software build server and then lurked in the software build server.
In March 2020, Solarwinds developers began to use TeamCity produced by JetBrains (TeamCity is a build management and continuous integration server provided by a Czech software development company) to build Orion's software update package. The compilation software started more than 100 compilation virtual machines . These virtual machines are created temporarily and will self-destruct once the compilation work is completed. At this time, the attacker encountered trouble. Each compilation virtual machine was responsible for compiling the code of different modules of the Orion network management software. They did not know which compilation virtual machine would compile the SolarWinds.Orion.Core.BusinessLayer.dll file or package the dll file. .
Batch Delivery Sunspot Tool
So the attacker specially designed and written a tool to deliver the Sunspot tool to more than 100 compiled virtual machines. On each compiled virtual machine, Sunspot will create a scheduled task to monitor the MSBuild.exe compilation process.
Once it is discovered that SolarWinds.Orion.Core.BusinessLayer.dll appears on the server, the legitimate source code file is immediately renamed and replaced with the dll file containing the backdoor Sunburst version. The whole process only takes a few seconds. It is very difficult to implement and also This shows how well-prepared the attackers have been, and also indirectly understands why the APT organization spent a year testing. Sunspot will then restore the original legitimate dll files, delete itself from all virtual machines, and clean up traces.
Use Orion update package to invade downstream customer companies
Solarwinds official website will distribute update patch installation packages of Orion software to its customers every once in a while. Since the spring of 2020, Solarwinds official website has released update patch installation packages to as many as 18,000 Orion customers. If these customers’ Orion servers are exposed to the outside world, Open access to the Internet will cause the Sunburst backdoor program built into the installation package to be triggered. (The following picture comes from the Internet)
When the victim installs the patch package, the SolarWinds.BusinessLayerHost.exe or SolarWinds.BusinessLayerHostx64.exe program will be run, and then the SolarWinds.Orion.Core.BusinessLayer.dll file will be called , and then the backdoor Sunburst hidden in this dll file will be triggered. , thus causing the customer's Orion network management server to be controlled.
Such a result would lead to 18,000 customer companies being infected with the Sunburst backdoor. Penetrating the intranets of 18,000 companies is obviously unrealistic, and the action is too large, and it is easy to expose the APT attack plan, so the APT organization selected about 50 Conduct lateral attacks on important targets , including the U.S. Department of the Treasury, Department of Justice, Microsoft, etc. As shown in the figure above, these are the published lists of all compromised vendors. For other intrusion targets of little value, the APT organization will send termination commands to cause the Sunburt backdoor to permanently exit and self-destruct.
It is worth noting that Google also uses the Orion network management software, but due to its strict internal security protection measures, the relevant person in charge of Google confidently stated that Google has not been attacked by the Solarwinds supply chain, and subsequent follow-up has proved that this is indeed the case.
Sunburst backdoor activated
Once the dll file containing the Sunbrust backdoor is called and executed, it will not immediately trigger the execution of malicious operations, but will randomly wait for 12 to 14 days. The purpose of this is to gradually bring the controlled servers of more than 18,000 Orion software users around the world online. , to avoid the centralized online being discovered and alarmed by the traffic monitoring equipment, causing the plan to be exposed.
The Sunburst backdoor will then use the DGA domain name to communicate with the C2, collect the victim's host environment and computer information and send it to the attacker. The attacker will use this information to determine whether the unit is worthy of further intrusion.
If the target is very important, the Teardrop tool or Teardrop tool will be further deployed . These two tools are malicious code loaders. The Teardrop loader will download 4 types of images, and combine the image steganography technology to extract the hidden shellcode, and then use Teardrop to Load execution.
If the target is not important, the attacker will issue instructions through the C2 to cause Sunburst to exit or be permanently disabled. Although the attackers infected thousands of servers, they only selected about 50 important units for deep penetration. This allowed them to concentrate on important targets and reduce the dispersion of energy when targeting a small number of important targets. , to prevent attacks from being captured by the security protection system.
Unintentionally leaving a backdoor sample
The attacker went to such great lengths to hide himself and detect excessive traffic that the security team spent months tracing the origin of the attack. By monitoring Solarwinds' company emails , the attacker learned that the victim had noticed the attack, so he decisively deleted the Sunspot tool of all virtual machines. So how did the security personnel discover the attack weapons and samples left by the APT organization?
As the saying goes, Skynet is full of details but not omitted . This originated from an accident caused by source code compilation. Around February 2020, an error occurred in a compiled virtual machine during the software building process. Then TeamCity created a virtual machine snapshot, which contained all the contents of the virtual machine when the failure occurred, making it easier for R&D engineers to analyze the failure. Under normal circumstances, SolarWinds R&D engineers would delete these snapshots later, but I don’t know why they were not deleted. Fortunately, the Sunspot tool was retained in this virtual machine snapshot, and even more fortunately, the software engineer never deleted this snapshot. This accident became the only clue to trace the entire Solarwinds supply chain attack .
The attacker stops and cleans up the traces
On or about June 4, 2020, a Solarwinds customer discovered some unusual traffic coming from the Orion network management server and contacted SolarWinds to discuss the matter. By monitoring the email exchanges of some personnel at Solarwinds, the APT organization discovered that its attack behavior might be exposed, so it deleted the Sunspot and Sunburst backdoor samples in the Orion software build environment.
On November 26, 2020, the attacker logged into Solarwinds’ VPN account for the last time, and then continued to monitor the email exchanges between Solarwinds executives and security personnel, presumably to learn about the progress of the attack.
On December 12, 2020, foreign security personnel reported the Solarwinds supply chain backdoor incident for the first time, causing an uproar in the security circle. However, everyone needs to note that it has been at least two years since the APT organization invaded Solarwinds without being discovered.
On December 18, 2020, FireEye, Microsoft, and GoDaddy took over the C2 domain name of the Sunburst backdoor communication and changed the resolution IP of the related domain name to 20.140.0.1. According to the later reverse analysis of the backdoor sample and its built-in rules, when the C2 When the IP address of domain name resolution is 20.140.0.0/15, the Sunburst backdoor will permanently terminate the operation (the attacker is very smart and controls the C2 domain name to resolve to different IPs, and then the Sunburst backdoor will perform different operations based on the domain name resolution IP) , this process ABC_123 will be explained in detail in the next article).
APT attack organization determination
There is still no official conclusion on which APT organization carried out the attack. Kaspersky researchers found that in terms of algorithm, the SUNBURST backdoor has many similarities with the Kazuar malware used by the Russian APT organization Turla , so it is judged to be related to the Russian APT organization; foreign insiders believe that it is an organization with Russian background, APT29. But there is no conclusive evidence; there are also reports that it was initiated by the US government, with its own people beating their own people.
Personal opinion: Judging from the implementation function and design concept of the backdoor, it does not look like it was implemented by the American national APT organization. In the U.S. network arsenal, any backdoor program chosen at random is more practical than the third-stage backdoor. There is really no need to use CobaltStrike remote control, which has been thoroughly analyzed by security personnel. Of course, I am not referring to the second-stage backdoor Sunburst.
To sum up, I personally prefer this view: the Russian APT29 organization launched this Solawinds supply chain attack.
Part4 Summary of Attack Events
1. The Sunspot tool and Sunburst backdoor written by this APT organization use very sophisticated technology, which corresponds to the Advanced technology of APT attacks.
2. In order to carry out this APT supply chain attack, the APT attacker prepared for up to a year in building a source code compilation and testing environment and writing backdoor code. Corresponding to the P of APT attack events, persistence and bypassing various protection.
3. This attack affected 18,000 Orion customer companies and successfully invaded about 100 core units in the United States. It corresponds to the T of APT attacks, which is harmful.
4. If the Oriaon customer company has configured the Orion network management software server to only communicate with the SolarWinds official website in advance according to Solarwinds' security operating specifications, or placed it behind a firewall for isolation, then the Sunburst backdoor attack will fail because it cannot connect externally, but Microsoft Many victims, including the company and Mandiant Security, failed to do so.
5. In the first few days of tracing the source of this attack, the team was required to communicate only through phone calls and external accounts to prevent attackers from learning the latest status of the traceability work by monitoring email exchanges .
6. ABC_123 will write a follow-up article specifically to introduce the design ideas of the Sunburst backdoor, how to bypass layers of traffic monitoring, and how the US network security company traced the source of this attack, so stay tuned.
The public account focuses on sharing network security technology, including APT event analysis, red team attack and defense, blue team analysis, penetration testing, code audit, etc. One article per week, 99% original, so stay tuned.
Contact me: 0day123abc#gmail.com(replace # with @)