Thousands of Linux systems infected by stealthy malware since 2021

Thousands of machines running Linux have been infected by a malware strain that’s notable for its stealth, the number of misconfigurations it can exploit, and the breadth of malicious activities it can perform, researchers reported Thursday.

The malware has been circulating since at least 2021. It gets installed by exploiting more than 20,000 common misconfigurations, a capability that may make millions of machines connected to the Internet potential targets, researchers from Aqua Security said. It can also exploit CVE-2023-33246, a vulnerability with a severity rating of 10 out of 10 that was patched last year in Apache RocketMQ, a messaging and streaming platform that’s found on many Linux machines.

Perfctl storm

The researchers are calling the malware Perfctl, the name of a malicious component that surreptitiously mines cryptocurrency. The unknown developers of the malware gave the process a name that combines the perf Linux monitoring tool and ctl, an abbreviation commonly used with command line tools. A signature characteristic of Perfctl is its use of process and file names that are identical or similar to those commonly found in Linux environments. The naming convention is one of the many ways the malware attempts to escape notice of infected users.

Perfctl further cloaks itself using a host of other tricks. One is that it installs many of its components as rootkits, a special class of malware that hides its presence from the operating system and administrative tools. Other stealth mechanisms include:

• Stopping activities that are easy to detect when a new user logs in
• Using a Unix socket over TOR for external communications
• Deleting its installation binary after execution and running as a background service thereafter
• Manipulating the Linux process pcap_loop through a technique known as hooking to prevent admin tools from recording the malicious traffic
• Suppressing mesg errors to avoid any visible warnings during execution.

The malware is designed to ensure persistence, meaning the ability to remain on the infected machine after reboots or attempts to delete core components. Two such techniques are (1) modifying the ~/.profile script, which sets up the environment during user login so the malware loads ahead of legitimate workloads expected to run on the server and (2) copying itself from memory to multiple disk locations. The hooking of pcap_loop can also provide persistence by allowing malicious activities to continue even after primary payloads are detected and removed.

Besides using the machine resources to mine cryptocurrency, Perfctl also turns the machine into a profit-making proxy that paying customers use to relay their Internet traffic. Aqua Security researchers have also observed the malware serving as a backdoor to install other families of malware.

Assaf Morag, Aqua Security’s threat intelligence director, wrote in an email:

Perfctl malware stands out as a significant threat due to its design, which enables it to evade detection while maintaining persistence on infected systems. This combination poses a challenge for defenders and indeed the malware has been linked to a growing number of reports and discussions across various forums, highlighting the distress and frustration of users who find themselves infected.

Perfctl uses a rootkit and changes some of the system utilities to hide the activity of the cryptominer and proxy-jacking software. It blends seamlessly into its environment with seemingly legitimate names. Additionally, Perfctl’s architecture enables it to perform a range of malicious activities, from data exfiltration to the deployment of additional payloads. Its versatility means that it can be leveraged for various malicious purposes, making it particularly dangerous for organizations and individuals alike.

“The malware always manages to restart”

While Perfctl and some of the malware it installs are detected by some antivirus software, Aqua Security researchers were unable to find any research reports on the malware. They were, however, able to find a wealth of threads on developer-related sites that discussed infections consistent with it.

This Reddit comment posted to the CentOS subreddit is typical. An admin noticed that two servers were infected with a cryptocurrency hijacker with the names perfcc and perfctl. The admin wanted help investigating the cause.

“I only became aware of the malware because my monitoring setup alerted me to 100% CPU utilization,” the admin wrote in the April 2023 post. “However, the process would stop immediately when I logged in via SSH or console. As soon as I logged out, the malware would resume running within a few seconds or minutes.” The admin continued:

I have attempted to remove the malware by following the steps outlined in other forums, but to no avail. The malware always manages to restart once I log out. I have also searched the entire system for the string "perfcc" and found the files listed below. However, removing them did not resolve the issue. as it keep respawn on each time rebooted.

Other discussions include: Reddit, Stack Overflow (Spanish), forobeta (Spanish),  brainycp (Russian), natnetwork (Indonesian), Proxmox (Deutsch), Camel2243 (Chinese), svrforum (Korean), exabytes, virtualmin, serverfault and many others.

After exploiting a vulnerability or misconfiguration, the exploit code downloads the main payload from a server, which, in most cases, has been hacked by the attacker and converted into a channel for distributing the malware anonymously. An attack that targeted the researchers’ honeypot named the payload httpd. Once executed, the file copies itself from memory to a new location in the /tmp directory, runs it, and then terminates the original process and deletes the downloaded binary.

Once moved to the /tmp directory, the file executes under a different name, which mimics the name of a known Linux process. The file hosted on the honeypot was named sh. From there, the file establishes a local command-and-control process and attempts to gain root system rights by exploiting CVE-2021-4043, a privilege-escalation vulnerability that was patched in 2021 in Gpac, a widely used open source multimedia framework.

The malware goes on to copy itself from memory to a handful of other disk locations, once again using names that appear as routine system files. The malware then drops a rootkit, a host of popular Linux utilities that have been modified to serve as rootkits, and the miner. In some cases, the malware also installs software for “proxy-jacking,” the term for surreptitiously routing traffic through the infected machine so the true origin of the data isn’t revealed.

The researchers continued:

As part of its command-and-control operation, the malware opens a Unix socket, creates two directories under the /tmp directory, and stores data there that influences its operation. This data includes host events, locations of the copies of itself, process names, communication logs, tokens, and additional log information. Additionally, the malware uses environment variables to store data that further affects its execution and behavior.

All the binaries are packed, stripped, and encrypted, indicating significant efforts to bypass defense mechanisms and hinder reverse engineering attempts. The malware also uses advanced evasion techniques, such as suspending its activity when it detects a new user in the btmp or utmp files and terminating any competing malware to maintain control over the infected system.

The diagram below captures the attack flow:

Credit: Aqua Security

The following image captures some of the names given to the malicious files that are installed:

Credit: Aqua Security

By extrapolating data such as the number of Linux servers connected to the Internet across various services and applications, as tracked by services such as Shodan and Censys, the researchers estimate that the number of machines infected by Perfctl is measured in the thousands. They say that the pool of vulnerable machines—meaning those that have yet to install the patch for CVE-2023-33246 or contain a vulnerable misconfiguration—is in the millions. The researchers have yet to measure the amount of cryptocurrency the malicious miners have generated.

People who want to determine if their device has been targeted or infected by Perfctl should look for indicators of compromise included in Thursday’s post. They should also be on the lookout for unusual spikes in CPU usage or sudden system slowdowns, particularly if they occur during idle times. To prevent infections, it's important that the patch for CVE-2023-33246 be installed and that the the misconfigurations identified by Aqua Security be fixed. Thursday’s report provides other steps for preventing infections.