A China-linked cyber espionage group known as Velvet Ant spent nearly a decade inside the internal network of an unnamed organization without being detected, according to the results of a forensic investigation published by cybersecurity firm Sygnia.

The group’s defining characteristic is the ability to maintain stealthy years-long persistence in target environments. In this particular case, booting them out also took considerable effort, as they managed to gain control of the full authentication stack by backdooring privilege access management (PAM) modules and OpenSSH binaries across multiple hosts.

Taking over the authentication layer

Sygnia’s report does not say how the group gained initial access to the target organization’s internet-facing servers, only that they:

• Deployed a modified version of the GS-Netcat utility on those servers to establish a reverse shell connection to a remote C2 server
• Worked their way deeper into the organization’s internal network by using modified Nginx configurations and a custom-built binary that establishes an SSH connection to an internal server when triggered via a HTTP POST request
• Used a custom implementation of the open-source ssspl SOCKS5 proxy server for tunneling and lateral movement.

But the most consequential phase of the intrusion was Velvet Ant’s takeover of the authentication layer: the group modified PAM modules responsible for verifying logins, and the OpenSSH binaries used for remote access.

The modified PAM module could either accept a hardcoded secret password to bypass authentication entirely, or do that and silently log every username and password entered by legitimate users to a hidden file.

“Nine distinct pam_unix.so variants were identified, each built in a separate compile environment – the level of effort required to produce and maintain these variants points to a well-resourced, deliberate operation,” the analysts noted.

The modified SSH binaries went further: they captured credentials from both incoming and outgoing connections, logged every command typed during active sessions, and stored everything in encrypted files disguised to blend into the filesystem.

For added stealth, a custom flag in these binaries could be used to suppress credential and key logging, and a separate flag enabled the binary to disguise the process name so that it would blend in with system processes.

As a third layer, they appended their own keys to the authorized_keys files on compromised servers, giving themselves password-free access that would survive even a full password rotation.

Eradication and detection challenges

Evicting Velvet Ant was not a matter of deleting a rogue file or disabling a service, and removing one or two of the three persistence mechanisms would not be enough.

“From an eradication perspective, replacing a malicious service is one thing, while replacing PAM modules and OpenSSH binaries is another. A wrong package, incompatible binary or a missing dependency can completely lock administrators out of a host. In critical infrastructure, that can turn remediation into a production outage,” Sygnia pointed out.

The wide variety of Linux distros and versions running on the affected server presented an added difficulty, as remediation packages had to be tweaked to work on each of them.

Finally, all of this had to be tested before the replacement packages were actually installed on production system, and the team prepared rollback options for various failure scenarios.

“[This operation] is a case study in why signature-based detection and alert-driven security operations fall short against a patient, capable threat actor,” they noted.

“There was no novel exploit to catch; no clearly malicious binary dropping into a monitored directory. Velvet Ant operates through pam_unix.so, sshd, and ssh – components that exist on virtually every Linux host in the environment, that behave normally for legitimate users, and that generate no anomalous log entries when backdoored. The attacker’s presence was, by design, indistinguishable from legitimate administrative activity.”

Sygnia advised organizations to deploy a variety of critical security controls and to engage in proactive threat hunting.

“Proactive threat hunting shifts the analytical frame from ‘what is known to be malicious’ to ‘what is inconsistent, unexpected, or unjustified in this environment.’ That distinction matters enormously in segmented or high-sensitivity networks, where the assumption of isolation can create a false sense of security,” they concluded.

Subscribe to our breaking news e-mail alert to never miss out on the latest breaches, vulnerabilities and cybersecurity threats. Subscribe here!