XZ Utils is a free general-purpose data compression software with a high compression ratio. It is based on the LZMA compression algorithm, which is an improved version of the Lempel-Ziv algorithm. As a popular compression tool, XZ Utils is widely bundled in various Linux distributions and software packages.

On March 29, 2024, security researcher Andres Freund discovered a backdoor in the upstream xz/liblzma package. The backdoor was injected into the distributed tarballs of the xz package, which is part of the build process for xz-utils and liblzma.

What is the XZ backdoor?

When specific conditions are met, the backdoor is triggered and modifies the Makefile of liblzma to interfere with its symbol resolution process at runtime. This manipulation causes the RSA_public_decrypt@....pl symbol to point to the attacker’s code. Given that some Linux distributions such as Debian and Red Hat or Fedora are building the xz package as part of their package build process, and along with a specific patched OpenSSH version, the backdoor gets included in their software packages.

Why is the XZ backdoor significant? During the public key authentication process of sshd, a commonly used program and protocol (SSH) used to connect to remote hosts, the backdoor code is executed, allowing the attacker to completely bypass authentication.

How did the malware get into the XZ package?

A backdoor available in widely used Linux distros such as Debian, Red Hat, and Ubuntu, allowing attackers to bypass authentication and gain unauthorized access to systems is by any measure a world-wide threatening event in cybersecurity. So how did it happen? how did the attacker manage to inject the backdoor into the XZ package?

The attacker, under the pseudonym JiaT75 (Jia Tan), created a GitHub account in 2021 and began contributing to various open-source projects. They submitted a patch to the libarchive project, which was merged without much discussion. This patch replaced a safe function with an unsafe variant, potentially introducing another vulnerability. Following that, Jia Tan began pressuring the maintainer of the xz package to add another maintainer to the project. After gaining trust, Jia Tan made their first commit to the xz repository, and over time, they became the second most active contributor to the project.

As a project contributor, Jia Tan introduced a series of changes to the xz package, and also to Google’s oss-fuzz project, would later prove pivotal in the creating the backdoor. Further actors were involved throughout 2021 and up until 2024, including Jia Tan themself, and unidentified personas such as Jigar Kumar and Dennis Ens. These actors were involved in pressuring maintainers to merge patches and make changes to the project that would ultimately lead to the backdoor being introduced.

In 2023, Jia Tan merged their first commit to the xz repository, which gave them full trust within the project. They also began pushing for the inclusion of the compromised version of xz in Debian. A number of other suspicious accounts with little former activity also pushed for its inclusion, including misoeater91 and krygorin4545. krygorin4545’s PGP key was created two days before joining the discussion.

How did Andres Freund discover the XZ backdoor?

Not all heroes wear capes, some use Valgrind.

What is Valgrind? Glad you asked! Valgrind is a tool developers use for memory debugging, memory leak detection, and profiling. It is widely used by system programmers to identify memory-related issues in their code. In case you’re wondering, system programmers are developer who run code that mostly interacts at a low level with the operating system, such as Rust, C and C++.

So back to our hero - Andres Freund. He discovered the backdoor after noticing unusual behavior associated with sshd. He observed that sshd was consuming a surprisingly large amount of CPU during the login process. Additionally, he encountered numerous errors while using the Valgrind tool for profiling and memory debugging, prompting him to investigate further.

Thanks to Andres’ sense of curiosity, keen observation and technical skills, the backdoor was discovered and brought to light, allowing the affected parties to take necessary actions to mitigate the security risk.

Our own JavaScript backdoor

So, finally coming around to the JavaScript perspective of the XZ backdoor and where does it meet us JavaScript developers.

5 years ago, on November 2018, the JavaScript ecosystem discovered one of the most significant and sophisticated supply chain attacks in the history of the npm registry known to that date - the event-stream incident.

The event-stream incident involved the introduction of a malicious package named flatmap-stream as a dependency to the popular event-stream package, which was downloaded nearly 8 million times in just 2.5 months. The malicious package was added by a new maintainer, right9ctrl, who was granted publishing rights by the original maintainer, dominictarr, after offering to help maintain the package.

Sounds familiar? History repeats, with a bang.

The flatmap-stream package contained obfuscated code that was designed to steal bitcoins from applications using the event-stream package. The malicious code was not detected for 2.5 months until a developer, Jayden Seric, noticed on October 29th 2018 an unexpected deprecation warning in the nodemon package and reported it upstream in an issue. The issue was later linked to the event-stream package, which led to the discovery of the malicious dependency. This report of a strange-looking dependency and a Node.js server bootstrap error, lead to the discovery of the backdoor.

The event-stream incident is a prime example of a supply chain attack in the open-source ecosystem, and as we are finding out 5 years later, the security and trust associated with open-source software is still a major concern. At the very least, the incident highlights the risks associated with using third-party dependencies in software projects and the potential security vulnerabilities that can arise from these dependencies.

Node.js and supply chain security in light of the XZ backdoor

The impact of the XZ backdoor is significant not solely because of the malware and its technical implications, but also because of trust, sustainability and security concerns in the open-source ecosystem that it brings with it.

Is trust now effectively eroding for welcoming new open source contributors?

To conclude, I’m jotted down a few thoughts on the XZ backdoor incident and its implications for open-source software and supply chain security:

• The backdoor code is only triggered in certain conditions, which makes it harder to detect. Pre-conditions include the OS being x86-64 Linux, the build process being part of a Debian or RPM package build, and the absence of certain environment variables. It also requires a specific patched OpenSSH version to build against the vulnerable lzma library, making it difficult for researchers to trace the backdoor. So effectively, the backdoor is incredibly stealthy and difficult to detect, let along the entire process that spanned over 3 years to set the stage and introduce it. It’s as if we discovered a time bomb that was set to explode in the future with pure luck.

• What if there’s a correlated history of this backdoor, typosquatting and malicious packages added to the npm registry? to PyPI or to RubyGems? Can we trace back the git-associated email address for Jia Tan’s contributions and other actors involved in the XZ backdoor?

• The important lesson of choosing a stable and LTS releases: the backdoor succeeded in being bundled in Debian unstable (sid), Debian testing (trixie) and Fedora rawhide. It’s a stark reminder to avoid the bleeding edge and to stick to stable and long-term support releases for production environments.

• with enough eyes, all bugs are shallow - Linus’s Law. The backdoor was discovered by Andres Freund, who noticed unusual behavior associated with sshd. This is literally the power of open source. How could this have been found if it was a closed source software at a vendor project? Would ‘an Andres Freund’ have been able to find it or would be at all interested in debugging it further if this was a proprietary software that slowed down? And what would be the turnaround ticket and support time for it?

• Spoofing git history, and git contributor accounts is relatively easily task to pull off. This is often referenced with repojacking, typosquatting and other forms of identity and integrity attacks. How can we prevent this? How do we take this into account? And no, it’s not about PGP signing commits. Verified commits are not a silver bullet and would literally add no value in this case so they wouldn’t have helped.

• Yagiz Nizipli, a core contributor and TSC member of the Node.js project had swiftly taken action to draft a proposal to limit Node.js repository access and permissions to maintainers who have not been active for a certain period of time. Sounds good to me, and an overall good practice following least privilege principle, to prevent a future backdoor from being introduced by inactive maintainers, whether intentionally or unintentionally.

• Open source contributions: a blessing or a curse? What yesterday was considered a welcoming act of collaboration, faithful effort and community spirit, today is a potential security risk. How do we balance the spirit and benefits of open source contributions with the risks of introducing a backdoor and their impact on supply chain security?

Time will tell.

In the meanwhile, stay safe out there and keep an eye out for the next backdoor. It might be closer than you think.