Ever since Log4Shell came into our lives, the internet has been flooded with daily Log4Shell CVEs updates, Log4j releases, as well as outdated recommendations and discredited mitigations. With all this data abundance, it is challenging to keep up or know what you should do next. Our research shows that among the organizations that mitigated the Log4j vulnerabilities 55% of their workloads are still vulnerable to Log4Shell (as of December 21, 2021). The main challenge here is understanding the existing infrastructure, and identifying the location of all vulnerable Log4j libraries.

Meanwhile, attackers are on the hunt to exploit Log4Shell in the wild. In days, they probed half of all corporate networks globally for the exploit. With more than a hundred variants of Log4Shell exploits, mishandling or slow handling of the situation could quickly escalate to your worst nightmare of data leaks, ransomware, or an infrastructure overtake.

This post covers all the latest updates, and presents you with guidelines to identify all vulnerable versions of Log4j in your cloud environment, hopefully before you leave for the holiday season with your beloved ones.

How to prioritize patching

Before we dive into the Log4Shell events and details, let’s start with a summary of the four Log4j vulnerabilities that were discovered in the past 10 days.

Wiz recommends security teams focus on resources that are still vulnerable to CVE-2021-44228 first, with log4j version lower than 2.15.0 (the original Log4Shell vulnerability), since it is easier to exploit and is still being heavily exploited in the wild. As new patches keep coming, ensure you’re patching to the latest version of log4j (2.17 as of December 21, 2021).

We've seen environments with hundreds of different workloads vulnerable to Log4Shell. Handling all of them at once is impossible. Therefore, we recommend prioritizing resources that are:

• Publicly exposed—Public exposure to the internet allows malicious actors to scan public addresses and try to send crafted messages to those addresses. VMs that are publicly facing and may have server logging via log4j can receive these messages and be compromised.

• Highly privileged—If a highly privileged VM is compromised, attackers can try to use the role the VM can assume to access resources and make changes in your environment, For example, it's quite common for VMs to be able to assume an AWS Role that has s3:* permissions. Compromised VMs with such permissions allow attackers to read every bucket in your environment or even delete buckets.

Then, you can move on to updating resources that are still vulnerable to the rest of the log4j vulnerabilities.

How it all began

On Dec 9, Alibaba's Cloud Security Team publicly disclosed a vulnerability with a 10/10 rating in a ubiquitous, most popular, Java logging library, Apache Log4j. This vulnerability allowed attackers to execute code remotely in various backends that use this library—thousands of cyber attacks followed. Less than a day after the CVE was published, Deutsche Telekom CERT tweeted about attacks in its honeypot infrastructure:

On Dec 14, by the time agile software developers patched their code, IT admins rushed to update their software and DevOps placed mitigations to prevent this RCE attack. Also, Log4j published a second 2.16.0 update. This update covered a non-default configuration vulnerability that was missing in the first update. At first, this CVE was rated low and allowed Denial of Service (DoS) attacks, so sysadmins were content with updating to 2.15.0 and postponed the update to 2.16.0 for later.

On Dec 18, the Apache Logging security team raised the second CVE rating from moderate 3.7 to critical 9.0 and changed the title from DoS to Remote Code Execution (RCE). And that’s not all - a third Log4j 2.17.0 update was published to fix yet another new 7.5 rated DoS vulnerability!

Log4Shell exploitation in the wild

Cloudflare reported seeing testing of the vulnerability on Dec 1, eight days before the vulnerability was made public. As of mid-December, this vulnerability has been actively exploited in the wild, primarily by botnets and coin miners. Two groups, in particular, stood out:

• Mirai—One of the most significant botnets targeting exposed networking devices running Linux. Several different variants of Mirai exploit Log4j vulnerabilities.

• Kinsing—Golang-based malware. It runs a cryptocurrency miner and attempts to spread itself to other hosts in the victim's environment. For more information regarding the IOCs and malware groups using the Log4Shell vulnerability, see the Log4Shell-IOCs page on github.

Ransomware groups were not lagging behind and joined the party, leveraging Log4Shell for initial access and lateral movement. Bitdefender researchers discovered Log4Shell attempts to deploy ransomware on Windows machines connected to compromised Minecraft servers. The Microsoft Threat Intelligence Center (MSTIC) reported tracking multiple nation-state activity originating from China, Iran, North Korea, and Turkey.

How does the Log4Shell attack work

These vulnerabilities come down to unsanitized user input that reaches software and tells it what to do. If you recall the SQL Injection, then you realize it’s the oldest play in the book, and that there is nothing new or fancy here. Log4j comes down to a feature called Lookups, that was introduced in Log4j 2.0.

Does this mean Log4J 1.x is not vulnerable to this attack? No, it might still be susceptible to a separately reported CVE-2021-4104 that, similarly to CVE-2021-44228, allows Remote Code Execution if your logging configuration has JMSAppender configured in it. Regardless, Log4j 1.x is not supported and has had no security patches since 2015.

Lookups allow users to pass variables and commands down to Log4j, which evaluates and handles them. For example, it is helpful to log a user's name to understand the context of the log, e.g. $${ctx:userName}. If the user can manipulate this variable, a malicious actor could leverage that to pass values that command Log4j to perform actions it shouldn't perform.

An attack commonly seen in the wild utilizes the HTTP user-agent, similar to the ShellShock of 2014. This HTTPS header usually contains the browser and the OS names and versions to allow servers to handle their users better. Usually, it looks like this: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko/20100101 Firefox/95.0

Here is an example of a user-agent value used in the wild by attackers:

If ${jdni:ldap:URL} ends up in Log4j, it will trigger loading and executing code from the URL that the attacker controls. CVE-2021-44228 covers exactly this scenario. The above command instructs Log4j to use Java Naming and Directory Interface (JNDI). JNDI is a Java runtime Application Programming Interface (API) that can retrieve objects from remote servers using a few protocols. The lookup command instructs connecting to 715de0e31919[.]bingsearchlib[.]com[:]39356 using the Lightweight Directory Access Protocol (LDAP) protocol, and fetching the object a. Both JNDI and LDAP are well-documented protocols that an attacker can leverage to execute code.

LDAP is not the only JNDI service, and Juniper reported a shift to use RMI instead of LDAP. The other two CVEs cover different scenarios using the same strategy: an attacker crafts malicious data input that bypasses various mitigations in Log4j and its configurations, which results in Remote Code Execution, data leaks, or Denial of Service.

Why is Log4Shell such a critical issue

As we have seen, Log4Shell raised three main challenges:

• Proper identification of impacted software— How do you even begin to search for software that uses another software that uses Log4j across your entire infrastructure? Can/should you even trust the chain to report the issue to you on time?

• Efficient and fast remediation—Even after locating all the relevant software, rolling out updates takes time. If updates are not available, patching is even more complex and error-prone.

• The resurrection of concepts considered long gone—Since SQL Injections first became known, sanitizing queries received an emphasis but did not become a security practice anywhere. In such an attack vector, it's not only the external servers that are at risk.

Consider user input that is stored in a DB, and is later retrieved and logged. A malicious payload sent by the user could get into the DB, and later, the logger would receive it. The next misconception is that sanitizing inputs in your external-facing server is enough to prevent such attacks. If you check the input and discard it before you store it, that would help you prevent the attack from the previous example. You could use "probably the most comprehensive Regex I’ve seen yet to identify Log4Shell exploitation attempts":

But imagine what would happen if the data received is compressed or encoded, like ZIP or base64:

The receiving server sanitizes the input and makes sure that it does not contain a malicious payload or contains base64, as it should. Later, the other service (that does not even suspect that the value originated from the user), decodes it using base64, and sends the actual payload to Log4j:

Which versions of Log4j can I use safely

As of Dec 28, you should upgrade to the latest Log4j version:

• If you use Java 8 (or later), upgrade the library to 2.17.1.

• If you use Java 7 or need JNDI, upgrade to 2.12.4.

• If you use Java 7 or need JNDI, upgrade to 2.3.2.

Log4j and many others suggested different mitigations. For example: fixing the issue by disabling lookups in certain situations using methods similar to log4j2.formatMsgNoLookups or using "%message{nolookups}". You should not use that recommendation anymore as it is insufficient in many cases and discredited.

Which software is affected by the Log4j vulnerability?

Exploiting Log4Shell is easy, and there are many resources and examples online. The use of vulnerable Log4j is so widespread that we suggest categorizing it:

• Appliance, various *aaS services, and third-party software that uses vulnerable Log4j

• Proprietary software that is developed or maintained internally and uses vulnerable Log4j

How to locate all the software that uses Log4j

For internally developed software, developers should follow the mitigation steps published by the Log4j security team. We suggest updating vulnerable libraries to the latest version. As proved in the last few days, other mitigations can leave your software vulnerable to further attacks. We do not recommend using Pattern Layout or other similar techniques on any unsanitized input from users.

Which other mitigations can I apply

Network-level mitigations could be an excellent temporary solution while you work on updating your stack. For example, you can set up Web Application Firewall (WAF) to prevent common attacks from reaching your potentially exploitable software, and prevent your servers from initiating outbound to unknown or black-listed domains.

Which third-party appliances, services, and software use Log4j?

Software vendors of appliances, services, and software that you use depend on open source as part of their stack. Their dependence on vulnerable Log4j makes you vulnerable too. All the major vendors used Log4j and reported that Log4Shell impacted them. All types of software are affected: management software like vCenter, backend solutions like Elastic, Kafka, and even Minecraft. Locating and updating the software is crucial.

Follow the steps provided by the software vendor to fix and mitigate the Log4Shell attack. Sometimes, manual actions are required even in the case of affected "managed services" by vendors like Amazon, Google, and Microsoft. The National Cyber Security Centre of Netherlands(NCSC-NL) is an excellent source for reports about Log4Shell and maintains a list of all the software, affected versions, and required mitigation steps.

Identifying if a software uses Log4j

Sometimes there is a need for a "black box" assessment of software to ensure it is not vulnerable. Here are some quick tips to help you out:

• If you're confident that the software never executes Java, it doesn't use Log4j inside it. Log4Shell does not impact projects like Log4net and Log4cxx.

• You can scan and locate log4j-core files.

• For your self-developed software, search for log4j references.

• The NCSC-NL offers a list of tools to help you detect if the software uses Log4j.

However, there are still other scenarios that are not covered above, such as recursive decencies that require deeper file analysis. With all the above detection challenges and more, Wiz can help (see below)✨

Answering all those questions is a tedious task that usually requires gaining full access to all your resources, and many tools to perform all those different queries. Manually checking whether a machine is vulnerable requires accessing it, even if it is not vulnerable.

Use Wiz to detect, track, and prioritize log4j remediation

Wiz detects log4j vulnerabilities on all cloud workloads, including VMs, containers, serverless functions and even virtual appliances. Wiz allows you to query your infrastructure stack at once, effortlessly. You can get immediate insights about all vulnerable appliances and services even if they are closed source, third-party software that uses Log4j, or your software embeds vulnerable Log4j in it. Additionally, customers can use the Wiz Security Graph to focus remediation effort on the workloads that are at greatest risk (e.g. workloads that are effectively exposed to the internet or ones that have access to sensitive resources).