Description
If you work in security, the chances are that you have spent the last several days urgently responding to the Log4Shell vulnerability (CVE-2021-44228), investigating where you have instances of Log4j in your environment, and questioning your vendors about their response. You have likely already read up on the implications and steps that need to be taken. This blog is for everyone else who wants to understand what’s going on and why the internet seems to be on fire again. And for you security professionals, we’ve also included some questions on the broader implications and long term view. You know, for all that spare time you have right now.
What is Log4Shell?
Log4Shell — also known as CVE-2021-44228 — is a critical vulnerability that enables remote code execution in systems using the Apache Foundation’s Log4j, which is an open-source Java library that is extensively used in commercial and open-source software products and utilities. For a more in-depth technical assessment of Log4Shell check out Rapid7’s AttackerKB analysis.
What is Log4j?
Log4j is one of the most common tools for sending text to be stored in log files and/or databases. It is used in millions of applications and websites in every organization all across the internet. For example, information is sent to keep track of website visitors, note when warnings or errors occur in processing, and help support teams’ triage problems.
Get answers to your Log4Shell questions from our experts
Sign up for our webinar on Thursday, December 16
So what’s the problem?
It turns out that Log4j doesn’t just log plain strings. Text strings that are formatted a certain way will be executed just like a line from a computer program. The problem is that this allows malicious actors to manipulate computers all over the internet into taking actions without the computer owners’ wishes or permission. Cyberattacks can use this to steal information, force actions, or extort the computer owners or operators.
This vulnerability is what we’re referring to as Log4Shell, or CVE-2021-44228. Log4j is the vulnerable technology. As this is a highly evolving situation, you can always head over to our main live blog on Log4Shell.
Is it really that big of a deal?
In a word, yes.
Caitlin Kiska, an information security engineer at Cardinal Health, put it this way: “Imagine there is a specific kind of bolt used in most of the cars and car parts in the world, and they just said that bolt needs to be replaced.” Glenn Thorpe, Rapid7’s Emergent Threat Response Manager added, “… and the presence of that bolt allows anyone to just take over the car.”
The first issue is Log4j’s widespread use. This little tool is used in countless systems across the internet, which makes remediation or mitigation of this into a huge task — and makes it more likely something might get missed.
The second issue is that attackers can use it in a variety of ways, so the sky is sort of the limit for them.
Perhaps the biggest concern of all is that it’s actually pretty easy to use the vulnerability for malicious purposes. Remote code execution vulnerabilities are always concerning, but you hope that they will be complicated to exploit and require specialist technical skill, limiting who can take advantage of them and slowing down the pace of attacks so that defenders can ideally take remediation action first. That’s not the case with Log4Shell. The Log4j tool is designed to send whatever data is inputted in the right format, so as long as attackers know how to form their commands into that format (which is not a secret), they can take advantage of this bug, and they currently are doing just that.
That sounds pretty bad. Is the situation hopeless?
Definitely not. The good news is that the Apache Foundation has updated Log4j to address the vulnerability. All organizations urgently need to check for the presence of this vulnerability in their environment and update affected systems to the latest patched version.
The first update — version 2.15.0 — was released on December 6, 2021. As exploitation ramped up in the wild, it became clear that the update did not fully remediate the issue in all use cases, a vulnerability that the National Vulnerability Database (NVD) codified as CVE-2021-45046.
As a result, on December 13, the Apache Foundation released version 2.16.0, which completely removes support for message lookup patterns, thus slamming the door on JNDI functionality completely and possibly adding to development team backlogs to update material sections on their codebase that handle logging.
That sounds straightforward, right?
Unfortunately, it’s likely going to be a pretty huge undertaking and likely require different phases of discovery and remediation/mitigation.
Remediation
The first course of action is to identify all vulnerable applications, which can be a mix of vendor-supplied solutions and in-house developed applications. NCSC NL is maintaining a list of impacted software, but organizations are encouraged to monitor vendor advisories and releases directly for the most up-to-date information.
For in-house developed applications, organizations — at a minimum — need to update their Log4j libraries to the latest version (which, as of 2021-12-14, is 2.16) and apply the mitigations described in Rapid7’s initial blog post on CVE-2021-44228, which includes adding a parameter to all Java startup scripts and strongly encourages updating Java virtual machine installations to the latest, safest versions. An additional resource, published by the Apache Security Team, provides a curated list of all affected Apache projects, which can be helpful to expedite identification and discovery.
If teams are performing “remote” checks (i.e., exploiting the vulnerability remotely as an attacker would) versus local filesystems “authenticated” checks, the remote checks should be both by IP address and by fully qualified domain name/virtual host name, as there may be different routing rules in play that scanning by IP address alone will not catch.
These mitigations must happen everywhere there is a vulnerable instance of Log4j. Do not assume that the issue applies only to internet-facing systems or live-running systems. There may be batch jobs that run hourly, daily, weekly, monthly, etc., on stored data that may contain exploit strings that could trigger execution.
Forensics
Attacks have been active since at least December 1, 2021, and there is evidence this weakness has been known about since at least March of 2021. Therefore, it is quite prudent to adopt an “assume breach” mindset. NCSC NL has a great resource page on ways you can detect exploitation attempts from application log files. Please be aware that this is not just a web-based attack. Initial, quite public, exploit showcases included changing the name of iOS devices and Tesla cars. Both those companies regularly pull metadata from their respective devices, and it seems those strings were passed to Log4j handlers somewhere in the processing chain. You should review logs from all internet-facing systems, as well as anywhere Log4j processing occurs.
Exploitation attempts will generally rely on pulling external resources in (as is the case with any attack after gaining initial access), so behavioral detections may have already caught or stopped some attacks. The Log4j weakness allows for rather clever data exfiltration paths, especially DNS. Attackers are pulling values from environment variables and files with known filesystems paths and creating dynamic domain names from them. That means organizations should review DNS query logs going as far back as possible. Note: This could take quite a bit of time and effort, but it must be done to ensure you’re not already a victim.
Proactive response
Out of an abundance of caution, organizations should also consider re-numbering critical IP segments (where Log4j lived), changing host names on critical systems (where Log4j lived), and resetting credentials — especially those associated with Amazon AWS and other cloud providers — in the event they have already been exfiltrated.
Who should be paying attention to this?
Pretty much every organization, regardless of size, sector, or geography. If you have any kind of web presence or internet connectivity, you need to pay attention and check your status. If you outsource all the technical aspects of your business, ask your vendors what they are doing about this issue.
Who is exploiting it and how?
Kind of… everyone.
“Benign” researchers (some independent, some part of cybersecurity firms) are using the exploit to gain an initial understanding of the base internet-facing attack surface.
Cyberattackers are also very active and are racing to take advantage of this vulnerability before organizations can get their patches in place. Botnets, such as Mirai, have been adapted to use the exploit code to both exfiltrate sensitive data and gain initial access to systems (some deep within organizations).
Ransomware groups have already sprung into action and weaponized the Log4j weakness to compromise organizations. Rapid7’s Project Heisenberg is collecting and publishing samples of all the unique attempts seen since December 11, 2021.
How are things likely to develop?
These initial campaigns are only the beginning. Sophisticated attacks from more serious and capable groups will appear over the coming days, weeks, and months, and they’ll likely focus on more enterprise-grade software that is known to be vulnerable.
Most of the initial attack attempts are via website-focused injection points (input fields, search boxes, headers, etc.). There will be more advanced campaigns that hit API endpoints (even “hidden” APIs that are part of company mobile apps) and try to find sneaky ways to get exploit strings past gate guards (like the iOS device renaming example).
Even organizations that have remediated deployed applications might miss some virtual machine or container images that get spun up regularly or infrequently. The Log4Shell attacks are easily automatable, and we’ll be seeing them as regularly as we see WannaCry and Conficker (yes, we still see quite a few exploits on the regular for those vulnerabilities).
Do we need to worry about similar vulnerabilities in other systems?
In the immediate term, security teams should narrow their attention to identify systems with the affected Log4j packages.
For the longer term, while it is impossible to forecast identification of similar vulnerabilities, we do know the ease and prevalence of CVE-2021-44228 demands the continued attention (been a long weekend for many) of security, infrastructure, and application teams.
Along with Log4Shell, we also have CVE-2021-4104 — reported on December 9, 2021 — a flaw in the Java logging library Apache Log4j in version 1.x. JMSAppender that is vulnerable to deserialization of untrusted data. This allows a remote attacker to execute code on the server if the deployed application is configured to use JMSAppender and to the attacker’s JMS Broker. Note this flaw only affects applications that are specifically configured to use JMSAppender, which is not the default, or when the attacker has write-access to the Log4j configuration for adding JMSAppender to the attacker’s JMS Broker.
Exploit vectors of Log4Shell and mitigations reported on Friday continue to evolve as reported by our partners at Snyk.io and Java Champion, Brian Vermeer — see “Editor’s note (Dec. 13, 2021)” — therefore, continued vigilance on this near and present threat is time well spent. Postmortem exercises (and there will be many) should absolutely include efforts to evolve software, open-source, and package dependency inventories and, given current impacts, better model threats from packages with similar uninspected lookup behavior.
Does this issue indicate that we should stop compiling systems and go back to building from scratch?
There definitely has been chatter regarding the reliance upon so many third-party dependencies in all areas of software development. We’ve seen many attempts at poisoning numerous ecosystems this year, everything from Python to JavaScript, and now we have a critical vulnerability in a near-ubiquitous component.
On one hand, there is merit in relying solely on code you develop in-house, built on the core features in your programming language of choice. You can make an argument that this would make attackers’ lives harder and reduce the bang they get for their buck.
However, it seems a bit daft to fully forego the volumes of pre-built, feature-rich, and highly useful components. And let’s not forget that not all software is created equally. The ideal is that community built and shared software will benefit from many more hands to the pump, more critical eyes, and a longer period to mature.
The lesson from the Log4Shell weakness seems pretty clear: Use, but verify, and support! Libraries such as Log4j benefit from wide community adoption, since they gain great features that can be harnessed quickly and effectively. However, you cannot just assume that all is and will be well in any given ecosystem, and you absolutely need to be part of the community vetting change and feature requests. If more eyes were on the initial request to add this fairly crazy feature (dynamic message execution) and more security-savvy folks were in the approval stream, you would very likely not be reading this post right now (and it’d be one of the most boring Marvel “What If…?” episodes ever).
Organizations should pull up a seat to the table, offer code creation and review support, and consider funding projects that become foundational or ubiquitous components in your application development environment.
How would a Software Bill of Materials (SBOM) have impacted this issue?
A Bill of Materials is an inventory, by definition, and conceptually should contribute to speeding the discovery timeline during emergent vulnerability response exercises, such as the Log4j incident we are communally involved in now.
An SBOM is intended to be a formal record that contains the details and supply chain relationships of various components used in software, kind of like an ingredients list for technology. In this case, those components would include java libraries used for logging (e.g. Log4j2).
SBOM requirements were included in the US Executive Order issued in May 2021. While there may be international variations, the concept and intended objects are uniform. For that reason, I will reference US progress for simplicity.
From: The Minimum Elements For a Software Bill of Materials (SBOM), issued Department of Commerce, in coordination with the National Telecommunications and Information Administration (NTIA), Jul 12, 2021
The question many Log4Shell responders — including CISOs, developers, engineers, sys admins, clients, and customers — are still grappling with is simply where affected versions of Log4j are in use within their technical ecosystems. Maintaining accurate inventories of assets has become increasingly challenging as our technical environments have become more complicated, interconnected, and wide-reaching. Our ever-growing reliance on internet-connected technologies, and the rise of shadow IT only make this problem worse.
Vulnerabilities in tools like Log4j, which is used broadly in a vast array of technologies and systems, highlight the need for more transparency and automation for asset and inventory management. Perhaps the longer-term substantive impact from Log4Shell will be to refocus investments and appreciation for the cruciality of an accurate inventory of software and associated components through an SBOM that can easily be queried and linked to dependencies.
The bottom line is that if we had lived in a timeline where SBOMs were required and in place for all software projects, identifying impacted products, devices, and ecosystems would have been much easier than it has been for Log4Shell and remediation would likely be faster and more effective.
How does Log4Shell impact my regulatory status — do I need to take special action to ensure compliance?
According to Rapid7’s resident policy and compliance expert, Harley Geiger, “Regulators may not have seen Log4Shell coming, but now that the vulnerability has been discovered, there will be an expectation that regulated companies address it. As organizations’ security programs address this widespread and serious vulnerability, those actions should be aligned with compliance requirements and reporting. For many regulated companies, the discovery of Log4Shell triggers a need to re-assess the company’s risks, the effectiveness of their safeguards to protect sensitive systems and information (including patching to the latest version), and the readiness of their incident response processes to address potential log4j exploitation. Many regulations also require these obligations to be passed on to service providers. If a Log4j exploitation results in a significant business disruption or breach of personal information, regulations may require the company to issue an incident report or breach notification.”
We also asked Harley whether we’re likely to see a public policy response. Here’s what he said…
“On a federal policy level, organizations should expect a renewed push for adoption of SBOM to help identify the presence of Log4j (and other vulnerabilities) in products and environments going forward. CISA has also required federal agencies to expedite mitigation of Log4j and has ramped up information sharing related to the vulnerability. This may add wind to the sails of cyber incident reporting legislation that is circulating in Congress at present.”
How do we know about all of this?
Well, a bunch of kids started wreaking havoc with Minecraft servers, and things just went downhill from there. While there is some truth to that, the reality is that an issue was filed on the Log4j project in late November 2021. Proof-of-concept code was released in early December, and active attacks started shortly thereafter. Some cybersecurity vendors have evidence of preliminary attacks starting on December 1, 2021.
Anyone monitoring the issue discussions (something both developers, defenders, and attackers do on the regular) would have noticed the gaping hole this “feature” created.
How long has it been around?
There is evidence of a request for this functionality to be added dating back to 2013. A talk at Black Hat USA 2016 mentioned several JNDI injection remote code execution vectors in general but did not point to Log4j directly.
Some proof-of-concept code targeting the JNDI lookup weakness in Log4j 2.x was also discovered back in March of 2021.
Fear, Uncertainty, and Doubt (FUD) is in copious supply today and likely to persist into the coming weeks and months. While adopting an “assumed breach” mindset isn’t relevant for every celebrity vulnerability, the prevalence and transitive dependency of the Log4j library along with the sophisticated obfuscation exploit techniques we are witnessing in real time point out that the question we should be considering isn’t, “How long has it been around?” Rather, it is, “How long should we be mentally preparing ourselves (and setting expectations) to clean it up?”
We are adding more questions as they come our way. If you have questions you’d like answered, please let us know.
[UPDATE: December 17, 2021, 6 PM ET]
I’ve heard that the update doesn’t work and you can still be exploited even if you have updated? Should I panic?
If you’ve updated to either v2.15 (the original fix) or v2.16 (the latest fix), you don’t need to panic. It’s true that the v2.15 update “was incomplete in certain non-default configurations,” and that was designated as a separate vulnerability: CVE-2021-45046. But the key words here are** “in certain non-default configurations.”**
Essentially, when a logging configuration uses a non-default Pattern Layout with a Context Lookup, attackers with control over Thread Context Map (MDC) input data can craft malicious input data using a JNDI Lookup pattern. This can result in DoS attacks, and we’ve now discovered it can also result in information leaks, and in some specific cases, RCE. This has resulted in the vulnerability being upgraded from a CVSS score of 3.7 to 9.0 on the Apache Foundation website – but the RCE is currently only being reported for macOS, and no in-the-wild-exploitation has been publicly reported, so it still isn’t time to panic. The bottom line is that you do need to update to v2.17 as soon as you can, but unless you have those non-default configurations, v2.15 probably has you covered.
I’ve heard that there’s now evidence of ransomware attacks against this vulnerability. Should I panic?
It is true that reports have started to appear of the ransomware group, Conti, leveraging CVE-2021-44228 (Log4Shell) to mount attacks. According to a report from AdvIntel, Conti is testing exploitation by targeting vulnerable Log4j2 instances in VMware vCenter “for lateral movement directly from the compromised network resulting in vCenter access affecting US and European victim networks from the pre-existent Cobalt Strike sessions.” While it’s not time to panic about this, we do expect to see much more widespread ransom-based exploitation to follow in coming weeks, and this is another reason to ensure you are on the latest version (v2.17) as soon as possible.
Is it OK to scan for vulnerable applications or systems?
If you own or lease systems and have appropriate authorization, it is absolutely fine to scan to identify vulnerable applications or systems – in fact, we strongly recommend you do so in your own environments so you can get patching.
Beyond that, while laws vary by country, most anti-hacking laws revolve around not exceeding authorized access or accessing systems without authorization, so scanning someone else’s assets without permission may fall foul of the law.
For example, as our resident US public policy expert, Harley Geiger, explains in this tweet, under the US anti-hacking law, the Computer Fraud and Abuse Act (CFAA), “If the test involves unauthorized exfiltration of nonpublic data from a target system or causes a target system to download your executable code w/o authorization, even if done in good faith, stop & make friends w a lawyer.”
