
If you work in security, the chances are that you have spent the last several days urgently responding to the Log4Shell vulnerability
(CVE-2021-44228), investigating where you have instances of Log4j in
your environment, and questioning your vendors about their response. You
have likely already read up on the implications and steps that need to
be taken. This blog is for everyone else who wants to understand what’s
going on and why the internet seems to be on fire again. And for you
security professionals, we’ve also included some questions on the
broader implications and long term view. You know, for all that spare
time you have right now.
What is Log4Shell?
Log4Shell — also known as CVE-2021-44228 — is a critical
vulnerability that enables remote code execution in systems using the
Apache Foundation’s Log4j, which is an open-source Java library that is
extensively used in commercial and open-source software products and
utilities. For a more in-depth technical assessment of Log4Shell check
out Rapid7’s AttackerKB analysis.
What is Log4j?
Log4j is one of the most common tools for sending text to be stored
in log files and/or databases. It is used in millions of applications
and websites in every organization all across the internet. For example,
information is sent to keep track of website visitors, note when
warnings or errors occur in processing, and help support teams’ triage
problems.
Get answers to your Log4Shell questions from our experts
Sign up for our webinar on Thursday, December 16
So what’s the problem?
It turns out that Log4j doesn’t just log plain strings. Text strings
that are formatted a certain way will be executed just like a line from a
computer program. The problem is that this allows malicious actors to
manipulate computers all over the internet into taking actions without
the computer owners’ wishes or permission. Cyberattacks can use this to
steal information, force actions, or extort the computer owners or
operators.
This vulnerability is what we’re referring to as Log4Shell, or
CVE-2021-44228. Log4j is the vulnerable technology. As this is a highly
evolving situation, you can always head over to our main live blog on Log4Shell.
Is it really that big of a deal?
In a word, yes.
Caitlin Kiska, an information security engineer at Cardinal Health, put it this way:
“Imagine there is a specific kind of bolt used in most of the cars and
car parts in the world, and they just said that bolt needs to be
replaced.” Glenn Thorpe, Rapid7’s Emergent Threat Response Manager
added, “… and the presence of that bolt allows anyone to just take over
the car.”
The first issue is Log4j’s widespread use. This little tool is used
in countless systems across the internet, which makes remediation or
mitigation of this into a huge task — and makes it more likely something
might get missed.
The second issue is that attackers can use it in a variety of ways, so the sky is sort of the limit for them.
Perhaps the biggest concern of all is that it’s actually pretty easy
to use the vulnerability for malicious purposes. Remote code execution
vulnerabilities are always concerning, but you hope that they will be
complicated to exploit and require specialist technical skill, limiting
who can take advantage of them and slowing down the pace of attacks so
that defenders can ideally take remediation action first. That’s not the
case with Log4Shell. The Log4j tool is designed to send whatever data
is inputted in the right format, so as long as attackers know how to
form their commands into that format (which is not a secret), they can
take advantage of this bug, and they currently are doing just that.
That sounds pretty bad. Is the situation hopeless?
Definitely not. The good news is that the Apache Foundation has updated Log4j
to address the vulnerability. All organizations urgently need to check
for the presence of this vulnerability in their environment and update
affected systems to the latest patched version.
The first update — version 2.15.0 — was released on December 6, 2021.
As exploitation ramped up in the wild, it became clear that the update
did not fully remediate the issue in all use cases, a vulnerability that
the National Vulnerability Database (NVD) codified as CVE-2021-45046.
As a result, on December 13, the Apache Foundation released version 2.16.0,
which completely removes support for message lookup patterns, thus
slamming the door on JNDI functionality completely and possibly adding
to development team backlogs to update material sections on their
codebase that handle logging.
That sounds straightforward, right?
Unfortunately, it’s likely going to be a pretty huge undertaking and
likely require different phases of discovery and remediation/mitigation.
Remediation
The first course of action is to identify all vulnerable
applications, which can be a mix of vendor-supplied solutions and
in-house developed applications. NCSC NL is maintaining a list
of impacted software, but organizations are encouraged to monitor
vendor advisories and releases directly for the most up-to-date
information.
For in-house developed applications, organizations — at a minimum —
need to update their Log4j libraries to the latest version (which, as of
2021-12-14, is 2.16) and apply the mitigations described in Rapid7’s initial blog post on CVE-2021-44228,
which includes adding a parameter to all Java startup scripts and
strongly encourages updating Java virtual machine installations to the
latest, safest versions. An additional resource, published by the Apache
Security Team, provides a curated list of all affected Apache projects, which can be helpful to expedite identification and discovery.
If teams are performing “remote” checks (i.e., exploiting the
vulnerability remotely as an attacker would) versus local filesystems
“authenticated” checks, the remote checks should be both by IP address
and by fully qualified domain name/virtual host name, as there may be
different routing rules in play that scanning by IP address alone will
not catch.
These mitigations must happen everywhere there is a vulnerable
instance of Log4j. Do not assume that the issue applies only to
internet-facing systems or live-running systems. There may be batch jobs
that run hourly, daily, weekly, monthly, etc., on stored data that may
contain exploit strings that could trigger execution.
Forensics
Attacks have been active since at least December 1, 2021, and there
is evidence this weakness has been known about since at least March of
2021. Therefore, it is quite prudent to adopt an “assume breach”
mindset. NCSC NL has a great resource page
on ways you can detect exploitation attempts from application log
files. Please be aware that this is not just a web-based attack.
Initial, quite public, exploit showcases included changing the name of
iOS devices and Tesla cars. Both those companies regularly pull metadata
from their respective devices, and it seems those strings were passed
to Log4j handlers somewhere in the processing chain. You should review
logs from all internet-facing systems, as well as anywhere Log4j
processing occurs.
Exploitation attempts will generally rely on pulling external
resources in (as is the case with any attack after gaining initial
access), so behavioral detections may have already caught or stopped
some attacks. The Log4j weakness allows for rather clever data
exfiltration paths, especially DNS. Attackers are pulling values from
environment variables and files with known filesystems paths and
creating dynamic domain names from them. That means organizations should
review DNS query logs going as far back as possible. Note: This could
take quite a bit of time and effort, but it must be done to ensure
you’re not already a victim.
Proactive response
Out of an abundance of caution, organizations should also consider
re-numbering critical IP segments (where Log4j lived), changing host
names on critical systems (where Log4j lived), and resetting credentials
— especially those associated with Amazon AWS and other cloud providers
— in the event they have already been exfiltrated.
Who should be paying attention to this?
Pretty much every organization, regardless of size, sector, or
geography. If you have any kind of web presence or internet
connectivity, you need to pay attention and check your status. If you
outsource all the technical aspects of your business, ask your vendors
what they are doing about this issue.
Who is exploiting it and how?
Kind of… everyone.
“Benign” researchers (some independent, some part of cybersecurity
firms) are using the exploit to gain an initial understanding of the
base internet-facing attack surface.
Cyberattackers are also very active
and are racing to take advantage of this vulnerability before
organizations can get their patches in place. Botnets, such as Mirai,
have been adapted to use the exploit code to both exfiltrate sensitive
data and gain initial access to systems (some deep within
organizations).
Ransomware groups have already sprung into action and weaponized the
Log4j weakness to compromise organizations. Rapid7’s Project Heisenberg
is collecting and publishing samples of all the unique attempts seen since December 11, 2021.
How are things likely to develop?
These initial campaigns are only the beginning. Sophisticated attacks
from more serious and capable groups will appear over the coming days,
weeks, and months, and they’ll likely focus on more enterprise-grade
software that is known to be vulnerable.
Most of the initial attack attempts are via website-focused injection
points (input fields, search boxes, headers, etc.). There will be more
advanced campaigns that hit API endpoints (even “hidden” APIs that are
part of company mobile apps) and try to find sneaky ways to get exploit
strings past gate guards (like the iOS device renaming example).
Even organizations that have remediated deployed applications might
miss some virtual machine or container images that get spun up regularly
or infrequently. The Log4Shell attacks are easily automatable, and
we’ll be seeing them as regularly as we see WannaCry and Conficker (yes,
we still see quite a few exploits on the regular for those
vulnerabilities).
Do we need to worry about similar vulnerabilities in other systems?
In the immediate term, security teams should narrow their attention to identify systems with the affected Log4j packages.
For the longer term, while it is impossible to forecast
identification of similar vulnerabilities, we do know the ease and
prevalence of CVE-2021-44228 demands the continued attention (been a long weekend for many) of security, infrastructure, and application teams.
Along with Log4Shell, we also have CVE-2021-4104
— reported on December 9, 2021 — a flaw in the Java logging library
Apache Log4j in version 1.x. JMSAppender that is vulnerable to
deserialization of untrusted data. This allows a remote attacker to
execute code on the server if the deployed application is configured to
use JMSAppender and to the attacker’s JMS Broker. Note this flaw only
affects applications that are specifically configured to use
JMSAppender, which is not the default, or when the attacker has
write-access to the Log4j configuration for adding JMSAppender to the
attacker’s JMS Broker.
Exploit vectors of Log4Shell and mitigations reported on Friday continue to evolve as reported
by our partners at Snyk.io and Java Champion, Brian Vermeer — see
“Editor’s note (Dec. 13, 2021)” — therefore, continued vigilance on this
near and present threat is time well spent. Postmortem exercises (and
there will be many) should absolutely include efforts to evolve
software, open-source, and package dependency inventories and, given
current impacts, better model threats from packages with similar
uninspected lookup behavior.
Does this issue indicate that we should stop compiling systems and go back to building from scratch?
There definitely has been chatter regarding the reliance upon so many
third-party dependencies in all areas of software development. We’ve
seen many attempts at poisoning numerous ecosystems this year,
everything from Python to JavaScript, and now we have a critical
vulnerability in a near-ubiquitous component.
On one hand, there is merit in relying solely on code you develop
in-house, built on the core features in your programming language of
choice. You can make an argument that this would make attackers’ lives
harder and reduce the bang they get for their buck.
However, it seems a bit daft to fully forego the volumes of
pre-built, feature-rich, and highly useful components. And let’s not
forget that not all software is created equally. The ideal is that
community built and shared software will benefit from many more hands to
the pump, more critical eyes, and a longer period to mature.
The lesson from the Log4Shell weakness seems pretty clear: Use, but
verify, and support! Libraries such as Log4j benefit from wide community
adoption, since they gain great features that can be harnessed quickly
and effectively. However, you cannot just assume that all is and will be
well in any given ecosystem, and you absolutely need to be part of the
community vetting change and feature requests. If more eyes were on the
initial request to add this fairly crazy feature (dynamic message
execution) and more security-savvy folks were in the approval stream,
you would very likely not be reading this post right now (and it’d be
one of the most boring Marvel “What If…?” episodes ever).
Organizations should pull up a seat to the table, offer code creation
and review support, and consider funding projects that become
foundational or ubiquitous components in your application development
environment.
How would a Software Bill of Materials (SBOM) have impacted this issue?
A Bill of Materials is an inventory, by definition, and conceptually
should contribute to speeding the discovery timeline during emergent
vulnerability response exercises, such as the Log4j incident we are
communally involved in now.
An SBOM is intended to be a formal record that contains the details
and supply chain relationships of various components used in software,
kind of like an ingredients list for technology. In this case, those
components would include java libraries used for logging (e.g. Log4j2).
SBOM requirements were included in the US Executive Order
issued in May 2021. While there may be international variations, the
concept and intended objects are uniform. For that reason, I will
reference US progress for simplicity.
From: The Minimum Elements For a Software Bill of Materials
(SBOM), issued Department of Commerce, in coordination with the National
Telecommunications and Information Administration (NTIA), Jul 12, 2021

The question many Log4Shell responders — including CISOs, developers,
engineers, sys admins, clients, and customers — are still grappling
with is simply where affected versions of Log4j are in use within their
technical ecosystems. Maintaining accurate inventories of assets has
become increasingly challenging as our technical environments have
become more complicated, interconnected, and wide-reaching. Our
ever-growing reliance on internet-connected technologies, and the rise
of shadow IT only make this problem worse.
Vulnerabilities in tools like Log4j, which is used broadly in a vast
array of technologies and systems, highlight the need for more
transparency and automation for asset and inventory management. Perhaps
the longer-term substantive impact from Log4Shell will be to refocus
investments and appreciation for the cruciality of an accurate inventory
of software and associated components through an SBOM that can easily
be queried and linked to dependencies.
The bottom line is that if we had lived in a timeline where SBOMs
were required and in place for all software projects, identifying
impacted products, devices, and ecosystems would have been much easier
than it has been for Log4Shell and remediation would likely be faster
and more effective.
How does Log4Shell impact my regulatory status — do I need to take special action to ensure compliance?
According to Rapid7’s resident policy and compliance expert, Harley
Geiger, “Regulators may not have seen Log4Shell coming, but now that the
vulnerability has been discovered, there will be an expectation that
regulated companies address it. As organizations’ security programs
address this widespread and serious vulnerability, those actions should
be aligned with compliance requirements and reporting. For many
regulated companies, the discovery of Log4Shell triggers a need to
re-assess the company’s risks, the effectiveness of their safeguards to
protect sensitive systems and information (including patching to the
latest version), and the readiness of their incident response processes
to address potential log4j exploitation. Many regulations also require
these obligations to be passed on to service providers. If a Log4j
exploitation results in a significant business disruption or breach of
personal information, regulations may require the company to issue an
incident report or breach notification.”
We also asked Harley whether we’re likely to see a public policy response. Here’s what he said…
“On a federal policy level, organizations should expect a renewed
push for adoption of SBOM to help identify the presence of Log4j (and
other vulnerabilities) in products and environments going forward. CISA
has also required federal agencies to expedite mitigation of Log4j and
has ramped up information sharing related to the vulnerability. This may
add wind to the sails of cyber incident reporting legislation that is
circulating in Congress at present.”
How do we know about all of this?
Well, a bunch of kids started wreaking havoc with Minecraft servers,
and things just went downhill from there. While there is some truth to
that, the reality is that an issue was filed on the Log4j project in
late November 2021. Proof-of-concept code was released in early
December, and active attacks started shortly thereafter. Some
cybersecurity vendors have evidence of preliminary attacks starting on
December 1, 2021.
Anyone monitoring the issue discussions (something both developers,
defenders, and attackers do on the regular) would have noticed the
gaping hole this “feature” created.
How long has it been around?
There is evidence of a request for this functionality to be added
dating back to 2013. A talk at Black Hat USA 2016 mentioned several JNDI
injection remote code execution vectors in general but did not point to
Log4j directly.
Some proof-of-concept code targeting the JNDI lookup weakness in Log4j 2.x was also discovered back in March of 2021.
Fear, Uncertainty, and Doubt (FUD) is in copious supply today and
likely to persist into the coming weeks and months. While adopting an
“assumed breach” mindset isn’t relevant for every celebrity
vulnerability, the prevalence and transitive dependency of the Log4j
library along with the sophisticated obfuscation exploit techniques we
are witnessing in real time point out that the question we should be
considering isn’t, “How long has it been around?” Rather, it is, “How
long should we be mentally preparing ourselves (and setting
expectations) to clean it up?”
We are adding more questions as they come our way. If you have questions you’d like answered, please let us know.
[UPDATE: December 17, 2021, 6 PM ET]
I’ve heard that the update doesn’t work and you can still be exploited even if you have updated? Should I panic?
If you’ve updated to either v2.15 (the original fix) or v2.16
(the latest fix), you don’t need to panic. It’s true that the v2.15
update “was incomplete in certain non-default configurations,” and that
was designated as a separate vulnerability: CVE-2021-45046. But the key
words here are** “in certain non-default configurations.”**
Essentially, when a logging configuration uses a non-default Pattern
Layout with a Context Lookup, attackers with control over Thread Context
Map (MDC) input data can craft malicious input data using a JNDI Lookup
pattern. This can result in DoS attacks, and we’ve now discovered it
can also result in information leaks, and in some specific cases, RCE.
This has resulted in the vulnerability being upgraded from a CVSS score
of 3.7 to 9.0 on the Apache Foundation website – but the RCE is currently only being reported for macOS,
and no in-the-wild-exploitation has been publicly reported, so it still
isn’t time to panic. The bottom line is that you do need to update to
v2.17 as soon as you can, but unless you have those non-default
configurations, v2.15 probably has you covered.
I’ve heard that there’s now evidence of ransomware attacks against this vulnerability. Should I panic?
It is true that reports have started to appear of the ransomware
group, Conti, leveraging CVE-2021-44228 (Log4Shell) to mount attacks.
According to a report
from AdvIntel, Conti is testing exploitation by targeting vulnerable
Log4j2 instances in VMware vCenter “for lateral movement directly from
the compromised network resulting in vCenter access affecting US and
European victim networks from the pre-existent Cobalt Strike sessions.”
While it’s not time to panic about this, we do expect to see much more
widespread ransom-based exploitation to follow in coming weeks, and this
is another reason to ensure you are on the latest version (v2.17) as soon as possible.
Is it OK to scan for vulnerable applications or systems?
If you own or lease systems and have appropriate authorization, it is
absolutely fine to scan to identify vulnerable applications or systems –
in fact, we strongly recommend you do so in your own environments so
you can get patching.
Beyond that, while laws vary by country, most anti-hacking laws
revolve around not exceeding authorized access or accessing systems
without authorization, so scanning someone else’s assets without
permission may fall foul of the law.
For example, as our resident US public policy expert, Harley Geiger, explains in this tweet,
under the US anti-hacking law, the Computer Fraud and Abuse Act (CFAA),
“If the test involves unauthorized exfiltration of nonpublic data from a
target system or causes a target system to download your executable
code w/o authorization, even if done in good faith, stop & make
friends w a lawyer.”
Get more critical insights about defending against Log4Shell
Check out our resource center