CVE-2021-44228 rules everything around us — or so it seemed, at least, for those breathless days in December 2021 when the full scope of Log4Shell
was starting to take hold and security teams were strapped for time and
resources as they scoured their organizations’ environments for
vulnerable instances of Apache Log4j. But now that the peak intensity
around this vulnerability has waned and we’ve had a chance to catch our
collective breath, where does the effort to patch and remediate stand?
What should security teams be focusing on today in the fight against
Log4Shell?
On Wednesday, February 16, Rapid7 experts Bob Rudis, Devin Krugly,
and Glenn Thorpe sat down for a webinar on the current state of the
Log4j vulnerability. They covered where Log4Shell stands now, what the
future might hold, and what organizations should be doing proactively to
ensure they’re as protected as possible against exploits.
Laying out the landscape
Glenn Thorpe, Rapid7’s Program Manager for Emergent Threat Response,
kicked things off with a recap and retrospective of Log4Shell and why it
seemingly set fire to the entire internet for a good portion of
December. The seriousness of this vulnerability is due to the
coming-together of several key factors, including:
- The ability for vulnerable systems to grant an attacker full administrative access
- The low level of skill required for exploitation — in many cases, attackers simply have to copy and paste
- The attack vector’s capability to run undetected over an encrypted channel
- The pervasiveness of the Log4j library, which means vulnerability
scanners alone can’t act as complete solutions against this threat
Put all this together, and it’s no surprise that the volume of
exploit attempts leveraging the Log4j vulnerability ramped up throughout
December 2021 and has continued to spike periodically throughout
January and February 2022. By January 10, ransomware using Log4Shell had
been observed, and on January 14, Rapid7’s MDR saw mass Log4j exploits in VMware products.
But while there’s certainly been plenty of Log4j patching done, the
picture on that front is far from complete. According to the latest CISA data (also here
as a daily-updated spreadsheet), there are still 320 cataloged software
products that are known to be affected by vulnerable Log4j as of
February 16, 2022 — and 1,406 still awaiting confirmation from the
vendor.
Log4j today: A new normal?
So, where does the effort to put out Log4j fires stand now? Devin
Krugly, Rapid7’s Practice Advisor for Vulnerability Risk Management,
thinks we’re in a better spot than we were in December — but we’re by no
means out of the woods.
“We’re effectively out of fire-fighting mode,” said Devin. That means
that, at this point, most security teams have identified the affected
systems, implemented mitigations, and patched vulnerable versions of
Log4j. But because of the complexity of today’s software supply chains,
there are often heavily nested dependencies within vendor systems — some
of which Log4j may still be implicated in. This means it’s essential to
have a solid inventory of vendor software products that may be using
Log4j and to ensure those instances of the library are updated and
patched.
“Don’t lose that momentum,” Glenn chimed in. “Don’t put that on your post-mortem action list and forget about it.”
This imperative is all the more critical because of a recent uptick
in Log4Shell activity. Rapid7’s Chief Data Scientist Bob Rudis laid out
some activity detected by the Project Heisenberg honeypot fleet
indicating a revival of Log4j activity in early and mid-February, much
of it from new infrastructure and scanning hosts that hadn’t been seen
before.
Amid this increase in activity, vulnerable instances of Log4j are anything but gone from the internet. In fact, data from Sonatype as of February 16, 2022 indicates 39% of Log4j downloads are _still _versions vulnerable to Log4Shell.
“We’re going to be seeing Log4j attempts on the internet, on the
regular, at a low level, forever," Bob said. Log4Shell is now in a
family with WannaCry and Conficker (yes, that Conficker) —
vulnerabilities that are around indefinitely, and which we’ll need to
continually monitor for as attackers use them to try to breach our
defenses.
Navigating life with Log4Shell
Adopting a defense-in-depth posture in the “new normal” of life with
Log4Shell is sure to come with its share of headaches. Luckily, Bob,
Devin, and Glenn shared some practical strategies that security teams
can adopt to keep their organizations’ defenses strong and avoid some
common pitfalls.
Go beyond compensating controls
“My vendor says they’ve removed the JNDI class from the JAR file —
does that mean their application is no longer vulnerable to Log4Shell?”
This question came up in a few different forms from our webinar
audience. The answer from our panelists was nuanced but crystal-clear:
maybe for now, but not forever.
Removing the JNDI class is a compensating control — one that provides
a quick fix for the vulnerability but doesn’t patch the core,
underlying problem via a full update. For example, when you do a backup,
you might unknowingly reintroduce the JNDI class after removing it —
or, as Devin pointed out, an attacker could chain together a replacement
for it.
These kinds of compensating or mitigating controls have their place
in a short-term response, but there’s simply no action that can replace
the work of upgrading all instances of Log4j to the most up-to-date
versions that contain patches for Log4Shell.
“Mitigate for speed, but not in perpetuity,” Glenn recommended.
Find the nooks and crannies
Today’s cloud-centric IT environments are increasingly ephemeral and
on-demand — a boost for innovation and speed, but that also means teams
can deploy workloads without security teams ever knowing about it.
Adopting an “Always Be Scanning” mindset, as Bob put it, is essential to
ensure vulnerable instances of Log4j aren’t introduced into your
environment.
Continually scanning your internet-facing components is a good and
necessary start — but the work doesn’t end there. As Devin pointed out,
finding the nooks and crannies where Log4j might crop up is critical.
This includes scouring containers and virtual machines, as well as
analyzing application and server logs for malicious JNDI strings. You
should also ensure your security operations center (SOC)
team can quickly and easily identify indicators that your environment
is being scanned for reconnaissance into Log4Shell exploit
opportunities.
“Involving the SOC team for alerting purposes, if you haven’t already
done that, is an absolutely necessity in this case," said Devin.
Get better at vendor management
It should be clear by now that in a post-Log4j world, organizations
must demand the highest possible level of visibility into their software
supply chain — and that means being clear, even tough, with vendors.
“Managing stuff on the internet is hard because organizations are
chaotic beings by nature, and you’re trying to control the chaos as a
security professional," said Bob. Setting yourself up success in this
context means having the highest level of vulnerability possible. After
all, how many other vulnerabilities just as bad as Log4Shell — or even
worse — might be out there lurking in the corners of your vendors’ code?
The upcoming US government requirements around Software Bill of Materials (SBOM)
for vendor procurement should go a long way toward raising expectations
for software vendors. Start asking vendors if they can produce an SBOM
that details remediation and update of any vulnerable instances of
Log4j.
These conversations don’t need to be adversarial — in fact, vendors
can be a key resource in the effort to defend against Log4Shell.
Especially for smaller organizations or under-resourced security teams,
relying on capable third parties can be a smart way to bolster your
defenses.
Only you can secure the software supply chain
OK, maybe that subhead is not literally true — a secure software
supply chain is a community-wide effort, to which we must all hold each
other accountable. The cloud-based digital ecosystem we all inhabit,
whether we like it or not, is fundamentally interconnected. A pervasive
vulnerability like Log4Shell is an unmistakable reminder of that fact.
It also serves as an opportunity to raise our expectations of
ourselves, our organizations, and our partners — and those choices do
start at home, with each security team as they update their
applications, continually scan their environments, and demand visibility
from their vendors. Those actions really do help create a more secure
internet for everyone.
So while we’ll be living with Log4Shell probably forever, it’ll be
living with us, too. And as scared as you are of the spider, it’s even
more scared of your boot.
Want to go more in-depth? Check out the full replay of our webinar, "Log4Shell Two Months Later: Lessons and Insights for Protectors."
Quick resources:
Bob, Devin, and Glenn mentioned a wealth of handy links in their discussion. Here are those resources for quick, easy reference.
Additional reading:
NEVER MISS A BLOG
![[Security Nation] Matthew Kienow on Open-Source Security and the Recog Framework](https://blog.rapid7.com/content/images/2022/03/security_nation_logo.jpg){kind=logo}
![[Security Nation] Matthew Kienow on Open-Source Security and the Recog Framework](https://blog.rapid7.com/content/images/2022/03/image1.png)
