Description
This post is co-authored by Blake Cifelli, Senior Advisory Services Consultant.
In today’s cybersecurity world, risks evolve faster than we can remediate them. To meet our goals and become resilient to these fast changes, we need the right balance of automation and human interaction. Enabling rapid response for protecting information systems is paramount, but how does a business reach this level of reaction?
How can organizations maintain a standard of excellence to their responses in high-risk situations?
Where do you even begin to respond to a critical vulnerability like the one in Apache’s Log4j Java library (a.k.a. Log4Shell)?
Most importantly, how do we transform the tactical actions that need to take place into an effective strategy to scale?
1. Empower personnel
The personnel with the knowledge about your various solutions must be empowered to make the decisions necessary to address your company’s information technology needs. If those team members don’t feel they can make those decisions, then they will defer to management — but managers may not know the intricacies of the solutions and could create a natural bottleneck, since there are going to be more decision points than managers to make decisions. Providing personnel with policy documents with uniform criteria for evaluating the risk these new vulnerabilities present, the ways to respond, and the time expectations is paramount for a timely resolution.
In a typical risk resolution process, there are many gates to safeguard our systems. This helps ensure that whatever change happens increases the solution’s confidentiality, integrity, or availability rather than diminishing it. However, a situation like Log4Shell needs to be treated like an incident response activity to quickly address the risk. Create a task force to effectively answer the important questions like:
- How do we find vulnerable systems?
- Which systems are vulnerable?
- What options are there for a fix? One size may not fit all.
- Who is going to track changes?
- Who is going to validate the fix is in place?
Utilizing a strong incident response procedure to answer all these questions will assist with prioritization and remediation to an acceptable level of risk.
2. Promote visibility
Any standard vulnerability management lifecycle process begins with identifying affected systems to assess and evaluate the scope of a vulnerability’s presence on the network. The approach should utilize both proactive and reactive efforts through a combination of tools and well-documented processes to streamline and scale the response effectively.
A proactive process would first involve having well-documented use of any such library versions internally in an inventory, so that discoverability and traceability are much more narrowly focused efforts. If you conduct authenticated vulnerability scans continuously on pre-scheduled frequencies, this will also help with identification of third-party software utilizing this library over time. Classifying system criticality within the vulnerability management tool will help you more effectively scale future remediation processes.
These proactive processes help jumpstart an initial response, but you’ll still need reactive efforts to help ensure effective and timely remediation. Vulnerability scanning tools will receive signature updates regarding this newly discovered vulnerability, which will require updating your vulnerability management tool and initiating one-off alternative scans that may deviate from pre-scheduled rotations. These alternative scans should include tiered phases, so the most critical systems receive scan priority, and then remaining systems are scanned in order of criticality. Leveraging the pre-existing system criticality classification will significantly expedite this process.
A security incident and event management (SIEM) tool can also assist with identifying, tracking, and alerting for any suspicious activity that may be tied to exploitation of this vulnerability. Host agents and network detection systems that report back to the SIEM should be closely monitored, and any activity or traffic that deviates from baselines should receive an active response. You may need to adjust logging and alerting rules and thresholds to ensure your efforts are strategically focused.
Tactical processes help you achieve this continuous identification, but you still need to orchestrate and execute them through strategic planning to remain timely, efficient, and effective. Well-documented asset inventories and appropriate system criticality classifications help you prioritize your efforts, while continuous vulnerability scans and leveraging vulnerability management and SIEM tools help to identify, track, and manage vulnerability exposure. Leadership should provide the direction to guide these activities from inception to implementation through effective communication and allocation of resources. Lay out a short-term roadmap for tracking objectives and quick wins as part of the remediation process, so you can quickly and concisely show how you’re tracking toward goals.
3. Implement prioritization and mitigation
Now that your team has successfully identified all affected systems, you’ll need to roll out patches to those systems on a continuous basis during the next phase to mitigate risk. Current enterprise-wide patching timelines may require adjustment due to the urgency associated with such critical vulnerabilities. Patch testing and rollout phases must be expedited to support a more timely and effective response.
Much like conducting our vulnerability scans in terms of system criticality prioritization, our patch management response should follow a similar approach, with the caveat that a pilot group or pilot system deemed non-critical should be patched first for testing purposes to ensure no adverse effects prior to rolling out patches in order of system criticality. If you’ve configured a full test environment is configured, you can test patches on critical systems first within that environment and then roll them out in production according to criticality. The testing timeline itself should be reduced throughout all standard phases of a testing cycle — you may even need to eliminate certain testing phases altogether. The rollout timelines for patches across all systems will need to be expedited as well to ensure as timely coverage as possible. If your environment has widespread use of the vulnerable library, you may require reductions in timelines of anywhere from 25% to 50%.
Emergency patching procedures should provide for timely testing and production rollouts within roughly half the time of a normal patching cycle, or 5 to 10 days at a maximum for critical systems to minimize breach potential as quickly as possible. Also keep in mind that some vulnerabilities may involve more than just application of a simple patch — configuration changes may also be necessary to further mitigate potential exploitation by an adversary.
4. Validate remediation
Now, you’ve deployed patches to all affected systems, so the mitigation efforts are complete, right? While you may want to shift your focus back to other tasks, it’s essential to maintain continuous identification processes to ensure that no stone remains unturned.
The vulnerability management validation phase leverages those reactive identification processes, in addition to patch management processes, to assist in efficient and effective vulnerability remediation for affected systems. This stage involves re-scanning initially identified vulnerable systems to assess successful patch application and performing additional open scans of the network to ensure that there are no lingering systems that may still be affected by the vulnerability but weren’t originally identified — or perhaps weren’t successfully patched as part of the patch management process. This cycle of continuous validation will remain in effect until “clean” scans are reported across the enterprise regarding this vulnerability.
Since the Log4j logging library is widely used throughout many enterprise applications and even unknowingly embedded in so many others, continuous validation will become crucial in ensuring your organization remains vigilant and can mitigate the vulnerability quickly and effectively as you continue to discover affected systems.
5. Regularly review risks
A vulnerability management lifecycle rarely ever comes to a true end. As adversaries and security evangelists further evaluate a specific vulnerability over time, new methods of exploitation are identified, affected versions increase in scope and scale, and recent patches and fixes are found to be ineffective. This leaves organizations potentially open to exposure and at a loss for the best path forward. Continuous review of the trends surrounding an ongoing critical vulnerability will help organizations ensure they remain both aware of the impact and the current mitigating measures that have been most successful. Additionally, leveraging other solutions can help further identify and launch a coordinated defense-in-depth response to any potential malicious activity that may be associated with such vulnerabilities.
Working to continuously identify, mitigate, validate, and review vulnerabilities throughout their inevitable course will require commitment and fortitude to achieve the best results, but once the tides have subsided with Log4Shell and you’ve successfully and securely endured one of the worst security vulnerability exposures in a decade by following these processes, you can rest assured that your incident response processes were well-tested during this endeavor — and your IT security budget should be more than solidified for the next few years to come.
Check out our additional resources for further insight of this vulnerability, mitigating measures, and tools available to assist.
- Log4Shell Resource Hub
- Log4Shell Resources for Rapid7 Customers
- Widespread Exploitation of Critical Remote Code Execution in Apache Log4j
- Update on Log4Shell’s Impact on Rapid7 Solutions and Systems
- Using InsightVM to Find Apache Log4j CVE-2021-44228
- InsightVM Docs page for Log4j
- Rapid7 analysis of Log4Shell on AttackerKB
Attacker Behavior Analytics Detection Rules viewed and sorted by rule priority
MITRE ATT&CK Matrix within Detection Rules
New investigations interface
MITRE ATT&CK tab within Investigations Evidence panel
Dashboard Library in InsightIDR
New “r7_context" Rapid7 Resource Name (RRN) data in Log Search
