Detecting Log4j (Log4Shell): Mitigating the impact on your organization

The discovery of Log4j has DevOps teams working tirelessly to mitigate the issue. Here are six actions your organization should be taking now.

At midnight last Thursday, we experienced one of the most notable infosec events in years. A new zero-day exploit in a popular logging package for Java, Log4j, was discovered. The exact origin and timeline are still being investigated, but it’s important to note that this was not just a vulnerability announcement. The information disclosed was rapidly followed by fully functional exploit code—and the exploit itself turned out to be trivial to execute. 

Over 3 billion devices run Java, and because there are only a handful of logging libraries, many of them are likely to run Log4j. Worse still, many internet-exposed target applications can be exploited by external users without authentication. And unlike some other notable open source vulnerabilities, such as the infamous Heartbleed or the recently disclosed Trojan Source, in this case no prior coordination took place “behind the scenes” to ensure that users had adequate time to plan their response.

Learn more about the Log4j vulnerability

Putting aside the deconstruction of the bug itself, here’s why the seasoned CISO should have slept late and enjoyed little more than their regular Friday routine.

The holes in the Swiss cheese

As aviation safety enthusiasts say, an incident or accident occurs when the holes in the Swiss cheese line up. That is to say, we have multiple layers of protections and controls that should stop the worst-case scenarios from manifesting. Known catastrophic events occur only when all those controls fail to perform. Cybersecurity is no different, and we’ve been talking about defense in depth forever, too.

Here are six factors that, when combined, should help mitigate the impact and protect the Friday of every CISO.

  • Vulnerability notification and remediation of the Log4j component. This is the obvious muscle to flex, and it essentially breaks down to several actions.
    • Minimize the lead time from vulnerability discovery to your team becoming aware
    • Maximize the ability to respond and identify which applications utilize the component
    • Streamline the ability to recover—quickly patch the affected applications or put compensating controls in place
  • Susceptibility to exploitation: input and output validation. Successful exploitation of this vulnerability requires that you’ve ignored several secure coding principles. Data from untrusted sources (i.e., user-controlled input) should generally not be concatenated into log files without sanitization. This is especially true when the data is from an unauthenticated origin, whether human users or other applications. Data flowing into sensitive areas such as databases and logs that may be rendered to others should be subjected to output validation. Not doing so is a log injection vulnerability in itself, which has long been known and categorized as CWE 117, as well as being described in the CERT Java coding rules. So our first line of defense is making sure our applications are free from these kinds of flaws.
  • Network architecture and network-level access controls. Successfully achieving the next stage of exploitation requires that an attacker can transfer a payload or exfiltrate data to or from an external system. This might include transferring some code to execute a shell, or even some crypto mining malware. Initiating communication with outside hosts should always be locked down to the bare minimum. Guidelines from the Center for Internet Security (CIS) cover this, as do many other control frameworks including MITRE ATT&CK, which has a whole category of mitigations in M1037 covering network filtering. For client-side applications, using sandboxing technology—even if only the built-in OS and JVM functionality—could further mitigate the range of exploitation paths available.
  • Application architectural risk. In a worst-case scenario, such as unauthenticated external users accessing application elements that must be exposed for the application to function, there are obvious defensive programming techniques that apply. Why does your login page need to allow a $ and { character anyway? Although it might make sense for free-text search fields to allow a wide range of characters, a well-designed application should know what type and range of information to expect and lock down server-side input filtering to limit the accepted data to just those values. A password could contain special characters, but allowing sensitive data such as passwords to propagate into logs is a no-no in itself, and this eventuality should be eliminated elsewhere, too. Interactive analysis tools can help visualize the application dataflow—including across application and microservices boundaries—so you can find out what data needs to go where and make sure appropriate filtering exists.
  • Logging, monitoring, and anomaly detection. No, this isn’t irony. Anomaly detection routines should spot an application doing something unusual. That could be attempting to connect to hosts it shouldn’t, or launching processes or accessing files that it doesn’t usually interact with. There is a long list of other layers of defensive and detective controls that could be configured as part of a sandbox environment to limit the impact of any successful exploitation or make real-world exploitation practically impossible. Or, if exploitation does occur, being able to show what was accessed and support the breach investigation team in determining the true business impact will be invaluable when you get to that stage.
  • Software component transparency, vendor management, and mature open source adoption. This brings us to the topic of how we address this kind of risk for software that we did not develop ourselves, including software embedded in products we use every day. For some companies, the disclosure of a software Bill of Materials (SBOM) is already routinely requested and received. Such a listing of software components helps streamline a response and investigation, since it identifies exactly who is using Log4j (either directly or indirectly), and which products or vendors to engage with for remediation. But there’s a second aspect to this story: mature management of open source. As Tim Mackey, principal security strategist at Synopsys Cybersecurity Research Center (CyRC), likes to say, “There is no such vendor as open source.” In the case of Log4j, it appears that the library was still actively maintained and supported as part of a broader community initiative via the Apache Foundation. But many libraries are developed without the support of specific groups or even just individuals, and then they are abandoned or forgotten about. Looking at the provenance of open source components, including aspects such as community support and maintenance, provides an indicator of where this kind of latent risk may manifest. This data can help developers choose which components to trust. What would have happened if, instead of Log4j, the affected component had been abandoned several years ago and the last maintainer could no longer be contacted? 

What this looks like in practice

A vulnerability response is a combination of people, process, and technology. Software composition analysis tools help identify and track library usage. When a new vulnerability emerges, a Synopsys Black Duck® research team investigates the issue. This particular vulnerability was not assigned a detailed CVE entry until several hours after it was disclosed, but Synopsys analysts had already allocated a Black Duck Security Advisory (BDSA) number and pushed the notification out to Synopsys customers. 

The next question is your ability to respond. Once Black Duck sends out an alert, security analysts on your team can see which applications are impacted. The developers who own those applications get notifications automatically too, sometimes via Teams, Slack, or even in a Jira ticket or email, depending on how Black Duck alert is configured. You then need to have the organizational machinery in place to rapidly roll out an update. This is where we must look at the DevOps capability itself. 

One of the key metrics from DevOps Research and Assessment (DORA), a cross-industry program to measure and benchmark organizational DevOps capabilities, is the speed at which a change can be pushed into production. According to DORA, elite performers can complete this cycle in less than an hour and deploy any change on demand. But only 26% of the 1,200 respondents surveyed in the latest State of DevOps Survey fall into the elite category. While this is a large number of organizations, the next tier, high performers, takes between a day to a week to complete the cycle. These organizations are equipped to respond less quickly, meaning that slotting in a security patch is not something that would be considered a “business as usual” activity.

This significant gap between elite and high performers illustrates why maximizing the uptake of foundational DevOps practices benefits everyone and underlines why security teams must partner with development to ensure that the business is adequately equipped to respond, whether tackling a fast fix for a security, quality, or any other kind of issue.

SODR_2021_1.max-2800x2800.pngFigure 1: Software delivery performance metrics chart from State of DevOps Survey

Finally, beyond scrambling to fix the underlying component, there are three reasons why you shouldn’t be affected. Good code hygiene, good network architecture, and confidence that centralized security and engineering teams have integrated appropriate security checks into CI/CD pipelines and provided actionable information to developers should protect you, and help you avoid security, quality, and safety risks for every software delivery. Static code checkers, including Coverity®, can find log injection vulnerabilities, and in the case of modern infrastructure-as-code configurations, overly permissive network rules can be caught by automated analysis. 

In a well-orchestrated AppSec program, determining whether your team or built-in automation ran the tools, and if they fixed the findings, should be a couple of clicks on the application security orchestration and correlation (ASOC) dashboard.

Furthermore, as we see increased adoption of SBOMs, championed recently by Executive Order 14028, software component information should flow more seamlessly between product vendors and their customers. Instead of calling every single supplier to confirm component information, the task now becomes a straightforward database search. Although users of software composition analysis products such as Black Duck already benefit from internal SBOM information for software that they develop in-house, we are on the cusp of broad SBOM adoption and distribution between organizations. Once established, this will be another powerful tool for mitigating and managing risk, and support more rapid response (perhaps even automated response) in these scenarios.

Putting it on autopilot

Of course, talking about all these things that should have already happened is a bit like the proverbial stable door and horse. It is important to highlight that security is not the job of any single person or department. For our mission to succeed, each one of these activities must be federated out into everybody’s job. And for organizations to achieve success, a range of different security activities must be woven into the culture of how IT systems are planned, constructed, and operated. Practices such as formulating nonfunctional security requirements and reviewing architecture at the design phase, for example, will help identify what data the application should process, and what requirements for security checks must be satisfied before software is released into production. 

At modern organizations, the highly automated software delivery process or pipeline is programmed to fail if these prerequisite steps have not been completed, making it impossible to launch a product with such vulnerabilities or missing mitigations in the first place. In the future, SBOM information might become a means by which zero-trust architectures can dynamically determine the level of trust and privilege software may execute—or when it should be stopped from executing.

The good news is that it is possible now to measure and benchmark organizational ability across all these critical areas, and illuminate any gaps and areas for improvement using frameworks such as Building Software in Maturity Model (BSIMM).

With this reassurance, Synopsys customers will have wrapped up their Friday on schedule and be ready to enjoy the weekend.

Find and fix your highest-priority vulnerabilities quickly

Learn how Black Duck SCA can help

 
Michael White

Posted by

Michael White

Michael White

Michael White is the director of solution strategy within Synopsys' Software Integrity Group. During his time at Synopsys, he has worked with leading software and IT organizations around the world to build programs to manage software security, safety, and reliability risks.


More from Open source and software supply chain risks