Posted by Ari Kauppi on September 11, 2017
At Synopsys, our R&D teams routinely organize internal hackathons to verify the Synopsys Software Integrity Portfolio’s performance in real-world environments. During one hackthon, focused on open source software, Tuomas Haanpää, from the Synopsys Fuzz Testing (Defensics) R&D group, ran our NFSv3 test suite against the Linux kernel and found several interesting errors.
Initial analysis found that anomalized NFSv3 READDIR/READDIRPLUS requests produced an error, and further repeated requests eventually froze the system under test (SUT).The high-level root cause for this issue is a typical case that’s often seen in larger systems: A front-end subsystem—in this case SunRPC—makes several assumptions about the back-end subsystems’s—in this case NFSDv3’s—behavior. However, these assumptions aren’t properly validated by either one, so when unexpected inputs in the NFS payload breaks those assumptions, the NULL pointer dereference leads to a failure. For a successful denial-of-service (DoS) attack, the intruder requires read-only access to a NFS mount.
Mitre has allocated CVE-2017-7645 for this remote DoS vulnerability. Our research indicates that, at the very least, every kernel since v2.6.32 is vulnerable without the patch.
An extended study of the root cause and reproduction efforts of CVE-2017-7645 revealed the potential for more issues lurking in the NFSDv3 implementation. Portions of the Linux kernel code base are quite old and have numerous pointer arithmetics and assumptions built in, indicating that we needed to proceed by fuzzing with sanitizers. The need to fuzz with sanitizers couldn’t be truer for complex systems because bugs can always be found even after conducting rigorous code reviews and using static analysis tools.
The Linux kernel supports Kernel Address Sanitizer (KASAN), a very powerful compile-time instrumentation method, in v4.0 and later. KASAN instruments every memory allocation, free and read, and write operation with sanity checks. Among other issues, KASAN can uncover most use-after-free, buffer overflow, and buffer overrun errors. The Synopsys Fuzz Testing R&D team selected KASAN to run in conjunction with Synopsys Fuzz Testing (Defensics).
When the Defensics NFS3 Server test suite ran with KASAN enabled in the SUT, a previously passing test case started to fail. KASAN noticed a read of freed memory region with anomalous NFSDv3 WRITE requests. It showed no payload to be written, but was actually writing 1MB of data to the target file.
Further investigation revealed the following about the written data:
When the issue is exploited, however, it does not leave any traces in any logs and the attack could go unnoticed. The malicious activity would look like normal NFS traffic, but it was, in fact, leaking information that should have been kept private.
This discovery reminded us that memory access issues that do not cause immediate crashes and go unnoticed can pose serious security threats to organizations. One example is Heartbleed, a vulnerability that unexpectedly leaks information although systems are working as expected.
Mitre allocated CVE-2017-7895 to the issue that I discovered. The issue seems to have been introduced in kernel v2.6.22—which is about 10 years ago.
The NFSDv4 implementation in Linux was developed much later than the NFSDv3 implementation. Although the NFSDv4 subsystem was designed with a better structure for input validation and error checking, Jani Tuovila from the Synopsys Fuzz Testing R&D team decided to verify with the Defensics NFSv4 Server test suite.
After performing the test, a failure was found. A remote DoS was caused when the SUT was handling an invalid error in the LAYOUTGET command, and it was later revealed that the remote DoS is even more severe if the source address is spoofable. The maliciously crafted request triggers an out-of-bounds read and pointer dereference. Because the vulnerability is in the error handling of the path, no actual access rights are required. The only requirement is that the spoofable source address be within an allowed IP range for NFS access.
Mitre allocated CVE-2017-8797 for this issue. At least every kernel since v4.4.0 is vulnerable.
Tuomas Haanpää, Jani Tuovila, and myself, from Synopsys’ Fuzz Testing R&D team, weren’t the only members involved in the Linux kernel NFS fuzzing project. Matti Kamunen and Marko Laakso also reached out to the Linux Foundation to ensure a safe, secure, and coordinated disclosure. A special thank you to Linux NFS maintainer J. Bruce Fields, who helped patch the discovered issues.
To prevent severe issues from going unnoticed, use state-of-the-art fuzzing solutions combined with in-depth instrumentation.