Synopsys researchers found this vulnerability using fuzzing, a negative testing technique in which deliberately malformed inputs are delivered to target software. When a failure occurs, a vulnerability has been located. In this case, the fuzzer delivered a DNS request over TLS with an invalid opcode, and named crashed with an assertion error.
While it is possible for developers to write negative unit tests, the process is slow and tedious. An automated fuzzer can create and deliver thousands or millions of badly formed inputs to thoroughly exercise the target software.
The fuzzer used by Synopsys researchers was Defensics®, which creates high-quality test cases by using generational (or model-based) fuzzing. It knows all the details of valid inputs, which means it can create fuzz test cases that are nearly correct, but anomalized in specific ways. The resulting test cases push the target software into specific states and exercise many code pathways.
As with any type of security testing, keeping up with the latest developments is important in fuzzing. DNS over TLS is new and not widely implemented or deployed. Updating our DNS test suite to support DNS over TLS shows how staying current is important, and how fuzzing can help improve the security of an entire ecosystem.
Software gets fuzzed one way or another. That is, when applications are released and deployed in the world, they will receive unexpected and malformed inputs, either by accident or as an attack. Proactive fuzzing during application development results in a hardened, more resilient application that is less risky to deploy and operate.