Effective fault detection for static analyzers via automated testing

Abstract

Static analyzers comprehend and analyze input programs without dynamically executing them to gather insights into and detect flaws in their properties and behaviors. These tools are indispensable for ensuring software quality and supporting various software engineering tasks, including vulnerability detection, privacy leakage identification, and malware analysis. Despite their widespread adoption in real-world software development and maintenance, static analyzers, like other computer programs, are susceptible to implementation faults, and it is a common practice for static analyzers to detect such faults via testing. Manually creating test cases for static analyzers, however, is highly time-consuming and labor-intensive because both constructing input programs to trigger specific analyses and deriving the correct analysis results for the input programs are non-trivial tasks. Meanwhile, existing research efforts to automatically generate test cases and uncover faults in static analyzers suffer from three important limitations that restrict their applicability. These efforts depend on dedicated oracles designed for specific programming languages or particular sets of static analyzers, have limited support for certain program elements, or overlook bugs reflected in only the intermediate representations constructed by the static analyzers but not the warnings they report. To address these limitations, we develop three novel techniques, namely STATFIER, AN­NATESTER, and SASCOPE. The STATFIER technique leverages semantics-preserving program transformations to derive valid variants from existing test input programs for static analyzers, and it discovers faults in the static analyzers via metamorphic testing. We systematically investigate the impact of program annotations on static analyzers and propose another metamorphic testing technique, AN­NATESTER, to automatically identify annotation-induced faults. Furthermore, we comprehensively study the root causes of program representation faults and their fix strategies and develop the SASCOPE technique to detect relevant faults via automated testing. We have implemented the techniques into three testing frameworks with the same names. Using the testing frameworks, we identify 141 faults in popular static analyzers. We have reported all identified faults to the respective developers via issue tracking systems, with 72 of them confirmed or fixed.