TY - GEN
T1 - A Preliminary Study on Open-Source Memory Vulnerability Detectors
AU - Nong, Yu
AU - Cai, Haipeng
N1 - Publisher Copyright:
© 2020 IEEE.
PY - 2020/2
Y1 - 2020/2
N2 - We present preliminary results of a study on memory vulnerability detectors based on (static and/or dynamic) program analysis. Against a public suite of 520 C/C++ programs as benchmarks which cover 14 different vulnerability categories, we measured the performance of five state-of-the-art detectors in terms of effectiveness and efficiency. Our study revealed that with respect to the particular set of benchmarks we chose: (1) the effectiveness of these studied detectors varied widely: 66.7% to 100% precision, 0% to 100% recall, and 0% to 100% F1 per category, indicating most of the techniques worked extremely well on certain kinds of vulnerabilities yet quite poorly on others, (2) these detectors were generally quite efficient: despite a few outliers, the average (per benchmark) time costs were around one second, (3) except for between the most and least accurate detectors, other pairs of detectors did not have statistically significant and large differences in accuracy in our pair-wise statistical testing. We also share insights into the failures and successes of these detectors obtained from our case studies.
AB - We present preliminary results of a study on memory vulnerability detectors based on (static and/or dynamic) program analysis. Against a public suite of 520 C/C++ programs as benchmarks which cover 14 different vulnerability categories, we measured the performance of five state-of-the-art detectors in terms of effectiveness and efficiency. Our study revealed that with respect to the particular set of benchmarks we chose: (1) the effectiveness of these studied detectors varied widely: 66.7% to 100% precision, 0% to 100% recall, and 0% to 100% F1 per category, indicating most of the techniques worked extremely well on certain kinds of vulnerabilities yet quite poorly on others, (2) these detectors were generally quite efficient: despite a few outliers, the average (per benchmark) time costs were around one second, (3) except for between the most and least accurate detectors, other pairs of detectors did not have statistically significant and large differences in accuracy in our pair-wise statistical testing. We also share insights into the failures and successes of these detectors obtained from our case studies.
KW - accuracy
KW - code analysis
KW - comparison
KW - detection
KW - efficiency
KW - precision
KW - recall
KW - Software vulnerability
UR - https://www.scopus.com/pages/publications/85083565068
U2 - 10.1109/SANER48275.2020.9054851
DO - 10.1109/SANER48275.2020.9054851
M3 - Conference contribution
AN - SCOPUS:85083565068
T3 - SANER 2020 - Proceedings of the 2020 IEEE 27th International Conference on Software Analysis, Evolution, and Reengineering
SP - 557
EP - 561
BT - SANER 2020 - Proceedings of the 2020 IEEE 27th International Conference on Software Analysis, Evolution, and Reengineering
A2 - Kontogiannis, Kostas
A2 - Khomh, Foutse
A2 - Chatzigeorgiou, Alexander
A2 - Fokaefs, Marios-Eleftherios
A2 - Zhou, Minghui
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 27th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2020
Y2 - 18 February 2020 through 21 February 2020
ER -