Estimating the accuracy of dynamic change-impact analysis using sensitivity analysis

The reliability and security of software are affected by its constant changes. For that reason, developers use change-impact analysis early to identify the potential consequences of changing a program location. Dynamic impact analysis, in particular, identifies potential impacts on concrete, typical executions. However, the accuracy (precision and recall) of dynamic impact analyses for predicting the actual impacts of changes has not been studied. In this paper, we present a novel approach based on sensitivity analysis and execution differencing to estimate, for the first time, the accuracy of dynamic impact analyses. Unlike approaches that only use software repositories, which might not be available or might contain insufficient changes, our approach makes changes to every part of the software to identify actually impacted code and compare it with the predictions of dynamic impact analysis. Using this approach in addition to changes made by other researchers on multiple Java subjects, we estimated the accuracy of the best method-level dynamic impact analysis in the literature. Our results suggest that dynamic impact analysis can be surprisingly inaccurate with an average precision of 47-52% and recall of 56-87%. This study offers insights to developers into the effectiveness of existing dynamic impact analyses and motivates the future development of more accurate analyses.