Black-box test case selection by relating code changes with previously fixed defects

Abstract

Software continuously changes to address new requirements and to fix defects. Regression testing is performed to ensure that the applied changes do not adversely affect existing functionality. The increasing number of test cases makes it infeasible to execute the whole regression test suite. Test case selection is adopted to select a subset of the test suite, which is associated with the changed parts of the software. These parts are assumed to be error-prone. We present and evaluate a test case selection approach in the context of black-box regression testing of embedded systems. In this context, it is challenging to relate test cases with a set of distinct source code elements to be able to select those test cases associated with the modified parts of the source code. We analyze previously fixed defects for this purpose. We relate test cases that detect these defects with the source files that are previously modified for fixing them. Then, we select test cases related with source code files that are modified in the subsequent revision. The strength of this relation is determined as the number of changes associated with fixed defects previously detected by the same test cases. We conduct a case study on 3 real projects from the consumer electronics domain. Results show that it is possible to detect from 65% up to 85% of the defects detected by the whole test suite by selecting from 30% up to 70% of the test cases.