Prioritization of test cases with varying test costs and fault severities for certification testing

Abstract

We present an industrial case study on the application of test case prioritization techniques in the context of certification testing in consumer electronics domain. Test execution times and fault severities are subject to high variations in this domain. As a result, most of the existing techniques and metrics turn out to be inappropriate for this application context. We discuss such deficiencies and the room for improvement based on our case study with the certification test suites of 3 Smart TV applications as real experimental objects. We also propose a new metric, LAPFD, which is based on the calculation of the average of the percentage of faults detected. This calculation is weighted according to the cost of test cases and calculated separately per severity class. Then, a lexicographic ordering is performed based on these classes. We compared the baseline (random) ordering of test cases with respect to an alternative ordering based on cost, measured as the test execution time. These alternative orderings are evaluated by using the LAPFD metric. We observed that cost-based ordering of test cases consistently outperformed random ordering. Another observation is that there is a large room for improvement regarding the effectiveness of test case prioritization in this application domain.