Impact of education and experience level on the effectiveness of exploratory testing: an industrial case study

Abstract

Exploratory Testing (ET) is a widely applied approach in practice. This approach relies on the knowledge and experience of test engineers and technicians who perform ET. Hence, we aimed at evaluating the impact of education and experience level on the effectiveness of ET. We conducted an industrial case study for this purpose. 19 practitioners, who have different educational backgrounds and experience levels, were involved in applying ET for testing a Digital TV system. We measured the number of detected failures and categorized these failures based on their severity. We evaluated effectiveness from two aspects: criticality of the detected failures and efficiency in terms of number of failures detected per unit of time. The results show that the efficiency of ET is significantly affected by both the educational background and experience. When it comes to the number of critical failures detected, we cannot observe any impact of education on results. However, experience level has a significant impact for this aspect as well.