Automated refinement of models for model-based testing using exploratory testing

Abstract

Model-based testing relies on models of the system under test to automatically generate test cases. Consequently, the effectiveness of the generated test cases depends on models. In general, these models are created manually, and as such, they are subject to errors like omission of certain system usage behavior. Such omitted behaviors are also omitted by the generated test cases. In practice, these faults are usually detected with exploratory testing. However, exploratory testing mainly relies on the knowledge and manual activities of experienced test engineers. In this paper, we introduce an approach and a toolset, ARME, for automatically refining system models based on recorded testing activities of these engineers. ARME compares the recorded execution traces with respect to the possible execution paths in test models. Then, these models are automatically refined to incorporate any omitted system behavior and update model parameters to focus on the mostly executed scenarios. The refined models can be used for generating more effective test cases. We applied our approach in the context of 3 industrial case studies to improve the models for model-based testing of a digital TV system. In all of these case studies, several critical faults were detected after generating test cases based on the refined models. These faults were not detected by the initial set of test cases. They were also missed during the exploratory testing activities.