Automatic Construction of Graphical User Interfaces Semantic Models Using Robots for Mobile Application Testing

With the growing adoption of rich GUIs (Graphical User Interfaces) in mobile applications, researchers and practitioners have focused on GUI-based testing of the applications. Constructing a GUI model for AUT (Application Under Test) is a conventional strategy; nevertheless, automated modelling is typically grounded in source code logic, often lacking representation of functional semantics. However, manually constructing a GUI model enriched with semantics is inefficient. In this work, a GUI semantic model is proposed, namely FSM-ES (Finite State Machine with Extended Semantics), which not only reflects the syntactical structure of the GUI, but also the semantics of its elements. Moreover, visual technology is used to capture GUI information and use semantic ontology to guide robots in order to complete gesture actions such as clicking and sliding on the GUI, achieving the establishment of FSM-ES. The case study demonstrates that, while ensuring coverage of the core functions of the AUT, GUI semantic model proposed in this paper is 52% faster than manual modelling. Moreover, it facilitates the generation of test cases focused on functional semantics.