Automated Field Usability Evaluation Using Generated Task Trees

Usability is an important aspect of any kind of product. This also applies for software like desktop applications and websites, as well as apps on mobile devices and smart TVs. In a competitive market, the usability of a software becomes a discriminator between success and failure. This is especially important for software, as alternatives are often close at hand and only one click away. Hence, the software development must strive for highly usable products. Usability engineering allows for continuously measuring and improving the usability of a software during its development and beyond. For this, it offers a broad variety of methods, that support detecting usability issues in early development stages on a prototype level, as well as during the operation of a final software. Unfortunately, most of these methods are applied manually, which increases the effort of their utilization. In this thesis, we describe a fully automated approach for usability evaluation. This approach is a user-oriented method to be applied in the field, i.e., during the operation of a software. For this, it first traces the usage of a software by recording user actions on key stroke level. From these recordings, it compiles a model of the Graphical User Interface (GUI) of a software, as well as a usage model in the form of task trees. Based on these models and the recorded actions, our approach performs a detection of 14 different so called usability smells. These smells are exceptional user behavior and indicate usability issues. The result of the application of our approach on a software is a list of findings for each of the smells. These findings provide detailed information about the user tasks that are affected by the related usability issues, as well as about the elements of the GUI that cause the issues. By applying it on two websites and one desktop application, we perform an in-depth validation of our approach in three case studies. In these case studies, we verify if task trees can be generated from recorded user actions and if they are representative for the user behavior. Furthermore, we apply the usability smell detection and analyze the corresponding results with respect to their validity. For this, we also compare the findings with the results of generally accepted usability evaluation methods. Finally, we conclude on the results and derive conditions for findings of our approach, which must be met to consider them as indicators for usability issues. The results of the case studies are promising. They show, that our approach can find, fully automated, a broad range of usability issues. In addition, we show, that the findings can reference in detail the elements of the GUI that cause a usability issue. Our approach is supplemental to established usability engineering methods and can be applied with minimal effort on a large scale.