Implications of Applying Usability Testing with Remote Users

Abstract

Some studies in the literature on remote asynchronous usability testing have indicated the existence of contextual factors related to remote-uncontrolled environments. Typically, in these environments, users take part in the usability test at any time although uncontrolled contextual factors might be present. Moreover, such settings might induce different interactions with the evaluated products, which consequently may influence the data collected in the usability test. Therefore, this research aims to explore these kinds of interactions to determine whether they differ from users’ interactions in the laboratory and, if so, how. The findings of this research are intended to contribute new knowledge about the implications of applying asynchronous usability testing to remote users. To meet this goal, three main studies are conducted: the first exploratory study is aimed at exploring what happens during testing sessions in users’ natural environments. The second empirical study involves two participant samples: one sample performed the test in their natural environment, and the other sample performed the text in a lab. The performances of both groups are compared to explore their differences. User-reported data regarding contextual factors are also explored. In the third controlled experimental study, stimulating contextual factors are applied during usability testing sessions to explore the users’ interactions. The results showed that usability testing outcomes were independent of the method itself. With respect to physical environments, contextual factors were the most influential in the outcomes of usability testing. Although interruptions had the highest negative influence, the extent of this influence differed based on the type of interruption applied. In-person interruptions were the most disruptive because they influenced, not only the number of errors and task-load measurements, but also the time taken to perform tasks. Instant messaging increased the number of errors and the task load. Phone interruptions did not have noticeable effects on performance, but increased stress, time pressure and frustration. Based on our results, we concluded that if remote asynchronous usability testing is used, then the influence of contextual factors should be expected. Hence, these factors should be collected during testing because awareness of them is vital in improving data interpretation.