User-Elicited and Validated Hovering Gestures for a Self-Powered Photovoltaic Sensor for Tabletop Interaction with Smart Home Devices

Abstract

Sustainable, energy-efficient, convenient and personalised living is driving the modern smart home revolution. However, despite the emergence of a myriad of innovative smart home technologies, the acceptance and adoption of these technologies remains low, due to several factors, including security and privacy issues, energy consumption concerns, and their higher complexity, which results in poor user experience and perceived lower usability. A user interface that is easy to maintain and use, with seamless integration in a home, could increase user satisfaction and adoption.

Photovoltaic (PV) panels and sheets harvest light energy, and can also be low-cost, large-area light sensors that are self-powered, battery-free, flexible and portable. They can be aesthetically pleasing with different colour patterns and transparency, and can be deployed on different surfaces throughout the home. They offer a unique opportunity as a low maintenance integrated user interface for seamless contextual control of multiple different smart home devices by multiple users through touchless hovering hand gesture interaction. However, there is a lack of research into the design of interactions and applications with such energy harvesting interfaces. Also, there are no universal hovering hand gesture sets, either user-elicited or experimentally validated, for surface interaction or for interaction with smart home devices.

This thesis aims to develop a universal hovering hand gesture vocabulary for touchless surface interaction by exploring the control of multiple smart home devices with multiple users. First, a survey and semi-structured interviews were conducted to determine user preferences for touchless hovering gesture control of smart home devices using a tabletop PV sensor. Then, a literature review was conducted to generate a list of (15) interactively controlled smart home devices and their associated (78) functions or referents. An end-user ‘guessability’ study was then conducted with 20 non-technical participants suggesting one gesture per referent, and then created gesture sets using an ‘agreement’ analysis method. An end-user 'identification' study was then conducted for validation, in which 20 non-technical participants suggested one referent per gesture, and then compared using the resulting agreed referents. A novel end-user ‘matching’ study was also conducted for extended validation, in which 20 new non-technical participants matched two lists of agreed gestures and referents, and then created a list of agreed gesture–function combinations for smart home control.

An end-user ‘production’ study that is suggested in the literature as a parallel to the guessability study was also conducted with 25 new non-technical participants who suggested ‘unlimited’ gestures per referent, and then created resulting agreed gesture sets. Finally, an early prototype with a graphical user interface and real-time gesture recognition system was developed to control various smart home devices. Signals from the PV sensor were experimentally collected as time-series data for each agreed user-elicited gesture, and supervised machine learning models were developed to recognise them with very high accuracy.

This thesis contributes by introducing a unified framework for the design, validation and experimental recognition of gesture vocabularies. From this process, it derives several user-defined gesture sets of original agreed, preferred, popular and primitive hovering gestures along with their taxonomies for hovering tabletop interaction. This thesis also presents, for the first time, a comparison of two well-known user elicitation methods, guessability and production, highlighting the key similarities and differences between the two methods. Finally, it provides practical guidelines for designing hovering surface interaction for smart home control.

The thesis demonstrates an easy-to-maintain, easy-to-use and sustainable interface that integrates seamless interaction with smart home devices using hovering surface interaction. The methodological findings of the thesis inform human-computer interaction researchers in the design of user-elicited interactions for large-area, versatile electromagnetic field sensors and gesture-driven applications. This work is an important step towards achieving wider user acceptance and adoption of smart home systems through the use of interactive devices.