The Use of Text Recognition, Lip Reading, and Object Detection for Protecting Sensitive Information from Shoulder Surfing Attacks

Abstract

The portability and convenience of laptops have propelled their use in public venues. However, the risk of unauthorized view of sensitive information displayed on these devices, including business data, emails, banking information, online trading information, and private chats, raises privacy concerns. In particular, shoulder-surfing attacks pose a significant threat, whereby individuals can steal sensitive information by looking over one’s shoulder. While researchers have developed various approaches to protect users' screens, such as text modification-based, gesture-based, and external tool-based, those methods have limitations in terms of effectiveness, protection, and usability. To address these limitations, this dissertation proposes, develops, and evaluates three novel methods for protecting sensitive information from shoulder-surfing attacks: detection and labeling (D&L), recognizing and labeling sensitive information in text entry (RLSITE), and “someone is close” (SIC). D&L is a method designed to protect sensitive information while browsing. It works by recognizing and labeling sensitive information in text entry and replacing it with a category label. The labeled and hidden sensitive information is then read to users through their headphones when they click the label. RLSITE is a method designed to protect sensitive information while typing. It works by automatically capturing and interpreting users' lip movements of the sensitive information, then replacing it with a category label and reading it to users through their headphones when they click the label. Finally, the SIC method automatically detects whether someone is close to a user. If so, it will alert the user while labeling the sensitive information and reading it to users through their headphones. The proposed methods have been empirically evaluated in controlled laboratory settings using various measures, including usability, effectiveness, and protection. Evaluation results demonstrate that D&L, RLSITE, and SIC outperform baseline methods in all measures. Furthermore, these innovations have significant practical implications, making them more resistant to shoulder-surfing attacks to browse or enter sensitive content on devices without compromising the usability of these devices.