Toward Leveraging Artificial Intelligence to Support the Identification of Accessibility Challenges
Abstract
Context: Today, mobile devices provide support to disabled people to make their life easier due to their high accessibility and capability, e.g., finding accessible locations, picture and voice-based communication, customized user interfaces, and vocabulary levels. These accessibility frameworks are directly integrated, as libraries, in various apps, providing them with accessibility functions. Just like any other software, these frameworks regularly encounter errors. App developers report these errors in the form of bug reports or by the user in user reviews. User reviews include insights that are useful for app evolution. These reports related to accessibility faults/issues need to be urgently fixed since their existence significantly hinders the usability of apps. However, recent studies have shown that developers may incorporate accessibility strategies in inspecting manually or partial reports to investigate if there are accessibility reports that exist. Unfortunately, these studies are limited to the developer. With the increase in the number of received reviews, manually analyzing them is tedious and time-consuming, especially when searching for accessibility reviews.
Objective: The goal of this thesis is to support the automated identification of accessibility in user reviews or bug reports, to help technology professionals prioritize their handling, and, thus, to create more inclusive apps. Particularly, we propose a model that takes as input accessibility user reviews or bug reports and learns their keyword-based features to make a classification decision, for a given review, on whether it is about accessibility or not. To complement this goal, we aim to reveal insights from deaf and hard-of-hearing students about Blackboard, which is one of the most common Learning Management systems (LMS) used by many universities, especially in the current COVID-19 pandemic. This occurs to explore how deaf and hard-of-hearing students have challenges and concerns in e-learning experiences during the sudden shift to online learning during COVID-19 in terms of accessibility.
Method: Our empirically-driven study follows a mixture of qualitative and quantitative methods. We text mine user reviews and bug reports documentation. We identify the accessibility reports and categorize them based on the BBC standards and guidelines for mobile accessibility and Web Content Accessibility Guidelines (WCAG 2.1). Then, we automatically classify a large set of user reviews and bug reports and identify among the various accessibility models presented in the literature. After that, we used a mixed-methods approach by conducting a survey and interviews to get the information we needed. This was done on deaf and hard-of-hearing students to identify the challenges and concerns in terms of accessibility in the e-learning platform Blackboard.
Result: We introduced models that can accurately identify accessibility reviews and bug reports and automate detecting them. Our models (1) outperform two baselines, namely a keyword-based detector and a random classifier; (2) our model achieves an accuracy of 91% with a relatively small training dataset; however, the accuracy improves as we increase the size of the training dataset. Our mixed methods with deaf and hard-of-hearing students have revealed several difficulties, such as inadequate support and inaccessibility of content from learning systems.
Conclusion: Our models can automatically classify app reviews and bug reports as accessibility-related or not so developers can easily detect accessibility issues with their products and improve them to more accessible and inclusive apps utilizing the users' input. Our goal is to create a sustainable change by including a model in the developer’s software maintenance pipeline and raising awareness of existing errors that hinder the accessibility of mobile apps, which is a pressing need. In light of our findings from the Blackboard case study, Blackboard and the course material are not easily accessible to deaf students and hard of hearing. Thus, deaf students find that learning is extremely stressful during the pandemic.
Description
Keywords
Software Accessibility, Machine Learning, HCI, NLP, Artificial Intelligence, User Review