Automated Repair of Accessibility Issues in Mobile Applications

Abstract

Mobile accessibility is more critical than ever due to the significant increase in mobile app usage, particularly among people with disabilities who rely on mobile devices to access essential information and services. People with vision and motor disabilities often use assistive technologies to interact with mobile applications. However, recent studies show that a significant percentage of mobile apps remain inaccessible due to layout accessibility issues, making them challenging to use for older adults and people with disabilities. Unfortunately, existing techniques are limited in helping developers debug these issues; they can only detect issues but not repair them. Therefore, the repair of layout accessibility issues remains a manual, labor-intensive, and error-prone process.

Automated repair of layout accessibility issues is complicated by several challenges. First, a repair must account for multiple issues holistically in order to preserve the relative consistency of the original app design. Second, due to the complex relationship between UI components, there is no straightforward way of identifying the set of elements and properties that need to be modified for a given issue. Third, assuming the relevant views and properties could be identified, the number of possible changes that need to be considered grows exponentially as more elements and properties need to be considered. Finally, a change in one element can create cascading changes that lead to new problems in other areas of the UI. Together, these challenges make a seemingly simple repair difficult to achieve.

In this dissertation, I introduce a repair framework that builds and analyzes models of the User Interface (UI) and leverages multi-objective genetic search algorithms to repair layout accessibility issues. To evaluate the effectiveness of the framework, I instantiated it to repair the different known types of layout accessibility issues in mobile apps. The empirical evaluation of these instantiations on real-world mobile apps demonstrated their effectiveness in repairing these issues. In addition, I conducted user studies to assess the impact of the repairs on the UI quality and aesthetics. The results demonstrated that the repaired UIs were not only more accessible but also did not distort or significantly change their original design. Overall, these results are positive and indicate that my repair framework can be highly effective in automatically repairing layout accessibility issues in mobile applications. Overall, my results confirm my dissertation's hypothesis that a repair framework employing a multi-objective genetic search-based approach can be highly effective in automatically repairing layout accessibility issues in mobile applications.