EVALUATING E-GOVERNMENT SYSTEMS USING LARGE LANGUAGE MODELS FROM SOFTWARE ENGINEERING BENCHMARKS: A CASE STUDY OF THE ZAKAT, TAX AND CUSTOMS AUTHORITY (ZATCA) IN THE KINGDOM OF SAUDI ARABIA

Abstract

The growth of e-government services has placed a greater emphasis on the need for designing e-government systems that are functional, but also usable, understandable, and meet users’ needs and expectations. In this regard, the e-systems of the Zakat, Tax and Customs Authority (ZATCA) – which is used in Saudi Arabia for managing zakat, tax, and customs services- are essential to provide tax, zakat and customs services to a variety of users, such as citizens, businesses, importers, exporters, customs brokers and travellers. Therefore, this study seeks to assess the usability and perceived requirements quality of ZATCA’s digital systems, as viewed from users' perspectives, and to explore the inter-relationships between usability, user experience, perceived requirements quality, and user satisfaction. The research is built upon well-established approaches to usability and software quality, and in line with the international standards such as ISO 9241-11:2018, ISO 9241-210:2019, ISO/IEC 25010:2023, and ISO/IEC 25030:2019. These standards have been widely applied in other studies to assess usability, user experience and quality of digital systems. The adoption of these standards allows the study to draw on previous work and offer a benchmarked assessment of e-government systems. The study used a mixed-methods approach in a descriptive and evaluative single-case study. The study used a bilingual online questionnaire distributed to a non-probability convenience sample with purposive inclusion criteria, targeting external users who had prior experience with ZATCA’s digital systems. The questionnaire contained the System Usability Scale (SUS), UMUX-Lite items, perceived requirements quality items, a measure of overall satisfaction, user-category questions, and open-ended questions. The study measured requirements quality from the users’ perspective and did not claim to directly audit ZATCA’s internal SRS, BRD, or development documentation. A total of 365 responses were used after cleaning. Descriptive statistics, Cronbach's alpha, Pearson correlation, one-way ANOVA and an independent-samples t-test were used to evaluate the quantitative data. The open-ended responses were analysed using thematic analysis and controlled LLM-based coding and classification. The findings show that the level of user satisfaction and usability of ZATCA's digital systems was moderately positive. The average SUS score was 68.253, with an average UMUX score of 3.778, average requirements quality of 3.649 and average satisfaction of 3.721. The results indicated high internal consistency for all scales of the study with Cronbach's alpha values of 0.969 for SUS, 0.894 for UMUX, and 0.948 for the requirements quality scale. The results found very strong positive and significant associations between usability and user experience, requirements quality and satisfaction. For instance, increased usability was strongly linked to increased satisfaction and better requirements quality. The inferential analysis revealed statistically significant differences among users. The highest levels of usability and satisfaction were indicated by individual taxpayers or zakat payers, while customs brokers and some business-related users indicated lower levels of satisfaction. The independent-samples t-test also revealed statistically significant between business/company representatives and individual taxpayers, with individual users showing more positive perceptions. The qualitative findings generally corroborated the quantitative findings by revealing common user complaints about system processes, terminology, guidance and access to services. In conclusion, this study suggests that ZATCA systems are at an acceptable level of operation; but there is room for improvement in usability, requirements and user-centered design. Improving guidance, terminology, and tailoring system design to various user segments can positively impact user satisfaction and the success of digital public service.