SACM - United Kingdom
Permanent URI for this collectionhttps://drepo.sdl.edu.sa/handle/20.500.14154/9667
Browse
2 results
Search Results
Item Restricted Evaluating Text Summarization with Goal-Oriented Metrics: A Case Study using Large Language Models (LLMs) and Empowered GQM(University of Birmingham, 2024-09) Altamimi, Rana; Bahsoon, RamiThis study evaluates the performance of Large Language Models (LLMs) in dialogue summarization tasks, focusing on Gemma and Flan-T5. Employing a mixed-methods approach, we utilized the SAMSum dataset and developed an enhanced Goal-Question-Metric (GQM) framework for comprehensive assessment. Our evaluation combined traditional quantitative metrics (ROUGE, BLEU) with qualitative assessments performed by GPT-4, addressing multiple dimensions of summary quality. Results revealed that Flan-T5 consistently outperformed Gemma across both quantitative and qualitative metrics. Flan-T5 excelled in lexical overlap measures (ROUGE-1: 53.03, BLEU: 13.91) and demonstrated superior performance in qualitative assessments, particularly in conciseness (81.84/100) and coherence (77.89/100). Gemma, while showing competence, lagged behind Flan-T5 in most metrics. This study highlights the effectiveness of Flan-T5 in dialogue summarization tasks and underscores the importance of a multi-faceted evaluation approach in assessing LLM performance. Our findings suggest that future developments in this field should focus on enhancing lexical fidelity and higher-level qualities such as coherence and conciseness. This study contributes to the growing body of research on LLM evaluation and offers insights for improving dialogue summarization techniques.32 0Item Restricted Model Driven Development of Mobile Health Applications(King's College London, 2024-09) Alwakeel, Lyan Abdulrahman; Zschaler, Steffen; Lano, KevinThe proliferation of mobile devices has created a demand for software applications that can be run on these devices. However, developing mobile applications that meet user requirements and can function on a variety of devices poses several challenges. This is especially true for mobile applications in critical domains like healthcare, where the stakes are high and quality is paramount. In response, software engineering has focused on improving the development process, methods, and tools to create high-quality mobile applications. One promising approach is Model-Driven Development (MDD), which generates low-level code from high-level models, enabling developers to “write once, run anywhere”. The MDD approach plays a substantial role in increasing software productivity and enhancing solution quality by automating the generation of implementations across various platforms, instead of relying on manual coding for each platform version. The selection of the right architecture design and back-end services is crucial for developing effective application solutions, and expertise in these choices can be encoded into an MDD process. While there have been some research efforts to apply MDD to mobile applications, the existing studies are limited and offer opportunities for further improvement. Currently, the published work primarily focuses on either generating user interfaces, developing simple data-centric applications, or applications with predefined functions. Therefore, this research introduces AppCraft, a framework based on MDD that is specifically designed for developing cross-platform mobile health (mHealth) applications. The AppCraft framework simplifies the generation of complex, intelligent, and high-quality self-management mHealth applications by leveraging MDD principles. Moreover, this research investigates the potential of AppCraft in integrating machine learning models within mHealth applications. By leveraging high-level models, AppCraft simplifies the integration process, providing significant benefits to both developers and machine learning engineers. This allows developers to accelerate the mobile application development process and enables ML engineers to test their models effectively. The research is based on design science research principles, which employ artefacts as proof-of-concept to validate research findings. The research commences by conducting a thorough review of existing MDD frameworks, mobile development approaches, and mobile architectures. This is followed by a systematic literature review focused on self-management mHealth applications. Drawing insights from this analysis, the AppCraft framework is developed as a result. The effectiveness of the AppCraft framework is assessed through a series of case studies in the healthcare domain, demonstrating substantial reductions in development time and effort. The evaluation validates the framework’s applicability, flexibility, and simplicity. Furthermore, the generated applications undergo comprehensive evaluation, affirming their efficiency, consistency, and usability.77 0