SACM - United Kingdom
Permanent URI for this collectionhttps://drepo.sdl.edu.sa/handle/20.500.14154/9667
Browse
2 results
Search Results
Item Restricted Evaluating Text Summarization with Goal-Oriented Metrics: A Case Study using Large Language Models (LLMs) and Empowered GQM(University of Birmingham, 2024-09) Altamimi, Rana; Bahsoon, RamiThis study evaluates the performance of Large Language Models (LLMs) in dialogue summarization tasks, focusing on Gemma and Flan-T5. Employing a mixed-methods approach, we utilized the SAMSum dataset and developed an enhanced Goal-Question-Metric (GQM) framework for comprehensive assessment. Our evaluation combined traditional quantitative metrics (ROUGE, BLEU) with qualitative assessments performed by GPT-4, addressing multiple dimensions of summary quality. Results revealed that Flan-T5 consistently outperformed Gemma across both quantitative and qualitative metrics. Flan-T5 excelled in lexical overlap measures (ROUGE-1: 53.03, BLEU: 13.91) and demonstrated superior performance in qualitative assessments, particularly in conciseness (81.84/100) and coherence (77.89/100). Gemma, while showing competence, lagged behind Flan-T5 in most metrics. This study highlights the effectiveness of Flan-T5 in dialogue summarization tasks and underscores the importance of a multi-faceted evaluation approach in assessing LLM performance. Our findings suggest that future developments in this field should focus on enhancing lexical fidelity and higher-level qualities such as coherence and conciseness. This study contributes to the growing body of research on LLM evaluation and offers insights for improving dialogue summarization techniques.32 0Item Restricted Applications and effectiveness of using artificial intelligence in the assessment of individuals with aphasia, A scoping review(Saudi Digital Library, 2023-11-03) Alhejji, Bader; Caroline, Haw; Stuart, CunninghamAim: This review aims to examine the current uses of artificial intelligence (AI) in evaluating individuals with aphasia and assessing the effectiveness of these AI-based tools. Methodology: The scoping review methodology was employed to comprehensively investigate the role of artificial intelligence (AI) and machine learning in aphasia assessment. This encompassed systematic searches across databases such as ScienceOpen and PubMed, supplemented by Google Scholar and StarPlus for grey literature inclusion. Eligibility criteria targeted machine learning and AI techniques applied to adult aphasia patients, with a focus on post-2010 publications. The data extraction process involved documenting study particulars, AI algorithms, outcome measures, and findings. Descriptive analysis and statistical methods facilitated AI approach identification, categorization, and accuracy assessment. The PRISMA checklist was utilized for study quality evaluation, promoting transparency and rigor. Findings: Predominant AI approaches within this review were Support Vector Machines (SVMs) and Convolutional Neural Networks (CNNs), prominently featured across the selected studies. AI-based tools for the assessment of aphasia exhibited an average effectiveness of 88.3%, drawn from insights gleaned from 24 studies with 1546 participants. Remarkably, one of the AI apps achieved an accuracy of 95%, underscoring SVMs and CNNs' technology potential for accurate and impactful aphasia assessment outcomes. These findings emphasize the effectiveness and capacity of SVMs and CNNs to enrich clinical practice and expand research in aphasia evaluation. Conclusion: The present study identifies AI-based systems for assessing aphasia as a promising field, yet it acknowledges limitations concerning patient privacy and the need for a comprehensive AI-based system that covers the entire assessment process of aphasia.30 0