Fine-Tuning Large Language Models: A Systematic Review of Methods, Challenges, and Domain- Specific Adaptations
No Thumbnail Available
Date
2025
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Saudi Digital Library
Abstract
Fine-tuning large language models (LLMs) has
emerged as a crucial step for adapting these powerful models to
specialized tasks and domains. In this paper, we present a
systematic literature review of recent techniques for fine-tuning
LLMs, the challenges encountered across different application
domains, and the strategies developed to address domain-
specific requirements. We identify four key requirements for
effective fine-tuning: (1) Parameter-efficient and scalable
methods that mitigate the resource cost of updating billion-
parameter models, (2) High-quality, low-cost data usage
techniques for curating or generating training data, (3) Domain
adaptability and knowledge integration approaches (including
retrieval augmentation and alignment with knowledge bases),
and (4) Robust evaluation and interpretability practices to
ensure fine-tuned models are accurate and trustworthy. We
analyze six representative papers in diverse domains –
healthcare (biomedical LLMs like Med-PaLM and BioGPT),
recommender systems (e.g. the DEALRec data-efficient tuning
framework), smart manufacturing (knowledge-graph-
augmented RAG pipelines), socially-informed AI (instruction-
tuned models like FLAN and LLaMA-based alignments), and
education (comparing specialized small models to GPT-4 with
retrieval). Through this analysis, we synthesize how each
approach fulfills or falls short of the identified requirements.
Our review highlights emerging trends such as parameter-
efficient fine-tuning (PEFT), retrieval augmented generation
(RAG), and multi-task instruction tuning as promising
directions to specialize LLMs while controlling cost and
maintaining performance. We discuss open challenges including
the trade-off between efficiency and performance, data bias and
scarcity, maintaining generalization across domains, and
improving evaluation metrics and interpretability. Finally, we
outline future research opportunities to further enhance the
fine-tuning of LLMs for domain-specific applications.
Description
Keywords
Fine-Tuning, Large Language Models (LLMs), Parameter-Efficient, Domain Adaptation, Systematic Literature Review