The Impact of LLMs Usage on Learning Outcomes for Software Development Students: A Focus on Prompt Engineering

Abstract

This study investigates the impact of large language model (LLM) usage, specifically ChatGPT, on student learning outcomes in programming education. The research adopts a mixed-methods approach, combining quantitative survey data from students and qualitative interviews with instructors. The study addresses three research questions: (1) the effect of LLM usage on undergraduate students' learning outcomes, (2) the influence of prompt engineering skills on this relationship, and (3) instructors' perceptions on these relationships. Quantitative data were collected from 159 students across two Saudi universities using a structured online survey with sections covering demographic information, LLM usage, self-reported programming understanding, and prompt engineering skills. Qualitative data were obtained through semi-structured interviews with programming instructors, covering LLM usage, prompt engineering skills, and their impact on student learning outcomes. The quantitative analysis utilized Partial Least Squares Structural Equation Modeling (PLS-SEM) to assess the measurement and structural models, including path coefficients, model explanatory power (R²), and predictive power (PLSpredict). Qualitative data were thematically analyzed using Atlas.ti to identify key themes related to instructor perspectives on the model. LLM usage positively impacts learning outcomes. While quantitative results did not show a significant moderating effect of prompt engineering skills, qualitative findings highlight its critical role in determining the positive effect of LLM usage on learning outcomes. The study emphasizes the importance of clear LLM usage policies and early prompt engineering training to promote meaningful engagement and maintain academic integrity in programming courses.