Evaluating the impact of Large Language Models on problem-solving skills in programming debugging of IT undergraduates

Abstract

This study investigates the impact of Large Language Models (LLMs) on problem-solving skills in source code debugging among IT undergraduates. A pre-, mid-, and post-experimental design was employed, including pre-test, mid-test, post-test (Prior), and post-test (Recent) phases to assess debugging performance with and without LLM assistance. The sample consisted of 87 students from the Department of Industrial Management, University of Kelaniya, Sri Lanka, stratified by gender, academic level, A/L stream, Z-score, and GPA. Results showed significant improvement in debugging accuracy, increasing from 46.53% in the pre-test to 69.51% in the post-test (Prior), indicating skill retention. Task efficiency also improved, with completion time reduced from 18 minutes to 10 minutes. However, transferability to new problems was moderate, with a post-test (Recent) accuracy of 58.40%. Higher academic levels, technical A/L streams, and mid-range GPAs were associated with better retention and adaptability. While LLMs enhanced immediate performance, the findings highlight the need to balance their use with independent practice to support long-term skill development. Limitations include resource constraints and short study duration, suggesting the need for longitudinal research. The study recommends structured integration of LLMs to optimize programming education outcomes.