A new study published in the Proceedings of the 2023 IEEE Frontiers in Education Conference explored using large language models (LLMs) like ChatGPT to generate feedback on introductory programming assignments. As programming classes grow and resources remain limited, automating personalized feedback could support overburdened instructors.
Researchers Natalie Kiesler, Dominic Lohr, and Hieke Keuning investigated whether ChatGPT could mimic human teaching assistants and identify errors in beginner code. They prompted ChatGPT with 99 student submissions to 4 introductory programming exercises, asking "What's wrong with my code?"
The study found that ChatGPT provided reasonably good feedback for compiler errors and basic fixes. It gave textual explanations of bugs and offered improved code for 65 out of 99 submissions. This level of personalized feedback exceeds most existing auto-grading tools.
However, ChatGPT's feedback quality was lower for logic errors and when multiple mistakes were present. 61 of 99 responses contained misleading information that could confuse novices. There was also high variability between different ChatGPT responses to the same code.
The authors conclude that ChatGPT shows promise for automatically generating programming feedback. However, novices need guidance on properly prompting the system and critically evaluating responses. Future work should explore engineering better prompts and comparing ChatGPT to human teaching assistants.
Educators interested in testing ChatGPT's programming abilities can find the full paper here [link]. As AI advances, it may soon mimic human tutors in providing the personalized and high-quality feedback key to mastering coding.
Source: Exploring the Potential of Large Language Models to Generate Formative Programming Feedback
Visit the Paper and Github for more details.
All the credit for this research belongs to the researchers who worked on this project.
Also, make sure to join our AI SubReddit, Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, awesome AI projects, AI guides/tutorial, Best AI tools, and more.
