Stretching AI's reach : assessing an AI-driven feedback system for extended academic writing

Abstract

Advances in large language models (LLMs) enable timely and scalable writing evaluation. Previous research has shown that LLM-driven conversational systems, such as ChatGPT, can provide feedback on short essays. However, it is unclear whether AI can effectively evaluate more demanding genres. This study investigates a custom-built writing feedback system developed at a Hong Kong university that uses OpenAI's GPT-4 Turbo (0125-preview) to provide rubric-based feedback on a 1500-word academic report. Guided by a detailed, rubric-aligned prompt, the system generated 333 feedback items from 37 undergraduates, which were analysed for accuracy, tone, and inclusion of examples. The analysis showed that most feedback was accurate and addressed both strengths and weaknesses, but over half lacked concrete examples. Often recycling phrases from rubric descriptors, the feedback was largely generic and occasionally inaccurate. Interview data from six students revealed that the AI feedback was valued for its coverage, efficiency, and constructive tone, yet its generic nature undermined its usefulness. Despite these limitations, students expressed interest in receiving both AI and teacher feedback for the efficiency and coverage that AI offers, alongside the specificity and relevance of teacher input. These findings suggest that employing a well-crafted prompt on an AI model with a large context window does not necessarily guarantee substantive feedback. Therefore, educators using AI-driven feedback systems should thoroughly assess these systems' capacity to handle extended academic writing. Future research could explore ways to refine prompts and system design for long-form writing assignments.