ChatGPT as an automated writing evaluation tool : how students perceive it and how it affects their writing

Abstract

Generative artificial intelligence (GAI) language models, exemplified by ChatGPT, are significantly helping language learners in writing practice by providing immediate formative feedback. This study investigates whether ChatGPT’s writing feedback influences graduate students’ academic writing abilities and compares it with combined instructor and ChatGPT feedback. The students receiving only ChatGPT feedback (Group A) showed significant improvements in mechanics, tone, grammar, APA formatting, and overall writing quality, with lesser gains in organisation and no significant change in content. In contrast, the students who received combined feedback (Group B) exhibited significant enhancements in all of the parameters that were evaluated, including the areas in which Group A lagged. However, the only significant between-group difference was in grammar, although there was a marginal difference in organisation that suggested a trend towards greater improvement in Group B. The participants expressed positive attitudes about using ChatGPT as an automated writing evaluation tool, especially for correcting grammar and enriching vocabulary.