A multiple LLM approach to automated essay scoring : comparing seven open-source models with human graders

Abstract

This study evaluates whether open-source large language models (LLMs) can support automated essay scoring (AES) of reflective writing in higher education. Forty-nine reflective essays from a foundation entrepreneurship and innovation course were scored by lecturers and independently graded by seven open-source LLMs (Llama 3 8B, DeepSeek R1 32B, Ministral 3 8B, Qwen 3 8B, GPT-OSS 20B, Gemma 3 27B, and Phi 3 14B) using the same rubric. Model performance was assessed via exact match, mean absolute error (MAE), Pearson and Spearman correlations, and paired t-tests/Wilcoxon tests against human scores. Ministral produced complete outputs and demonstrated the most balanced alignment with human grading, showing no significant difference in mean or median scores. Other models exhibited larger deviations, conservative bias, or missing outputs, raising scalability concerns. Results suggest that LLMs may serve as decision-support tools but do not yet replace human judgement for reflective assessment.