Fluent but misaligned : an NLP approach to measuring uptake in classroom AI-avatar dialogue

Abstract

Generative AI can sound fluent yet still miss what students mean, especially during brief classroom brainstorming. We analysed chat logs from a mathematics-for-game-development lesson in which 25 Hong Kong undergraduates worked in small groups and interacted with a generative AI avatar. We treated each student turn and the following avatar reply as a dyad and used Natural Language Processing (NLP) to measure three classroom-relevant signals: meaning continuity (Sentence-BERT cosine similarity), keyword carryover (Jaccard overlap of content-word lemmas), and interaction mode (Language Style Matching). Results showed an opportunity-structure effect: when students wrote longer, more content-rich turns, the avatar’s replies were more consistently on-topic and carried forward more task terms. We also observed two productive patterns (on-topic support with shared terminology or with paraphrase) and a risk pattern, *keyword echo*, where term reuse looked responsive but meaning continuity was weak. These measures offer educators a practical way to interpret classroom GenAI dialogue beyond surface fluency.