Simultaneous interpreting with auto-subtitling : investigating viewer cognitive effort, stress, and comprehension

Abstract

Simultaneous interpreting (SI) enables real-time cross-language communication without significant delays and is vital for fast-paced environments such as multilingual conferences. Automatic subtitles, powered by artificial intelligence (AI), is an important mode of audiovisual translation and has been widely deployed by virtual conferencing platforms to help users overcome language barriers. While the cognitive and emotional impacts of SI have been explored in prior studies, research directly comparing SI, auto-subtitling, and their combined use remains limited. This study investigates and compares the effectiveness of three interlingual translation modes, auto-subtitling, SI, and a combined dual-modality approach, on comprehension, cognitive load, and stress levels. Mandarin Chinese-speaking participants viewed a video presentation delivered in Arabic, a language they did not understand. Participants were divided into three groups: Group A relied on automatic subtitles in Simplified Chinese characters, Group B relied on SI in Mandarin, and Group C used a combination of both methods. Analysis of electroencephalographic data and comprehension test results revealed no statistically significant differences in content comprehension across the groups. However, Group A reported the poorest viewing experience, with the highest stress levels, while Group B expended the greatest cognitive effort. Group C exhibited the lowest levels of cognitive effort and stress, underscoring the advantages of dual-modality systems. These findings suggest that combining accurate automatic subtitles with professional interpreting may enhance accessibility, reduce cognitive demands, and improve the viewing experience, offering valuable insights into the integration of AI-driven technologies in SI.