Multi-modal EEG datasets and benchmarks for EEG-based neural decoding research

Abstract

The growing availability of publicly shared EEG datasets involving audio, video, and linguistic stimuli offers great potential for advancing multi-modal neural decoding research. However, significant heterogeneity in file formats, annotation structures, and temporal alignment across datasets has hindered their direct usability and cross-dataset benchmarking. In this study, we present a unified preprocessing pipeline applied to six diverse EEG datasets that cover auditory attention, language comprehension, and visual perception. The proposed pipeline standardizes EEG representations and multimodal annotations, allowing the resulting data to be aligned in time and directly usable for large-scale modeling tasks. We summarize the resulting unified database and highlight its consistency in structure and format. To demonstrate usability, we provide benchmark classification results on label-eligible datasets using both traditional methods (rLDA) and deep learning (EEGNet). The resulting resource enables consistent multimodal EEG analyses and provides a structured basis for advancing research on cross-modal alignment and neural decoding. The accompanying benchmark experiments serve to illustrate the usability of the standardized datasets; achieving competitive performance was not the aim of these evaluations.