Distilling attention knowledge for speaker verification

Abstract

In recent years, the adoption of pre-trained speech models as feature extractors for speaker verification (SV) has surged. To reduce model complexity, knowledge from these pre-trained models has been transferred to compact student models, enabling them to achieve performance unattainable by conventional methods. While label-level and feature-level knowledge distillation (KD) have both yielded promising results in SV, attention-level KD remains under explored. Different from the other two methods, attention-level KD aims to guide the student network to focus more on the informative and important parts of the feature. To explore the attention-level KD in SV tasks, in this paper, we develop two distinct attention maps that enable the student model to learn the teacher’s attention along both the frequency and temporal dimensions, named Frequency-attentive KD (FREQ-AKD) and Temporal-attentive KD (TEMPO-AKD), respectively. Experiments conducted on the VoxCeleb and CN-Celeb datasets demonstrate the effectiveness of both FREQ-AKD and TEMPO-AKD.