Decentralized federated distillation with protected pruned models in few global epochs

Abstract

Decentralized Federated Distillation (DFD) has emerged as a significant research direction since it not only supports heterogeneous model training but also naturally avoids privacy risks and communication bottlenecks stemming from the central server. DFD shows great potential in various training scenarios, especially in cross-silo federated environments. Existing DFD algorithms have limitations, including reliance on public datasets for distillation, insufficient privacy protection, and high communication overhead. This paper introduces Ring-Distill, a novel DFD algorithm designed specifically for cross-silo federated environments, effectively addressing the aforementioned limitations. Ring-Distill allows clients to complete federated training within a few global epochs (i.e., communication rounds) without sharing public datasets, e.g., N global epochs for a system involving N clients. To protect privacy and further reduce communication overhead, Ring-Distill contains a privacy-oriented automatic model pruning mechanism (PAMP) which can automatically create the privacy-preserving compressed model called proxy model for each client. These proxy models are employed for client-to-client distillation during each global epoch. Furthermore, Ring-Distill contains a historical model-based distillation (HMD) mechanism that allows local models to transfer knowledge from multiple historical proxy model replicas stored locally. Due to historical proxy model replicas, the HMD mechanism not only improves the distillation performance but also effectively mitigates client dropout issues. Theoretical analysis and comprehensive experiments show that Ring-Distill has significant advantages in terms of accuracy, privacy, and communication cost.