论文标题
自适应蒸馏从异质客户分散学习的自适应蒸馏
Adaptive Distillation for Decentralized Learning from Heterogeneous Clients
论文作者
论文摘要
本文通过要求一组客户共享与自己的数据资源预先训练的本地模型，以解决分散学习的问题，以实现高性能的全球模型。我们对客户模型体系结构和数据分布都多样化的特定情况特别感兴趣，这使得采用常规方法（例如联合学习和网络共同依据）并非平凡。为此，我们提出了一种新的分散学习方法，称为通过自适应蒸馏（DLAD）分散学习。给定客户模型的集合和大量未标记的蒸馏样品，提出的DLAD 1）汇总了客户端模型的输出，同时自适应地强调了在给定蒸馏样本中具有更高信心的人和2）训练全局模型以模仿聚集的输出。我们对多个公共数据集（MNIST，CIFAR-10和CINIC-10）进行了广泛的实验评估证明了该方法的有效性。
This paper addresses the problem of decentralized learning to achieve a high-performance global model by asking a group of clients to share local models pre-trained with their own data resources. We are particularly interested in a specific case where both the client model architectures and data distributions are diverse, which makes it nontrivial to adopt conventional approaches such as Federated Learning and network co-distillation. To this end, we propose a new decentralized learning method called Decentralized Learning via Adaptive Distillation (DLAD). Given a collection of client models and a large number of unlabeled distillation samples, the proposed DLAD 1) aggregates the outputs of the client models while adaptively emphasizing those with higher confidence in given distillation samples and 2) trains the global model to imitate the aggregated outputs. Our extensive experimental evaluation on multiple public datasets (MNIST, CIFAR-10, and CINIC-10) demonstrates the effectiveness of the proposed method.
