Learning by imitating the classics: Mitigating class imbalance in federated learning via simulated centralized learning

Federated learning (FL) is a distributed machine learning framework in which multiple clients update their local models in parallel and then aggregate them to generate a global model. However, when local data on different clients are class imbalanced, local models trained on different clients are usually divergent, which can degrade the performance of the global model. To address the problem, most previous research has focused on reducing the divergence of local models across clients. Nonetheless, these methods may not be effective when the level of class imbalance is significant. We find that, to train a global model using imbalanced data of multiple clients, if we first aggregate their data in a central server and then implement centralized training, the obtained global model is much less affected by the class imbalance. The above centralized learning (CL) approach inspires us to design a method called FL via Simulated CL (FedSCL). The FedSCL method mimics the data-sampling and training process of CL by serially sampling a batch of local data from a randomly selected client and using it to update the global model. This is done in parallel with model aggregation for every few steps to improve efficiency and stability during training. Experimental results reveal that FedSCL achieves performance improvements of up to 2.90%, 5.44%, 9.51%, and 3.91% on the MNIST, FMNIST, CIFAR-10, and Tiny-ImageNet-200 datasets, respectively, compared to FedAvg. The paper also provides theoretical analysis of the parallel strategy used in FedSCL.