AI4OPT Seminar Series

Date: Thursday, September 29, 2022

Time: Noon – 1:00 pm

Meeting Link: https://gatech.zoom.us/j/99381428980

Speaker: Satyen Kale

Optimization Algorithms for Heterogeneous Clients in Federated Learning

Abstract:  Federated Learning has emerged as an important paradigm in modern large-scale machine learning, where the training data remains distributed over a vast number of clients, which may be phones, network sensors, hospitals, etc. A major challenge in the design of optimization methods for Federated Learning is the heterogeneity (i.e. non-iid nature) of client data. This problem affects the currently dominant algorithm deployed in practice known as Federated Averaging (FedAvg): we provide results for FedAvg quantifying the degree to which this problem causes unstable or slow convergence. We develop two optimization algorithms to handle this problem for two different settings of Federated Learning. In the cross-silo setting of Federated Learning, we propose a new algorithm called SCAFFOLD which uses control variates to correct for client heterogeneity and prove that SCAFFOLD requires significantly fewer communication rounds and is not affected by data heterogeneity or client sampling. In the cross-device setting of Federated Learning, we propose a general framework called Mime which mitigates client-drift and adapts arbitrary centralized optimization algorithms (e.g. SGD, Adam, etc.) to Federated Learning via a combination of control-variates and server-level statistics (e.g. momentum) at every client-update step. Our theoretical and empirical analyses establish the superiority of SCAFFOLD and Mime over other baselines in their respective settings.

Bio: Satyen Kale is a research scientist at Google Research working in the New York office. His current research is the design of efficient and practical algorithms for fundamental problems in Machine Learning and Optimization. More specifically, he is interested in decision-making under uncertainty, statistical learning theory, combinatorial optimization, and continuous optimization. His research has been recognized with several awards: a Best Paper Award at ICML 2015, a Best Paper Award at ICLR 2018, and a Best Student Paper Award at COLT 2018. He was a program chair for COLT 2017 and ALT 2019.