Advancing Distributed and Federated Optimization Theory.

The widespread adoption of large-scale machine learning has highlighted the critical need for efficient and robust distributed optimization techniques, particularly in federated learning settings. This doctoral thesis comprehensively addresses several fundamental challenges at the intersection of optimization theory and practical distributed systems. The research makes seven distinct contributions. It provides theoretical backing for local gradient steps to accelerate communication (ProxSkip), eliminates neighborhood error in stochastic local updates (Variance Reduced ProxSkip), and demonstrates that local steps remain effective even with partial client participation. Further, it shows how server-side stepsizes and sampling without replacement improve convergence in heterogeneous environments. The thesis also explores gradient compression, proving that compressing gradient differences is superior to compressing raw gradients. It establishes that Byzantine robustness can be achieved simultaneously with partial participation using gradient-difference clipping. Finally, it develops the first theoretical framework for low-rank adaptation based on randomized asymmetric chains, offering new insights into fine-tuning large models. These contributions introduce novel algorithms, establish rigorous guarantees under realistic assumptions, and are supported by numerical experiments, significantly advancing the field of distributed and federated optimization.