CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study</title>  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <meta content="Federated LearningGradient DescentOptimization." lang="en" name="keywords"/> <base href="/html/2502.00182v2/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S1" title="In Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S2" title="In Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>From Gradient Descent to FedAvg</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S3" title="In Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Experimental Setup</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S3.SS1" title="In 3 Experimental Setup ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.1 </span>Basic Configuration</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S3.SS2" title="In 3 Experimental Setup ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.2 </span>FL Setup</span></a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S3.SS2.SSS1" title="In 3.2 FL Setup ‣ 3 Experimental Setup ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.2.1 </span>Evaluation</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S3.SS2.SSS2" title="In 3.2 FL Setup ‣ 3 Experimental Setup ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.2.2 </span>Transition from Centralized to Federated Learning</span></a></li> </ol> </li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4" title="In Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Experimental Results</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.SS1" title="In 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.1 </span>Impact of Hyperparameters under IID Condition</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.SS2" title="In 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2 </span>Partial Participation and Imbalanced Data under IID Condition</span></a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.SS2.SSS1" title="In 4.2 Partial Participation and Imbalanced Data under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2.1 </span>Impact of Partial Participation</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.SS2.SSS2" title="In 4.2 Partial Participation and Imbalanced Data under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2.2 </span>Impact of Imbalanced Data.</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.SS3" title="In 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.3 </span>Training Dynamics Under Non-IID Setting</span></a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.SS3.SSS1" title="In 4.3 Training Dynamics Under Non-IID Setting ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.3.1 </span>Non-IID with Dirichlet Distribution</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.SS3.SSS2" title="In 4.3 Training Dynamics Under Non-IID Setting ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.3.2 </span>Impact of Hyperparameters under Non-IID Condition</span></a></li> </ol> </li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S5" title="In Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Understanding FL from a Loss Landscape Perspective</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S6" title="In Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>Discussion and Conclusion</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S6.SS0.SSS1" title="In 6 Discussion and Conclusion ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6.0.1 </span>Acknowledgements</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S6.SS0.SSS2" title="In 6 Discussion and Conclusion ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6.0.2 </span><span class="ltx_ERROR undefined">\discintname</span></span></a></li> </ol> </li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line"><span class="ltx_note ltx_role_institutetext" id="id1"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">institutetext: </span>University of Stavanger, Norway <br class="ltx_break"/><span class="ltx_note ltx_role_email" id="id1.1"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">email: </span>{jungwon.seo, f.ozgur.catak, chunming.rong}@uis.no</span></span></span> </span></span></span> <h1 class="ltx_title ltx_title_document">Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study</h1> <div class="ltx_authors"> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Jungwon Seo </span></span> <span class="ltx_author_before"> </span><span class="ltx_creator ltx_role_author"> <span class="ltx_personname"> Ferhat Ozgur Catak </span></span> <span class="ltx_author_before"> </span><span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Chunming Rong </span></span> </div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract</h6> <p class="ltx_p" id="id1.id1">As privacy concerns and data regulations grow, federated learning (FL) has emerged as a promising approach for training machine learning models across decentralized data sources without sharing raw data. However, a significant challenge in FL is that client data are often non-IID (non-independent and identically distributed), leading to reduced performance compared to centralized learning. While many methods have been proposed to address this issue, their underlying mechanisms are often viewed from different perspectives. Through a comprehensive investigation from gradient descent to FL, and from IID to non-IID data settings, we find that inconsistencies in client loss landscapes primarily cause performance degradation in non-IID scenarios. From this understanding, we observe that existing methods can be grouped into two main strategies: (i) adjusting parameter update paths and (ii) modifying client loss landscapes. These findings offer a clear perspective on addressing non-IID challenges in FL and help guide future research in the field.</p> </div> <div class="ltx_keywords"> <h6 class="ltx_title ltx_title_keywords">Keywords: </h6>Federated LearningGradient DescentOptimization. </div> <div class="ltx_para" id="p1"> <span class="ltx_ERROR undefined" id="p1.1">{textblock*}</span> <p class="ltx_p" id="p1.2">(-40mm,-70mm) <span class="ltx_text ltx_font_italic" id="p1.2.1" style="font-size:80%;">Presented at the 36th NIKT 25–27 November 2024, Bergen, Norway<span class="ltx_text ltx_font_upright" id="p1.2.1.1"> </span></span></p> </div> <section class="ltx_section" id="S1"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2> <div class="ltx_para" id="S1.p1"> <p class="ltx_p" id="S1.p1.1">Federated Learning (FL) is a distributed machine learning framework that prioritizes privacy preservation <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib14" title="">14</a>]</cite>. Unlike traditional centralized methods, where client data is aggregated on a central server for model training, FL allows clients to keep their data locally. Clients collaborate by sharing only their locally trained models with a central server, which aggregates these models to build a global model. This approach enables learning from diverse, distributed datasets without exposing data, making FL especially valuable for privacy-sensitive applications.</p> </div> <div class="ltx_para" id="S1.p2"> <p class="ltx_p" id="S1.p2.1">Although similar distributed learning frameworks have existed before <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib3" title="">3</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib23" title="">23</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib4" title="">4</a>]</cite>, the term "Federated Learning" was officially coined by McMahan et al. in 2016 <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib14" title="">14</a>]</cite>. Since then, particularly in deep learning, FL has gained significant attention in both research and industry <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib6" title="">6</a>]</cite>. One of the primary challenges in FL is the heterogeneity of data across clients <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib22" title="">22</a>]</cite>. When client data distributions are non-IID (not Independent and Identically Distributed), the learning direction of individual clients may diverge from that of the global model. This divergence, often referred to as "<span class="ltx_text ltx_font_italic" id="S1.p2.1.1">client drift</span>", can degrade the overall performance of the global model and, in extreme cases, may even hinder convergence <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib7" title="">7</a>]</cite>. Numerous studies have addressed this issue, primarily from an optimization perspective, proposing both theoretical and empirical methods <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib13" title="">13</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib7" title="">7</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib1" title="">1</a>]</cite>.</p> </div> <div class="ltx_para" id="S1.p3"> <p class="ltx_p" id="S1.p3.1">As the field has matured, a variety of new perspectives have emerged. Some researchers argue that FL’s challenges resemble those found in catastrophic forgetting, similar to continual learning <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib10" title="">10</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib9" title="">9</a>]</cite>, while others suggest that the smoothness of the loss function, inspired by sharpness-aware training, plays a crucial role <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib15" title="">15</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib20" title="">20</a>]</cite>. Additionally, some point to differences in learned representations, relating to concepts from contrastive learning <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib12" title="">12</a>]</cite>.</p> </div> <div class="ltx_para" id="S1.p4"> <p class="ltx_p" id="S1.p4.1">Initially, the FL problem was perceived as a straightforward optimization task. However, the incorporation of new perspectives has introduced additional layers of complexity, making it challenging to identify the critical areas of focus. Moreover, most existing studies primarily investigate FL under non-IID settings, leaving a gap in our understanding of how FL behaves from scratch. As a result, FL has become a "black box within a black box," exacerbating the opacity already present in deep learning models. This multitude of perspectives has made it difficult for researchers to interpret proposed methods and to determine the most promising research directions.</p> </div> <div class="ltx_para" id="S1.p5"> <p class="ltx_p" id="S1.p5.1">To address this, we propose a systematic investigation of FL, starting with IID settings and gradually exploring its behavior across various hyperparameters and conditions. By grounding our analysis in gradient descent and extending it to FL, we aim to uncover the mechanisms driving its performance and explain the unexpected phenomena observed in complex, non-IID environments. Through this approach, we hope to clarify FL’s core challenges and offer direction for future research.</p> </div> <div class="ltx_para" id="S1.p6"> <p class="ltx_p" id="S1.p6.1">The contributions of this study are as follows:</p> <ul class="ltx_itemize" id="S1.I1"> <li class="ltx_item" id="S1.I1.i1" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S1.I1.i1.p1"> <p class="ltx_p" id="S1.I1.i1.p1.1"><span class="ltx_text ltx_font_bold" id="S1.I1.i1.p1.1.1">Optimization Algorithm Review</span>: We provide a clear review of optimization algorithms, from gradient descent to FedAvg.</p> </div> </li> <li class="ltx_item" id="S1.I1.i2" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S1.I1.i2.p1"> <p class="ltx_p" id="S1.I1.i2.p1.1"><span class="ltx_text ltx_font_bold" id="S1.I1.i2.p1.1.1">Experiments in IID Settings</span>: We conduct extensive experiments on FedAvg under IID conditions, exploring various hyperparameters and FL settings to gain insights into its behavior.</p> </div> </li> <li class="ltx_item" id="S1.I1.i3" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S1.I1.i3.p1"> <p class="ltx_p" id="S1.I1.i3.p1.1"><span class="ltx_text ltx_font_bold" id="S1.I1.i3.p1.1.1">Non-IID Analysis</span>: Using insights from the IID experiments, we analyze FL performance in non-IID settings, interpreting the observed phenomena.</p> </div> </li> <li class="ltx_item" id="S1.I1.i4" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S1.I1.i4.p1"> <p class="ltx_p" id="S1.I1.i4.p1.1"><span class="ltx_text ltx_font_bold" id="S1.I1.i4.p1.1.1">Categorizing Approaches for Non-IID Challenges</span>: We classify existing and future methods into two strategies: (a) update path adjustment and (b) loss landscape modification, addressing non-IID data in FL.</p> </div> </li> </ul> </div> </section> <section class="ltx_section" id="S2"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">2 </span>From Gradient Descent to FedAvg</h2> <div class="ltx_para" id="S2.p1"> <p class="ltx_p" id="S2.p1.5">Gradient Descent (GD) is one of the most fundamental algorithms in machine learning for optimizing model parameters, such as those in neural networks <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib17" title="">17</a>]</cite>. Consider an objective function <math alttext="F(\theta)" class="ltx_Math" display="inline" id="S2.p1.1.m1.1"><semantics id="S2.p1.1.m1.1a"><mrow id="S2.p1.1.m1.1.2" xref="S2.p1.1.m1.1.2.cmml"><mi id="S2.p1.1.m1.1.2.2" xref="S2.p1.1.m1.1.2.2.cmml">F</mi><mo id="S2.p1.1.m1.1.2.1" xref="S2.p1.1.m1.1.2.1.cmml">⁢</mo><mrow id="S2.p1.1.m1.1.2.3.2" xref="S2.p1.1.m1.1.2.cmml"><mo id="S2.p1.1.m1.1.2.3.2.1" stretchy="false" xref="S2.p1.1.m1.1.2.cmml">(</mo><mi id="S2.p1.1.m1.1.1" xref="S2.p1.1.m1.1.1.cmml">θ</mi><mo id="S2.p1.1.m1.1.2.3.2.2" stretchy="false" xref="S2.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p1.1.m1.1b"><apply id="S2.p1.1.m1.1.2.cmml" xref="S2.p1.1.m1.1.2"><times id="S2.p1.1.m1.1.2.1.cmml" xref="S2.p1.1.m1.1.2.1"></times><ci id="S2.p1.1.m1.1.2.2.cmml" xref="S2.p1.1.m1.1.2.2">𝐹</ci><ci id="S2.p1.1.m1.1.1.cmml" xref="S2.p1.1.m1.1.1">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.1.m1.1c">F(\theta)</annotation><annotation encoding="application/x-llamapun" id="S2.p1.1.m1.1d">italic_F ( italic_θ )</annotation></semantics></math> that we seek to minimize, where <math alttext="\theta" class="ltx_Math" display="inline" id="S2.p1.2.m2.1"><semantics id="S2.p1.2.m2.1a"><mi id="S2.p1.2.m2.1.1" xref="S2.p1.2.m2.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S2.p1.2.m2.1b"><ci id="S2.p1.2.m2.1.1.cmml" xref="S2.p1.2.m2.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.2.m2.1c">\theta</annotation><annotation encoding="application/x-llamapun" id="S2.p1.2.m2.1d">italic_θ</annotation></semantics></math> represents the model parameters, with <math alttext="\theta\in\mathbb{R}^{d}" class="ltx_Math" display="inline" id="S2.p1.3.m3.1"><semantics id="S2.p1.3.m3.1a"><mrow id="S2.p1.3.m3.1.1" xref="S2.p1.3.m3.1.1.cmml"><mi id="S2.p1.3.m3.1.1.2" xref="S2.p1.3.m3.1.1.2.cmml">θ</mi><mo id="S2.p1.3.m3.1.1.1" xref="S2.p1.3.m3.1.1.1.cmml">∈</mo><msup id="S2.p1.3.m3.1.1.3" xref="S2.p1.3.m3.1.1.3.cmml"><mi id="S2.p1.3.m3.1.1.3.2" xref="S2.p1.3.m3.1.1.3.2.cmml">ℝ</mi><mi id="S2.p1.3.m3.1.1.3.3" xref="S2.p1.3.m3.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.p1.3.m3.1b"><apply id="S2.p1.3.m3.1.1.cmml" xref="S2.p1.3.m3.1.1"><in id="S2.p1.3.m3.1.1.1.cmml" xref="S2.p1.3.m3.1.1.1"></in><ci id="S2.p1.3.m3.1.1.2.cmml" xref="S2.p1.3.m3.1.1.2">𝜃</ci><apply id="S2.p1.3.m3.1.1.3.cmml" xref="S2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.p1.3.m3.1.1.3.1.cmml" xref="S2.p1.3.m3.1.1.3">superscript</csymbol><ci id="S2.p1.3.m3.1.1.3.2.cmml" xref="S2.p1.3.m3.1.1.3.2">ℝ</ci><ci id="S2.p1.3.m3.1.1.3.3.cmml" xref="S2.p1.3.m3.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.3.m3.1c">\theta\in\mathbb{R}^{d}</annotation><annotation encoding="application/x-llamapun" id="S2.p1.3.m3.1d">italic_θ ∈ blackboard_R start_POSTSUPERSCRIPT italic_d end_POSTSUPERSCRIPT</annotation></semantics></math>, and <math alttext="d" class="ltx_Math" display="inline" id="S2.p1.4.m4.1"><semantics id="S2.p1.4.m4.1a"><mi id="S2.p1.4.m4.1.1" xref="S2.p1.4.m4.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.p1.4.m4.1b"><ci id="S2.p1.4.m4.1.1.cmml" xref="S2.p1.4.m4.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.4.m4.1c">d</annotation><annotation encoding="application/x-llamapun" id="S2.p1.4.m4.1d">italic_d</annotation></semantics></math> is the dimensionality of <math alttext="\theta" class="ltx_Math" display="inline" id="S2.p1.5.m5.1"><semantics id="S2.p1.5.m5.1a"><mi id="S2.p1.5.m5.1.1" xref="S2.p1.5.m5.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S2.p1.5.m5.1b"><ci id="S2.p1.5.m5.1.1.cmml" xref="S2.p1.5.m5.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.5.m5.1c">\theta</annotation><annotation encoding="application/x-llamapun" id="S2.p1.5.m5.1d">italic_θ</annotation></semantics></math>. The parameter update rule in GD is given by:</p> </div> <div class="ltx_para" id="S2.p2"> <table class="ltx_equation ltx_eqn_table" id="S2.E1"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="\theta^{t+1}=\theta^{t}-\eta\nabla F(\theta^{t};\mathcal{D})" class="ltx_Math" display="block" id="S2.E1.m1.2"><semantics id="S2.E1.m1.2a"><mrow id="S2.E1.m1.2.2" xref="S2.E1.m1.2.2.cmml"><msup id="S2.E1.m1.2.2.3" xref="S2.E1.m1.2.2.3.cmml"><mi id="S2.E1.m1.2.2.3.2" xref="S2.E1.m1.2.2.3.2.cmml">θ</mi><mrow id="S2.E1.m1.2.2.3.3" xref="S2.E1.m1.2.2.3.3.cmml"><mi id="S2.E1.m1.2.2.3.3.2" xref="S2.E1.m1.2.2.3.3.2.cmml">t</mi><mo id="S2.E1.m1.2.2.3.3.1" xref="S2.E1.m1.2.2.3.3.1.cmml">+</mo><mn id="S2.E1.m1.2.2.3.3.3" xref="S2.E1.m1.2.2.3.3.3.cmml">1</mn></mrow></msup><mo id="S2.E1.m1.2.2.2" xref="S2.E1.m1.2.2.2.cmml">=</mo><mrow id="S2.E1.m1.2.2.1" xref="S2.E1.m1.2.2.1.cmml"><msup id="S2.E1.m1.2.2.1.3" xref="S2.E1.m1.2.2.1.3.cmml"><mi id="S2.E1.m1.2.2.1.3.2" xref="S2.E1.m1.2.2.1.3.2.cmml">θ</mi><mi id="S2.E1.m1.2.2.1.3.3" xref="S2.E1.m1.2.2.1.3.3.cmml">t</mi></msup><mo id="S2.E1.m1.2.2.1.2" xref="S2.E1.m1.2.2.1.2.cmml">−</mo><mrow id="S2.E1.m1.2.2.1.1" xref="S2.E1.m1.2.2.1.1.cmml"><mi id="S2.E1.m1.2.2.1.1.3" xref="S2.E1.m1.2.2.1.1.3.cmml">η</mi><mo id="S2.E1.m1.2.2.1.1.2" lspace="0.167em" xref="S2.E1.m1.2.2.1.1.2.cmml">⁢</mo><mrow id="S2.E1.m1.2.2.1.1.4" xref="S2.E1.m1.2.2.1.1.4.cmml"><mo id="S2.E1.m1.2.2.1.1.4.1" rspace="0.167em" xref="S2.E1.m1.2.2.1.1.4.1.cmml">∇</mo><mi id="S2.E1.m1.2.2.1.1.4.2" xref="S2.E1.m1.2.2.1.1.4.2.cmml">F</mi></mrow><mo id="S2.E1.m1.2.2.1.1.2a" xref="S2.E1.m1.2.2.1.1.2.cmml">⁢</mo><mrow id="S2.E1.m1.2.2.1.1.1.1" xref="S2.E1.m1.2.2.1.1.1.2.cmml"><mo id="S2.E1.m1.2.2.1.1.1.1.2" stretchy="false" xref="S2.E1.m1.2.2.1.1.1.2.cmml">(</mo><msup id="S2.E1.m1.2.2.1.1.1.1.1" xref="S2.E1.m1.2.2.1.1.1.1.1.cmml"><mi id="S2.E1.m1.2.2.1.1.1.1.1.2" xref="S2.E1.m1.2.2.1.1.1.1.1.2.cmml">θ</mi><mi id="S2.E1.m1.2.2.1.1.1.1.1.3" xref="S2.E1.m1.2.2.1.1.1.1.1.3.cmml">t</mi></msup><mo id="S2.E1.m1.2.2.1.1.1.1.3" xref="S2.E1.m1.2.2.1.1.1.2.cmml">;</mo><mi class="ltx_font_mathcaligraphic" id="S2.E1.m1.1.1" xref="S2.E1.m1.1.1.cmml">𝒟</mi><mo id="S2.E1.m1.2.2.1.1.1.1.4" stretchy="false" xref="S2.E1.m1.2.2.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.2b"><apply id="S2.E1.m1.2.2.cmml" xref="S2.E1.m1.2.2"><eq id="S2.E1.m1.2.2.2.cmml" xref="S2.E1.m1.2.2.2"></eq><apply id="S2.E1.m1.2.2.3.cmml" xref="S2.E1.m1.2.2.3"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.3.1.cmml" xref="S2.E1.m1.2.2.3">superscript</csymbol><ci id="S2.E1.m1.2.2.3.2.cmml" xref="S2.E1.m1.2.2.3.2">𝜃</ci><apply id="S2.E1.m1.2.2.3.3.cmml" xref="S2.E1.m1.2.2.3.3"><plus id="S2.E1.m1.2.2.3.3.1.cmml" xref="S2.E1.m1.2.2.3.3.1"></plus><ci id="S2.E1.m1.2.2.3.3.2.cmml" xref="S2.E1.m1.2.2.3.3.2">𝑡</ci><cn id="S2.E1.m1.2.2.3.3.3.cmml" type="integer" xref="S2.E1.m1.2.2.3.3.3">1</cn></apply></apply><apply id="S2.E1.m1.2.2.1.cmml" xref="S2.E1.m1.2.2.1"><minus id="S2.E1.m1.2.2.1.2.cmml" xref="S2.E1.m1.2.2.1.2"></minus><apply id="S2.E1.m1.2.2.1.3.cmml" xref="S2.E1.m1.2.2.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.1.3.1.cmml" xref="S2.E1.m1.2.2.1.3">superscript</csymbol><ci id="S2.E1.m1.2.2.1.3.2.cmml" xref="S2.E1.m1.2.2.1.3.2">𝜃</ci><ci id="S2.E1.m1.2.2.1.3.3.cmml" xref="S2.E1.m1.2.2.1.3.3">𝑡</ci></apply><apply id="S2.E1.m1.2.2.1.1.cmml" xref="S2.E1.m1.2.2.1.1"><times id="S2.E1.m1.2.2.1.1.2.cmml" xref="S2.E1.m1.2.2.1.1.2"></times><ci id="S2.E1.m1.2.2.1.1.3.cmml" xref="S2.E1.m1.2.2.1.1.3">𝜂</ci><apply id="S2.E1.m1.2.2.1.1.4.cmml" xref="S2.E1.m1.2.2.1.1.4"><ci id="S2.E1.m1.2.2.1.1.4.1.cmml" xref="S2.E1.m1.2.2.1.1.4.1">∇</ci><ci id="S2.E1.m1.2.2.1.1.4.2.cmml" xref="S2.E1.m1.2.2.1.1.4.2">𝐹</ci></apply><list id="S2.E1.m1.2.2.1.1.1.2.cmml" xref="S2.E1.m1.2.2.1.1.1.1"><apply id="S2.E1.m1.2.2.1.1.1.1.1.cmml" xref="S2.E1.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E1.m1.2.2.1.1.1.1.1">superscript</csymbol><ci id="S2.E1.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.E1.m1.2.2.1.1.1.1.1.2">𝜃</ci><ci id="S2.E1.m1.2.2.1.1.1.1.1.3.cmml" xref="S2.E1.m1.2.2.1.1.1.1.1.3">𝑡</ci></apply><ci id="S2.E1.m1.1.1.cmml" xref="S2.E1.m1.1.1">𝒟</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.2c">\theta^{t+1}=\theta^{t}-\eta\nabla F(\theta^{t};\mathcal{D})</annotation><annotation encoding="application/x-llamapun" id="S2.E1.m1.2d">italic_θ start_POSTSUPERSCRIPT italic_t + 1 end_POSTSUPERSCRIPT = italic_θ start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT - italic_η ∇ italic_F ( italic_θ start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ; caligraphic_D )</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S2.p3"> <p class="ltx_p" id="S2.p3.6">Here, <math alttext="t" class="ltx_Math" display="inline" id="S2.p3.1.m1.1"><semantics id="S2.p3.1.m1.1a"><mi id="S2.p3.1.m1.1.1" xref="S2.p3.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.p3.1.m1.1b"><ci id="S2.p3.1.m1.1.1.cmml" xref="S2.p3.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.1.m1.1c">t</annotation><annotation encoding="application/x-llamapun" id="S2.p3.1.m1.1d">italic_t</annotation></semantics></math> denotes the iteration step, and <math alttext="\nabla F(\theta^{t};\mathcal{D})" class="ltx_Math" display="inline" id="S2.p3.2.m2.2"><semantics id="S2.p3.2.m2.2a"><mrow id="S2.p3.2.m2.2.2" xref="S2.p3.2.m2.2.2.cmml"><mrow id="S2.p3.2.m2.2.2.3" xref="S2.p3.2.m2.2.2.3.cmml"><mo id="S2.p3.2.m2.2.2.3.1" rspace="0.167em" xref="S2.p3.2.m2.2.2.3.1.cmml">∇</mo><mi id="S2.p3.2.m2.2.2.3.2" xref="S2.p3.2.m2.2.2.3.2.cmml">F</mi></mrow><mo id="S2.p3.2.m2.2.2.2" xref="S2.p3.2.m2.2.2.2.cmml">⁢</mo><mrow id="S2.p3.2.m2.2.2.1.1" xref="S2.p3.2.m2.2.2.1.2.cmml"><mo id="S2.p3.2.m2.2.2.1.1.2" stretchy="false" xref="S2.p3.2.m2.2.2.1.2.cmml">(</mo><msup id="S2.p3.2.m2.2.2.1.1.1" xref="S2.p3.2.m2.2.2.1.1.1.cmml"><mi id="S2.p3.2.m2.2.2.1.1.1.2" xref="S2.p3.2.m2.2.2.1.1.1.2.cmml">θ</mi><mi id="S2.p3.2.m2.2.2.1.1.1.3" xref="S2.p3.2.m2.2.2.1.1.1.3.cmml">t</mi></msup><mo id="S2.p3.2.m2.2.2.1.1.3" xref="S2.p3.2.m2.2.2.1.2.cmml">;</mo><mi class="ltx_font_mathcaligraphic" id="S2.p3.2.m2.1.1" xref="S2.p3.2.m2.1.1.cmml">𝒟</mi><mo id="S2.p3.2.m2.2.2.1.1.4" stretchy="false" xref="S2.p3.2.m2.2.2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p3.2.m2.2b"><apply id="S2.p3.2.m2.2.2.cmml" xref="S2.p3.2.m2.2.2"><times id="S2.p3.2.m2.2.2.2.cmml" xref="S2.p3.2.m2.2.2.2"></times><apply id="S2.p3.2.m2.2.2.3.cmml" xref="S2.p3.2.m2.2.2.3"><ci id="S2.p3.2.m2.2.2.3.1.cmml" xref="S2.p3.2.m2.2.2.3.1">∇</ci><ci id="S2.p3.2.m2.2.2.3.2.cmml" xref="S2.p3.2.m2.2.2.3.2">𝐹</ci></apply><list id="S2.p3.2.m2.2.2.1.2.cmml" xref="S2.p3.2.m2.2.2.1.1"><apply id="S2.p3.2.m2.2.2.1.1.1.cmml" xref="S2.p3.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.p3.2.m2.2.2.1.1.1.1.cmml" xref="S2.p3.2.m2.2.2.1.1.1">superscript</csymbol><ci id="S2.p3.2.m2.2.2.1.1.1.2.cmml" xref="S2.p3.2.m2.2.2.1.1.1.2">𝜃</ci><ci id="S2.p3.2.m2.2.2.1.1.1.3.cmml" xref="S2.p3.2.m2.2.2.1.1.1.3">𝑡</ci></apply><ci id="S2.p3.2.m2.1.1.cmml" xref="S2.p3.2.m2.1.1">𝒟</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.2.m2.2c">\nabla F(\theta^{t};\mathcal{D})</annotation><annotation encoding="application/x-llamapun" id="S2.p3.2.m2.2d">∇ italic_F ( italic_θ start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ; caligraphic_D )</annotation></semantics></math> represents the gradient of the objective function with respect to <math alttext="\theta^{t}" class="ltx_Math" display="inline" id="S2.p3.3.m3.1"><semantics id="S2.p3.3.m3.1a"><msup id="S2.p3.3.m3.1.1" xref="S2.p3.3.m3.1.1.cmml"><mi id="S2.p3.3.m3.1.1.2" xref="S2.p3.3.m3.1.1.2.cmml">θ</mi><mi id="S2.p3.3.m3.1.1.3" xref="S2.p3.3.m3.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S2.p3.3.m3.1b"><apply id="S2.p3.3.m3.1.1.cmml" xref="S2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S2.p3.3.m3.1.1.1.cmml" xref="S2.p3.3.m3.1.1">superscript</csymbol><ci id="S2.p3.3.m3.1.1.2.cmml" xref="S2.p3.3.m3.1.1.2">𝜃</ci><ci id="S2.p3.3.m3.1.1.3.cmml" xref="S2.p3.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.3.m3.1c">\theta^{t}</annotation><annotation encoding="application/x-llamapun" id="S2.p3.3.m3.1d">italic_θ start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT</annotation></semantics></math>, evaluated on the dataset <math alttext="\mathcal{D}" class="ltx_Math" display="inline" id="S2.p3.4.m4.1"><semantics id="S2.p3.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p3.4.m4.1.1" xref="S2.p3.4.m4.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S2.p3.4.m4.1b"><ci id="S2.p3.4.m4.1.1.cmml" xref="S2.p3.4.m4.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.4.m4.1c">\mathcal{D}</annotation><annotation encoding="application/x-llamapun" id="S2.p3.4.m4.1d">caligraphic_D</annotation></semantics></math>. By updating the parameters in the negative direction of the gradient with a learning rate <math alttext="\eta" class="ltx_Math" display="inline" id="S2.p3.5.m5.1"><semantics id="S2.p3.5.m5.1a"><mi id="S2.p3.5.m5.1.1" xref="S2.p3.5.m5.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S2.p3.5.m5.1b"><ci id="S2.p3.5.m5.1.1.cmml" xref="S2.p3.5.m5.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.5.m5.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S2.p3.5.m5.1d">italic_η</annotation></semantics></math>, the parameters move towards a more optimal solution. This iterative process is repeated until the gradient <math alttext="\nabla F(\theta^{t})" class="ltx_Math" display="inline" id="S2.p3.6.m6.1"><semantics id="S2.p3.6.m6.1a"><mrow id="S2.p3.6.m6.1.1" xref="S2.p3.6.m6.1.1.cmml"><mrow id="S2.p3.6.m6.1.1.3" xref="S2.p3.6.m6.1.1.3.cmml"><mo id="S2.p3.6.m6.1.1.3.1" rspace="0.167em" xref="S2.p3.6.m6.1.1.3.1.cmml">∇</mo><mi id="S2.p3.6.m6.1.1.3.2" xref="S2.p3.6.m6.1.1.3.2.cmml">F</mi></mrow><mo id="S2.p3.6.m6.1.1.2" xref="S2.p3.6.m6.1.1.2.cmml">⁢</mo><mrow id="S2.p3.6.m6.1.1.1.1" xref="S2.p3.6.m6.1.1.1.1.1.cmml"><mo id="S2.p3.6.m6.1.1.1.1.2" stretchy="false" xref="S2.p3.6.m6.1.1.1.1.1.cmml">(</mo><msup id="S2.p3.6.m6.1.1.1.1.1" xref="S2.p3.6.m6.1.1.1.1.1.cmml"><mi id="S2.p3.6.m6.1.1.1.1.1.2" xref="S2.p3.6.m6.1.1.1.1.1.2.cmml">θ</mi><mi id="S2.p3.6.m6.1.1.1.1.1.3" xref="S2.p3.6.m6.1.1.1.1.1.3.cmml">t</mi></msup><mo id="S2.p3.6.m6.1.1.1.1.3" stretchy="false" xref="S2.p3.6.m6.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p3.6.m6.1b"><apply id="S2.p3.6.m6.1.1.cmml" xref="S2.p3.6.m6.1.1"><times id="S2.p3.6.m6.1.1.2.cmml" xref="S2.p3.6.m6.1.1.2"></times><apply id="S2.p3.6.m6.1.1.3.cmml" xref="S2.p3.6.m6.1.1.3"><ci id="S2.p3.6.m6.1.1.3.1.cmml" xref="S2.p3.6.m6.1.1.3.1">∇</ci><ci id="S2.p3.6.m6.1.1.3.2.cmml" xref="S2.p3.6.m6.1.1.3.2">𝐹</ci></apply><apply id="S2.p3.6.m6.1.1.1.1.1.cmml" xref="S2.p3.6.m6.1.1.1.1"><csymbol cd="ambiguous" id="S2.p3.6.m6.1.1.1.1.1.1.cmml" xref="S2.p3.6.m6.1.1.1.1">superscript</csymbol><ci id="S2.p3.6.m6.1.1.1.1.1.2.cmml" xref="S2.p3.6.m6.1.1.1.1.1.2">𝜃</ci><ci id="S2.p3.6.m6.1.1.1.1.1.3.cmml" xref="S2.p3.6.m6.1.1.1.1.1.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.6.m6.1c">\nabla F(\theta^{t})</annotation><annotation encoding="application/x-llamapun" id="S2.p3.6.m6.1d">∇ italic_F ( italic_θ start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT )</annotation></semantics></math> converges to zero, indicating that the model has reached a local or global minimum.</p> </div> <div class="ltx_para" id="S2.p4"> <p class="ltx_p" id="S2.p4.2">A major limitation of GD lies in its dependence on a static dataset <math alttext="\mathcal{D}" class="ltx_Math" display="inline" id="S2.p4.1.m1.1"><semantics id="S2.p4.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p4.1.m1.1.1" xref="S2.p4.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S2.p4.1.m1.1b"><ci id="S2.p4.1.m1.1.1.cmml" xref="S2.p4.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p4.1.m1.1c">\mathcal{D}</annotation><annotation encoding="application/x-llamapun" id="S2.p4.1.m1.1d">caligraphic_D</annotation></semantics></math>. This becomes problematic when the dataset is too large to fit into a single device’s memory or when full-batch training becomes computationally expensive. GD requires computing the gradient over all data points and using the averaged gradient for each update at iteration <math alttext="t" class="ltx_Math" display="inline" id="S2.p4.2.m2.1"><semantics id="S2.p4.2.m2.1a"><mi id="S2.p4.2.m2.1.1" xref="S2.p4.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.p4.2.m2.1b"><ci id="S2.p4.2.m2.1.1.cmml" xref="S2.p4.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p4.2.m2.1c">t</annotation><annotation encoding="application/x-llamapun" id="S2.p4.2.m2.1d">italic_t</annotation></semantics></math>. Expanding the update rule in Eq. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S2.E1" title="In 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">1</span></a>, we obtain:</p> </div> <div class="ltx_para" id="S2.p5"> <table class="ltx_equation ltx_eqn_table" id="S2.E2"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="\theta^{t+1}=\theta^{t}-\eta\frac{1}{|\mathcal{D}|}\sum_{i\in\mathcal{D}}% \nabla F(\theta^{t};x_{i},y_{i})" class="ltx_Math" display="block" id="S2.E2.m1.4"><semantics id="S2.E2.m1.4a"><mrow id="S2.E2.m1.4.4" xref="S2.E2.m1.4.4.cmml"><msup id="S2.E2.m1.4.4.5" xref="S2.E2.m1.4.4.5.cmml"><mi id="S2.E2.m1.4.4.5.2" xref="S2.E2.m1.4.4.5.2.cmml">θ</mi><mrow id="S2.E2.m1.4.4.5.3" xref="S2.E2.m1.4.4.5.3.cmml"><mi id="S2.E2.m1.4.4.5.3.2" xref="S2.E2.m1.4.4.5.3.2.cmml">t</mi><mo id="S2.E2.m1.4.4.5.3.1" xref="S2.E2.m1.4.4.5.3.1.cmml">+</mo><mn id="S2.E2.m1.4.4.5.3.3" xref="S2.E2.m1.4.4.5.3.3.cmml">1</mn></mrow></msup><mo id="S2.E2.m1.4.4.4" xref="S2.E2.m1.4.4.4.cmml">=</mo><mrow id="S2.E2.m1.4.4.3" xref="S2.E2.m1.4.4.3.cmml"><msup id="S2.E2.m1.4.4.3.5" xref="S2.E2.m1.4.4.3.5.cmml"><mi id="S2.E2.m1.4.4.3.5.2" xref="S2.E2.m1.4.4.3.5.2.cmml">θ</mi><mi id="S2.E2.m1.4.4.3.5.3" xref="S2.E2.m1.4.4.3.5.3.cmml">t</mi></msup><mo id="S2.E2.m1.4.4.3.4" xref="S2.E2.m1.4.4.3.4.cmml">−</mo><mrow id="S2.E2.m1.4.4.3.3" xref="S2.E2.m1.4.4.3.3.cmml"><mi id="S2.E2.m1.4.4.3.3.5" xref="S2.E2.m1.4.4.3.3.5.cmml">η</mi><mo id="S2.E2.m1.4.4.3.3.4" xref="S2.E2.m1.4.4.3.3.4.cmml">⁢</mo><mfrac id="S2.E2.m1.1.1" xref="S2.E2.m1.1.1.cmml"><mn id="S2.E2.m1.1.1.3" xref="S2.E2.m1.1.1.3.cmml">1</mn><mrow id="S2.E2.m1.1.1.1.3" xref="S2.E2.m1.1.1.1.2.cmml"><mo id="S2.E2.m1.1.1.1.3.1" stretchy="false" xref="S2.E2.m1.1.1.1.2.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S2.E2.m1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.cmml">𝒟</mi><mo id="S2.E2.m1.1.1.1.3.2" stretchy="false" xref="S2.E2.m1.1.1.1.2.1.cmml">|</mo></mrow></mfrac><mo id="S2.E2.m1.4.4.3.3.4a" xref="S2.E2.m1.4.4.3.3.4.cmml">⁢</mo><mrow id="S2.E2.m1.4.4.3.3.3" xref="S2.E2.m1.4.4.3.3.3.cmml"><munder id="S2.E2.m1.4.4.3.3.3.4" xref="S2.E2.m1.4.4.3.3.3.4.cmml"><mo id="S2.E2.m1.4.4.3.3.3.4.2" movablelimits="false" xref="S2.E2.m1.4.4.3.3.3.4.2.cmml">∑</mo><mrow id="S2.E2.m1.4.4.3.3.3.4.3" xref="S2.E2.m1.4.4.3.3.3.4.3.cmml"><mi id="S2.E2.m1.4.4.3.3.3.4.3.2" xref="S2.E2.m1.4.4.3.3.3.4.3.2.cmml">i</mi><mo id="S2.E2.m1.4.4.3.3.3.4.3.1" xref="S2.E2.m1.4.4.3.3.3.4.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.E2.m1.4.4.3.3.3.4.3.3" xref="S2.E2.m1.4.4.3.3.3.4.3.3.cmml">𝒟</mi></mrow></munder><mrow id="S2.E2.m1.4.4.3.3.3.3" xref="S2.E2.m1.4.4.3.3.3.3.cmml"><mrow id="S2.E2.m1.4.4.3.3.3.3.5" xref="S2.E2.m1.4.4.3.3.3.3.5.cmml"><mo id="S2.E2.m1.4.4.3.3.3.3.5.1" rspace="0.167em" xref="S2.E2.m1.4.4.3.3.3.3.5.1.cmml">∇</mo><mi id="S2.E2.m1.4.4.3.3.3.3.5.2" xref="S2.E2.m1.4.4.3.3.3.3.5.2.cmml">F</mi></mrow><mo id="S2.E2.m1.4.4.3.3.3.3.4" xref="S2.E2.m1.4.4.3.3.3.3.4.cmml">⁢</mo><mrow id="S2.E2.m1.4.4.3.3.3.3.3.3" xref="S2.E2.m1.4.4.3.3.3.3.3.4.cmml"><mo id="S2.E2.m1.4.4.3.3.3.3.3.3.4" stretchy="false" xref="S2.E2.m1.4.4.3.3.3.3.3.4.cmml">(</mo><msup id="S2.E2.m1.2.2.1.1.1.1.1.1.1" xref="S2.E2.m1.2.2.1.1.1.1.1.1.1.cmml"><mi id="S2.E2.m1.2.2.1.1.1.1.1.1.1.2" xref="S2.E2.m1.2.2.1.1.1.1.1.1.1.2.cmml">θ</mi><mi id="S2.E2.m1.2.2.1.1.1.1.1.1.1.3" xref="S2.E2.m1.2.2.1.1.1.1.1.1.1.3.cmml">t</mi></msup><mo id="S2.E2.m1.4.4.3.3.3.3.3.3.5" xref="S2.E2.m1.4.4.3.3.3.3.3.4.cmml">;</mo><msub id="S2.E2.m1.3.3.2.2.2.2.2.2.2" xref="S2.E2.m1.3.3.2.2.2.2.2.2.2.cmml"><mi id="S2.E2.m1.3.3.2.2.2.2.2.2.2.2" xref="S2.E2.m1.3.3.2.2.2.2.2.2.2.2.cmml">x</mi><mi id="S2.E2.m1.3.3.2.2.2.2.2.2.2.3" xref="S2.E2.m1.3.3.2.2.2.2.2.2.2.3.cmml">i</mi></msub><mo id="S2.E2.m1.4.4.3.3.3.3.3.3.6" xref="S2.E2.m1.4.4.3.3.3.3.3.4.cmml">,</mo><msub id="S2.E2.m1.4.4.3.3.3.3.3.3.3" xref="S2.E2.m1.4.4.3.3.3.3.3.3.3.cmml"><mi id="S2.E2.m1.4.4.3.3.3.3.3.3.3.2" xref="S2.E2.m1.4.4.3.3.3.3.3.3.3.2.cmml">y</mi><mi id="S2.E2.m1.4.4.3.3.3.3.3.3.3.3" xref="S2.E2.m1.4.4.3.3.3.3.3.3.3.3.cmml">i</mi></msub><mo id="S2.E2.m1.4.4.3.3.3.3.3.3.7" stretchy="false" xref="S2.E2.m1.4.4.3.3.3.3.3.4.cmml">)</mo></mrow></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.4b"><apply id="S2.E2.m1.4.4.cmml" xref="S2.E2.m1.4.4"><eq id="S2.E2.m1.4.4.4.cmml" xref="S2.E2.m1.4.4.4"></eq><apply id="S2.E2.m1.4.4.5.cmml" xref="S2.E2.m1.4.4.5"><csymbol cd="ambiguous" id="S2.E2.m1.4.4.5.1.cmml" xref="S2.E2.m1.4.4.5">superscript</csymbol><ci id="S2.E2.m1.4.4.5.2.cmml" xref="S2.E2.m1.4.4.5.2">𝜃</ci><apply id="S2.E2.m1.4.4.5.3.cmml" xref="S2.E2.m1.4.4.5.3"><plus id="S2.E2.m1.4.4.5.3.1.cmml" xref="S2.E2.m1.4.4.5.3.1"></plus><ci id="S2.E2.m1.4.4.5.3.2.cmml" xref="S2.E2.m1.4.4.5.3.2">𝑡</ci><cn id="S2.E2.m1.4.4.5.3.3.cmml" type="integer" xref="S2.E2.m1.4.4.5.3.3">1</cn></apply></apply><apply id="S2.E2.m1.4.4.3.cmml" xref="S2.E2.m1.4.4.3"><minus id="S2.E2.m1.4.4.3.4.cmml" xref="S2.E2.m1.4.4.3.4"></minus><apply id="S2.E2.m1.4.4.3.5.cmml" xref="S2.E2.m1.4.4.3.5"><csymbol cd="ambiguous" id="S2.E2.m1.4.4.3.5.1.cmml" xref="S2.E2.m1.4.4.3.5">superscript</csymbol><ci id="S2.E2.m1.4.4.3.5.2.cmml" xref="S2.E2.m1.4.4.3.5.2">𝜃</ci><ci id="S2.E2.m1.4.4.3.5.3.cmml" xref="S2.E2.m1.4.4.3.5.3">𝑡</ci></apply><apply id="S2.E2.m1.4.4.3.3.cmml" xref="S2.E2.m1.4.4.3.3"><times id="S2.E2.m1.4.4.3.3.4.cmml" xref="S2.E2.m1.4.4.3.3.4"></times><ci id="S2.E2.m1.4.4.3.3.5.cmml" xref="S2.E2.m1.4.4.3.3.5">𝜂</ci><apply id="S2.E2.m1.1.1.cmml" xref="S2.E2.m1.1.1"><divide id="S2.E2.m1.1.1.2.cmml" xref="S2.E2.m1.1.1"></divide><cn id="S2.E2.m1.1.1.3.cmml" type="integer" xref="S2.E2.m1.1.1.3">1</cn><apply id="S2.E2.m1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.3"><abs id="S2.E2.m1.1.1.1.2.1.cmml" xref="S2.E2.m1.1.1.1.3.1"></abs><ci id="S2.E2.m1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1">𝒟</ci></apply></apply><apply id="S2.E2.m1.4.4.3.3.3.cmml" xref="S2.E2.m1.4.4.3.3.3"><apply id="S2.E2.m1.4.4.3.3.3.4.cmml" xref="S2.E2.m1.4.4.3.3.3.4"><csymbol cd="ambiguous" id="S2.E2.m1.4.4.3.3.3.4.1.cmml" xref="S2.E2.m1.4.4.3.3.3.4">subscript</csymbol><sum id="S2.E2.m1.4.4.3.3.3.4.2.cmml" xref="S2.E2.m1.4.4.3.3.3.4.2"></sum><apply id="S2.E2.m1.4.4.3.3.3.4.3.cmml" xref="S2.E2.m1.4.4.3.3.3.4.3"><in id="S2.E2.m1.4.4.3.3.3.4.3.1.cmml" xref="S2.E2.m1.4.4.3.3.3.4.3.1"></in><ci id="S2.E2.m1.4.4.3.3.3.4.3.2.cmml" xref="S2.E2.m1.4.4.3.3.3.4.3.2">𝑖</ci><ci id="S2.E2.m1.4.4.3.3.3.4.3.3.cmml" xref="S2.E2.m1.4.4.3.3.3.4.3.3">𝒟</ci></apply></apply><apply id="S2.E2.m1.4.4.3.3.3.3.cmml" xref="S2.E2.m1.4.4.3.3.3.3"><times id="S2.E2.m1.4.4.3.3.3.3.4.cmml" xref="S2.E2.m1.4.4.3.3.3.3.4"></times><apply id="S2.E2.m1.4.4.3.3.3.3.5.cmml" xref="S2.E2.m1.4.4.3.3.3.3.5"><ci id="S2.E2.m1.4.4.3.3.3.3.5.1.cmml" xref="S2.E2.m1.4.4.3.3.3.3.5.1">∇</ci><ci id="S2.E2.m1.4.4.3.3.3.3.5.2.cmml" xref="S2.E2.m1.4.4.3.3.3.3.5.2">𝐹</ci></apply><list id="S2.E2.m1.4.4.3.3.3.3.3.4.cmml" xref="S2.E2.m1.4.4.3.3.3.3.3.3"><apply id="S2.E2.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.2.2.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.2.2.1.1.1.1.1.1.1">superscript</csymbol><ci id="S2.E2.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S2.E2.m1.2.2.1.1.1.1.1.1.1.2">𝜃</ci><ci id="S2.E2.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S2.E2.m1.2.2.1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.E2.m1.3.3.2.2.2.2.2.2.2.cmml" xref="S2.E2.m1.3.3.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E2.m1.3.3.2.2.2.2.2.2.2.1.cmml" xref="S2.E2.m1.3.3.2.2.2.2.2.2.2">subscript</csymbol><ci id="S2.E2.m1.3.3.2.2.2.2.2.2.2.2.cmml" xref="S2.E2.m1.3.3.2.2.2.2.2.2.2.2">𝑥</ci><ci id="S2.E2.m1.3.3.2.2.2.2.2.2.2.3.cmml" xref="S2.E2.m1.3.3.2.2.2.2.2.2.2.3">𝑖</ci></apply><apply id="S2.E2.m1.4.4.3.3.3.3.3.3.3.cmml" xref="S2.E2.m1.4.4.3.3.3.3.3.3.3"><csymbol cd="ambiguous" id="S2.E2.m1.4.4.3.3.3.3.3.3.3.1.cmml" xref="S2.E2.m1.4.4.3.3.3.3.3.3.3">subscript</csymbol><ci id="S2.E2.m1.4.4.3.3.3.3.3.3.3.2.cmml" xref="S2.E2.m1.4.4.3.3.3.3.3.3.3.2">𝑦</ci><ci id="S2.E2.m1.4.4.3.3.3.3.3.3.3.3.cmml" xref="S2.E2.m1.4.4.3.3.3.3.3.3.3.3">𝑖</ci></apply></list></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.4c">\theta^{t+1}=\theta^{t}-\eta\frac{1}{|\mathcal{D}|}\sum_{i\in\mathcal{D}}% \nabla F(\theta^{t};x_{i},y_{i})</annotation><annotation encoding="application/x-llamapun" id="S2.E2.m1.4d">italic_θ start_POSTSUPERSCRIPT italic_t + 1 end_POSTSUPERSCRIPT = italic_θ start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT - italic_η divide start_ARG 1 end_ARG start_ARG | caligraphic_D | end_ARG ∑ start_POSTSUBSCRIPT italic_i ∈ caligraphic_D end_POSTSUBSCRIPT ∇ italic_F ( italic_θ start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ; italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S2.p6"> <p class="ltx_p" id="S2.p6.3">Here, <math alttext="x_{i}" class="ltx_Math" display="inline" id="S2.p6.1.m1.1"><semantics id="S2.p6.1.m1.1a"><msub id="S2.p6.1.m1.1.1" xref="S2.p6.1.m1.1.1.cmml"><mi id="S2.p6.1.m1.1.1.2" xref="S2.p6.1.m1.1.1.2.cmml">x</mi><mi id="S2.p6.1.m1.1.1.3" xref="S2.p6.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p6.1.m1.1b"><apply id="S2.p6.1.m1.1.1.cmml" xref="S2.p6.1.m1.1.1"><csymbol cd="ambiguous" id="S2.p6.1.m1.1.1.1.cmml" xref="S2.p6.1.m1.1.1">subscript</csymbol><ci id="S2.p6.1.m1.1.1.2.cmml" xref="S2.p6.1.m1.1.1.2">𝑥</ci><ci id="S2.p6.1.m1.1.1.3.cmml" xref="S2.p6.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p6.1.m1.1c">x_{i}</annotation><annotation encoding="application/x-llamapun" id="S2.p6.1.m1.1d">italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT</annotation></semantics></math> and <math alttext="y_{i}" class="ltx_Math" display="inline" id="S2.p6.2.m2.1"><semantics id="S2.p6.2.m2.1a"><msub id="S2.p6.2.m2.1.1" xref="S2.p6.2.m2.1.1.cmml"><mi id="S2.p6.2.m2.1.1.2" xref="S2.p6.2.m2.1.1.2.cmml">y</mi><mi id="S2.p6.2.m2.1.1.3" xref="S2.p6.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p6.2.m2.1b"><apply id="S2.p6.2.m2.1.1.cmml" xref="S2.p6.2.m2.1.1"><csymbol cd="ambiguous" id="S2.p6.2.m2.1.1.1.cmml" xref="S2.p6.2.m2.1.1">subscript</csymbol><ci id="S2.p6.2.m2.1.1.2.cmml" xref="S2.p6.2.m2.1.1.2">𝑦</ci><ci id="S2.p6.2.m2.1.1.3.cmml" xref="S2.p6.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p6.2.m2.1c">y_{i}</annotation><annotation encoding="application/x-llamapun" id="S2.p6.2.m2.1d">italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT</annotation></semantics></math> represent the feature and label pairs from the dataset <math alttext="\mathcal{D}" class="ltx_Math" display="inline" id="S2.p6.3.m3.1"><semantics id="S2.p6.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p6.3.m3.1.1" xref="S2.p6.3.m3.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S2.p6.3.m3.1b"><ci id="S2.p6.3.m3.1.1.cmml" xref="S2.p6.3.m3.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p6.3.m3.1c">\mathcal{D}</annotation><annotation encoding="application/x-llamapun" id="S2.p6.3.m3.1d">caligraphic_D</annotation></semantics></math> in a supervised learning scenario. Eq. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S2.E2" title="In 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">2</span></a> highlights that, before each update, the gradient must be computed and averaged across all data points. Each data point contributes a distinct gradient, and the average of these gradients determines the final update direction. This gradient averaging is fundamental to optimization and closely relates to the parameter averaging concept in Federated Averaging (FedAvg). FedAvg can thus be viewed as a generalization of traditional optimization methods, a connection that will be elaborated upon.</p> </div> <div class="ltx_para" id="S2.p7"> <p class="ltx_p" id="S2.p7.1">To address GD’s inefficiencies, SGD was introduced. Unlike GD, which updates after computing the gradient over the entire dataset, SGD updates the parameters after each individual data sample. This leads to faster updates and reduced memory usage, making it suitable for large-scale problems. However, this method can be unstable, as computing gradients from a single data point may introduce bias and slow convergence in terms of model performance, despite the higher computational efficiency.</p> </div> <figure class="ltx_figure" id="S2.F1"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="140" id="S2.F1.g1" src="x1.png" width="664"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Illustration of how different GD methods navigate the loss landscape</figcaption> </figure> <div class="ltx_para" id="S2.p8"> <p class="ltx_p" id="S2.p8.2">To balance the trade-offs between GD and SGD, mini-batch SGD was developed. It computes gradients using a batch of size <math alttext="B" class="ltx_Math" display="inline" id="S2.p8.1.m1.1"><semantics id="S2.p8.1.m1.1a"><mi id="S2.p8.1.m1.1.1" xref="S2.p8.1.m1.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.p8.1.m1.1b"><ci id="S2.p8.1.m1.1.1.cmml" xref="S2.p8.1.m1.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p8.1.m1.1c">B</annotation><annotation encoding="application/x-llamapun" id="S2.p8.1.m1.1d">italic_B</annotation></semantics></math> for each update, reducing SGD’s instability while improving GD’s efficiency. GD, SGD, and mini-batch SGD can be seen as special cases within a broader framework, where the batch size <math alttext="B" class="ltx_Math" display="inline" id="S2.p8.2.m2.1"><semantics id="S2.p8.2.m2.1a"><mi id="S2.p8.2.m2.1.1" xref="S2.p8.2.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.p8.2.m2.1b"><ci id="S2.p8.2.m2.1.1.cmml" xref="S2.p8.2.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p8.2.m2.1c">B</annotation><annotation encoding="application/x-llamapun" id="S2.p8.2.m2.1d">italic_B</annotation></semantics></math> ranges from 1 (SGD) to the entire dataset (GD), as shown in Algorithm <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:alg.sgd</span>, with their respective optimization paths illustrated in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S2.F1" title="Figure 1 ‣ 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">1</span></a>.</p> </div> <div class="ltx_para" id="S2.p9"> <p class="ltx_p" id="S2.p9.1">The concept of gradient averaging for training extends naturally to distributed and parallel computing, as gradient calculations are independent until aggregation. Scaling SGD process across multiple devices (e.g., GPUs or distributed systems) accelerates training and mitigates the limitations of centralized hardware. While there are various approaches to parallel SGD <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib3" title="">3</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib23" title="">23</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib4" title="">4</a>]</cite>, in this work, we focus on aggregating gradients from each device per batch of each client, as described in Algorithm <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#alg1" title="Algorithm 1 ‣ 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">1</span></a>. In this distributed learning approach, we introduce a parameter, <math alttext="K" class="ltx_Math" display="inline" id="S2.p9.1.m1.1"><semantics id="S2.p9.1.m1.1a"><mi id="S2.p9.1.m1.1.1" xref="S2.p9.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.p9.1.m1.1b"><ci id="S2.p9.1.m1.1.1.cmml" xref="S2.p9.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p9.1.m1.1c">K</annotation><annotation encoding="application/x-llamapun" id="S2.p9.1.m1.1d">italic_K</annotation></semantics></math>, representing the number of devices. Gradients are computed on each device, followed by averaging locally and then globally.</p> </div> <figure class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top" id="alg1"> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span class="ltx_text ltx_font_bold" id="alg1.2.1.1">Algorithm 1</span> </span> Parallel Stochastic Gradient Descent (Parallel SGD) </figcaption> <div class="ltx_listing ltx_listing" id="alg1.3"> <div class="ltx_listingline" id="alg1.l1"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg1.l1.1.1.1" style="font-size:80%;">1:</span></span> <span class="ltx_text ltx_font_bold" id="alg1.l1.2">Input:</span> <math alttext="\eta" class="ltx_Math" display="inline" id="alg1.l1.m1.1"><semantics id="alg1.l1.m1.1a"><mi id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><ci id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="alg1.l1.m1.1d">italic_η</annotation></semantics></math>, <math alttext="B" class="ltx_Math" display="inline" id="alg1.l1.m2.1"><semantics id="alg1.l1.m2.1a"><mi id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.1b"><ci id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.1c">B</annotation><annotation encoding="application/x-llamapun" id="alg1.l1.m2.1d">italic_B</annotation></semantics></math>, <math alttext="E" class="ltx_Math" display="inline" id="alg1.l1.m3.1"><semantics id="alg1.l1.m3.1a"><mi id="alg1.l1.m3.1.1" xref="alg1.l1.m3.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m3.1b"><ci id="alg1.l1.m3.1.1.cmml" xref="alg1.l1.m3.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m3.1c">E</annotation><annotation encoding="application/x-llamapun" id="alg1.l1.m3.1d">italic_E</annotation></semantics></math>, Number of devices <math alttext="K" class="ltx_Math" display="inline" id="alg1.l1.m4.1"><semantics id="alg1.l1.m4.1a"><mi id="alg1.l1.m4.1.1" xref="alg1.l1.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m4.1b"><ci id="alg1.l1.m4.1.1.cmml" xref="alg1.l1.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m4.1c">K</annotation><annotation encoding="application/x-llamapun" id="alg1.l1.m4.1d">italic_K</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg1.l2"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg1.l2.1.1.1" style="font-size:80%;">2:</span></span> <span class="ltx_text ltx_font_bold" id="alg1.l2.2">for</span> <math alttext="n=1" class="ltx_Math" display="inline" id="alg1.l2.m1.1"><semantics id="alg1.l2.m1.1a"><mrow id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml"><mi id="alg1.l2.m1.1.1.2" xref="alg1.l2.m1.1.1.2.cmml">n</mi><mo id="alg1.l2.m1.1.1.1" xref="alg1.l2.m1.1.1.1.cmml">=</mo><mn id="alg1.l2.m1.1.1.3" xref="alg1.l2.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1"><eq id="alg1.l2.m1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1"></eq><ci id="alg1.l2.m1.1.1.2.cmml" xref="alg1.l2.m1.1.1.2">𝑛</ci><cn id="alg1.l2.m1.1.1.3.cmml" type="integer" xref="alg1.l2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">n=1</annotation><annotation encoding="application/x-llamapun" id="alg1.l2.m1.1d">italic_n = 1</annotation></semantics></math> to <math alttext="E" class="ltx_Math" display="inline" id="alg1.l2.m2.1"><semantics id="alg1.l2.m2.1a"><mi id="alg1.l2.m2.1.1" xref="alg1.l2.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="alg1.l2.m2.1b"><ci id="alg1.l2.m2.1.1.cmml" xref="alg1.l2.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m2.1c">E</annotation><annotation encoding="application/x-llamapun" id="alg1.l2.m2.1d">italic_E</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg1.l2.3">do</span> </div> <div class="ltx_listingline" id="alg1.l3"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg1.l3.1.1.1" style="font-size:80%;">3:</span></span> <span class="ltx_text ltx_font_bold" id="alg1.l3.2">for</span> each device <math alttext="k=1" class="ltx_Math" display="inline" id="alg1.l3.m1.1"><semantics id="alg1.l3.m1.1a"><mrow id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml"><mi id="alg1.l3.m1.1.1.2" xref="alg1.l3.m1.1.1.2.cmml">k</mi><mo id="alg1.l3.m1.1.1.1" xref="alg1.l3.m1.1.1.1.cmml">=</mo><mn id="alg1.l3.m1.1.1.3" xref="alg1.l3.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><apply id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1"><eq id="alg1.l3.m1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1"></eq><ci id="alg1.l3.m1.1.1.2.cmml" xref="alg1.l3.m1.1.1.2">𝑘</ci><cn id="alg1.l3.m1.1.1.3.cmml" type="integer" xref="alg1.l3.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">k=1</annotation><annotation encoding="application/x-llamapun" id="alg1.l3.m1.1d">italic_k = 1</annotation></semantics></math> to <math alttext="K" class="ltx_Math" display="inline" id="alg1.l3.m2.1"><semantics id="alg1.l3.m2.1a"><mi id="alg1.l3.m2.1.1" xref="alg1.l3.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="alg1.l3.m2.1b"><ci id="alg1.l3.m2.1.1.cmml" xref="alg1.l3.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m2.1c">K</annotation><annotation encoding="application/x-llamapun" id="alg1.l3.m2.1d">italic_K</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg1.l3.3">in parallel</span> <span class="ltx_text ltx_font_bold" id="alg1.l3.4">do</span> </div> <div class="ltx_listingline" id="alg1.l4"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg1.l4.1.1.1" style="font-size:80%;">4:</span></span> Uniformly select a random mini-batch <math alttext="B_{k}" class="ltx_Math" display="inline" id="alg1.l4.m1.1"><semantics id="alg1.l4.m1.1a"><msub id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml"><mi id="alg1.l4.m1.1.1.2" xref="alg1.l4.m1.1.1.2.cmml">B</mi><mi id="alg1.l4.m1.1.1.3" xref="alg1.l4.m1.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><apply id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.1.cmml" xref="alg1.l4.m1.1.1">subscript</csymbol><ci id="alg1.l4.m1.1.1.2.cmml" xref="alg1.l4.m1.1.1.2">𝐵</ci><ci id="alg1.l4.m1.1.1.3.cmml" xref="alg1.l4.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">B_{k}</annotation><annotation encoding="application/x-llamapun" id="alg1.l4.m1.1d">italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> from <math alttext="\mathcal{D}_{k}" class="ltx_Math" display="inline" id="alg1.l4.m2.1"><semantics id="alg1.l4.m2.1a"><msub id="alg1.l4.m2.1.1" xref="alg1.l4.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l4.m2.1.1.2" xref="alg1.l4.m2.1.1.2.cmml">𝒟</mi><mi id="alg1.l4.m2.1.1.3" xref="alg1.l4.m2.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.l4.m2.1b"><apply id="alg1.l4.m2.1.1.cmml" xref="alg1.l4.m2.1.1"><csymbol cd="ambiguous" id="alg1.l4.m2.1.1.1.cmml" xref="alg1.l4.m2.1.1">subscript</csymbol><ci id="alg1.l4.m2.1.1.2.cmml" xref="alg1.l4.m2.1.1.2">𝒟</ci><ci id="alg1.l4.m2.1.1.3.cmml" xref="alg1.l4.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m2.1c">\mathcal{D}_{k}</annotation><annotation encoding="application/x-llamapun" id="alg1.l4.m2.1d">caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg1.l5"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg1.l5.1.1.1" style="font-size:80%;">5:</span></span> Compute gradient on device <math alttext="k" class="ltx_Math" display="inline" id="alg1.l5.m1.1"><semantics id="alg1.l5.m1.1a"><mi id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><ci id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">k</annotation><annotation encoding="application/x-llamapun" id="alg1.l5.m1.1d">italic_k</annotation></semantics></math>: <math alttext="g_{k}=\frac{1}{|B_{k}|}\sum_{i\in B_{k}}\nabla_{\theta}F(\theta;x_{i},y_{i})" class="ltx_Math" display="inline" id="alg1.l5.m2.4"><semantics id="alg1.l5.m2.4a"><mrow id="alg1.l5.m2.4.4" xref="alg1.l5.m2.4.4.cmml"><msub id="alg1.l5.m2.4.4.4" xref="alg1.l5.m2.4.4.4.cmml"><mi id="alg1.l5.m2.4.4.4.2" xref="alg1.l5.m2.4.4.4.2.cmml">g</mi><mi id="alg1.l5.m2.4.4.4.3" xref="alg1.l5.m2.4.4.4.3.cmml">k</mi></msub><mo id="alg1.l5.m2.4.4.3" xref="alg1.l5.m2.4.4.3.cmml">=</mo><mrow id="alg1.l5.m2.4.4.2" xref="alg1.l5.m2.4.4.2.cmml"><mfrac id="alg1.l5.m2.1.1" xref="alg1.l5.m2.1.1.cmml"><mn id="alg1.l5.m2.1.1.3" xref="alg1.l5.m2.1.1.3.cmml">1</mn><mrow id="alg1.l5.m2.1.1.1.1" xref="alg1.l5.m2.1.1.1.2.cmml"><mo id="alg1.l5.m2.1.1.1.1.2" stretchy="false" xref="alg1.l5.m2.1.1.1.2.1.cmml">|</mo><msub id="alg1.l5.m2.1.1.1.1.1" xref="alg1.l5.m2.1.1.1.1.1.cmml"><mi id="alg1.l5.m2.1.1.1.1.1.2" xref="alg1.l5.m2.1.1.1.1.1.2.cmml">B</mi><mi id="alg1.l5.m2.1.1.1.1.1.3" xref="alg1.l5.m2.1.1.1.1.1.3.cmml">k</mi></msub><mo id="alg1.l5.m2.1.1.1.1.3" stretchy="false" xref="alg1.l5.m2.1.1.1.2.1.cmml">|</mo></mrow></mfrac><mo id="alg1.l5.m2.4.4.2.3" xref="alg1.l5.m2.4.4.2.3.cmml">⁢</mo><mrow id="alg1.l5.m2.4.4.2.2" xref="alg1.l5.m2.4.4.2.2.cmml"><msub id="alg1.l5.m2.4.4.2.2.3" xref="alg1.l5.m2.4.4.2.2.3.cmml"><mo id="alg1.l5.m2.4.4.2.2.3.2" xref="alg1.l5.m2.4.4.2.2.3.2.cmml">∑</mo><mrow id="alg1.l5.m2.4.4.2.2.3.3" xref="alg1.l5.m2.4.4.2.2.3.3.cmml"><mi id="alg1.l5.m2.4.4.2.2.3.3.2" xref="alg1.l5.m2.4.4.2.2.3.3.2.cmml">i</mi><mo id="alg1.l5.m2.4.4.2.2.3.3.1" xref="alg1.l5.m2.4.4.2.2.3.3.1.cmml">∈</mo><msub id="alg1.l5.m2.4.4.2.2.3.3.3" xref="alg1.l5.m2.4.4.2.2.3.3.3.cmml"><mi id="alg1.l5.m2.4.4.2.2.3.3.3.2" xref="alg1.l5.m2.4.4.2.2.3.3.3.2.cmml">B</mi><mi id="alg1.l5.m2.4.4.2.2.3.3.3.3" xref="alg1.l5.m2.4.4.2.2.3.3.3.3.cmml">k</mi></msub></mrow></msub><mrow id="alg1.l5.m2.4.4.2.2.2" xref="alg1.l5.m2.4.4.2.2.2.cmml"><mrow id="alg1.l5.m2.4.4.2.2.2.4" xref="alg1.l5.m2.4.4.2.2.2.4.cmml"><msub id="alg1.l5.m2.4.4.2.2.2.4.1" xref="alg1.l5.m2.4.4.2.2.2.4.1.cmml"><mo id="alg1.l5.m2.4.4.2.2.2.4.1.2" rspace="0.167em" xref="alg1.l5.m2.4.4.2.2.2.4.1.2.cmml">∇</mo><mi id="alg1.l5.m2.4.4.2.2.2.4.1.3" xref="alg1.l5.m2.4.4.2.2.2.4.1.3.cmml">θ</mi></msub><mi id="alg1.l5.m2.4.4.2.2.2.4.2" xref="alg1.l5.m2.4.4.2.2.2.4.2.cmml">F</mi></mrow><mo id="alg1.l5.m2.4.4.2.2.2.3" xref="alg1.l5.m2.4.4.2.2.2.3.cmml">⁢</mo><mrow id="alg1.l5.m2.4.4.2.2.2.2.2" xref="alg1.l5.m2.4.4.2.2.2.2.3.cmml"><mo id="alg1.l5.m2.4.4.2.2.2.2.2.3" stretchy="false" xref="alg1.l5.m2.4.4.2.2.2.2.3.cmml">(</mo><mi id="alg1.l5.m2.2.2" xref="alg1.l5.m2.2.2.cmml">θ</mi><mo id="alg1.l5.m2.4.4.2.2.2.2.2.4" xref="alg1.l5.m2.4.4.2.2.2.2.3.cmml">;</mo><msub id="alg1.l5.m2.3.3.1.1.1.1.1.1" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.cmml"><mi id="alg1.l5.m2.3.3.1.1.1.1.1.1.2" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.2.cmml">x</mi><mi id="alg1.l5.m2.3.3.1.1.1.1.1.1.3" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="alg1.l5.m2.4.4.2.2.2.2.2.5" xref="alg1.l5.m2.4.4.2.2.2.2.3.cmml">,</mo><msub id="alg1.l5.m2.4.4.2.2.2.2.2.2" xref="alg1.l5.m2.4.4.2.2.2.2.2.2.cmml"><mi id="alg1.l5.m2.4.4.2.2.2.2.2.2.2" xref="alg1.l5.m2.4.4.2.2.2.2.2.2.2.cmml">y</mi><mi id="alg1.l5.m2.4.4.2.2.2.2.2.2.3" xref="alg1.l5.m2.4.4.2.2.2.2.2.2.3.cmml">i</mi></msub><mo id="alg1.l5.m2.4.4.2.2.2.2.2.6" stretchy="false" xref="alg1.l5.m2.4.4.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m2.4b"><apply id="alg1.l5.m2.4.4.cmml" xref="alg1.l5.m2.4.4"><eq id="alg1.l5.m2.4.4.3.cmml" xref="alg1.l5.m2.4.4.3"></eq><apply id="alg1.l5.m2.4.4.4.cmml" xref="alg1.l5.m2.4.4.4"><csymbol cd="ambiguous" id="alg1.l5.m2.4.4.4.1.cmml" xref="alg1.l5.m2.4.4.4">subscript</csymbol><ci id="alg1.l5.m2.4.4.4.2.cmml" xref="alg1.l5.m2.4.4.4.2">𝑔</ci><ci id="alg1.l5.m2.4.4.4.3.cmml" xref="alg1.l5.m2.4.4.4.3">𝑘</ci></apply><apply id="alg1.l5.m2.4.4.2.cmml" xref="alg1.l5.m2.4.4.2"><times id="alg1.l5.m2.4.4.2.3.cmml" xref="alg1.l5.m2.4.4.2.3"></times><apply id="alg1.l5.m2.1.1.cmml" xref="alg1.l5.m2.1.1"><divide id="alg1.l5.m2.1.1.2.cmml" xref="alg1.l5.m2.1.1"></divide><cn id="alg1.l5.m2.1.1.3.cmml" type="integer" xref="alg1.l5.m2.1.1.3">1</cn><apply id="alg1.l5.m2.1.1.1.2.cmml" xref="alg1.l5.m2.1.1.1.1"><abs id="alg1.l5.m2.1.1.1.2.1.cmml" xref="alg1.l5.m2.1.1.1.1.2"></abs><apply id="alg1.l5.m2.1.1.1.1.1.cmml" xref="alg1.l5.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m2.1.1.1.1.1.1.cmml" xref="alg1.l5.m2.1.1.1.1.1">subscript</csymbol><ci id="alg1.l5.m2.1.1.1.1.1.2.cmml" xref="alg1.l5.m2.1.1.1.1.1.2">𝐵</ci><ci id="alg1.l5.m2.1.1.1.1.1.3.cmml" xref="alg1.l5.m2.1.1.1.1.1.3">𝑘</ci></apply></apply></apply><apply id="alg1.l5.m2.4.4.2.2.cmml" xref="alg1.l5.m2.4.4.2.2"><apply id="alg1.l5.m2.4.4.2.2.3.cmml" xref="alg1.l5.m2.4.4.2.2.3"><csymbol cd="ambiguous" id="alg1.l5.m2.4.4.2.2.3.1.cmml" xref="alg1.l5.m2.4.4.2.2.3">subscript</csymbol><sum id="alg1.l5.m2.4.4.2.2.3.2.cmml" xref="alg1.l5.m2.4.4.2.2.3.2"></sum><apply id="alg1.l5.m2.4.4.2.2.3.3.cmml" xref="alg1.l5.m2.4.4.2.2.3.3"><in id="alg1.l5.m2.4.4.2.2.3.3.1.cmml" xref="alg1.l5.m2.4.4.2.2.3.3.1"></in><ci id="alg1.l5.m2.4.4.2.2.3.3.2.cmml" xref="alg1.l5.m2.4.4.2.2.3.3.2">𝑖</ci><apply id="alg1.l5.m2.4.4.2.2.3.3.3.cmml" xref="alg1.l5.m2.4.4.2.2.3.3.3"><csymbol cd="ambiguous" id="alg1.l5.m2.4.4.2.2.3.3.3.1.cmml" xref="alg1.l5.m2.4.4.2.2.3.3.3">subscript</csymbol><ci id="alg1.l5.m2.4.4.2.2.3.3.3.2.cmml" xref="alg1.l5.m2.4.4.2.2.3.3.3.2">𝐵</ci><ci id="alg1.l5.m2.4.4.2.2.3.3.3.3.cmml" xref="alg1.l5.m2.4.4.2.2.3.3.3.3">𝑘</ci></apply></apply></apply><apply id="alg1.l5.m2.4.4.2.2.2.cmml" xref="alg1.l5.m2.4.4.2.2.2"><times id="alg1.l5.m2.4.4.2.2.2.3.cmml" xref="alg1.l5.m2.4.4.2.2.2.3"></times><apply id="alg1.l5.m2.4.4.2.2.2.4.cmml" xref="alg1.l5.m2.4.4.2.2.2.4"><apply id="alg1.l5.m2.4.4.2.2.2.4.1.cmml" xref="alg1.l5.m2.4.4.2.2.2.4.1"><csymbol cd="ambiguous" id="alg1.l5.m2.4.4.2.2.2.4.1.1.cmml" xref="alg1.l5.m2.4.4.2.2.2.4.1">subscript</csymbol><ci id="alg1.l5.m2.4.4.2.2.2.4.1.2.cmml" xref="alg1.l5.m2.4.4.2.2.2.4.1.2">∇</ci><ci id="alg1.l5.m2.4.4.2.2.2.4.1.3.cmml" xref="alg1.l5.m2.4.4.2.2.2.4.1.3">𝜃</ci></apply><ci id="alg1.l5.m2.4.4.2.2.2.4.2.cmml" xref="alg1.l5.m2.4.4.2.2.2.4.2">𝐹</ci></apply><list id="alg1.l5.m2.4.4.2.2.2.2.3.cmml" xref="alg1.l5.m2.4.4.2.2.2.2.2"><ci id="alg1.l5.m2.2.2.cmml" xref="alg1.l5.m2.2.2">𝜃</ci><apply id="alg1.l5.m2.3.3.1.1.1.1.1.1.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l5.m2.3.3.1.1.1.1.1.1.2.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.2">𝑥</ci><ci id="alg1.l5.m2.3.3.1.1.1.1.1.1.3.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="alg1.l5.m2.4.4.2.2.2.2.2.2.cmml" xref="alg1.l5.m2.4.4.2.2.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l5.m2.4.4.2.2.2.2.2.2.1.cmml" xref="alg1.l5.m2.4.4.2.2.2.2.2.2">subscript</csymbol><ci id="alg1.l5.m2.4.4.2.2.2.2.2.2.2.cmml" xref="alg1.l5.m2.4.4.2.2.2.2.2.2.2">𝑦</ci><ci id="alg1.l5.m2.4.4.2.2.2.2.2.2.3.cmml" xref="alg1.l5.m2.4.4.2.2.2.2.2.2.3">𝑖</ci></apply></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m2.4c">g_{k}=\frac{1}{|B_{k}|}\sum_{i\in B_{k}}\nabla_{\theta}F(\theta;x_{i},y_{i})</annotation><annotation encoding="application/x-llamapun" id="alg1.l5.m2.4d">italic_g start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG | italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT | end_ARG ∑ start_POSTSUBSCRIPT italic_i ∈ italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∇ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT italic_F ( italic_θ ; italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg1.l6"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg1.l6.1.1.1" style="font-size:80%;">6:</span></span> Aggregate gradients: <math alttext="g=\frac{1}{K}\sum_{k=1}^{K}g_{k}" class="ltx_Math" display="inline" id="alg1.l6.m1.1"><semantics id="alg1.l6.m1.1a"><mrow id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml"><mi id="alg1.l6.m1.1.1.2" xref="alg1.l6.m1.1.1.2.cmml">g</mi><mo id="alg1.l6.m1.1.1.1" xref="alg1.l6.m1.1.1.1.cmml">=</mo><mrow id="alg1.l6.m1.1.1.3" xref="alg1.l6.m1.1.1.3.cmml"><mfrac id="alg1.l6.m1.1.1.3.2" xref="alg1.l6.m1.1.1.3.2.cmml"><mn id="alg1.l6.m1.1.1.3.2.2" xref="alg1.l6.m1.1.1.3.2.2.cmml">1</mn><mi id="alg1.l6.m1.1.1.3.2.3" xref="alg1.l6.m1.1.1.3.2.3.cmml">K</mi></mfrac><mo id="alg1.l6.m1.1.1.3.1" xref="alg1.l6.m1.1.1.3.1.cmml">⁢</mo><mrow id="alg1.l6.m1.1.1.3.3" xref="alg1.l6.m1.1.1.3.3.cmml"><msubsup id="alg1.l6.m1.1.1.3.3.1" xref="alg1.l6.m1.1.1.3.3.1.cmml"><mo id="alg1.l6.m1.1.1.3.3.1.2.2" xref="alg1.l6.m1.1.1.3.3.1.2.2.cmml">∑</mo><mrow id="alg1.l6.m1.1.1.3.3.1.2.3" xref="alg1.l6.m1.1.1.3.3.1.2.3.cmml"><mi id="alg1.l6.m1.1.1.3.3.1.2.3.2" xref="alg1.l6.m1.1.1.3.3.1.2.3.2.cmml">k</mi><mo id="alg1.l6.m1.1.1.3.3.1.2.3.1" xref="alg1.l6.m1.1.1.3.3.1.2.3.1.cmml">=</mo><mn id="alg1.l6.m1.1.1.3.3.1.2.3.3" xref="alg1.l6.m1.1.1.3.3.1.2.3.3.cmml">1</mn></mrow><mi id="alg1.l6.m1.1.1.3.3.1.3" xref="alg1.l6.m1.1.1.3.3.1.3.cmml">K</mi></msubsup><msub id="alg1.l6.m1.1.1.3.3.2" xref="alg1.l6.m1.1.1.3.3.2.cmml"><mi id="alg1.l6.m1.1.1.3.3.2.2" xref="alg1.l6.m1.1.1.3.3.2.2.cmml">g</mi><mi id="alg1.l6.m1.1.1.3.3.2.3" xref="alg1.l6.m1.1.1.3.3.2.3.cmml">k</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><apply id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1"><eq id="alg1.l6.m1.1.1.1.cmml" xref="alg1.l6.m1.1.1.1"></eq><ci id="alg1.l6.m1.1.1.2.cmml" xref="alg1.l6.m1.1.1.2">𝑔</ci><apply id="alg1.l6.m1.1.1.3.cmml" xref="alg1.l6.m1.1.1.3"><times id="alg1.l6.m1.1.1.3.1.cmml" xref="alg1.l6.m1.1.1.3.1"></times><apply id="alg1.l6.m1.1.1.3.2.cmml" xref="alg1.l6.m1.1.1.3.2"><divide id="alg1.l6.m1.1.1.3.2.1.cmml" xref="alg1.l6.m1.1.1.3.2"></divide><cn id="alg1.l6.m1.1.1.3.2.2.cmml" type="integer" xref="alg1.l6.m1.1.1.3.2.2">1</cn><ci id="alg1.l6.m1.1.1.3.2.3.cmml" xref="alg1.l6.m1.1.1.3.2.3">𝐾</ci></apply><apply id="alg1.l6.m1.1.1.3.3.cmml" xref="alg1.l6.m1.1.1.3.3"><apply id="alg1.l6.m1.1.1.3.3.1.cmml" xref="alg1.l6.m1.1.1.3.3.1"><csymbol cd="ambiguous" id="alg1.l6.m1.1.1.3.3.1.1.cmml" xref="alg1.l6.m1.1.1.3.3.1">superscript</csymbol><apply id="alg1.l6.m1.1.1.3.3.1.2.cmml" xref="alg1.l6.m1.1.1.3.3.1"><csymbol cd="ambiguous" id="alg1.l6.m1.1.1.3.3.1.2.1.cmml" xref="alg1.l6.m1.1.1.3.3.1">subscript</csymbol><sum id="alg1.l6.m1.1.1.3.3.1.2.2.cmml" xref="alg1.l6.m1.1.1.3.3.1.2.2"></sum><apply id="alg1.l6.m1.1.1.3.3.1.2.3.cmml" xref="alg1.l6.m1.1.1.3.3.1.2.3"><eq id="alg1.l6.m1.1.1.3.3.1.2.3.1.cmml" xref="alg1.l6.m1.1.1.3.3.1.2.3.1"></eq><ci id="alg1.l6.m1.1.1.3.3.1.2.3.2.cmml" xref="alg1.l6.m1.1.1.3.3.1.2.3.2">𝑘</ci><cn id="alg1.l6.m1.1.1.3.3.1.2.3.3.cmml" type="integer" xref="alg1.l6.m1.1.1.3.3.1.2.3.3">1</cn></apply></apply><ci id="alg1.l6.m1.1.1.3.3.1.3.cmml" xref="alg1.l6.m1.1.1.3.3.1.3">𝐾</ci></apply><apply id="alg1.l6.m1.1.1.3.3.2.cmml" xref="alg1.l6.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="alg1.l6.m1.1.1.3.3.2.1.cmml" xref="alg1.l6.m1.1.1.3.3.2">subscript</csymbol><ci id="alg1.l6.m1.1.1.3.3.2.2.cmml" xref="alg1.l6.m1.1.1.3.3.2.2">𝑔</ci><ci id="alg1.l6.m1.1.1.3.3.2.3.cmml" xref="alg1.l6.m1.1.1.3.3.2.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">g=\frac{1}{K}\sum_{k=1}^{K}g_{k}</annotation><annotation encoding="application/x-llamapun" id="alg1.l6.m1.1d">italic_g = divide start_ARG 1 end_ARG start_ARG italic_K end_ARG ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_g start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg1.l7"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg1.l7.1.1.1" style="font-size:80%;">7:</span></span> Update parameters: <math alttext="\theta\leftarrow\theta-\eta\cdot g" class="ltx_Math" display="inline" id="alg1.l7.m1.1"><semantics id="alg1.l7.m1.1a"><mrow id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1.cmml"><mi id="alg1.l7.m1.1.1.2" xref="alg1.l7.m1.1.1.2.cmml">θ</mi><mo id="alg1.l7.m1.1.1.1" stretchy="false" xref="alg1.l7.m1.1.1.1.cmml">←</mo><mrow id="alg1.l7.m1.1.1.3" xref="alg1.l7.m1.1.1.3.cmml"><mi id="alg1.l7.m1.1.1.3.2" xref="alg1.l7.m1.1.1.3.2.cmml">θ</mi><mo id="alg1.l7.m1.1.1.3.1" xref="alg1.l7.m1.1.1.3.1.cmml">−</mo><mrow id="alg1.l7.m1.1.1.3.3" xref="alg1.l7.m1.1.1.3.3.cmml"><mi id="alg1.l7.m1.1.1.3.3.2" xref="alg1.l7.m1.1.1.3.3.2.cmml">η</mi><mo id="alg1.l7.m1.1.1.3.3.1" lspace="0.222em" rspace="0.222em" xref="alg1.l7.m1.1.1.3.3.1.cmml">⋅</mo><mi id="alg1.l7.m1.1.1.3.3.3" xref="alg1.l7.m1.1.1.3.3.3.cmml">g</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.1b"><apply id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1"><ci id="alg1.l7.m1.1.1.1.cmml" xref="alg1.l7.m1.1.1.1">←</ci><ci id="alg1.l7.m1.1.1.2.cmml" xref="alg1.l7.m1.1.1.2">𝜃</ci><apply id="alg1.l7.m1.1.1.3.cmml" xref="alg1.l7.m1.1.1.3"><minus id="alg1.l7.m1.1.1.3.1.cmml" xref="alg1.l7.m1.1.1.3.1"></minus><ci id="alg1.l7.m1.1.1.3.2.cmml" xref="alg1.l7.m1.1.1.3.2">𝜃</ci><apply id="alg1.l7.m1.1.1.3.3.cmml" xref="alg1.l7.m1.1.1.3.3"><ci id="alg1.l7.m1.1.1.3.3.1.cmml" xref="alg1.l7.m1.1.1.3.3.1">⋅</ci><ci id="alg1.l7.m1.1.1.3.3.2.cmml" xref="alg1.l7.m1.1.1.3.3.2">𝜂</ci><ci id="alg1.l7.m1.1.1.3.3.3.cmml" xref="alg1.l7.m1.1.1.3.3.3">𝑔</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.1c">\theta\leftarrow\theta-\eta\cdot g</annotation><annotation encoding="application/x-llamapun" id="alg1.l7.m1.1d">italic_θ ← italic_θ - italic_η ⋅ italic_g</annotation></semantics></math> </div> </div> </figure> <div class="ltx_para" id="S2.p10"> <p class="ltx_p" id="S2.p10.1">Despite the potential benefits of distributed and parallel processing, communication bottlenecks can arise, particularly during gradient aggregation. While intra-device parallelism (e.g., multi-GPU operations) tends to be efficient, frequent gradient communication between devices introduces overhead, offsetting the advantages of parallelization <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib18" title="">18</a>]</cite>.</p> </div> <div class="ltx_para" id="S2.p11"> <p class="ltx_p" id="S2.p11.2">A natural approach to mitigate this issue is to reduce the communication frequency. Several studies have investigated methods where each device updates parameters independently for a set duration, merging gradients less frequently. In LocalSGD <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib19" title="">19</a>]</cite>, each device performs training for <math alttext="I" class="ltx_Math" display="inline" id="S2.p11.1.m1.1"><semantics id="S2.p11.1.m1.1a"><mi id="S2.p11.1.m1.1.1" xref="S2.p11.1.m1.1.1.cmml">I</mi><annotation-xml encoding="MathML-Content" id="S2.p11.1.m1.1b"><ci id="S2.p11.1.m1.1.1.cmml" xref="S2.p11.1.m1.1.1">𝐼</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p11.1.m1.1c">I</annotation><annotation encoding="application/x-llamapun" id="S2.p11.1.m1.1d">italic_I</annotation></semantics></math> local epochs, followed by gradient merging every <math alttext="I+1" class="ltx_Math" display="inline" id="S2.p11.2.m2.1"><semantics id="S2.p11.2.m2.1a"><mrow id="S2.p11.2.m2.1.1" xref="S2.p11.2.m2.1.1.cmml"><mi id="S2.p11.2.m2.1.1.2" xref="S2.p11.2.m2.1.1.2.cmml">I</mi><mo id="S2.p11.2.m2.1.1.1" xref="S2.p11.2.m2.1.1.1.cmml">+</mo><mn id="S2.p11.2.m2.1.1.3" xref="S2.p11.2.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.p11.2.m2.1b"><apply id="S2.p11.2.m2.1.1.cmml" xref="S2.p11.2.m2.1.1"><plus id="S2.p11.2.m2.1.1.1.cmml" xref="S2.p11.2.m2.1.1.1"></plus><ci id="S2.p11.2.m2.1.1.2.cmml" xref="S2.p11.2.m2.1.1.2">𝐼</ci><cn id="S2.p11.2.m2.1.1.3.cmml" type="integer" xref="S2.p11.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p11.2.m2.1c">I+1</annotation><annotation encoding="application/x-llamapun" id="S2.p11.2.m2.1d">italic_I + 1</annotation></semantics></math> epochs. The globally updated model is then used as the starting point for subsequent training, as described in Algorithm <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#alg2" title="Algorithm 2 ‣ 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">2</span></a>.</p> </div> <figure class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top" id="alg2"> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span class="ltx_text ltx_font_bold" id="alg2.2.1.1">Algorithm 2</span> </span> Local Stochastic Gradient Descent (Local SGD) </figcaption> <div class="ltx_listing ltx_listing" id="alg2.3"> <div class="ltx_listingline" id="alg2.l1"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l1.1.1.1" style="font-size:80%;">1:</span></span> <span class="ltx_text ltx_font_bold" id="alg2.l1.2">Input:</span> <math alttext="\eta" class="ltx_Math" display="inline" id="alg2.l1.m1.1"><semantics id="alg2.l1.m1.1a"><mi id="alg2.l1.m1.1.1" xref="alg2.l1.m1.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m1.1b"><ci id="alg2.l1.m1.1.1.cmml" xref="alg2.l1.m1.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m1.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="alg2.l1.m1.1d">italic_η</annotation></semantics></math>, <math alttext="B" class="ltx_Math" display="inline" id="alg2.l1.m2.1"><semantics id="alg2.l1.m2.1a"><mi id="alg2.l1.m2.1.1" xref="alg2.l1.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m2.1b"><ci id="alg2.l1.m2.1.1.cmml" xref="alg2.l1.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m2.1c">B</annotation><annotation encoding="application/x-llamapun" id="alg2.l1.m2.1d">italic_B</annotation></semantics></math>, <math alttext="E" class="ltx_Math" display="inline" id="alg2.l1.m3.1"><semantics id="alg2.l1.m3.1a"><mi id="alg2.l1.m3.1.1" xref="alg2.l1.m3.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m3.1b"><ci id="alg2.l1.m3.1.1.cmml" xref="alg2.l1.m3.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m3.1c">E</annotation><annotation encoding="application/x-llamapun" id="alg2.l1.m3.1d">italic_E</annotation></semantics></math>, <math alttext="K" class="ltx_Math" display="inline" id="alg2.l1.m4.1"><semantics id="alg2.l1.m4.1a"><mi id="alg2.l1.m4.1.1" xref="alg2.l1.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m4.1b"><ci id="alg2.l1.m4.1.1.cmml" xref="alg2.l1.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m4.1c">K</annotation><annotation encoding="application/x-llamapun" id="alg2.l1.m4.1d">italic_K</annotation></semantics></math>, Synchronization period <math alttext="I" class="ltx_Math" display="inline" id="alg2.l1.m5.1"><semantics id="alg2.l1.m5.1a"><mi id="alg2.l1.m5.1.1" xref="alg2.l1.m5.1.1.cmml">I</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m5.1b"><ci id="alg2.l1.m5.1.1.cmml" xref="alg2.l1.m5.1.1">𝐼</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m5.1c">I</annotation><annotation encoding="application/x-llamapun" id="alg2.l1.m5.1d">italic_I</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg2.l2"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l2.1.1.1" style="font-size:80%;">2:</span></span> <span class="ltx_text ltx_font_bold" id="alg2.l2.2">for</span> <math alttext="n=1" class="ltx_Math" display="inline" id="alg2.l2.m1.1"><semantics id="alg2.l2.m1.1a"><mrow id="alg2.l2.m1.1.1" xref="alg2.l2.m1.1.1.cmml"><mi id="alg2.l2.m1.1.1.2" xref="alg2.l2.m1.1.1.2.cmml">n</mi><mo id="alg2.l2.m1.1.1.1" xref="alg2.l2.m1.1.1.1.cmml">=</mo><mn id="alg2.l2.m1.1.1.3" xref="alg2.l2.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l2.m1.1b"><apply id="alg2.l2.m1.1.1.cmml" xref="alg2.l2.m1.1.1"><eq id="alg2.l2.m1.1.1.1.cmml" xref="alg2.l2.m1.1.1.1"></eq><ci id="alg2.l2.m1.1.1.2.cmml" xref="alg2.l2.m1.1.1.2">𝑛</ci><cn id="alg2.l2.m1.1.1.3.cmml" type="integer" xref="alg2.l2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m1.1c">n=1</annotation><annotation encoding="application/x-llamapun" id="alg2.l2.m1.1d">italic_n = 1</annotation></semantics></math> to <math alttext="E" class="ltx_Math" display="inline" id="alg2.l2.m2.1"><semantics id="alg2.l2.m2.1a"><mi id="alg2.l2.m2.1.1" xref="alg2.l2.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="alg2.l2.m2.1b"><ci id="alg2.l2.m2.1.1.cmml" xref="alg2.l2.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m2.1c">E</annotation><annotation encoding="application/x-llamapun" id="alg2.l2.m2.1d">italic_E</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg2.l2.3">do</span> </div> <div class="ltx_listingline" id="alg2.l3"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l3.1.1.1" style="font-size:80%;">3:</span></span> <span class="ltx_text ltx_font_bold" id="alg2.l3.2">for</span> each device <math alttext="k=1" class="ltx_Math" display="inline" id="alg2.l3.m1.1"><semantics id="alg2.l3.m1.1a"><mrow id="alg2.l3.m1.1.1" xref="alg2.l3.m1.1.1.cmml"><mi id="alg2.l3.m1.1.1.2" xref="alg2.l3.m1.1.1.2.cmml">k</mi><mo id="alg2.l3.m1.1.1.1" xref="alg2.l3.m1.1.1.1.cmml">=</mo><mn id="alg2.l3.m1.1.1.3" xref="alg2.l3.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l3.m1.1b"><apply id="alg2.l3.m1.1.1.cmml" xref="alg2.l3.m1.1.1"><eq id="alg2.l3.m1.1.1.1.cmml" xref="alg2.l3.m1.1.1.1"></eq><ci id="alg2.l3.m1.1.1.2.cmml" xref="alg2.l3.m1.1.1.2">𝑘</ci><cn id="alg2.l3.m1.1.1.3.cmml" type="integer" xref="alg2.l3.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l3.m1.1c">k=1</annotation><annotation encoding="application/x-llamapun" id="alg2.l3.m1.1d">italic_k = 1</annotation></semantics></math> to <math alttext="K" class="ltx_Math" display="inline" id="alg2.l3.m2.1"><semantics id="alg2.l3.m2.1a"><mi id="alg2.l3.m2.1.1" xref="alg2.l3.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="alg2.l3.m2.1b"><ci id="alg2.l3.m2.1.1.cmml" xref="alg2.l3.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l3.m2.1c">K</annotation><annotation encoding="application/x-llamapun" id="alg2.l3.m2.1d">italic_K</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg2.l3.3">in parallel</span> <span class="ltx_text ltx_font_bold" id="alg2.l3.4">do</span> </div> <div class="ltx_listingline" id="alg2.l4"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l4.1.1.1" style="font-size:80%;">4:</span></span> Initialize local parameters: <math alttext="\theta_{k}\leftarrow\theta" class="ltx_Math" display="inline" id="alg2.l4.m1.1"><semantics id="alg2.l4.m1.1a"><mrow id="alg2.l4.m1.1.1" xref="alg2.l4.m1.1.1.cmml"><msub id="alg2.l4.m1.1.1.2" xref="alg2.l4.m1.1.1.2.cmml"><mi id="alg2.l4.m1.1.1.2.2" xref="alg2.l4.m1.1.1.2.2.cmml">θ</mi><mi id="alg2.l4.m1.1.1.2.3" xref="alg2.l4.m1.1.1.2.3.cmml">k</mi></msub><mo id="alg2.l4.m1.1.1.1" stretchy="false" xref="alg2.l4.m1.1.1.1.cmml">←</mo><mi id="alg2.l4.m1.1.1.3" xref="alg2.l4.m1.1.1.3.cmml">θ</mi></mrow><annotation-xml encoding="MathML-Content" id="alg2.l4.m1.1b"><apply id="alg2.l4.m1.1.1.cmml" xref="alg2.l4.m1.1.1"><ci id="alg2.l4.m1.1.1.1.cmml" xref="alg2.l4.m1.1.1.1">←</ci><apply id="alg2.l4.m1.1.1.2.cmml" xref="alg2.l4.m1.1.1.2"><csymbol cd="ambiguous" id="alg2.l4.m1.1.1.2.1.cmml" xref="alg2.l4.m1.1.1.2">subscript</csymbol><ci id="alg2.l4.m1.1.1.2.2.cmml" xref="alg2.l4.m1.1.1.2.2">𝜃</ci><ci id="alg2.l4.m1.1.1.2.3.cmml" xref="alg2.l4.m1.1.1.2.3">𝑘</ci></apply><ci id="alg2.l4.m1.1.1.3.cmml" xref="alg2.l4.m1.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.m1.1c">\theta_{k}\leftarrow\theta</annotation><annotation encoding="application/x-llamapun" id="alg2.l4.m1.1d">italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ← italic_θ</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg2.l5"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l5.1.1.1" style="font-size:80%;">5:</span></span> <span class="ltx_text ltx_font_bold" id="alg2.l5.2">for</span> <math alttext="t=1" class="ltx_Math" display="inline" id="alg2.l5.m1.1"><semantics id="alg2.l5.m1.1a"><mrow id="alg2.l5.m1.1.1" xref="alg2.l5.m1.1.1.cmml"><mi id="alg2.l5.m1.1.1.2" xref="alg2.l5.m1.1.1.2.cmml">t</mi><mo id="alg2.l5.m1.1.1.1" xref="alg2.l5.m1.1.1.1.cmml">=</mo><mn id="alg2.l5.m1.1.1.3" xref="alg2.l5.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l5.m1.1b"><apply id="alg2.l5.m1.1.1.cmml" xref="alg2.l5.m1.1.1"><eq id="alg2.l5.m1.1.1.1.cmml" xref="alg2.l5.m1.1.1.1"></eq><ci id="alg2.l5.m1.1.1.2.cmml" xref="alg2.l5.m1.1.1.2">𝑡</ci><cn id="alg2.l5.m1.1.1.3.cmml" type="integer" xref="alg2.l5.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m1.1c">t=1</annotation><annotation encoding="application/x-llamapun" id="alg2.l5.m1.1d">italic_t = 1</annotation></semantics></math> to <math alttext="I" class="ltx_Math" display="inline" id="alg2.l5.m2.1"><semantics id="alg2.l5.m2.1a"><mi id="alg2.l5.m2.1.1" xref="alg2.l5.m2.1.1.cmml">I</mi><annotation-xml encoding="MathML-Content" id="alg2.l5.m2.1b"><ci id="alg2.l5.m2.1.1.cmml" xref="alg2.l5.m2.1.1">𝐼</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m2.1c">I</annotation><annotation encoding="application/x-llamapun" id="alg2.l5.m2.1d">italic_I</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg2.l5.3">do</span> </div> <div class="ltx_listingline" id="alg2.l6"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l6.1.1.1" style="font-size:80%;">6:</span></span> Uniformly select a random mini-batch <math alttext="B_{k}^{t}" class="ltx_Math" display="inline" id="alg2.l6.m1.1"><semantics id="alg2.l6.m1.1a"><msubsup id="alg2.l6.m1.1.1" xref="alg2.l6.m1.1.1.cmml"><mi id="alg2.l6.m1.1.1.2.2" xref="alg2.l6.m1.1.1.2.2.cmml">B</mi><mi id="alg2.l6.m1.1.1.2.3" xref="alg2.l6.m1.1.1.2.3.cmml">k</mi><mi id="alg2.l6.m1.1.1.3" xref="alg2.l6.m1.1.1.3.cmml">t</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg2.l6.m1.1b"><apply id="alg2.l6.m1.1.1.cmml" xref="alg2.l6.m1.1.1"><csymbol cd="ambiguous" id="alg2.l6.m1.1.1.1.cmml" xref="alg2.l6.m1.1.1">superscript</csymbol><apply id="alg2.l6.m1.1.1.2.cmml" xref="alg2.l6.m1.1.1"><csymbol cd="ambiguous" id="alg2.l6.m1.1.1.2.1.cmml" xref="alg2.l6.m1.1.1">subscript</csymbol><ci id="alg2.l6.m1.1.1.2.2.cmml" xref="alg2.l6.m1.1.1.2.2">𝐵</ci><ci id="alg2.l6.m1.1.1.2.3.cmml" xref="alg2.l6.m1.1.1.2.3">𝑘</ci></apply><ci id="alg2.l6.m1.1.1.3.cmml" xref="alg2.l6.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m1.1c">B_{k}^{t}</annotation><annotation encoding="application/x-llamapun" id="alg2.l6.m1.1d">italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT</annotation></semantics></math> from <math alttext="\mathcal{D}_{k}" class="ltx_Math" display="inline" id="alg2.l6.m2.1"><semantics id="alg2.l6.m2.1a"><msub id="alg2.l6.m2.1.1" xref="alg2.l6.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l6.m2.1.1.2" xref="alg2.l6.m2.1.1.2.cmml">𝒟</mi><mi id="alg2.l6.m2.1.1.3" xref="alg2.l6.m2.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="alg2.l6.m2.1b"><apply id="alg2.l6.m2.1.1.cmml" xref="alg2.l6.m2.1.1"><csymbol cd="ambiguous" id="alg2.l6.m2.1.1.1.cmml" xref="alg2.l6.m2.1.1">subscript</csymbol><ci id="alg2.l6.m2.1.1.2.cmml" xref="alg2.l6.m2.1.1.2">𝒟</ci><ci id="alg2.l6.m2.1.1.3.cmml" xref="alg2.l6.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m2.1c">\mathcal{D}_{k}</annotation><annotation encoding="application/x-llamapun" id="alg2.l6.m2.1d">caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg2.l7"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l7.1.1.1" style="font-size:80%;">7:</span></span> Compute gradient on device <math alttext="k" class="ltx_Math" display="inline" id="alg2.l7.m1.1"><semantics id="alg2.l7.m1.1a"><mi id="alg2.l7.m1.1.1" xref="alg2.l7.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="alg2.l7.m1.1b"><ci id="alg2.l7.m1.1.1.cmml" xref="alg2.l7.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l7.m1.1c">k</annotation><annotation encoding="application/x-llamapun" id="alg2.l7.m1.1d">italic_k</annotation></semantics></math>: <math alttext="g_{k}=\frac{1}{|B_{k}^{t}|}\sum_{i\in B_{k}^{t}}\nabla_{\theta}F(\theta_{k};x_% {i},y_{i})" class="ltx_Math" display="inline" id="alg2.l7.m2.4"><semantics id="alg2.l7.m2.4a"><mrow id="alg2.l7.m2.4.4" xref="alg2.l7.m2.4.4.cmml"><msub id="alg2.l7.m2.4.4.5" xref="alg2.l7.m2.4.4.5.cmml"><mi id="alg2.l7.m2.4.4.5.2" xref="alg2.l7.m2.4.4.5.2.cmml">g</mi><mi id="alg2.l7.m2.4.4.5.3" xref="alg2.l7.m2.4.4.5.3.cmml">k</mi></msub><mo id="alg2.l7.m2.4.4.4" xref="alg2.l7.m2.4.4.4.cmml">=</mo><mrow id="alg2.l7.m2.4.4.3" xref="alg2.l7.m2.4.4.3.cmml"><mfrac id="alg2.l7.m2.1.1" xref="alg2.l7.m2.1.1.cmml"><mn id="alg2.l7.m2.1.1.3" xref="alg2.l7.m2.1.1.3.cmml">1</mn><mrow id="alg2.l7.m2.1.1.1.1" xref="alg2.l7.m2.1.1.1.2.cmml"><mo id="alg2.l7.m2.1.1.1.1.2" stretchy="false" xref="alg2.l7.m2.1.1.1.2.1.cmml">|</mo><msubsup id="alg2.l7.m2.1.1.1.1.1" xref="alg2.l7.m2.1.1.1.1.1.cmml"><mi id="alg2.l7.m2.1.1.1.1.1.2.2" xref="alg2.l7.m2.1.1.1.1.1.2.2.cmml">B</mi><mi id="alg2.l7.m2.1.1.1.1.1.2.3" xref="alg2.l7.m2.1.1.1.1.1.2.3.cmml">k</mi><mi id="alg2.l7.m2.1.1.1.1.1.3" xref="alg2.l7.m2.1.1.1.1.1.3.cmml">t</mi></msubsup><mo id="alg2.l7.m2.1.1.1.1.3" stretchy="false" xref="alg2.l7.m2.1.1.1.2.1.cmml">|</mo></mrow></mfrac><mo id="alg2.l7.m2.4.4.3.4" xref="alg2.l7.m2.4.4.3.4.cmml">⁢</mo><mrow id="alg2.l7.m2.4.4.3.3" xref="alg2.l7.m2.4.4.3.3.cmml"><msub id="alg2.l7.m2.4.4.3.3.4" xref="alg2.l7.m2.4.4.3.3.4.cmml"><mo id="alg2.l7.m2.4.4.3.3.4.2" xref="alg2.l7.m2.4.4.3.3.4.2.cmml">∑</mo><mrow id="alg2.l7.m2.4.4.3.3.4.3" xref="alg2.l7.m2.4.4.3.3.4.3.cmml"><mi id="alg2.l7.m2.4.4.3.3.4.3.2" xref="alg2.l7.m2.4.4.3.3.4.3.2.cmml">i</mi><mo id="alg2.l7.m2.4.4.3.3.4.3.1" xref="alg2.l7.m2.4.4.3.3.4.3.1.cmml">∈</mo><msubsup id="alg2.l7.m2.4.4.3.3.4.3.3" xref="alg2.l7.m2.4.4.3.3.4.3.3.cmml"><mi id="alg2.l7.m2.4.4.3.3.4.3.3.2.2" xref="alg2.l7.m2.4.4.3.3.4.3.3.2.2.cmml">B</mi><mi id="alg2.l7.m2.4.4.3.3.4.3.3.2.3" xref="alg2.l7.m2.4.4.3.3.4.3.3.2.3.cmml">k</mi><mi id="alg2.l7.m2.4.4.3.3.4.3.3.3" xref="alg2.l7.m2.4.4.3.3.4.3.3.3.cmml">t</mi></msubsup></mrow></msub><mrow id="alg2.l7.m2.4.4.3.3.3" xref="alg2.l7.m2.4.4.3.3.3.cmml"><mrow id="alg2.l7.m2.4.4.3.3.3.5" xref="alg2.l7.m2.4.4.3.3.3.5.cmml"><msub id="alg2.l7.m2.4.4.3.3.3.5.1" xref="alg2.l7.m2.4.4.3.3.3.5.1.cmml"><mo id="alg2.l7.m2.4.4.3.3.3.5.1.2" rspace="0.167em" xref="alg2.l7.m2.4.4.3.3.3.5.1.2.cmml">∇</mo><mi id="alg2.l7.m2.4.4.3.3.3.5.1.3" xref="alg2.l7.m2.4.4.3.3.3.5.1.3.cmml">θ</mi></msub><mi id="alg2.l7.m2.4.4.3.3.3.5.2" xref="alg2.l7.m2.4.4.3.3.3.5.2.cmml">F</mi></mrow><mo id="alg2.l7.m2.4.4.3.3.3.4" xref="alg2.l7.m2.4.4.3.3.3.4.cmml">⁢</mo><mrow id="alg2.l7.m2.4.4.3.3.3.3.3" xref="alg2.l7.m2.4.4.3.3.3.3.4.cmml"><mo id="alg2.l7.m2.4.4.3.3.3.3.3.4" stretchy="false" xref="alg2.l7.m2.4.4.3.3.3.3.4.cmml">(</mo><msub id="alg2.l7.m2.2.2.1.1.1.1.1.1" xref="alg2.l7.m2.2.2.1.1.1.1.1.1.cmml"><mi id="alg2.l7.m2.2.2.1.1.1.1.1.1.2" xref="alg2.l7.m2.2.2.1.1.1.1.1.1.2.cmml">θ</mi><mi id="alg2.l7.m2.2.2.1.1.1.1.1.1.3" xref="alg2.l7.m2.2.2.1.1.1.1.1.1.3.cmml">k</mi></msub><mo id="alg2.l7.m2.4.4.3.3.3.3.3.5" xref="alg2.l7.m2.4.4.3.3.3.3.4.cmml">;</mo><msub id="alg2.l7.m2.3.3.2.2.2.2.2.2" xref="alg2.l7.m2.3.3.2.2.2.2.2.2.cmml"><mi id="alg2.l7.m2.3.3.2.2.2.2.2.2.2" xref="alg2.l7.m2.3.3.2.2.2.2.2.2.2.cmml">x</mi><mi id="alg2.l7.m2.3.3.2.2.2.2.2.2.3" xref="alg2.l7.m2.3.3.2.2.2.2.2.2.3.cmml">i</mi></msub><mo id="alg2.l7.m2.4.4.3.3.3.3.3.6" xref="alg2.l7.m2.4.4.3.3.3.3.4.cmml">,</mo><msub id="alg2.l7.m2.4.4.3.3.3.3.3.3" xref="alg2.l7.m2.4.4.3.3.3.3.3.3.cmml"><mi id="alg2.l7.m2.4.4.3.3.3.3.3.3.2" xref="alg2.l7.m2.4.4.3.3.3.3.3.3.2.cmml">y</mi><mi id="alg2.l7.m2.4.4.3.3.3.3.3.3.3" xref="alg2.l7.m2.4.4.3.3.3.3.3.3.3.cmml">i</mi></msub><mo id="alg2.l7.m2.4.4.3.3.3.3.3.7" stretchy="false" xref="alg2.l7.m2.4.4.3.3.3.3.4.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l7.m2.4b"><apply id="alg2.l7.m2.4.4.cmml" xref="alg2.l7.m2.4.4"><eq id="alg2.l7.m2.4.4.4.cmml" xref="alg2.l7.m2.4.4.4"></eq><apply id="alg2.l7.m2.4.4.5.cmml" xref="alg2.l7.m2.4.4.5"><csymbol cd="ambiguous" id="alg2.l7.m2.4.4.5.1.cmml" xref="alg2.l7.m2.4.4.5">subscript</csymbol><ci id="alg2.l7.m2.4.4.5.2.cmml" xref="alg2.l7.m2.4.4.5.2">𝑔</ci><ci id="alg2.l7.m2.4.4.5.3.cmml" xref="alg2.l7.m2.4.4.5.3">𝑘</ci></apply><apply id="alg2.l7.m2.4.4.3.cmml" xref="alg2.l7.m2.4.4.3"><times id="alg2.l7.m2.4.4.3.4.cmml" xref="alg2.l7.m2.4.4.3.4"></times><apply id="alg2.l7.m2.1.1.cmml" xref="alg2.l7.m2.1.1"><divide id="alg2.l7.m2.1.1.2.cmml" xref="alg2.l7.m2.1.1"></divide><cn id="alg2.l7.m2.1.1.3.cmml" type="integer" xref="alg2.l7.m2.1.1.3">1</cn><apply id="alg2.l7.m2.1.1.1.2.cmml" xref="alg2.l7.m2.1.1.1.1"><abs id="alg2.l7.m2.1.1.1.2.1.cmml" xref="alg2.l7.m2.1.1.1.1.2"></abs><apply id="alg2.l7.m2.1.1.1.1.1.cmml" xref="alg2.l7.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l7.m2.1.1.1.1.1.1.cmml" xref="alg2.l7.m2.1.1.1.1.1">superscript</csymbol><apply id="alg2.l7.m2.1.1.1.1.1.2.cmml" xref="alg2.l7.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l7.m2.1.1.1.1.1.2.1.cmml" xref="alg2.l7.m2.1.1.1.1.1">subscript</csymbol><ci id="alg2.l7.m2.1.1.1.1.1.2.2.cmml" xref="alg2.l7.m2.1.1.1.1.1.2.2">𝐵</ci><ci id="alg2.l7.m2.1.1.1.1.1.2.3.cmml" xref="alg2.l7.m2.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="alg2.l7.m2.1.1.1.1.1.3.cmml" xref="alg2.l7.m2.1.1.1.1.1.3">𝑡</ci></apply></apply></apply><apply id="alg2.l7.m2.4.4.3.3.cmml" xref="alg2.l7.m2.4.4.3.3"><apply id="alg2.l7.m2.4.4.3.3.4.cmml" xref="alg2.l7.m2.4.4.3.3.4"><csymbol cd="ambiguous" id="alg2.l7.m2.4.4.3.3.4.1.cmml" xref="alg2.l7.m2.4.4.3.3.4">subscript</csymbol><sum id="alg2.l7.m2.4.4.3.3.4.2.cmml" xref="alg2.l7.m2.4.4.3.3.4.2"></sum><apply id="alg2.l7.m2.4.4.3.3.4.3.cmml" xref="alg2.l7.m2.4.4.3.3.4.3"><in id="alg2.l7.m2.4.4.3.3.4.3.1.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.1"></in><ci id="alg2.l7.m2.4.4.3.3.4.3.2.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.2">𝑖</ci><apply id="alg2.l7.m2.4.4.3.3.4.3.3.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.3"><csymbol cd="ambiguous" id="alg2.l7.m2.4.4.3.3.4.3.3.1.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.3">superscript</csymbol><apply id="alg2.l7.m2.4.4.3.3.4.3.3.2.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.3"><csymbol cd="ambiguous" id="alg2.l7.m2.4.4.3.3.4.3.3.2.1.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.3">subscript</csymbol><ci id="alg2.l7.m2.4.4.3.3.4.3.3.2.2.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.3.2.2">𝐵</ci><ci id="alg2.l7.m2.4.4.3.3.4.3.3.2.3.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.3.2.3">𝑘</ci></apply><ci id="alg2.l7.m2.4.4.3.3.4.3.3.3.cmml" xref="alg2.l7.m2.4.4.3.3.4.3.3.3">𝑡</ci></apply></apply></apply><apply id="alg2.l7.m2.4.4.3.3.3.cmml" xref="alg2.l7.m2.4.4.3.3.3"><times id="alg2.l7.m2.4.4.3.3.3.4.cmml" xref="alg2.l7.m2.4.4.3.3.3.4"></times><apply id="alg2.l7.m2.4.4.3.3.3.5.cmml" xref="alg2.l7.m2.4.4.3.3.3.5"><apply id="alg2.l7.m2.4.4.3.3.3.5.1.cmml" xref="alg2.l7.m2.4.4.3.3.3.5.1"><csymbol cd="ambiguous" id="alg2.l7.m2.4.4.3.3.3.5.1.1.cmml" xref="alg2.l7.m2.4.4.3.3.3.5.1">subscript</csymbol><ci id="alg2.l7.m2.4.4.3.3.3.5.1.2.cmml" xref="alg2.l7.m2.4.4.3.3.3.5.1.2">∇</ci><ci id="alg2.l7.m2.4.4.3.3.3.5.1.3.cmml" xref="alg2.l7.m2.4.4.3.3.3.5.1.3">𝜃</ci></apply><ci id="alg2.l7.m2.4.4.3.3.3.5.2.cmml" xref="alg2.l7.m2.4.4.3.3.3.5.2">𝐹</ci></apply><list id="alg2.l7.m2.4.4.3.3.3.3.4.cmml" xref="alg2.l7.m2.4.4.3.3.3.3.3"><apply id="alg2.l7.m2.2.2.1.1.1.1.1.1.cmml" xref="alg2.l7.m2.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l7.m2.2.2.1.1.1.1.1.1.1.cmml" xref="alg2.l7.m2.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="alg2.l7.m2.2.2.1.1.1.1.1.1.2.cmml" xref="alg2.l7.m2.2.2.1.1.1.1.1.1.2">𝜃</ci><ci id="alg2.l7.m2.2.2.1.1.1.1.1.1.3.cmml" xref="alg2.l7.m2.2.2.1.1.1.1.1.1.3">𝑘</ci></apply><apply id="alg2.l7.m2.3.3.2.2.2.2.2.2.cmml" xref="alg2.l7.m2.3.3.2.2.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l7.m2.3.3.2.2.2.2.2.2.1.cmml" xref="alg2.l7.m2.3.3.2.2.2.2.2.2">subscript</csymbol><ci id="alg2.l7.m2.3.3.2.2.2.2.2.2.2.cmml" xref="alg2.l7.m2.3.3.2.2.2.2.2.2.2">𝑥</ci><ci id="alg2.l7.m2.3.3.2.2.2.2.2.2.3.cmml" xref="alg2.l7.m2.3.3.2.2.2.2.2.2.3">𝑖</ci></apply><apply id="alg2.l7.m2.4.4.3.3.3.3.3.3.cmml" xref="alg2.l7.m2.4.4.3.3.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l7.m2.4.4.3.3.3.3.3.3.1.cmml" xref="alg2.l7.m2.4.4.3.3.3.3.3.3">subscript</csymbol><ci id="alg2.l7.m2.4.4.3.3.3.3.3.3.2.cmml" xref="alg2.l7.m2.4.4.3.3.3.3.3.3.2">𝑦</ci><ci id="alg2.l7.m2.4.4.3.3.3.3.3.3.3.cmml" xref="alg2.l7.m2.4.4.3.3.3.3.3.3.3">𝑖</ci></apply></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l7.m2.4c">g_{k}=\frac{1}{|B_{k}^{t}|}\sum_{i\in B_{k}^{t}}\nabla_{\theta}F(\theta_{k};x_% {i},y_{i})</annotation><annotation encoding="application/x-llamapun" id="alg2.l7.m2.4d">italic_g start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG | italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT | end_ARG ∑ start_POSTSUBSCRIPT italic_i ∈ italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT end_POSTSUBSCRIPT ∇ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT italic_F ( italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ; italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg2.l8"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l8.1.1.1" style="font-size:80%;">8:</span></span> Update local parameters: <math alttext="\theta_{k}\leftarrow\theta_{k}-\eta\cdot g_{k}" class="ltx_Math" display="inline" id="alg2.l8.m1.1"><semantics id="alg2.l8.m1.1a"><mrow id="alg2.l8.m1.1.1" xref="alg2.l8.m1.1.1.cmml"><msub id="alg2.l8.m1.1.1.2" xref="alg2.l8.m1.1.1.2.cmml"><mi id="alg2.l8.m1.1.1.2.2" xref="alg2.l8.m1.1.1.2.2.cmml">θ</mi><mi id="alg2.l8.m1.1.1.2.3" xref="alg2.l8.m1.1.1.2.3.cmml">k</mi></msub><mo id="alg2.l8.m1.1.1.1" stretchy="false" xref="alg2.l8.m1.1.1.1.cmml">←</mo><mrow id="alg2.l8.m1.1.1.3" xref="alg2.l8.m1.1.1.3.cmml"><msub id="alg2.l8.m1.1.1.3.2" xref="alg2.l8.m1.1.1.3.2.cmml"><mi id="alg2.l8.m1.1.1.3.2.2" xref="alg2.l8.m1.1.1.3.2.2.cmml">θ</mi><mi id="alg2.l8.m1.1.1.3.2.3" xref="alg2.l8.m1.1.1.3.2.3.cmml">k</mi></msub><mo id="alg2.l8.m1.1.1.3.1" xref="alg2.l8.m1.1.1.3.1.cmml">−</mo><mrow id="alg2.l8.m1.1.1.3.3" xref="alg2.l8.m1.1.1.3.3.cmml"><mi id="alg2.l8.m1.1.1.3.3.2" xref="alg2.l8.m1.1.1.3.3.2.cmml">η</mi><mo id="alg2.l8.m1.1.1.3.3.1" lspace="0.222em" rspace="0.222em" xref="alg2.l8.m1.1.1.3.3.1.cmml">⋅</mo><msub id="alg2.l8.m1.1.1.3.3.3" xref="alg2.l8.m1.1.1.3.3.3.cmml"><mi id="alg2.l8.m1.1.1.3.3.3.2" xref="alg2.l8.m1.1.1.3.3.3.2.cmml">g</mi><mi id="alg2.l8.m1.1.1.3.3.3.3" xref="alg2.l8.m1.1.1.3.3.3.3.cmml">k</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l8.m1.1b"><apply id="alg2.l8.m1.1.1.cmml" xref="alg2.l8.m1.1.1"><ci id="alg2.l8.m1.1.1.1.cmml" xref="alg2.l8.m1.1.1.1">←</ci><apply id="alg2.l8.m1.1.1.2.cmml" xref="alg2.l8.m1.1.1.2"><csymbol cd="ambiguous" id="alg2.l8.m1.1.1.2.1.cmml" xref="alg2.l8.m1.1.1.2">subscript</csymbol><ci id="alg2.l8.m1.1.1.2.2.cmml" xref="alg2.l8.m1.1.1.2.2">𝜃</ci><ci id="alg2.l8.m1.1.1.2.3.cmml" xref="alg2.l8.m1.1.1.2.3">𝑘</ci></apply><apply id="alg2.l8.m1.1.1.3.cmml" xref="alg2.l8.m1.1.1.3"><minus id="alg2.l8.m1.1.1.3.1.cmml" xref="alg2.l8.m1.1.1.3.1"></minus><apply id="alg2.l8.m1.1.1.3.2.cmml" xref="alg2.l8.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg2.l8.m1.1.1.3.2.1.cmml" xref="alg2.l8.m1.1.1.3.2">subscript</csymbol><ci id="alg2.l8.m1.1.1.3.2.2.cmml" xref="alg2.l8.m1.1.1.3.2.2">𝜃</ci><ci id="alg2.l8.m1.1.1.3.2.3.cmml" xref="alg2.l8.m1.1.1.3.2.3">𝑘</ci></apply><apply id="alg2.l8.m1.1.1.3.3.cmml" xref="alg2.l8.m1.1.1.3.3"><ci id="alg2.l8.m1.1.1.3.3.1.cmml" xref="alg2.l8.m1.1.1.3.3.1">⋅</ci><ci id="alg2.l8.m1.1.1.3.3.2.cmml" xref="alg2.l8.m1.1.1.3.3.2">𝜂</ci><apply id="alg2.l8.m1.1.1.3.3.3.cmml" xref="alg2.l8.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="alg2.l8.m1.1.1.3.3.3.1.cmml" xref="alg2.l8.m1.1.1.3.3.3">subscript</csymbol><ci id="alg2.l8.m1.1.1.3.3.3.2.cmml" xref="alg2.l8.m1.1.1.3.3.3.2">𝑔</ci><ci id="alg2.l8.m1.1.1.3.3.3.3.cmml" xref="alg2.l8.m1.1.1.3.3.3.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.m1.1c">\theta_{k}\leftarrow\theta_{k}-\eta\cdot g_{k}</annotation><annotation encoding="application/x-llamapun" id="alg2.l8.m1.1d">italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ← italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT - italic_η ⋅ italic_g start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg2.l9"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg2.l9.1.1.1" style="font-size:80%;">9:</span></span> Synchronize: Aggregate parameters across devices: <math alttext="\theta\leftarrow\frac{1}{K}\sum_{k=1}^{K}\theta_{k}" class="ltx_Math" display="inline" id="alg2.l9.m1.1"><semantics id="alg2.l9.m1.1a"><mrow id="alg2.l9.m1.1.1" xref="alg2.l9.m1.1.1.cmml"><mi id="alg2.l9.m1.1.1.2" xref="alg2.l9.m1.1.1.2.cmml">θ</mi><mo id="alg2.l9.m1.1.1.1" stretchy="false" xref="alg2.l9.m1.1.1.1.cmml">←</mo><mrow id="alg2.l9.m1.1.1.3" xref="alg2.l9.m1.1.1.3.cmml"><mfrac id="alg2.l9.m1.1.1.3.2" xref="alg2.l9.m1.1.1.3.2.cmml"><mn id="alg2.l9.m1.1.1.3.2.2" xref="alg2.l9.m1.1.1.3.2.2.cmml">1</mn><mi id="alg2.l9.m1.1.1.3.2.3" xref="alg2.l9.m1.1.1.3.2.3.cmml">K</mi></mfrac><mo id="alg2.l9.m1.1.1.3.1" xref="alg2.l9.m1.1.1.3.1.cmml">⁢</mo><mrow id="alg2.l9.m1.1.1.3.3" xref="alg2.l9.m1.1.1.3.3.cmml"><msubsup id="alg2.l9.m1.1.1.3.3.1" xref="alg2.l9.m1.1.1.3.3.1.cmml"><mo id="alg2.l9.m1.1.1.3.3.1.2.2" xref="alg2.l9.m1.1.1.3.3.1.2.2.cmml">∑</mo><mrow id="alg2.l9.m1.1.1.3.3.1.2.3" xref="alg2.l9.m1.1.1.3.3.1.2.3.cmml"><mi id="alg2.l9.m1.1.1.3.3.1.2.3.2" xref="alg2.l9.m1.1.1.3.3.1.2.3.2.cmml">k</mi><mo id="alg2.l9.m1.1.1.3.3.1.2.3.1" xref="alg2.l9.m1.1.1.3.3.1.2.3.1.cmml">=</mo><mn id="alg2.l9.m1.1.1.3.3.1.2.3.3" xref="alg2.l9.m1.1.1.3.3.1.2.3.3.cmml">1</mn></mrow><mi id="alg2.l9.m1.1.1.3.3.1.3" xref="alg2.l9.m1.1.1.3.3.1.3.cmml">K</mi></msubsup><msub id="alg2.l9.m1.1.1.3.3.2" xref="alg2.l9.m1.1.1.3.3.2.cmml"><mi id="alg2.l9.m1.1.1.3.3.2.2" xref="alg2.l9.m1.1.1.3.3.2.2.cmml">θ</mi><mi id="alg2.l9.m1.1.1.3.3.2.3" xref="alg2.l9.m1.1.1.3.3.2.3.cmml">k</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l9.m1.1b"><apply id="alg2.l9.m1.1.1.cmml" xref="alg2.l9.m1.1.1"><ci id="alg2.l9.m1.1.1.1.cmml" xref="alg2.l9.m1.1.1.1">←</ci><ci id="alg2.l9.m1.1.1.2.cmml" xref="alg2.l9.m1.1.1.2">𝜃</ci><apply id="alg2.l9.m1.1.1.3.cmml" xref="alg2.l9.m1.1.1.3"><times id="alg2.l9.m1.1.1.3.1.cmml" xref="alg2.l9.m1.1.1.3.1"></times><apply id="alg2.l9.m1.1.1.3.2.cmml" xref="alg2.l9.m1.1.1.3.2"><divide id="alg2.l9.m1.1.1.3.2.1.cmml" xref="alg2.l9.m1.1.1.3.2"></divide><cn id="alg2.l9.m1.1.1.3.2.2.cmml" type="integer" xref="alg2.l9.m1.1.1.3.2.2">1</cn><ci id="alg2.l9.m1.1.1.3.2.3.cmml" xref="alg2.l9.m1.1.1.3.2.3">𝐾</ci></apply><apply id="alg2.l9.m1.1.1.3.3.cmml" xref="alg2.l9.m1.1.1.3.3"><apply id="alg2.l9.m1.1.1.3.3.1.cmml" xref="alg2.l9.m1.1.1.3.3.1"><csymbol cd="ambiguous" id="alg2.l9.m1.1.1.3.3.1.1.cmml" xref="alg2.l9.m1.1.1.3.3.1">superscript</csymbol><apply id="alg2.l9.m1.1.1.3.3.1.2.cmml" xref="alg2.l9.m1.1.1.3.3.1"><csymbol cd="ambiguous" id="alg2.l9.m1.1.1.3.3.1.2.1.cmml" xref="alg2.l9.m1.1.1.3.3.1">subscript</csymbol><sum id="alg2.l9.m1.1.1.3.3.1.2.2.cmml" xref="alg2.l9.m1.1.1.3.3.1.2.2"></sum><apply id="alg2.l9.m1.1.1.3.3.1.2.3.cmml" xref="alg2.l9.m1.1.1.3.3.1.2.3"><eq id="alg2.l9.m1.1.1.3.3.1.2.3.1.cmml" xref="alg2.l9.m1.1.1.3.3.1.2.3.1"></eq><ci id="alg2.l9.m1.1.1.3.3.1.2.3.2.cmml" xref="alg2.l9.m1.1.1.3.3.1.2.3.2">𝑘</ci><cn id="alg2.l9.m1.1.1.3.3.1.2.3.3.cmml" type="integer" xref="alg2.l9.m1.1.1.3.3.1.2.3.3">1</cn></apply></apply><ci id="alg2.l9.m1.1.1.3.3.1.3.cmml" xref="alg2.l9.m1.1.1.3.3.1.3">𝐾</ci></apply><apply id="alg2.l9.m1.1.1.3.3.2.cmml" xref="alg2.l9.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="alg2.l9.m1.1.1.3.3.2.1.cmml" xref="alg2.l9.m1.1.1.3.3.2">subscript</csymbol><ci id="alg2.l9.m1.1.1.3.3.2.2.cmml" xref="alg2.l9.m1.1.1.3.3.2.2">𝜃</ci><ci id="alg2.l9.m1.1.1.3.3.2.3.cmml" xref="alg2.l9.m1.1.1.3.3.2.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l9.m1.1c">\theta\leftarrow\frac{1}{K}\sum_{k=1}^{K}\theta_{k}</annotation><annotation encoding="application/x-llamapun" id="alg2.l9.m1.1d">italic_θ ← divide start_ARG 1 end_ARG start_ARG italic_K end_ARG ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> </div> </div> </figure> <div class="ltx_para" id="S2.p12"> <p class="ltx_p" id="S2.p12.1">LocalSGD can be viewed as a special case of FedAvg, more aligned with distributed learning. It uses centrally managed, evenly partitioned IID data across all devices, reducing gradient variance and closely approximating centralized training performance. While sharing FedAvg’s core mechanics, LocalSGD assumes balanced IID data, making it more suited for distributed learning.</p> </div> <div class="ltx_para" id="S2.p13"> <p class="ltx_p" id="S2.p13.1">In contrast, FedAvg (Algorithm. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#alg3" title="Algorithm 3 ‣ 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">3</span></a>) assumes partial participation of client (Line <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#alg3.l3" title="In Algorithm 3 ‣ 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">3</span></a>) and non-IID data across clients, requiring adjustments in gradient aggregation. Client updates are weighted by dataset size, with larger datasets contributing more to the global model, as noted in line <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#alg3.l12" title="In Algorithm 3 ‣ 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">12</span></a>.</p> </div> <figure class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top" id="alg3"> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span class="ltx_text ltx_font_bold" id="alg3.2.1.1">Algorithm 3</span> </span> Federated Averaging (FedAvg)</figcaption> <div class="ltx_listing ltx_listing" id="alg3.3"> <div class="ltx_listingline" id="alg3.l1"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l1.1.1.1" style="font-size:80%;">1:</span></span> <span class="ltx_text ltx_font_bold" id="alg3.l1.2">Input:</span> Global <math alttext="\eta_{g}" class="ltx_Math" display="inline" id="alg3.l1.m1.1"><semantics id="alg3.l1.m1.1a"><msub id="alg3.l1.m1.1.1" xref="alg3.l1.m1.1.1.cmml"><mi id="alg3.l1.m1.1.1.2" xref="alg3.l1.m1.1.1.2.cmml">η</mi><mi id="alg3.l1.m1.1.1.3" xref="alg3.l1.m1.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l1.m1.1b"><apply id="alg3.l1.m1.1.1.cmml" xref="alg3.l1.m1.1.1"><csymbol cd="ambiguous" id="alg3.l1.m1.1.1.1.cmml" xref="alg3.l1.m1.1.1">subscript</csymbol><ci id="alg3.l1.m1.1.1.2.cmml" xref="alg3.l1.m1.1.1.2">𝜂</ci><ci id="alg3.l1.m1.1.1.3.cmml" xref="alg3.l1.m1.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m1.1c">\eta_{g}</annotation><annotation encoding="application/x-llamapun" id="alg3.l1.m1.1d">italic_η start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT</annotation></semantics></math>, Local <math alttext="\eta_{l}" class="ltx_Math" display="inline" id="alg3.l1.m2.1"><semantics id="alg3.l1.m2.1a"><msub id="alg3.l1.m2.1.1" xref="alg3.l1.m2.1.1.cmml"><mi id="alg3.l1.m2.1.1.2" xref="alg3.l1.m2.1.1.2.cmml">η</mi><mi id="alg3.l1.m2.1.1.3" xref="alg3.l1.m2.1.1.3.cmml">l</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l1.m2.1b"><apply id="alg3.l1.m2.1.1.cmml" xref="alg3.l1.m2.1.1"><csymbol cd="ambiguous" id="alg3.l1.m2.1.1.1.cmml" xref="alg3.l1.m2.1.1">subscript</csymbol><ci id="alg3.l1.m2.1.1.2.cmml" xref="alg3.l1.m2.1.1.2">𝜂</ci><ci id="alg3.l1.m2.1.1.3.cmml" xref="alg3.l1.m2.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m2.1c">\eta_{l}</annotation><annotation encoding="application/x-llamapun" id="alg3.l1.m2.1d">italic_η start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT</annotation></semantics></math>, <math alttext="B" class="ltx_Math" display="inline" id="alg3.l1.m3.1"><semantics id="alg3.l1.m3.1a"><mi id="alg3.l1.m3.1.1" xref="alg3.l1.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m3.1b"><ci id="alg3.l1.m3.1.1.cmml" xref="alg3.l1.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m3.1c">B</annotation><annotation encoding="application/x-llamapun" id="alg3.l1.m3.1d">italic_B</annotation></semantics></math>, <math alttext="E" class="ltx_Math" display="inline" id="alg3.l1.m4.1"><semantics id="alg3.l1.m4.1a"><mi id="alg3.l1.m4.1.1" xref="alg3.l1.m4.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m4.1b"><ci id="alg3.l1.m4.1.1.cmml" xref="alg3.l1.m4.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m4.1c">E</annotation><annotation encoding="application/x-llamapun" id="alg3.l1.m4.1d">italic_E</annotation></semantics></math>, <math alttext="K" class="ltx_Math" display="inline" id="alg3.l1.m5.1"><semantics id="alg3.l1.m5.1a"><mi id="alg3.l1.m5.1.1" xref="alg3.l1.m5.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m5.1b"><ci id="alg3.l1.m5.1.1.cmml" xref="alg3.l1.m5.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m5.1c">K</annotation><annotation encoding="application/x-llamapun" id="alg3.l1.m5.1d">italic_K</annotation></semantics></math>, rounds <math alttext="R" class="ltx_Math" display="inline" id="alg3.l1.m6.1"><semantics id="alg3.l1.m6.1a"><mi id="alg3.l1.m6.1.1" xref="alg3.l1.m6.1.1.cmml">R</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m6.1b"><ci id="alg3.l1.m6.1.1.cmml" xref="alg3.l1.m6.1.1">𝑅</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m6.1c">R</annotation><annotation encoding="application/x-llamapun" id="alg3.l1.m6.1d">italic_R</annotation></semantics></math>, client sampling ratio <math alttext="C" class="ltx_Math" display="inline" id="alg3.l1.m7.1"><semantics id="alg3.l1.m7.1a"><mi id="alg3.l1.m7.1.1" xref="alg3.l1.m7.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m7.1b"><ci id="alg3.l1.m7.1.1.cmml" xref="alg3.l1.m7.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m7.1c">C</annotation><annotation encoding="application/x-llamapun" id="alg3.l1.m7.1d">italic_C</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg3.l2"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l2.1.1.1" style="font-size:80%;">2:</span></span> <span class="ltx_text ltx_font_bold" id="alg3.l2.2">for</span> each round <math alttext="r=1" class="ltx_Math" display="inline" id="alg3.l2.m1.1"><semantics id="alg3.l2.m1.1a"><mrow id="alg3.l2.m1.1.1" xref="alg3.l2.m1.1.1.cmml"><mi id="alg3.l2.m1.1.1.2" xref="alg3.l2.m1.1.1.2.cmml">r</mi><mo id="alg3.l2.m1.1.1.1" xref="alg3.l2.m1.1.1.1.cmml">=</mo><mn id="alg3.l2.m1.1.1.3" xref="alg3.l2.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg3.l2.m1.1b"><apply id="alg3.l2.m1.1.1.cmml" xref="alg3.l2.m1.1.1"><eq id="alg3.l2.m1.1.1.1.cmml" xref="alg3.l2.m1.1.1.1"></eq><ci id="alg3.l2.m1.1.1.2.cmml" xref="alg3.l2.m1.1.1.2">𝑟</ci><cn id="alg3.l2.m1.1.1.3.cmml" type="integer" xref="alg3.l2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l2.m1.1c">r=1</annotation><annotation encoding="application/x-llamapun" id="alg3.l2.m1.1d">italic_r = 1</annotation></semantics></math> to <math alttext="R" class="ltx_Math" display="inline" id="alg3.l2.m2.1"><semantics id="alg3.l2.m2.1a"><mi id="alg3.l2.m2.1.1" xref="alg3.l2.m2.1.1.cmml">R</mi><annotation-xml encoding="MathML-Content" id="alg3.l2.m2.1b"><ci id="alg3.l2.m2.1.1.cmml" xref="alg3.l2.m2.1.1">𝑅</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l2.m2.1c">R</annotation><annotation encoding="application/x-llamapun" id="alg3.l2.m2.1d">italic_R</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg3.l2.3">do</span> </div> <div class="ltx_listingline" id="alg3.l3"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l3.1.1.1" style="font-size:80%;">3:</span></span> Randomly select a subset <math alttext="\mathcal{S}" class="ltx_Math" display="inline" id="alg3.l3.m1.1"><semantics id="alg3.l3.m1.1a"><mi class="ltx_font_mathcaligraphic" id="alg3.l3.m1.1.1" xref="alg3.l3.m1.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="alg3.l3.m1.1b"><ci id="alg3.l3.m1.1.1.cmml" xref="alg3.l3.m1.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l3.m1.1c">\mathcal{S}</annotation><annotation encoding="application/x-llamapun" id="alg3.l3.m1.1d">caligraphic_S</annotation></semantics></math> of <math alttext="\max(C\cdot K,1)" class="ltx_Math" display="inline" id="alg3.l3.m2.3"><semantics id="alg3.l3.m2.3a"><mrow id="alg3.l3.m2.3.3.1" xref="alg3.l3.m2.3.3.2.cmml"><mi id="alg3.l3.m2.1.1" xref="alg3.l3.m2.1.1.cmml">max</mi><mo id="alg3.l3.m2.3.3.1a" xref="alg3.l3.m2.3.3.2.cmml">⁡</mo><mrow id="alg3.l3.m2.3.3.1.1" xref="alg3.l3.m2.3.3.2.cmml"><mo id="alg3.l3.m2.3.3.1.1.2" stretchy="false" xref="alg3.l3.m2.3.3.2.cmml">(</mo><mrow id="alg3.l3.m2.3.3.1.1.1" xref="alg3.l3.m2.3.3.1.1.1.cmml"><mi id="alg3.l3.m2.3.3.1.1.1.2" xref="alg3.l3.m2.3.3.1.1.1.2.cmml">C</mi><mo id="alg3.l3.m2.3.3.1.1.1.1" lspace="0.222em" rspace="0.222em" xref="alg3.l3.m2.3.3.1.1.1.1.cmml">⋅</mo><mi id="alg3.l3.m2.3.3.1.1.1.3" xref="alg3.l3.m2.3.3.1.1.1.3.cmml">K</mi></mrow><mo id="alg3.l3.m2.3.3.1.1.3" xref="alg3.l3.m2.3.3.2.cmml">,</mo><mn id="alg3.l3.m2.2.2" xref="alg3.l3.m2.2.2.cmml">1</mn><mo id="alg3.l3.m2.3.3.1.1.4" stretchy="false" xref="alg3.l3.m2.3.3.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l3.m2.3b"><apply id="alg3.l3.m2.3.3.2.cmml" xref="alg3.l3.m2.3.3.1"><max id="alg3.l3.m2.1.1.cmml" xref="alg3.l3.m2.1.1"></max><apply id="alg3.l3.m2.3.3.1.1.1.cmml" xref="alg3.l3.m2.3.3.1.1.1"><ci id="alg3.l3.m2.3.3.1.1.1.1.cmml" xref="alg3.l3.m2.3.3.1.1.1.1">⋅</ci><ci id="alg3.l3.m2.3.3.1.1.1.2.cmml" xref="alg3.l3.m2.3.3.1.1.1.2">𝐶</ci><ci id="alg3.l3.m2.3.3.1.1.1.3.cmml" xref="alg3.l3.m2.3.3.1.1.1.3">𝐾</ci></apply><cn id="alg3.l3.m2.2.2.cmml" type="integer" xref="alg3.l3.m2.2.2">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l3.m2.3c">\max(C\cdot K,1)</annotation><annotation encoding="application/x-llamapun" id="alg3.l3.m2.3d">roman_max ( italic_C ⋅ italic_K , 1 )</annotation></semantics></math> clients </div> <div class="ltx_listingline" id="alg3.l4"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l4.1.1.1" style="font-size:80%;">4:</span></span> <span class="ltx_text ltx_font_bold" id="alg3.l4.2">for</span> each client <math alttext="k\in\mathcal{S}" class="ltx_Math" display="inline" id="alg3.l4.m1.1"><semantics id="alg3.l4.m1.1a"><mrow id="alg3.l4.m1.1.1" xref="alg3.l4.m1.1.1.cmml"><mi id="alg3.l4.m1.1.1.2" xref="alg3.l4.m1.1.1.2.cmml">k</mi><mo id="alg3.l4.m1.1.1.1" xref="alg3.l4.m1.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="alg3.l4.m1.1.1.3" xref="alg3.l4.m1.1.1.3.cmml">𝒮</mi></mrow><annotation-xml encoding="MathML-Content" id="alg3.l4.m1.1b"><apply id="alg3.l4.m1.1.1.cmml" xref="alg3.l4.m1.1.1"><in id="alg3.l4.m1.1.1.1.cmml" xref="alg3.l4.m1.1.1.1"></in><ci id="alg3.l4.m1.1.1.2.cmml" xref="alg3.l4.m1.1.1.2">𝑘</ci><ci id="alg3.l4.m1.1.1.3.cmml" xref="alg3.l4.m1.1.1.3">𝒮</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l4.m1.1c">k\in\mathcal{S}</annotation><annotation encoding="application/x-llamapun" id="alg3.l4.m1.1d">italic_k ∈ caligraphic_S</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg3.l4.3">in parallel</span> <span class="ltx_text ltx_font_bold" id="alg3.l4.4">do</span> </div> <div class="ltx_listingline" id="alg3.l5"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l5.1.1.1" style="font-size:80%;">5:</span></span> Initialize local model: <math alttext="\theta_{k}\leftarrow\theta" class="ltx_Math" display="inline" id="alg3.l5.m1.1"><semantics id="alg3.l5.m1.1a"><mrow id="alg3.l5.m1.1.1" xref="alg3.l5.m1.1.1.cmml"><msub id="alg3.l5.m1.1.1.2" xref="alg3.l5.m1.1.1.2.cmml"><mi id="alg3.l5.m1.1.1.2.2" xref="alg3.l5.m1.1.1.2.2.cmml">θ</mi><mi id="alg3.l5.m1.1.1.2.3" xref="alg3.l5.m1.1.1.2.3.cmml">k</mi></msub><mo id="alg3.l5.m1.1.1.1" stretchy="false" xref="alg3.l5.m1.1.1.1.cmml">←</mo><mi id="alg3.l5.m1.1.1.3" xref="alg3.l5.m1.1.1.3.cmml">θ</mi></mrow><annotation-xml encoding="MathML-Content" id="alg3.l5.m1.1b"><apply id="alg3.l5.m1.1.1.cmml" xref="alg3.l5.m1.1.1"><ci id="alg3.l5.m1.1.1.1.cmml" xref="alg3.l5.m1.1.1.1">←</ci><apply id="alg3.l5.m1.1.1.2.cmml" xref="alg3.l5.m1.1.1.2"><csymbol cd="ambiguous" id="alg3.l5.m1.1.1.2.1.cmml" xref="alg3.l5.m1.1.1.2">subscript</csymbol><ci id="alg3.l5.m1.1.1.2.2.cmml" xref="alg3.l5.m1.1.1.2.2">𝜃</ci><ci id="alg3.l5.m1.1.1.2.3.cmml" xref="alg3.l5.m1.1.1.2.3">𝑘</ci></apply><ci id="alg3.l5.m1.1.1.3.cmml" xref="alg3.l5.m1.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l5.m1.1c">\theta_{k}\leftarrow\theta</annotation><annotation encoding="application/x-llamapun" id="alg3.l5.m1.1d">italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ← italic_θ</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg3.l6"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l6.1.1.1" style="font-size:80%;">6:</span></span> <span class="ltx_text ltx_font_bold" id="alg3.l6.2">for</span> each local epoch <math alttext="n=1" class="ltx_Math" display="inline" id="alg3.l6.m1.1"><semantics id="alg3.l6.m1.1a"><mrow id="alg3.l6.m1.1.1" xref="alg3.l6.m1.1.1.cmml"><mi id="alg3.l6.m1.1.1.2" xref="alg3.l6.m1.1.1.2.cmml">n</mi><mo id="alg3.l6.m1.1.1.1" xref="alg3.l6.m1.1.1.1.cmml">=</mo><mn id="alg3.l6.m1.1.1.3" xref="alg3.l6.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg3.l6.m1.1b"><apply id="alg3.l6.m1.1.1.cmml" xref="alg3.l6.m1.1.1"><eq id="alg3.l6.m1.1.1.1.cmml" xref="alg3.l6.m1.1.1.1"></eq><ci id="alg3.l6.m1.1.1.2.cmml" xref="alg3.l6.m1.1.1.2">𝑛</ci><cn id="alg3.l6.m1.1.1.3.cmml" type="integer" xref="alg3.l6.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l6.m1.1c">n=1</annotation><annotation encoding="application/x-llamapun" id="alg3.l6.m1.1d">italic_n = 1</annotation></semantics></math> to <math alttext="E" class="ltx_Math" display="inline" id="alg3.l6.m2.1"><semantics id="alg3.l6.m2.1a"><mi id="alg3.l6.m2.1.1" xref="alg3.l6.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="alg3.l6.m2.1b"><ci id="alg3.l6.m2.1.1.cmml" xref="alg3.l6.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l6.m2.1c">E</annotation><annotation encoding="application/x-llamapun" id="alg3.l6.m2.1d">italic_E</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg3.l6.3">do</span> </div> <div class="ltx_listingline" id="alg3.l7"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l7.1.1.1" style="font-size:80%;">7:</span></span> <span class="ltx_text ltx_font_bold" id="alg3.l7.2">for</span> each mini-batch <math alttext="B_{k}" class="ltx_Math" display="inline" id="alg3.l7.m1.1"><semantics id="alg3.l7.m1.1a"><msub id="alg3.l7.m1.1.1" xref="alg3.l7.m1.1.1.cmml"><mi id="alg3.l7.m1.1.1.2" xref="alg3.l7.m1.1.1.2.cmml">B</mi><mi id="alg3.l7.m1.1.1.3" xref="alg3.l7.m1.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l7.m1.1b"><apply id="alg3.l7.m1.1.1.cmml" xref="alg3.l7.m1.1.1"><csymbol cd="ambiguous" id="alg3.l7.m1.1.1.1.cmml" xref="alg3.l7.m1.1.1">subscript</csymbol><ci id="alg3.l7.m1.1.1.2.cmml" xref="alg3.l7.m1.1.1.2">𝐵</ci><ci id="alg3.l7.m1.1.1.3.cmml" xref="alg3.l7.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l7.m1.1c">B_{k}</annotation><annotation encoding="application/x-llamapun" id="alg3.l7.m1.1d">italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> from local dataset <math alttext="\mathcal{D}_{k}" class="ltx_Math" display="inline" id="alg3.l7.m2.1"><semantics id="alg3.l7.m2.1a"><msub id="alg3.l7.m2.1.1" xref="alg3.l7.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg3.l7.m2.1.1.2" xref="alg3.l7.m2.1.1.2.cmml">𝒟</mi><mi id="alg3.l7.m2.1.1.3" xref="alg3.l7.m2.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l7.m2.1b"><apply id="alg3.l7.m2.1.1.cmml" xref="alg3.l7.m2.1.1"><csymbol cd="ambiguous" id="alg3.l7.m2.1.1.1.cmml" xref="alg3.l7.m2.1.1">subscript</csymbol><ci id="alg3.l7.m2.1.1.2.cmml" xref="alg3.l7.m2.1.1.2">𝒟</ci><ci id="alg3.l7.m2.1.1.3.cmml" xref="alg3.l7.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l7.m2.1c">\mathcal{D}_{k}</annotation><annotation encoding="application/x-llamapun" id="alg3.l7.m2.1d">caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> <span class="ltx_text ltx_font_bold" id="alg3.l7.3">do</span> </div> <div class="ltx_listingline" id="alg3.l8"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l8.1.1.1" style="font-size:80%;">8:</span></span> Compute gradient: <math alttext="g_{k}=\frac{1}{|B_{k}|}\sum_{(x_{i},y_{i})\in B_{k}}\nabla_{\theta}F(\theta_{k% };x_{i},y_{i})" class="ltx_Math" display="inline" id="alg3.l8.m1.6"><semantics id="alg3.l8.m1.6a"><mrow id="alg3.l8.m1.6.6" xref="alg3.l8.m1.6.6.cmml"><msub id="alg3.l8.m1.6.6.5" xref="alg3.l8.m1.6.6.5.cmml"><mi id="alg3.l8.m1.6.6.5.2" xref="alg3.l8.m1.6.6.5.2.cmml">g</mi><mi id="alg3.l8.m1.6.6.5.3" xref="alg3.l8.m1.6.6.5.3.cmml">k</mi></msub><mo id="alg3.l8.m1.6.6.4" xref="alg3.l8.m1.6.6.4.cmml">=</mo><mrow id="alg3.l8.m1.6.6.3" xref="alg3.l8.m1.6.6.3.cmml"><mfrac id="alg3.l8.m1.1.1" xref="alg3.l8.m1.1.1.cmml"><mn id="alg3.l8.m1.1.1.3" xref="alg3.l8.m1.1.1.3.cmml">1</mn><mrow id="alg3.l8.m1.1.1.1.1" xref="alg3.l8.m1.1.1.1.2.cmml"><mo id="alg3.l8.m1.1.1.1.1.2" stretchy="false" xref="alg3.l8.m1.1.1.1.2.1.cmml">|</mo><msub id="alg3.l8.m1.1.1.1.1.1" xref="alg3.l8.m1.1.1.1.1.1.cmml"><mi id="alg3.l8.m1.1.1.1.1.1.2" xref="alg3.l8.m1.1.1.1.1.1.2.cmml">B</mi><mi id="alg3.l8.m1.1.1.1.1.1.3" xref="alg3.l8.m1.1.1.1.1.1.3.cmml">k</mi></msub><mo id="alg3.l8.m1.1.1.1.1.3" stretchy="false" xref="alg3.l8.m1.1.1.1.2.1.cmml">|</mo></mrow></mfrac><mo id="alg3.l8.m1.6.6.3.4" xref="alg3.l8.m1.6.6.3.4.cmml">⁢</mo><mrow id="alg3.l8.m1.6.6.3.3" xref="alg3.l8.m1.6.6.3.3.cmml"><msub id="alg3.l8.m1.6.6.3.3.4" xref="alg3.l8.m1.6.6.3.3.4.cmml"><mo id="alg3.l8.m1.6.6.3.3.4.2" xref="alg3.l8.m1.6.6.3.3.4.2.cmml">∑</mo><mrow id="alg3.l8.m1.3.3.2" xref="alg3.l8.m1.3.3.2.cmml"><mrow id="alg3.l8.m1.3.3.2.2.2" xref="alg3.l8.m1.3.3.2.2.3.cmml"><mo id="alg3.l8.m1.3.3.2.2.2.3" stretchy="false" xref="alg3.l8.m1.3.3.2.2.3.cmml">(</mo><msub id="alg3.l8.m1.2.2.1.1.1.1" xref="alg3.l8.m1.2.2.1.1.1.1.cmml"><mi id="alg3.l8.m1.2.2.1.1.1.1.2" xref="alg3.l8.m1.2.2.1.1.1.1.2.cmml">x</mi><mi id="alg3.l8.m1.2.2.1.1.1.1.3" xref="alg3.l8.m1.2.2.1.1.1.1.3.cmml">i</mi></msub><mo id="alg3.l8.m1.3.3.2.2.2.4" xref="alg3.l8.m1.3.3.2.2.3.cmml">,</mo><msub id="alg3.l8.m1.3.3.2.2.2.2" xref="alg3.l8.m1.3.3.2.2.2.2.cmml"><mi id="alg3.l8.m1.3.3.2.2.2.2.2" xref="alg3.l8.m1.3.3.2.2.2.2.2.cmml">y</mi><mi id="alg3.l8.m1.3.3.2.2.2.2.3" xref="alg3.l8.m1.3.3.2.2.2.2.3.cmml">i</mi></msub><mo id="alg3.l8.m1.3.3.2.2.2.5" stretchy="false" xref="alg3.l8.m1.3.3.2.2.3.cmml">)</mo></mrow><mo id="alg3.l8.m1.3.3.2.3" xref="alg3.l8.m1.3.3.2.3.cmml">∈</mo><msub id="alg3.l8.m1.3.3.2.4" xref="alg3.l8.m1.3.3.2.4.cmml"><mi id="alg3.l8.m1.3.3.2.4.2" xref="alg3.l8.m1.3.3.2.4.2.cmml">B</mi><mi id="alg3.l8.m1.3.3.2.4.3" xref="alg3.l8.m1.3.3.2.4.3.cmml">k</mi></msub></mrow></msub><mrow id="alg3.l8.m1.6.6.3.3.3" xref="alg3.l8.m1.6.6.3.3.3.cmml"><mrow id="alg3.l8.m1.6.6.3.3.3.5" xref="alg3.l8.m1.6.6.3.3.3.5.cmml"><msub id="alg3.l8.m1.6.6.3.3.3.5.1" xref="alg3.l8.m1.6.6.3.3.3.5.1.cmml"><mo id="alg3.l8.m1.6.6.3.3.3.5.1.2" rspace="0.167em" xref="alg3.l8.m1.6.6.3.3.3.5.1.2.cmml">∇</mo><mi id="alg3.l8.m1.6.6.3.3.3.5.1.3" xref="alg3.l8.m1.6.6.3.3.3.5.1.3.cmml">θ</mi></msub><mi id="alg3.l8.m1.6.6.3.3.3.5.2" xref="alg3.l8.m1.6.6.3.3.3.5.2.cmml">F</mi></mrow><mo id="alg3.l8.m1.6.6.3.3.3.4" xref="alg3.l8.m1.6.6.3.3.3.4.cmml">⁢</mo><mrow id="alg3.l8.m1.6.6.3.3.3.3.3" xref="alg3.l8.m1.6.6.3.3.3.3.4.cmml"><mo id="alg3.l8.m1.6.6.3.3.3.3.3.4" stretchy="false" xref="alg3.l8.m1.6.6.3.3.3.3.4.cmml">(</mo><msub id="alg3.l8.m1.4.4.1.1.1.1.1.1" xref="alg3.l8.m1.4.4.1.1.1.1.1.1.cmml"><mi id="alg3.l8.m1.4.4.1.1.1.1.1.1.2" xref="alg3.l8.m1.4.4.1.1.1.1.1.1.2.cmml">θ</mi><mi id="alg3.l8.m1.4.4.1.1.1.1.1.1.3" xref="alg3.l8.m1.4.4.1.1.1.1.1.1.3.cmml">k</mi></msub><mo id="alg3.l8.m1.6.6.3.3.3.3.3.5" xref="alg3.l8.m1.6.6.3.3.3.3.4.cmml">;</mo><msub id="alg3.l8.m1.5.5.2.2.2.2.2.2" xref="alg3.l8.m1.5.5.2.2.2.2.2.2.cmml"><mi id="alg3.l8.m1.5.5.2.2.2.2.2.2.2" xref="alg3.l8.m1.5.5.2.2.2.2.2.2.2.cmml">x</mi><mi id="alg3.l8.m1.5.5.2.2.2.2.2.2.3" xref="alg3.l8.m1.5.5.2.2.2.2.2.2.3.cmml">i</mi></msub><mo id="alg3.l8.m1.6.6.3.3.3.3.3.6" xref="alg3.l8.m1.6.6.3.3.3.3.4.cmml">,</mo><msub id="alg3.l8.m1.6.6.3.3.3.3.3.3" xref="alg3.l8.m1.6.6.3.3.3.3.3.3.cmml"><mi id="alg3.l8.m1.6.6.3.3.3.3.3.3.2" xref="alg3.l8.m1.6.6.3.3.3.3.3.3.2.cmml">y</mi><mi id="alg3.l8.m1.6.6.3.3.3.3.3.3.3" xref="alg3.l8.m1.6.6.3.3.3.3.3.3.3.cmml">i</mi></msub><mo id="alg3.l8.m1.6.6.3.3.3.3.3.7" stretchy="false" xref="alg3.l8.m1.6.6.3.3.3.3.4.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l8.m1.6b"><apply id="alg3.l8.m1.6.6.cmml" xref="alg3.l8.m1.6.6"><eq id="alg3.l8.m1.6.6.4.cmml" xref="alg3.l8.m1.6.6.4"></eq><apply id="alg3.l8.m1.6.6.5.cmml" xref="alg3.l8.m1.6.6.5"><csymbol cd="ambiguous" id="alg3.l8.m1.6.6.5.1.cmml" xref="alg3.l8.m1.6.6.5">subscript</csymbol><ci id="alg3.l8.m1.6.6.5.2.cmml" xref="alg3.l8.m1.6.6.5.2">𝑔</ci><ci id="alg3.l8.m1.6.6.5.3.cmml" xref="alg3.l8.m1.6.6.5.3">𝑘</ci></apply><apply id="alg3.l8.m1.6.6.3.cmml" xref="alg3.l8.m1.6.6.3"><times id="alg3.l8.m1.6.6.3.4.cmml" xref="alg3.l8.m1.6.6.3.4"></times><apply id="alg3.l8.m1.1.1.cmml" xref="alg3.l8.m1.1.1"><divide id="alg3.l8.m1.1.1.2.cmml" xref="alg3.l8.m1.1.1"></divide><cn id="alg3.l8.m1.1.1.3.cmml" type="integer" xref="alg3.l8.m1.1.1.3">1</cn><apply id="alg3.l8.m1.1.1.1.2.cmml" xref="alg3.l8.m1.1.1.1.1"><abs id="alg3.l8.m1.1.1.1.2.1.cmml" xref="alg3.l8.m1.1.1.1.1.2"></abs><apply id="alg3.l8.m1.1.1.1.1.1.cmml" xref="alg3.l8.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l8.m1.1.1.1.1.1.1.cmml" xref="alg3.l8.m1.1.1.1.1.1">subscript</csymbol><ci id="alg3.l8.m1.1.1.1.1.1.2.cmml" xref="alg3.l8.m1.1.1.1.1.1.2">𝐵</ci><ci id="alg3.l8.m1.1.1.1.1.1.3.cmml" xref="alg3.l8.m1.1.1.1.1.1.3">𝑘</ci></apply></apply></apply><apply id="alg3.l8.m1.6.6.3.3.cmml" xref="alg3.l8.m1.6.6.3.3"><apply id="alg3.l8.m1.6.6.3.3.4.cmml" xref="alg3.l8.m1.6.6.3.3.4"><csymbol cd="ambiguous" id="alg3.l8.m1.6.6.3.3.4.1.cmml" xref="alg3.l8.m1.6.6.3.3.4">subscript</csymbol><sum id="alg3.l8.m1.6.6.3.3.4.2.cmml" xref="alg3.l8.m1.6.6.3.3.4.2"></sum><apply id="alg3.l8.m1.3.3.2.cmml" xref="alg3.l8.m1.3.3.2"><in id="alg3.l8.m1.3.3.2.3.cmml" xref="alg3.l8.m1.3.3.2.3"></in><interval closure="open" id="alg3.l8.m1.3.3.2.2.3.cmml" xref="alg3.l8.m1.3.3.2.2.2"><apply id="alg3.l8.m1.2.2.1.1.1.1.cmml" xref="alg3.l8.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l8.m1.2.2.1.1.1.1.1.cmml" xref="alg3.l8.m1.2.2.1.1.1.1">subscript</csymbol><ci id="alg3.l8.m1.2.2.1.1.1.1.2.cmml" xref="alg3.l8.m1.2.2.1.1.1.1.2">𝑥</ci><ci id="alg3.l8.m1.2.2.1.1.1.1.3.cmml" xref="alg3.l8.m1.2.2.1.1.1.1.3">𝑖</ci></apply><apply id="alg3.l8.m1.3.3.2.2.2.2.cmml" xref="alg3.l8.m1.3.3.2.2.2.2"><csymbol cd="ambiguous" id="alg3.l8.m1.3.3.2.2.2.2.1.cmml" xref="alg3.l8.m1.3.3.2.2.2.2">subscript</csymbol><ci id="alg3.l8.m1.3.3.2.2.2.2.2.cmml" xref="alg3.l8.m1.3.3.2.2.2.2.2">𝑦</ci><ci id="alg3.l8.m1.3.3.2.2.2.2.3.cmml" xref="alg3.l8.m1.3.3.2.2.2.2.3">𝑖</ci></apply></interval><apply id="alg3.l8.m1.3.3.2.4.cmml" xref="alg3.l8.m1.3.3.2.4"><csymbol cd="ambiguous" id="alg3.l8.m1.3.3.2.4.1.cmml" xref="alg3.l8.m1.3.3.2.4">subscript</csymbol><ci id="alg3.l8.m1.3.3.2.4.2.cmml" xref="alg3.l8.m1.3.3.2.4.2">𝐵</ci><ci id="alg3.l8.m1.3.3.2.4.3.cmml" xref="alg3.l8.m1.3.3.2.4.3">𝑘</ci></apply></apply></apply><apply id="alg3.l8.m1.6.6.3.3.3.cmml" xref="alg3.l8.m1.6.6.3.3.3"><times id="alg3.l8.m1.6.6.3.3.3.4.cmml" xref="alg3.l8.m1.6.6.3.3.3.4"></times><apply id="alg3.l8.m1.6.6.3.3.3.5.cmml" xref="alg3.l8.m1.6.6.3.3.3.5"><apply id="alg3.l8.m1.6.6.3.3.3.5.1.cmml" xref="alg3.l8.m1.6.6.3.3.3.5.1"><csymbol cd="ambiguous" id="alg3.l8.m1.6.6.3.3.3.5.1.1.cmml" xref="alg3.l8.m1.6.6.3.3.3.5.1">subscript</csymbol><ci id="alg3.l8.m1.6.6.3.3.3.5.1.2.cmml" xref="alg3.l8.m1.6.6.3.3.3.5.1.2">∇</ci><ci id="alg3.l8.m1.6.6.3.3.3.5.1.3.cmml" xref="alg3.l8.m1.6.6.3.3.3.5.1.3">𝜃</ci></apply><ci id="alg3.l8.m1.6.6.3.3.3.5.2.cmml" xref="alg3.l8.m1.6.6.3.3.3.5.2">𝐹</ci></apply><list id="alg3.l8.m1.6.6.3.3.3.3.4.cmml" xref="alg3.l8.m1.6.6.3.3.3.3.3"><apply id="alg3.l8.m1.4.4.1.1.1.1.1.1.cmml" xref="alg3.l8.m1.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l8.m1.4.4.1.1.1.1.1.1.1.cmml" xref="alg3.l8.m1.4.4.1.1.1.1.1.1">subscript</csymbol><ci id="alg3.l8.m1.4.4.1.1.1.1.1.1.2.cmml" xref="alg3.l8.m1.4.4.1.1.1.1.1.1.2">𝜃</ci><ci id="alg3.l8.m1.4.4.1.1.1.1.1.1.3.cmml" xref="alg3.l8.m1.4.4.1.1.1.1.1.1.3">𝑘</ci></apply><apply id="alg3.l8.m1.5.5.2.2.2.2.2.2.cmml" xref="alg3.l8.m1.5.5.2.2.2.2.2.2"><csymbol cd="ambiguous" id="alg3.l8.m1.5.5.2.2.2.2.2.2.1.cmml" xref="alg3.l8.m1.5.5.2.2.2.2.2.2">subscript</csymbol><ci id="alg3.l8.m1.5.5.2.2.2.2.2.2.2.cmml" xref="alg3.l8.m1.5.5.2.2.2.2.2.2.2">𝑥</ci><ci id="alg3.l8.m1.5.5.2.2.2.2.2.2.3.cmml" xref="alg3.l8.m1.5.5.2.2.2.2.2.2.3">𝑖</ci></apply><apply id="alg3.l8.m1.6.6.3.3.3.3.3.3.cmml" xref="alg3.l8.m1.6.6.3.3.3.3.3.3"><csymbol cd="ambiguous" id="alg3.l8.m1.6.6.3.3.3.3.3.3.1.cmml" xref="alg3.l8.m1.6.6.3.3.3.3.3.3">subscript</csymbol><ci id="alg3.l8.m1.6.6.3.3.3.3.3.3.2.cmml" xref="alg3.l8.m1.6.6.3.3.3.3.3.3.2">𝑦</ci><ci id="alg3.l8.m1.6.6.3.3.3.3.3.3.3.cmml" xref="alg3.l8.m1.6.6.3.3.3.3.3.3.3">𝑖</ci></apply></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l8.m1.6c">g_{k}=\frac{1}{|B_{k}|}\sum_{(x_{i},y_{i})\in B_{k}}\nabla_{\theta}F(\theta_{k% };x_{i},y_{i})</annotation><annotation encoding="application/x-llamapun" id="alg3.l8.m1.6d">italic_g start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG | italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT | end_ARG ∑ start_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ∈ italic_B start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∇ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT italic_F ( italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ; italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg3.l9"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l9.1.1.1" style="font-size:80%;">9:</span></span> Update local model: <math alttext="\theta_{k}\leftarrow\theta_{k}-\eta_{l}\cdot g_{k}" class="ltx_Math" display="inline" id="alg3.l9.m1.1"><semantics id="alg3.l9.m1.1a"><mrow id="alg3.l9.m1.1.1" xref="alg3.l9.m1.1.1.cmml"><msub id="alg3.l9.m1.1.1.2" xref="alg3.l9.m1.1.1.2.cmml"><mi id="alg3.l9.m1.1.1.2.2" xref="alg3.l9.m1.1.1.2.2.cmml">θ</mi><mi id="alg3.l9.m1.1.1.2.3" xref="alg3.l9.m1.1.1.2.3.cmml">k</mi></msub><mo id="alg3.l9.m1.1.1.1" stretchy="false" xref="alg3.l9.m1.1.1.1.cmml">←</mo><mrow id="alg3.l9.m1.1.1.3" xref="alg3.l9.m1.1.1.3.cmml"><msub id="alg3.l9.m1.1.1.3.2" xref="alg3.l9.m1.1.1.3.2.cmml"><mi id="alg3.l9.m1.1.1.3.2.2" xref="alg3.l9.m1.1.1.3.2.2.cmml">θ</mi><mi id="alg3.l9.m1.1.1.3.2.3" xref="alg3.l9.m1.1.1.3.2.3.cmml">k</mi></msub><mo id="alg3.l9.m1.1.1.3.1" xref="alg3.l9.m1.1.1.3.1.cmml">−</mo><mrow id="alg3.l9.m1.1.1.3.3" xref="alg3.l9.m1.1.1.3.3.cmml"><msub id="alg3.l9.m1.1.1.3.3.2" xref="alg3.l9.m1.1.1.3.3.2.cmml"><mi id="alg3.l9.m1.1.1.3.3.2.2" xref="alg3.l9.m1.1.1.3.3.2.2.cmml">η</mi><mi id="alg3.l9.m1.1.1.3.3.2.3" xref="alg3.l9.m1.1.1.3.3.2.3.cmml">l</mi></msub><mo id="alg3.l9.m1.1.1.3.3.1" lspace="0.222em" rspace="0.222em" xref="alg3.l9.m1.1.1.3.3.1.cmml">⋅</mo><msub id="alg3.l9.m1.1.1.3.3.3" xref="alg3.l9.m1.1.1.3.3.3.cmml"><mi id="alg3.l9.m1.1.1.3.3.3.2" xref="alg3.l9.m1.1.1.3.3.3.2.cmml">g</mi><mi id="alg3.l9.m1.1.1.3.3.3.3" xref="alg3.l9.m1.1.1.3.3.3.3.cmml">k</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l9.m1.1b"><apply id="alg3.l9.m1.1.1.cmml" xref="alg3.l9.m1.1.1"><ci id="alg3.l9.m1.1.1.1.cmml" xref="alg3.l9.m1.1.1.1">←</ci><apply id="alg3.l9.m1.1.1.2.cmml" xref="alg3.l9.m1.1.1.2"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.2.1.cmml" xref="alg3.l9.m1.1.1.2">subscript</csymbol><ci id="alg3.l9.m1.1.1.2.2.cmml" xref="alg3.l9.m1.1.1.2.2">𝜃</ci><ci id="alg3.l9.m1.1.1.2.3.cmml" xref="alg3.l9.m1.1.1.2.3">𝑘</ci></apply><apply id="alg3.l9.m1.1.1.3.cmml" xref="alg3.l9.m1.1.1.3"><minus id="alg3.l9.m1.1.1.3.1.cmml" xref="alg3.l9.m1.1.1.3.1"></minus><apply id="alg3.l9.m1.1.1.3.2.cmml" xref="alg3.l9.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.3.2.1.cmml" xref="alg3.l9.m1.1.1.3.2">subscript</csymbol><ci id="alg3.l9.m1.1.1.3.2.2.cmml" xref="alg3.l9.m1.1.1.3.2.2">𝜃</ci><ci id="alg3.l9.m1.1.1.3.2.3.cmml" xref="alg3.l9.m1.1.1.3.2.3">𝑘</ci></apply><apply id="alg3.l9.m1.1.1.3.3.cmml" xref="alg3.l9.m1.1.1.3.3"><ci id="alg3.l9.m1.1.1.3.3.1.cmml" xref="alg3.l9.m1.1.1.3.3.1">⋅</ci><apply id="alg3.l9.m1.1.1.3.3.2.cmml" xref="alg3.l9.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.3.3.2.1.cmml" xref="alg3.l9.m1.1.1.3.3.2">subscript</csymbol><ci id="alg3.l9.m1.1.1.3.3.2.2.cmml" xref="alg3.l9.m1.1.1.3.3.2.2">𝜂</ci><ci id="alg3.l9.m1.1.1.3.3.2.3.cmml" xref="alg3.l9.m1.1.1.3.3.2.3">𝑙</ci></apply><apply id="alg3.l9.m1.1.1.3.3.3.cmml" xref="alg3.l9.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.3.3.3.1.cmml" xref="alg3.l9.m1.1.1.3.3.3">subscript</csymbol><ci id="alg3.l9.m1.1.1.3.3.3.2.cmml" xref="alg3.l9.m1.1.1.3.3.3.2">𝑔</ci><ci id="alg3.l9.m1.1.1.3.3.3.3.cmml" xref="alg3.l9.m1.1.1.3.3.3.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l9.m1.1c">\theta_{k}\leftarrow\theta_{k}-\eta_{l}\cdot g_{k}</annotation><annotation encoding="application/x-llamapun" id="alg3.l9.m1.1d">italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ← italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT - italic_η start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ⋅ italic_g start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg3.l10"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l10.1.1.1" style="font-size:80%;">10:</span></span> Send back to server: (Option I) <math alttext="\Delta\theta_{k}=\theta_{k}-\theta" class="ltx_Math" display="inline" id="alg3.l10.m1.1"><semantics id="alg3.l10.m1.1a"><mrow id="alg3.l10.m1.1.1" xref="alg3.l10.m1.1.1.cmml"><mrow id="alg3.l10.m1.1.1.2" xref="alg3.l10.m1.1.1.2.cmml"><mi id="alg3.l10.m1.1.1.2.2" mathvariant="normal" xref="alg3.l10.m1.1.1.2.2.cmml">Δ</mi><mo id="alg3.l10.m1.1.1.2.1" xref="alg3.l10.m1.1.1.2.1.cmml">⁢</mo><msub id="alg3.l10.m1.1.1.2.3" xref="alg3.l10.m1.1.1.2.3.cmml"><mi id="alg3.l10.m1.1.1.2.3.2" xref="alg3.l10.m1.1.1.2.3.2.cmml">θ</mi><mi id="alg3.l10.m1.1.1.2.3.3" xref="alg3.l10.m1.1.1.2.3.3.cmml">k</mi></msub></mrow><mo id="alg3.l10.m1.1.1.1" xref="alg3.l10.m1.1.1.1.cmml">=</mo><mrow id="alg3.l10.m1.1.1.3" xref="alg3.l10.m1.1.1.3.cmml"><msub id="alg3.l10.m1.1.1.3.2" xref="alg3.l10.m1.1.1.3.2.cmml"><mi id="alg3.l10.m1.1.1.3.2.2" xref="alg3.l10.m1.1.1.3.2.2.cmml">θ</mi><mi id="alg3.l10.m1.1.1.3.2.3" xref="alg3.l10.m1.1.1.3.2.3.cmml">k</mi></msub><mo id="alg3.l10.m1.1.1.3.1" xref="alg3.l10.m1.1.1.3.1.cmml">−</mo><mi id="alg3.l10.m1.1.1.3.3" xref="alg3.l10.m1.1.1.3.3.cmml">θ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l10.m1.1b"><apply id="alg3.l10.m1.1.1.cmml" xref="alg3.l10.m1.1.1"><eq id="alg3.l10.m1.1.1.1.cmml" xref="alg3.l10.m1.1.1.1"></eq><apply id="alg3.l10.m1.1.1.2.cmml" xref="alg3.l10.m1.1.1.2"><times id="alg3.l10.m1.1.1.2.1.cmml" xref="alg3.l10.m1.1.1.2.1"></times><ci id="alg3.l10.m1.1.1.2.2.cmml" xref="alg3.l10.m1.1.1.2.2">Δ</ci><apply id="alg3.l10.m1.1.1.2.3.cmml" xref="alg3.l10.m1.1.1.2.3"><csymbol cd="ambiguous" id="alg3.l10.m1.1.1.2.3.1.cmml" xref="alg3.l10.m1.1.1.2.3">subscript</csymbol><ci id="alg3.l10.m1.1.1.2.3.2.cmml" xref="alg3.l10.m1.1.1.2.3.2">𝜃</ci><ci id="alg3.l10.m1.1.1.2.3.3.cmml" xref="alg3.l10.m1.1.1.2.3.3">𝑘</ci></apply></apply><apply id="alg3.l10.m1.1.1.3.cmml" xref="alg3.l10.m1.1.1.3"><minus id="alg3.l10.m1.1.1.3.1.cmml" xref="alg3.l10.m1.1.1.3.1"></minus><apply id="alg3.l10.m1.1.1.3.2.cmml" xref="alg3.l10.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg3.l10.m1.1.1.3.2.1.cmml" xref="alg3.l10.m1.1.1.3.2">subscript</csymbol><ci id="alg3.l10.m1.1.1.3.2.2.cmml" xref="alg3.l10.m1.1.1.3.2.2">𝜃</ci><ci id="alg3.l10.m1.1.1.3.2.3.cmml" xref="alg3.l10.m1.1.1.3.2.3">𝑘</ci></apply><ci id="alg3.l10.m1.1.1.3.3.cmml" xref="alg3.l10.m1.1.1.3.3">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l10.m1.1c">\Delta\theta_{k}=\theta_{k}-\theta</annotation><annotation encoding="application/x-llamapun" id="alg3.l10.m1.1d">roman_Δ italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT = italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT - italic_θ</annotation></semantics></math> or (Option II) <math alttext="\theta_{k}" class="ltx_Math" display="inline" id="alg3.l10.m2.1"><semantics id="alg3.l10.m2.1a"><msub id="alg3.l10.m2.1.1" xref="alg3.l10.m2.1.1.cmml"><mi id="alg3.l10.m2.1.1.2" xref="alg3.l10.m2.1.1.2.cmml">θ</mi><mi id="alg3.l10.m2.1.1.3" xref="alg3.l10.m2.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l10.m2.1b"><apply id="alg3.l10.m2.1.1.cmml" xref="alg3.l10.m2.1.1"><csymbol cd="ambiguous" id="alg3.l10.m2.1.1.1.cmml" xref="alg3.l10.m2.1.1">subscript</csymbol><ci id="alg3.l10.m2.1.1.2.cmml" xref="alg3.l10.m2.1.1.2">𝜃</ci><ci id="alg3.l10.m2.1.1.3.cmml" xref="alg3.l10.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l10.m2.1c">\theta_{k}</annotation><annotation encoding="application/x-llamapun" id="alg3.l10.m2.1d">italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg3.l11"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l11.1.1.1" style="font-size:80%;">11:</span></span> Aggregate the total number of data points: <math alttext="M=\sum_{k\in\mathcal{S}}|\mathcal{D}_{k}|" class="ltx_Math" display="inline" id="alg3.l11.m1.1"><semantics id="alg3.l11.m1.1a"><mrow id="alg3.l11.m1.1.1" xref="alg3.l11.m1.1.1.cmml"><mi id="alg3.l11.m1.1.1.3" xref="alg3.l11.m1.1.1.3.cmml">M</mi><mo id="alg3.l11.m1.1.1.2" rspace="0.111em" xref="alg3.l11.m1.1.1.2.cmml">=</mo><mrow id="alg3.l11.m1.1.1.1" xref="alg3.l11.m1.1.1.1.cmml"><msub id="alg3.l11.m1.1.1.1.2" xref="alg3.l11.m1.1.1.1.2.cmml"><mo id="alg3.l11.m1.1.1.1.2.2" rspace="0em" xref="alg3.l11.m1.1.1.1.2.2.cmml">∑</mo><mrow id="alg3.l11.m1.1.1.1.2.3" xref="alg3.l11.m1.1.1.1.2.3.cmml"><mi id="alg3.l11.m1.1.1.1.2.3.2" xref="alg3.l11.m1.1.1.1.2.3.2.cmml">k</mi><mo id="alg3.l11.m1.1.1.1.2.3.1" xref="alg3.l11.m1.1.1.1.2.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="alg3.l11.m1.1.1.1.2.3.3" xref="alg3.l11.m1.1.1.1.2.3.3.cmml">𝒮</mi></mrow></msub><mrow id="alg3.l11.m1.1.1.1.1.1" xref="alg3.l11.m1.1.1.1.1.2.cmml"><mo id="alg3.l11.m1.1.1.1.1.1.2" stretchy="false" xref="alg3.l11.m1.1.1.1.1.2.1.cmml">|</mo><msub id="alg3.l11.m1.1.1.1.1.1.1" xref="alg3.l11.m1.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg3.l11.m1.1.1.1.1.1.1.2" xref="alg3.l11.m1.1.1.1.1.1.1.2.cmml">𝒟</mi><mi id="alg3.l11.m1.1.1.1.1.1.1.3" xref="alg3.l11.m1.1.1.1.1.1.1.3.cmml">k</mi></msub><mo id="alg3.l11.m1.1.1.1.1.1.3" stretchy="false" xref="alg3.l11.m1.1.1.1.1.2.1.cmml">|</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l11.m1.1b"><apply id="alg3.l11.m1.1.1.cmml" xref="alg3.l11.m1.1.1"><eq id="alg3.l11.m1.1.1.2.cmml" xref="alg3.l11.m1.1.1.2"></eq><ci id="alg3.l11.m1.1.1.3.cmml" xref="alg3.l11.m1.1.1.3">𝑀</ci><apply id="alg3.l11.m1.1.1.1.cmml" xref="alg3.l11.m1.1.1.1"><apply id="alg3.l11.m1.1.1.1.2.cmml" xref="alg3.l11.m1.1.1.1.2"><csymbol cd="ambiguous" id="alg3.l11.m1.1.1.1.2.1.cmml" xref="alg3.l11.m1.1.1.1.2">subscript</csymbol><sum id="alg3.l11.m1.1.1.1.2.2.cmml" xref="alg3.l11.m1.1.1.1.2.2"></sum><apply id="alg3.l11.m1.1.1.1.2.3.cmml" xref="alg3.l11.m1.1.1.1.2.3"><in id="alg3.l11.m1.1.1.1.2.3.1.cmml" xref="alg3.l11.m1.1.1.1.2.3.1"></in><ci id="alg3.l11.m1.1.1.1.2.3.2.cmml" xref="alg3.l11.m1.1.1.1.2.3.2">𝑘</ci><ci id="alg3.l11.m1.1.1.1.2.3.3.cmml" xref="alg3.l11.m1.1.1.1.2.3.3">𝒮</ci></apply></apply><apply id="alg3.l11.m1.1.1.1.1.2.cmml" xref="alg3.l11.m1.1.1.1.1.1"><abs id="alg3.l11.m1.1.1.1.1.2.1.cmml" xref="alg3.l11.m1.1.1.1.1.1.2"></abs><apply id="alg3.l11.m1.1.1.1.1.1.1.cmml" xref="alg3.l11.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l11.m1.1.1.1.1.1.1.1.cmml" xref="alg3.l11.m1.1.1.1.1.1.1">subscript</csymbol><ci id="alg3.l11.m1.1.1.1.1.1.1.2.cmml" xref="alg3.l11.m1.1.1.1.1.1.1.2">𝒟</ci><ci id="alg3.l11.m1.1.1.1.1.1.1.3.cmml" xref="alg3.l11.m1.1.1.1.1.1.1.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l11.m1.1c">M=\sum_{k\in\mathcal{S}}|\mathcal{D}_{k}|</annotation><annotation encoding="application/x-llamapun" id="alg3.l11.m1.1d">italic_M = ∑ start_POSTSUBSCRIPT italic_k ∈ caligraphic_S end_POSTSUBSCRIPT | caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT |</annotation></semantics></math> </div> <div class="ltx_listingline" id="alg3.l12"> <span class="ltx_tag ltx_tag_listingline"><span class="ltx_text" id="alg3.l12.1.1.1" style="font-size:80%;">12:</span></span> Update global model: (I) <math alttext="\theta\leftarrow\theta+\eta_{g}\cdot\sum_{k\in\mathcal{S}}\frac{|\mathcal{D}_{% k}|}{M}\cdot\Delta\theta_{k}" class="ltx_Math" display="inline" id="alg3.l12.m1.1"><semantics id="alg3.l12.m1.1a"><mrow id="alg3.l12.m1.1.2" xref="alg3.l12.m1.1.2.cmml"><mi id="alg3.l12.m1.1.2.2" xref="alg3.l12.m1.1.2.2.cmml">θ</mi><mo id="alg3.l12.m1.1.2.1" stretchy="false" xref="alg3.l12.m1.1.2.1.cmml">←</mo><mrow id="alg3.l12.m1.1.2.3" xref="alg3.l12.m1.1.2.3.cmml"><mi id="alg3.l12.m1.1.2.3.2" xref="alg3.l12.m1.1.2.3.2.cmml">θ</mi><mo id="alg3.l12.m1.1.2.3.1" xref="alg3.l12.m1.1.2.3.1.cmml">+</mo><mrow id="alg3.l12.m1.1.2.3.3" xref="alg3.l12.m1.1.2.3.3.cmml"><msub id="alg3.l12.m1.1.2.3.3.2" xref="alg3.l12.m1.1.2.3.3.2.cmml"><mi id="alg3.l12.m1.1.2.3.3.2.2" xref="alg3.l12.m1.1.2.3.3.2.2.cmml">η</mi><mi id="alg3.l12.m1.1.2.3.3.2.3" xref="alg3.l12.m1.1.2.3.3.2.3.cmml">g</mi></msub><mo id="alg3.l12.m1.1.2.3.3.1" lspace="0.222em" rspace="0.055em" xref="alg3.l12.m1.1.2.3.3.1.cmml">⋅</mo><mrow id="alg3.l12.m1.1.2.3.3.3" xref="alg3.l12.m1.1.2.3.3.3.cmml"><msub id="alg3.l12.m1.1.2.3.3.3.1" xref="alg3.l12.m1.1.2.3.3.3.1.cmml"><mo id="alg3.l12.m1.1.2.3.3.3.1.2" xref="alg3.l12.m1.1.2.3.3.3.1.2.cmml">∑</mo><mrow id="alg3.l12.m1.1.2.3.3.3.1.3" xref="alg3.l12.m1.1.2.3.3.3.1.3.cmml"><mi id="alg3.l12.m1.1.2.3.3.3.1.3.2" xref="alg3.l12.m1.1.2.3.3.3.1.3.2.cmml">k</mi><mo id="alg3.l12.m1.1.2.3.3.3.1.3.1" xref="alg3.l12.m1.1.2.3.3.3.1.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="alg3.l12.m1.1.2.3.3.3.1.3.3" xref="alg3.l12.m1.1.2.3.3.3.1.3.3.cmml">𝒮</mi></mrow></msub><mrow id="alg3.l12.m1.1.2.3.3.3.2" xref="alg3.l12.m1.1.2.3.3.3.2.cmml"><mrow id="alg3.l12.m1.1.2.3.3.3.2.2" xref="alg3.l12.m1.1.2.3.3.3.2.2.cmml"><mfrac id="alg3.l12.m1.1.1" xref="alg3.l12.m1.1.1.cmml"><mrow id="alg3.l12.m1.1.1.1.1" xref="alg3.l12.m1.1.1.1.2.cmml"><mo id="alg3.l12.m1.1.1.1.1.2" stretchy="false" xref="alg3.l12.m1.1.1.1.2.1.cmml">|</mo><msub id="alg3.l12.m1.1.1.1.1.1" xref="alg3.l12.m1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg3.l12.m1.1.1.1.1.1.2" xref="alg3.l12.m1.1.1.1.1.1.2.cmml">𝒟</mi><mi id="alg3.l12.m1.1.1.1.1.1.3" xref="alg3.l12.m1.1.1.1.1.1.3.cmml">k</mi></msub><mo id="alg3.l12.m1.1.1.1.1.3" stretchy="false" xref="alg3.l12.m1.1.1.1.2.1.cmml">|</mo></mrow><mi id="alg3.l12.m1.1.1.3" xref="alg3.l12.m1.1.1.3.cmml">M</mi></mfrac><mo id="alg3.l12.m1.1.2.3.3.3.2.2.1" lspace="0.222em" rspace="0.222em" xref="alg3.l12.m1.1.2.3.3.3.2.2.1.cmml">⋅</mo><mi id="alg3.l12.m1.1.2.3.3.3.2.2.2" mathvariant="normal" xref="alg3.l12.m1.1.2.3.3.3.2.2.2.cmml">Δ</mi></mrow><mo id="alg3.l12.m1.1.2.3.3.3.2.1" xref="alg3.l12.m1.1.2.3.3.3.2.1.cmml">⁢</mo><msub id="alg3.l12.m1.1.2.3.3.3.2.3" xref="alg3.l12.m1.1.2.3.3.3.2.3.cmml"><mi id="alg3.l12.m1.1.2.3.3.3.2.3.2" xref="alg3.l12.m1.1.2.3.3.3.2.3.2.cmml">θ</mi><mi id="alg3.l12.m1.1.2.3.3.3.2.3.3" xref="alg3.l12.m1.1.2.3.3.3.2.3.3.cmml">k</mi></msub></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l12.m1.1b"><apply id="alg3.l12.m1.1.2.cmml" xref="alg3.l12.m1.1.2"><ci id="alg3.l12.m1.1.2.1.cmml" xref="alg3.l12.m1.1.2.1">←</ci><ci id="alg3.l12.m1.1.2.2.cmml" xref="alg3.l12.m1.1.2.2">𝜃</ci><apply id="alg3.l12.m1.1.2.3.cmml" xref="alg3.l12.m1.1.2.3"><plus id="alg3.l12.m1.1.2.3.1.cmml" xref="alg3.l12.m1.1.2.3.1"></plus><ci id="alg3.l12.m1.1.2.3.2.cmml" xref="alg3.l12.m1.1.2.3.2">𝜃</ci><apply id="alg3.l12.m1.1.2.3.3.cmml" xref="alg3.l12.m1.1.2.3.3"><ci id="alg3.l12.m1.1.2.3.3.1.cmml" xref="alg3.l12.m1.1.2.3.3.1">⋅</ci><apply id="alg3.l12.m1.1.2.3.3.2.cmml" xref="alg3.l12.m1.1.2.3.3.2"><csymbol cd="ambiguous" id="alg3.l12.m1.1.2.3.3.2.1.cmml" xref="alg3.l12.m1.1.2.3.3.2">subscript</csymbol><ci id="alg3.l12.m1.1.2.3.3.2.2.cmml" xref="alg3.l12.m1.1.2.3.3.2.2">𝜂</ci><ci id="alg3.l12.m1.1.2.3.3.2.3.cmml" xref="alg3.l12.m1.1.2.3.3.2.3">𝑔</ci></apply><apply id="alg3.l12.m1.1.2.3.3.3.cmml" xref="alg3.l12.m1.1.2.3.3.3"><apply id="alg3.l12.m1.1.2.3.3.3.1.cmml" xref="alg3.l12.m1.1.2.3.3.3.1"><csymbol cd="ambiguous" id="alg3.l12.m1.1.2.3.3.3.1.1.cmml" xref="alg3.l12.m1.1.2.3.3.3.1">subscript</csymbol><sum id="alg3.l12.m1.1.2.3.3.3.1.2.cmml" xref="alg3.l12.m1.1.2.3.3.3.1.2"></sum><apply id="alg3.l12.m1.1.2.3.3.3.1.3.cmml" xref="alg3.l12.m1.1.2.3.3.3.1.3"><in id="alg3.l12.m1.1.2.3.3.3.1.3.1.cmml" xref="alg3.l12.m1.1.2.3.3.3.1.3.1"></in><ci id="alg3.l12.m1.1.2.3.3.3.1.3.2.cmml" xref="alg3.l12.m1.1.2.3.3.3.1.3.2">𝑘</ci><ci id="alg3.l12.m1.1.2.3.3.3.1.3.3.cmml" xref="alg3.l12.m1.1.2.3.3.3.1.3.3">𝒮</ci></apply></apply><apply id="alg3.l12.m1.1.2.3.3.3.2.cmml" xref="alg3.l12.m1.1.2.3.3.3.2"><times id="alg3.l12.m1.1.2.3.3.3.2.1.cmml" xref="alg3.l12.m1.1.2.3.3.3.2.1"></times><apply id="alg3.l12.m1.1.2.3.3.3.2.2.cmml" xref="alg3.l12.m1.1.2.3.3.3.2.2"><ci id="alg3.l12.m1.1.2.3.3.3.2.2.1.cmml" xref="alg3.l12.m1.1.2.3.3.3.2.2.1">⋅</ci><apply id="alg3.l12.m1.1.1.cmml" xref="alg3.l12.m1.1.1"><divide id="alg3.l12.m1.1.1.2.cmml" xref="alg3.l12.m1.1.1"></divide><apply id="alg3.l12.m1.1.1.1.2.cmml" xref="alg3.l12.m1.1.1.1.1"><abs id="alg3.l12.m1.1.1.1.2.1.cmml" xref="alg3.l12.m1.1.1.1.1.2"></abs><apply id="alg3.l12.m1.1.1.1.1.1.cmml" xref="alg3.l12.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l12.m1.1.1.1.1.1.1.cmml" xref="alg3.l12.m1.1.1.1.1.1">subscript</csymbol><ci id="alg3.l12.m1.1.1.1.1.1.2.cmml" xref="alg3.l12.m1.1.1.1.1.1.2">𝒟</ci><ci id="alg3.l12.m1.1.1.1.1.1.3.cmml" xref="alg3.l12.m1.1.1.1.1.1.3">𝑘</ci></apply></apply><ci id="alg3.l12.m1.1.1.3.cmml" xref="alg3.l12.m1.1.1.3">𝑀</ci></apply><ci id="alg3.l12.m1.1.2.3.3.3.2.2.2.cmml" xref="alg3.l12.m1.1.2.3.3.3.2.2.2">Δ</ci></apply><apply id="alg3.l12.m1.1.2.3.3.3.2.3.cmml" xref="alg3.l12.m1.1.2.3.3.3.2.3"><csymbol cd="ambiguous" id="alg3.l12.m1.1.2.3.3.3.2.3.1.cmml" xref="alg3.l12.m1.1.2.3.3.3.2.3">subscript</csymbol><ci id="alg3.l12.m1.1.2.3.3.3.2.3.2.cmml" xref="alg3.l12.m1.1.2.3.3.3.2.3.2">𝜃</ci><ci id="alg3.l12.m1.1.2.3.3.3.2.3.3.cmml" xref="alg3.l12.m1.1.2.3.3.3.2.3.3">𝑘</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l12.m1.1c">\theta\leftarrow\theta+\eta_{g}\cdot\sum_{k\in\mathcal{S}}\frac{|\mathcal{D}_{% k}|}{M}\cdot\Delta\theta_{k}</annotation><annotation encoding="application/x-llamapun" id="alg3.l12.m1.1d">italic_θ ← italic_θ + italic_η start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT ⋅ ∑ start_POSTSUBSCRIPT italic_k ∈ caligraphic_S end_POSTSUBSCRIPT divide start_ARG | caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT | end_ARG start_ARG italic_M end_ARG ⋅ roman_Δ italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> or (II) <math alttext="\theta\leftarrow\sum_{k\in\mathcal{S}}\frac{|\mathcal{D}_{k}|}{M}\cdot\theta_{k}" class="ltx_Math" display="inline" id="alg3.l12.m2.1"><semantics id="alg3.l12.m2.1a"><mrow id="alg3.l12.m2.1.2" xref="alg3.l12.m2.1.2.cmml"><mi id="alg3.l12.m2.1.2.2" xref="alg3.l12.m2.1.2.2.cmml">θ</mi><mo id="alg3.l12.m2.1.2.1" rspace="0.111em" stretchy="false" xref="alg3.l12.m2.1.2.1.cmml">←</mo><mrow id="alg3.l12.m2.1.2.3" xref="alg3.l12.m2.1.2.3.cmml"><msub id="alg3.l12.m2.1.2.3.1" xref="alg3.l12.m2.1.2.3.1.cmml"><mo id="alg3.l12.m2.1.2.3.1.2" xref="alg3.l12.m2.1.2.3.1.2.cmml">∑</mo><mrow id="alg3.l12.m2.1.2.3.1.3" xref="alg3.l12.m2.1.2.3.1.3.cmml"><mi id="alg3.l12.m2.1.2.3.1.3.2" xref="alg3.l12.m2.1.2.3.1.3.2.cmml">k</mi><mo id="alg3.l12.m2.1.2.3.1.3.1" xref="alg3.l12.m2.1.2.3.1.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="alg3.l12.m2.1.2.3.1.3.3" xref="alg3.l12.m2.1.2.3.1.3.3.cmml">𝒮</mi></mrow></msub><mrow id="alg3.l12.m2.1.2.3.2" xref="alg3.l12.m2.1.2.3.2.cmml"><mfrac id="alg3.l12.m2.1.1" xref="alg3.l12.m2.1.1.cmml"><mrow id="alg3.l12.m2.1.1.1.1" xref="alg3.l12.m2.1.1.1.2.cmml"><mo id="alg3.l12.m2.1.1.1.1.2" stretchy="false" xref="alg3.l12.m2.1.1.1.2.1.cmml">|</mo><msub id="alg3.l12.m2.1.1.1.1.1" xref="alg3.l12.m2.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg3.l12.m2.1.1.1.1.1.2" xref="alg3.l12.m2.1.1.1.1.1.2.cmml">𝒟</mi><mi id="alg3.l12.m2.1.1.1.1.1.3" xref="alg3.l12.m2.1.1.1.1.1.3.cmml">k</mi></msub><mo id="alg3.l12.m2.1.1.1.1.3" stretchy="false" xref="alg3.l12.m2.1.1.1.2.1.cmml">|</mo></mrow><mi id="alg3.l12.m2.1.1.3" xref="alg3.l12.m2.1.1.3.cmml">M</mi></mfrac><mo id="alg3.l12.m2.1.2.3.2.1" lspace="0.222em" rspace="0.222em" xref="alg3.l12.m2.1.2.3.2.1.cmml">⋅</mo><msub id="alg3.l12.m2.1.2.3.2.2" xref="alg3.l12.m2.1.2.3.2.2.cmml"><mi id="alg3.l12.m2.1.2.3.2.2.2" xref="alg3.l12.m2.1.2.3.2.2.2.cmml">θ</mi><mi id="alg3.l12.m2.1.2.3.2.2.3" xref="alg3.l12.m2.1.2.3.2.2.3.cmml">k</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l12.m2.1b"><apply id="alg3.l12.m2.1.2.cmml" xref="alg3.l12.m2.1.2"><ci id="alg3.l12.m2.1.2.1.cmml" xref="alg3.l12.m2.1.2.1">←</ci><ci id="alg3.l12.m2.1.2.2.cmml" xref="alg3.l12.m2.1.2.2">𝜃</ci><apply id="alg3.l12.m2.1.2.3.cmml" xref="alg3.l12.m2.1.2.3"><apply id="alg3.l12.m2.1.2.3.1.cmml" xref="alg3.l12.m2.1.2.3.1"><csymbol cd="ambiguous" id="alg3.l12.m2.1.2.3.1.1.cmml" xref="alg3.l12.m2.1.2.3.1">subscript</csymbol><sum id="alg3.l12.m2.1.2.3.1.2.cmml" xref="alg3.l12.m2.1.2.3.1.2"></sum><apply id="alg3.l12.m2.1.2.3.1.3.cmml" xref="alg3.l12.m2.1.2.3.1.3"><in id="alg3.l12.m2.1.2.3.1.3.1.cmml" xref="alg3.l12.m2.1.2.3.1.3.1"></in><ci id="alg3.l12.m2.1.2.3.1.3.2.cmml" xref="alg3.l12.m2.1.2.3.1.3.2">𝑘</ci><ci id="alg3.l12.m2.1.2.3.1.3.3.cmml" xref="alg3.l12.m2.1.2.3.1.3.3">𝒮</ci></apply></apply><apply id="alg3.l12.m2.1.2.3.2.cmml" xref="alg3.l12.m2.1.2.3.2"><ci id="alg3.l12.m2.1.2.3.2.1.cmml" xref="alg3.l12.m2.1.2.3.2.1">⋅</ci><apply id="alg3.l12.m2.1.1.cmml" xref="alg3.l12.m2.1.1"><divide id="alg3.l12.m2.1.1.2.cmml" xref="alg3.l12.m2.1.1"></divide><apply id="alg3.l12.m2.1.1.1.2.cmml" xref="alg3.l12.m2.1.1.1.1"><abs id="alg3.l12.m2.1.1.1.2.1.cmml" xref="alg3.l12.m2.1.1.1.1.2"></abs><apply id="alg3.l12.m2.1.1.1.1.1.cmml" xref="alg3.l12.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l12.m2.1.1.1.1.1.1.cmml" xref="alg3.l12.m2.1.1.1.1.1">subscript</csymbol><ci id="alg3.l12.m2.1.1.1.1.1.2.cmml" xref="alg3.l12.m2.1.1.1.1.1.2">𝒟</ci><ci id="alg3.l12.m2.1.1.1.1.1.3.cmml" xref="alg3.l12.m2.1.1.1.1.1.3">𝑘</ci></apply></apply><ci id="alg3.l12.m2.1.1.3.cmml" xref="alg3.l12.m2.1.1.3">𝑀</ci></apply><apply id="alg3.l12.m2.1.2.3.2.2.cmml" xref="alg3.l12.m2.1.2.3.2.2"><csymbol cd="ambiguous" id="alg3.l12.m2.1.2.3.2.2.1.cmml" xref="alg3.l12.m2.1.2.3.2.2">subscript</csymbol><ci id="alg3.l12.m2.1.2.3.2.2.2.cmml" xref="alg3.l12.m2.1.2.3.2.2.2">𝜃</ci><ci id="alg3.l12.m2.1.2.3.2.2.3.cmml" xref="alg3.l12.m2.1.2.3.2.2.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l12.m2.1c">\theta\leftarrow\sum_{k\in\mathcal{S}}\frac{|\mathcal{D}_{k}|}{M}\cdot\theta_{k}</annotation><annotation encoding="application/x-llamapun" id="alg3.l12.m2.1d">italic_θ ← ∑ start_POSTSUBSCRIPT italic_k ∈ caligraphic_S end_POSTSUBSCRIPT divide start_ARG | caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT | end_ARG start_ARG italic_M end_ARG ⋅ italic_θ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> </div> </div> </figure> <div class="ltx_para" id="S2.p14"> <p class="ltx_p" id="S2.p14.1">Originally, FedAvg exchanges local model parameters (Option II) instead of updates (Option I), which are mathematically equivalent. However, sending updates allows for more flexibility in modifying the aggregation process, enabling techniques such as global learning rate adjustments and optimizers <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib7" title="">7</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib16" title="">16</a>]</cite>. Moreover, this formulation provides a clearer conceptual framework for understanding FL. From the perspective of Option I in line <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#alg3.l12" title="In Algorithm 3 ‣ 2 From Gradient Descent to FedAvg ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">12</span></a>, the updates from each client resemble gradients from individual data samples in GD. Therefore, FL is not merely parameter mixing but remains a structured optimization process. The following sections provide an empirical analysis of how FL works in practice.</p> </div> </section> <section class="ltx_section" id="S3"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">3 </span>Experimental Setup</h2> <section class="ltx_subsection" id="S3.SS1"> <h3 class="ltx_title ltx_title_subsection"> <span class="ltx_tag ltx_tag_subsection">3.1 </span>Basic Configuration</h3> <div class="ltx_para" id="S3.SS1.p1"> <p class="ltx_p" id="S3.SS1.p1.3">We conduct our experiments using the CIFAR-10 dataset (50,000 instances in the training set and 10,000 in the test set), a standard benchmark in FL research. The model architecture is a convolutional neural network (CNN) with two convolutional layers followed by ReLU activations and max-pooling. It includes two fully connected layers, leading to a 10-class output, as described in <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib14" title="">14</a>]</cite>. The task is a classification problem, with cross-entropy loss serving as the objective function. To ensure a controlled experimental environment, we deliberately exclude additional techniques such as weight decay, data augmentation, and advanced optimizers (e.g., momentum or adaptive techniques). This approach allows us to isolate the impact of the hyperparameters <math alttext="B" class="ltx_Math" display="inline" id="S3.SS1.p1.1.m1.1"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">B</annotation><annotation encoding="application/x-llamapun" id="S3.SS1.p1.1.m1.1d">italic_B</annotation></semantics></math>, <math alttext="E" class="ltx_Math" display="inline" id="S3.SS1.p1.2.m2.1"><semantics id="S3.SS1.p1.2.m2.1a"><mi id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><ci id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">E</annotation><annotation encoding="application/x-llamapun" id="S3.SS1.p1.2.m2.1d">italic_E</annotation></semantics></math>, and <math alttext="\eta" class="ltx_Math" display="inline" id="S3.SS1.p1.3.m3.1"><semantics id="S3.SS1.p1.3.m3.1a"><mi id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><ci id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S3.SS1.p1.3.m3.1d">italic_η</annotation></semantics></math> on the SGD process.</p> </div> </section> <section class="ltx_subsection" id="S3.SS2"> <h3 class="ltx_title ltx_title_subsection"> <span class="ltx_tag ltx_tag_subsection">3.2 </span>FL Setup</h3> <section class="ltx_subsubsection" id="S3.SS2.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> <span class="ltx_tag ltx_tag_subsubsection">3.2.1 </span>Evaluation</h4> <div class="ltx_para" id="S3.SS2.SSS1.p1"> <p class="ltx_p" id="S3.SS2.SSS1.p1.1">We evaluate model performance by analyzing both convergence speed and generalization quality. While rapid convergence is desirable, it does not inherently guarantee strong generalization, making it crucial to track both aspects throughout training. In our setup, global model accuracy (Top-1) is measured server-side using a dedicated test set. Test loss is also computed server-side to assess generalization, while training loss is computed locally by clients. The server averages client-side training losses to monitor overall optimization progress.</p> </div> </section> <section class="ltx_subsubsection" id="S3.SS2.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> <span class="ltx_tag ltx_tag_subsubsection">3.2.2 </span>Transition from Centralized to Federated Learning</h4> <div class="ltx_para" id="S3.SS2.SSS2.p1"> <p class="ltx_p" id="S3.SS2.SSS2.p1.4">To set initial hyperparameters, we first establish reasonable baseline performance in a centralized learning (CL) setup, corresponding to <math alttext="K=1" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p1.1.m1.1"><semantics id="S3.SS2.SSS2.p1.1.m1.1a"><mrow id="S3.SS2.SSS2.p1.1.m1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.1.1.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.cmml">K</mi><mo id="S3.SS2.SSS2.p1.1.m1.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p1.1.m1.1.1.3" xref="S3.SS2.SSS2.p1.1.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.1.m1.1b"><apply id="S3.SS2.SSS2.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1"><eq id="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1"></eq><ci id="S3.SS2.SSS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.2">𝐾</ci><cn id="S3.SS2.SSS2.p1.1.m1.1.1.3.cmml" type="integer" xref="S3.SS2.SSS2.p1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.1.m1.1c">K=1</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p1.1.m1.1d">italic_K = 1</annotation></semantics></math>. We use <math alttext="B=500" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p1.2.m2.1"><semantics id="S3.SS2.SSS2.p1.2.m2.1a"><mrow id="S3.SS2.SSS2.p1.2.m2.1.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.cmml"><mi id="S3.SS2.SSS2.p1.2.m2.1.1.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.cmml">B</mi><mo id="S3.SS2.SSS2.p1.2.m2.1.1.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p1.2.m2.1.1.3" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.cmml">500</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.2.m2.1b"><apply id="S3.SS2.SSS2.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1"><eq id="S3.SS2.SSS2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.1"></eq><ci id="S3.SS2.SSS2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2">𝐵</ci><cn id="S3.SS2.SSS2.p1.2.m2.1.1.3.cmml" type="integer" xref="S3.SS2.SSS2.p1.2.m2.1.1.3">500</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.2.m2.1c">B=500</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p1.2.m2.1d">italic_B = 500</annotation></semantics></math>, <math alttext="\eta=0.005" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p1.3.m3.1"><semantics id="S3.SS2.SSS2.p1.3.m3.1a"><mrow id="S3.SS2.SSS2.p1.3.m3.1.1" xref="S3.SS2.SSS2.p1.3.m3.1.1.cmml"><mi id="S3.SS2.SSS2.p1.3.m3.1.1.2" xref="S3.SS2.SSS2.p1.3.m3.1.1.2.cmml">η</mi><mo id="S3.SS2.SSS2.p1.3.m3.1.1.1" xref="S3.SS2.SSS2.p1.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p1.3.m3.1.1.3" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.cmml">0.005</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.3.m3.1b"><apply id="S3.SS2.SSS2.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1"><eq id="S3.SS2.SSS2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.1"></eq><ci id="S3.SS2.SSS2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.2">𝜂</ci><cn id="S3.SS2.SSS2.p1.3.m3.1.1.3.cmml" type="float" xref="S3.SS2.SSS2.p1.3.m3.1.1.3">0.005</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.3.m3.1c">\eta=0.005</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p1.3.m3.1d">italic_η = 0.005</annotation></semantics></math>, and <math alttext="E=1" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p1.4.m4.1"><semantics id="S3.SS2.SSS2.p1.4.m4.1a"><mrow id="S3.SS2.SSS2.p1.4.m4.1.1" xref="S3.SS2.SSS2.p1.4.m4.1.1.cmml"><mi id="S3.SS2.SSS2.p1.4.m4.1.1.2" xref="S3.SS2.SSS2.p1.4.m4.1.1.2.cmml">E</mi><mo id="S3.SS2.SSS2.p1.4.m4.1.1.1" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p1.4.m4.1.1.3" xref="S3.SS2.SSS2.p1.4.m4.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.4.m4.1b"><apply id="S3.SS2.SSS2.p1.4.m4.1.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1"><eq id="S3.SS2.SSS2.p1.4.m4.1.1.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1"></eq><ci id="S3.SS2.SSS2.p1.4.m4.1.1.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.2">𝐸</ci><cn id="S3.SS2.SSS2.p1.4.m4.1.1.3.cmml" type="integer" xref="S3.SS2.SSS2.p1.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.4.m4.1c">E=1</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p1.4.m4.1d">italic_E = 1</annotation></semantics></math>, yielding a final accuracy of around 68%. These hyperparameters are not fine-tuned for peak performance in CL; rather, the aim is to observe how performance evolves as we transition from CL to FL.</p> </div> <div class="ltx_para" id="S3.SS2.SSS2.p2"> <p class="ltx_p" id="S3.SS2.SSS2.p2.4">When <math alttext="K" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p2.1.m1.1"><semantics id="S3.SS2.SSS2.p2.1.m1.1a"><mi id="S3.SS2.SSS2.p2.1.m1.1.1" xref="S3.SS2.SSS2.p2.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.1.m1.1b"><ci id="S3.SS2.SSS2.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.1.m1.1c">K</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p2.1.m1.1d">italic_K</annotation></semantics></math> changes, the dataset is proportionally divided among <math alttext="K" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p2.2.m2.1"><semantics id="S3.SS2.SSS2.p2.2.m2.1a"><mi id="S3.SS2.SSS2.p2.2.m2.1.1" xref="S3.SS2.SSS2.p2.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.2.m2.1b"><ci id="S3.SS2.SSS2.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.2.m2.1c">K</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p2.2.m2.1d">italic_K</annotation></semantics></math> clients. For <math alttext="K=1" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p2.3.m3.1"><semantics id="S3.SS2.SSS2.p2.3.m3.1a"><mrow id="S3.SS2.SSS2.p2.3.m3.1.1" xref="S3.SS2.SSS2.p2.3.m3.1.1.cmml"><mi id="S3.SS2.SSS2.p2.3.m3.1.1.2" xref="S3.SS2.SSS2.p2.3.m3.1.1.2.cmml">K</mi><mo id="S3.SS2.SSS2.p2.3.m3.1.1.1" xref="S3.SS2.SSS2.p2.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p2.3.m3.1.1.3" xref="S3.SS2.SSS2.p2.3.m3.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.3.m3.1b"><apply id="S3.SS2.SSS2.p2.3.m3.1.1.cmml" xref="S3.SS2.SSS2.p2.3.m3.1.1"><eq id="S3.SS2.SSS2.p2.3.m3.1.1.1.cmml" xref="S3.SS2.SSS2.p2.3.m3.1.1.1"></eq><ci id="S3.SS2.SSS2.p2.3.m3.1.1.2.cmml" xref="S3.SS2.SSS2.p2.3.m3.1.1.2">𝐾</ci><cn id="S3.SS2.SSS2.p2.3.m3.1.1.3.cmml" type="integer" xref="S3.SS2.SSS2.p2.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.3.m3.1c">K=1</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p2.3.m3.1d">italic_K = 1</annotation></semantics></math> (CL), a single client holds the entire training dataset of 50,000 samples, while for <math alttext="K=10" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p2.4.m4.1"><semantics id="S3.SS2.SSS2.p2.4.m4.1a"><mrow id="S3.SS2.SSS2.p2.4.m4.1.1" xref="S3.SS2.SSS2.p2.4.m4.1.1.cmml"><mi id="S3.SS2.SSS2.p2.4.m4.1.1.2" xref="S3.SS2.SSS2.p2.4.m4.1.1.2.cmml">K</mi><mo id="S3.SS2.SSS2.p2.4.m4.1.1.1" xref="S3.SS2.SSS2.p2.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p2.4.m4.1.1.3" xref="S3.SS2.SSS2.p2.4.m4.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.4.m4.1b"><apply id="S3.SS2.SSS2.p2.4.m4.1.1.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1"><eq id="S3.SS2.SSS2.p2.4.m4.1.1.1.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.1"></eq><ci id="S3.SS2.SSS2.p2.4.m4.1.1.2.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.2">𝐾</ci><cn id="S3.SS2.SSS2.p2.4.m4.1.1.3.cmml" type="integer" xref="S3.SS2.SSS2.p2.4.m4.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.4.m4.1c">K=10</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p2.4.m4.1d">italic_K = 10</annotation></semantics></math>, each client is allocated 5,000 unique training samples. We initialize with a balanced IID dataset, ensuring that all clients have an equal distribution of labels and an identical number of samples. The effect of varying the number of clients on FL training dynamics is shown in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S3.F2" title="Figure 2 ‣ 3.2.2 Transition from Centralized to Federated Learning ‣ 3.2 FL Setup ‣ 3 Experimental Setup ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">2</span></a>.</p> </div> <figure class="ltx_figure" id="S3.F2"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="230" id="S3.F2.g1" src="x2.png" width="830"/> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Impact of varying <math alttext="K" class="ltx_Math" display="inline" id="S3.F2.2.m1.1"><semantics id="S3.F2.2.m1.1b"><mi id="S3.F2.2.m1.1.1" xref="S3.F2.2.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.F2.2.m1.1c"><ci id="S3.F2.2.m1.1.1.cmml" xref="S3.F2.2.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.2.m1.1d">K</annotation><annotation encoding="application/x-llamapun" id="S3.F2.2.m1.1e">italic_K</annotation></semantics></math> with consistent hyperparameters.</figcaption> </figure> <div class="ltx_para" id="S3.SS2.SSS2.p3"> <p class="ltx_p" id="S3.SS2.SSS2.p3.3">In this setup, as <math alttext="K" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p3.1.m1.1"><semantics id="S3.SS2.SSS2.p3.1.m1.1a"><mi id="S3.SS2.SSS2.p3.1.m1.1.1" xref="S3.SS2.SSS2.p3.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p3.1.m1.1b"><ci id="S3.SS2.SSS2.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p3.1.m1.1c">K</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p3.1.m1.1d">italic_K</annotation></semantics></math> increases, there is a noticeable decline in performance within a fixed number of communication rounds, highlighting a common oversight. When transitioning from CL to FL with a limited dataset, the critical factor that must be consistently controlled for a fair comparison is the <span class="ltx_text ltx_font_bold" id="S3.SS2.SSS2.p3.3.1">amount</span> of effective updates per round, <math alttext="u" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p3.2.m2.1"><semantics id="S3.SS2.SSS2.p3.2.m2.1a"><mi id="S3.SS2.SSS2.p3.2.m2.1.1" xref="S3.SS2.SSS2.p3.2.m2.1.1.cmml">u</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p3.2.m2.1b"><ci id="S3.SS2.SSS2.p3.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1">𝑢</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p3.2.m2.1c">u</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p3.2.m2.1d">italic_u</annotation></semantics></math>, rather than the <span class="ltx_text ltx_font_bold" id="S3.SS2.SSS2.p3.3.2">number</span>, due to the influence of <math alttext="\eta" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p3.3.m3.1"><semantics id="S3.SS2.SSS2.p3.3.m3.1a"><mi id="S3.SS2.SSS2.p3.3.m3.1.1" xref="S3.SS2.SSS2.p3.3.m3.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p3.3.m3.1b"><ci id="S3.SS2.SSS2.p3.3.m3.1.1.cmml" xref="S3.SS2.SSS2.p3.3.m3.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p3.3.m3.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p3.3.m3.1d">italic_η</annotation></semantics></math>. It is defined as:</p> </div> <div class="ltx_para" id="S3.SS2.SSS2.p4"> <table class="ltx_equation ltx_eqn_table" id="S3.E3"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="u=\eta\cdot\frac{E\cdot|\mathcal{D}|}{B\cdot K}" class="ltx_Math" display="block" id="S3.E3.m1.1"><semantics id="S3.E3.m1.1a"><mrow id="S3.E3.m1.1.2" xref="S3.E3.m1.1.2.cmml"><mi id="S3.E3.m1.1.2.2" xref="S3.E3.m1.1.2.2.cmml">u</mi><mo id="S3.E3.m1.1.2.1" xref="S3.E3.m1.1.2.1.cmml">=</mo><mrow id="S3.E3.m1.1.2.3" xref="S3.E3.m1.1.2.3.cmml"><mi id="S3.E3.m1.1.2.3.2" xref="S3.E3.m1.1.2.3.2.cmml">η</mi><mo id="S3.E3.m1.1.2.3.1" lspace="0.222em" rspace="0.222em" xref="S3.E3.m1.1.2.3.1.cmml">⋅</mo><mfrac id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml"><mrow id="S3.E3.m1.1.1.1" xref="S3.E3.m1.1.1.1.cmml"><mi id="S3.E3.m1.1.1.1.3" xref="S3.E3.m1.1.1.1.3.cmml">E</mi><mo id="S3.E3.m1.1.1.1.2" lspace="0.222em" rspace="0.222em" xref="S3.E3.m1.1.1.1.2.cmml">⋅</mo><mrow id="S3.E3.m1.1.1.1.4.2" xref="S3.E3.m1.1.1.1.4.1.cmml"><mo id="S3.E3.m1.1.1.1.4.2.1" stretchy="false" xref="S3.E3.m1.1.1.1.4.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S3.E3.m1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.cmml">𝒟</mi><mo id="S3.E3.m1.1.1.1.4.2.2" stretchy="false" xref="S3.E3.m1.1.1.1.4.1.1.cmml">|</mo></mrow></mrow><mrow id="S3.E3.m1.1.1.3" xref="S3.E3.m1.1.1.3.cmml"><mi id="S3.E3.m1.1.1.3.2" xref="S3.E3.m1.1.1.3.2.cmml">B</mi><mo id="S3.E3.m1.1.1.3.1" lspace="0.222em" rspace="0.222em" xref="S3.E3.m1.1.1.3.1.cmml">⋅</mo><mi id="S3.E3.m1.1.1.3.3" xref="S3.E3.m1.1.1.3.3.cmml">K</mi></mrow></mfrac></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.1b"><apply id="S3.E3.m1.1.2.cmml" xref="S3.E3.m1.1.2"><eq id="S3.E3.m1.1.2.1.cmml" xref="S3.E3.m1.1.2.1"></eq><ci id="S3.E3.m1.1.2.2.cmml" xref="S3.E3.m1.1.2.2">𝑢</ci><apply id="S3.E3.m1.1.2.3.cmml" xref="S3.E3.m1.1.2.3"><ci id="S3.E3.m1.1.2.3.1.cmml" xref="S3.E3.m1.1.2.3.1">⋅</ci><ci id="S3.E3.m1.1.2.3.2.cmml" xref="S3.E3.m1.1.2.3.2">𝜂</ci><apply id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1"><divide id="S3.E3.m1.1.1.2.cmml" xref="S3.E3.m1.1.1"></divide><apply id="S3.E3.m1.1.1.1.cmml" xref="S3.E3.m1.1.1.1"><ci id="S3.E3.m1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.2">⋅</ci><ci id="S3.E3.m1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.3">𝐸</ci><apply id="S3.E3.m1.1.1.1.4.1.cmml" xref="S3.E3.m1.1.1.1.4.2"><abs id="S3.E3.m1.1.1.1.4.1.1.cmml" xref="S3.E3.m1.1.1.1.4.2.1"></abs><ci id="S3.E3.m1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1">𝒟</ci></apply></apply><apply id="S3.E3.m1.1.1.3.cmml" xref="S3.E3.m1.1.1.3"><ci id="S3.E3.m1.1.1.3.1.cmml" xref="S3.E3.m1.1.1.3.1">⋅</ci><ci id="S3.E3.m1.1.1.3.2.cmml" xref="S3.E3.m1.1.1.3.2">𝐵</ci><ci id="S3.E3.m1.1.1.3.3.cmml" xref="S3.E3.m1.1.1.3.3">𝐾</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.1c">u=\eta\cdot\frac{E\cdot|\mathcal{D}|}{B\cdot K}</annotation><annotation encoding="application/x-llamapun" id="S3.E3.m1.1d">italic_u = italic_η ⋅ divide start_ARG italic_E ⋅ | caligraphic_D | end_ARG start_ARG italic_B ⋅ italic_K end_ARG</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SS2.SSS2.p5"> <p class="ltx_p" id="S3.SS2.SSS2.p5.5">Therefore, local epoch <math alttext="E" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p5.1.m1.1"><semantics id="S3.SS2.SSS2.p5.1.m1.1a"><mi id="S3.SS2.SSS2.p5.1.m1.1.1" xref="S3.SS2.SSS2.p5.1.m1.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p5.1.m1.1b"><ci id="S3.SS2.SSS2.p5.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p5.1.m1.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p5.1.m1.1c">E</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p5.1.m1.1d">italic_E</annotation></semantics></math>, batch size <math alttext="B" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p5.2.m2.1"><semantics id="S3.SS2.SSS2.p5.2.m2.1a"><mi id="S3.SS2.SSS2.p5.2.m2.1.1" xref="S3.SS2.SSS2.p5.2.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p5.2.m2.1b"><ci id="S3.SS2.SSS2.p5.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p5.2.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p5.2.m2.1c">B</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p5.2.m2.1d">italic_B</annotation></semantics></math>, and learning rate <math alttext="\eta" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p5.3.m3.1"><semantics id="S3.SS2.SSS2.p5.3.m3.1a"><mi id="S3.SS2.SSS2.p5.3.m3.1.1" xref="S3.SS2.SSS2.p5.3.m3.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p5.3.m3.1b"><ci id="S3.SS2.SSS2.p5.3.m3.1.1.cmml" xref="S3.SS2.SSS2.p5.3.m3.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p5.3.m3.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p5.3.m3.1d">italic_η</annotation></semantics></math> must be adjusted based on each client’s dataset size <math alttext="\frac{|\mathcal{D}|}{K}" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p5.4.m4.1"><semantics id="S3.SS2.SSS2.p5.4.m4.1a"><mfrac id="S3.SS2.SSS2.p5.4.m4.1.1" xref="S3.SS2.SSS2.p5.4.m4.1.1.cmml"><mrow id="S3.SS2.SSS2.p5.4.m4.1.1.1.3" xref="S3.SS2.SSS2.p5.4.m4.1.1.1.2.cmml"><mo id="S3.SS2.SSS2.p5.4.m4.1.1.1.3.1" stretchy="false" xref="S3.SS2.SSS2.p5.4.m4.1.1.1.2.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS2.p5.4.m4.1.1.1.1" xref="S3.SS2.SSS2.p5.4.m4.1.1.1.1.cmml">𝒟</mi><mo id="S3.SS2.SSS2.p5.4.m4.1.1.1.3.2" stretchy="false" xref="S3.SS2.SSS2.p5.4.m4.1.1.1.2.1.cmml">|</mo></mrow><mi id="S3.SS2.SSS2.p5.4.m4.1.1.3" xref="S3.SS2.SSS2.p5.4.m4.1.1.3.cmml">K</mi></mfrac><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p5.4.m4.1b"><apply id="S3.SS2.SSS2.p5.4.m4.1.1.cmml" xref="S3.SS2.SSS2.p5.4.m4.1.1"><divide id="S3.SS2.SSS2.p5.4.m4.1.1.2.cmml" xref="S3.SS2.SSS2.p5.4.m4.1.1"></divide><apply id="S3.SS2.SSS2.p5.4.m4.1.1.1.2.cmml" xref="S3.SS2.SSS2.p5.4.m4.1.1.1.3"><abs id="S3.SS2.SSS2.p5.4.m4.1.1.1.2.1.cmml" xref="S3.SS2.SSS2.p5.4.m4.1.1.1.3.1"></abs><ci id="S3.SS2.SSS2.p5.4.m4.1.1.1.1.cmml" xref="S3.SS2.SSS2.p5.4.m4.1.1.1.1">𝒟</ci></apply><ci id="S3.SS2.SSS2.p5.4.m4.1.1.3.cmml" xref="S3.SS2.SSS2.p5.4.m4.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p5.4.m4.1c">\frac{|\mathcal{D}|}{K}</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p5.4.m4.1d">divide start_ARG | caligraphic_D | end_ARG start_ARG italic_K end_ARG</annotation></semantics></math> to ensure all scenarios achieve an equal <math alttext="u" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p5.5.m5.1"><semantics id="S3.SS2.SSS2.p5.5.m5.1a"><mi id="S3.SS2.SSS2.p5.5.m5.1.1" xref="S3.SS2.SSS2.p5.5.m5.1.1.cmml">u</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p5.5.m5.1b"><ci id="S3.SS2.SSS2.p5.5.m5.1.1.cmml" xref="S3.SS2.SSS2.p5.5.m5.1.1">𝑢</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p5.5.m5.1c">u</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p5.5.m5.1d">italic_u</annotation></semantics></math>.</p> </div> <figure class="ltx_figure" id="S3.F3"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="184" id="S3.F3.g1" src="x3.png" width="830"/> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Impact of matched <math alttext="B" class="ltx_Math" display="inline" id="S3.F3.5.m1.1"><semantics id="S3.F3.5.m1.1b"><mi id="S3.F3.5.m1.1.1" xref="S3.F3.5.m1.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.F3.5.m1.1c"><ci id="S3.F3.5.m1.1.1.cmml" xref="S3.F3.5.m1.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.5.m1.1d">B</annotation><annotation encoding="application/x-llamapun" id="S3.F3.5.m1.1e">italic_B</annotation></semantics></math>, <math alttext="E" class="ltx_Math" display="inline" id="S3.F3.6.m2.1"><semantics id="S3.F3.6.m2.1b"><mi id="S3.F3.6.m2.1.1" xref="S3.F3.6.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S3.F3.6.m2.1c"><ci id="S3.F3.6.m2.1.1.cmml" xref="S3.F3.6.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.6.m2.1d">E</annotation><annotation encoding="application/x-llamapun" id="S3.F3.6.m2.1e">italic_E</annotation></semantics></math>, and <math alttext="\eta" class="ltx_Math" display="inline" id="S3.F3.7.m3.1"><semantics id="S3.F3.7.m3.1b"><mi id="S3.F3.7.m3.1.1" xref="S3.F3.7.m3.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.F3.7.m3.1c"><ci id="S3.F3.7.m3.1.1.cmml" xref="S3.F3.7.m3.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.7.m3.1d">\eta</annotation><annotation encoding="application/x-llamapun" id="S3.F3.7.m3.1e">italic_η</annotation></semantics></math> on performance with different <math alttext="K" class="ltx_Math" display="inline" id="S3.F3.8.m4.1"><semantics id="S3.F3.8.m4.1b"><mi id="S3.F3.8.m4.1.1" xref="S3.F3.8.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.F3.8.m4.1c"><ci id="S3.F3.8.m4.1.1.cmml" xref="S3.F3.8.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.8.m4.1d">K</annotation><annotation encoding="application/x-llamapun" id="S3.F3.8.m4.1e">italic_K</annotation></semantics></math>.</figcaption> </figure> <div class="ltx_para" id="S3.SS2.SSS2.p6"> <p class="ltx_p" id="S3.SS2.SSS2.p6.9">In Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S3.F3" title="Figure 3 ‣ 3.2.2 Transition from Centralized to Federated Learning ‣ 3.2 FL Setup ‣ 3 Experimental Setup ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">3</span></a>, we present three accuracy charts corresponding to adjustments in <math alttext="B" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.1.m1.1"><semantics id="S3.SS2.SSS2.p6.1.m1.1a"><mi id="S3.SS2.SSS2.p6.1.m1.1.1" xref="S3.SS2.SSS2.p6.1.m1.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.1.m1.1b"><ci id="S3.SS2.SSS2.p6.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p6.1.m1.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.1.m1.1c">B</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.1.m1.1d">italic_B</annotation></semantics></math>, <math alttext="E" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.2.m2.1"><semantics id="S3.SS2.SSS2.p6.2.m2.1a"><mi id="S3.SS2.SSS2.p6.2.m2.1.1" xref="S3.SS2.SSS2.p6.2.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.2.m2.1b"><ci id="S3.SS2.SSS2.p6.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p6.2.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.2.m2.1c">E</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.2.m2.1d">italic_E</annotation></semantics></math>, and <math alttext="\eta" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.3.m3.1"><semantics id="S3.SS2.SSS2.p6.3.m3.1a"><mi id="S3.SS2.SSS2.p6.3.m3.1.1" xref="S3.SS2.SSS2.p6.3.m3.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.3.m3.1b"><ci id="S3.SS2.SSS2.p6.3.m3.1.1.cmml" xref="S3.SS2.SSS2.p6.3.m3.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.3.m3.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.3.m3.1d">italic_η</annotation></semantics></math>. Unlike the previous fixed hyperparameter setting, these adjustments align performance and convergence rates across different client counts. When <math alttext="E" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.4.m4.1"><semantics id="S3.SS2.SSS2.p6.4.m4.1a"><mi id="S3.SS2.SSS2.p6.4.m4.1.1" xref="S3.SS2.SSS2.p6.4.m4.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.4.m4.1b"><ci id="S3.SS2.SSS2.p6.4.m4.1.1.cmml" xref="S3.SS2.SSS2.p6.4.m4.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.4.m4.1c">E</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.4.m4.1d">italic_E</annotation></semantics></math> is increased to 50 and <math alttext="\eta" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.5.m5.1"><semantics id="S3.SS2.SSS2.p6.5.m5.1a"><mi id="S3.SS2.SSS2.p6.5.m5.1.1" xref="S3.SS2.SSS2.p6.5.m5.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.5.m5.1b"><ci id="S3.SS2.SSS2.p6.5.m5.1.1.cmml" xref="S3.SS2.SSS2.p6.5.m5.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.5.m5.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.5.m5.1d">italic_η</annotation></semantics></math> is set to 0.25, it results in an extreme case, which we disregard in our analysis. Unless specified otherwise, the base setting for all experiments is <math alttext="K=10" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.6.m6.1"><semantics id="S3.SS2.SSS2.p6.6.m6.1a"><mrow id="S3.SS2.SSS2.p6.6.m6.1.1" xref="S3.SS2.SSS2.p6.6.m6.1.1.cmml"><mi id="S3.SS2.SSS2.p6.6.m6.1.1.2" xref="S3.SS2.SSS2.p6.6.m6.1.1.2.cmml">K</mi><mo id="S3.SS2.SSS2.p6.6.m6.1.1.1" xref="S3.SS2.SSS2.p6.6.m6.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p6.6.m6.1.1.3" xref="S3.SS2.SSS2.p6.6.m6.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.6.m6.1b"><apply id="S3.SS2.SSS2.p6.6.m6.1.1.cmml" xref="S3.SS2.SSS2.p6.6.m6.1.1"><eq id="S3.SS2.SSS2.p6.6.m6.1.1.1.cmml" xref="S3.SS2.SSS2.p6.6.m6.1.1.1"></eq><ci id="S3.SS2.SSS2.p6.6.m6.1.1.2.cmml" xref="S3.SS2.SSS2.p6.6.m6.1.1.2">𝐾</ci><cn id="S3.SS2.SSS2.p6.6.m6.1.1.3.cmml" type="integer" xref="S3.SS2.SSS2.p6.6.m6.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.6.m6.1c">K=10</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.6.m6.1d">italic_K = 10</annotation></semantics></math>, <math alttext="E=1" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.7.m7.1"><semantics id="S3.SS2.SSS2.p6.7.m7.1a"><mrow id="S3.SS2.SSS2.p6.7.m7.1.1" xref="S3.SS2.SSS2.p6.7.m7.1.1.cmml"><mi id="S3.SS2.SSS2.p6.7.m7.1.1.2" xref="S3.SS2.SSS2.p6.7.m7.1.1.2.cmml">E</mi><mo id="S3.SS2.SSS2.p6.7.m7.1.1.1" xref="S3.SS2.SSS2.p6.7.m7.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p6.7.m7.1.1.3" xref="S3.SS2.SSS2.p6.7.m7.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.7.m7.1b"><apply id="S3.SS2.SSS2.p6.7.m7.1.1.cmml" xref="S3.SS2.SSS2.p6.7.m7.1.1"><eq id="S3.SS2.SSS2.p6.7.m7.1.1.1.cmml" xref="S3.SS2.SSS2.p6.7.m7.1.1.1"></eq><ci id="S3.SS2.SSS2.p6.7.m7.1.1.2.cmml" xref="S3.SS2.SSS2.p6.7.m7.1.1.2">𝐸</ci><cn id="S3.SS2.SSS2.p6.7.m7.1.1.3.cmml" type="integer" xref="S3.SS2.SSS2.p6.7.m7.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.7.m7.1c">E=1</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.7.m7.1d">italic_E = 1</annotation></semantics></math>, <math alttext="B=50" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.8.m8.1"><semantics id="S3.SS2.SSS2.p6.8.m8.1a"><mrow id="S3.SS2.SSS2.p6.8.m8.1.1" xref="S3.SS2.SSS2.p6.8.m8.1.1.cmml"><mi id="S3.SS2.SSS2.p6.8.m8.1.1.2" xref="S3.SS2.SSS2.p6.8.m8.1.1.2.cmml">B</mi><mo id="S3.SS2.SSS2.p6.8.m8.1.1.1" xref="S3.SS2.SSS2.p6.8.m8.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p6.8.m8.1.1.3" xref="S3.SS2.SSS2.p6.8.m8.1.1.3.cmml">50</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.8.m8.1b"><apply id="S3.SS2.SSS2.p6.8.m8.1.1.cmml" xref="S3.SS2.SSS2.p6.8.m8.1.1"><eq id="S3.SS2.SSS2.p6.8.m8.1.1.1.cmml" xref="S3.SS2.SSS2.p6.8.m8.1.1.1"></eq><ci id="S3.SS2.SSS2.p6.8.m8.1.1.2.cmml" xref="S3.SS2.SSS2.p6.8.m8.1.1.2">𝐵</ci><cn id="S3.SS2.SSS2.p6.8.m8.1.1.3.cmml" type="integer" xref="S3.SS2.SSS2.p6.8.m8.1.1.3">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.8.m8.1c">B=50</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.8.m8.1d">italic_B = 50</annotation></semantics></math>, and <math alttext="\eta=0.005" class="ltx_Math" display="inline" id="S3.SS2.SSS2.p6.9.m9.1"><semantics id="S3.SS2.SSS2.p6.9.m9.1a"><mrow id="S3.SS2.SSS2.p6.9.m9.1.1" xref="S3.SS2.SSS2.p6.9.m9.1.1.cmml"><mi id="S3.SS2.SSS2.p6.9.m9.1.1.2" xref="S3.SS2.SSS2.p6.9.m9.1.1.2.cmml">η</mi><mo id="S3.SS2.SSS2.p6.9.m9.1.1.1" xref="S3.SS2.SSS2.p6.9.m9.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p6.9.m9.1.1.3" xref="S3.SS2.SSS2.p6.9.m9.1.1.3.cmml">0.005</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p6.9.m9.1b"><apply id="S3.SS2.SSS2.p6.9.m9.1.1.cmml" xref="S3.SS2.SSS2.p6.9.m9.1.1"><eq id="S3.SS2.SSS2.p6.9.m9.1.1.1.cmml" xref="S3.SS2.SSS2.p6.9.m9.1.1.1"></eq><ci id="S3.SS2.SSS2.p6.9.m9.1.1.2.cmml" xref="S3.SS2.SSS2.p6.9.m9.1.1.2">𝜂</ci><cn id="S3.SS2.SSS2.p6.9.m9.1.1.3.cmml" type="float" xref="S3.SS2.SSS2.p6.9.m9.1.1.3">0.005</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p6.9.m9.1c">\eta=0.005</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.SSS2.p6.9.m9.1d">italic_η = 0.005</annotation></semantics></math>, with variations applied to a single parameter in each experiment.</p> </div> </section> </section> </section> <section class="ltx_section" id="S4"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">4 </span>Experimental Results</h2> <section class="ltx_subsection" id="S4.SS1"> <h3 class="ltx_title ltx_title_subsection"> <span class="ltx_tag ltx_tag_subsection">4.1 </span>Impact of Hyperparameters under IID Condition</h3> <figure class="ltx_figure" id="S4.F4"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="369" id="S4.F4.g1" src="x4.png" width="830"/> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Training dynamics when <math alttext="K=10" class="ltx_Math" display="inline" id="S4.F4.5.m1.1"><semantics id="S4.F4.5.m1.1b"><mrow id="S4.F4.5.m1.1.1" xref="S4.F4.5.m1.1.1.cmml"><mi id="S4.F4.5.m1.1.1.2" xref="S4.F4.5.m1.1.1.2.cmml">K</mi><mo id="S4.F4.5.m1.1.1.1" xref="S4.F4.5.m1.1.1.1.cmml">=</mo><mn id="S4.F4.5.m1.1.1.3" xref="S4.F4.5.m1.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.F4.5.m1.1c"><apply id="S4.F4.5.m1.1.1.cmml" xref="S4.F4.5.m1.1.1"><eq id="S4.F4.5.m1.1.1.1.cmml" xref="S4.F4.5.m1.1.1.1"></eq><ci id="S4.F4.5.m1.1.1.2.cmml" xref="S4.F4.5.m1.1.1.2">𝐾</ci><cn id="S4.F4.5.m1.1.1.3.cmml" type="integer" xref="S4.F4.5.m1.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.5.m1.1d">K=10</annotation><annotation encoding="application/x-llamapun" id="S4.F4.5.m1.1e">italic_K = 10</annotation></semantics></math>. The first row shows results for varying <math alttext="E" class="ltx_Math" display="inline" id="S4.F4.6.m2.1"><semantics id="S4.F4.6.m2.1b"><mi id="S4.F4.6.m2.1.1" xref="S4.F4.6.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S4.F4.6.m2.1c"><ci id="S4.F4.6.m2.1.1.cmml" xref="S4.F4.6.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.6.m2.1d">E</annotation><annotation encoding="application/x-llamapun" id="S4.F4.6.m2.1e">italic_E</annotation></semantics></math>, the second for different <math alttext="B" class="ltx_Math" display="inline" id="S4.F4.7.m3.1"><semantics id="S4.F4.7.m3.1b"><mi id="S4.F4.7.m3.1.1" xref="S4.F4.7.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S4.F4.7.m3.1c"><ci id="S4.F4.7.m3.1.1.cmml" xref="S4.F4.7.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.7.m3.1d">B</annotation><annotation encoding="application/x-llamapun" id="S4.F4.7.m3.1e">italic_B</annotation></semantics></math>, and the third for various <math alttext="\eta" class="ltx_Math" display="inline" id="S4.F4.8.m4.1"><semantics id="S4.F4.8.m4.1b"><mi id="S4.F4.8.m4.1.1" xref="S4.F4.8.m4.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S4.F4.8.m4.1c"><ci id="S4.F4.8.m4.1.1.cmml" xref="S4.F4.8.m4.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.8.m4.1d">\eta</annotation><annotation encoding="application/x-llamapun" id="S4.F4.8.m4.1e">italic_η</annotation></semantics></math>.</figcaption> </figure> <div class="ltx_para" id="S4.SS1.p1"> <p class="ltx_p" id="S4.SS1.p1.7">We first assess how three hyperparameters—<math alttext="E" class="ltx_Math" display="inline" id="S4.SS1.p1.1.m1.1"><semantics id="S4.SS1.p1.1.m1.1a"><mi id="S4.SS1.p1.1.m1.1.1" xref="S4.SS1.p1.1.m1.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.1.m1.1b"><ci id="S4.SS1.p1.1.m1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.1.m1.1c">E</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.1.m1.1d">italic_E</annotation></semantics></math>, <math alttext="\eta" class="ltx_Math" display="inline" id="S4.SS1.p1.2.m2.1"><semantics id="S4.SS1.p1.2.m2.1a"><mi id="S4.SS1.p1.2.m2.1.1" xref="S4.SS1.p1.2.m2.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.2.m2.1b"><ci id="S4.SS1.p1.2.m2.1.1.cmml" xref="S4.SS1.p1.2.m2.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.2.m2.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.2.m2.1d">italic_η</annotation></semantics></math>, and <math alttext="B" class="ltx_Math" display="inline" id="S4.SS1.p1.3.m3.1"><semantics id="S4.SS1.p1.3.m3.1a"><mi id="S4.SS1.p1.3.m3.1.1" xref="S4.SS1.p1.3.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.3.m3.1b"><ci id="S4.SS1.p1.3.m3.1.1.cmml" xref="S4.SS1.p1.3.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.3.m3.1c">B</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.3.m3.1d">italic_B</annotation></semantics></math>—affect test accuracy, test loss, and training loss. As illustrated in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F4" title="Figure 4 ‣ 4.1 Impact of Hyperparameters under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">4</span></a>, increasing <math alttext="E" class="ltx_Math" display="inline" id="S4.SS1.p1.4.m4.1"><semantics id="S4.SS1.p1.4.m4.1a"><mi id="S4.SS1.p1.4.m4.1.1" xref="S4.SS1.p1.4.m4.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.4.m4.1b"><ci id="S4.SS1.p1.4.m4.1.1.cmml" xref="S4.SS1.p1.4.m4.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.4.m4.1c">E</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.4.m4.1d">italic_E</annotation></semantics></math> or <math alttext="\eta" class="ltx_Math" display="inline" id="S4.SS1.p1.5.m5.1"><semantics id="S4.SS1.p1.5.m5.1a"><mi id="S4.SS1.p1.5.m5.1.1" xref="S4.SS1.p1.5.m5.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.5.m5.1b"><ci id="S4.SS1.p1.5.m5.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.5.m5.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.5.m5.1d">italic_η</annotation></semantics></math>, or decreasing <math alttext="B" class="ltx_Math" display="inline" id="S4.SS1.p1.6.m6.1"><semantics id="S4.SS1.p1.6.m6.1a"><mi id="S4.SS1.p1.6.m6.1.1" xref="S4.SS1.p1.6.m6.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.6.m6.1b"><ci id="S4.SS1.p1.6.m6.1.1.cmml" xref="S4.SS1.p1.6.m6.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.6.m6.1c">B</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.6.m6.1d">italic_B</annotation></semantics></math>, speeds up convergence. This is evident from the curvature of the accuracy plots and the more rapid decline in training loss, which is consistent with our update amount equation <math alttext="u" class="ltx_Math" display="inline" id="S4.SS1.p1.7.m7.1"><semantics id="S4.SS1.p1.7.m7.1a"><mi id="S4.SS1.p1.7.m7.1.1" xref="S4.SS1.p1.7.m7.1.1.cmml">u</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.7.m7.1b"><ci id="S4.SS1.p1.7.m7.1.1.cmml" xref="S4.SS1.p1.7.m7.1.1">𝑢</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.7.m7.1c">u</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.7.m7.1d">italic_u</annotation></semantics></math> (Eq. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S3.E3" title="In 3.2.2 Transition from Centralized to Federated Learning ‣ 3.2 FL Setup ‣ 3 Experimental Setup ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">3</span></a>). Notably, the point of overfitting occurs earlier with faster convergence, as marked by the minimum test loss (indicated by the circle).</p> </div> <div class="ltx_para" id="S4.SS1.p2"> <p class="ltx_p" id="S4.SS1.p2.4">However, some notable observations can be made. While higher <math alttext="E" class="ltx_Math" display="inline" id="S4.SS1.p2.1.m1.1"><semantics id="S4.SS1.p2.1.m1.1a"><mi id="S4.SS1.p2.1.m1.1.1" xref="S4.SS1.p2.1.m1.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.1.m1.1b"><ci id="S4.SS1.p2.1.m1.1.1.cmml" xref="S4.SS1.p2.1.m1.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.1.m1.1c">E</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.1.m1.1d">italic_E</annotation></semantics></math> values result in faster convergence, they lead to suboptimal minimum test loss and final accuracy, with lower epochs (e.g., <math alttext="E=1" class="ltx_Math" display="inline" id="S4.SS1.p2.2.m2.1"><semantics id="S4.SS1.p2.2.m2.1a"><mrow id="S4.SS1.p2.2.m2.1.1" xref="S4.SS1.p2.2.m2.1.1.cmml"><mi id="S4.SS1.p2.2.m2.1.1.2" xref="S4.SS1.p2.2.m2.1.1.2.cmml">E</mi><mo id="S4.SS1.p2.2.m2.1.1.1" xref="S4.SS1.p2.2.m2.1.1.1.cmml">=</mo><mn id="S4.SS1.p2.2.m2.1.1.3" xref="S4.SS1.p2.2.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.2.m2.1b"><apply id="S4.SS1.p2.2.m2.1.1.cmml" xref="S4.SS1.p2.2.m2.1.1"><eq id="S4.SS1.p2.2.m2.1.1.1.cmml" xref="S4.SS1.p2.2.m2.1.1.1"></eq><ci id="S4.SS1.p2.2.m2.1.1.2.cmml" xref="S4.SS1.p2.2.m2.1.1.2">𝐸</ci><cn id="S4.SS1.p2.2.m2.1.1.3.cmml" type="integer" xref="S4.SS1.p2.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.2.m2.1c">E=1</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.2.m2.1d">italic_E = 1</annotation></semantics></math>) providing better performance (first row in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F4" title="Figure 4 ‣ 4.1 Impact of Hyperparameters under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">4</span></a>). In contrast, the smallest batch size (<math alttext="B=5" class="ltx_Math" display="inline" id="S4.SS1.p2.3.m3.1"><semantics id="S4.SS1.p2.3.m3.1a"><mrow id="S4.SS1.p2.3.m3.1.1" xref="S4.SS1.p2.3.m3.1.1.cmml"><mi id="S4.SS1.p2.3.m3.1.1.2" xref="S4.SS1.p2.3.m3.1.1.2.cmml">B</mi><mo id="S4.SS1.p2.3.m3.1.1.1" xref="S4.SS1.p2.3.m3.1.1.1.cmml">=</mo><mn id="S4.SS1.p2.3.m3.1.1.3" xref="S4.SS1.p2.3.m3.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.3.m3.1b"><apply id="S4.SS1.p2.3.m3.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1"><eq id="S4.SS1.p2.3.m3.1.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1.1"></eq><ci id="S4.SS1.p2.3.m3.1.1.2.cmml" xref="S4.SS1.p2.3.m3.1.1.2">𝐵</ci><cn id="S4.SS1.p2.3.m3.1.1.3.cmml" type="integer" xref="S4.SS1.p2.3.m3.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.3.m3.1c">B=5</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.3.m3.1d">italic_B = 5</annotation></semantics></math>) achieves the highest final accuracy and the lowest test loss (second row in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F4" title="Figure 4 ‣ 4.1 Impact of Hyperparameters under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">4</span></a>). Similarly, higher <math alttext="\eta" class="ltx_Math" display="inline" id="S4.SS1.p2.4.m4.1"><semantics id="S4.SS1.p2.4.m4.1a"><mi id="S4.SS1.p2.4.m4.1.1" xref="S4.SS1.p2.4.m4.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.4.m4.1b"><ci id="S4.SS1.p2.4.m4.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.4.m4.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.4.m4.1d">italic_η</annotation></semantics></math> produce the best final accuracy and the lowest test loss (third row in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F4" title="Figure 4 ‣ 4.1 Impact of Hyperparameters under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">4</span></a>) while both of them provide faster convergence. Although the differences are not substantial under IID conditions, they become more pronounced in non-IID settings, which we will discuss later.</p> </div> </section> <section class="ltx_subsection" id="S4.SS2"> <h3 class="ltx_title ltx_title_subsection"> <span class="ltx_tag ltx_tag_subsection">4.2 </span>Partial Participation and Imbalanced Data under IID Condition</h3> <div class="ltx_para" id="S4.SS2.p1"> <p class="ltx_p" id="S4.SS2.p1.1">This section explores the effects of partial client participation and imbalanced data in the context of FL under IID conditions. While these are not hyperparameters, they are key aspects of the experimental setup. In FL, the server cannot always ensure that all clients participate in every training round. As a result, the server either waits for all clients or proceeds with only those that respond in time, leading to partial participation (PP), where only a subset of clients contribute to model updates in each round.</p> </div> <div class="ltx_para" id="S4.SS2.p2"> <p class="ltx_p" id="S4.SS2.p2.1">Even under IID conditions, where each client has an equal proportion of labels, the number of data samples per client can vary, creating imbalanced datasets. Some clients may have significantly more data than others.</p> </div> <div class="ltx_para" id="S4.SS2.p3"> <p class="ltx_p" id="S4.SS2.p3.1">We now examine how partial participation and data imbalance influence training dynamics, particularly with regard to convergence and model performance.</p> </div> <section class="ltx_subsubsection" id="S4.SS2.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> <span class="ltx_tag ltx_tag_subsubsection">4.2.1 </span>Impact of Partial Participation</h4> <div class="ltx_para" id="S4.SS2.SSS1.p1"> <p class="ltx_p" id="S4.SS2.SSS1.p1.1">To evaluate the impact of partial participation (PP), we vary the number of clients involved in each communication round, randomly selecting 1, 2, 5, or 10 out of a total of 10 clients. While one might assume that involving more clients per round would result in better performance, our experiments under IID conditions reveal minimal performance differences across varying levels of participation, as shown in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F5" title="Figure 5 ‣ 4.2.1 Impact of Partial Participation ‣ 4.2 Partial Participation and Imbalanced Data under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">5</span></a>.</p> </div> <figure class="ltx_figure" id="S4.F5"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="138" id="S4.F5.g1" src="x5.png" width="830"/> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Training Dynamics under Partial Participation with IID Condition</figcaption> </figure> <div class="ltx_para" id="S4.SS2.SSS1.p2"> <p class="ltx_p" id="S4.SS2.SSS1.p2.1">This result suggests that in the IID setting, an update from a single client remains valid for others, allowing them to resume training locally from that state without significant performance degradation. Although fewer clients participating per round introduce more fluctuations, the final accuracy still converges to nearly the same level.</p> </div> </section> <section class="ltx_subsubsection" id="S4.SS2.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> <span class="ltx_tag ltx_tag_subsubsection">4.2.2 </span>Impact of Imbalanced Data.</h4> <div class="ltx_para" id="S4.SS2.SSS2.p1"> <p class="ltx_p" id="S4.SS2.SSS2.p1.1">In our experiments with imbalanced datasets, we construct the environment using a Standard Gaussian Mixture (SGM) model. Here, an SGM value of 0 corresponds to a balanced dataset, while higher SGM values indicate increasing levels of data imbalance, as illustrated in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F6" title="Figure 6 ‣ 4.2.2 Impact of Imbalanced Data. ‣ 4.2 Partial Participation and Imbalanced Data under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">6</span></a>.</p> </div> <div class="ltx_pagination ltx_role_newpage"></div> <figure class="ltx_figure" id="S4.F6"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="180" id="S4.F6.g1" src="x6.png" width="830"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Data partitioning with IID labels across different levels of SGM</figcaption> </figure> <div class="ltx_para" id="S4.SS2.SSS2.p2"> <p class="ltx_p" id="S4.SS2.SSS2.p2.1">For imbalanced datasets, we explore two aggregation strategies: <span class="ltx_text ltx_font_bold" id="S4.SS2.SSS2.p2.1.1">weighted aggregation</span>, used in FedAvg, where the contribution of each client is proportional to its dataset size, and <span class="ltx_text ltx_font_bold" id="S4.SS2.SSS2.p2.1.2">naive aggregation</span>, where updates from all clients are averaged equally, regardless of dataset size.</p> </div> <figure class="ltx_figure" id="S4.F7"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="230" id="S4.F7.g1" src="x7.png" width="830"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Performance under varying levels of data imbalance. The first row shows results using weighted aggregation, while the second row uses naive averaging.</figcaption> </figure> <div class="ltx_para" id="S4.SS2.SSS2.p3"> <p class="ltx_p" id="S4.SS2.SSS2.p3.2">Contrary to intuitive expectations that increased imbalance would degrade performance, our experiments reveal that <span class="ltx_text ltx_font_bold" id="S4.SS2.SSS2.p3.2.1">weighted aggregation</span> actually accelerates convergence in IID settings. As shown in the first row of Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F7" title="Figure 7 ‣ 4.2.2 Impact of Imbalanced Data. ‣ 4.2 Partial Participation and Imbalanced Data under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">7</span></a>, models converge faster as imbalance increases, especially for SGM 0.9 (purple), which shows higher accuracy, earlier overfitting, and lower training loss between rounds 40-70. This is because, in weighted aggregation, clients with larger datasets exert more influence, effectively increasing the total number of updates. For instance, when Client A has 10 data points and Client B has 90, the weighted average of updates is <math alttext="\frac{10\times 10+90\times 90}{100}=82" class="ltx_Math" display="inline" id="S4.SS2.SSS2.p3.1.m1.1"><semantics id="S4.SS2.SSS2.p3.1.m1.1a"><mrow id="S4.SS2.SSS2.p3.1.m1.1.1" xref="S4.SS2.SSS2.p3.1.m1.1.1.cmml"><mfrac id="S4.SS2.SSS2.p3.1.m1.1.1.2" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.cmml"><mrow id="S4.SS2.SSS2.p3.1.m1.1.1.2.2" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.cmml"><mrow id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.cmml"><mn id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.2" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.2.cmml">10</mn><mo id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.1" lspace="0.222em" rspace="0.222em" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.1.cmml">×</mo><mn id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.3" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.3.cmml">10</mn></mrow><mo id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.1" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.1.cmml">+</mo><mrow id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.cmml"><mn id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.2" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.2.cmml">90</mn><mo id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.1" lspace="0.222em" rspace="0.222em" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.1.cmml">×</mo><mn id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.3" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.3.cmml">90</mn></mrow></mrow><mn id="S4.SS2.SSS2.p3.1.m1.1.1.2.3" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.3.cmml">100</mn></mfrac><mo id="S4.SS2.SSS2.p3.1.m1.1.1.1" xref="S4.SS2.SSS2.p3.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS2.p3.1.m1.1.1.3" xref="S4.SS2.SSS2.p3.1.m1.1.1.3.cmml">82</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS2.p3.1.m1.1b"><apply id="S4.SS2.SSS2.p3.1.m1.1.1.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1"><eq id="S4.SS2.SSS2.p3.1.m1.1.1.1.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.1"></eq><apply id="S4.SS2.SSS2.p3.1.m1.1.1.2.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.2"><divide id="S4.SS2.SSS2.p3.1.m1.1.1.2.1.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.2"></divide><apply id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2"><plus id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.1.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.1"></plus><apply id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2"><times id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.1.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.1"></times><cn id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.2.cmml" type="integer" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.2">10</cn><cn id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.3.cmml" type="integer" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.2.3">10</cn></apply><apply id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3"><times id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.1.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.1"></times><cn id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.2.cmml" type="integer" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.2">90</cn><cn id="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.3.cmml" type="integer" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.2.3.3">90</cn></apply></apply><cn id="S4.SS2.SSS2.p3.1.m1.1.1.2.3.cmml" type="integer" xref="S4.SS2.SSS2.p3.1.m1.1.1.2.3">100</cn></apply><cn id="S4.SS2.SSS2.p3.1.m1.1.1.3.cmml" type="integer" xref="S4.SS2.SSS2.p3.1.m1.1.1.3">82</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS2.p3.1.m1.1c">\frac{10\times 10+90\times 90}{100}=82</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.SSS2.p3.1.m1.1d">divide start_ARG 10 × 10 + 90 × 90 end_ARG start_ARG 100 end_ARG = 82</annotation></semantics></math>. In contrast, with balanced datasets (50 data points each), the average number of updates is <math alttext="\frac{50\times 50+50\times 50}{100}=50" class="ltx_Math" display="inline" id="S4.SS2.SSS2.p3.2.m2.1"><semantics id="S4.SS2.SSS2.p3.2.m2.1a"><mrow id="S4.SS2.SSS2.p3.2.m2.1.1" xref="S4.SS2.SSS2.p3.2.m2.1.1.cmml"><mfrac id="S4.SS2.SSS2.p3.2.m2.1.1.2" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.cmml"><mrow id="S4.SS2.SSS2.p3.2.m2.1.1.2.2" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.cmml"><mrow id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.cmml"><mn id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.2" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.2.cmml">50</mn><mo id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.1" lspace="0.222em" rspace="0.222em" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.1.cmml">×</mo><mn id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.3" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.3.cmml">50</mn></mrow><mo id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.1" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.1.cmml">+</mo><mrow id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.cmml"><mn id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.2" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.2.cmml">50</mn><mo id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.1" lspace="0.222em" rspace="0.222em" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.1.cmml">×</mo><mn id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.3" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.3.cmml">50</mn></mrow></mrow><mn id="S4.SS2.SSS2.p3.2.m2.1.1.2.3" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.3.cmml">100</mn></mfrac><mo id="S4.SS2.SSS2.p3.2.m2.1.1.1" xref="S4.SS2.SSS2.p3.2.m2.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS2.p3.2.m2.1.1.3" xref="S4.SS2.SSS2.p3.2.m2.1.1.3.cmml">50</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS2.p3.2.m2.1b"><apply id="S4.SS2.SSS2.p3.2.m2.1.1.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1"><eq id="S4.SS2.SSS2.p3.2.m2.1.1.1.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.1"></eq><apply id="S4.SS2.SSS2.p3.2.m2.1.1.2.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.2"><divide id="S4.SS2.SSS2.p3.2.m2.1.1.2.1.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.2"></divide><apply id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2"><plus id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.1.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.1"></plus><apply id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2"><times id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.1.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.1"></times><cn id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.2.cmml" type="integer" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.2">50</cn><cn id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.3.cmml" type="integer" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.2.3">50</cn></apply><apply id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3"><times id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.1.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.1"></times><cn id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.2.cmml" type="integer" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.2">50</cn><cn id="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.3.cmml" type="integer" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.2.3.3">50</cn></apply></apply><cn id="S4.SS2.SSS2.p3.2.m2.1.1.2.3.cmml" type="integer" xref="S4.SS2.SSS2.p3.2.m2.1.1.2.3">100</cn></apply><cn id="S4.SS2.SSS2.p3.2.m2.1.1.3.cmml" type="integer" xref="S4.SS2.SSS2.p3.2.m2.1.1.3">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS2.p3.2.m2.1c">\frac{50\times 50+50\times 50}{100}=50</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.SSS2.p3.2.m2.1d">divide start_ARG 50 × 50 + 50 × 50 end_ARG start_ARG 100 end_ARG = 50</annotation></semantics></math>. Thus, in imbalanced scenarios, the total number of updates per round is higher, leading to faster convergence. Furthermore, when data is IID, the imbalance does not negatively impact performance—in fact, it can be advantageous.</p> </div> <div class="ltx_para" id="S4.SS2.SSS2.p4"> <p class="ltx_p" id="S4.SS2.SSS2.p4.1">We also conducted experiments using <span class="ltx_text ltx_font_bold" id="S4.SS2.SSS2.p4.1.1">naive aggregation</span>, which showed minimal performance differences across varying levels of imbalance, as shown in the second row of Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F7" title="Figure 7 ‣ 4.2.2 Impact of Imbalanced Data. ‣ 4.2 Partial Participation and Imbalanced Data under IID Condition ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">7</span></a>. This outcome can be understood by considering the number of updates: whether the dataset sizes are imbalanced (10 and 90) or balanced (50 and 50), naive averaging still results in an equal average of 50 updates per round. Interpreting updates as vectors, we can infer that when data is IID, the update vectors are aligned in direction, meaning that as long as the magnitudes are adjusted appropriately, the combined update remains the same. This suggests that in IID settings, updates from individual clients are consistently aligned in direction.</p> </div> </section> </section> <section class="ltx_subsection" id="S4.SS3"> <h3 class="ltx_title ltx_title_subsection"> <span class="ltx_tag ltx_tag_subsection">4.3 </span>Training Dynamics Under Non-IID Setting</h3> <section class="ltx_subsubsection" id="S4.SS3.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> <span class="ltx_tag ltx_tag_subsubsection">4.3.1 </span>Non-IID with Dirichlet Distribution</h4> <div class="ltx_para" id="S4.SS3.SSS1.p1"> <p class="ltx_p" id="S4.SS3.SSS1.p1.3">Next, we experiment non-IID label distribution with a Dirichlet distribution for data partitioning across clients, which introduces varying degrees of non-IID behavior. In this setup, the degree of non-IID distribution is controlled by the Dirichlet <math alttext="\alpha" class="ltx_Math" display="inline" id="S4.SS3.SSS1.p1.1.m1.1"><semantics id="S4.SS3.SSS1.p1.1.m1.1a"><mi id="S4.SS3.SSS1.p1.1.m1.1.1" xref="S4.SS3.SSS1.p1.1.m1.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS1.p1.1.m1.1b"><ci id="S4.SS3.SSS1.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS1.p1.1.m1.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS1.p1.1.m1.1c">\alpha</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS1.p1.1.m1.1d">italic_α</annotation></semantics></math>: a lower <math alttext="\alpha" class="ltx_Math" display="inline" id="S4.SS3.SSS1.p1.2.m2.1"><semantics id="S4.SS3.SSS1.p1.2.m2.1a"><mi id="S4.SS3.SSS1.p1.2.m2.1.1" xref="S4.SS3.SSS1.p1.2.m2.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS1.p1.2.m2.1b"><ci id="S4.SS3.SSS1.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS1.p1.2.m2.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS1.p1.2.m2.1c">\alpha</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS1.p1.2.m2.1d">italic_α</annotation></semantics></math> leads to more distinct distributions across clients, while a higher <math alttext="\alpha" class="ltx_Math" display="inline" id="S4.SS3.SSS1.p1.3.m3.1"><semantics id="S4.SS3.SSS1.p1.3.m3.1a"><mi id="S4.SS3.SSS1.p1.3.m3.1.1" xref="S4.SS3.SSS1.p1.3.m3.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS1.p1.3.m3.1b"><ci id="S4.SS3.SSS1.p1.3.m3.1.1.cmml" xref="S4.SS3.SSS1.p1.3.m3.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS1.p1.3.m3.1c">\alpha</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS1.p1.3.m3.1d">italic_α</annotation></semantics></math> results in more similar distributions. The resulting distributions are visualized in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F8" title="Figure 8 ‣ 4.3.1 Non-IID with Dirichlet Distribution ‣ 4.3 Training Dynamics Under Non-IID Setting ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">8</span></a>.</p> </div> <figure class="ltx_figure" id="S4.F8"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="175" id="S4.F8.g1" src="x8.png" width="830"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Data partitioning with varying Dirichlet parameter <math alttext="\alpha" class="ltx_Math" display="inline" id="S4.F8.2.m1.1"><semantics id="S4.F8.2.m1.1b"><mi id="S4.F8.2.m1.1.1" xref="S4.F8.2.m1.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.F8.2.m1.1c"><ci id="S4.F8.2.m1.1.1.cmml" xref="S4.F8.2.m1.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F8.2.m1.1d">\alpha</annotation><annotation encoding="application/x-llamapun" id="S4.F8.2.m1.1e">italic_α</annotation></semantics></math>.</figcaption> </figure> <div class="ltx_para" id="S4.SS3.SSS1.p2"> <p class="ltx_p" id="S4.SS3.SSS1.p2.1">In Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F9" title="Figure 9 ‣ 4.3.1 Non-IID with Dirichlet Distribution ‣ 4.3 Training Dynamics Under Non-IID Setting ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">9</span></a>, each line represents the average of three independent runs, with the shaded region indicating the min-max range across these runs. This approach accounts for the randomness introduced by the non-IID distribution of the dataset. Additionally, in the case of PP, the selection of clients in each round can have a significant impact on the results.</p> </div> <figure class="ltx_figure" id="S4.F9"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="230" id="S4.F9.g1" src="x9.png" width="830"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Performance across varying <math alttext="\alpha" class="ltx_Math" display="inline" id="S4.F9.2.m1.1"><semantics id="S4.F9.2.m1.1b"><mi id="S4.F9.2.m1.1.1" xref="S4.F9.2.m1.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.F9.2.m1.1c"><ci id="S4.F9.2.m1.1.1.cmml" xref="S4.F9.2.m1.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F9.2.m1.1d">\alpha</annotation><annotation encoding="application/x-llamapun" id="S4.F9.2.m1.1e">italic_α</annotation></semantics></math> (Top) and PP (Bottom).</figcaption> </figure> <div class="ltx_para" id="S4.SS3.SSS1.p3"> <p class="ltx_p" id="S4.SS3.SSS1.p3.3">We observe that as the data distribution becomes more non-IID (i.e., as <math alttext="\alpha" class="ltx_Math" display="inline" id="S4.SS3.SSS1.p3.1.m1.1"><semantics id="S4.SS3.SSS1.p3.1.m1.1a"><mi id="S4.SS3.SSS1.p3.1.m1.1.1" xref="S4.SS3.SSS1.p3.1.m1.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS1.p3.1.m1.1b"><ci id="S4.SS3.SSS1.p3.1.m1.1.1.cmml" xref="S4.SS3.SSS1.p3.1.m1.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS1.p3.1.m1.1c">\alpha</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS1.p3.1.m1.1d">italic_α</annotation></semantics></math> decreases), model performance degrades (from 69% to 55%) and convergence slows. While lower <math alttext="\alpha" class="ltx_Math" display="inline" id="S4.SS3.SSS1.p3.2.m2.1"><semantics id="S4.SS3.SSS1.p3.2.m2.1a"><mi id="S4.SS3.SSS1.p3.2.m2.1.1" xref="S4.SS3.SSS1.p3.2.m2.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS1.p3.2.m2.1b"><ci id="S4.SS3.SSS1.p3.2.m2.1.1.cmml" xref="S4.SS3.SSS1.p3.2.m2.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS1.p3.2.m2.1c">\alpha</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS1.p3.2.m2.1d">italic_α</annotation></semantics></math> values do eventually converge within the limited communication rounds, the resulting solutions are highly suboptimal. Furthermore, unlike in IID settings, the level of PP plays a critical role in FL performance. When <math alttext="\alpha" class="ltx_Math" display="inline" id="S4.SS3.SSS1.p3.3.m3.1"><semantics id="S4.SS3.SSS1.p3.3.m3.1a"><mi id="S4.SS3.SSS1.p3.3.m3.1.1" xref="S4.SS3.SSS1.p3.3.m3.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS1.p3.3.m3.1b"><ci id="S4.SS3.SSS1.p3.3.m3.1.1.cmml" xref="S4.SS3.SSS1.p3.3.m3.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS1.p3.3.m3.1c">\alpha</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS1.p3.3.m3.1d">italic_α</annotation></semantics></math> is fixed at 0.1, lower levels of PP lead to significantly lower final accuracy, higher test loss, and increased fluctuations compared to full participation. This indicates that updates from different clients are not valid for others.</p> </div> <div class="ltx_pagination ltx_role_newpage"></div> </section> <section class="ltx_subsubsection" id="S4.SS3.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> <span class="ltx_tag ltx_tag_subsubsection">4.3.2 </span>Impact of Hyperparameters under Non-IID Condition</h4> <div class="ltx_para" id="S4.SS3.SSS2.p1"> <p class="ltx_p" id="S4.SS3.SSS2.p1.1">We investigate the effects of varying hyperparameters, as shown in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S4.F10" title="Figure 10 ‣ 4.3.2 Impact of Hyperparameters under Non-IID Condition ‣ 4.3 Training Dynamics Under Non-IID Setting ‣ 4 Experimental Results ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">10</span></a>.</p> </div> <figure class="ltx_figure" id="S4.F10"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="369" id="S4.F10.g1" src="x10.png" width="830"/> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Test accuracy, test loss, and training loss for <math alttext="K=10" class="ltx_Math" display="inline" id="S4.F10.5.m1.1"><semantics id="S4.F10.5.m1.1b"><mrow id="S4.F10.5.m1.1.1" xref="S4.F10.5.m1.1.1.cmml"><mi id="S4.F10.5.m1.1.1.2" xref="S4.F10.5.m1.1.1.2.cmml">K</mi><mo id="S4.F10.5.m1.1.1.1" xref="S4.F10.5.m1.1.1.1.cmml">=</mo><mn id="S4.F10.5.m1.1.1.3" xref="S4.F10.5.m1.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.F10.5.m1.1c"><apply id="S4.F10.5.m1.1.1.cmml" xref="S4.F10.5.m1.1.1"><eq id="S4.F10.5.m1.1.1.1.cmml" xref="S4.F10.5.m1.1.1.1"></eq><ci id="S4.F10.5.m1.1.1.2.cmml" xref="S4.F10.5.m1.1.1.2">𝐾</ci><cn id="S4.F10.5.m1.1.1.3.cmml" type="integer" xref="S4.F10.5.m1.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F10.5.m1.1d">K=10</annotation><annotation encoding="application/x-llamapun" id="S4.F10.5.m1.1e">italic_K = 10</annotation></semantics></math>. The first row shows results for varying <math alttext="E" class="ltx_Math" display="inline" id="S4.F10.6.m2.1"><semantics id="S4.F10.6.m2.1b"><mi id="S4.F10.6.m2.1.1" xref="S4.F10.6.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S4.F10.6.m2.1c"><ci id="S4.F10.6.m2.1.1.cmml" xref="S4.F10.6.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F10.6.m2.1d">E</annotation><annotation encoding="application/x-llamapun" id="S4.F10.6.m2.1e">italic_E</annotation></semantics></math>, the second for different <math alttext="B" class="ltx_Math" display="inline" id="S4.F10.7.m3.1"><semantics id="S4.F10.7.m3.1b"><mi id="S4.F10.7.m3.1.1" xref="S4.F10.7.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S4.F10.7.m3.1c"><ci id="S4.F10.7.m3.1.1.cmml" xref="S4.F10.7.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F10.7.m3.1d">B</annotation><annotation encoding="application/x-llamapun" id="S4.F10.7.m3.1e">italic_B</annotation></semantics></math>, and the third for various <math alttext="\eta" class="ltx_Math" display="inline" id="S4.F10.8.m4.1"><semantics id="S4.F10.8.m4.1b"><mi id="S4.F10.8.m4.1.1" xref="S4.F10.8.m4.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S4.F10.8.m4.1c"><ci id="S4.F10.8.m4.1.1.cmml" xref="S4.F10.8.m4.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F10.8.m4.1d">\eta</annotation><annotation encoding="application/x-llamapun" id="S4.F10.8.m4.1e">italic_η</annotation></semantics></math>.</figcaption> </figure> <div class="ltx_para" id="S4.SS3.SSS2.p2"> <p class="ltx_p" id="S4.SS3.SSS2.p2.13">First, increasing <math alttext="E" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.1.m1.1"><semantics id="S4.SS3.SSS2.p2.1.m1.1a"><mi id="S4.SS3.SSS2.p2.1.m1.1.1" xref="S4.SS3.SSS2.p2.1.m1.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.1.m1.1b"><ci id="S4.SS3.SSS2.p2.1.m1.1.1.cmml" xref="S4.SS3.SSS2.p2.1.m1.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.1.m1.1c">E</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.1.m1.1d">italic_E</annotation></semantics></math> results in faster convergence, and unlike the IID setting, a higher number of epochs leads to improved final performance. The difference in accuracy between the smallest and largest <math alttext="E" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.2.m2.1"><semantics id="S4.SS3.SSS2.p2.2.m2.1a"><mi id="S4.SS3.SSS2.p2.2.m2.1.1" xref="S4.SS3.SSS2.p2.2.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.2.m2.1b"><ci id="S4.SS3.SSS2.p2.2.m2.1.1.cmml" xref="S4.SS3.SSS2.p2.2.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.2.m2.1c">E</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.2.m2.1d">italic_E</annotation></semantics></math> is <math alttext="+" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.3.m3.1"><semantics id="S4.SS3.SSS2.p2.3.m3.1a"><mo id="S4.SS3.SSS2.p2.3.m3.1.1" xref="S4.SS3.SSS2.p2.3.m3.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.3.m3.1b"><plus id="S4.SS3.SSS2.p2.3.m3.1.1.cmml" xref="S4.SS3.SSS2.p2.3.m3.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.3.m3.1c">+</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.3.m3.1d">+</annotation></semantics></math>1.7% (from 56.16% to 57.86%), whereas in the IID setting, this difference is <math alttext="-" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.4.m4.1"><semantics id="S4.SS3.SSS2.p2.4.m4.1a"><mo id="S4.SS3.SSS2.p2.4.m4.1.1" xref="S4.SS3.SSS2.p2.4.m4.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.4.m4.1b"><minus id="S4.SS3.SSS2.p2.4.m4.1.1.cmml" xref="S4.SS3.SSS2.p2.4.m4.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.4.m4.1c">-</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.4.m4.1d">-</annotation></semantics></math>1.55% (from 68.75% to 67.2%). Second, lower values of <math alttext="B" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.5.m5.1"><semantics id="S4.SS3.SSS2.p2.5.m5.1a"><mi id="S4.SS3.SSS2.p2.5.m5.1.1" xref="S4.SS3.SSS2.p2.5.m5.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.5.m5.1b"><ci id="S4.SS3.SSS2.p2.5.m5.1.1.cmml" xref="S4.SS3.SSS2.p2.5.m5.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.5.m5.1c">B</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.5.m5.1d">italic_B</annotation></semantics></math> result in both higher final accuracy and faster convergence. The accuracy gap between the smallest and largest <math alttext="B" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.6.m6.1"><semantics id="S4.SS3.SSS2.p2.6.m6.1a"><mi id="S4.SS3.SSS2.p2.6.m6.1.1" xref="S4.SS3.SSS2.p2.6.m6.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.6.m6.1b"><ci id="S4.SS3.SSS2.p2.6.m6.1.1.cmml" xref="S4.SS3.SSS2.p2.6.m6.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.6.m6.1c">B</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.6.m6.1d">italic_B</annotation></semantics></math> is <math alttext="+" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.7.m7.1"><semantics id="S4.SS3.SSS2.p2.7.m7.1a"><mo id="S4.SS3.SSS2.p2.7.m7.1.1" xref="S4.SS3.SSS2.p2.7.m7.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.7.m7.1b"><plus id="S4.SS3.SSS2.p2.7.m7.1.1.cmml" xref="S4.SS3.SSS2.p2.7.m7.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.7.m7.1c">+</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.7.m7.1d">+</annotation></semantics></math>6.45% (from 53.74% to 60.19%), compared to <math alttext="+" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.8.m8.1"><semantics id="S4.SS3.SSS2.p2.8.m8.1a"><mo id="S4.SS3.SSS2.p2.8.m8.1.1" xref="S4.SS3.SSS2.p2.8.m8.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.8.m8.1b"><plus id="S4.SS3.SSS2.p2.8.m8.1.1.cmml" xref="S4.SS3.SSS2.p2.8.m8.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.8.m8.1c">+</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.8.m8.1d">+</annotation></semantics></math>3.57% (from 67.77% to 71.34%) in the IID case. Finally, <math alttext="\eta" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.9.m9.1"><semantics id="S4.SS3.SSS2.p2.9.m9.1a"><mi id="S4.SS3.SSS2.p2.9.m9.1.1" xref="S4.SS3.SSS2.p2.9.m9.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.9.m9.1b"><ci id="S4.SS3.SSS2.p2.9.m9.1.1.cmml" xref="S4.SS3.SSS2.p2.9.m9.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.9.m9.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.9.m9.1d">italic_η</annotation></semantics></math> plays a crucial role in the non-IID scenario. The accuracy difference between the smallest and largest <math alttext="\eta" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.10.m10.1"><semantics id="S4.SS3.SSS2.p2.10.m10.1a"><mi id="S4.SS3.SSS2.p2.10.m10.1.1" xref="S4.SS3.SSS2.p2.10.m10.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.10.m10.1b"><ci id="S4.SS3.SSS2.p2.10.m10.1.1.cmml" xref="S4.SS3.SSS2.p2.10.m10.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.10.m10.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.10.m10.1d">italic_η</annotation></semantics></math> is <math alttext="+" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.11.m11.1"><semantics id="S4.SS3.SSS2.p2.11.m11.1a"><mo id="S4.SS3.SSS2.p2.11.m11.1.1" xref="S4.SS3.SSS2.p2.11.m11.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.11.m11.1b"><plus id="S4.SS3.SSS2.p2.11.m11.1.1.cmml" xref="S4.SS3.SSS2.p2.11.m11.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.11.m11.1c">+</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.11.m11.1d">+</annotation></semantics></math>15.48% (from 46.43% to 61.91%), compared to <math alttext="+" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.12.m12.1"><semantics id="S4.SS3.SSS2.p2.12.m12.1a"><mo id="S4.SS3.SSS2.p2.12.m12.1.1" xref="S4.SS3.SSS2.p2.12.m12.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.12.m12.1b"><plus id="S4.SS3.SSS2.p2.12.m12.1.1.cmml" xref="S4.SS3.SSS2.p2.12.m12.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.12.m12.1c">+</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.12.m12.1d">+</annotation></semantics></math>7.79% (from 63.22% to 71.01%) in the IID setting. Notably, lowest test loss decreases as <math alttext="\eta" class="ltx_Math" display="inline" id="S4.SS3.SSS2.p2.13.m13.1"><semantics id="S4.SS3.SSS2.p2.13.m13.1a"><mi id="S4.SS3.SSS2.p2.13.m13.1.1" xref="S4.SS3.SSS2.p2.13.m13.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS2.p2.13.m13.1b"><ci id="S4.SS3.SSS2.p2.13.m13.1.1.cmml" xref="S4.SS3.SSS2.p2.13.m13.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS2.p2.13.m13.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.SSS2.p2.13.m13.1d">italic_η</annotation></semantics></math> increases.</p> </div> </section> </section> </section> <section class="ltx_section" id="S5"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">5 </span>Understanding FL from a Loss Landscape Perspective</h2> <div class="ltx_para" id="S5.p1"> <p class="ltx_p" id="S5.p1.1">We have performed a series of experiments across datasets ranging from IID to non-IID, incorporating both partial participation and imbalanced data. As expected, the performance of the global model declines as the level of non-IIDness increases. The most commonly cited explanation for this behavior is client drift. <span class="ltx_text ltx_font_italic" id="S5.p1.1.1">But what drives this phenomenon?</span></p> </div> <div class="ltx_para" id="S5.p2"> <p class="ltx_p" id="S5.p2.1">To uncover the underlying causes, we examine the problem through the lens of the loss landscape <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib11" title="">11</a>]</cite>. The loss landscape is shaped by the interaction between the loss function, dataset, and model parameters. It provides valuable insights into how the loss changes as model parameters are updated for a fixed dataset.</p> </div> <div class="ltx_pagination ltx_role_newpage"></div> <figure class="ltx_figure" id="S5.F11"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="236" id="S5.F11.g1" src="x11.png" width="581"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>Toy example illustrating the loss landscape and update paths for two clients with IID and Non-IID datasets.</figcaption> </figure> <div class="ltx_para" id="S5.p3"> <p class="ltx_p" id="S5.p3.1">In FL, each client operates on a distinct dataset, meaning each experiences a unique loss landscape <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib2" title="">2</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib21" title="">21</a>]</cite>. Even though all clients use the same model and loss function, the variations in datasets lead to different optimization paths. As a result, updates from different clients are often misaligned, causing client drift. As each client optimizes locally, their models may converge to different local optima. When these locally optimized models are aggregated, the result may fail to reach the global optimum, as illustrated in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S5.F11" title="Figure 11 ‣ 5 Understanding FL from a Loss Landscape Perspective ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">11</span></a>.</p> </div> <div class="ltx_para" id="S5.p4"> <p class="ltx_p" id="S5.p4.1">In contrast, under IID conditions, these differences are minimized as the clients’ loss landscapes are more similar, leading to better-aligned update directions. Since directly visualizing the loss landscape in FL is difficult due to the high dimensionality and variability in client data, we instead performed two additional experiments to substantiate this observation.</p> </div> <div class="ltx_para" id="S5.p5"> <p class="ltx_p" id="S5.p5.1">First, we selected 5 clients from both IID and non-IID settings when <math alttext="K" class="ltx_Math" display="inline" id="S5.p5.1.m1.1"><semantics id="S5.p5.1.m1.1a"><mi id="S5.p5.1.m1.1.1" xref="S5.p5.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S5.p5.1.m1.1b"><ci id="S5.p5.1.m1.1.1.cmml" xref="S5.p5.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p5.1.m1.1c">K</annotation><annotation encoding="application/x-llamapun" id="S5.p5.1.m1.1d">italic_K</annotation></semantics></math> is 10 and traced their individual training loss. As shown in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S5.F12" title="Figure 12 ‣ 5 Understanding FL from a Loss Landscape Perspective ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">12</span></a>, under non-IID conditions, individual loss values decrease at different rates, while in the IID setting, the loss values for each client decrease more uniformly. This suggests that the local optimization processes in non-IID settings follow different loss landscapes, leading to varying loss values across clients.</p> </div> <figure class="ltx_figure" id="S5.F12"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="207" id="S5.F12.g1" src="x12.png" width="830"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Client training loss: Top row - IID, Bottom row - Non-IID.</figcaption> </figure> <div class="ltx_para" id="S5.p6"> <p class="ltx_p" id="S5.p6.1">Secondly, in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#S5.F13" title="Figure 13 ‣ 5 Understanding FL from a Loss Landscape Perspective ‣ Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study"><span class="ltx_text ltx_ref_tag">13</span></a>, we measure the layer-wise (conv1, conv2, fc1, and fc2) cosine similarity of local updates (<math alttext="\Delta\theta" class="ltx_Math" display="inline" id="S5.p6.1.m1.1"><semantics id="S5.p6.1.m1.1a"><mrow id="S5.p6.1.m1.1.1" xref="S5.p6.1.m1.1.1.cmml"><mi id="S5.p6.1.m1.1.1.2" mathvariant="normal" xref="S5.p6.1.m1.1.1.2.cmml">Δ</mi><mo id="S5.p6.1.m1.1.1.1" xref="S5.p6.1.m1.1.1.1.cmml">⁢</mo><mi id="S5.p6.1.m1.1.1.3" xref="S5.p6.1.m1.1.1.3.cmml">θ</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.p6.1.m1.1b"><apply id="S5.p6.1.m1.1.1.cmml" xref="S5.p6.1.m1.1.1"><times id="S5.p6.1.m1.1.1.1.cmml" xref="S5.p6.1.m1.1.1.1"></times><ci id="S5.p6.1.m1.1.1.2.cmml" xref="S5.p6.1.m1.1.1.2">Δ</ci><ci id="S5.p6.1.m1.1.1.3.cmml" xref="S5.p6.1.m1.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.1.m1.1c">\Delta\theta</annotation><annotation encoding="application/x-llamapun" id="S5.p6.1.m1.1d">roman_Δ italic_θ</annotation></semantics></math>) between pairs of clients, averaged across all combinations from 10 clients. As anticipated, in the IID setting, the cosine similarity between client updates remains consistently higher compared to the non-IID setting.</p> <div class="ltx_pagination ltx_role_newpage"></div> </div> <figure class="ltx_figure" id="S5.F13"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="138" id="S5.F13.g1" src="x13.png" width="830"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>Cosine similarity of local updates from different clients for each layer</figcaption> </figure> <div class="ltx_para" id="S5.p7"> <p class="ltx_p" id="S5.p7.1">These findings show that non-IID conditions lead to more diverse loss landscapes and less aligned client updates, while IID settings promote better alignment through consistent loss landscapes.</p> </div> <div class="ltx_para" id="S5.p8"> <p class="ltx_p" id="S5.p8.1">To guide the global model toward optimal performance in non-IID settings, two main strategies emerge:</p> <ol class="ltx_enumerate" id="S5.I1"> <li class="ltx_item" id="S5.I1.i1" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">1.</span> <div class="ltx_para" id="S5.I1.i1.p1"> <p class="ltx_p" id="S5.I1.i1.p1.1"><span class="ltx_text ltx_font_bold" id="S5.I1.i1.p1.1.1">Adjusting the update path</span>, where optimization steps are tailored to direct the model toward the global optimum without altering the underlying loss landscape.</p> </div> </li> <li class="ltx_item" id="S5.I1.i2" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">2.</span> <div class="ltx_para" id="S5.I1.i2.p1"> <p class="ltx_p" id="S5.I1.i2.p1.1"><span class="ltx_text ltx_font_bold" id="S5.I1.i2.p1.1.1">Modifying the loss landscape</span>, which aims to increase the overlap between local optima of individual clients, thus promoting more consistent convergence across clients.</p> </div> </li> </ol> <p class="ltx_p" id="S5.p8.2">For example, methods like SCAFFOLD <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib7" title="">7</a>]</cite>, which employs control variates, and FedOpt <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib16" title="">16</a>]</cite>, which uses server-side optimizers (e.g., Adam, Yogi) or weight perturbation <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib15" title="">15</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib9" title="">9</a>]</cite>, focus on refining update directions. In contrast, strategies that modify the loss landscape add terms to the objective function <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib13" title="">13</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib12" title="">12</a>, <a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib10" title="">10</a>]</cite>. Hybrid approaches like FedDyn <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib5" title="">5</a>]</cite> address both aspects simultaneously. However, a deeper analysis of these methods is necessary to fully understand their impact.</p> </div> </section> <section class="ltx_section" id="S6"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">6 </span>Discussion and Conclusion</h2> <div class="ltx_para" id="S6.p1"> <p class="ltx_p" id="S6.p1.4">Our experiments consistently show that a higher <math alttext="\eta" class="ltx_Math" display="inline" id="S6.p1.1.m1.1"><semantics id="S6.p1.1.m1.1a"><mi id="S6.p1.1.m1.1.1" xref="S6.p1.1.m1.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S6.p1.1.m1.1b"><ci id="S6.p1.1.m1.1.1.cmml" xref="S6.p1.1.m1.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.1.m1.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S6.p1.1.m1.1d">italic_η</annotation></semantics></math> and lower <math alttext="B" class="ltx_Math" display="inline" id="S6.p1.2.m2.1"><semantics id="S6.p1.2.m2.1a"><mi id="S6.p1.2.m2.1.1" xref="S6.p1.2.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S6.p1.2.m2.1b"><ci id="S6.p1.2.m2.1.1.cmml" xref="S6.p1.2.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.2.m2.1c">B</annotation><annotation encoding="application/x-llamapun" id="S6.p1.2.m2.1d">italic_B</annotation></semantics></math> improve performance in both IID and non-IID settings. According to our categorization, these hyperparameters are part of <span class="ltx_text ltx_font_bold" id="S6.p1.4.1">adjusting the update path</span>. A larger <math alttext="\eta" class="ltx_Math" display="inline" id="S6.p1.3.m3.1"><semantics id="S6.p1.3.m3.1a"><mi id="S6.p1.3.m3.1.1" xref="S6.p1.3.m3.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S6.p1.3.m3.1b"><ci id="S6.p1.3.m3.1.1.cmml" xref="S6.p1.3.m3.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.3.m3.1c">\eta</annotation><annotation encoding="application/x-llamapun" id="S6.p1.3.m3.1d">italic_η</annotation></semantics></math> leads to bigger steps in gradient descent, which can skip sharp minima and converge on flatter ones, although too large of a step risks divergence. Similarly, a smaller <math alttext="B" class="ltx_Math" display="inline" id="S6.p1.4.m4.1"><semantics id="S6.p1.4.m4.1a"><mi id="S6.p1.4.m4.1.1" xref="S6.p1.4.m4.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S6.p1.4.m4.1b"><ci id="S6.p1.4.m4.1.1.cmml" xref="S6.p1.4.m4.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.4.m4.1c">B</annotation><annotation encoding="application/x-llamapun" id="S6.p1.4.m4.1d">italic_B</annotation></semantics></math> aids in escaping sharp minima and encourages convergence to flatter minima <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib11" title="">11</a>]</cite>, improving generalization <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2502.00182v2#bib.bib8" title="">8</a>]</cite> and potentially increasing the likelihood of finding overlapping local optima, a topic that requires further investigation.</p> </div> <div class="ltx_para" id="S6.p2"> <p class="ltx_p" id="S6.p2.1">In conclusion, our exploration of optimization algorithms, from GD to FL, reveals that client drift stems from inconsistencies in loss landscapes. Based on these findings, we suggest that methods addressing performance degradation in non-IID settings can be categorized into two main strategies: adjusting the update path or modifying the loss landscape. While the motivations behind these methods may vary, they generally align with these categories. We hope this analysis provides valuable insights for new FL researchers and contributes to ongoing discussions in the field.</p> </div> <div class="ltx_para" id="S6.p3"> <span class="ltx_ERROR undefined" id="S6.p3.1">{credits}</span> </div> <section class="ltx_subsubsection" id="S6.SS0.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> <span class="ltx_tag ltx_tag_subsubsection">6.0.1 </span>Acknowledgements</h4> <div class="ltx_para" id="S6.SS0.SSS1.p1"> <p class="ltx_p" id="S6.SS0.SSS1.p1.1">The authors acknowledge the Research Council of Norway and the industry partners of NCS2030 – RCN project number 331644 – for their support.</p> </div> </section> <section class="ltx_subsubsection" id="S6.SS0.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> <span class="ltx_tag ltx_tag_subsubsection">6.0.2 </span><span class="ltx_ERROR undefined" id="S6.SS0.SSS2.1.1">\discintname</span> </h4> <div class="ltx_para" id="S6.SS0.SSS2.p1"> <p class="ltx_p" id="S6.SS0.SSS2.p1.1">The authors have no competing interests to declare that are relevant to the content of this article.</p> </div> </section> </section> <section class="ltx_bibliography" id="bib"> <h2 class="ltx_title ltx_title_bibliography">References</h2> <ul class="ltx_biblist"> <li class="ltx_bibitem" id="bib.bib1"> <span class="ltx_tag ltx_tag_bibitem">[1]</span> <span class="ltx_bibblock"> Acar, D.A.E., Zhao, Y., Matas, R., Mattina, M., Whatmough, P., Saligrama, V.: Federated learning based on dynamic regularization. In: ICLR (2021) </span> </li> <li class="ltx_bibitem" id="bib.bib2"> <span class="ltx_tag ltx_tag_bibitem">[2]</span> <span class="ltx_bibblock"> Al-Shedivat, M., Gillenwater, J., Xing, E., Rostamizadeh, A.: Federated learning via posterior averaging: A new perspective and practical algorithms. In: ICLR (2021) </span> </li> <li class="ltx_bibitem" id="bib.bib3"> <span class="ltx_tag ltx_tag_bibitem">[3]</span> <span class="ltx_bibblock"> Chu, C.T., Kim, S., Lin, Y.A., Yu, Y., Bradski, G., Olukotun, K., Ng, A.: Map-reduce for machine learning on multicore. NeurIPS (2006) </span> </li> <li class="ltx_bibitem" id="bib.bib4"> <span class="ltx_tag ltx_tag_bibitem">[4]</span> <span class="ltx_bibblock"> Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., et al.: Large scale distributed deep networks. NeurIPS (2012) </span> </li> <li class="ltx_bibitem" id="bib.bib5"> <span class="ltx_tag ltx_tag_bibitem">[5]</span> <span class="ltx_bibblock"> Durmus, A.E., Yue, Z., Ramon, M., Matthew, M., Paul, W., Venkatesh, S.: Federated learning based on dynamic regularization. In: ICLR (2021) </span> </li> <li class="ltx_bibitem" id="bib.bib6"> <span class="ltx_tag ltx_tag_bibitem">[6]</span> <span class="ltx_bibblock"> Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A.N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al.: Advances and open problems in federated learning. Foundations and trends in machine learning (2021) </span> </li> <li class="ltx_bibitem" id="bib.bib7"> <span class="ltx_tag ltx_tag_bibitem">[7]</span> <span class="ltx_bibblock"> Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: Scaffold: Stochastic controlled averaging for federated learning. In: ICML. PMLR (2020) </span> </li> <li class="ltx_bibitem" id="bib.bib8"> <span class="ltx_tag ltx_tag_bibitem">[8]</span> <span class="ltx_bibblock"> Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On large-batch training for deep learning: Generalization gap and sharp minima. In: ICLR (2017) </span> </li> <li class="ltx_bibitem" id="bib.bib9"> <span class="ltx_tag ltx_tag_bibitem">[9]</span> <span class="ltx_bibblock"> Lee, G., Jeong, M., Kim, S., Oh, J., Yun, S.Y.: Fedsol: Stabilized orthogonal learning with proximal restrictions in federated learning. In: CVPR (2024) </span> </li> <li class="ltx_bibitem" id="bib.bib10"> <span class="ltx_tag ltx_tag_bibitem">[10]</span> <span class="ltx_bibblock"> Lee, G., Jeong, M., Shin, Y., Bae, S., Yun, S.Y.: Preservation of the global knowledge by not-true distillation in federated learning. NeurIPS (2022) </span> </li> <li class="ltx_bibitem" id="bib.bib11"> <span class="ltx_tag ltx_tag_bibitem">[11]</span> <span class="ltx_bibblock"> Li, H., Xu, Z., Taylor, G., Studer, C., Goldstein, T.: Visualizing the loss landscape of neural nets. NeurIPS (2018) </span> </li> <li class="ltx_bibitem" id="bib.bib12"> <span class="ltx_tag ltx_tag_bibitem">[12]</span> <span class="ltx_bibblock"> Li, Q., He, B., Song, D.: Model-contrastive federated learning. In: CVPR (2021) </span> </li> <li class="ltx_bibitem" id="bib.bib13"> <span class="ltx_tag ltx_tag_bibitem">[13]</span> <span class="ltx_bibblock"> Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. MLSys (2020) </span> </li> <li class="ltx_bibitem" id="bib.bib14"> <span class="ltx_tag ltx_tag_bibitem">[14]</span> <span class="ltx_bibblock"> McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: AISTATS. PMLR (2017) </span> </li> <li class="ltx_bibitem" id="bib.bib15"> <span class="ltx_tag ltx_tag_bibitem">[15]</span> <span class="ltx_bibblock"> Qu, Z., Li, X., Duan, R., Liu, Y., Tang, B., Lu, Z.: Generalized federated learning via sharpness aware minimization. In: ICML. PMLR (2022) </span> </li> <li class="ltx_bibitem" id="bib.bib16"> <span class="ltx_tag ltx_tag_bibitem">[16]</span> <span class="ltx_bibblock"> Reddi, S.J., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., McMahan, H.B.: Adaptive federated optimization. In: ICLR (2021) </span> </li> <li class="ltx_bibitem" id="bib.bib17"> <span class="ltx_tag ltx_tag_bibitem">[17]</span> <span class="ltx_bibblock"> Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016) </span> </li> <li class="ltx_bibitem" id="bib.bib18"> <span class="ltx_tag ltx_tag_bibitem">[18]</span> <span class="ltx_bibblock"> Sergeev, A., Del Balso, M.: Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799 (2018) </span> </li> <li class="ltx_bibitem" id="bib.bib19"> <span class="ltx_tag ltx_tag_bibitem">[19]</span> <span class="ltx_bibblock"> Stich, S.U.: Local SGD converges fast and communicates little. In: ICLR (2019) </span> </li> <li class="ltx_bibitem" id="bib.bib20"> <span class="ltx_tag ltx_tag_bibitem">[20]</span> <span class="ltx_bibblock"> Sun, Y., Shen, L., Chen, S., Ding, L., Tao, D.: Dynamic regularized sharpness aware minimization in federated learning: Approaching global consistency and smooth landscape. In: ICML. PMLR (2023) </span> </li> <li class="ltx_bibitem" id="bib.bib21"> <span class="ltx_tag ltx_tag_bibitem">[21]</span> <span class="ltx_bibblock"> Zhang, J., Li, Z., Li, B., Xu, J., Wu, S., Ding, S., Wu, C.: Federated learning with label distribution skew via logits calibration. In: ICML. PMLR (2022) </span> </li> <li class="ltx_bibitem" id="bib.bib22"> <span class="ltx_tag ltx_tag_bibitem">[22]</span> <span class="ltx_bibblock"> Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., Chandra, V.: Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 (2018) </span> </li> <li class="ltx_bibitem" id="bib.bib23"> <span class="ltx_tag ltx_tag_bibitem">[23]</span> <span class="ltx_bibblock"> Zinkevich, M., Weimer, M., Li, L., Smola, A.: Parallelized stochastic gradient descent. NeurIPS (2010) </span> </li> </ul> </section> <div class="ltx_pagination ltx_role_newpage"></div> </article> </div> <footer class="ltx_page_footer"> <div class="ltx_page_logo">Generated on Fri Feb 7 14:32:08 2025 by <a class="ltx_LaTeXML_logo" href="http://dlmf.nist.gov/LaTeXML/"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img alt="Mascot Sammy" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg=="/></a> </div></footer> </div> </body> </html>