CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks</title>  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <meta content=" 6G, distributed inference, large language models, over-the-air computation, tensor parallelism. " lang="en" name="keywords"/> <base href="/html/2503.14882v1/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S1" title="In Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">I Introduction</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S1.SS1" title="In I Introduction ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">I-A Contributions</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S1.SS2" title="In I Introduction ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">I-B Organization and Notations</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2" title="In Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">II System Model and Problem Formulation</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.SS1" title="In II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">II-A Distributed On-Device LLM Inference System</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.SS2" title="In II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">II-B Tensor Parallelism</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.SS2.SSS1" title="In II-B Tensor Parallelism ‣ II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">II-B1 Tensor Parallelism for MLP Layer</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.SS2.SSS2" title="In II-B Tensor Parallelism ‣ II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">II-B2 Tensor Parallelism for Self-Attention Layer</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.SS3" title="In II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">II-C Over-the-Air All-Reduce</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.SS4" title="In II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">II-D Problem Formulation</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3" title="In Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III Algorithm Development</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3.SS1" title="In III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III-A Problem Decomposition</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3.SS1.SSS1" title="In III-A Problem Decomposition ‣ III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III-A1 Short-term transceiver optimization for given model assignment policy <math alttext="\mathbf{m}" class="ltx_Math" display="inline"><semantics><mi>𝐦</mi><annotation-xml encoding="MathML-Content"><ci>𝐦</ci></annotation-xml><annotation encoding="application/x-tex">\mathbf{m}</annotation><annotation encoding="application/x-llamapun">bold_m</annotation></semantics></math> and channel condition <math alttext="\mathbf{h}" class="ltx_Math" display="inline"><semantics><mi>𝐡</mi><annotation-xml encoding="MathML-Content"><ci>𝐡</ci></annotation-xml><annotation encoding="application/x-tex">\mathbf{h}</annotation><annotation encoding="application/x-llamapun">bold_h</annotation></semantics></math></a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3.SS1.SSS2" title="In III-A Problem Decomposition ‣ III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III-A2 Long-term model assignment optimization based on the optimal solution <math alttext="\mathbf{a}^{*}(\mathbf{m}),\left\{b_{n}^{*}(\mathbf{m})\right\}" class="ltx_Math" display="inline"><semantics><mrow><mrow><msup><mi>𝐚</mi><mo>∗</mo></msup><mo>⁢</mo><mrow><mo stretchy="false">(</mo><mi>𝐦</mi><mo stretchy="false">)</mo></mrow></mrow><mo>,</mo><mrow><mo>{</mo><mrow><msubsup><mi>b</mi><mi>n</mi><mo>∗</mo></msubsup><mo>⁢</mo><mrow><mo stretchy="false">(</mo><mi>𝐦</mi><mo stretchy="false">)</mo></mrow></mrow><mo>}</mo></mrow></mrow><annotation-xml encoding="MathML-Content"><list><apply><times></times><apply><csymbol cd="ambiguous">superscript</csymbol><ci>𝐚</ci><times></times></apply><ci>𝐦</ci></apply><set><apply><times></times><apply><csymbol cd="ambiguous">superscript</csymbol><apply><csymbol cd="ambiguous">subscript</csymbol><ci>𝑏</ci><ci>𝑛</ci></apply><times></times></apply><ci>𝐦</ci></apply></set></list></annotation-xml><annotation encoding="application/x-tex">\mathbf{a}^{*}(\mathbf{m}),\left\{b_{n}^{*}(\mathbf{m})\right\}</annotation><annotation encoding="application/x-llamapun">bold_a start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ( bold_m ) , { italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ( bold_m ) }</annotation></semantics></math> to problem <math alttext="\mathcal{P}_{s}" class="ltx_Math" display="inline"><semantics><msub><mi class="ltx_font_mathcaligraphic">𝒫</mi><mi>s</mi></msub><annotation-xml encoding="MathML-Content"><apply><csymbol cd="ambiguous">subscript</csymbol><ci>𝒫</ci><ci>𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex">\mathcal{P}_{s}</annotation><annotation encoding="application/x-llamapun">caligraphic_P start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT</annotation></semantics></math></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3.SS2" title="In III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III-B Short-Term Transceiver Optimization for <math alttext="\mathcal{P}_{s}" class="ltx_Math" display="inline"><semantics><msub><mi class="ltx_font_mathcaligraphic">𝒫</mi><mi>s</mi></msub><annotation-xml encoding="MathML-Content"><apply><csymbol cd="ambiguous">subscript</csymbol><ci>𝒫</ci><ci>𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex">\mathcal{P}_{s}</annotation><annotation encoding="application/x-llamapun">caligraphic_P start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT</annotation></semantics></math></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3.SS3" title="In III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III-C Long-Term Model Assignment Optimization for <math alttext="\mathcal{P}_{l}" class="ltx_Math" display="inline"><semantics><msub><mi class="ltx_font_mathcaligraphic">𝒫</mi><mi>l</mi></msub><annotation-xml encoding="MathML-Content"><apply><csymbol cd="ambiguous">subscript</csymbol><ci>𝒫</ci><ci>𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex">\mathcal{P}_{l}</annotation><annotation encoding="application/x-llamapun">caligraphic_P start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT</annotation></semantics></math></a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3.SS3.SSS1" title="In III-C Long-Term Model Assignment Optimization for 𝒫ₗ ‣ III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III-C1 Step 1</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3.SS3.SSS2" title="In III-C Long-Term Model Assignment Optimization for 𝒫ₗ ‣ III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III-C2 Step 2</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3.SS4" title="In III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III-D Convergence Analysis</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S4" title="In Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">IV Extension to Multi-Antenna Devices</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S4.SS1" title="In IV Extension to Multi-Antenna Devices ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">IV-A Problem Formulation</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S4.SS2" title="In IV Extension to Multi-Antenna Devices ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">IV-B Algorithm Development</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S4.SS2.SSS1" title="In IV-B Algorithm Development ‣ IV Extension to Multi-Antenna Devices ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">IV-B1 Short-term transceiver optimization for given model assignment policy <math alttext="\mathbf{m}" class="ltx_Math" display="inline"><semantics><mi>𝐦</mi><annotation-xml encoding="MathML-Content"><ci>𝐦</ci></annotation-xml><annotation encoding="application/x-tex">\mathbf{m}</annotation><annotation encoding="application/x-llamapun">bold_m</annotation></semantics></math> and channel condition <math alttext="\mathbf{H}" class="ltx_Math" display="inline"><semantics><mi>𝐇</mi><annotation-xml encoding="MathML-Content"><ci>𝐇</ci></annotation-xml><annotation encoding="application/x-tex">\mathbf{H}</annotation><annotation encoding="application/x-llamapun">bold_H</annotation></semantics></math></a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S4.SS2.SSS2" title="In IV-B Algorithm Development ‣ IV Extension to Multi-Antenna Devices ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">IV-B2 Long-term model assignment optimization based on the optimal solution <math alttext="\mathbf{A}^{*}(\mathbf{m}),\left\{\mathbf{B}_{n}^{*}(\mathbf{m})\right\}" class="ltx_Math" display="inline"><semantics><mrow><mrow><msup><mi>𝐀</mi><mo>∗</mo></msup><mo>⁢</mo><mrow><mo stretchy="false">(</mo><mi>𝐦</mi><mo stretchy="false">)</mo></mrow></mrow><mo>,</mo><mrow><mo>{</mo><mrow><msubsup><mi>𝐁</mi><mi>n</mi><mo>∗</mo></msubsup><mo>⁢</mo><mrow><mo stretchy="false">(</mo><mi>𝐦</mi><mo stretchy="false">)</mo></mrow></mrow><mo>}</mo></mrow></mrow><annotation-xml encoding="MathML-Content"><list><apply><times></times><apply><csymbol cd="ambiguous">superscript</csymbol><ci>𝐀</ci><times></times></apply><ci>𝐦</ci></apply><set><apply><times></times><apply><csymbol cd="ambiguous">superscript</csymbol><apply><csymbol cd="ambiguous">subscript</csymbol><ci>𝐁</ci><ci>𝑛</ci></apply><times></times></apply><ci>𝐦</ci></apply></set></list></annotation-xml><annotation encoding="application/x-tex">\mathbf{A}^{*}(\mathbf{m}),\left\{\mathbf{B}_{n}^{*}(\mathbf{m})\right\}</annotation><annotation encoding="application/x-llamapun">bold_A start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ( bold_m ) , { bold_B start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ( bold_m ) }</annotation></semantics></math> to problem <math alttext="\mathcal{P}_{s}" class="ltx_Math" display="inline"><semantics><msub><mi class="ltx_font_mathcaligraphic">𝒫</mi><mi>s</mi></msub><annotation-xml encoding="MathML-Content"><apply><csymbol cd="ambiguous">subscript</csymbol><ci>𝒫</ci><ci>𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex">\mathcal{P}_{s}</annotation><annotation encoding="application/x-llamapun">caligraphic_P start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT</annotation></semantics></math></a></li> </ol> </li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S5" title="In Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">V Simulation Results</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S5.SS1" title="In V Simulation Results ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">V-A Simulation Setups</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S5.SS1.SSS1" title="In V-A Simulation Setups ‣ V Simulation Results ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">V-A1 LLM Inference Model Setting</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S5.SS1.SSS2" title="In V-A Simulation Setups ‣ V Simulation Results ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">V-A2 Communication Model Setting</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S5.SS2" title="In V Simulation Results ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">V-B Algorithm Convergence</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S5.SS3" title="In V Simulation Results ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">V-C Performance Evaluation</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S5.SS4" title="In V Simulation Results ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">V-D Comparison with Centralized Inference Approach</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S6" title="In Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">VI Conclusion</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#Sx1.SS1" title="In Appendix ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">VI-A Proof of Lemma 3</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#Sx1.SS2" title="In Appendix ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">VI-B Proof of Theorem 1</a></li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line"> <h1 class="ltx_title ltx_title_document">Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks</h1> <div class="ltx_authors"> Kai Zhang, Hengtao He, Member, IEEE, Shenghui Song, Senior Member, IEEE, Jun Zhang, Fellow, IEEE, and Khaled B. Letaief, Fellow, IEEE </div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract</h6> Large language models (LLMs) have demonstrated remarkable success across various application domains, but their enormous sizes and computational demands pose significant challenges for deployment on resource-constrained edge devices. To address this issue, we propose a novel distributed on-device LLM inference framework that leverages tensor parallelism to partition the neural network tensors (e.g., weight matrices) of one LLM across multiple edge devices for collaborative inference. A key challenge in tensor parallelism is the frequent all-reduce operations for aggregating intermediate layer outputs across participating devices, which incurs significant communication overhead. To alleviate this bottleneck, we propose an over-the-air computation (AirComp) approach that harnesses the analog superposition property of wireless multiple-access channels to perform fast all-reduce steps. To utilize the heterogeneous computational capabilities of edge devices and mitigate communication distortions, we investigate a joint model assignment and transceiver optimization problem to minimize the average transmission error. The resulting mixed-timescale stochastic non-convex optimization problem is intractable, and we propose an efficient two-stage algorithm to solve it. Moreover, we prove that the proposed algorithm converges almost surely to a stationary point of the original problem. Comprehensive simulation results will show that the proposed framework outperforms existing benchmark schemes, achieving up to 5x inference speed acceleration and improving inference accuracy. </div> <div class="ltx_keywords"> <h6 class="ltx_title ltx_title_keywords">Index Terms: </h6> 6G, distributed inference, large language models, over-the-air computation, tensor parallelism. </div> ††footnotetext: Part of this work has been accepted for presentation at the 2025 IEEE Int. Conf. Commun. (ICC), Montreal, Canada <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib1" title="">1</a>]</cite>. The authors are with the Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong (email: kzhangbn@connect.ust.hk, eehthe@ust.hk, eeshsong@ust.hk, eejzhang@ust.hk, eekhaled@ust.hk). (The corresponding author is Hengtao He.) <section class="ltx_section" id="S1"> <h2 class="ltx_title ltx_title_section"> I Introduction </h2> <div class="ltx_para" id="S1.p1"> The advent of large language models (LLMs) has marked a significant breakthrough in artifical intelligence (AI), demonstrating superior performance and adaptability in a wide range of applications, such as natural language processing <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib2" title="">2</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib3" title="">3</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib4" title="">4</a>]</cite>, embodied intelligence <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib5" title="">5</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib6" title="">6</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib7" title="">7</a>]</cite>, and wireless communications <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib8" title="">8</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib9" title="">9</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib10" title="">10</a>]</cite>. The efficacy of LLMs is primarily attributed to the vast model scale with billions of parameters, which enables them to capture complex semantic relationships and contextual nuances, leading to superior performance across diverse tasks. However, the substantial computational and memory requirements of LLMs present significant challenges for the deployment on resource-constrained edge devices. For instance, the LLaMA3 model <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib11" title="">11</a>]</cite> with 13 billion parameters requires 40GB of RAM, which far exceeds the capabilities of most edge devices. Consequently, most existing LLMs rely on cloud-based infrastructure, which limits the feasibility of LLM deployment and raises concerns about data privacy and inference latency, especially in sensitive domains like healthcare and finance. To address these challenges, distributed LLM inference has recently been proposed as a promising solution, which distributes the large models and computational workloads across multiple devices <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib12" title="">12</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib13" title="">13</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib14" title="">14</a>]</cite>. This strategy allows each device to handle smaller and more manageable model segments, thereby reducing the burden on individual devices and strengthening privacy protections. Furthermore, advancements in communication technologies, such as the 5G and future 6G wireless networks, enhance the feasibility of distributed LLM inference for real-time applications <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib15" title="">15</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib16" title="">16</a>]</cite>. </div> <div class="ltx_para" id="S1.p2"> Communication overhead is a critical factor affecting the performance of distributed LLM inference systems. To enhance communication efficiency, several recent studies have been conducted <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib17" title="">17</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib18" title="">18</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib19" title="">19</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib20" title="">20</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib21" title="">21</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib22" title="">22</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib23" title="">23</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib24" title="">24</a>]</cite>. In <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib17" title="">17</a>]</cite>, Zhang et al. proposed a collaborative edge computing framework that distributes different layers of LLMs across the edge device and cloud server. They developed a joint device selection and model partitioning algorithm to minimize inference latency and maximize throughput. In <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib18" title="">18</a>]</cite>, Yuan et al. considered splitting LLMs into several sub-models, where the resource-intensive components were offloaded to the server through non-orthogonal multiple-access (NOMA) channels. They further proposed a gradient descent-based algorithm to find the optimal trade-off between inference delay and energy consumption. In <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib19" title="">19</a>]</cite>, He et al. developed an active inference method to address the joint task offloading and resource allocation problem for distributed LLM inference over cloud-edge computing frameworks. Similarly, Chen et al. <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib20" title="">20</a>]</cite> proposed a reinforcement learning algorithm that optimizes the splitting point of LLMs between the edge device and cloud server to reduce the communication overhead under varying wireless network conditions. Furthermore, task-oriented communications have been utilized to optimize end-to-end inference throughput, accuracy, and latency, which can further enhance the communication efficiency of distributed LLM inference systems <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib21" title="">21</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib22" title="">22</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib23" title="">23</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib24" title="">24</a>]</cite>. </div> <div class="ltx_para" id="S1.p3"> Despite significant advances in distributed LLM inference, most existing works <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib17" title="">17</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib18" title="">18</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib20" title="">20</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib19" title="">19</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib21" title="">21</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib22" title="">22</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib23" title="">23</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib24" title="">24</a>]</cite> primarily focus on the device-cloud collaborative inference. This architecture, however, faces substantial challenges in terms of feasibility and scalability due to its reliance on a powerful centralized cloud server with high computational capability. Moreover, prior works have generally employed the pipeline parallelism architectures, which are associated with inherent disadvantages such as pipeline bubbles <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib25" title="">25</a>]</cite>. These bubbles occur when downstream devices are forced to remain idle while waiting for upstream computations to complete, leading to poor utilization of computational resources. To address these limitations, distributed on-device LLM inference leveraging tensor parallelism has recently been proposed as a promising solution <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib26" title="">26</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib27" title="">27</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib28" title="">28</a>]</cite>. This approach divides large neural network tensors (e.g., weight matrices) of LLMs into smaller segments and distributes them across multiple edge devices. It not only eliminates the reliance on a powerful central server but also enables concurrent processing of model segments across devices, significantly improving the utilization of computation and communication resources. Nevertheless, a critical challenge in tensor parallelism is the frequent all-reduce operations required to aggregate intermediate layer outputs across devices. These communication-intensive all-reduce steps can cause substantial latency in practical wireless networks and hinder real-time inference, necessitating efficient communication strategies to fully achieve the benefits of tensor parallelism. </div> <div class="ltx_para" id="S1.p4"> In this paper, we propose a communication-efficient framework for distributed on-device LLM inference with tensor parallelism. Specifically, we propose an over-the-air computation (AirComp) approach to facilitate fast all-reduce operations. AirComp leverages the superposition property of wireless multiple-access channels, allowing simultaneous transmissions from multiple devices to be naturally summed at the receiver <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib29" title="">29</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib30" title="">30</a>]</cite>. This method reduces the communication latency and bandwidth requirement compared to traditional techniques that treat communication and computation separately. Most recently, AirComp has gained popularity in various applications such as edge computing <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib31" title="">31</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib32" title="">32</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib33" title="">33</a>]</cite>, federated learning <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib34" title="">34</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib35" title="">35</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib36" title="">36</a>]</cite>, and distributed sensing <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib37" title="">37</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib38" title="">38</a>, <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib39" title="">39</a>]</cite>. Table <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S1.T1" title="TABLE I ‣ I Introduction ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">I</a> shows a thorough survey of recent state-of-the-art frameworks on distributed parallel computing and AirComp for both model training and inference tasks. </div> <figure class="ltx_table" id="S1.T1"> <table class="ltx_tabular ltx_centering ltx_align_middle" id="S1.T1.18"> <tr class="ltx_tr" id="S1.T1.18.19"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_tt" id="S1.T1.18.19.1" style="padding:1.8pt 2.7pt;"> Reference </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" id="S1.T1.18.19.2" style="padding:1.8pt 2.7pt;"> Application Scenario </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" id="S1.T1.18.19.3" style="padding:1.8pt 2.7pt;"> Parallelism Method </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" id="S1.T1.18.19.4" style="padding:1.8pt 2.7pt;"> Antenna Configuration </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" id="S1.T1.18.19.5" style="padding:1.8pt 2.7pt;"> Optimization Objective </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" id="S1.T1.18.19.6" style="padding:1.8pt 2.7pt;"> Large-Scale LLM </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" id="S1.T1.18.19.7" style="padding:1.8pt 2.7pt;"> Device Heterogeneity </td> </tr> <tr class="ltx_tr" id="S1.T1.2.2"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.2.2.3" style="padding:1.8pt 2.7pt;"> K. Yang et al. (2020) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib34" title="">34</a>]</cite> </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.2.2.4" style="padding:1.8pt 2.7pt;">Training</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.2.2.5" style="padding:1.8pt 2.7pt;"> Data Parallelism </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.2.2.6" style="padding:1.8pt 2.7pt;">Multi-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.2.2.7" style="padding:1.8pt 2.7pt;"> Device Participation </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.1.1.1" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.1.1.1.m1.1"><semantics id="S1.T1.1.1.1.m1.1a"><mo id="S1.T1.1.1.1.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.1.1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.1.1.1.m1.1b"><times id="S1.T1.1.1.1.m1.1.1.cmml" xref="S1.T1.1.1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.1.1.1.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.1.1.1.m1.1d">×</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.2.2.2" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.2.2.2.m1.1"><semantics id="S1.T1.2.2.2.m1.1a"><mo id="S1.T1.2.2.2.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.2.2.2.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.2.2.2.m1.1b"><times id="S1.T1.2.2.2.m1.1.1.cmml" xref="S1.T1.2.2.2.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.2.2.2.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.2.2.2.m1.1d">×</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S1.T1.4.4"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.4.4.3" style="padding:1.8pt 2.7pt;"> X. Fan et al. (2021) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib40" title="">40</a>]</cite> </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.4.4.4" style="padding:1.8pt 2.7pt;">Training</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.4.4.5" style="padding:1.8pt 2.7pt;">Data Parallelism</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.4.4.6" style="padding:1.8pt 2.7pt;">Single-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.4.4.7" style="padding:1.8pt 2.7pt;">Convergence Rate</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.3.3.1" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.3.3.1.m1.1"><semantics id="S1.T1.3.3.1.m1.1a"><mo id="S1.T1.3.3.1.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.3.3.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.3.3.1.m1.1b"><times id="S1.T1.3.3.1.m1.1.1.cmml" xref="S1.T1.3.3.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.3.3.1.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.3.3.1.m1.1d">×</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.4.4.2" style="padding:1.8pt 2.7pt;"><math alttext="\checkmark" class="ltx_Math" display="inline" id="S1.T1.4.4.2.m1.1"><semantics id="S1.T1.4.4.2.m1.1a"><mi id="S1.T1.4.4.2.m1.1.1" mathcolor="#000000" mathsize="90%" mathvariant="normal" xref="S1.T1.4.4.2.m1.1.1.cmml">✓</mi><annotation-xml encoding="MathML-Content" id="S1.T1.4.4.2.m1.1b"><ci id="S1.T1.4.4.2.m1.1.1.cmml" xref="S1.T1.4.4.2.m1.1.1">✓</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.4.4.2.m1.1c">\checkmark</annotation><annotation encoding="application/x-llamapun" id="S1.T1.4.4.2.m1.1d">✓</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S1.T1.6.6"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.6.6.3" style="padding:1.8pt 2.7pt;"> T. Sery et al. (2021) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib36" title="">36</a>]</cite> </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.6.6.4" style="padding:1.8pt 2.7pt;">Training</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.6.6.5" style="padding:1.8pt 2.7pt;">Data Parallelism</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.6.6.6" style="padding:1.8pt 2.7pt;">Single-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.6.6.7" style="padding:1.8pt 2.7pt;">Communication Distortion</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.5.5.1" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.5.5.1.m1.1"><semantics id="S1.T1.5.5.1.m1.1a"><mo id="S1.T1.5.5.1.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.5.5.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.5.5.1.m1.1b"><times id="S1.T1.5.5.1.m1.1.1.cmml" xref="S1.T1.5.5.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.5.5.1.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.5.5.1.m1.1d">×</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.6.6.2" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.6.6.2.m1.1"><semantics id="S1.T1.6.6.2.m1.1a"><mo id="S1.T1.6.6.2.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.6.6.2.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.6.6.2.m1.1b"><times id="S1.T1.6.6.2.m1.1.1.cmml" xref="S1.T1.6.6.2.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.6.6.2.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.6.6.2.m1.1d">×</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S1.T1.8.8"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.8.8.3" style="padding:1.8pt 2.7pt;"> Y. Liang et al. (2024) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib41" title="">41</a>]</cite> </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.8.8.4" style="padding:1.8pt 2.7pt;">Training</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.8.8.5" style="padding:1.8pt 2.7pt;"> Data Parallelism </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.8.8.6" style="padding:1.8pt 2.7pt;">Single-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.8.8.7" style="padding:1.8pt 2.7pt;"> Training Latency and Energy Consumption </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.7.7.1" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.7.7.1.m1.1"><semantics id="S1.T1.7.7.1.m1.1a"><mo id="S1.T1.7.7.1.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.7.7.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.7.7.1.m1.1b"><times id="S1.T1.7.7.1.m1.1.1.cmml" xref="S1.T1.7.7.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.7.7.1.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.7.7.1.m1.1d">×</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.8.8.2" style="padding:1.8pt 2.7pt;"><math alttext="\checkmark" class="ltx_Math" display="inline" id="S1.T1.8.8.2.m1.1"><semantics id="S1.T1.8.8.2.m1.1a"><mi id="S1.T1.8.8.2.m1.1.1" mathcolor="#000000" mathsize="90%" mathvariant="normal" xref="S1.T1.8.8.2.m1.1.1.cmml">✓</mi><annotation-xml encoding="MathML-Content" id="S1.T1.8.8.2.m1.1b"><ci id="S1.T1.8.8.2.m1.1.1.cmml" xref="S1.T1.8.8.2.m1.1.1">✓</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.8.8.2.m1.1c">\checkmark</annotation><annotation encoding="application/x-llamapun" id="S1.T1.8.8.2.m1.1d">✓</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S1.T1.10.10"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.10.10.3" style="padding:1.8pt 2.7pt;"> H. Sun et al. (2024) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib42" title="">42</a>]</cite> </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.10.10.4" style="padding:1.8pt 2.7pt;">Training</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.10.10.5" style="padding:1.8pt 2.7pt;"> Data and Model Parallelism </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.10.10.6" style="padding:1.8pt 2.7pt;">Multi-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.10.10.7" style="padding:1.8pt 2.7pt;">Convergence Rate</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.9.9.1" style="padding:1.8pt 2.7pt;"><math alttext="\checkmark" class="ltx_Math" display="inline" id="S1.T1.9.9.1.m1.1"><semantics id="S1.T1.9.9.1.m1.1a"><mi id="S1.T1.9.9.1.m1.1.1" mathcolor="#000000" mathsize="90%" mathvariant="normal" xref="S1.T1.9.9.1.m1.1.1.cmml">✓</mi><annotation-xml encoding="MathML-Content" id="S1.T1.9.9.1.m1.1b"><ci id="S1.T1.9.9.1.m1.1.1.cmml" xref="S1.T1.9.9.1.m1.1.1">✓</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.9.9.1.m1.1c">\checkmark</annotation><annotation encoding="application/x-llamapun" id="S1.T1.9.9.1.m1.1d">✓</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.10.10.2" style="padding:1.8pt 2.7pt;"><math alttext="\checkmark" class="ltx_Math" display="inline" id="S1.T1.10.10.2.m1.1"><semantics id="S1.T1.10.10.2.m1.1a"><mi id="S1.T1.10.10.2.m1.1.1" mathcolor="#000000" mathsize="90%" mathvariant="normal" xref="S1.T1.10.10.2.m1.1.1.cmml">✓</mi><annotation-xml encoding="MathML-Content" id="S1.T1.10.10.2.m1.1b"><ci id="S1.T1.10.10.2.m1.1.1.cmml" xref="S1.T1.10.10.2.m1.1.1">✓</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.10.10.2.m1.1c">\checkmark</annotation><annotation encoding="application/x-llamapun" id="S1.T1.10.10.2.m1.1d">✓</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S1.T1.12.12"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.12.12.3" style="padding:1.8pt 2.7pt;"> Z. Zhuang et al. (2023) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib43" title="">43</a>]</cite> </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.12.12.4" style="padding:1.8pt 2.7pt;">Inference</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.12.12.5" style="padding:1.8pt 2.7pt;"> Data and Model Parallelism </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.12.12.6" style="padding:1.8pt 2.7pt;">Multi-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.12.12.7" style="padding:1.8pt 2.7pt;"> Minimum Pair-Wise Discriminant Gain </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.11.11.1" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.11.11.1.m1.1"><semantics id="S1.T1.11.11.1.m1.1a"><mo id="S1.T1.11.11.1.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.11.11.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.11.11.1.m1.1b"><times id="S1.T1.11.11.1.m1.1.1.cmml" xref="S1.T1.11.11.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.11.11.1.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.11.11.1.m1.1d">×</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.12.12.2" style="padding:1.8pt 2.7pt;"><math alttext="\checkmark" class="ltx_Math" display="inline" id="S1.T1.12.12.2.m1.1"><semantics id="S1.T1.12.12.2.m1.1a"><mi id="S1.T1.12.12.2.m1.1.1" mathcolor="#000000" mathsize="90%" mathvariant="normal" xref="S1.T1.12.12.2.m1.1.1.cmml">✓</mi><annotation-xml encoding="MathML-Content" id="S1.T1.12.12.2.m1.1b"><ci id="S1.T1.12.12.2.m1.1.1.cmml" xref="S1.T1.12.12.2.m1.1.1">✓</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.12.12.2.m1.1c">\checkmark</annotation><annotation encoding="application/x-llamapun" id="S1.T1.12.12.2.m1.1d">✓</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S1.T1.14.14"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.14.14.3" style="padding:1.8pt 2.7pt;"> D. Wen et al. (2023) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib24" title="">24</a>]</cite> </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.14.14.4" style="padding:1.8pt 2.7pt;">Inference</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.14.14.5" style="padding:1.8pt 2.7pt;"> Data Parallelism </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.14.14.6" style="padding:1.8pt 2.7pt;">Multi-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.14.14.7" style="padding:1.8pt 2.7pt;"> Discriminant Gain </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.13.13.1" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.13.13.1.m1.1"><semantics id="S1.T1.13.13.1.m1.1a"><mo id="S1.T1.13.13.1.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.13.13.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.13.13.1.m1.1b"><times id="S1.T1.13.13.1.m1.1.1.cmml" xref="S1.T1.13.13.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.13.13.1.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.13.13.1.m1.1d">×</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.14.14.2" style="padding:1.8pt 2.7pt;"><math alttext="\checkmark" class="ltx_Math" display="inline" id="S1.T1.14.14.2.m1.1"><semantics id="S1.T1.14.14.2.m1.1a"><mi id="S1.T1.14.14.2.m1.1.1" mathcolor="#000000" mathsize="90%" mathvariant="normal" xref="S1.T1.14.14.2.m1.1.1.cmml">✓</mi><annotation-xml encoding="MathML-Content" id="S1.T1.14.14.2.m1.1b"><ci id="S1.T1.14.14.2.m1.1.1.cmml" xref="S1.T1.14.14.2.m1.1.1">✓</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.14.14.2.m1.1c">\checkmark</annotation><annotation encoding="application/x-llamapun" id="S1.T1.14.14.2.m1.1d">✓</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S1.T1.16.16"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.16.16.3" style="padding:1.8pt 2.7pt;"> P. Yang et al. (2024) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib44" title="">44</a>]</cite> </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.16.16.4" style="padding:1.8pt 2.7pt;">Inference</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.16.16.5" style="padding:1.8pt 2.7pt;"> Data and Model Parallelism </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.16.16.6" style="padding:1.8pt 2.7pt;">Multi-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.16.16.7" style="padding:1.8pt 2.7pt;"> Communication Distortion </td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.15.15.1" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.15.15.1.m1.1"><semantics id="S1.T1.15.15.1.m1.1a"><mo id="S1.T1.15.15.1.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.15.15.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.15.15.1.m1.1b"><times id="S1.T1.15.15.1.m1.1.1.cmml" xref="S1.T1.15.15.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.15.15.1.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.15.15.1.m1.1d">×</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S1.T1.16.16.2" style="padding:1.8pt 2.7pt;"><math alttext="\times" class="ltx_Math" display="inline" id="S1.T1.16.16.2.m1.1"><semantics id="S1.T1.16.16.2.m1.1a"><mo id="S1.T1.16.16.2.m1.1.1" mathcolor="#000000" mathsize="90%" xref="S1.T1.16.16.2.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.T1.16.16.2.m1.1b"><times id="S1.T1.16.16.2.m1.1.1.cmml" xref="S1.T1.16.16.2.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.16.16.2.m1.1c">\times</annotation><annotation encoding="application/x-llamapun" id="S1.T1.16.16.2.m1.1d">×</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S1.T1.18.18"> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_l ltx_border_r ltx_border_t" id="S1.T1.18.18.3" style="padding:1.8pt 2.7pt;">This paper</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" id="S1.T1.18.18.4" style="padding:1.8pt 2.7pt;">Inference</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" id="S1.T1.18.18.5" style="padding:1.8pt 2.7pt;">Tensor Parallelism</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" id="S1.T1.18.18.6" style="padding:1.8pt 2.7pt;">Multi-Antenna</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" id="S1.T1.18.18.7" style="padding:1.8pt 2.7pt;">Communication Distortion</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" id="S1.T1.17.17.1" style="padding:1.8pt 2.7pt;"><math alttext="\checkmark" class="ltx_Math" display="inline" id="S1.T1.17.17.1.m1.1"><semantics id="S1.T1.17.17.1.m1.1a"><mi id="S1.T1.17.17.1.m1.1.1" mathcolor="#000000" mathsize="90%" mathvariant="normal" xref="S1.T1.17.17.1.m1.1.1.cmml">✓</mi><annotation-xml encoding="MathML-Content" id="S1.T1.17.17.1.m1.1b"><ci id="S1.T1.17.17.1.m1.1.1.cmml" xref="S1.T1.17.17.1.m1.1.1">✓</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.17.17.1.m1.1c">\checkmark</annotation><annotation encoding="application/x-llamapun" id="S1.T1.17.17.1.m1.1d">✓</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" id="S1.T1.18.18.2" style="padding:1.8pt 2.7pt;"><math alttext="\checkmark" class="ltx_Math" display="inline" id="S1.T1.18.18.2.m1.1"><semantics id="S1.T1.18.18.2.m1.1a"><mi id="S1.T1.18.18.2.m1.1.1" mathcolor="#000000" mathsize="90%" mathvariant="normal" xref="S1.T1.18.18.2.m1.1.1.cmml">✓</mi><annotation-xml encoding="MathML-Content" id="S1.T1.18.18.2.m1.1b"><ci id="S1.T1.18.18.2.m1.1.1.cmml" xref="S1.T1.18.18.2.m1.1.1">✓</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.18.18.2.m1.1c">\checkmark</annotation><annotation encoding="application/x-llamapun" id="S1.T1.18.18.2.m1.1d">✓</annotation></semantics></math></td> </tr> </table> <figcaption class="ltx_caption ltx_centering" style="font-size:90%;color:#000000;">TABLE I: Overview of Over-the-Air Computation for Distributed Learning and Inference</figcaption> </figure> <figure class="ltx_figure" id="S1.F1"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="218" id="S1.F1.g1" src="x1.png" width="598"/> <figcaption class="ltx_caption ltx_centering">Fig. 1: An illustration of the distributed on-device LLM inference system, showing the system workflow and visualizing tensor parallelism for (a) MLP and (b) self-attention layers.</figcaption> </figure> <div class="ltx_para" id="S1.p5"> The performance of the proposed distributed LLM inference system, however, is heavily influenced by the communication efficiency, particularly given the limited energy resources of edge devices. Thus, to improve the inference performance, we investigate a joint model assignment and transceiver optimization problem aimed at minimizing the average transmission mean-squared error (MSE). The formulated joint optimization is crucial considering the heterogeneous computation capabilities of edge devices and varying wireless channel conditions. Optimal model assignment ensures that each device processes a suitable portion of the model based on its computational capability (e.g., memory size and compute power), while transceiver optimization minimizes the communication distortions during the AirComp process. To simplify the problem and gain key insights, we initially consider the scenario of single-antenna edge devices. We then extend the framework to a multi-antenna configuration, leveraging spatial multiplexing to further enhance communication efficiency and reduce inference latency. Furthermore, the formulated joint model assignment and transceiver optimization problem is intractable due to its mixed-timescale, stochastic, and non-convex property. Specifically, the model assignment policy should be determined at the beginning of inference based on long-term statistical channel state information (CSI), while the transceiver design adapts dynamically to the CSI in each all-reduce step. To address the mixed-timescale optimization problem, we develop an efficient two-stage algorithm by employing semidefinite relaxation (SDR) and stochastic successive convex approximation (SCA). We note that although existing wireless optimization techniques (e.g., SDR and SCA algorithms) have been well studied, their tailored application to distributed LLM inference brings unique challenges and technical requirements. Specifically, our framework addresses unique challenges arising from large-scale distributed LLM inference, including the frequent aggregation of high-dimensional tensors, mixed-timescale optimization involving long-term model assignment and short-term transceiver adaptation, handling of heterogeneous device capabilities, multi-antenna AirComp beamforming designs, and stringent energy constraints. </div> <section class="ltx_subsection" id="S1.SS1"> <h3 class="ltx_title ltx_title_subsection"> I-A Contributions </h3> <div class="ltx_para" id="S1.SS1.p1"> The main contributions of this paper are summarized as follows. </div> <div class="ltx_para" id="S1.SS1.p2"> 1) We propose a novel distributed on-device LLM inference framework by employing tensor parallelism and AirComp. While tensor parallelism effectively distributes computational workloads across edge devices, its frequent all-reduce operations incur significant communication overhead, which offsets the computational benefits and becomes a major bottleneck for inference performance. To address this challenge, we develop a communication-efficient AirComp all-reduce approach by exploiting the signal superposition property of wireless multiple-access channels. </div> <div class="ltx_para" id="S1.SS1.p3"> 2) To utilize the heterogeneous computational capabilities of edge devices and mitigate communication distortions, we investigate a joint model assignment and transceiver optimization problem to minimize the average transmission MSE. The formulated mixed-timescale stochastic non-convex optimization problem is inherently intractable. Thus, we develop an efficient two-stage algorithm that decomposes the original problem into short-term transceiver optimization and long-term model assignment optimization subproblems. The resulting subproblems are further solved by employing SDR and stochastic SCA, respectively. The proposed algorithm requires no prior knowledge of channel statistics, and it converges almost surely to a stationary point of the original problem. </div> <div class="ltx_para" id="S1.SS1.p4"> 3) We validate the effectiveness of the proposed framework through simulations with two state-of-the-art open-source LLMs and a real-world text dataset. Simulation results demonstrate that the proposed algorithm outperforms benchmark schemes across various network settings, achieving up to 5x inference speed acceleration and improving inference accuracy. </div> </section> <section class="ltx_subsection" id="S1.SS2"> <h3 class="ltx_title ltx_title_subsection"> I-B Organization and Notations </h3> <div class="ltx_para" id="S1.SS2.p1"> The rest of this paper is organized as follows. In Section <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2" title="II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">II</a>, we elaborate on the system model and present the problem formulation. In Section <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S3" title="III Algorithm Development ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">III</a>, we develop a two-stage algorithm and prove its convergence. In Section <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S4" title="IV Extension to Multi-Antenna Devices ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">IV</a>, we extend the algorithm for multi-antenna edge devices. Simulation results are presented in Section <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S5" title="V Simulation Results ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">V</a>, and we conclude the paper in Section <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S6" title="VI Conclusion ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">VI</a>. </div> <div class="ltx_para" id="S1.SS2.p2"> Notations: Column vectors and matrices are denoted by boldface lowercase and boldface capital letters, respectively. The symbol <math alttext="\mathbb{R}" class="ltx_Math" display="inline" id="S1.SS2.p2.1.m1.1"><semantics id="S1.SS2.p2.1.m1.1a"><mi id="S1.SS2.p2.1.m1.1.1" xref="S1.SS2.p2.1.m1.1.1.cmml">ℝ</mi><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.1.m1.1b"><ci id="S1.SS2.p2.1.m1.1.1.cmml" xref="S1.SS2.p2.1.m1.1.1">ℝ</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.1.m1.1c">\mathbb{R}</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.1.m1.1d">blackboard_R</annotation></semantics></math> denotes the set of real numbers. <math alttext="\mathbb{C}^{M\times N}" class="ltx_Math" display="inline" id="S1.SS2.p2.2.m2.1"><semantics id="S1.SS2.p2.2.m2.1a"><msup id="S1.SS2.p2.2.m2.1.1" xref="S1.SS2.p2.2.m2.1.1.cmml"><mi id="S1.SS2.p2.2.m2.1.1.2" xref="S1.SS2.p2.2.m2.1.1.2.cmml">ℂ</mi><mrow id="S1.SS2.p2.2.m2.1.1.3" xref="S1.SS2.p2.2.m2.1.1.3.cmml"><mi id="S1.SS2.p2.2.m2.1.1.3.2" xref="S1.SS2.p2.2.m2.1.1.3.2.cmml">M</mi><mo id="S1.SS2.p2.2.m2.1.1.3.1" lspace="0.222em" rspace="0.222em" xref="S1.SS2.p2.2.m2.1.1.3.1.cmml">×</mo><mi id="S1.SS2.p2.2.m2.1.1.3.3" xref="S1.SS2.p2.2.m2.1.1.3.3.cmml">N</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.2.m2.1b"><apply id="S1.SS2.p2.2.m2.1.1.cmml" xref="S1.SS2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S1.SS2.p2.2.m2.1.1.1.cmml" xref="S1.SS2.p2.2.m2.1.1">superscript</csymbol><ci id="S1.SS2.p2.2.m2.1.1.2.cmml" xref="S1.SS2.p2.2.m2.1.1.2">ℂ</ci><apply id="S1.SS2.p2.2.m2.1.1.3.cmml" xref="S1.SS2.p2.2.m2.1.1.3"><times id="S1.SS2.p2.2.m2.1.1.3.1.cmml" xref="S1.SS2.p2.2.m2.1.1.3.1"></times><ci id="S1.SS2.p2.2.m2.1.1.3.2.cmml" xref="S1.SS2.p2.2.m2.1.1.3.2">𝑀</ci><ci id="S1.SS2.p2.2.m2.1.1.3.3.cmml" xref="S1.SS2.p2.2.m2.1.1.3.3">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.2.m2.1c">\mathbb{C}^{M\times N}</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.2.m2.1d">blackboard_C start_POSTSUPERSCRIPT italic_M × italic_N end_POSTSUPERSCRIPT</annotation></semantics></math> represents the space of the <math alttext="M\times N" class="ltx_Math" display="inline" id="S1.SS2.p2.3.m3.1"><semantics id="S1.SS2.p2.3.m3.1a"><mrow id="S1.SS2.p2.3.m3.1.1" xref="S1.SS2.p2.3.m3.1.1.cmml"><mi id="S1.SS2.p2.3.m3.1.1.2" xref="S1.SS2.p2.3.m3.1.1.2.cmml">M</mi><mo id="S1.SS2.p2.3.m3.1.1.1" lspace="0.222em" rspace="0.222em" xref="S1.SS2.p2.3.m3.1.1.1.cmml">×</mo><mi id="S1.SS2.p2.3.m3.1.1.3" xref="S1.SS2.p2.3.m3.1.1.3.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.3.m3.1b"><apply id="S1.SS2.p2.3.m3.1.1.cmml" xref="S1.SS2.p2.3.m3.1.1"><times id="S1.SS2.p2.3.m3.1.1.1.cmml" xref="S1.SS2.p2.3.m3.1.1.1"></times><ci id="S1.SS2.p2.3.m3.1.1.2.cmml" xref="S1.SS2.p2.3.m3.1.1.2">𝑀</ci><ci id="S1.SS2.p2.3.m3.1.1.3.cmml" xref="S1.SS2.p2.3.m3.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.3.m3.1c">M\times N</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.3.m3.1d">italic_M × italic_N</annotation></semantics></math> complex-valued matrices. <math alttext="(\cdot)^{\mathsf{T}}" class="ltx_Math" display="inline" id="S1.SS2.p2.4.m4.1"><semantics id="S1.SS2.p2.4.m4.1a"><msup id="S1.SS2.p2.4.m4.1.2" xref="S1.SS2.p2.4.m4.1.2.cmml"><mrow id="S1.SS2.p2.4.m4.1.2.2.2" xref="S1.SS2.p2.4.m4.1.2.cmml"><mo id="S1.SS2.p2.4.m4.1.2.2.2.1" stretchy="false" xref="S1.SS2.p2.4.m4.1.2.cmml">(</mo><mo id="S1.SS2.p2.4.m4.1.1" lspace="0em" rspace="0em" xref="S1.SS2.p2.4.m4.1.1.cmml">⋅</mo><mo id="S1.SS2.p2.4.m4.1.2.2.2.2" stretchy="false" xref="S1.SS2.p2.4.m4.1.2.cmml">)</mo></mrow><mi id="S1.SS2.p2.4.m4.1.2.3" xref="S1.SS2.p2.4.m4.1.2.3.cmml">𝖳</mi></msup><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.4.m4.1b"><apply id="S1.SS2.p2.4.m4.1.2.cmml" xref="S1.SS2.p2.4.m4.1.2"><csymbol cd="ambiguous" id="S1.SS2.p2.4.m4.1.2.1.cmml" xref="S1.SS2.p2.4.m4.1.2">superscript</csymbol><ci id="S1.SS2.p2.4.m4.1.1.cmml" xref="S1.SS2.p2.4.m4.1.1">⋅</ci><ci id="S1.SS2.p2.4.m4.1.2.3.cmml" xref="S1.SS2.p2.4.m4.1.2.3">𝖳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.4.m4.1c">(\cdot)^{\mathsf{T}}</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.4.m4.1d">( ⋅ ) start_POSTSUPERSCRIPT sansserif_T end_POSTSUPERSCRIPT</annotation></semantics></math> and <math alttext="(\cdot)^{\mathsf{H}}" class="ltx_Math" display="inline" id="S1.SS2.p2.5.m5.1"><semantics id="S1.SS2.p2.5.m5.1a"><msup id="S1.SS2.p2.5.m5.1.2" xref="S1.SS2.p2.5.m5.1.2.cmml"><mrow id="S1.SS2.p2.5.m5.1.2.2.2" xref="S1.SS2.p2.5.m5.1.2.cmml"><mo id="S1.SS2.p2.5.m5.1.2.2.2.1" stretchy="false" xref="S1.SS2.p2.5.m5.1.2.cmml">(</mo><mo id="S1.SS2.p2.5.m5.1.1" lspace="0em" rspace="0em" xref="S1.SS2.p2.5.m5.1.1.cmml">⋅</mo><mo id="S1.SS2.p2.5.m5.1.2.2.2.2" stretchy="false" xref="S1.SS2.p2.5.m5.1.2.cmml">)</mo></mrow><mi id="S1.SS2.p2.5.m5.1.2.3" xref="S1.SS2.p2.5.m5.1.2.3.cmml">𝖧</mi></msup><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.5.m5.1b"><apply id="S1.SS2.p2.5.m5.1.2.cmml" xref="S1.SS2.p2.5.m5.1.2"><csymbol cd="ambiguous" id="S1.SS2.p2.5.m5.1.2.1.cmml" xref="S1.SS2.p2.5.m5.1.2">superscript</csymbol><ci id="S1.SS2.p2.5.m5.1.1.cmml" xref="S1.SS2.p2.5.m5.1.1">⋅</ci><ci id="S1.SS2.p2.5.m5.1.2.3.cmml" xref="S1.SS2.p2.5.m5.1.2.3">𝖧</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.5.m5.1c">(\cdot)^{\mathsf{H}}</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.5.m5.1d">( ⋅ ) start_POSTSUPERSCRIPT sansserif_H end_POSTSUPERSCRIPT</annotation></semantics></math> stand for the transpose and the conjugate transpose of their arguments, respectively. <math alttext="\textup{tr}(\mathbf{A})" class="ltx_Math" display="inline" id="S1.SS2.p2.6.m6.1"><semantics id="S1.SS2.p2.6.m6.1a"><mrow id="S1.SS2.p2.6.m6.1.2" xref="S1.SS2.p2.6.m6.1.2.cmml"><mtext id="S1.SS2.p2.6.m6.1.2.2" xref="S1.SS2.p2.6.m6.1.2.2a.cmml">tr</mtext><mo id="S1.SS2.p2.6.m6.1.2.1" xref="S1.SS2.p2.6.m6.1.2.1.cmml">⁢</mo><mrow id="S1.SS2.p2.6.m6.1.2.3.2" xref="S1.SS2.p2.6.m6.1.2.cmml"><mo id="S1.SS2.p2.6.m6.1.2.3.2.1" stretchy="false" xref="S1.SS2.p2.6.m6.1.2.cmml">(</mo><mi id="S1.SS2.p2.6.m6.1.1" xref="S1.SS2.p2.6.m6.1.1.cmml">𝐀</mi><mo id="S1.SS2.p2.6.m6.1.2.3.2.2" stretchy="false" xref="S1.SS2.p2.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.6.m6.1b"><apply id="S1.SS2.p2.6.m6.1.2.cmml" xref="S1.SS2.p2.6.m6.1.2"><times id="S1.SS2.p2.6.m6.1.2.1.cmml" xref="S1.SS2.p2.6.m6.1.2.1"></times><ci id="S1.SS2.p2.6.m6.1.2.2a.cmml" xref="S1.SS2.p2.6.m6.1.2.2"><mtext id="S1.SS2.p2.6.m6.1.2.2.cmml" xref="S1.SS2.p2.6.m6.1.2.2">tr</mtext></ci><ci id="S1.SS2.p2.6.m6.1.1.cmml" xref="S1.SS2.p2.6.m6.1.1">𝐀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.6.m6.1c">\textup{tr}(\mathbf{A})</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.6.m6.1d">tr ( bold_A )</annotation></semantics></math> denote the trace of matrix <math alttext="\mathbf{A}" class="ltx_Math" display="inline" id="S1.SS2.p2.7.m7.1"><semantics id="S1.SS2.p2.7.m7.1a"><mi id="S1.SS2.p2.7.m7.1.1" xref="S1.SS2.p2.7.m7.1.1.cmml">𝐀</mi><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.7.m7.1b"><ci id="S1.SS2.p2.7.m7.1.1.cmml" xref="S1.SS2.p2.7.m7.1.1">𝐀</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.7.m7.1c">\mathbf{A}</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.7.m7.1d">bold_A</annotation></semantics></math>. <math alttext="\mathbb{E}[\cdot]" class="ltx_Math" display="inline" id="S1.SS2.p2.8.m8.1"><semantics id="S1.SS2.p2.8.m8.1a"><mrow id="S1.SS2.p2.8.m8.1.2" xref="S1.SS2.p2.8.m8.1.2.cmml"><mi id="S1.SS2.p2.8.m8.1.2.2" xref="S1.SS2.p2.8.m8.1.2.2.cmml">𝔼</mi><mo id="S1.SS2.p2.8.m8.1.2.1" xref="S1.SS2.p2.8.m8.1.2.1.cmml">⁢</mo><mrow id="S1.SS2.p2.8.m8.1.2.3.2" xref="S1.SS2.p2.8.m8.1.2.3.1.cmml"><mo id="S1.SS2.p2.8.m8.1.2.3.2.1" stretchy="false" xref="S1.SS2.p2.8.m8.1.2.3.1.1.cmml">[</mo><mo id="S1.SS2.p2.8.m8.1.1" lspace="0em" rspace="0em" xref="S1.SS2.p2.8.m8.1.1.cmml">⋅</mo><mo id="S1.SS2.p2.8.m8.1.2.3.2.2" stretchy="false" xref="S1.SS2.p2.8.m8.1.2.3.1.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.8.m8.1b"><apply id="S1.SS2.p2.8.m8.1.2.cmml" xref="S1.SS2.p2.8.m8.1.2"><times id="S1.SS2.p2.8.m8.1.2.1.cmml" xref="S1.SS2.p2.8.m8.1.2.1"></times><ci id="S1.SS2.p2.8.m8.1.2.2.cmml" xref="S1.SS2.p2.8.m8.1.2.2">𝔼</ci><apply id="S1.SS2.p2.8.m8.1.2.3.1.cmml" xref="S1.SS2.p2.8.m8.1.2.3.2"><csymbol cd="latexml" id="S1.SS2.p2.8.m8.1.2.3.1.1.cmml" xref="S1.SS2.p2.8.m8.1.2.3.2.1">delimited-[]</csymbol><ci id="S1.SS2.p2.8.m8.1.1.cmml" xref="S1.SS2.p2.8.m8.1.1">⋅</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.8.m8.1c">\mathbb{E}[\cdot]</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.8.m8.1d">blackboard_E [ ⋅ ]</annotation></semantics></math> denotes the expectation operation. <math alttext="\nabla" class="ltx_Math" display="inline" id="S1.SS2.p2.9.m9.1"><semantics id="S1.SS2.p2.9.m9.1a"><mo id="S1.SS2.p2.9.m9.1.1" xref="S1.SS2.p2.9.m9.1.1.cmml">∇</mo><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.9.m9.1b"><ci id="S1.SS2.p2.9.m9.1.1.cmml" xref="S1.SS2.p2.9.m9.1.1">∇</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.9.m9.1c">\nabla</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.9.m9.1d">∇</annotation></semantics></math> represents the gradient operator. <math alttext="|\cdot|" class="ltx_math_unparsed" display="inline" id="S1.SS2.p2.10.m10.1"><semantics id="S1.SS2.p2.10.m10.1a"><mrow id="S1.SS2.p2.10.m10.1b"><mo fence="false" id="S1.SS2.p2.10.m10.1.1" stretchy="false">|</mo><mo id="S1.SS2.p2.10.m10.1.2" lspace="0em" rspace="0em">⋅</mo><mo fence="false" id="S1.SS2.p2.10.m10.1.3" stretchy="false">|</mo></mrow><annotation encoding="application/x-tex" id="S1.SS2.p2.10.m10.1c">|\cdot|</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.10.m10.1d">| ⋅ |</annotation></semantics></math> and <math alttext="\|\cdot\|" class="ltx_math_unparsed" display="inline" id="S1.SS2.p2.11.m11.1"><semantics id="S1.SS2.p2.11.m11.1a"><mrow id="S1.SS2.p2.11.m11.1b"><mo id="S1.SS2.p2.11.m11.1.1" rspace="0em">∥</mo><mo id="S1.SS2.p2.11.m11.1.2" lspace="0em" rspace="0em">⋅</mo><mo id="S1.SS2.p2.11.m11.1.3" lspace="0em">∥</mo></mrow><annotation encoding="application/x-tex" id="S1.SS2.p2.11.m11.1c">\|\cdot\|</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.11.m11.1d">∥ ⋅ ∥</annotation></semantics></math> stand for the <math alttext="\ell_{1}" class="ltx_Math" display="inline" id="S1.SS2.p2.12.m12.1"><semantics id="S1.SS2.p2.12.m12.1a"><msub id="S1.SS2.p2.12.m12.1.1" xref="S1.SS2.p2.12.m12.1.1.cmml"><mi id="S1.SS2.p2.12.m12.1.1.2" mathvariant="normal" xref="S1.SS2.p2.12.m12.1.1.2.cmml">ℓ</mi><mn id="S1.SS2.p2.12.m12.1.1.3" xref="S1.SS2.p2.12.m12.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.12.m12.1b"><apply id="S1.SS2.p2.12.m12.1.1.cmml" xref="S1.SS2.p2.12.m12.1.1"><csymbol cd="ambiguous" id="S1.SS2.p2.12.m12.1.1.1.cmml" xref="S1.SS2.p2.12.m12.1.1">subscript</csymbol><ci id="S1.SS2.p2.12.m12.1.1.2.cmml" xref="S1.SS2.p2.12.m12.1.1.2">ℓ</ci><cn id="S1.SS2.p2.12.m12.1.1.3.cmml" type="integer" xref="S1.SS2.p2.12.m12.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.12.m12.1c">\ell_{1}</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.12.m12.1d">roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> and <math alttext="\ell_{2}" class="ltx_Math" display="inline" id="S1.SS2.p2.13.m13.1"><semantics id="S1.SS2.p2.13.m13.1a"><msub id="S1.SS2.p2.13.m13.1.1" xref="S1.SS2.p2.13.m13.1.1.cmml"><mi id="S1.SS2.p2.13.m13.1.1.2" mathvariant="normal" xref="S1.SS2.p2.13.m13.1.1.2.cmml">ℓ</mi><mn id="S1.SS2.p2.13.m13.1.1.3" xref="S1.SS2.p2.13.m13.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S1.SS2.p2.13.m13.1b"><apply id="S1.SS2.p2.13.m13.1.1.cmml" xref="S1.SS2.p2.13.m13.1.1"><csymbol cd="ambiguous" id="S1.SS2.p2.13.m13.1.1.1.cmml" xref="S1.SS2.p2.13.m13.1.1">subscript</csymbol><ci id="S1.SS2.p2.13.m13.1.1.2.cmml" xref="S1.SS2.p2.13.m13.1.1.2">ℓ</ci><cn id="S1.SS2.p2.13.m13.1.1.3.cmml" type="integer" xref="S1.SS2.p2.13.m13.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS2.p2.13.m13.1c">\ell_{2}</annotation><annotation encoding="application/x-llamapun" id="S1.SS2.p2.13.m13.1d">roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math> norm of vectors. </div> </section> </section> <section class="ltx_section" id="S2"> <h2 class="ltx_title ltx_title_section"> II System Model and Problem Formulation </h2> <div class="ltx_para" id="S2.p1"> In this section, we first elaborate on the proposed distributed on-device LLM inference system, followed by proposing the communication-efficient AirComp all-reduce approach. To minimize the average transmission MSE, we then formulate a joint model assignment and transceiver optimization problem. </div> <section class="ltx_subsection" id="S2.SS1"> <h3 class="ltx_title ltx_title_subsection"> II-A Distributed On-Device LLM Inference System </h3> <div class="ltx_para" id="S2.SS1.p1"> To deploy LLMs on resource-limited edge devices, distributed on-device inference with tensor parallelism has been proposed. This method involves partitioning large neural network tensors (e.g., weight matrices) of LLMs into smaller segments and distributing them across multiple edge devices for simultaneous processing. The complete workflow of the proposed distributed on-device LLM inference system is illustrated in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S1.F1" title="Fig. 1 ‣ I Introduction ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">1</a>. When a device initiates an inference request, the edge server dynamically identifies available local devices and partitions the model parameters. Then, each device loads its assigned model segment into memory and performs forward computation. After each layer of the LLM is computed, an all-reduce operation aggregates the intermediate layer outputs from all devices, ensuring synchronization and consistency across devices during inference. In the proposed distributed inference framework, the device shares its input (typically token embeddings rather than raw data) with other participating devices. For scenarios demanding strict confidentiality, encryption schemes (e.g., homomorphic encryption) or secure enclaves can be adopted to mitigate privacy leakage. Furthermore, we highlight two typical scenarios illustrating real-world, trusted environments particularly suitable for our distributed inference framework, as shown in the following. <ul class="ltx_itemize" id="S2.I1"> <li class="ltx_item" id="S2.I1.i1" style="list-style-type:none;"> • <div class="ltx_para" id="S2.I1.i1.p1"> Organizational or HPC Clusters: Large institutions (e.g., corporate data centers, national labs, or university HPC centers) often host massive LLMs that exceed the capacity of a single node. In these clusters, multiple servers within the same security domain can distribute model segments or layers among them, securely exchanging raw input data via internal networks. Since all compute nodes reside in the same trusted infrastructure (with well-defined access control, encryption, and compliance policies), they can fully leverage parallelization to reduce per-inference latency and alleviate memory bottlenecks, without risking data exposure to external environments. </div> </li> <li class="ltx_item" id="S2.I1.i2" style="list-style-type:none;"> • <div class="ltx_para" id="S2.I1.i2.p1"> Single-User or Local Edge Scenarios: Individual users or small teams may possess multiple personal devices or localized edge servers (e.g., the home server or on-premises GPU node). These devices operate within a single-user network or closed local environment, allowing them to share raw inputs without breaching privacy. By splitting the LLM’s parameters or layers across these trusted devices, users can achieve faster response times and reduced memory load per device. These benefits are especially valuable for real-time applications (e.g., smart home assistants or AR/VR), where offloading data to external clouds may be undesirable or impractical. </div> </li> </ul> </div> </section> <section class="ltx_subsection" id="S2.SS2"> <h3 class="ltx_title ltx_title_subsection"> II-B Tensor Parallelism </h3> <div class="ltx_para" id="S2.SS2.p1"> LLMs are primarily built on the Transformer architecture, which typically consists of dozens of Transformer layers <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib45" title="">45</a>]</cite>. Each Transformer layer includes a self-attention mechanism and a multi-layer perceptron (MLP). To achieve efficient distributed inference, tensor parallelism partitions both the self-attention and MLP layers within each Transformer block into smaller tensor segments, as shown in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S1.F1" title="Fig. 1 ‣ I Introduction ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">1</a>. We note that both pipeline parallelism and tensor parallelism are two prevalent model partitioning strategies widely adopted in distributed inference frameworks. While pipeline parallelism partitions the model across layers, tensor parallelism partitions computations within each layer across multiple devices. Tensor parallelism is particularly attractive for on-device inference due to its inherent advantages in significantly reducing idle times (pipeline bubbles), achieving finer-grained memory allocation, and, when combined with AirComp-based aggregation, greatly minimizing communication overhead. These properties highlight the practical benefits and superior suitability of tensor parallelism for resource-constrained and latency-sensitive inference scenarios considered in this work. </div> <section class="ltx_subsubsection" id="S2.SS2.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> II-B1 Tensor Parallelism for MLP Layer</h4> <figure class="ltx_figure" id="S2.F2"> <img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="145" id="S2.F2.1.1.g1" src="x2.png" width="281"/> <figcaption class="ltx_caption ltx_centering">Fig. 2: Illustration of MLP matrix multiplication for conventional unpartitioned approach and tensor parallelism with two devices.</figcaption> </figure> <div class="ltx_para" id="S2.SS2.SSS1.p1"> For a typical 2-layer MLP within the Transformer block, the forward computation involves two main linear transformations, separated by a non-linear activation function (e.g., ReLU or GeLU). We formulate the computation of the MLP layer by taking the ReLU activation as an example. Mathematically, it is expressed as follows, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E1"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E1X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{Z}=\max(\mathbf{0},\mathbf{XW})\mathbf{U}," class="ltx_Math" display="inline" id="S2.E1X.2.1.1.m1.4"><semantics id="S2.E1X.2.1.1.m1.4a"><mrow id="S2.E1X.2.1.1.m1.4.4.1" xref="S2.E1X.2.1.1.m1.4.4.1.1.cmml"><mrow id="S2.E1X.2.1.1.m1.4.4.1.1" xref="S2.E1X.2.1.1.m1.4.4.1.1.cmml"><mi id="S2.E1X.2.1.1.m1.4.4.1.1.2" mathcolor="#000000" xref="S2.E1X.2.1.1.m1.4.4.1.1.2.cmml">𝐙</mi><mo id="S2.E1X.2.1.1.m1.4.4.1.1.1" mathcolor="#000000" xref="S2.E1X.2.1.1.m1.4.4.1.1.1.cmml">=</mo><mrow id="S2.E1X.2.1.1.m1.4.4.1.1.3" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.cmml"><mrow id="S2.E1X.2.1.1.m1.4.4.1.1.3.2.2" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.2.1.cmml"><mi id="S2.E1X.2.1.1.m1.1.1" mathcolor="#000000" xref="S2.E1X.2.1.1.m1.1.1.cmml">max</mi><mo id="S2.E1X.2.1.1.m1.4.4.1.1.3.2.2a" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.2.1.cmml">⁡</mo><mrow id="S2.E1X.2.1.1.m1.4.4.1.1.3.2.2.1" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.2.1.cmml"><mo id="S2.E1X.2.1.1.m1.4.4.1.1.3.2.2.1.1" mathcolor="#000000" stretchy="false" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.2.1.cmml">(</mo><mn id="S2.E1X.2.1.1.m1.2.2" mathcolor="#000000" xref="S2.E1X.2.1.1.m1.2.2.cmml">𝟎</mn><mo id="S2.E1X.2.1.1.m1.4.4.1.1.3.2.2.1.2" mathcolor="#000000" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.2.1.cmml">,</mo><mi id="S2.E1X.2.1.1.m1.3.3" mathcolor="#000000" xref="S2.E1X.2.1.1.m1.3.3.cmml">𝐗𝐖</mi><mo id="S2.E1X.2.1.1.m1.4.4.1.1.3.2.2.1.3" mathcolor="#000000" stretchy="false" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.2.1.cmml">)</mo></mrow></mrow><mo id="S2.E1X.2.1.1.m1.4.4.1.1.3.1" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.1.cmml">⁢</mo><mi id="S2.E1X.2.1.1.m1.4.4.1.1.3.3" mathcolor="#000000" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.3.cmml">𝐔</mi></mrow></mrow><mo id="S2.E1X.2.1.1.m1.4.4.1.2" mathcolor="#000000" xref="S2.E1X.2.1.1.m1.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E1X.2.1.1.m1.4b"><apply id="S2.E1X.2.1.1.m1.4.4.1.1.cmml" xref="S2.E1X.2.1.1.m1.4.4.1"><eq id="S2.E1X.2.1.1.m1.4.4.1.1.1.cmml" xref="S2.E1X.2.1.1.m1.4.4.1.1.1"></eq><ci id="S2.E1X.2.1.1.m1.4.4.1.1.2.cmml" xref="S2.E1X.2.1.1.m1.4.4.1.1.2">𝐙</ci><apply id="S2.E1X.2.1.1.m1.4.4.1.1.3.cmml" xref="S2.E1X.2.1.1.m1.4.4.1.1.3"><times id="S2.E1X.2.1.1.m1.4.4.1.1.3.1.cmml" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.1"></times><apply id="S2.E1X.2.1.1.m1.4.4.1.1.3.2.1.cmml" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.2.2"><max id="S2.E1X.2.1.1.m1.1.1.cmml" xref="S2.E1X.2.1.1.m1.1.1"></max><cn id="S2.E1X.2.1.1.m1.2.2.cmml" type="integer" xref="S2.E1X.2.1.1.m1.2.2">0</cn><ci id="S2.E1X.2.1.1.m1.3.3.cmml" xref="S2.E1X.2.1.1.m1.3.3">𝐗𝐖</ci></apply><ci id="S2.E1X.2.1.1.m1.4.4.1.1.3.3.cmml" xref="S2.E1X.2.1.1.m1.4.4.1.1.3.3">𝐔</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1X.2.1.1.m1.4c">\displaystyle\mathbf{Z}=\max(\mathbf{0},\mathbf{XW})\mathbf{U},</annotation><annotation encoding="application/x-llamapun" id="S2.E1X.2.1.1.m1.4d">bold_Z = roman_max ( bold_0 , bold_XW ) bold_U ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(1)</td> </tr> </tbody> </table> where <math alttext="\mathbf{X}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.1.1.m1.1"><semantics id="S2.SS2.SSS1.p1.1.1.m1.1a"><mi id="S2.SS2.SSS1.p1.1.1.m1.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.1.1.m1.1.1.cmml">𝐗</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.1.1.m1.1b"><ci id="S2.SS2.SSS1.p1.1.1.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.1.1.m1.1.1">𝐗</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.1.1.m1.1c">\mathbf{X}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.1.1.m1.1d">bold_X</annotation></semantics></math> is the input to the MLP layer, <math alttext="\mathbf{Z}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.2.2.m2.1"><semantics id="S2.SS2.SSS1.p1.2.2.m2.1a"><mi id="S2.SS2.SSS1.p1.2.2.m2.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.2.2.m2.1.1.cmml">𝐙</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.2.2.m2.1b"><ci id="S2.SS2.SSS1.p1.2.2.m2.1.1.cmml" xref="S2.SS2.SSS1.p1.2.2.m2.1.1">𝐙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.2.2.m2.1c">\mathbf{Z}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.2.2.m2.1d">bold_Z</annotation></semantics></math> is the output, and <math alttext="\mathbf{W}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.3.3.m3.1"><semantics id="S2.SS2.SSS1.p1.3.3.m3.1a"><mi id="S2.SS2.SSS1.p1.3.3.m3.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.3.3.m3.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.3.3.m3.1b"><ci id="S2.SS2.SSS1.p1.3.3.m3.1.1.cmml" xref="S2.SS2.SSS1.p1.3.3.m3.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.3.3.m3.1c">\mathbf{W}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.3.3.m3.1d">bold_W</annotation></semantics></math> and <math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.4.4.m4.1"><semantics id="S2.SS2.SSS1.p1.4.4.m4.1a"><mi id="S2.SS2.SSS1.p1.4.4.m4.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.4.4.m4.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.4.4.m4.1b"><ci id="S2.SS2.SSS1.p1.4.4.m4.1.1.cmml" xref="S2.SS2.SSS1.p1.4.4.m4.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.4.4.m4.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.4.4.m4.1d">bold_U</annotation></semantics></math> are the weight matrices, respectively. Our framework can be readily generalized to other activation functions, such as GeLU function: <math alttext="\mathbf{Z}=\text{GeLU}(\mathbf{XW})\mathbf{U},~{}\text{where}~{}\text{GeLU}(x)% =x\Phi(x)," class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.5.5.m5.4"><semantics id="S2.SS2.SSS1.p1.5.5.m5.4a"><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1"><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.3.cmml"><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.cmml"><mi id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.2.cmml">𝐙</mi><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.1.cmml">=</mo><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.cmml"><mtext id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.2a.cmml">GeLU</mtext><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.1" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.1.cmml">⁢</mo><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.3.2" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.cmml"><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.3.2.1" mathcolor="#000000" stretchy="false" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.cmml">(</mo><mi id="S2.SS2.SSS1.p1.5.5.m5.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.1.1.cmml">𝐗𝐖</mi><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.3.2.2" mathcolor="#000000" stretchy="false" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.cmml">)</mo></mrow><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.1a" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.1.cmml">⁢</mo><mi id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.4" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.4.cmml">𝐔</mi></mrow></mrow><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.3" mathcolor="#000000" rspace="0.497em" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.3a.cmml">,</mo><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.cmml"><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.cmml"><mtext id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.2a.cmml">where</mtext><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.1" lspace="0.330em" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.1.cmml">⁢</mo><mtext id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.3a.cmml">GeLU</mtext><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.1a" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.1.cmml">⁢</mo><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.4.2" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.cmml"><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.4.2.1" mathcolor="#000000" stretchy="false" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.cmml">(</mo><mi id="S2.SS2.SSS1.p1.5.5.m5.2.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.2.2.cmml">x</mi><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.4.2.2" mathcolor="#000000" stretchy="false" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.cmml">)</mo></mrow></mrow><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.1.cmml">=</mo><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.cmml"><mi id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.2.cmml">x</mi><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.1" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.1.cmml">⁢</mo><mi id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.3" mathcolor="#000000" mathvariant="normal" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.3.cmml">Φ</mi><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.1a" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.1.cmml">⁢</mo><mrow id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.4.2" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.cmml"><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.4.2.1" mathcolor="#000000" stretchy="false" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.cmml">(</mo><mi id="S2.SS2.SSS1.p1.5.5.m5.3.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.3.3.cmml">x</mi><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.4.2.2" mathcolor="#000000" stretchy="false" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.2" mathcolor="#000000">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.5.5.m5.4b"><apply id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.3.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.3a.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.3">formulae-sequence</csymbol><apply id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1"><eq id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.1.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.1"></eq><ci id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.2.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.2">𝐙</ci><apply id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3"><times id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.1.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.1"></times><ci id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.2a.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.2"><mtext id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.2.cmml" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.2">GeLU</mtext></ci><ci id="S2.SS2.SSS1.p1.5.5.m5.1.1.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.1.1">𝐗𝐖</ci><ci id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.4.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.1.1.3.4">𝐔</ci></apply></apply><apply id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2"><eq id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.1.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.1"></eq><apply id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2"><times id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.1.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.1"></times><ci id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.2a.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.2"><mtext id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.2.cmml" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.2">where</mtext></ci><ci id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.3a.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.3"><mtext id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.3.cmml" mathcolor="#000000" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.2.3">GeLU</mtext></ci><ci id="S2.SS2.SSS1.p1.5.5.m5.2.2.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.2.2">𝑥</ci></apply><apply id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3"><times id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.1.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.1"></times><ci id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.2.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.2">𝑥</ci><ci id="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.3.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.4.4.1.1.2.2.3.3">Φ</ci><ci id="S2.SS2.SSS1.p1.5.5.m5.3.3.cmml" xref="S2.SS2.SSS1.p1.5.5.m5.3.3">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.5.5.m5.4c">\mathbf{Z}=\text{GeLU}(\mathbf{XW})\mathbf{U},~{}\text{where}~{}\text{GeLU}(x)% =x\Phi(x),</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.5.5.m5.4d">bold_Z = GeLU ( bold_XW ) bold_U , where GeLU ( italic_x ) = italic_x roman_Φ ( italic_x ) ,</annotation></semantics></math> with <math alttext="\Phi(x)" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.6.6.m6.1"><semantics id="S2.SS2.SSS1.p1.6.6.m6.1a"><mrow id="S2.SS2.SSS1.p1.6.6.m6.1.2" xref="S2.SS2.SSS1.p1.6.6.m6.1.2.cmml"><mi id="S2.SS2.SSS1.p1.6.6.m6.1.2.2" mathcolor="#000000" mathvariant="normal" xref="S2.SS2.SSS1.p1.6.6.m6.1.2.2.cmml">Φ</mi><mo id="S2.SS2.SSS1.p1.6.6.m6.1.2.1" xref="S2.SS2.SSS1.p1.6.6.m6.1.2.1.cmml">⁢</mo><mrow id="S2.SS2.SSS1.p1.6.6.m6.1.2.3.2" xref="S2.SS2.SSS1.p1.6.6.m6.1.2.cmml"><mo id="S2.SS2.SSS1.p1.6.6.m6.1.2.3.2.1" mathcolor="#000000" stretchy="false" xref="S2.SS2.SSS1.p1.6.6.m6.1.2.cmml">(</mo><mi id="S2.SS2.SSS1.p1.6.6.m6.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.6.6.m6.1.1.cmml">x</mi><mo id="S2.SS2.SSS1.p1.6.6.m6.1.2.3.2.2" mathcolor="#000000" stretchy="false" xref="S2.SS2.SSS1.p1.6.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.6.6.m6.1b"><apply id="S2.SS2.SSS1.p1.6.6.m6.1.2.cmml" xref="S2.SS2.SSS1.p1.6.6.m6.1.2"><times id="S2.SS2.SSS1.p1.6.6.m6.1.2.1.cmml" xref="S2.SS2.SSS1.p1.6.6.m6.1.2.1"></times><ci id="S2.SS2.SSS1.p1.6.6.m6.1.2.2.cmml" xref="S2.SS2.SSS1.p1.6.6.m6.1.2.2">Φ</ci><ci id="S2.SS2.SSS1.p1.6.6.m6.1.1.cmml" xref="S2.SS2.SSS1.p1.6.6.m6.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.6.6.m6.1c">\Phi(x)</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.6.6.m6.1d">roman_Φ ( italic_x )</annotation></semantics></math> representing the cumulative distribution function of the standard Gaussian distribution. The traditional centralized inference approach loads the entire weight matrices <math alttext="\mathbf{W}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.7.m1.1"><semantics id="S2.SS2.SSS1.p1.7.m1.1a"><mi id="S2.SS2.SSS1.p1.7.m1.1.1" xref="S2.SS2.SSS1.p1.7.m1.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.7.m1.1b"><ci id="S2.SS2.SSS1.p1.7.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.7.m1.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.7.m1.1c">\mathbf{W}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.7.m1.1d">bold_W</annotation></semantics></math> and <math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.8.m2.1"><semantics id="S2.SS2.SSS1.p1.8.m2.1a"><mi id="S2.SS2.SSS1.p1.8.m2.1.1" xref="S2.SS2.SSS1.p1.8.m2.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.8.m2.1b"><ci id="S2.SS2.SSS1.p1.8.m2.1.1.cmml" xref="S2.SS2.SSS1.p1.8.m2.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.8.m2.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.8.m2.1d">bold_U</annotation></semantics></math> into memory and performs full matrix multiplications on a single device, which is usually impractical for resource-limited edge devices. To overcome this challenge, tensor parallelism distributes the weight matrices <math alttext="\mathbf{W}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.9.m3.1"><semantics id="S2.SS2.SSS1.p1.9.m3.1a"><mi id="S2.SS2.SSS1.p1.9.m3.1.1" xref="S2.SS2.SSS1.p1.9.m3.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.9.m3.1b"><ci id="S2.SS2.SSS1.p1.9.m3.1.1.cmml" xref="S2.SS2.SSS1.p1.9.m3.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.9.m3.1c">\mathbf{W}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.9.m3.1d">bold_W</annotation></semantics></math> and <math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.10.m4.1"><semantics id="S2.SS2.SSS1.p1.10.m4.1a"><mi id="S2.SS2.SSS1.p1.10.m4.1.1" xref="S2.SS2.SSS1.p1.10.m4.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.10.m4.1b"><ci id="S2.SS2.SSS1.p1.10.m4.1.1.cmml" xref="S2.SS2.SSS1.p1.10.m4.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.10.m4.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.10.m4.1d">bold_U</annotation></semantics></math> across <math alttext="N" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.11.m5.1"><semantics id="S2.SS2.SSS1.p1.11.m5.1a"><mi id="S2.SS2.SSS1.p1.11.m5.1.1" xref="S2.SS2.SSS1.p1.11.m5.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.11.m5.1b"><ci id="S2.SS2.SSS1.p1.11.m5.1.1.cmml" xref="S2.SS2.SSS1.p1.11.m5.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.11.m5.1c">N</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.11.m5.1d">italic_N</annotation></semantics></math> devices. The weight matrices <math alttext="\mathbf{W}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.12.7.m1.1"><semantics id="S2.SS2.SSS1.p1.12.7.m1.1a"><mi id="S2.SS2.SSS1.p1.12.7.m1.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.12.7.m1.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.12.7.m1.1b"><ci id="S2.SS2.SSS1.p1.12.7.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.12.7.m1.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.12.7.m1.1c">\mathbf{W}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.12.7.m1.1d">bold_W</annotation></semantics></math> and <math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.13.8.m2.1"><semantics id="S2.SS2.SSS1.p1.13.8.m2.1a"><mi id="S2.SS2.SSS1.p1.13.8.m2.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.13.8.m2.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.13.8.m2.1b"><ci id="S2.SS2.SSS1.p1.13.8.m2.1.1.cmml" xref="S2.SS2.SSS1.p1.13.8.m2.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.13.8.m2.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.13.8.m2.1d">bold_U</annotation></semantics></math> have dimensions <math alttext="\displaystyle d\times d_{\text{hidden}}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.14.9.m3.1"><semantics id="S2.SS2.SSS1.p1.14.9.m3.1a"><mrow id="S2.SS2.SSS1.p1.14.9.m3.1.1" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.cmml"><mi id="S2.SS2.SSS1.p1.14.9.m3.1.1.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.2.cmml">d</mi><mo id="S2.SS2.SSS1.p1.14.9.m3.1.1.1" lspace="0.222em" mathcolor="#000000" rspace="0.222em" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.1.cmml">×</mo><msub id="S2.SS2.SSS1.p1.14.9.m3.1.1.3" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.3.cmml"><mi id="S2.SS2.SSS1.p1.14.9.m3.1.1.3.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.3.2.cmml">d</mi><mtext id="S2.SS2.SSS1.p1.14.9.m3.1.1.3.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.3.3a.cmml">hidden</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.14.9.m3.1b"><apply id="S2.SS2.SSS1.p1.14.9.m3.1.1.cmml" xref="S2.SS2.SSS1.p1.14.9.m3.1.1"><times id="S2.SS2.SSS1.p1.14.9.m3.1.1.1.cmml" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.1"></times><ci id="S2.SS2.SSS1.p1.14.9.m3.1.1.2.cmml" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.2">𝑑</ci><apply id="S2.SS2.SSS1.p1.14.9.m3.1.1.3.cmml" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.14.9.m3.1.1.3.1.cmml" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.3">subscript</csymbol><ci id="S2.SS2.SSS1.p1.14.9.m3.1.1.3.2.cmml" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.3.2">𝑑</ci><ci id="S2.SS2.SSS1.p1.14.9.m3.1.1.3.3a.cmml" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.3.3"><mtext id="S2.SS2.SSS1.p1.14.9.m3.1.1.3.3.cmml" mathcolor="#000000" mathsize="70%" xref="S2.SS2.SSS1.p1.14.9.m3.1.1.3.3">hidden</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.14.9.m3.1c">\displaystyle d\times d_{\text{hidden}}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.14.9.m3.1d">italic_d × italic_d start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT</annotation></semantics></math> and <math alttext="\displaystyle d_{\text{hidden}}\times d" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.15.10.m4.1"><semantics id="S2.SS2.SSS1.p1.15.10.m4.1a"><mrow id="S2.SS2.SSS1.p1.15.10.m4.1.1" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.cmml"><msub id="S2.SS2.SSS1.p1.15.10.m4.1.1.2" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.2.cmml"><mi id="S2.SS2.SSS1.p1.15.10.m4.1.1.2.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.2.2.cmml">d</mi><mtext id="S2.SS2.SSS1.p1.15.10.m4.1.1.2.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.2.3a.cmml">hidden</mtext></msub><mo id="S2.SS2.SSS1.p1.15.10.m4.1.1.1" lspace="0.222em" mathcolor="#000000" rspace="0.222em" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.1.cmml">×</mo><mi id="S2.SS2.SSS1.p1.15.10.m4.1.1.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.3.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.15.10.m4.1b"><apply id="S2.SS2.SSS1.p1.15.10.m4.1.1.cmml" xref="S2.SS2.SSS1.p1.15.10.m4.1.1"><times id="S2.SS2.SSS1.p1.15.10.m4.1.1.1.cmml" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.1"></times><apply id="S2.SS2.SSS1.p1.15.10.m4.1.1.2.cmml" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.15.10.m4.1.1.2.1.cmml" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.2">subscript</csymbol><ci id="S2.SS2.SSS1.p1.15.10.m4.1.1.2.2.cmml" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.2.2">𝑑</ci><ci id="S2.SS2.SSS1.p1.15.10.m4.1.1.2.3a.cmml" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.2.3"><mtext id="S2.SS2.SSS1.p1.15.10.m4.1.1.2.3.cmml" mathcolor="#000000" mathsize="70%" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.2.3">hidden</mtext></ci></apply><ci id="S2.SS2.SSS1.p1.15.10.m4.1.1.3.cmml" xref="S2.SS2.SSS1.p1.15.10.m4.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.15.10.m4.1c">\displaystyle d_{\text{hidden}}\times d</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.15.10.m4.1d">italic_d start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT × italic_d</annotation></semantics></math>, respectively. As shown in Fig. <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.F2" title="Fig. 2 ‣ II-B1 Tensor Parallelism for MLP Layer ‣ II-B Tensor Parallelism ‣ II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">2</a>, the weight matrix <math alttext="\mathbf{W}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.16.11.m5.1"><semantics id="S2.SS2.SSS1.p1.16.11.m5.1a"><mi id="S2.SS2.SSS1.p1.16.11.m5.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.16.11.m5.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.16.11.m5.1b"><ci id="S2.SS2.SSS1.p1.16.11.m5.1.1.cmml" xref="S2.SS2.SSS1.p1.16.11.m5.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.16.11.m5.1c">\mathbf{W}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.16.11.m5.1d">bold_W</annotation></semantics></math> is partitioned column-wise into multiple slices as <table class="ltx_equationgroup ltx_eqn_table" id="S2.E2"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E2X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle\mathbf{W}" class="ltx_Math" display="inline" id="S2.E2X.2.1.1.m1.1"><semantics id="S2.E2X.2.1.1.m1.1a"><mi id="S2.E2X.2.1.1.m1.1.1" mathcolor="#000000" xref="S2.E2X.2.1.1.m1.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.E2X.2.1.1.m1.1b"><ci id="S2.E2X.2.1.1.m1.1.1.cmml" xref="S2.E2X.2.1.1.m1.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E2X.2.1.1.m1.1c">\displaystyle\mathbf{W}</annotation><annotation encoding="application/x-llamapun" id="S2.E2X.2.1.1.m1.1d">bold_W</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="2">(2)</td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E2Xa"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle=[\mathbf{W}_{1}\in\mathbb{R}^{d\times d^{1}_{\text{hidden}}},% \mathbf{W}_{2}\in\mathbb{R}^{d\times d^{2}_{\text{hidden}}},\ldots,\mathbf{W}_% {N}\in\mathbb{R}^{d\times d^{N}_{\text{hidden}}}]," class="ltx_Math" display="inline" id="S2.E2Xa.2.1.1.m1.2"><semantics id="S2.E2Xa.2.1.1.m1.2a"><mrow id="S2.E2Xa.2.1.1.m1.2.2.1" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.cmml"><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.3.cmml"></mi><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.2.cmml">=</mo><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.2.cmml"><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.2" mathcolor="#000000" stretchy="false" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.2.1.cmml">[</mo><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.3.cmml"><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.cmml"><msub id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.2.cmml">𝐖</mi><mn id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.3.cmml">1</mn></msub><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.1" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.cmml">∈</mo><msup id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.2.cmml">ℝ</mi><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.2.cmml">d</mi><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.1" lspace="0.222em" mathcolor="#000000" rspace="0.222em" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.1.cmml">×</mo><msubsup id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.2.cmml">d</mi><mtext id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.3a.cmml">hidden</mtext><mn id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.3.cmml">1</mn></msubsup></mrow></msup></mrow><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.3a.cmml">,</mo><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.3.cmml"><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.cmml"><msub id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.2.cmml">𝐖</mi><mn id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.3.cmml">2</mn></msub><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.2.cmml">∈</mo><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.2.cmml"><msup id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.2.cmml">ℝ</mi><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.2.cmml">d</mi><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.1" lspace="0.222em" mathcolor="#000000" rspace="0.222em" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.1.cmml">×</mo><msubsup id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.2.cmml">d</mi><mtext id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.3a.cmml">hidden</mtext><mn id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.3.cmml">2</mn></msubsup></mrow></msup><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.2.cmml">,</mo><mi id="S2.E2Xa.2.1.1.m1.1.1" mathcolor="#000000" mathvariant="normal" xref="S2.E2Xa.2.1.1.m1.1.1.cmml">…</mi></mrow></mrow><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.3a.cmml">,</mo><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.cmml"><msub id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.2.cmml">𝐖</mi><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.3.cmml">N</mi></msub><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.1" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.1.cmml">∈</mo><msup id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.2.cmml">ℝ</mi><mrow id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.2.cmml">d</mi><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.1" lspace="0.222em" mathcolor="#000000" rspace="0.222em" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.1.cmml">×</mo><msubsup id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.cmml"><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.2.cmml">d</mi><mtext id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.3a.cmml">hidden</mtext><mi id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.3" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.3.cmml">N</mi></msubsup></mrow></msup></mrow></mrow></mrow><mo id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.3" mathcolor="#000000" stretchy="false" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="S2.E2Xa.2.1.1.m1.2.2.1.2" mathcolor="#000000" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E2Xa.2.1.1.m1.2b"><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1"><eq id="S2.E2Xa.2.1.1.m1.2.2.1.1.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.2"></eq><csymbol cd="latexml" id="S2.E2Xa.2.1.1.m1.2.2.1.1.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.3">absent</csymbol><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1"><csymbol cd="latexml" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.2.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.2">delimited-[]</csymbol><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.3a.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1"><in id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.1"></in><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.2">𝐖</ci><cn id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.3.cmml" type="integer" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.3">1</cn></apply><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.2">ℝ</ci><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3"><times id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.1"></times><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.2">𝑑</ci><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3">subscript</csymbol><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3">superscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.2">𝑑</ci><cn id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.3.cmml" type="integer" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.2.3">1</cn></apply><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.3a.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.3"><mtext id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.3.cmml" mathsize="50%" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.3">hidden</mtext></ci></apply></apply></apply></apply><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.3a.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1"><in id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.2"></in><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3">subscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.2">𝐖</ci><cn id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.3.cmml" type="integer" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.3.3">2</cn></apply><list id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1"><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1">superscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.2">ℝ</ci><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3"><times id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.1"></times><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.2">𝑑</ci><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3">subscript</csymbol><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3">superscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.2">𝑑</ci><cn id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.3.cmml" type="integer" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.2.3">2</cn></apply><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.3a.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.3"><mtext id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.3.cmml" mathsize="50%" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.1.1.1.1.1.3.3.3">hidden</mtext></ci></apply></apply></apply><ci id="S2.E2Xa.2.1.1.m1.1.1.cmml" xref="S2.E2Xa.2.1.1.m1.1.1">…</ci></list></apply><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2"><in id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.1"></in><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2">subscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.2">𝐖</ci><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.2.3">𝑁</ci></apply><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3">superscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.2">ℝ</ci><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3"><times id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.1"></times><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.2">𝑑</ci><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3">subscript</csymbol><apply id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3"><csymbol cd="ambiguous" id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.1.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3">superscript</csymbol><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.2.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.2">𝑑</ci><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.3.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.2.3">𝑁</ci></apply><ci id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.3a.cmml" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.3"><mtext id="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.3.cmml" mathsize="50%" xref="S2.E2Xa.2.1.1.m1.2.2.1.1.1.1.1.2.2.2.2.3.3.3.3">hidden</mtext></ci></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2Xa.2.1.1.m1.2c">\displaystyle=[\mathbf{W}_{1}\in\mathbb{R}^{d\times d^{1}_{\text{hidden}}},% \mathbf{W}_{2}\in\mathbb{R}^{d\times d^{2}_{\text{hidden}}},\ldots,\mathbf{W}_% {N}\in\mathbb{R}^{d\times d^{N}_{\text{hidden}}}],</annotation><annotation encoding="application/x-llamapun" id="S2.E2Xa.2.1.1.m1.2d">= [ bold_W start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_d × italic_d start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT end_POSTSUPERSCRIPT , bold_W start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_d × italic_d start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT end_POSTSUPERSCRIPT , … , bold_W start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_d × italic_d start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ] ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> </tbody> </table> where <math alttext="\mathbf{W}_{n}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.17.1.m1.1"><semantics id="S2.SS2.SSS1.p1.17.1.m1.1a"><msub id="S2.SS2.SSS1.p1.17.1.m1.1.1" xref="S2.SS2.SSS1.p1.17.1.m1.1.1.cmml"><mi id="S2.SS2.SSS1.p1.17.1.m1.1.1.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.17.1.m1.1.1.2.cmml">𝐖</mi><mi id="S2.SS2.SSS1.p1.17.1.m1.1.1.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.17.1.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.17.1.m1.1b"><apply id="S2.SS2.SSS1.p1.17.1.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.17.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.17.1.m1.1.1.1.cmml" xref="S2.SS2.SSS1.p1.17.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.SSS1.p1.17.1.m1.1.1.2.cmml" xref="S2.SS2.SSS1.p1.17.1.m1.1.1.2">𝐖</ci><ci id="S2.SS2.SSS1.p1.17.1.m1.1.1.3.cmml" xref="S2.SS2.SSS1.p1.17.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.17.1.m1.1c">\mathbf{W}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.17.1.m1.1d">bold_W start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> represents the portion of the weight matrix <math alttext="\mathbf{W}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.18.2.m2.1"><semantics id="S2.SS2.SSS1.p1.18.2.m2.1a"><mi id="S2.SS2.SSS1.p1.18.2.m2.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.18.2.m2.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.18.2.m2.1b"><ci id="S2.SS2.SSS1.p1.18.2.m2.1.1.cmml" xref="S2.SS2.SSS1.p1.18.2.m2.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.18.2.m2.1c">\mathbf{W}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.18.2.m2.1d">bold_W</annotation></semantics></math> assigned to device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.19.3.m3.1"><semantics id="S2.SS2.SSS1.p1.19.3.m3.1a"><mi id="S2.SS2.SSS1.p1.19.3.m3.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.19.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.19.3.m3.1b"><ci id="S2.SS2.SSS1.p1.19.3.m3.1.1.cmml" xref="S2.SS2.SSS1.p1.19.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.19.3.m3.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.19.3.m3.1d">italic_n</annotation></semantics></math> and <math alttext="d_{\text{hidden}}=\sum_{n=1}^{N}d^{n}_{\text{hidden}}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.20.4.m4.1"><semantics id="S2.SS2.SSS1.p1.20.4.m4.1a"><mrow id="S2.SS2.SSS1.p1.20.4.m4.1.1" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.cmml"><msub id="S2.SS2.SSS1.p1.20.4.m4.1.1.2" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.2.cmml"><mi id="S2.SS2.SSS1.p1.20.4.m4.1.1.2.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.2.2.cmml">d</mi><mtext id="S2.SS2.SSS1.p1.20.4.m4.1.1.2.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.2.3a.cmml">hidden</mtext></msub><mo id="S2.SS2.SSS1.p1.20.4.m4.1.1.1" mathcolor="#000000" rspace="0.111em" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.1.cmml">=</mo><mrow id="S2.SS2.SSS1.p1.20.4.m4.1.1.3" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.cmml"><msubsup id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.cmml"><mo id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.2.cmml">∑</mo><mrow id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.cmml"><mi id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.2.cmml">n</mi><mo id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.1.cmml">=</mo><mn id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.3.cmml">1</mn></mrow><mi id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.3.cmml">N</mi></msubsup><msubsup id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.cmml"><mi id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.2.cmml">d</mi><mtext id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.3a.cmml">hidden</mtext><mi id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.3.cmml">n</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.20.4.m4.1b"><apply id="S2.SS2.SSS1.p1.20.4.m4.1.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1"><eq id="S2.SS2.SSS1.p1.20.4.m4.1.1.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.1"></eq><apply id="S2.SS2.SSS1.p1.20.4.m4.1.1.2.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.20.4.m4.1.1.2.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.2">subscript</csymbol><ci id="S2.SS2.SSS1.p1.20.4.m4.1.1.2.2.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.2.2">𝑑</ci><ci id="S2.SS2.SSS1.p1.20.4.m4.1.1.2.3a.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.2.3"><mtext id="S2.SS2.SSS1.p1.20.4.m4.1.1.2.3.cmml" mathcolor="#000000" mathsize="70%" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.2.3">hidden</mtext></ci></apply><apply id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3"><apply id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1">superscript</csymbol><apply id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1">subscript</csymbol><sum id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.2.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.2"></sum><apply id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3"><eq id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.1"></eq><ci id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.2.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.2">𝑛</ci><cn id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.3.cmml" type="integer" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.2.3.3">1</cn></apply></apply><ci id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.3.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.1.3">𝑁</ci></apply><apply id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2">subscript</csymbol><apply id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.1.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2">superscript</csymbol><ci id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.2.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.2">𝑑</ci><ci id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.3.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.2.3">𝑛</ci></apply><ci id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.3a.cmml" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.3"><mtext id="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.3.cmml" mathcolor="#000000" mathsize="70%" xref="S2.SS2.SSS1.p1.20.4.m4.1.1.3.2.3">hidden</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.20.4.m4.1c">d_{\text{hidden}}=\sum_{n=1}^{N}d^{n}_{\text{hidden}}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.20.4.m4.1d">italic_d start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT italic_d start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT</annotation></semantics></math>. Similarly, the weight matrix <math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.21.5.m5.1"><semantics id="S2.SS2.SSS1.p1.21.5.m5.1a"><mi id="S2.SS2.SSS1.p1.21.5.m5.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.21.5.m5.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.21.5.m5.1b"><ci id="S2.SS2.SSS1.p1.21.5.m5.1.1.cmml" xref="S2.SS2.SSS1.p1.21.5.m5.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.21.5.m5.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.21.5.m5.1d">bold_U</annotation></semantics></math> is partitioned row-wise as <table class="ltx_equationgroup ltx_eqn_table" id="S2.E3"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E3X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{U}=\begin{bmatrix}\mathbf{U}_{1}\in\mathbb{R}^{d^{1}_{% \text{hidden}}\times d}\\ \mathbf{U}_{2}\in\mathbb{R}^{d^{2}_{\text{hidden}}\times d}\\ \ldots\\ \mathbf{U}_{N}\in\mathbb{R}^{d^{N}_{\text{hidden}}\times d}\end{bmatrix}," class="ltx_Math" display="inline" id="S2.E3X.2.1.1.m1.1"><semantics id="S2.E3X.2.1.1.m1.1a"><mrow id="S2.E3X.2.1.1.m1.1.1.1" xref="S2.E3X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E3X.2.1.1.m1.1.1.1.1" xref="S2.E3X.2.1.1.m1.1.1.1.1.cmml"><mi id="S2.E3X.2.1.1.m1.1.1.1.1.2" mathcolor="#000000" xref="S2.E3X.2.1.1.m1.1.1.1.1.2.cmml">𝐔</mi><mo id="S2.E3X.2.1.1.m1.1.1.1.1.1" mathcolor="#000000" xref="S2.E3X.2.1.1.m1.1.1.1.1.1.cmml">=</mo><mrow id="S2.E3.m1.1.1.1.1.1.1a.3" xref="S2.E3.m1.1.1.1.1.1.1a.2.cmml"><mo id="S2.E3.m1.1.1.1.1.1.1a.3.1" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1a.2.1.cmml">[</mo><mtable id="S2.E3.m1.1.1.1.1.1.1.1.1.mf" rowspacing="0pt" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtr id="S2.E3.m1.1.1.1.1.1.1.1.1.mfa" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtd id="S2.E3.m1.1.1.1.1.1.1.1.1.mfb" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.cmml"><msub id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.2.cmml">𝐔</mi><mn id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.3.cmml">1</mn></msub><mo id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.1" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.1.cmml">∈</mo><msup id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.2.cmml">ℝ</mi><mrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.cmml"><msubsup id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.2.cmml">d</mi><mtext id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.3a.cmml">hidden</mtext><mn id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.3.cmml">1</mn></msubsup><mo id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.1" lspace="0.222em" mathcolor="#000000" rspace="0.222em" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.1.cmml">×</mo><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow></mtd></mtr><mtr id="S2.E3.m1.1.1.1.1.1.1.1.1.mfc" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtd id="S2.E3.m1.1.1.1.1.1.1.1.1.mfd" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.cmml"><msub id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.2.cmml">𝐔</mi><mn id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.3.cmml">2</mn></msub><mo id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.1" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.1.cmml">∈</mo><msup id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.2.cmml">ℝ</mi><mrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.cmml"><msubsup id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.2.cmml">d</mi><mtext id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.3a.cmml">hidden</mtext><mn id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.3.cmml">2</mn></msubsup><mo id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.1" lspace="0.222em" mathcolor="#000000" rspace="0.222em" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.1.cmml">×</mo><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow></mtd></mtr><mtr id="S2.E3.m1.1.1.1.1.1.1.1.1.mfe" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtd id="S2.E3.m1.1.1.1.1.1.1.1.1.mff" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.3.1.1" mathcolor="#000000" mathvariant="normal" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.3.1.1.cmml">…</mi></mtd></mtr><mtr id="S2.E3.m1.1.1.1.1.1.1.1.1.mfg" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtd id="S2.E3.m1.1.1.1.1.1.1.1.1.mfh" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml"><mrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.cmml"><msub id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.2.cmml">𝐔</mi><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.3.cmml">N</mi></msub><mo id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.1" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.1.cmml">∈</mo><msup id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.2.cmml">ℝ</mi><mrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.cmml"><msubsup id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.2.cmml">d</mi><mtext id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.3a.cmml">hidden</mtext><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.3.cmml">N</mi></msubsup><mo id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.1" lspace="0.222em" mathcolor="#000000" rspace="0.222em" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.1.cmml">×</mo><mi id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.3" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow></mtd></mtr></mtable><mo id="S2.E3.m1.1.1.1.1.1.1a.3.2" mathcolor="#000000" xref="S2.E3.m1.1.1.1.1.1.1a.2.1.cmml">]</mo></mrow></mrow><mo id="S2.E3X.2.1.1.m1.1.1.1.2" mathcolor="#000000" xref="S2.E3X.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E3X.2.1.1.m1.1b"><apply id="S2.E3X.2.1.1.m1.1.1.1.1.cmml" xref="S2.E3X.2.1.1.m1.1.1.1"><eq id="S2.E3X.2.1.1.m1.1.1.1.1.1.cmml" xref="S2.E3X.2.1.1.m1.1.1.1.1.1"></eq><ci id="S2.E3X.2.1.1.m1.1.1.1.1.2.cmml" xref="S2.E3X.2.1.1.m1.1.1.1.1.2">𝐔</ci><apply id="S2.E3.m1.1.1.1.1.1.1a.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1a.3"><csymbol cd="latexml" id="S2.E3.m1.1.1.1.1.1.1a.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1a.3.1">matrix</csymbol><matrix id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf"><matrixrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mfa.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf"><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1"><in id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.1"></in><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2">subscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.2">𝐔</ci><cn id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.3.cmml" type="integer" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.3">1</cn></apply><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3">superscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.2">ℝ</ci><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3"><times id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.1"></times><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2">subscript</csymbol><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2">superscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.2">𝑑</ci><cn id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.3.cmml" type="integer" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.2.3">1</cn></apply><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.3a.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.3"><mtext id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.3.cmml" mathsize="50%" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.2.3">hidden</mtext></ci></apply><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.3.3">𝑑</ci></apply></apply></apply></matrixrow><matrixrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mfb.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf"><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1"><in id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.1"></in><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2">subscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.2">𝐔</ci><cn id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.3.cmml" type="integer" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.3">2</cn></apply><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3">superscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.2">ℝ</ci><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3"><times id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.1"></times><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2">subscript</csymbol><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2">superscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.2">𝑑</ci><cn id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.3.cmml" type="integer" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.2.3">2</cn></apply><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.3a.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.3"><mtext id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.3.cmml" mathsize="50%" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.2.3">hidden</mtext></ci></apply><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.3.3">𝑑</ci></apply></apply></apply></matrixrow><matrixrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mfc.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf"><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.3.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.3.1.1">…</ci></matrixrow><matrixrow id="S2.E3.m1.1.1.1.1.1.1.1.1.mfd.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf"><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1"><in id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.1"></in><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2">subscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.2">𝐔</ci><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.3">𝑁</ci></apply><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3">superscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.2">ℝ</ci><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3"><times id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.1"></times><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2">subscript</csymbol><apply id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2">superscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.2">𝑑</ci><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.2.3">𝑁</ci></apply><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.3a.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.3"><mtext id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.3.cmml" mathsize="50%" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.2.3">hidden</mtext></ci></apply><ci id="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.3.3">𝑑</ci></apply></apply></apply></matrixrow></matrix></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3X.2.1.1.m1.1c">\displaystyle\mathbf{U}=\begin{bmatrix}\mathbf{U}_{1}\in\mathbb{R}^{d^{1}_{% \text{hidden}}\times d}\\ \mathbf{U}_{2}\in\mathbb{R}^{d^{2}_{\text{hidden}}\times d}\\ \ldots\\ \mathbf{U}_{N}\in\mathbb{R}^{d^{N}_{\text{hidden}}\times d}\end{bmatrix},</annotation><annotation encoding="application/x-llamapun" id="S2.E3X.2.1.1.m1.1d">bold_U = [ start_ARG start_ROW start_CELL bold_U start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_d start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT × italic_d end_POSTSUPERSCRIPT end_CELL end_ROW start_ROW start_CELL bold_U start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_d start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT × italic_d end_POSTSUPERSCRIPT end_CELL end_ROW start_ROW start_CELL … end_CELL end_ROW start_ROW start_CELL bold_U start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_d start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT start_POSTSUBSCRIPT hidden end_POSTSUBSCRIPT × italic_d end_POSTSUPERSCRIPT end_CELL end_ROW end_ARG ] ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(3)</td> </tr> </tbody> </table> where <math alttext="\mathbf{U}_{n}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.22.1.m1.1"><semantics id="S2.SS2.SSS1.p1.22.1.m1.1a"><msub id="S2.SS2.SSS1.p1.22.1.m1.1.1" xref="S2.SS2.SSS1.p1.22.1.m1.1.1.cmml"><mi id="S2.SS2.SSS1.p1.22.1.m1.1.1.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.22.1.m1.1.1.2.cmml">𝐔</mi><mi id="S2.SS2.SSS1.p1.22.1.m1.1.1.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.22.1.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.22.1.m1.1b"><apply id="S2.SS2.SSS1.p1.22.1.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.22.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.22.1.m1.1.1.1.cmml" xref="S2.SS2.SSS1.p1.22.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.SSS1.p1.22.1.m1.1.1.2.cmml" xref="S2.SS2.SSS1.p1.22.1.m1.1.1.2">𝐔</ci><ci id="S2.SS2.SSS1.p1.22.1.m1.1.1.3.cmml" xref="S2.SS2.SSS1.p1.22.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.22.1.m1.1c">\mathbf{U}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.22.1.m1.1d">bold_U start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> represents the portion of the weight matrix <math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.23.2.m2.1"><semantics id="S2.SS2.SSS1.p1.23.2.m2.1a"><mi id="S2.SS2.SSS1.p1.23.2.m2.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.23.2.m2.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.23.2.m2.1b"><ci id="S2.SS2.SSS1.p1.23.2.m2.1.1.cmml" xref="S2.SS2.SSS1.p1.23.2.m2.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.23.2.m2.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.23.2.m2.1d">bold_U</annotation></semantics></math> assigned to device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.24.3.m3.1"><semantics id="S2.SS2.SSS1.p1.24.3.m3.1a"><mi id="S2.SS2.SSS1.p1.24.3.m3.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.24.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.24.3.m3.1b"><ci id="S2.SS2.SSS1.p1.24.3.m3.1.1.cmml" xref="S2.SS2.SSS1.p1.24.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.24.3.m3.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.24.3.m3.1d">italic_n</annotation></semantics></math>. Then, each device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.25.m1.1"><semantics id="S2.SS2.SSS1.p1.25.m1.1a"><mi id="S2.SS2.SSS1.p1.25.m1.1.1" xref="S2.SS2.SSS1.p1.25.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.25.m1.1b"><ci id="S2.SS2.SSS1.p1.25.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.25.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.25.m1.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.25.m1.1d">italic_n</annotation></semantics></math> can perform the forward computation on its respective model segment as follows, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E4"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E4X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{Z}_{n}=\mathrm{max}(0,\mathbf{X}\mathbf{W}_{n})\mathbf{U}% _{n}," class="ltx_Math" display="inline" id="S2.E4X.2.1.1.m1.2"><semantics id="S2.E4X.2.1.1.m1.2a"><mrow id="S2.E4X.2.1.1.m1.2.2.1" xref="S2.E4X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S2.E4X.2.1.1.m1.2.2.1.1" xref="S2.E4X.2.1.1.m1.2.2.1.1.cmml"><msub id="S2.E4X.2.1.1.m1.2.2.1.1.3" xref="S2.E4X.2.1.1.m1.2.2.1.1.3.cmml"><mi id="S2.E4X.2.1.1.m1.2.2.1.1.3.2" xref="S2.E4X.2.1.1.m1.2.2.1.1.3.2.cmml">𝐙</mi><mi id="S2.E4X.2.1.1.m1.2.2.1.1.3.3" xref="S2.E4X.2.1.1.m1.2.2.1.1.3.3.cmml">n</mi></msub><mo id="S2.E4X.2.1.1.m1.2.2.1.1.2" xref="S2.E4X.2.1.1.m1.2.2.1.1.2.cmml">=</mo><mrow id="S2.E4X.2.1.1.m1.2.2.1.1.1" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.cmml"><mi id="S2.E4X.2.1.1.m1.2.2.1.1.1.3" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.3.cmml">max</mi><mo id="S2.E4X.2.1.1.m1.2.2.1.1.1.2" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.2.cmml">⁢</mo><mrow id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.2.cmml"><mo id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.2" stretchy="false" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.2.cmml">(</mo><mn id="S2.E4X.2.1.1.m1.1.1" xref="S2.E4X.2.1.1.m1.1.1.cmml">0</mn><mo id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.3" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.2.cmml">,</mo><msub id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.cmml"><mi id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.2" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml">𝐗𝐖</mi><mi id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.4" stretchy="false" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.2.cmml">)</mo></mrow><mo id="S2.E4X.2.1.1.m1.2.2.1.1.1.2a" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.2.cmml">⁢</mo><msub id="S2.E4X.2.1.1.m1.2.2.1.1.1.4" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.4.cmml"><mi id="S2.E4X.2.1.1.m1.2.2.1.1.1.4.2" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.4.2.cmml">𝐔</mi><mi id="S2.E4X.2.1.1.m1.2.2.1.1.1.4.3" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.4.3.cmml">n</mi></msub></mrow></mrow><mo id="S2.E4X.2.1.1.m1.2.2.1.2" xref="S2.E4X.2.1.1.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E4X.2.1.1.m1.2b"><apply id="S2.E4X.2.1.1.m1.2.2.1.1.cmml" xref="S2.E4X.2.1.1.m1.2.2.1"><eq id="S2.E4X.2.1.1.m1.2.2.1.1.2.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.2"></eq><apply id="S2.E4X.2.1.1.m1.2.2.1.1.3.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.3"><csymbol cd="ambiguous" id="S2.E4X.2.1.1.m1.2.2.1.1.3.1.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.3">subscript</csymbol><ci id="S2.E4X.2.1.1.m1.2.2.1.1.3.2.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.3.2">𝐙</ci><ci id="S2.E4X.2.1.1.m1.2.2.1.1.3.3.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.3.3">𝑛</ci></apply><apply id="S2.E4X.2.1.1.m1.2.2.1.1.1.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1"><times id="S2.E4X.2.1.1.m1.2.2.1.1.1.2.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.2"></times><ci id="S2.E4X.2.1.1.m1.2.2.1.1.1.3.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.3">max</ci><interval closure="open" id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.2.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1"><cn id="S2.E4X.2.1.1.m1.1.1.cmml" type="integer" xref="S2.E4X.2.1.1.m1.1.1">0</cn><apply id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.2">𝐗𝐖</ci><ci id="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3">𝑛</ci></apply></interval><apply id="S2.E4X.2.1.1.m1.2.2.1.1.1.4.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.4"><csymbol cd="ambiguous" id="S2.E4X.2.1.1.m1.2.2.1.1.1.4.1.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.4">subscript</csymbol><ci id="S2.E4X.2.1.1.m1.2.2.1.1.1.4.2.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.4.2">𝐔</ci><ci id="S2.E4X.2.1.1.m1.2.2.1.1.1.4.3.cmml" xref="S2.E4X.2.1.1.m1.2.2.1.1.1.4.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E4X.2.1.1.m1.2c">\displaystyle\mathbf{Z}_{n}=\mathrm{max}(0,\mathbf{X}\mathbf{W}_{n})\mathbf{U}% _{n},</annotation><annotation encoding="application/x-llamapun" id="S2.E4X.2.1.1.m1.2d">bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT = roman_max ( 0 , bold_XW start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) bold_U start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(4)</td> </tr> </tbody> </table> where <math alttext="\mathbf{Z}_{n}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.26.m1.1"><semantics id="S2.SS2.SSS1.p1.26.m1.1a"><msub id="S2.SS2.SSS1.p1.26.m1.1.1" xref="S2.SS2.SSS1.p1.26.m1.1.1.cmml"><mi id="S2.SS2.SSS1.p1.26.m1.1.1.2" xref="S2.SS2.SSS1.p1.26.m1.1.1.2.cmml">𝐙</mi><mi id="S2.SS2.SSS1.p1.26.m1.1.1.3" xref="S2.SS2.SSS1.p1.26.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.26.m1.1b"><apply id="S2.SS2.SSS1.p1.26.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.26.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.26.m1.1.1.1.cmml" xref="S2.SS2.SSS1.p1.26.m1.1.1">subscript</csymbol><ci id="S2.SS2.SSS1.p1.26.m1.1.1.2.cmml" xref="S2.SS2.SSS1.p1.26.m1.1.1.2">𝐙</ci><ci id="S2.SS2.SSS1.p1.26.m1.1.1.3.cmml" xref="S2.SS2.SSS1.p1.26.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.26.m1.1c">\mathbf{Z}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.26.m1.1d">bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> is the partial output produced by device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.27.m2.1"><semantics id="S2.SS2.SSS1.p1.27.m2.1a"><mi id="S2.SS2.SSS1.p1.27.m2.1.1" xref="S2.SS2.SSS1.p1.27.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.27.m2.1b"><ci id="S2.SS2.SSS1.p1.27.m2.1.1.cmml" xref="S2.SS2.SSS1.p1.27.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.27.m2.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.27.m2.1d">italic_n</annotation></semantics></math>. Once all devices obtain their local outputs <math alttext="\mathbf{Z}_{n}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.28.m3.1"><semantics id="S2.SS2.SSS1.p1.28.m3.1a"><msub id="S2.SS2.SSS1.p1.28.m3.1.1" xref="S2.SS2.SSS1.p1.28.m3.1.1.cmml"><mi id="S2.SS2.SSS1.p1.28.m3.1.1.2" xref="S2.SS2.SSS1.p1.28.m3.1.1.2.cmml">𝐙</mi><mi id="S2.SS2.SSS1.p1.28.m3.1.1.3" xref="S2.SS2.SSS1.p1.28.m3.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.28.m3.1b"><apply id="S2.SS2.SSS1.p1.28.m3.1.1.cmml" xref="S2.SS2.SSS1.p1.28.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.28.m3.1.1.1.cmml" xref="S2.SS2.SSS1.p1.28.m3.1.1">subscript</csymbol><ci id="S2.SS2.SSS1.p1.28.m3.1.1.2.cmml" xref="S2.SS2.SSS1.p1.28.m3.1.1.2">𝐙</ci><ci id="S2.SS2.SSS1.p1.28.m3.1.1.3.cmml" xref="S2.SS2.SSS1.p1.28.m3.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.28.m3.1c">\mathbf{Z}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.28.m3.1d">bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math>, an all-reduce operation is performed to aggregate the partial outputs from all devices as follows, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E5"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E5X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{Z}=\sum_{n=1}^{N}\mathbf{Z}_{n}." class="ltx_Math" display="inline" id="S2.E5X.2.1.1.m1.1"><semantics id="S2.E5X.2.1.1.m1.1a"><mrow id="S2.E5X.2.1.1.m1.1.1.1" xref="S2.E5X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E5X.2.1.1.m1.1.1.1.1" xref="S2.E5X.2.1.1.m1.1.1.1.1.cmml"><mi id="S2.E5X.2.1.1.m1.1.1.1.1.2" xref="S2.E5X.2.1.1.m1.1.1.1.1.2.cmml">𝐙</mi><mo id="S2.E5X.2.1.1.m1.1.1.1.1.1" xref="S2.E5X.2.1.1.m1.1.1.1.1.1.cmml">=</mo><mrow id="S2.E5X.2.1.1.m1.1.1.1.1.3" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.cmml"><mstyle displaystyle="true" id="S2.E5X.2.1.1.m1.1.1.1.1.3.1" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.cmml"><munderover id="S2.E5X.2.1.1.m1.1.1.1.1.3.1a" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.cmml"><mo id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.2" movablelimits="false" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.2.cmml">∑</mo><mrow id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.cmml"><mi id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.2" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.2.cmml">n</mi><mo id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.1" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.1.cmml">=</mo><mn id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.3" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.3" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.3.cmml">N</mi></munderover></mstyle><msub id="S2.E5X.2.1.1.m1.1.1.1.1.3.2" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.2.cmml"><mi id="S2.E5X.2.1.1.m1.1.1.1.1.3.2.2" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.2.2.cmml">𝐙</mi><mi id="S2.E5X.2.1.1.m1.1.1.1.1.3.2.3" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.2.3.cmml">n</mi></msub></mrow></mrow><mo id="S2.E5X.2.1.1.m1.1.1.1.2" lspace="0em" xref="S2.E5X.2.1.1.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E5X.2.1.1.m1.1b"><apply id="S2.E5X.2.1.1.m1.1.1.1.1.cmml" xref="S2.E5X.2.1.1.m1.1.1.1"><eq id="S2.E5X.2.1.1.m1.1.1.1.1.1.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.1"></eq><ci id="S2.E5X.2.1.1.m1.1.1.1.1.2.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.2">𝐙</ci><apply id="S2.E5X.2.1.1.m1.1.1.1.1.3.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3"><apply id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.1.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1">superscript</csymbol><apply id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.1.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1">subscript</csymbol><sum id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.2.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.2"></sum><apply id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3"><eq id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.1.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.1"></eq><ci id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.2.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.2">𝑛</ci><cn id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.3.cmml" type="integer" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.2.3.3">1</cn></apply></apply><ci id="S2.E5X.2.1.1.m1.1.1.1.1.3.1.3.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.1.3">𝑁</ci></apply><apply id="S2.E5X.2.1.1.m1.1.1.1.1.3.2.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E5X.2.1.1.m1.1.1.1.1.3.2.1.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.E5X.2.1.1.m1.1.1.1.1.3.2.2.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.2.2">𝐙</ci><ci id="S2.E5X.2.1.1.m1.1.1.1.1.3.2.3.cmml" xref="S2.E5X.2.1.1.m1.1.1.1.1.3.2.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E5X.2.1.1.m1.1c">\displaystyle\mathbf{Z}=\sum_{n=1}^{N}\mathbf{Z}_{n}.</annotation><annotation encoding="application/x-llamapun" id="S2.E5X.2.1.1.m1.1d">bold_Z = ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT .</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(5)</td> </tr> </tbody> </table> The validity of this aggregation can be explained by considering how the model parameters are partitioned across the devices. Specifically, concatenating the column slices <math alttext="\mathbf{W}_{n}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.29.1.m1.1"><semantics id="S2.SS2.SSS1.p1.29.1.m1.1a"><msub id="S2.SS2.SSS1.p1.29.1.m1.1.1" xref="S2.SS2.SSS1.p1.29.1.m1.1.1.cmml"><mi id="S2.SS2.SSS1.p1.29.1.m1.1.1.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.29.1.m1.1.1.2.cmml">𝐖</mi><mi id="S2.SS2.SSS1.p1.29.1.m1.1.1.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.29.1.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.29.1.m1.1b"><apply id="S2.SS2.SSS1.p1.29.1.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.29.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.29.1.m1.1.1.1.cmml" xref="S2.SS2.SSS1.p1.29.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.SSS1.p1.29.1.m1.1.1.2.cmml" xref="S2.SS2.SSS1.p1.29.1.m1.1.1.2">𝐖</ci><ci id="S2.SS2.SSS1.p1.29.1.m1.1.1.3.cmml" xref="S2.SS2.SSS1.p1.29.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.29.1.m1.1c">\mathbf{W}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.29.1.m1.1d">bold_W start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> reproduces <math alttext="\mathbf{W}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.30.2.m2.1"><semantics id="S2.SS2.SSS1.p1.30.2.m2.1a"><mi id="S2.SS2.SSS1.p1.30.2.m2.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.30.2.m2.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.30.2.m2.1b"><ci id="S2.SS2.SSS1.p1.30.2.m2.1.1.cmml" xref="S2.SS2.SSS1.p1.30.2.m2.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.30.2.m2.1c">\mathbf{W}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.30.2.m2.1d">bold_W</annotation></semantics></math>, and stacking the row slices <math alttext="\mathbf{U}_{n}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.31.3.m3.1"><semantics id="S2.SS2.SSS1.p1.31.3.m3.1a"><msub id="S2.SS2.SSS1.p1.31.3.m3.1.1" xref="S2.SS2.SSS1.p1.31.3.m3.1.1.cmml"><mi id="S2.SS2.SSS1.p1.31.3.m3.1.1.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.31.3.m3.1.1.2.cmml">𝐔</mi><mi id="S2.SS2.SSS1.p1.31.3.m3.1.1.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.31.3.m3.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.31.3.m3.1b"><apply id="S2.SS2.SSS1.p1.31.3.m3.1.1.cmml" xref="S2.SS2.SSS1.p1.31.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.31.3.m3.1.1.1.cmml" xref="S2.SS2.SSS1.p1.31.3.m3.1.1">subscript</csymbol><ci id="S2.SS2.SSS1.p1.31.3.m3.1.1.2.cmml" xref="S2.SS2.SSS1.p1.31.3.m3.1.1.2">𝐔</ci><ci id="S2.SS2.SSS1.p1.31.3.m3.1.1.3.cmml" xref="S2.SS2.SSS1.p1.31.3.m3.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.31.3.m3.1c">\mathbf{U}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.31.3.m3.1d">bold_U start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> recovers <math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.32.4.m4.1"><semantics id="S2.SS2.SSS1.p1.32.4.m4.1a"><mi id="S2.SS2.SSS1.p1.32.4.m4.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.32.4.m4.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.32.4.m4.1b"><ci id="S2.SS2.SSS1.p1.32.4.m4.1.1.cmml" xref="S2.SS2.SSS1.p1.32.4.m4.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.32.4.m4.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.32.4.m4.1d">bold_U</annotation></semantics></math>. Consequently, the original unpartitioned MLP output <math alttext="\mathbf{Z}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.33.5.m5.1"><semantics id="S2.SS2.SSS1.p1.33.5.m5.1a"><mi id="S2.SS2.SSS1.p1.33.5.m5.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.33.5.m5.1.1.cmml">𝐙</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.33.5.m5.1b"><ci id="S2.SS2.SSS1.p1.33.5.m5.1.1.cmml" xref="S2.SS2.SSS1.p1.33.5.m5.1.1">𝐙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.33.5.m5.1c">\mathbf{Z}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.33.5.m5.1d">bold_Z</annotation></semantics></math> can be expressed as <table class="ltx_equationgroup ltx_eqn_table" id="S2.E6"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E6X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{Z}" class="ltx_Math" display="inline" id="S2.E6X.2.1.1.m1.1"><semantics id="S2.E6X.2.1.1.m1.1a"><mi id="S2.E6X.2.1.1.m1.1.1" mathcolor="#000000" xref="S2.E6X.2.1.1.m1.1.1.cmml">𝐙</mi><annotation-xml encoding="MathML-Content" id="S2.E6X.2.1.1.m1.1b"><ci id="S2.E6X.2.1.1.m1.1.1.cmml" xref="S2.E6X.2.1.1.m1.1.1">𝐙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E6X.2.1.1.m1.1c">\displaystyle\mathbf{Z}</annotation><annotation encoding="application/x-llamapun" id="S2.E6X.2.1.1.m1.1d">bold_Z</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle=\max(\mathbf{0},\mathbf{XW})\mathbf{U}" class="ltx_Math" display="inline" id="S2.E6X.3.2.2.m1.3"><semantics id="S2.E6X.3.2.2.m1.3a"><mrow id="S2.E6X.3.2.2.m1.3.4" xref="S2.E6X.3.2.2.m1.3.4.cmml"><mi id="S2.E6X.3.2.2.m1.3.4.2" xref="S2.E6X.3.2.2.m1.3.4.2.cmml"></mi><mo id="S2.E6X.3.2.2.m1.3.4.1" mathcolor="#000000" xref="S2.E6X.3.2.2.m1.3.4.1.cmml">=</mo><mrow id="S2.E6X.3.2.2.m1.3.4.3" xref="S2.E6X.3.2.2.m1.3.4.3.cmml"><mrow id="S2.E6X.3.2.2.m1.3.4.3.2.2" xref="S2.E6X.3.2.2.m1.3.4.3.2.1.cmml"><mi id="S2.E6X.3.2.2.m1.1.1" mathcolor="#000000" xref="S2.E6X.3.2.2.m1.1.1.cmml">max</mi><mo id="S2.E6X.3.2.2.m1.3.4.3.2.2a" xref="S2.E6X.3.2.2.m1.3.4.3.2.1.cmml">⁡</mo><mrow id="S2.E6X.3.2.2.m1.3.4.3.2.2.1" xref="S2.E6X.3.2.2.m1.3.4.3.2.1.cmml"><mo id="S2.E6X.3.2.2.m1.3.4.3.2.2.1.1" mathcolor="#000000" stretchy="false" xref="S2.E6X.3.2.2.m1.3.4.3.2.1.cmml">(</mo><mn id="S2.E6X.3.2.2.m1.2.2" mathcolor="#000000" xref="S2.E6X.3.2.2.m1.2.2.cmml">𝟎</mn><mo id="S2.E6X.3.2.2.m1.3.4.3.2.2.1.2" mathcolor="#000000" xref="S2.E6X.3.2.2.m1.3.4.3.2.1.cmml">,</mo><mi id="S2.E6X.3.2.2.m1.3.3" mathcolor="#000000" xref="S2.E6X.3.2.2.m1.3.3.cmml">𝐗𝐖</mi><mo id="S2.E6X.3.2.2.m1.3.4.3.2.2.1.3" mathcolor="#000000" stretchy="false" xref="S2.E6X.3.2.2.m1.3.4.3.2.1.cmml">)</mo></mrow></mrow><mo id="S2.E6X.3.2.2.m1.3.4.3.1" xref="S2.E6X.3.2.2.m1.3.4.3.1.cmml">⁢</mo><mi id="S2.E6X.3.2.2.m1.3.4.3.3" mathcolor="#000000" xref="S2.E6X.3.2.2.m1.3.4.3.3.cmml">𝐔</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E6X.3.2.2.m1.3b"><apply id="S2.E6X.3.2.2.m1.3.4.cmml" xref="S2.E6X.3.2.2.m1.3.4"><eq id="S2.E6X.3.2.2.m1.3.4.1.cmml" xref="S2.E6X.3.2.2.m1.3.4.1"></eq><csymbol cd="latexml" id="S2.E6X.3.2.2.m1.3.4.2.cmml" xref="S2.E6X.3.2.2.m1.3.4.2">absent</csymbol><apply id="S2.E6X.3.2.2.m1.3.4.3.cmml" xref="S2.E6X.3.2.2.m1.3.4.3"><times id="S2.E6X.3.2.2.m1.3.4.3.1.cmml" xref="S2.E6X.3.2.2.m1.3.4.3.1"></times><apply id="S2.E6X.3.2.2.m1.3.4.3.2.1.cmml" xref="S2.E6X.3.2.2.m1.3.4.3.2.2"><max id="S2.E6X.3.2.2.m1.1.1.cmml" xref="S2.E6X.3.2.2.m1.1.1"></max><cn id="S2.E6X.3.2.2.m1.2.2.cmml" type="integer" xref="S2.E6X.3.2.2.m1.2.2">0</cn><ci id="S2.E6X.3.2.2.m1.3.3.cmml" xref="S2.E6X.3.2.2.m1.3.3">𝐗𝐖</ci></apply><ci id="S2.E6X.3.2.2.m1.3.4.3.3.cmml" xref="S2.E6X.3.2.2.m1.3.4.3.3">𝐔</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E6X.3.2.2.m1.3c">\displaystyle=\max(\mathbf{0},\mathbf{XW})\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.E6X.3.2.2.m1.3d">= roman_max ( bold_0 , bold_XW ) bold_U</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="4">(6)</td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E6Xa"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle=\max(\mathbf{0},\mathbf{X[\mathbf{W}_{1},\mathbf{W}_{2},\ldots,% \mathbf{W}_{N}]})\begin{bmatrix}\mathbf{U}_{1}\\ \mathbf{U}_{2}\\ \ldots\\ \mathbf{U}_{N}\end{bmatrix}" class="ltx_Math" display="inline" id="S2.E6Xa.2.1.1.m1.4"><semantics id="S2.E6Xa.2.1.1.m1.4a"><mrow id="S2.E6Xa.2.1.1.m1.4.4" xref="S2.E6Xa.2.1.1.m1.4.4.cmml"><mi id="S2.E6Xa.2.1.1.m1.4.4.3" xref="S2.E6Xa.2.1.1.m1.4.4.3.cmml"></mi><mo id="S2.E6Xa.2.1.1.m1.4.4.2" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.2.cmml">=</mo><mrow id="S2.E6Xa.2.1.1.m1.4.4.1" xref="S2.E6Xa.2.1.1.m1.4.4.1.cmml"><mrow id="S2.E6Xa.2.1.1.m1.4.4.1.1.1" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.2.cmml"><mi id="S2.E6Xa.2.1.1.m1.2.2" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.2.2.cmml">max</mi><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1a" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.2.cmml">⁡</mo><mrow id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.2.cmml"><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.2" mathcolor="#000000" stretchy="false" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.2.cmml">(</mo><mn id="S2.E6Xa.2.1.1.m1.3.3" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.3.3.cmml">𝟎</mn><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.3" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.2.cmml">,</mo><mrow id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml"><mi id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.5" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.5.cmml">𝐗</mi><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.4" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.4.cmml">⁢</mo><mrow id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.4.cmml"><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.4" mathcolor="#000000" stretchy="false" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.4.cmml">[</mo><msub id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.2" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.2.cmml">𝐖</mi><mn id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.3" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.3.cmml">𝟏</mn></msub><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.5" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.4.cmml">,</mo><msub id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.cmml"><mi id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2.cmml">𝐖</mi><mn id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3.cmml">𝟐</mn></msub><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.6" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.4.cmml">,</mo><mi id="S2.E6Xa.2.1.1.m1.1.1" mathcolor="#000000" mathvariant="normal" xref="S2.E6Xa.2.1.1.m1.1.1.cmml">…</mi><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.7" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.4.cmml">,</mo><msub id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.cmml"><mi id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.2" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.2.cmml">𝐖</mi><mi id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.3" mathcolor="#000000" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.3.cmml">𝐍</mi></msub><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.8" mathcolor="#000000" stretchy="false" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.4.cmml">]</mo></mrow></mrow><mo id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.4" mathcolor="#000000" stretchy="false" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.2.cmml">)</mo></mrow></mrow><mo id="S2.E6Xa.2.1.1.m1.4.4.1.2" xref="S2.E6Xa.2.1.1.m1.4.4.1.2.cmml">⁢</mo><mrow id="S2.E6.m1.1.1.1.1.1.1a.3" xref="S2.E6.m1.1.1.1.1.1.1a.2.cmml"><mo id="S2.E6.m1.1.1.1.1.1.1a.3.1" mathcolor="#000000" xref="S2.E6.m1.1.1.1.1.1.1a.2.1.cmml">[</mo><mtable id="S2.E6.m1.1.1.1.1.1.1.1.1.mf" rowspacing="0pt" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtr id="S2.E6.m1.1.1.1.1.1.1.1.1.mfa" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtd id="S2.E6.m1.1.1.1.1.1.1.1.1.mfb" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><msub id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.cmml"><mi id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2" mathcolor="#000000" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.cmml">𝐔</mi><mn id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3" mathcolor="#000000" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.cmml">1</mn></msub></mtd></mtr><mtr id="S2.E6.m1.1.1.1.1.1.1.1.1.mfc" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtd id="S2.E6.m1.1.1.1.1.1.1.1.1.mfd" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><msub id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.cmml"><mi id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2" mathcolor="#000000" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.cmml">𝐔</mi><mn id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3" mathcolor="#000000" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.cmml">2</mn></msub></mtd></mtr><mtr id="S2.E6.m1.1.1.1.1.1.1.1.1.mfe" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtd id="S2.E6.m1.1.1.1.1.1.1.1.1.mff" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><mi id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.3.1.1" mathcolor="#000000" mathvariant="normal" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.3.1.1.cmml">…</mi></mtd></mtr><mtr id="S2.E6.m1.1.1.1.1.1.1.1.1.mfg" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><mtd id="S2.E6.m1.1.1.1.1.1.1.1.1.mfh" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml"><msub id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.cmml"><mi id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2" mathcolor="#000000" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.cmml">𝐔</mi><mi id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3" mathcolor="#000000" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.cmml">N</mi></msub></mtd></mtr></mtable><mo id="S2.E6.m1.1.1.1.1.1.1a.3.2" mathcolor="#000000" xref="S2.E6.m1.1.1.1.1.1.1a.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E6Xa.2.1.1.m1.4b"><apply id="S2.E6Xa.2.1.1.m1.4.4.cmml" xref="S2.E6Xa.2.1.1.m1.4.4"><eq id="S2.E6Xa.2.1.1.m1.4.4.2.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.2"></eq><csymbol cd="latexml" id="S2.E6Xa.2.1.1.m1.4.4.3.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.3">absent</csymbol><apply id="S2.E6Xa.2.1.1.m1.4.4.1.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1"><times id="S2.E6Xa.2.1.1.m1.4.4.1.2.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.2"></times><apply id="S2.E6Xa.2.1.1.m1.4.4.1.1.2.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1"><max id="S2.E6Xa.2.1.1.m1.2.2.cmml" xref="S2.E6Xa.2.1.1.m1.2.2"></max><cn id="S2.E6Xa.2.1.1.m1.3.3.cmml" type="integer" xref="S2.E6Xa.2.1.1.m1.3.3">0</cn><apply id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1"><times id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.4.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.4"></times><ci id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.5.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.5">𝐗</ci><list id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.4.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3"><apply id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.2">𝐖</ci><cn id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.3.cmml" type="integer" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.1.3">1</cn></apply><apply id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2">𝐖</ci><cn id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3.cmml" type="integer" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3">2</cn></apply><ci id="S2.E6Xa.2.1.1.m1.1.1.cmml" xref="S2.E6Xa.2.1.1.m1.1.1">…</ci><apply id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.1.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3">subscript</csymbol><ci id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.2.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.2">𝐖</ci><ci id="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.3.cmml" xref="S2.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.3.3.3">𝐍</ci></apply></list></apply></apply><apply id="S2.E6.m1.1.1.1.1.1.1a.2.cmml" xref="S2.E6.m1.1.1.1.1.1.1a.3"><csymbol cd="latexml" id="S2.E6.m1.1.1.1.1.1.1a.2.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1a.3.1">matrix</csymbol><matrix id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf"><matrixrow id="S2.E6.m1.1.1.1.1.1.1.1.1.mfa.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf"><apply id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1">subscript</csymbol><ci id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.2">𝐔</ci><cn id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3.cmml" type="integer" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.1.1.1.3">1</cn></apply></matrixrow><matrixrow id="S2.E6.m1.1.1.1.1.1.1.1.1.mfb.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf"><apply id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1">subscript</csymbol><ci id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.2">𝐔</ci><cn id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3.cmml" type="integer" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.2.1.1.3">2</cn></apply></matrixrow><matrixrow id="S2.E6.m1.1.1.1.1.1.1.1.1.mfc.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf"><ci id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.3.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.3.1.1">…</ci></matrixrow><matrixrow id="S2.E6.m1.1.1.1.1.1.1.1.1.mfd.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf"><apply id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1">subscript</csymbol><ci id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.2">𝐔</ci><ci id="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.1.mf.4.1.1.3">𝑁</ci></apply></matrixrow></matrix></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E6Xa.2.1.1.m1.4c">\displaystyle=\max(\mathbf{0},\mathbf{X[\mathbf{W}_{1},\mathbf{W}_{2},\ldots,% \mathbf{W}_{N}]})\begin{bmatrix}\mathbf{U}_{1}\\ \mathbf{U}_{2}\\ \ldots\\ \mathbf{U}_{N}\end{bmatrix}</annotation><annotation encoding="application/x-llamapun" id="S2.E6Xa.2.1.1.m1.4d">= roman_max ( bold_0 , bold_X [ bold_W start_POSTSUBSCRIPT bold_1 end_POSTSUBSCRIPT , bold_W start_POSTSUBSCRIPT bold_2 end_POSTSUBSCRIPT , … , bold_W start_POSTSUBSCRIPT bold_N end_POSTSUBSCRIPT ] ) [ start_ARG start_ROW start_CELL bold_U start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_CELL end_ROW start_ROW start_CELL bold_U start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_CELL end_ROW start_ROW start_CELL … end_CELL end_ROW start_ROW start_CELL bold_U start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT end_CELL end_ROW end_ARG ]</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E6Xb"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle\mathrel{\stackrel{{\scriptstyle\makebox[0.0pt]{\mbox{\tiny(a)}}}% }{{=}}}\sum_{n=1}^{N}\mathrm{max}(0,\mathbf{X}\mathbf{W}_{n})\mathbf{U}_{n}" class="ltx_Math" display="inline" id="S2.E6Xb.2.1.1.m1.2"><semantics id="S2.E6Xb.2.1.1.m1.2a"><mrow id="S2.E6Xb.2.1.1.m1.2.2" xref="S2.E6Xb.2.1.1.m1.2.2.cmml"><mi id="S2.E6Xb.2.1.1.m1.2.2.3" xref="S2.E6Xb.2.1.1.m1.2.2.3.cmml"></mi><mover id="S2.E6Xb.2.1.1.m1.2.2.2" xref="S2.E6Xb.2.1.1.m1.2.2.2.cmml"><mo id="S2.E6Xb.2.1.1.m1.2.2.2.2" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.2.2.cmml">=</mo><mpadded id="S2.E6Xb.2.1.1.m1.2.2.2.3" width="0.0pt" xref="S2.E6Xb.2.1.1.m1.2.2.2.3b.cmml"><mtext id="S2.E6Xb.2.1.1.m1.2.2.2.3a" mathcolor="#000000" mathsize="71%" xref="S2.E6Xb.2.1.1.m1.2.2.2.3b.cmml">(a)</mtext></mpadded></mover><mrow id="S2.E6Xb.2.1.1.m1.2.2.1" xref="S2.E6Xb.2.1.1.m1.2.2.1.cmml"><mstyle displaystyle="true" id="S2.E6Xb.2.1.1.m1.2.2.1.2" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.cmml"><munderover id="S2.E6Xb.2.1.1.m1.2.2.1.2a" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.cmml"><mo id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.2" mathcolor="#000000" movablelimits="false" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.2.cmml">∑</mo><mrow id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.cmml"><mi id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.2" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.2.cmml">n</mi><mo id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.1" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.1.cmml">=</mo><mn id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.3" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.E6Xb.2.1.1.m1.2.2.1.2.3" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.3.cmml">N</mi></munderover></mstyle><mrow id="S2.E6Xb.2.1.1.m1.2.2.1.1" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.cmml"><mi id="S2.E6Xb.2.1.1.m1.2.2.1.1.3" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.3.cmml">max</mi><mo id="S2.E6Xb.2.1.1.m1.2.2.1.1.2" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.2.cmml">⁢</mo><mrow id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.2.cmml"><mo id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.2" mathcolor="#000000" stretchy="false" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.2.cmml">(</mo><mn id="S2.E6Xb.2.1.1.m1.1.1" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.1.1.cmml">0</mn><mo id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.3" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.2.cmml">,</mo><msub id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.cmml"><mi id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.2" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.2.cmml">𝐗𝐖</mi><mi id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.3" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.4" mathcolor="#000000" stretchy="false" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.2.cmml">)</mo></mrow><mo id="S2.E6Xb.2.1.1.m1.2.2.1.1.2a" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.2.cmml">⁢</mo><msub id="S2.E6Xb.2.1.1.m1.2.2.1.1.4" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.4.cmml"><mi id="S2.E6Xb.2.1.1.m1.2.2.1.1.4.2" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.4.2.cmml">𝐔</mi><mi id="S2.E6Xb.2.1.1.m1.2.2.1.1.4.3" mathcolor="#000000" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.4.3.cmml">n</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E6Xb.2.1.1.m1.2b"><apply id="S2.E6Xb.2.1.1.m1.2.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2"><apply id="S2.E6Xb.2.1.1.m1.2.2.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.2"><csymbol cd="ambiguous" id="S2.E6Xb.2.1.1.m1.2.2.2.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.2">superscript</csymbol><eq id="S2.E6Xb.2.1.1.m1.2.2.2.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.2.2"></eq><ci id="S2.E6Xb.2.1.1.m1.2.2.2.3b.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.2.3"><mpadded id="S2.E6Xb.2.1.1.m1.2.2.2.3.cmml" width="0.0pt" xref="S2.E6Xb.2.1.1.m1.2.2.2.3"><mtext id="S2.E6Xb.2.1.1.m1.2.2.2.3a.cmml" mathsize="50%" xref="S2.E6Xb.2.1.1.m1.2.2.2.3">(a)</mtext></mpadded></ci></apply><csymbol cd="latexml" id="S2.E6Xb.2.1.1.m1.2.2.3.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.3">absent</csymbol><apply id="S2.E6Xb.2.1.1.m1.2.2.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1"><apply id="S2.E6Xb.2.1.1.m1.2.2.1.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2"><csymbol cd="ambiguous" id="S2.E6Xb.2.1.1.m1.2.2.1.2.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2">superscript</csymbol><apply id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2"><csymbol cd="ambiguous" id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2">subscript</csymbol><sum id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.2"></sum><apply id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3"><eq id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.1"></eq><ci id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.2">𝑛</ci><cn id="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.3.cmml" type="integer" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.2.3.3">1</cn></apply></apply><ci id="S2.E6Xb.2.1.1.m1.2.2.1.2.3.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.2.3">𝑁</ci></apply><apply id="S2.E6Xb.2.1.1.m1.2.2.1.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1"><times id="S2.E6Xb.2.1.1.m1.2.2.1.1.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.2"></times><ci id="S2.E6Xb.2.1.1.m1.2.2.1.1.3.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.3">max</ci><interval closure="open" id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1"><cn id="S2.E6Xb.2.1.1.m1.1.1.cmml" type="integer" xref="S2.E6Xb.2.1.1.m1.1.1">0</cn><apply id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.2">𝐗𝐖</ci><ci id="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.3.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.1.1.1.3">𝑛</ci></apply></interval><apply id="S2.E6Xb.2.1.1.m1.2.2.1.1.4.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.4"><csymbol cd="ambiguous" id="S2.E6Xb.2.1.1.m1.2.2.1.1.4.1.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.4">subscript</csymbol><ci id="S2.E6Xb.2.1.1.m1.2.2.1.1.4.2.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.4.2">𝐔</ci><ci id="S2.E6Xb.2.1.1.m1.2.2.1.1.4.3.cmml" xref="S2.E6Xb.2.1.1.m1.2.2.1.1.4.3">𝑛</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E6Xb.2.1.1.m1.2c">\displaystyle\mathrel{\stackrel{{\scriptstyle\makebox[0.0pt]{\mbox{\tiny(a)}}}% }{{=}}}\sum_{n=1}^{N}\mathrm{max}(0,\mathbf{X}\mathbf{W}_{n})\mathbf{U}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.E6Xb.2.1.1.m1.2d">start_RELOP SUPERSCRIPTOP start_ARG = end_ARG start_ARG (a) end_ARG end_RELOP ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT roman_max ( 0 , bold_XW start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) bold_U start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E6Xc"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle=\sum_{n=1}^{N}\mathbf{Z}_{n}," class="ltx_Math" display="inline" id="S2.E6Xc.2.1.1.m1.1"><semantics id="S2.E6Xc.2.1.1.m1.1a"><mrow id="S2.E6Xc.2.1.1.m1.1.1.1" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E6Xc.2.1.1.m1.1.1.1.1" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.cmml"><mi id="S2.E6Xc.2.1.1.m1.1.1.1.1.2" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.2.cmml"></mi><mo id="S2.E6Xc.2.1.1.m1.1.1.1.1.1" mathcolor="#000000" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.1.cmml">=</mo><mrow id="S2.E6Xc.2.1.1.m1.1.1.1.1.3" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.cmml"><mstyle displaystyle="true" id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.cmml"><munderover id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1a" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.cmml"><mo id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.2" mathcolor="#000000" movablelimits="false" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.2.cmml">∑</mo><mrow id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.cmml"><mi id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.2" mathcolor="#000000" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.2.cmml">n</mi><mo id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.1" mathcolor="#000000" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.1.cmml">=</mo><mn id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.3" mathcolor="#000000" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.3" mathcolor="#000000" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.3.cmml">N</mi></munderover></mstyle><msub id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.cmml"><mi id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.2" mathcolor="#000000" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.2.cmml">𝐙</mi><mi id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.3" mathcolor="#000000" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.3.cmml">n</mi></msub></mrow></mrow><mo id="S2.E6Xc.2.1.1.m1.1.1.1.2" mathcolor="#000000" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E6Xc.2.1.1.m1.1b"><apply id="S2.E6Xc.2.1.1.m1.1.1.1.1.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1"><eq id="S2.E6Xc.2.1.1.m1.1.1.1.1.1.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.1"></eq><csymbol cd="latexml" id="S2.E6Xc.2.1.1.m1.1.1.1.1.2.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.2">absent</csymbol><apply id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3"><apply id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.1.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1">superscript</csymbol><apply id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.1.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1">subscript</csymbol><sum id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.2.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.2"></sum><apply id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3"><eq id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.1.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.1"></eq><ci id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.2.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.2">𝑛</ci><cn id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.3.cmml" type="integer" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.2.3.3">1</cn></apply></apply><ci id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.3.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.1.3">𝑁</ci></apply><apply id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.1.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.2.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.2">𝐙</ci><ci id="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.3.cmml" xref="S2.E6Xc.2.1.1.m1.1.1.1.1.3.2.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E6Xc.2.1.1.m1.1c">\displaystyle=\sum_{n=1}^{N}\mathbf{Z}_{n},</annotation><annotation encoding="application/x-llamapun" id="S2.E6Xc.2.1.1.m1.1d">= ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> </tbody> </table> where (a) follows from the element-wise property of activation functions (e.g., ReLU, GeLU). Therefore, aggregating partial results <math alttext="\mathbf{Z}_{n}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.34.1.m1.1"><semantics id="S2.SS2.SSS1.p1.34.1.m1.1a"><msub id="S2.SS2.SSS1.p1.34.1.m1.1.1" xref="S2.SS2.SSS1.p1.34.1.m1.1.1.cmml"><mi id="S2.SS2.SSS1.p1.34.1.m1.1.1.2" mathcolor="#000000" xref="S2.SS2.SSS1.p1.34.1.m1.1.1.2.cmml">𝐙</mi><mi id="S2.SS2.SSS1.p1.34.1.m1.1.1.3" mathcolor="#000000" xref="S2.SS2.SSS1.p1.34.1.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.34.1.m1.1b"><apply id="S2.SS2.SSS1.p1.34.1.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.34.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.34.1.m1.1.1.1.cmml" xref="S2.SS2.SSS1.p1.34.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.SSS1.p1.34.1.m1.1.1.2.cmml" xref="S2.SS2.SSS1.p1.34.1.m1.1.1.2">𝐙</ci><ci id="S2.SS2.SSS1.p1.34.1.m1.1.1.3.cmml" xref="S2.SS2.SSS1.p1.34.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.34.1.m1.1c">\mathbf{Z}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.34.1.m1.1d">bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> reconstructs the original unpartitioned output <math alttext="\mathbf{Z}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.35.2.m2.1"><semantics id="S2.SS2.SSS1.p1.35.2.m2.1a"><mi id="S2.SS2.SSS1.p1.35.2.m2.1.1" mathcolor="#000000" xref="S2.SS2.SSS1.p1.35.2.m2.1.1.cmml">𝐙</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.35.2.m2.1b"><ci id="S2.SS2.SSS1.p1.35.2.m2.1.1.cmml" xref="S2.SS2.SSS1.p1.35.2.m2.1.1">𝐙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.35.2.m2.1c">\mathbf{Z}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.35.2.m2.1d">bold_Z</annotation></semantics></math> (i.e., Eq. (<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.E5" title="In II-B1 Tensor Parallelism for MLP Layer ‣ II-B Tensor Parallelism ‣ II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">5</a>) holds). After aggregation, the final output <math alttext="\mathbf{Z}" class="ltx_Math" display="inline" id="S2.SS2.SSS1.p1.36.m1.1"><semantics id="S2.SS2.SSS1.p1.36.m1.1a"><mi id="S2.SS2.SSS1.p1.36.m1.1.1" xref="S2.SS2.SSS1.p1.36.m1.1.1.cmml">𝐙</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.36.m1.1b"><ci id="S2.SS2.SSS1.p1.36.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.36.m1.1.1">𝐙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.36.m1.1c">\mathbf{Z}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS1.p1.36.m1.1d">bold_Z</annotation></semantics></math> of the MLP layer is broadcasted to all devices, ensuring synchronization and consistency across devices for the subsequent layer’s computation. </div> </section> <section class="ltx_subsubsection" id="S2.SS2.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> II-B2 Tensor Parallelism for Self-Attention Layer</h4> <div class="ltx_para" id="S2.SS2.SSS2.p1"> For the self-attention layer, tensor parallelism similarly partitions its query (<math alttext="\mathbf{Q}" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.1.m1.1"><semantics id="S2.SS2.SSS2.p1.1.m1.1a"><mi id="S2.SS2.SSS2.p1.1.m1.1.1" xref="S2.SS2.SSS2.p1.1.m1.1.1.cmml">𝐐</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.1.m1.1b"><ci id="S2.SS2.SSS2.p1.1.m1.1.1.cmml" xref="S2.SS2.SSS2.p1.1.m1.1.1">𝐐</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.1.m1.1c">\mathbf{Q}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.1.m1.1d">bold_Q</annotation></semantics></math>), key (<math alttext="\mathbf{K}" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.2.m2.1"><semantics id="S2.SS2.SSS2.p1.2.m2.1a"><mi id="S2.SS2.SSS2.p1.2.m2.1.1" xref="S2.SS2.SSS2.p1.2.m2.1.1.cmml">𝐊</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.2.m2.1b"><ci id="S2.SS2.SSS2.p1.2.m2.1.1.cmml" xref="S2.SS2.SSS2.p1.2.m2.1.1">𝐊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.2.m2.1c">\mathbf{K}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.2.m2.1d">bold_K</annotation></semantics></math>), value (<math alttext="\mathbf{V}" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.3.m3.1"><semantics id="S2.SS2.SSS2.p1.3.m3.1a"><mi id="S2.SS2.SSS2.p1.3.m3.1.1" xref="S2.SS2.SSS2.p1.3.m3.1.1.cmml">𝐕</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.3.m3.1b"><ci id="S2.SS2.SSS2.p1.3.m3.1.1.cmml" xref="S2.SS2.SSS2.p1.3.m3.1.1">𝐕</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.3.m3.1c">\mathbf{V}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.3.m3.1d">bold_V</annotation></semantics></math>), and transformation (<math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.4.m4.1"><semantics id="S2.SS2.SSS2.p1.4.m4.1a"><mi id="S2.SS2.SSS2.p1.4.m4.1.1" xref="S2.SS2.SSS2.p1.4.m4.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.4.m4.1b"><ci id="S2.SS2.SSS2.p1.4.m4.1.1.cmml" xref="S2.SS2.SSS2.p1.4.m4.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.4.m4.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.4.m4.1d">bold_U</annotation></semantics></math>) matrices across edge devices. In the traditional centralized computation of the self-attention layer, the output <math alttext="\mathbf{Z}" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.5.m5.1"><semantics id="S2.SS2.SSS2.p1.5.m5.1a"><mi id="S2.SS2.SSS2.p1.5.m5.1.1" xref="S2.SS2.SSS2.p1.5.m5.1.1.cmml">𝐙</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.5.m5.1b"><ci id="S2.SS2.SSS2.p1.5.m5.1.1.cmml" xref="S2.SS2.SSS2.p1.5.m5.1.1">𝐙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.5.m5.1c">\mathbf{Z}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.5.m5.1d">bold_Z</annotation></semantics></math> can be derived as follows, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E7"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E7X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{Z}=\mathrm{softmax}\left(\frac{\mathbf{X}\mathbf{Q}(% \mathbf{X}\mathbf{K})^{\mathsf{T}}}{\sqrt{d_{k}}}\right)\mathbf{V}\mathbf{U}," class="ltx_Math" display="inline" id="S2.E7X.2.1.1.m1.2"><semantics id="S2.E7X.2.1.1.m1.2a"><mrow id="S2.E7X.2.1.1.m1.2.2.1" xref="S2.E7X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S2.E7X.2.1.1.m1.2.2.1.1" xref="S2.E7X.2.1.1.m1.2.2.1.1.cmml"><mi id="S2.E7X.2.1.1.m1.2.2.1.1.2" xref="S2.E7X.2.1.1.m1.2.2.1.1.2.cmml">𝐙</mi><mo id="S2.E7X.2.1.1.m1.2.2.1.1.1" xref="S2.E7X.2.1.1.m1.2.2.1.1.1.cmml">=</mo><mrow id="S2.E7X.2.1.1.m1.2.2.1.1.3" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.cmml"><mi id="S2.E7X.2.1.1.m1.2.2.1.1.3.2" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.2.cmml">softmax</mi><mo id="S2.E7X.2.1.1.m1.2.2.1.1.3.1" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.1.cmml">⁢</mo><mrow id="S2.E7X.2.1.1.m1.2.2.1.1.3.3.2" xref="S2.E7X.2.1.1.m1.1.1.cmml"><mo id="S2.E7X.2.1.1.m1.2.2.1.1.3.3.2.1" xref="S2.E7X.2.1.1.m1.1.1.cmml">(</mo><mstyle displaystyle="true" id="S2.E7X.2.1.1.m1.1.1" xref="S2.E7X.2.1.1.m1.1.1.cmml"><mfrac id="S2.E7X.2.1.1.m1.1.1a" xref="S2.E7X.2.1.1.m1.1.1.cmml"><mrow id="S2.E7X.2.1.1.m1.1.1.1" xref="S2.E7X.2.1.1.m1.1.1.1.cmml"><mi id="S2.E7X.2.1.1.m1.1.1.1.3" xref="S2.E7X.2.1.1.m1.1.1.1.3.cmml">𝐗𝐐</mi><mo id="S2.E7X.2.1.1.m1.1.1.1.2" xref="S2.E7X.2.1.1.m1.1.1.1.2.cmml">⁢</mo><msup id="S2.E7X.2.1.1.m1.1.1.1.4" xref="S2.E7X.2.1.1.m1.1.1.1.4.cmml"><mrow id="S2.E7X.2.1.1.m1.1.1.1.4.2.2" xref="S2.E7X.2.1.1.m1.1.1.1.4.cmml"><mo id="S2.E7X.2.1.1.m1.1.1.1.4.2.2.1" stretchy="false" xref="S2.E7X.2.1.1.m1.1.1.1.4.cmml">(</mo><mi id="S2.E7X.2.1.1.m1.1.1.1.1" xref="S2.E7X.2.1.1.m1.1.1.1.1.cmml">𝐗𝐊</mi><mo id="S2.E7X.2.1.1.m1.1.1.1.4.2.2.2" stretchy="false" xref="S2.E7X.2.1.1.m1.1.1.1.4.cmml">)</mo></mrow><mi id="S2.E7X.2.1.1.m1.1.1.1.4.3" xref="S2.E7X.2.1.1.m1.1.1.1.4.3.cmml">𝖳</mi></msup></mrow><msqrt id="S2.E7X.2.1.1.m1.1.1.3" xref="S2.E7X.2.1.1.m1.1.1.3.cmml"><msub id="S2.E7X.2.1.1.m1.1.1.3.2" xref="S2.E7X.2.1.1.m1.1.1.3.2.cmml"><mi id="S2.E7X.2.1.1.m1.1.1.3.2.2" xref="S2.E7X.2.1.1.m1.1.1.3.2.2.cmml">d</mi><mi id="S2.E7X.2.1.1.m1.1.1.3.2.3" xref="S2.E7X.2.1.1.m1.1.1.3.2.3.cmml">k</mi></msub></msqrt></mfrac></mstyle><mo id="S2.E7X.2.1.1.m1.2.2.1.1.3.3.2.2" xref="S2.E7X.2.1.1.m1.1.1.cmml">)</mo></mrow><mo id="S2.E7X.2.1.1.m1.2.2.1.1.3.1a" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.1.cmml">⁢</mo><mi id="S2.E7X.2.1.1.m1.2.2.1.1.3.4" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.4.cmml">𝐕𝐔</mi></mrow></mrow><mo id="S2.E7X.2.1.1.m1.2.2.1.2" xref="S2.E7X.2.1.1.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E7X.2.1.1.m1.2b"><apply id="S2.E7X.2.1.1.m1.2.2.1.1.cmml" xref="S2.E7X.2.1.1.m1.2.2.1"><eq id="S2.E7X.2.1.1.m1.2.2.1.1.1.cmml" xref="S2.E7X.2.1.1.m1.2.2.1.1.1"></eq><ci id="S2.E7X.2.1.1.m1.2.2.1.1.2.cmml" xref="S2.E7X.2.1.1.m1.2.2.1.1.2">𝐙</ci><apply id="S2.E7X.2.1.1.m1.2.2.1.1.3.cmml" xref="S2.E7X.2.1.1.m1.2.2.1.1.3"><times id="S2.E7X.2.1.1.m1.2.2.1.1.3.1.cmml" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.1"></times><ci id="S2.E7X.2.1.1.m1.2.2.1.1.3.2.cmml" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.2">softmax</ci><apply id="S2.E7X.2.1.1.m1.1.1.cmml" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.3.2"><divide id="S2.E7X.2.1.1.m1.1.1.2.cmml" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.3.2"></divide><apply id="S2.E7X.2.1.1.m1.1.1.1.cmml" xref="S2.E7X.2.1.1.m1.1.1.1"><times id="S2.E7X.2.1.1.m1.1.1.1.2.cmml" xref="S2.E7X.2.1.1.m1.1.1.1.2"></times><ci id="S2.E7X.2.1.1.m1.1.1.1.3.cmml" xref="S2.E7X.2.1.1.m1.1.1.1.3">𝐗𝐐</ci><apply id="S2.E7X.2.1.1.m1.1.1.1.4.cmml" xref="S2.E7X.2.1.1.m1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E7X.2.1.1.m1.1.1.1.4.1.cmml" xref="S2.E7X.2.1.1.m1.1.1.1.4">superscript</csymbol><ci id="S2.E7X.2.1.1.m1.1.1.1.1.cmml" xref="S2.E7X.2.1.1.m1.1.1.1.1">𝐗𝐊</ci><ci id="S2.E7X.2.1.1.m1.1.1.1.4.3.cmml" xref="S2.E7X.2.1.1.m1.1.1.1.4.3">𝖳</ci></apply></apply><apply id="S2.E7X.2.1.1.m1.1.1.3.cmml" xref="S2.E7X.2.1.1.m1.1.1.3"><root id="S2.E7X.2.1.1.m1.1.1.3a.cmml" xref="S2.E7X.2.1.1.m1.1.1.3"></root><apply id="S2.E7X.2.1.1.m1.1.1.3.2.cmml" xref="S2.E7X.2.1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E7X.2.1.1.m1.1.1.3.2.1.cmml" xref="S2.E7X.2.1.1.m1.1.1.3.2">subscript</csymbol><ci id="S2.E7X.2.1.1.m1.1.1.3.2.2.cmml" xref="S2.E7X.2.1.1.m1.1.1.3.2.2">𝑑</ci><ci id="S2.E7X.2.1.1.m1.1.1.3.2.3.cmml" xref="S2.E7X.2.1.1.m1.1.1.3.2.3">𝑘</ci></apply></apply></apply><ci id="S2.E7X.2.1.1.m1.2.2.1.1.3.4.cmml" xref="S2.E7X.2.1.1.m1.2.2.1.1.3.4">𝐕𝐔</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E7X.2.1.1.m1.2c">\displaystyle\mathbf{Z}=\mathrm{softmax}\left(\frac{\mathbf{X}\mathbf{Q}(% \mathbf{X}\mathbf{K})^{\mathsf{T}}}{\sqrt{d_{k}}}\right)\mathbf{V}\mathbf{U},</annotation><annotation encoding="application/x-llamapun" id="S2.E7X.2.1.1.m1.2d">bold_Z = roman_softmax ( divide start_ARG bold_XQ ( bold_XK ) start_POSTSUPERSCRIPT sansserif_T end_POSTSUPERSCRIPT end_ARG start_ARG square-root start_ARG italic_d start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_ARG end_ARG ) bold_VU ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(7)</td> </tr> </tbody> </table> where <math alttext="\mathbf{X}" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.6.m1.1"><semantics id="S2.SS2.SSS2.p1.6.m1.1a"><mi id="S2.SS2.SSS2.p1.6.m1.1.1" xref="S2.SS2.SSS2.p1.6.m1.1.1.cmml">𝐗</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.6.m1.1b"><ci id="S2.SS2.SSS2.p1.6.m1.1.1.cmml" xref="S2.SS2.SSS2.p1.6.m1.1.1">𝐗</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.6.m1.1c">\mathbf{X}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.6.m1.1d">bold_X</annotation></semantics></math> denotes the input, and <math alttext="d_{k}" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.7.m2.1"><semantics id="S2.SS2.SSS2.p1.7.m2.1a"><msub id="S2.SS2.SSS2.p1.7.m2.1.1" xref="S2.SS2.SSS2.p1.7.m2.1.1.cmml"><mi id="S2.SS2.SSS2.p1.7.m2.1.1.2" xref="S2.SS2.SSS2.p1.7.m2.1.1.2.cmml">d</mi><mi id="S2.SS2.SSS2.p1.7.m2.1.1.3" xref="S2.SS2.SSS2.p1.7.m2.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.7.m2.1b"><apply id="S2.SS2.SSS2.p1.7.m2.1.1.cmml" xref="S2.SS2.SSS2.p1.7.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS2.p1.7.m2.1.1.1.cmml" xref="S2.SS2.SSS2.p1.7.m2.1.1">subscript</csymbol><ci id="S2.SS2.SSS2.p1.7.m2.1.1.2.cmml" xref="S2.SS2.SSS2.p1.7.m2.1.1.2">𝑑</ci><ci id="S2.SS2.SSS2.p1.7.m2.1.1.3.cmml" xref="S2.SS2.SSS2.p1.7.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.7.m2.1c">d_{k}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.7.m2.1d">italic_d start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> denotes the dimension of the key vectors. In tensor parallelism, the memory-intensive weight matrices are splited and distributed across <math alttext="N" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.8.m3.1"><semantics id="S2.SS2.SSS2.p1.8.m3.1a"><mi id="S2.SS2.SSS2.p1.8.m3.1.1" xref="S2.SS2.SSS2.p1.8.m3.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.8.m3.1b"><ci id="S2.SS2.SSS2.p1.8.m3.1.1.cmml" xref="S2.SS2.SSS2.p1.8.m3.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.8.m3.1c">N</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.8.m3.1d">italic_N</annotation></semantics></math> edge devices as follows, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E8"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E8X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{Q}" class="ltx_Math" display="inline" id="S2.E8X.2.1.1.m1.1"><semantics id="S2.E8X.2.1.1.m1.1a"><mi id="S2.E8X.2.1.1.m1.1.1" xref="S2.E8X.2.1.1.m1.1.1.cmml">𝐐</mi><annotation-xml encoding="MathML-Content" id="S2.E8X.2.1.1.m1.1b"><ci id="S2.E8X.2.1.1.m1.1.1.cmml" xref="S2.E8X.2.1.1.m1.1.1">𝐐</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E8X.2.1.1.m1.1c">\displaystyle\mathbf{Q}</annotation><annotation encoding="application/x-llamapun" id="S2.E8X.2.1.1.m1.1d">bold_Q</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle=[\mathbf{Q}_{1},\ldots,\mathbf{Q}_{N}]," class="ltx_Math" display="inline" id="S2.E8X.3.2.2.m1.2"><semantics id="S2.E8X.3.2.2.m1.2a"><mrow id="S2.E8X.3.2.2.m1.2.2.1" xref="S2.E8X.3.2.2.m1.2.2.1.1.cmml"><mrow id="S2.E8X.3.2.2.m1.2.2.1.1" xref="S2.E8X.3.2.2.m1.2.2.1.1.cmml"><mi id="S2.E8X.3.2.2.m1.2.2.1.1.4" xref="S2.E8X.3.2.2.m1.2.2.1.1.4.cmml"></mi><mo id="S2.E8X.3.2.2.m1.2.2.1.1.3" xref="S2.E8X.3.2.2.m1.2.2.1.1.3.cmml">=</mo><mrow id="S2.E8X.3.2.2.m1.2.2.1.1.2.2" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.3.cmml"><mo id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.3" stretchy="false" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.3.cmml">[</mo><msub id="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1" xref="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.cmml"><mi id="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.2" xref="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.2.cmml">𝐐</mi><mn id="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.3" xref="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.4" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.3.cmml">,</mo><mi id="S2.E8X.3.2.2.m1.1.1" mathvariant="normal" xref="S2.E8X.3.2.2.m1.1.1.cmml">…</mi><mo id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.5" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.3.cmml">,</mo><msub id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.cmml"><mi id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.2" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.2.cmml">𝐐</mi><mi id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.3" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.3.cmml">N</mi></msub><mo id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.6" stretchy="false" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.3.cmml">]</mo></mrow></mrow><mo id="S2.E8X.3.2.2.m1.2.2.1.2" xref="S2.E8X.3.2.2.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E8X.3.2.2.m1.2b"><apply id="S2.E8X.3.2.2.m1.2.2.1.1.cmml" xref="S2.E8X.3.2.2.m1.2.2.1"><eq id="S2.E8X.3.2.2.m1.2.2.1.1.3.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.3"></eq><csymbol cd="latexml" id="S2.E8X.3.2.2.m1.2.2.1.1.4.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.4">absent</csymbol><list id="S2.E8X.3.2.2.m1.2.2.1.1.2.3.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.2"><apply id="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.2">𝐐</ci><cn id="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.3.cmml" type="integer" xref="S2.E8X.3.2.2.m1.2.2.1.1.1.1.1.3">1</cn></apply><ci id="S2.E8X.3.2.2.m1.1.1.cmml" xref="S2.E8X.3.2.2.m1.1.1">…</ci><apply id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.1.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2">subscript</csymbol><ci id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.2.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.2">𝐐</ci><ci id="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.3.cmml" xref="S2.E8X.3.2.2.m1.2.2.1.1.2.2.2.3">𝑁</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E8X.3.2.2.m1.2c">\displaystyle=[\mathbf{Q}_{1},\ldots,\mathbf{Q}_{N}],</annotation><annotation encoding="application/x-llamapun" id="S2.E8X.3.2.2.m1.2d">= [ bold_Q start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , bold_Q start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ] ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="4">(8)</td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E8Xa"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{K}" class="ltx_Math" display="inline" id="S2.E8Xa.2.1.1.m1.1"><semantics id="S2.E8Xa.2.1.1.m1.1a"><mi id="S2.E8Xa.2.1.1.m1.1.1" xref="S2.E8Xa.2.1.1.m1.1.1.cmml">𝐊</mi><annotation-xml encoding="MathML-Content" id="S2.E8Xa.2.1.1.m1.1b"><ci id="S2.E8Xa.2.1.1.m1.1.1.cmml" xref="S2.E8Xa.2.1.1.m1.1.1">𝐊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E8Xa.2.1.1.m1.1c">\displaystyle\mathbf{K}</annotation><annotation encoding="application/x-llamapun" id="S2.E8Xa.2.1.1.m1.1d">bold_K</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle=[\mathbf{K}_{1},\ldots,\mathbf{K}_{N}]," class="ltx_Math" display="inline" id="S2.E8Xa.3.2.2.m1.2"><semantics id="S2.E8Xa.3.2.2.m1.2a"><mrow id="S2.E8Xa.3.2.2.m1.2.2.1" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.cmml"><mrow id="S2.E8Xa.3.2.2.m1.2.2.1.1" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.cmml"><mi id="S2.E8Xa.3.2.2.m1.2.2.1.1.4" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.4.cmml"></mi><mo id="S2.E8Xa.3.2.2.m1.2.2.1.1.3" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.3.cmml">=</mo><mrow id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.3.cmml"><mo id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.3" stretchy="false" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.3.cmml">[</mo><msub id="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.cmml"><mi id="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.2" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.2.cmml">𝐊</mi><mn id="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.3" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.4" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.3.cmml">,</mo><mi id="S2.E8Xa.3.2.2.m1.1.1" mathvariant="normal" xref="S2.E8Xa.3.2.2.m1.1.1.cmml">…</mi><mo id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.5" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.3.cmml">,</mo><msub id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.cmml"><mi id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.2" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.2.cmml">𝐊</mi><mi id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.3" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.3.cmml">N</mi></msub><mo id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.6" stretchy="false" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.3.cmml">]</mo></mrow></mrow><mo id="S2.E8Xa.3.2.2.m1.2.2.1.2" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E8Xa.3.2.2.m1.2b"><apply id="S2.E8Xa.3.2.2.m1.2.2.1.1.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1"><eq id="S2.E8Xa.3.2.2.m1.2.2.1.1.3.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.3"></eq><csymbol cd="latexml" id="S2.E8Xa.3.2.2.m1.2.2.1.1.4.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.4">absent</csymbol><list id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.3.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2"><apply id="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.2">𝐊</ci><cn id="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.3.cmml" type="integer" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.1.1.1.3">1</cn></apply><ci id="S2.E8Xa.3.2.2.m1.1.1.cmml" xref="S2.E8Xa.3.2.2.m1.1.1">…</ci><apply id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.1.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2">subscript</csymbol><ci id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.2.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.2">𝐊</ci><ci id="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.3.cmml" xref="S2.E8Xa.3.2.2.m1.2.2.1.1.2.2.2.3">𝑁</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E8Xa.3.2.2.m1.2c">\displaystyle=[\mathbf{K}_{1},\ldots,\mathbf{K}_{N}],</annotation><annotation encoding="application/x-llamapun" id="S2.E8Xa.3.2.2.m1.2d">= [ bold_K start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , bold_K start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ] ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E8Xb"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{V}" class="ltx_Math" display="inline" id="S2.E8Xb.2.1.1.m1.1"><semantics id="S2.E8Xb.2.1.1.m1.1a"><mi id="S2.E8Xb.2.1.1.m1.1.1" xref="S2.E8Xb.2.1.1.m1.1.1.cmml">𝐕</mi><annotation-xml encoding="MathML-Content" id="S2.E8Xb.2.1.1.m1.1b"><ci id="S2.E8Xb.2.1.1.m1.1.1.cmml" xref="S2.E8Xb.2.1.1.m1.1.1">𝐕</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E8Xb.2.1.1.m1.1c">\displaystyle\mathbf{V}</annotation><annotation encoding="application/x-llamapun" id="S2.E8Xb.2.1.1.m1.1d">bold_V</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle=[\mathbf{V}_{1},\ldots,\mathbf{V}_{N}]," class="ltx_Math" display="inline" id="S2.E8Xb.3.2.2.m1.2"><semantics id="S2.E8Xb.3.2.2.m1.2a"><mrow id="S2.E8Xb.3.2.2.m1.2.2.1" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.cmml"><mrow id="S2.E8Xb.3.2.2.m1.2.2.1.1" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.cmml"><mi id="S2.E8Xb.3.2.2.m1.2.2.1.1.4" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.4.cmml"></mi><mo id="S2.E8Xb.3.2.2.m1.2.2.1.1.3" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.3.cmml">=</mo><mrow id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.3.cmml"><mo id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.3" stretchy="false" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.3.cmml">[</mo><msub id="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.cmml"><mi id="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.2" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.2.cmml">𝐕</mi><mn id="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.3" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.4" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.3.cmml">,</mo><mi id="S2.E8Xb.3.2.2.m1.1.1" mathvariant="normal" xref="S2.E8Xb.3.2.2.m1.1.1.cmml">…</mi><mo id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.5" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.3.cmml">,</mo><msub id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.cmml"><mi id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.2" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.2.cmml">𝐕</mi><mi id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.3" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.3.cmml">N</mi></msub><mo id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.6" stretchy="false" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.3.cmml">]</mo></mrow></mrow><mo id="S2.E8Xb.3.2.2.m1.2.2.1.2" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E8Xb.3.2.2.m1.2b"><apply id="S2.E8Xb.3.2.2.m1.2.2.1.1.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1"><eq id="S2.E8Xb.3.2.2.m1.2.2.1.1.3.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.3"></eq><csymbol cd="latexml" id="S2.E8Xb.3.2.2.m1.2.2.1.1.4.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.4">absent</csymbol><list id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.3.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2"><apply id="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.2">𝐕</ci><cn id="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.3.cmml" type="integer" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.1.1.1.3">1</cn></apply><ci id="S2.E8Xb.3.2.2.m1.1.1.cmml" xref="S2.E8Xb.3.2.2.m1.1.1">…</ci><apply id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.1.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2">subscript</csymbol><ci id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.2.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.2">𝐕</ci><ci id="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.3.cmml" xref="S2.E8Xb.3.2.2.m1.2.2.1.1.2.2.2.3">𝑁</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E8Xb.3.2.2.m1.2c">\displaystyle=[\mathbf{V}_{1},\ldots,\mathbf{V}_{N}],</annotation><annotation encoding="application/x-llamapun" id="S2.E8Xb.3.2.2.m1.2d">= [ bold_V start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , bold_V start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ] ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E8Xc"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{U}" class="ltx_Math" display="inline" id="S2.E8Xc.2.1.1.m1.1"><semantics id="S2.E8Xc.2.1.1.m1.1a"><mi id="S2.E8Xc.2.1.1.m1.1.1" xref="S2.E8Xc.2.1.1.m1.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.E8Xc.2.1.1.m1.1b"><ci id="S2.E8Xc.2.1.1.m1.1.1.cmml" xref="S2.E8Xc.2.1.1.m1.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E8Xc.2.1.1.m1.1c">\displaystyle\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S2.E8Xc.2.1.1.m1.1d">bold_U</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle=[\mathbf{U}_{1}^{\mathsf{T}},\ldots,\mathbf{U}_{N}^{\mathsf{T}}]% ^{\mathsf{T}}." class="ltx_Math" display="inline" id="S2.E8Xc.3.2.2.m1.2"><semantics id="S2.E8Xc.3.2.2.m1.2a"><mrow id="S2.E8Xc.3.2.2.m1.2.2.1" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.cmml"><mrow id="S2.E8Xc.3.2.2.m1.2.2.1.1" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.cmml"><mi id="S2.E8Xc.3.2.2.m1.2.2.1.1.4" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.4.cmml"></mi><mo id="S2.E8Xc.3.2.2.m1.2.2.1.1.3" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.3.cmml">=</mo><msup id="S2.E8Xc.3.2.2.m1.2.2.1.1.2" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.cmml"><mrow id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.3.cmml"><mo id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.3" stretchy="false" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.3.cmml">[</mo><msubsup id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.cmml"><mi id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.2" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.2.cmml">𝐔</mi><mn id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.3" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.3.cmml">1</mn><mi id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.3" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.3.cmml">𝖳</mi></msubsup><mo id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.4" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.3.cmml">,</mo><mi id="S2.E8Xc.3.2.2.m1.1.1" mathvariant="normal" xref="S2.E8Xc.3.2.2.m1.1.1.cmml">…</mi><mo id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.5" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.3.cmml">,</mo><msubsup id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.cmml"><mi id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.2" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.2.cmml">𝐔</mi><mi id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.3" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.3.cmml">N</mi><mi id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.3" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.3.cmml">𝖳</mi></msubsup><mo id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.6" stretchy="false" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.3.cmml">]</mo></mrow><mi id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.4" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.4.cmml">𝖳</mi></msup></mrow><mo id="S2.E8Xc.3.2.2.m1.2.2.1.2" lspace="0em" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E8Xc.3.2.2.m1.2b"><apply id="S2.E8Xc.3.2.2.m1.2.2.1.1.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1"><eq id="S2.E8Xc.3.2.2.m1.2.2.1.1.3.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.3"></eq><csymbol cd="latexml" id="S2.E8Xc.3.2.2.m1.2.2.1.1.4.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.4">absent</csymbol><apply id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2"><csymbol cd="ambiguous" id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.3.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2">superscript</csymbol><list id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.3.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2"><apply id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1">superscript</csymbol><apply id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.1.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.2.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.2">𝐔</ci><cn id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.3.cmml" type="integer" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.2.3">1</cn></apply><ci id="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.1.1.1.1.3">𝖳</ci></apply><ci id="S2.E8Xc.3.2.2.m1.1.1.cmml" xref="S2.E8Xc.3.2.2.m1.1.1">…</ci><apply id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.1.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2">superscript</csymbol><apply id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.1.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2">subscript</csymbol><ci id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.2.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.2">𝐔</ci><ci id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.3.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.2.3">𝑁</ci></apply><ci id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.3.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.2.2.2.3">𝖳</ci></apply></list><ci id="S2.E8Xc.3.2.2.m1.2.2.1.1.2.4.cmml" xref="S2.E8Xc.3.2.2.m1.2.2.1.1.2.4">𝖳</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E8Xc.3.2.2.m1.2c">\displaystyle=[\mathbf{U}_{1}^{\mathsf{T}},\ldots,\mathbf{U}_{N}^{\mathsf{T}}]% ^{\mathsf{T}}.</annotation><annotation encoding="application/x-llamapun" id="S2.E8Xc.3.2.2.m1.2d">= [ bold_U start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT sansserif_T end_POSTSUPERSCRIPT , … , bold_U start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT start_POSTSUPERSCRIPT sansserif_T end_POSTSUPERSCRIPT ] start_POSTSUPERSCRIPT sansserif_T end_POSTSUPERSCRIPT .</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> </tbody> </table> Then, each device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.9.m1.1"><semantics id="S2.SS2.SSS2.p1.9.m1.1a"><mi id="S2.SS2.SSS2.p1.9.m1.1.1" xref="S2.SS2.SSS2.p1.9.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.9.m1.1b"><ci id="S2.SS2.SSS2.p1.9.m1.1.1.cmml" xref="S2.SS2.SSS2.p1.9.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.9.m1.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.9.m1.1d">italic_n</annotation></semantics></math> performs local computation on its corresponding portion of the query, key, value, and transformation matrices as follows, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E9"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E9X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathbf{Z}_{n}=\mathrm{softmax}\left(\frac{\mathbf{X}\mathbf{Q}_{% n}(\mathbf{X}\mathbf{K}_{n})^{\mathsf{T}}}{\sqrt{d_{k}}}\right)\mathbf{V}_{n}% \mathbf{U}_{n}." class="ltx_Math" display="inline" id="S2.E9X.2.1.1.m1.2"><semantics id="S2.E9X.2.1.1.m1.2a"><mrow id="S2.E9X.2.1.1.m1.2.2.1" xref="S2.E9X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S2.E9X.2.1.1.m1.2.2.1.1" xref="S2.E9X.2.1.1.m1.2.2.1.1.cmml"><msub id="S2.E9X.2.1.1.m1.2.2.1.1.2" xref="S2.E9X.2.1.1.m1.2.2.1.1.2.cmml"><mi id="S2.E9X.2.1.1.m1.2.2.1.1.2.2" xref="S2.E9X.2.1.1.m1.2.2.1.1.2.2.cmml">𝐙</mi><mi id="S2.E9X.2.1.1.m1.2.2.1.1.2.3" xref="S2.E9X.2.1.1.m1.2.2.1.1.2.3.cmml">n</mi></msub><mo id="S2.E9X.2.1.1.m1.2.2.1.1.1" xref="S2.E9X.2.1.1.m1.2.2.1.1.1.cmml">=</mo><mrow id="S2.E9X.2.1.1.m1.2.2.1.1.3" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.cmml"><mi id="S2.E9X.2.1.1.m1.2.2.1.1.3.2" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.2.cmml">softmax</mi><mo id="S2.E9X.2.1.1.m1.2.2.1.1.3.1" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.1.cmml">⁢</mo><mrow id="S2.E9X.2.1.1.m1.2.2.1.1.3.3.2" xref="S2.E9X.2.1.1.m1.1.1.cmml"><mo id="S2.E9X.2.1.1.m1.2.2.1.1.3.3.2.1" xref="S2.E9X.2.1.1.m1.1.1.cmml">(</mo><mstyle displaystyle="true" id="S2.E9X.2.1.1.m1.1.1" xref="S2.E9X.2.1.1.m1.1.1.cmml"><mfrac id="S2.E9X.2.1.1.m1.1.1a" xref="S2.E9X.2.1.1.m1.1.1.cmml"><mrow id="S2.E9X.2.1.1.m1.1.1.1" xref="S2.E9X.2.1.1.m1.1.1.1.cmml"><msub id="S2.E9X.2.1.1.m1.1.1.1.3" xref="S2.E9X.2.1.1.m1.1.1.1.3.cmml"><mi id="S2.E9X.2.1.1.m1.1.1.1.3.2" xref="S2.E9X.2.1.1.m1.1.1.1.3.2.cmml">𝐗𝐐</mi><mi id="S2.E9X.2.1.1.m1.1.1.1.3.3" xref="S2.E9X.2.1.1.m1.1.1.1.3.3.cmml">n</mi></msub><mo id="S2.E9X.2.1.1.m1.1.1.1.2" xref="S2.E9X.2.1.1.m1.1.1.1.2.cmml">⁢</mo><msup id="S2.E9X.2.1.1.m1.1.1.1.1" xref="S2.E9X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E9X.2.1.1.m1.1.1.1.1.1.1" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.cmml"><mo id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.2" stretchy="false" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.cmml"><mi id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.2" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.2.cmml">𝐗𝐊</mi><mi id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.3" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.3" stretchy="false" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S2.E9X.2.1.1.m1.1.1.1.1.3" xref="S2.E9X.2.1.1.m1.1.1.1.1.3.cmml">𝖳</mi></msup></mrow><msqrt id="S2.E9X.2.1.1.m1.1.1.3" xref="S2.E9X.2.1.1.m1.1.1.3.cmml"><msub id="S2.E9X.2.1.1.m1.1.1.3.2" xref="S2.E9X.2.1.1.m1.1.1.3.2.cmml"><mi id="S2.E9X.2.1.1.m1.1.1.3.2.2" xref="S2.E9X.2.1.1.m1.1.1.3.2.2.cmml">d</mi><mi id="S2.E9X.2.1.1.m1.1.1.3.2.3" xref="S2.E9X.2.1.1.m1.1.1.3.2.3.cmml">k</mi></msub></msqrt></mfrac></mstyle><mo id="S2.E9X.2.1.1.m1.2.2.1.1.3.3.2.2" xref="S2.E9X.2.1.1.m1.1.1.cmml">)</mo></mrow><mo id="S2.E9X.2.1.1.m1.2.2.1.1.3.1a" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.1.cmml">⁢</mo><msub id="S2.E9X.2.1.1.m1.2.2.1.1.3.4" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.4.cmml"><mi id="S2.E9X.2.1.1.m1.2.2.1.1.3.4.2" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.4.2.cmml">𝐕</mi><mi id="S2.E9X.2.1.1.m1.2.2.1.1.3.4.3" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.4.3.cmml">n</mi></msub><mo id="S2.E9X.2.1.1.m1.2.2.1.1.3.1b" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.1.cmml">⁢</mo><msub id="S2.E9X.2.1.1.m1.2.2.1.1.3.5" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.5.cmml"><mi id="S2.E9X.2.1.1.m1.2.2.1.1.3.5.2" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.5.2.cmml">𝐔</mi><mi id="S2.E9X.2.1.1.m1.2.2.1.1.3.5.3" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.5.3.cmml">n</mi></msub></mrow></mrow><mo id="S2.E9X.2.1.1.m1.2.2.1.2" lspace="0em" xref="S2.E9X.2.1.1.m1.2.2.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E9X.2.1.1.m1.2b"><apply id="S2.E9X.2.1.1.m1.2.2.1.1.cmml" xref="S2.E9X.2.1.1.m1.2.2.1"><eq id="S2.E9X.2.1.1.m1.2.2.1.1.1.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.1"></eq><apply id="S2.E9X.2.1.1.m1.2.2.1.1.2.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.2"><csymbol cd="ambiguous" id="S2.E9X.2.1.1.m1.2.2.1.1.2.1.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.2">subscript</csymbol><ci id="S2.E9X.2.1.1.m1.2.2.1.1.2.2.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.2.2">𝐙</ci><ci id="S2.E9X.2.1.1.m1.2.2.1.1.2.3.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.2.3">𝑛</ci></apply><apply id="S2.E9X.2.1.1.m1.2.2.1.1.3.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3"><times id="S2.E9X.2.1.1.m1.2.2.1.1.3.1.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.1"></times><ci id="S2.E9X.2.1.1.m1.2.2.1.1.3.2.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.2">softmax</ci><apply id="S2.E9X.2.1.1.m1.1.1.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.3.2"><divide id="S2.E9X.2.1.1.m1.1.1.2.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.3.2"></divide><apply id="S2.E9X.2.1.1.m1.1.1.1.cmml" xref="S2.E9X.2.1.1.m1.1.1.1"><times id="S2.E9X.2.1.1.m1.1.1.1.2.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.2"></times><apply id="S2.E9X.2.1.1.m1.1.1.1.3.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E9X.2.1.1.m1.1.1.1.3.1.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.3">subscript</csymbol><ci id="S2.E9X.2.1.1.m1.1.1.1.3.2.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.3.2">𝐗𝐐</ci><ci id="S2.E9X.2.1.1.m1.1.1.1.3.3.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.3.3">𝑛</ci></apply><apply id="S2.E9X.2.1.1.m1.1.1.1.1.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E9X.2.1.1.m1.1.1.1.1.2.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.1">superscript</csymbol><apply id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.2">𝐗𝐊</ci><ci id="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.1.1.1.1.3">𝑛</ci></apply><ci id="S2.E9X.2.1.1.m1.1.1.1.1.3.cmml" xref="S2.E9X.2.1.1.m1.1.1.1.1.3">𝖳</ci></apply></apply><apply id="S2.E9X.2.1.1.m1.1.1.3.cmml" xref="S2.E9X.2.1.1.m1.1.1.3"><root id="S2.E9X.2.1.1.m1.1.1.3a.cmml" xref="S2.E9X.2.1.1.m1.1.1.3"></root><apply id="S2.E9X.2.1.1.m1.1.1.3.2.cmml" xref="S2.E9X.2.1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E9X.2.1.1.m1.1.1.3.2.1.cmml" xref="S2.E9X.2.1.1.m1.1.1.3.2">subscript</csymbol><ci id="S2.E9X.2.1.1.m1.1.1.3.2.2.cmml" xref="S2.E9X.2.1.1.m1.1.1.3.2.2">𝑑</ci><ci id="S2.E9X.2.1.1.m1.1.1.3.2.3.cmml" xref="S2.E9X.2.1.1.m1.1.1.3.2.3">𝑘</ci></apply></apply></apply><apply id="S2.E9X.2.1.1.m1.2.2.1.1.3.4.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.4"><csymbol cd="ambiguous" id="S2.E9X.2.1.1.m1.2.2.1.1.3.4.1.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.4">subscript</csymbol><ci id="S2.E9X.2.1.1.m1.2.2.1.1.3.4.2.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.4.2">𝐕</ci><ci id="S2.E9X.2.1.1.m1.2.2.1.1.3.4.3.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.4.3">𝑛</ci></apply><apply id="S2.E9X.2.1.1.m1.2.2.1.1.3.5.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.5"><csymbol cd="ambiguous" id="S2.E9X.2.1.1.m1.2.2.1.1.3.5.1.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.5">subscript</csymbol><ci id="S2.E9X.2.1.1.m1.2.2.1.1.3.5.2.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.5.2">𝐔</ci><ci id="S2.E9X.2.1.1.m1.2.2.1.1.3.5.3.cmml" xref="S2.E9X.2.1.1.m1.2.2.1.1.3.5.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E9X.2.1.1.m1.2c">\displaystyle\mathbf{Z}_{n}=\mathrm{softmax}\left(\frac{\mathbf{X}\mathbf{Q}_{% n}(\mathbf{X}\mathbf{K}_{n})^{\mathsf{T}}}{\sqrt{d_{k}}}\right)\mathbf{V}_{n}% \mathbf{U}_{n}.</annotation><annotation encoding="application/x-llamapun" id="S2.E9X.2.1.1.m1.2d">bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT = roman_softmax ( divide start_ARG bold_XQ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( bold_XK start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT sansserif_T end_POSTSUPERSCRIPT end_ARG start_ARG square-root start_ARG italic_d start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_ARG end_ARG ) bold_V start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT bold_U start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT .</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(9)</td> </tr> </tbody> </table> Once all devices obtain their local outputs <math alttext="\mathbf{Z}_{n}" class="ltx_Math" display="inline" id="S2.SS2.SSS2.p1.10.m1.1"><semantics id="S2.SS2.SSS2.p1.10.m1.1a"><msub id="S2.SS2.SSS2.p1.10.m1.1.1" xref="S2.SS2.SSS2.p1.10.m1.1.1.cmml"><mi id="S2.SS2.SSS2.p1.10.m1.1.1.2" xref="S2.SS2.SSS2.p1.10.m1.1.1.2.cmml">𝐙</mi><mi id="S2.SS2.SSS2.p1.10.m1.1.1.3" xref="S2.SS2.SSS2.p1.10.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p1.10.m1.1b"><apply id="S2.SS2.SSS2.p1.10.m1.1.1.cmml" xref="S2.SS2.SSS2.p1.10.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.SSS2.p1.10.m1.1.1.1.cmml" xref="S2.SS2.SSS2.p1.10.m1.1.1">subscript</csymbol><ci id="S2.SS2.SSS2.p1.10.m1.1.1.2.cmml" xref="S2.SS2.SSS2.p1.10.m1.1.1.2">𝐙</ci><ci id="S2.SS2.SSS2.p1.10.m1.1.1.3.cmml" xref="S2.SS2.SSS2.p1.10.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p1.10.m1.1c">\mathbf{Z}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.SSS2.p1.10.m1.1d">bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math>, a similar all-reduce operation is required to gather and combine the partial outputs from devices as shown in (<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.E5" title="In II-B1 Tensor Parallelism for MLP Layer ‣ II-B Tensor Parallelism ‣ II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">5</a>). </div> </section> </section> <section class="ltx_subsection" id="S2.SS3"> <h3 class="ltx_title ltx_title_subsection"> II-C Over-the-Air All-Reduce </h3> <div class="ltx_para" id="S2.SS3.p1"> Employing tensor parallelism for distributed LLM inference requires frequent all-reduce operations, which cause significant communication overhead in practical wireless networks. To address this issue, we propose a communication-efficient AirComp all-reduce approach. The AirComp aggregates distributed data efficiently by leveraging the signal superposition property of wireless multiple-access channels, allowing simultaneous transmissions to compute nomographic functions (e.g., arithmetic mean) <cite class="ltx_cite ltx_citemacro_cite">[<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#bib.bib46" title="">46</a>]</cite>. In the proposed distributed LLM inference system, the aggregation of intermediate layer outputs in the all-reduce step aligns with this operation, making the AirComp suitable to mitigate communication overhead. Note that edge devices performing AirComp must achieve symbol-level synchronization to ensure their transmitted signals arrive concurrently at the receiver, minimizing aggregation errors due to timing offsets. In our framework, synchronization among edge devices can be practically realized through the well-established timing advance (TA) mechanism. Specifically, the edge server estimates each device’s timing offset and instructs each device to adjust its signal transmission timing via dedicated TA commands. By aligning transmissions precisely, edge devices can ensure simultaneous arrival and accurate signal aggregation at the receiver. </div> <div class="ltx_para" id="S2.SS3.p2"> We consider a wireless network consisting of an edge server with <math alttext="N_{r}" class="ltx_Math" display="inline" id="S2.SS3.p2.1.m1.1"><semantics id="S2.SS3.p2.1.m1.1a"><msub id="S2.SS3.p2.1.m1.1.1" xref="S2.SS3.p2.1.m1.1.1.cmml"><mi id="S2.SS3.p2.1.m1.1.1.2" xref="S2.SS3.p2.1.m1.1.1.2.cmml">N</mi><mi id="S2.SS3.p2.1.m1.1.1.3" xref="S2.SS3.p2.1.m1.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.1.m1.1b"><apply id="S2.SS3.p2.1.m1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.1.m1.1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1">subscript</csymbol><ci id="S2.SS3.p2.1.m1.1.1.2.cmml" xref="S2.SS3.p2.1.m1.1.1.2">𝑁</ci><ci id="S2.SS3.p2.1.m1.1.1.3.cmml" xref="S2.SS3.p2.1.m1.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.1.m1.1c">N_{r}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.1.m1.1d">italic_N start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT</annotation></semantics></math> antennas and <math alttext="N" class="ltx_Math" display="inline" id="S2.SS3.p2.2.m2.1"><semantics id="S2.SS3.p2.2.m2.1a"><mi id="S2.SS3.p2.2.m2.1.1" xref="S2.SS3.p2.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.2.m2.1b"><ci id="S2.SS3.p2.2.m2.1.1.cmml" xref="S2.SS3.p2.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.2.m2.1c">N</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.2.m2.1d">italic_N</annotation></semantics></math> single-antenna edge devices. We further extend the proposed framework to a more general scenario invloving multi-antenna edge devices in Section <a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S4" title="IV Extension to Multi-Antenna Devices ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">IV</a>. The uplink channels from edge devices to the server are block-fading, where channel statistics remain constant throughout the inference process, with channel states varying independently across different time intervals. Let <math alttext="z_{n}" class="ltx_Math" display="inline" id="S2.SS3.p2.3.m3.1"><semantics id="S2.SS3.p2.3.m3.1a"><msub id="S2.SS3.p2.3.m3.1.1" xref="S2.SS3.p2.3.m3.1.1.cmml"><mi id="S2.SS3.p2.3.m3.1.1.2" xref="S2.SS3.p2.3.m3.1.1.2.cmml">z</mi><mi id="S2.SS3.p2.3.m3.1.1.3" xref="S2.SS3.p2.3.m3.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.3.m3.1b"><apply id="S2.SS3.p2.3.m3.1.1.cmml" xref="S2.SS3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.3.m3.1.1.1.cmml" xref="S2.SS3.p2.3.m3.1.1">subscript</csymbol><ci id="S2.SS3.p2.3.m3.1.1.2.cmml" xref="S2.SS3.p2.3.m3.1.1.2">𝑧</ci><ci id="S2.SS3.p2.3.m3.1.1.3.cmml" xref="S2.SS3.p2.3.m3.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.3.m3.1c">z_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.3.m3.1d">italic_z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> denote the per-round transmitted entry of device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS3.p2.4.m4.1"><semantics id="S2.SS3.p2.4.m4.1a"><mi id="S2.SS3.p2.4.m4.1.1" xref="S2.SS3.p2.4.m4.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.4.m4.1b"><ci id="S2.SS3.p2.4.m4.1.1.cmml" xref="S2.SS3.p2.4.m4.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.4.m4.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.4.m4.1d">italic_n</annotation></semantics></math>’s intermediate layer output <math alttext="\mathbf{Z}_{n}" class="ltx_Math" display="inline" id="S2.SS3.p2.5.m5.1"><semantics id="S2.SS3.p2.5.m5.1a"><msub id="S2.SS3.p2.5.m5.1.1" xref="S2.SS3.p2.5.m5.1.1.cmml"><mi id="S2.SS3.p2.5.m5.1.1.2" xref="S2.SS3.p2.5.m5.1.1.2.cmml">𝐙</mi><mi id="S2.SS3.p2.5.m5.1.1.3" xref="S2.SS3.p2.5.m5.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.5.m5.1b"><apply id="S2.SS3.p2.5.m5.1.1.cmml" xref="S2.SS3.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.5.m5.1.1.1.cmml" xref="S2.SS3.p2.5.m5.1.1">subscript</csymbol><ci id="S2.SS3.p2.5.m5.1.1.2.cmml" xref="S2.SS3.p2.5.m5.1.1.2">𝐙</ci><ci id="S2.SS3.p2.5.m5.1.1.3.cmml" xref="S2.SS3.p2.5.m5.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.5.m5.1c">\mathbf{Z}_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.5.m5.1d">bold_Z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math>, which has a complete dimensionality of <math alttext="L_{0}" class="ltx_Math" display="inline" id="S2.SS3.p2.6.m6.1"><semantics id="S2.SS3.p2.6.m6.1a"><msub id="S2.SS3.p2.6.m6.1.1" xref="S2.SS3.p2.6.m6.1.1.cmml"><mi id="S2.SS3.p2.6.m6.1.1.2" xref="S2.SS3.p2.6.m6.1.1.2.cmml">L</mi><mn id="S2.SS3.p2.6.m6.1.1.3" xref="S2.SS3.p2.6.m6.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.6.m6.1b"><apply id="S2.SS3.p2.6.m6.1.1.cmml" xref="S2.SS3.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.6.m6.1.1.1.cmml" xref="S2.SS3.p2.6.m6.1.1">subscript</csymbol><ci id="S2.SS3.p2.6.m6.1.1.2.cmml" xref="S2.SS3.p2.6.m6.1.1.2">𝐿</ci><cn id="S2.SS3.p2.6.m6.1.1.3.cmml" type="integer" xref="S2.SS3.p2.6.m6.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.6.m6.1c">L_{0}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.6.m6.1d">italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT</annotation></semantics></math>. To reduce transmission power, the transmitted symbols are normalized to have zero mean and unit variance, i.e., <math alttext="\mathbb{E}[\|z_{n}\|^{2}]=1" class="ltx_Math" display="inline" id="S2.SS3.p2.7.m7.1"><semantics id="S2.SS3.p2.7.m7.1a"><mrow id="S2.SS3.p2.7.m7.1.1" xref="S2.SS3.p2.7.m7.1.1.cmml"><mrow id="S2.SS3.p2.7.m7.1.1.1" xref="S2.SS3.p2.7.m7.1.1.1.cmml"><mi id="S2.SS3.p2.7.m7.1.1.1.3" xref="S2.SS3.p2.7.m7.1.1.1.3.cmml">𝔼</mi><mo id="S2.SS3.p2.7.m7.1.1.1.2" xref="S2.SS3.p2.7.m7.1.1.1.2.cmml">⁢</mo><mrow id="S2.SS3.p2.7.m7.1.1.1.1.1" xref="S2.SS3.p2.7.m7.1.1.1.1.2.cmml"><mo id="S2.SS3.p2.7.m7.1.1.1.1.1.2" stretchy="false" xref="S2.SS3.p2.7.m7.1.1.1.1.2.1.cmml">[</mo><msup id="S2.SS3.p2.7.m7.1.1.1.1.1.1" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.cmml"><mrow id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.2.cmml"><mo id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.2" stretchy="false" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.2.1.cmml">‖</mo><msub id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.2" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.2.cmml">z</mi><mi id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.3" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.3" stretchy="false" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.SS3.p2.7.m7.1.1.1.1.1.1.3" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.3.cmml">2</mn></msup><mo id="S2.SS3.p2.7.m7.1.1.1.1.1.3" stretchy="false" xref="S2.SS3.p2.7.m7.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="S2.SS3.p2.7.m7.1.1.2" xref="S2.SS3.p2.7.m7.1.1.2.cmml">=</mo><mn id="S2.SS3.p2.7.m7.1.1.3" xref="S2.SS3.p2.7.m7.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.7.m7.1b"><apply id="S2.SS3.p2.7.m7.1.1.cmml" xref="S2.SS3.p2.7.m7.1.1"><eq id="S2.SS3.p2.7.m7.1.1.2.cmml" xref="S2.SS3.p2.7.m7.1.1.2"></eq><apply id="S2.SS3.p2.7.m7.1.1.1.cmml" xref="S2.SS3.p2.7.m7.1.1.1"><times id="S2.SS3.p2.7.m7.1.1.1.2.cmml" xref="S2.SS3.p2.7.m7.1.1.1.2"></times><ci id="S2.SS3.p2.7.m7.1.1.1.3.cmml" xref="S2.SS3.p2.7.m7.1.1.1.3">𝔼</ci><apply id="S2.SS3.p2.7.m7.1.1.1.1.2.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1"><csymbol cd="latexml" id="S2.SS3.p2.7.m7.1.1.1.1.2.1.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S2.SS3.p2.7.m7.1.1.1.1.1.1.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.7.m7.1.1.1.1.1.1.2.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1">superscript</csymbol><apply id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.2.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.2">norm</csymbol><apply id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.2">𝑧</ci><ci id="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><cn id="S2.SS3.p2.7.m7.1.1.1.1.1.1.3.cmml" type="integer" xref="S2.SS3.p2.7.m7.1.1.1.1.1.1.3">2</cn></apply></apply></apply><cn id="S2.SS3.p2.7.m7.1.1.3.cmml" type="integer" xref="S2.SS3.p2.7.m7.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.7.m7.1c">\mathbb{E}[\|z_{n}\|^{2}]=1</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.7.m7.1d">blackboard_E [ ∥ italic_z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ∥ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] = 1</annotation></semantics></math>, where the normalization factor is uniform for all devices and can be inverted at the server. Given synchronized symbol boundaries, all devices transmit their intermediate layer outputs simultaneously. To mitigate the distortion of received signals caused by channel noise, aggregation beamforming is adopted. Let <math alttext="\mathbf{a}\in\mathbb{C}^{N_{r}\times 1}" class="ltx_Math" display="inline" id="S2.SS3.p2.8.m8.1"><semantics id="S2.SS3.p2.8.m8.1a"><mrow id="S2.SS3.p2.8.m8.1.1" xref="S2.SS3.p2.8.m8.1.1.cmml"><mi id="S2.SS3.p2.8.m8.1.1.2" xref="S2.SS3.p2.8.m8.1.1.2.cmml">𝐚</mi><mo id="S2.SS3.p2.8.m8.1.1.1" xref="S2.SS3.p2.8.m8.1.1.1.cmml">∈</mo><msup id="S2.SS3.p2.8.m8.1.1.3" xref="S2.SS3.p2.8.m8.1.1.3.cmml"><mi id="S2.SS3.p2.8.m8.1.1.3.2" xref="S2.SS3.p2.8.m8.1.1.3.2.cmml">ℂ</mi><mrow id="S2.SS3.p2.8.m8.1.1.3.3" xref="S2.SS3.p2.8.m8.1.1.3.3.cmml"><msub id="S2.SS3.p2.8.m8.1.1.3.3.2" xref="S2.SS3.p2.8.m8.1.1.3.3.2.cmml"><mi id="S2.SS3.p2.8.m8.1.1.3.3.2.2" xref="S2.SS3.p2.8.m8.1.1.3.3.2.2.cmml">N</mi><mi id="S2.SS3.p2.8.m8.1.1.3.3.2.3" xref="S2.SS3.p2.8.m8.1.1.3.3.2.3.cmml">r</mi></msub><mo id="S2.SS3.p2.8.m8.1.1.3.3.1" lspace="0.222em" rspace="0.222em" xref="S2.SS3.p2.8.m8.1.1.3.3.1.cmml">×</mo><mn id="S2.SS3.p2.8.m8.1.1.3.3.3" xref="S2.SS3.p2.8.m8.1.1.3.3.3.cmml">1</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.8.m8.1b"><apply id="S2.SS3.p2.8.m8.1.1.cmml" xref="S2.SS3.p2.8.m8.1.1"><in id="S2.SS3.p2.8.m8.1.1.1.cmml" xref="S2.SS3.p2.8.m8.1.1.1"></in><ci id="S2.SS3.p2.8.m8.1.1.2.cmml" xref="S2.SS3.p2.8.m8.1.1.2">𝐚</ci><apply id="S2.SS3.p2.8.m8.1.1.3.cmml" xref="S2.SS3.p2.8.m8.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p2.8.m8.1.1.3.1.cmml" xref="S2.SS3.p2.8.m8.1.1.3">superscript</csymbol><ci id="S2.SS3.p2.8.m8.1.1.3.2.cmml" xref="S2.SS3.p2.8.m8.1.1.3.2">ℂ</ci><apply id="S2.SS3.p2.8.m8.1.1.3.3.cmml" xref="S2.SS3.p2.8.m8.1.1.3.3"><times id="S2.SS3.p2.8.m8.1.1.3.3.1.cmml" xref="S2.SS3.p2.8.m8.1.1.3.3.1"></times><apply id="S2.SS3.p2.8.m8.1.1.3.3.2.cmml" xref="S2.SS3.p2.8.m8.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.SS3.p2.8.m8.1.1.3.3.2.1.cmml" xref="S2.SS3.p2.8.m8.1.1.3.3.2">subscript</csymbol><ci id="S2.SS3.p2.8.m8.1.1.3.3.2.2.cmml" xref="S2.SS3.p2.8.m8.1.1.3.3.2.2">𝑁</ci><ci id="S2.SS3.p2.8.m8.1.1.3.3.2.3.cmml" xref="S2.SS3.p2.8.m8.1.1.3.3.2.3">𝑟</ci></apply><cn id="S2.SS3.p2.8.m8.1.1.3.3.3.cmml" type="integer" xref="S2.SS3.p2.8.m8.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.8.m8.1c">\mathbf{a}\in\mathbb{C}^{N_{r}\times 1}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.8.m8.1d">bold_a ∈ blackboard_C start_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT × 1 end_POSTSUPERSCRIPT</annotation></semantics></math> denote the aggregation beamforming vector at the edge server. After the AirComp, the received signal at the server is given by, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E10"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E10X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle\hat{z}=\mathbf{a}^{\mathsf{H}}\sum_{n=1}^{N}\mathbf{h}_{n}b_{n}z% _{n}+\mathbf{a}^{\mathsf{H}}\mathbf{n}," class="ltx_Math" display="inline" id="S2.E10X.2.1.1.m1.1"><semantics id="S2.E10X.2.1.1.m1.1a"><mrow id="S2.E10X.2.1.1.m1.1.1.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E10X.2.1.1.m1.1.1.1.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.cmml"><mover accent="true" id="S2.E10X.2.1.1.m1.1.1.1.1.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.2.cmml"><mi id="S2.E10X.2.1.1.m1.1.1.1.1.2.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.2.2.cmml">z</mi><mo id="S2.E10X.2.1.1.m1.1.1.1.1.2.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.2.1.cmml">^</mo></mover><mo id="S2.E10X.2.1.1.m1.1.1.1.1.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.1.cmml">=</mo><mrow id="S2.E10X.2.1.1.m1.1.1.1.1.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.cmml"><mrow id="S2.E10X.2.1.1.m1.1.1.1.1.3.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.cmml"><msup id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.cmml"><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.2.cmml">𝐚</mi><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.3.cmml">𝖧</mi></msup><mo id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.1.cmml">⁢</mo><mrow id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.cmml"><mstyle displaystyle="true" id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.cmml"><munderover id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1a" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.cmml"><mo id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.2" movablelimits="false" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.2.cmml">∑</mo><mrow id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.cmml"><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.2.cmml">n</mi><mo id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.1.cmml">=</mo><mn id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.3.cmml">N</mi></munderover></mstyle><mrow id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.cmml"><msub id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.cmml"><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.2.cmml">𝐡</mi><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.3.cmml">n</mi></msub><mo id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.1.cmml">⁢</mo><msub id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.cmml"><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.2.cmml">b</mi><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.3.cmml">n</mi></msub><mo id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.1a" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.1.cmml">⁢</mo><msub id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.cmml"><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.2.cmml">z</mi><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.3.cmml">n</mi></msub></mrow></mrow></mrow><mo id="S2.E10X.2.1.1.m1.1.1.1.1.3.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.1.cmml">+</mo><mrow id="S2.E10X.2.1.1.m1.1.1.1.1.3.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.cmml"><msup id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.cmml"><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.2.cmml">𝐚</mi><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.3.cmml">𝖧</mi></msup><mo id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.1" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.1.cmml">⁢</mo><mi id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.3" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.3.cmml">𝐧</mi></mrow></mrow></mrow><mo id="S2.E10X.2.1.1.m1.1.1.1.2" xref="S2.E10X.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E10X.2.1.1.m1.1b"><apply id="S2.E10X.2.1.1.m1.1.1.1.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1"><eq id="S2.E10X.2.1.1.m1.1.1.1.1.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.1"></eq><apply id="S2.E10X.2.1.1.m1.1.1.1.1.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.2"><ci id="S2.E10X.2.1.1.m1.1.1.1.1.2.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.2.1">^</ci><ci id="S2.E10X.2.1.1.m1.1.1.1.1.2.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.2.2">𝑧</ci></apply><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3"><plus id="S2.E10X.2.1.1.m1.1.1.1.1.3.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.1"></plus><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2"><times id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.1"></times><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2"><csymbol cd="ambiguous" id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2">superscript</csymbol><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.2">𝐚</ci><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.2.3">𝖧</ci></apply><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3"><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1"><csymbol cd="ambiguous" id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1">superscript</csymbol><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1"><csymbol cd="ambiguous" id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1">subscript</csymbol><sum id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.2"></sum><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3"><eq id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.1"></eq><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.2">𝑛</ci><cn id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.3.cmml" type="integer" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.2.3.3">1</cn></apply></apply><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.1.3">𝑁</ci></apply><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2"><times id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.1"></times><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2"><csymbol cd="ambiguous" id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2">subscript</csymbol><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.2">𝐡</ci><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.2.3">𝑛</ci></apply><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3"><csymbol cd="ambiguous" id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3">subscript</csymbol><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.2">𝑏</ci><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.3.3">𝑛</ci></apply><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4"><csymbol cd="ambiguous" id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4">subscript</csymbol><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.2">𝑧</ci><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.2.3.2.4.3">𝑛</ci></apply></apply></apply></apply><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3"><times id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.1"></times><apply id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.1.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2">superscript</csymbol><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.2.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.2">𝐚</ci><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.2.3">𝖧</ci></apply><ci id="S2.E10X.2.1.1.m1.1.1.1.1.3.3.3.cmml" xref="S2.E10X.2.1.1.m1.1.1.1.1.3.3.3">𝐧</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E10X.2.1.1.m1.1c">\displaystyle\hat{z}=\mathbf{a}^{\mathsf{H}}\sum_{n=1}^{N}\mathbf{h}_{n}b_{n}z% _{n}+\mathbf{a}^{\mathsf{H}}\mathbf{n},</annotation><annotation encoding="application/x-llamapun" id="S2.E10X.2.1.1.m1.1d">over^ start_ARG italic_z end_ARG = bold_a start_POSTSUPERSCRIPT sansserif_H end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT bold_h start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT + bold_a start_POSTSUPERSCRIPT sansserif_H end_POSTSUPERSCRIPT bold_n ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(10)</td> </tr> </tbody> </table> where <math alttext="\mathbf{h}_{n}\in\mathbb{C}^{N_{r}\times 1}" class="ltx_Math" display="inline" id="S2.SS3.p2.9.m1.1"><semantics id="S2.SS3.p2.9.m1.1a"><mrow id="S2.SS3.p2.9.m1.1.1" xref="S2.SS3.p2.9.m1.1.1.cmml"><msub id="S2.SS3.p2.9.m1.1.1.2" xref="S2.SS3.p2.9.m1.1.1.2.cmml"><mi id="S2.SS3.p2.9.m1.1.1.2.2" xref="S2.SS3.p2.9.m1.1.1.2.2.cmml">𝐡</mi><mi id="S2.SS3.p2.9.m1.1.1.2.3" xref="S2.SS3.p2.9.m1.1.1.2.3.cmml">n</mi></msub><mo id="S2.SS3.p2.9.m1.1.1.1" xref="S2.SS3.p2.9.m1.1.1.1.cmml">∈</mo><msup id="S2.SS3.p2.9.m1.1.1.3" xref="S2.SS3.p2.9.m1.1.1.3.cmml"><mi id="S2.SS3.p2.9.m1.1.1.3.2" xref="S2.SS3.p2.9.m1.1.1.3.2.cmml">ℂ</mi><mrow id="S2.SS3.p2.9.m1.1.1.3.3" xref="S2.SS3.p2.9.m1.1.1.3.3.cmml"><msub id="S2.SS3.p2.9.m1.1.1.3.3.2" xref="S2.SS3.p2.9.m1.1.1.3.3.2.cmml"><mi id="S2.SS3.p2.9.m1.1.1.3.3.2.2" xref="S2.SS3.p2.9.m1.1.1.3.3.2.2.cmml">N</mi><mi id="S2.SS3.p2.9.m1.1.1.3.3.2.3" xref="S2.SS3.p2.9.m1.1.1.3.3.2.3.cmml">r</mi></msub><mo id="S2.SS3.p2.9.m1.1.1.3.3.1" lspace="0.222em" rspace="0.222em" xref="S2.SS3.p2.9.m1.1.1.3.3.1.cmml">×</mo><mn id="S2.SS3.p2.9.m1.1.1.3.3.3" xref="S2.SS3.p2.9.m1.1.1.3.3.3.cmml">1</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.9.m1.1b"><apply id="S2.SS3.p2.9.m1.1.1.cmml" xref="S2.SS3.p2.9.m1.1.1"><in id="S2.SS3.p2.9.m1.1.1.1.cmml" xref="S2.SS3.p2.9.m1.1.1.1"></in><apply id="S2.SS3.p2.9.m1.1.1.2.cmml" xref="S2.SS3.p2.9.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p2.9.m1.1.1.2.1.cmml" xref="S2.SS3.p2.9.m1.1.1.2">subscript</csymbol><ci id="S2.SS3.p2.9.m1.1.1.2.2.cmml" xref="S2.SS3.p2.9.m1.1.1.2.2">𝐡</ci><ci id="S2.SS3.p2.9.m1.1.1.2.3.cmml" xref="S2.SS3.p2.9.m1.1.1.2.3">𝑛</ci></apply><apply id="S2.SS3.p2.9.m1.1.1.3.cmml" xref="S2.SS3.p2.9.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p2.9.m1.1.1.3.1.cmml" xref="S2.SS3.p2.9.m1.1.1.3">superscript</csymbol><ci id="S2.SS3.p2.9.m1.1.1.3.2.cmml" xref="S2.SS3.p2.9.m1.1.1.3.2">ℂ</ci><apply id="S2.SS3.p2.9.m1.1.1.3.3.cmml" xref="S2.SS3.p2.9.m1.1.1.3.3"><times id="S2.SS3.p2.9.m1.1.1.3.3.1.cmml" xref="S2.SS3.p2.9.m1.1.1.3.3.1"></times><apply id="S2.SS3.p2.9.m1.1.1.3.3.2.cmml" xref="S2.SS3.p2.9.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.SS3.p2.9.m1.1.1.3.3.2.1.cmml" xref="S2.SS3.p2.9.m1.1.1.3.3.2">subscript</csymbol><ci id="S2.SS3.p2.9.m1.1.1.3.3.2.2.cmml" xref="S2.SS3.p2.9.m1.1.1.3.3.2.2">𝑁</ci><ci id="S2.SS3.p2.9.m1.1.1.3.3.2.3.cmml" xref="S2.SS3.p2.9.m1.1.1.3.3.2.3">𝑟</ci></apply><cn id="S2.SS3.p2.9.m1.1.1.3.3.3.cmml" type="integer" xref="S2.SS3.p2.9.m1.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.9.m1.1c">\mathbf{h}_{n}\in\mathbb{C}^{N_{r}\times 1}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.9.m1.1d">bold_h start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ∈ blackboard_C start_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT × 1 end_POSTSUPERSCRIPT</annotation></semantics></math> denotes the uplink channel from device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS3.p2.10.m2.1"><semantics id="S2.SS3.p2.10.m2.1a"><mi id="S2.SS3.p2.10.m2.1.1" xref="S2.SS3.p2.10.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.10.m2.1b"><ci id="S2.SS3.p2.10.m2.1.1.cmml" xref="S2.SS3.p2.10.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.10.m2.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.10.m2.1d">italic_n</annotation></semantics></math> to the server, <math alttext="b_{n}" class="ltx_Math" display="inline" id="S2.SS3.p2.11.m3.1"><semantics id="S2.SS3.p2.11.m3.1a"><msub id="S2.SS3.p2.11.m3.1.1" xref="S2.SS3.p2.11.m3.1.1.cmml"><mi id="S2.SS3.p2.11.m3.1.1.2" xref="S2.SS3.p2.11.m3.1.1.2.cmml">b</mi><mi id="S2.SS3.p2.11.m3.1.1.3" xref="S2.SS3.p2.11.m3.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.11.m3.1b"><apply id="S2.SS3.p2.11.m3.1.1.cmml" xref="S2.SS3.p2.11.m3.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.11.m3.1.1.1.cmml" xref="S2.SS3.p2.11.m3.1.1">subscript</csymbol><ci id="S2.SS3.p2.11.m3.1.1.2.cmml" xref="S2.SS3.p2.11.m3.1.1.2">𝑏</ci><ci id="S2.SS3.p2.11.m3.1.1.3.cmml" xref="S2.SS3.p2.11.m3.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.11.m3.1c">b_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.11.m3.1d">italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> is the transmit power of device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS3.p2.12.m4.1"><semantics id="S2.SS3.p2.12.m4.1a"><mi id="S2.SS3.p2.12.m4.1.1" xref="S2.SS3.p2.12.m4.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.12.m4.1b"><ci id="S2.SS3.p2.12.m4.1.1.cmml" xref="S2.SS3.p2.12.m4.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.12.m4.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.12.m4.1d">italic_n</annotation></semantics></math>, and <math alttext="\mathbf{n}\sim\mathcal{CN}\left(0,\sigma^{2}\mathbf{I}\right)" class="ltx_Math" display="inline" id="S2.SS3.p2.13.m5.2"><semantics id="S2.SS3.p2.13.m5.2a"><mrow id="S2.SS3.p2.13.m5.2.2" xref="S2.SS3.p2.13.m5.2.2.cmml"><mi id="S2.SS3.p2.13.m5.2.2.3" xref="S2.SS3.p2.13.m5.2.2.3.cmml">𝐧</mi><mo id="S2.SS3.p2.13.m5.2.2.2" xref="S2.SS3.p2.13.m5.2.2.2.cmml">∼</mo><mrow id="S2.SS3.p2.13.m5.2.2.1" xref="S2.SS3.p2.13.m5.2.2.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p2.13.m5.2.2.1.3" xref="S2.SS3.p2.13.m5.2.2.1.3.cmml">𝒞</mi><mo id="S2.SS3.p2.13.m5.2.2.1.2" xref="S2.SS3.p2.13.m5.2.2.1.2.cmml">⁢</mo><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p2.13.m5.2.2.1.4" xref="S2.SS3.p2.13.m5.2.2.1.4.cmml">𝒩</mi><mo id="S2.SS3.p2.13.m5.2.2.1.2a" xref="S2.SS3.p2.13.m5.2.2.1.2.cmml">⁢</mo><mrow id="S2.SS3.p2.13.m5.2.2.1.1.1" xref="S2.SS3.p2.13.m5.2.2.1.1.2.cmml"><mo id="S2.SS3.p2.13.m5.2.2.1.1.1.2" xref="S2.SS3.p2.13.m5.2.2.1.1.2.cmml">(</mo><mn id="S2.SS3.p2.13.m5.1.1" xref="S2.SS3.p2.13.m5.1.1.cmml">0</mn><mo id="S2.SS3.p2.13.m5.2.2.1.1.1.3" xref="S2.SS3.p2.13.m5.2.2.1.1.2.cmml">,</mo><mrow id="S2.SS3.p2.13.m5.2.2.1.1.1.1" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.cmml"><msup id="S2.SS3.p2.13.m5.2.2.1.1.1.1.2" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.cmml"><mi id="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.2" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.2.cmml">σ</mi><mn id="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.3" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.3.cmml">2</mn></msup><mo id="S2.SS3.p2.13.m5.2.2.1.1.1.1.1" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.1.cmml">⁢</mo><mi id="S2.SS3.p2.13.m5.2.2.1.1.1.1.3" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.3.cmml">𝐈</mi></mrow><mo id="S2.SS3.p2.13.m5.2.2.1.1.1.4" xref="S2.SS3.p2.13.m5.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.13.m5.2b"><apply id="S2.SS3.p2.13.m5.2.2.cmml" xref="S2.SS3.p2.13.m5.2.2"><csymbol cd="latexml" id="S2.SS3.p2.13.m5.2.2.2.cmml" xref="S2.SS3.p2.13.m5.2.2.2">similar-to</csymbol><ci id="S2.SS3.p2.13.m5.2.2.3.cmml" xref="S2.SS3.p2.13.m5.2.2.3">𝐧</ci><apply id="S2.SS3.p2.13.m5.2.2.1.cmml" xref="S2.SS3.p2.13.m5.2.2.1"><times id="S2.SS3.p2.13.m5.2.2.1.2.cmml" xref="S2.SS3.p2.13.m5.2.2.1.2"></times><ci id="S2.SS3.p2.13.m5.2.2.1.3.cmml" xref="S2.SS3.p2.13.m5.2.2.1.3">𝒞</ci><ci id="S2.SS3.p2.13.m5.2.2.1.4.cmml" xref="S2.SS3.p2.13.m5.2.2.1.4">𝒩</ci><interval closure="open" id="S2.SS3.p2.13.m5.2.2.1.1.2.cmml" xref="S2.SS3.p2.13.m5.2.2.1.1.1"><cn id="S2.SS3.p2.13.m5.1.1.cmml" type="integer" xref="S2.SS3.p2.13.m5.1.1">0</cn><apply id="S2.SS3.p2.13.m5.2.2.1.1.1.1.cmml" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1"><times id="S2.SS3.p2.13.m5.2.2.1.1.1.1.1.cmml" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.1"></times><apply id="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.cmml" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.1.cmml" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.2">superscript</csymbol><ci id="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.2.cmml" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.2">𝜎</ci><cn id="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.3.cmml" type="integer" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.2.3">2</cn></apply><ci id="S2.SS3.p2.13.m5.2.2.1.1.1.1.3.cmml" xref="S2.SS3.p2.13.m5.2.2.1.1.1.1.3">𝐈</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.13.m5.2c">\mathbf{n}\sim\mathcal{CN}\left(0,\sigma^{2}\mathbf{I}\right)</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.13.m5.2d">bold_n ∼ caligraphic_C caligraphic_N ( 0 , italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT bold_I )</annotation></semantics></math> denotes the additive white Gaussian noise vector with <math alttext="\sigma^{2}" class="ltx_Math" display="inline" id="S2.SS3.p2.14.m6.1"><semantics id="S2.SS3.p2.14.m6.1a"><msup id="S2.SS3.p2.14.m6.1.1" xref="S2.SS3.p2.14.m6.1.1.cmml"><mi id="S2.SS3.p2.14.m6.1.1.2" xref="S2.SS3.p2.14.m6.1.1.2.cmml">σ</mi><mn id="S2.SS3.p2.14.m6.1.1.3" xref="S2.SS3.p2.14.m6.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.14.m6.1b"><apply id="S2.SS3.p2.14.m6.1.1.cmml" xref="S2.SS3.p2.14.m6.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.14.m6.1.1.1.cmml" xref="S2.SS3.p2.14.m6.1.1">superscript</csymbol><ci id="S2.SS3.p2.14.m6.1.1.2.cmml" xref="S2.SS3.p2.14.m6.1.1.2">𝜎</ci><cn id="S2.SS3.p2.14.m6.1.1.3.cmml" type="integer" xref="S2.SS3.p2.14.m6.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.14.m6.1c">\sigma^{2}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.14.m6.1d">italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT</annotation></semantics></math> being the noise power. In the single-antenna setting, each device employs only a scalar transmit-power coefficient <math alttext="b_{n}" class="ltx_Math" display="inline" id="S2.SS3.p2.15.1.m1.1"><semantics id="S2.SS3.p2.15.1.m1.1a"><msub id="S2.SS3.p2.15.1.m1.1.1" xref="S2.SS3.p2.15.1.m1.1.1.cmml"><mi id="S2.SS3.p2.15.1.m1.1.1.2" mathcolor="#000000" xref="S2.SS3.p2.15.1.m1.1.1.2.cmml">b</mi><mi id="S2.SS3.p2.15.1.m1.1.1.3" mathcolor="#000000" xref="S2.SS3.p2.15.1.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.15.1.m1.1b"><apply id="S2.SS3.p2.15.1.m1.1.1.cmml" xref="S2.SS3.p2.15.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.15.1.m1.1.1.1.cmml" xref="S2.SS3.p2.15.1.m1.1.1">subscript</csymbol><ci id="S2.SS3.p2.15.1.m1.1.1.2.cmml" xref="S2.SS3.p2.15.1.m1.1.1.2">𝑏</ci><ci id="S2.SS3.p2.15.1.m1.1.1.3.cmml" xref="S2.SS3.p2.15.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.15.1.m1.1c">b_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.15.1.m1.1d">italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> (instead of a beamforming vector) to scale its transmitted scalar entry <math alttext="z_{n}" class="ltx_Math" display="inline" id="S2.SS3.p2.16.2.m2.1"><semantics id="S2.SS3.p2.16.2.m2.1a"><msub id="S2.SS3.p2.16.2.m2.1.1" xref="S2.SS3.p2.16.2.m2.1.1.cmml"><mi id="S2.SS3.p2.16.2.m2.1.1.2" mathcolor="#000000" xref="S2.SS3.p2.16.2.m2.1.1.2.cmml">z</mi><mi id="S2.SS3.p2.16.2.m2.1.1.3" mathcolor="#000000" xref="S2.SS3.p2.16.2.m2.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.16.2.m2.1b"><apply id="S2.SS3.p2.16.2.m2.1.1.cmml" xref="S2.SS3.p2.16.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.16.2.m2.1.1.1.cmml" xref="S2.SS3.p2.16.2.m2.1.1">subscript</csymbol><ci id="S2.SS3.p2.16.2.m2.1.1.2.cmml" xref="S2.SS3.p2.16.2.m2.1.1.2">𝑧</ci><ci id="S2.SS3.p2.16.2.m2.1.1.3.cmml" xref="S2.SS3.p2.16.2.m2.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.16.2.m2.1c">z_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.16.2.m2.1d">italic_z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math>. The distortion of <math alttext="\hat{z}" class="ltx_Math" display="inline" id="S2.SS3.p2.17.m7.1"><semantics id="S2.SS3.p2.17.m7.1a"><mover accent="true" id="S2.SS3.p2.17.m7.1.1" xref="S2.SS3.p2.17.m7.1.1.cmml"><mi id="S2.SS3.p2.17.m7.1.1.2" xref="S2.SS3.p2.17.m7.1.1.2.cmml">z</mi><mo id="S2.SS3.p2.17.m7.1.1.1" xref="S2.SS3.p2.17.m7.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.17.m7.1b"><apply id="S2.SS3.p2.17.m7.1.1.cmml" xref="S2.SS3.p2.17.m7.1.1"><ci id="S2.SS3.p2.17.m7.1.1.1.cmml" xref="S2.SS3.p2.17.m7.1.1.1">^</ci><ci id="S2.SS3.p2.17.m7.1.1.2.cmml" xref="S2.SS3.p2.17.m7.1.1.2">𝑧</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.17.m7.1c">\hat{z}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.17.m7.1d">over^ start_ARG italic_z end_ARG</annotation></semantics></math> with respect to the desired target summation <math alttext="z=\sum_{n=1}^{N}z_{n}" class="ltx_Math" display="inline" id="S2.SS3.p2.18.m8.1"><semantics id="S2.SS3.p2.18.m8.1a"><mrow id="S2.SS3.p2.18.m8.1.1" xref="S2.SS3.p2.18.m8.1.1.cmml"><mi id="S2.SS3.p2.18.m8.1.1.2" xref="S2.SS3.p2.18.m8.1.1.2.cmml">z</mi><mo id="S2.SS3.p2.18.m8.1.1.1" rspace="0.111em" xref="S2.SS3.p2.18.m8.1.1.1.cmml">=</mo><mrow id="S2.SS3.p2.18.m8.1.1.3" xref="S2.SS3.p2.18.m8.1.1.3.cmml"><msubsup id="S2.SS3.p2.18.m8.1.1.3.1" xref="S2.SS3.p2.18.m8.1.1.3.1.cmml"><mo id="S2.SS3.p2.18.m8.1.1.3.1.2.2" xref="S2.SS3.p2.18.m8.1.1.3.1.2.2.cmml">∑</mo><mrow id="S2.SS3.p2.18.m8.1.1.3.1.2.3" xref="S2.SS3.p2.18.m8.1.1.3.1.2.3.cmml"><mi id="S2.SS3.p2.18.m8.1.1.3.1.2.3.2" xref="S2.SS3.p2.18.m8.1.1.3.1.2.3.2.cmml">n</mi><mo id="S2.SS3.p2.18.m8.1.1.3.1.2.3.1" xref="S2.SS3.p2.18.m8.1.1.3.1.2.3.1.cmml">=</mo><mn id="S2.SS3.p2.18.m8.1.1.3.1.2.3.3" xref="S2.SS3.p2.18.m8.1.1.3.1.2.3.3.cmml">1</mn></mrow><mi id="S2.SS3.p2.18.m8.1.1.3.1.3" xref="S2.SS3.p2.18.m8.1.1.3.1.3.cmml">N</mi></msubsup><msub id="S2.SS3.p2.18.m8.1.1.3.2" xref="S2.SS3.p2.18.m8.1.1.3.2.cmml"><mi id="S2.SS3.p2.18.m8.1.1.3.2.2" xref="S2.SS3.p2.18.m8.1.1.3.2.2.cmml">z</mi><mi id="S2.SS3.p2.18.m8.1.1.3.2.3" xref="S2.SS3.p2.18.m8.1.1.3.2.3.cmml">n</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.18.m8.1b"><apply id="S2.SS3.p2.18.m8.1.1.cmml" xref="S2.SS3.p2.18.m8.1.1"><eq id="S2.SS3.p2.18.m8.1.1.1.cmml" xref="S2.SS3.p2.18.m8.1.1.1"></eq><ci id="S2.SS3.p2.18.m8.1.1.2.cmml" xref="S2.SS3.p2.18.m8.1.1.2">𝑧</ci><apply id="S2.SS3.p2.18.m8.1.1.3.cmml" xref="S2.SS3.p2.18.m8.1.1.3"><apply id="S2.SS3.p2.18.m8.1.1.3.1.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1"><csymbol cd="ambiguous" id="S2.SS3.p2.18.m8.1.1.3.1.1.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1">superscript</csymbol><apply id="S2.SS3.p2.18.m8.1.1.3.1.2.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1"><csymbol cd="ambiguous" id="S2.SS3.p2.18.m8.1.1.3.1.2.1.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1">subscript</csymbol><sum id="S2.SS3.p2.18.m8.1.1.3.1.2.2.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1.2.2"></sum><apply id="S2.SS3.p2.18.m8.1.1.3.1.2.3.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1.2.3"><eq id="S2.SS3.p2.18.m8.1.1.3.1.2.3.1.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1.2.3.1"></eq><ci id="S2.SS3.p2.18.m8.1.1.3.1.2.3.2.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1.2.3.2">𝑛</ci><cn id="S2.SS3.p2.18.m8.1.1.3.1.2.3.3.cmml" type="integer" xref="S2.SS3.p2.18.m8.1.1.3.1.2.3.3">1</cn></apply></apply><ci id="S2.SS3.p2.18.m8.1.1.3.1.3.cmml" xref="S2.SS3.p2.18.m8.1.1.3.1.3">𝑁</ci></apply><apply id="S2.SS3.p2.18.m8.1.1.3.2.cmml" xref="S2.SS3.p2.18.m8.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS3.p2.18.m8.1.1.3.2.1.cmml" xref="S2.SS3.p2.18.m8.1.1.3.2">subscript</csymbol><ci id="S2.SS3.p2.18.m8.1.1.3.2.2.cmml" xref="S2.SS3.p2.18.m8.1.1.3.2.2">𝑧</ci><ci id="S2.SS3.p2.18.m8.1.1.3.2.3.cmml" xref="S2.SS3.p2.18.m8.1.1.3.2.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.18.m8.1c">z=\sum_{n=1}^{N}z_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.18.m8.1d">italic_z = ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> is measured by the MSE, which is defined as <table class="ltx_equationgroup ltx_eqn_table" id="S2.E11"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E11X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\textup{MSE}(\hat{z},z)=\mathbb{E}\left[\|\hat{z}-z\|^{2}\right]." class="ltx_Math" display="inline" id="S2.E11X.2.1.1.m1.3"><semantics id="S2.E11X.2.1.1.m1.3a"><mrow id="S2.E11X.2.1.1.m1.3.3.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.cmml"><mrow id="S2.E11X.2.1.1.m1.3.3.1.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.cmml"><mrow id="S2.E11X.2.1.1.m1.3.3.1.1.3" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.cmml"><mtext id="S2.E11X.2.1.1.m1.3.3.1.1.3.2" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.2a.cmml">MSE</mtext><mo id="S2.E11X.2.1.1.m1.3.3.1.1.3.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.1.cmml">⁢</mo><mrow id="S2.E11X.2.1.1.m1.3.3.1.1.3.3.2" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.3.1.cmml"><mo id="S2.E11X.2.1.1.m1.3.3.1.1.3.3.2.1" stretchy="false" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.3.1.cmml">(</mo><mover accent="true" id="S2.E11X.2.1.1.m1.1.1" xref="S2.E11X.2.1.1.m1.1.1.cmml"><mi id="S2.E11X.2.1.1.m1.1.1.2" xref="S2.E11X.2.1.1.m1.1.1.2.cmml">z</mi><mo id="S2.E11X.2.1.1.m1.1.1.1" xref="S2.E11X.2.1.1.m1.1.1.1.cmml">^</mo></mover><mo id="S2.E11X.2.1.1.m1.3.3.1.1.3.3.2.2" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.3.1.cmml">,</mo><mi id="S2.E11X.2.1.1.m1.2.2" xref="S2.E11X.2.1.1.m1.2.2.cmml">z</mi><mo id="S2.E11X.2.1.1.m1.3.3.1.1.3.3.2.3" stretchy="false" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.3.1.cmml">)</mo></mrow></mrow><mo id="S2.E11X.2.1.1.m1.3.3.1.1.2" xref="S2.E11X.2.1.1.m1.3.3.1.1.2.cmml">=</mo><mrow id="S2.E11X.2.1.1.m1.3.3.1.1.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.cmml"><mi id="S2.E11X.2.1.1.m1.3.3.1.1.1.3" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.3.cmml">𝔼</mi><mo id="S2.E11X.2.1.1.m1.3.3.1.1.1.2" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.2.cmml">⁢</mo><mrow id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.2.cmml"><mo id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.2" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.2.1.cmml">[</mo><msup id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.cmml"><mrow id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.2.cmml"><mo id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.2" stretchy="false" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.2" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.2.cmml">z</mi><mo id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.1.cmml">^</mo></mover><mo id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mi id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.3" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.3.cmml">z</mi></mrow><mo id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.3" stretchy="false" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.3" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.3.cmml">2</mn></msup><mo id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.3" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><mo id="S2.E11X.2.1.1.m1.3.3.1.2" lspace="0em" xref="S2.E11X.2.1.1.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E11X.2.1.1.m1.3b"><apply id="S2.E11X.2.1.1.m1.3.3.1.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1"><eq id="S2.E11X.2.1.1.m1.3.3.1.1.2.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.2"></eq><apply id="S2.E11X.2.1.1.m1.3.3.1.1.3.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.3"><times id="S2.E11X.2.1.1.m1.3.3.1.1.3.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.1"></times><ci id="S2.E11X.2.1.1.m1.3.3.1.1.3.2a.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.2"><mtext id="S2.E11X.2.1.1.m1.3.3.1.1.3.2.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.2">MSE</mtext></ci><interval closure="open" id="S2.E11X.2.1.1.m1.3.3.1.1.3.3.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.3.3.2"><apply id="S2.E11X.2.1.1.m1.1.1.cmml" xref="S2.E11X.2.1.1.m1.1.1"><ci id="S2.E11X.2.1.1.m1.1.1.1.cmml" xref="S2.E11X.2.1.1.m1.1.1.1">^</ci><ci id="S2.E11X.2.1.1.m1.1.1.2.cmml" xref="S2.E11X.2.1.1.m1.1.1.2">𝑧</ci></apply><ci id="S2.E11X.2.1.1.m1.2.2.cmml" xref="S2.E11X.2.1.1.m1.2.2">𝑧</ci></interval></apply><apply id="S2.E11X.2.1.1.m1.3.3.1.1.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1"><times id="S2.E11X.2.1.1.m1.3.3.1.1.1.2.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.2"></times><ci id="S2.E11X.2.1.1.m1.3.3.1.1.1.3.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.3">𝔼</ci><apply id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.2.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1"><csymbol cd="latexml" id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.2.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.2.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1">superscript</csymbol><apply id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.2">norm</csymbol><apply id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1"><minus id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1"></minus><apply id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2"><ci id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.1">^</ci><ci id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2.2">𝑧</ci></apply><ci id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.1.1.1.3">𝑧</ci></apply></apply><cn id="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.3.cmml" type="integer" xref="S2.E11X.2.1.1.m1.3.3.1.1.1.1.1.1.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E11X.2.1.1.m1.3c">\displaystyle\textup{MSE}(\hat{z},z)=\mathbb{E}\left[\|\hat{z}-z\|^{2}\right].</annotation><annotation encoding="application/x-llamapun" id="S2.E11X.2.1.1.m1.3d">MSE ( over^ start_ARG italic_z end_ARG , italic_z ) = blackboard_E [ ∥ over^ start_ARG italic_z end_ARG - italic_z ∥ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] .</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(11)</td> </tr> </tbody> </table> The MSE serves as a metric to evaluate the performance of the AirComp all-reduce operations. As shown in the simulations later, the inference accuracy of the distributed on-device LLM inference system is greatly influenced by the transmission error during the AirComp phase. By substituting (<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.E10" title="In II-C Over-the-Air All-Reduce ‣ II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">10</a>) into (<a class="ltx_ref" href="https://arxiv.org/html/2503.14882v1#S2.E11" title="In II-C Over-the-Air All-Reduce ‣ II System Model and Problem Formulation ‣ Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks">11</a>), the MSE can be explicitly represented as a function of aggregation beamforming vector <math alttext="\mathbf{a}" class="ltx_Math" display="inline" id="S2.SS3.p2.19.m1.1"><semantics id="S2.SS3.p2.19.m1.1a"><mi id="S2.SS3.p2.19.m1.1.1" xref="S2.SS3.p2.19.m1.1.1.cmml">𝐚</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.19.m1.1b"><ci id="S2.SS3.p2.19.m1.1.1.cmml" xref="S2.SS3.p2.19.m1.1.1">𝐚</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.19.m1.1c">\mathbf{a}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.19.m1.1d">bold_a</annotation></semantics></math> and transmitter scalars <math alttext="\left\{b_{n}\right\}_{n=1}^{N}" class="ltx_Math" display="inline" id="S2.SS3.p2.20.m2.1"><semantics id="S2.SS3.p2.20.m2.1a"><msubsup id="S2.SS3.p2.20.m2.1.1" xref="S2.SS3.p2.20.m2.1.1.cmml"><mrow id="S2.SS3.p2.20.m2.1.1.1.1.1" xref="S2.SS3.p2.20.m2.1.1.1.1.2.cmml"><mo id="S2.SS3.p2.20.m2.1.1.1.1.1.2" xref="S2.SS3.p2.20.m2.1.1.1.1.2.cmml">{</mo><msub id="S2.SS3.p2.20.m2.1.1.1.1.1.1" xref="S2.SS3.p2.20.m2.1.1.1.1.1.1.cmml"><mi id="S2.SS3.p2.20.m2.1.1.1.1.1.1.2" xref="S2.SS3.p2.20.m2.1.1.1.1.1.1.2.cmml">b</mi><mi id="S2.SS3.p2.20.m2.1.1.1.1.1.1.3" xref="S2.SS3.p2.20.m2.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.SS3.p2.20.m2.1.1.1.1.1.3" xref="S2.SS3.p2.20.m2.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS3.p2.20.m2.1.1.1.3" xref="S2.SS3.p2.20.m2.1.1.1.3.cmml"><mi id="S2.SS3.p2.20.m2.1.1.1.3.2" xref="S2.SS3.p2.20.m2.1.1.1.3.2.cmml">n</mi><mo id="S2.SS3.p2.20.m2.1.1.1.3.1" xref="S2.SS3.p2.20.m2.1.1.1.3.1.cmml">=</mo><mn id="S2.SS3.p2.20.m2.1.1.1.3.3" xref="S2.SS3.p2.20.m2.1.1.1.3.3.cmml">1</mn></mrow><mi id="S2.SS3.p2.20.m2.1.1.3" xref="S2.SS3.p2.20.m2.1.1.3.cmml">N</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.20.m2.1b"><apply id="S2.SS3.p2.20.m2.1.1.cmml" xref="S2.SS3.p2.20.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.20.m2.1.1.2.cmml" xref="S2.SS3.p2.20.m2.1.1">superscript</csymbol><apply id="S2.SS3.p2.20.m2.1.1.1.cmml" xref="S2.SS3.p2.20.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.20.m2.1.1.1.2.cmml" xref="S2.SS3.p2.20.m2.1.1">subscript</csymbol><set id="S2.SS3.p2.20.m2.1.1.1.1.2.cmml" xref="S2.SS3.p2.20.m2.1.1.1.1.1"><apply id="S2.SS3.p2.20.m2.1.1.1.1.1.1.cmml" xref="S2.SS3.p2.20.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.20.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS3.p2.20.m2.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS3.p2.20.m2.1.1.1.1.1.1.2.cmml" xref="S2.SS3.p2.20.m2.1.1.1.1.1.1.2">𝑏</ci><ci id="S2.SS3.p2.20.m2.1.1.1.1.1.1.3.cmml" xref="S2.SS3.p2.20.m2.1.1.1.1.1.1.3">𝑛</ci></apply></set><apply id="S2.SS3.p2.20.m2.1.1.1.3.cmml" xref="S2.SS3.p2.20.m2.1.1.1.3"><eq id="S2.SS3.p2.20.m2.1.1.1.3.1.cmml" xref="S2.SS3.p2.20.m2.1.1.1.3.1"></eq><ci id="S2.SS3.p2.20.m2.1.1.1.3.2.cmml" xref="S2.SS3.p2.20.m2.1.1.1.3.2">𝑛</ci><cn id="S2.SS3.p2.20.m2.1.1.1.3.3.cmml" type="integer" xref="S2.SS3.p2.20.m2.1.1.1.3.3">1</cn></apply></apply><ci id="S2.SS3.p2.20.m2.1.1.3.cmml" xref="S2.SS3.p2.20.m2.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.20.m2.1c">\left\{b_{n}\right\}_{n=1}^{N}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p2.20.m2.1d">{ italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT</annotation></semantics></math> as follows, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E12"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E12X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle\textup{MSE}(\mathbf{a},\left\{b_{n}\right\})=\sum_{n=1}^{N}\left% \|\mathbf{a}^{\mathsf{H}}\mathbf{h}_{n}b_{n}-1\right\|^{2}+\sigma^{2}\mathbf{a% }^{\mathsf{H}}\mathbf{a}." class="ltx_Math" display="inline" id="S2.E12X.2.1.1.m1.2"><semantics id="S2.E12X.2.1.1.m1.2a"><mrow id="S2.E12X.2.1.1.m1.2.2.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S2.E12X.2.1.1.m1.2.2.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.cmml"><mtext id="S2.E12X.2.1.1.m1.2.2.1.1.1.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.3a.cmml">MSE</mtext><mo id="S2.E12X.2.1.1.m1.2.2.1.1.1.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.2.cmml">⁢</mo><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.2.cmml"><mo id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.2" stretchy="false" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.2.cmml">(</mo><mi id="S2.E12X.2.1.1.m1.1.1" xref="S2.E12X.2.1.1.m1.1.1.cmml">𝐚</mi><mo id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.2.cmml">,</mo><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml"><mo id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml">{</mo><msub id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.2.cmml">b</mi><mi id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml">}</mo></mrow><mo id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.4" stretchy="false" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S2.E12X.2.1.1.m1.2.2.1.1.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.3.cmml">=</mo><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.cmml"><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.2.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.cmml"><mstyle displaystyle="true" id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.cmml"><munderover id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2a" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.cmml"><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.2" movablelimits="false" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.2.cmml">∑</mo><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.cmml"><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.2.cmml">n</mi><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.1.cmml">=</mo><mn id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.3.cmml">N</mi></munderover></mstyle><msup id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.cmml"><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.2.cmml"><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.2.1.cmml">‖</mo><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.cmml"><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.cmml"><msup id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.cmml"><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.2.cmml">𝐚</mi><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.3.cmml">𝖧</mi></msup><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.1.cmml">⁢</mo><msub id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.cmml"><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.2.cmml">𝐡</mi><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.3.cmml">n</mi></msub><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.1a" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.1.cmml">⁢</mo><msub id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.cmml"><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.2.cmml">b</mi><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.3.cmml">n</mi></msub></mrow><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.1.cmml">−</mo><mn id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.3.cmml">1</mn></mrow><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.3.cmml">2</mn></msup></mrow><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.2.cmml">+</mo><mrow id="S2.E12X.2.1.1.m1.2.2.1.1.2.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.cmml"><msup id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.cmml"><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.2.cmml">σ</mi><mn id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.3.cmml">2</mn></msup><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.1" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.1.cmml">⁢</mo><msup id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.cmml"><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.2" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.2.cmml">𝐚</mi><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.3" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.3.cmml">𝖧</mi></msup><mo id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.1a" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.1.cmml">⁢</mo><mi id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.4" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.4.cmml">𝐚</mi></mrow></mrow></mrow><mo id="S2.E12X.2.1.1.m1.2.2.1.2" lspace="0em" xref="S2.E12X.2.1.1.m1.2.2.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E12X.2.1.1.m1.2b"><apply id="S2.E12X.2.1.1.m1.2.2.1.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1"><eq id="S2.E12X.2.1.1.m1.2.2.1.1.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.3"></eq><apply id="S2.E12X.2.1.1.m1.2.2.1.1.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1"><times id="S2.E12X.2.1.1.m1.2.2.1.1.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.2"></times><ci id="S2.E12X.2.1.1.m1.2.2.1.1.1.3a.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.3"><mtext id="S2.E12X.2.1.1.m1.2.2.1.1.1.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.3">MSE</mtext></ci><interval closure="open" id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1"><ci id="S2.E12X.2.1.1.m1.1.1.cmml" xref="S2.E12X.2.1.1.m1.1.1">𝐚</ci><set id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1"><apply id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.2">𝑏</ci><ci id="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></set></interval></apply><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2"><plus id="S2.E12X.2.1.1.m1.2.2.1.1.2.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.2"></plus><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1"><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2">superscript</csymbol><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2">subscript</csymbol><sum id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.2"></sum><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3"><eq id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.1"></eq><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.2">𝑛</ci><cn id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.3.cmml" type="integer" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.2.3.3">1</cn></apply></apply><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.2.3">𝑁</ci></apply><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1">superscript</csymbol><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1"><csymbol cd="latexml" id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.2.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.2">norm</csymbol><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1"><minus id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.1"></minus><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2"><times id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.1"></times><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2">superscript</csymbol><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.2">𝐚</ci><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.2.3">𝖧</ci></apply><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.2">𝐡</ci><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.3.3">𝑛</ci></apply><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4">subscript</csymbol><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.2">𝑏</ci><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.2.4.3">𝑛</ci></apply></apply><cn id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.3.cmml" type="integer" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.1.1.1.3">1</cn></apply></apply><cn id="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.3.cmml" type="integer" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.1.1.3">2</cn></apply></apply><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3"><times id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.1"></times><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2">superscript</csymbol><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.2">𝜎</ci><cn id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.3.cmml" type="integer" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.2.3">2</cn></apply><apply id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3"><csymbol cd="ambiguous" id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.1.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3">superscript</csymbol><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.2.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.2">𝐚</ci><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.3.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.3.3">𝖧</ci></apply><ci id="S2.E12X.2.1.1.m1.2.2.1.1.2.3.4.cmml" xref="S2.E12X.2.1.1.m1.2.2.1.1.2.3.4">𝐚</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E12X.2.1.1.m1.2c">\displaystyle\textup{MSE}(\mathbf{a},\left\{b_{n}\right\})=\sum_{n=1}^{N}\left% \|\mathbf{a}^{\mathsf{H}}\mathbf{h}_{n}b_{n}-1\right\|^{2}+\sigma^{2}\mathbf{a% }^{\mathsf{H}}\mathbf{a}.</annotation><annotation encoding="application/x-llamapun" id="S2.E12X.2.1.1.m1.2d">MSE ( bold_a , { italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } ) = ∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT ∥ bold_a start_POSTSUPERSCRIPT sansserif_H end_POSTSUPERSCRIPT bold_h start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT - 1 ∥ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT bold_a start_POSTSUPERSCRIPT sansserif_H end_POSTSUPERSCRIPT bold_a .</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(12)</td> </tr> </tbody> </table> </div> <div class="ltx_para" id="S2.SS3.p3"> Edge devices involved in inference tasks typically have limited energy supply. Thus, we assume that for each device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS3.p3.1.m1.1"><semantics id="S2.SS3.p3.1.m1.1a"><mi id="S2.SS3.p3.1.m1.1.1" xref="S2.SS3.p3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.1.m1.1b"><ci id="S2.SS3.p3.1.m1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.1.m1.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.1.m1.1d">italic_n</annotation></semantics></math>, the energy consumption for both the forward computation of each LLM layer and the transmission of the intermediate output cannot exceed the maximum power budget <math alttext="P_{n}^{\textup{max}}" class="ltx_Math" display="inline" id="S2.SS3.p3.2.m2.1"><semantics id="S2.SS3.p3.2.m2.1a"><msubsup id="S2.SS3.p3.2.m2.1.1" xref="S2.SS3.p3.2.m2.1.1.cmml"><mi id="S2.SS3.p3.2.m2.1.1.2.2" xref="S2.SS3.p3.2.m2.1.1.2.2.cmml">P</mi><mi id="S2.SS3.p3.2.m2.1.1.2.3" xref="S2.SS3.p3.2.m2.1.1.2.3.cmml">n</mi><mtext id="S2.SS3.p3.2.m2.1.1.3" xref="S2.SS3.p3.2.m2.1.1.3a.cmml">max</mtext></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.2.m2.1b"><apply id="S2.SS3.p3.2.m2.1.1.cmml" xref="S2.SS3.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.2.m2.1.1.1.cmml" xref="S2.SS3.p3.2.m2.1.1">superscript</csymbol><apply id="S2.SS3.p3.2.m2.1.1.2.cmml" xref="S2.SS3.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.2.m2.1.1.2.1.cmml" xref="S2.SS3.p3.2.m2.1.1">subscript</csymbol><ci id="S2.SS3.p3.2.m2.1.1.2.2.cmml" xref="S2.SS3.p3.2.m2.1.1.2.2">𝑃</ci><ci id="S2.SS3.p3.2.m2.1.1.2.3.cmml" xref="S2.SS3.p3.2.m2.1.1.2.3">𝑛</ci></apply><ci id="S2.SS3.p3.2.m2.1.1.3a.cmml" xref="S2.SS3.p3.2.m2.1.1.3"><mtext id="S2.SS3.p3.2.m2.1.1.3.cmml" mathsize="70%" xref="S2.SS3.p3.2.m2.1.1.3">max</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.2.m2.1c">P_{n}^{\textup{max}}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.2.m2.1d">italic_P start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT max end_POSTSUPERSCRIPT</annotation></semantics></math>. To model the computation energy consumption, we first introduce a model assignment vector <math alttext="\mathbf{m}=[m_{1},\ldots,m_{N}]" class="ltx_Math" display="inline" id="S2.SS3.p3.3.m3.3"><semantics id="S2.SS3.p3.3.m3.3a"><mrow id="S2.SS3.p3.3.m3.3.3" xref="S2.SS3.p3.3.m3.3.3.cmml"><mi id="S2.SS3.p3.3.m3.3.3.4" xref="S2.SS3.p3.3.m3.3.3.4.cmml">𝐦</mi><mo id="S2.SS3.p3.3.m3.3.3.3" xref="S2.SS3.p3.3.m3.3.3.3.cmml">=</mo><mrow id="S2.SS3.p3.3.m3.3.3.2.2" xref="S2.SS3.p3.3.m3.3.3.2.3.cmml"><mo id="S2.SS3.p3.3.m3.3.3.2.2.3" stretchy="false" xref="S2.SS3.p3.3.m3.3.3.2.3.cmml">[</mo><msub id="S2.SS3.p3.3.m3.2.2.1.1.1" xref="S2.SS3.p3.3.m3.2.2.1.1.1.cmml"><mi id="S2.SS3.p3.3.m3.2.2.1.1.1.2" xref="S2.SS3.p3.3.m3.2.2.1.1.1.2.cmml">m</mi><mn id="S2.SS3.p3.3.m3.2.2.1.1.1.3" xref="S2.SS3.p3.3.m3.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS3.p3.3.m3.3.3.2.2.4" xref="S2.SS3.p3.3.m3.3.3.2.3.cmml">,</mo><mi id="S2.SS3.p3.3.m3.1.1" mathvariant="normal" xref="S2.SS3.p3.3.m3.1.1.cmml">…</mi><mo id="S2.SS3.p3.3.m3.3.3.2.2.5" xref="S2.SS3.p3.3.m3.3.3.2.3.cmml">,</mo><msub id="S2.SS3.p3.3.m3.3.3.2.2.2" xref="S2.SS3.p3.3.m3.3.3.2.2.2.cmml"><mi id="S2.SS3.p3.3.m3.3.3.2.2.2.2" xref="S2.SS3.p3.3.m3.3.3.2.2.2.2.cmml">m</mi><mi id="S2.SS3.p3.3.m3.3.3.2.2.2.3" xref="S2.SS3.p3.3.m3.3.3.2.2.2.3.cmml">N</mi></msub><mo id="S2.SS3.p3.3.m3.3.3.2.2.6" stretchy="false" xref="S2.SS3.p3.3.m3.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.3.m3.3b"><apply id="S2.SS3.p3.3.m3.3.3.cmml" xref="S2.SS3.p3.3.m3.3.3"><eq id="S2.SS3.p3.3.m3.3.3.3.cmml" xref="S2.SS3.p3.3.m3.3.3.3"></eq><ci id="S2.SS3.p3.3.m3.3.3.4.cmml" xref="S2.SS3.p3.3.m3.3.3.4">𝐦</ci><list id="S2.SS3.p3.3.m3.3.3.2.3.cmml" xref="S2.SS3.p3.3.m3.3.3.2.2"><apply id="S2.SS3.p3.3.m3.2.2.1.1.1.cmml" xref="S2.SS3.p3.3.m3.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.3.m3.2.2.1.1.1.1.cmml" xref="S2.SS3.p3.3.m3.2.2.1.1.1">subscript</csymbol><ci id="S2.SS3.p3.3.m3.2.2.1.1.1.2.cmml" xref="S2.SS3.p3.3.m3.2.2.1.1.1.2">𝑚</ci><cn id="S2.SS3.p3.3.m3.2.2.1.1.1.3.cmml" type="integer" xref="S2.SS3.p3.3.m3.2.2.1.1.1.3">1</cn></apply><ci id="S2.SS3.p3.3.m3.1.1.cmml" xref="S2.SS3.p3.3.m3.1.1">…</ci><apply id="S2.SS3.p3.3.m3.3.3.2.2.2.cmml" xref="S2.SS3.p3.3.m3.3.3.2.2.2"><csymbol cd="ambiguous" id="S2.SS3.p3.3.m3.3.3.2.2.2.1.cmml" xref="S2.SS3.p3.3.m3.3.3.2.2.2">subscript</csymbol><ci id="S2.SS3.p3.3.m3.3.3.2.2.2.2.cmml" xref="S2.SS3.p3.3.m3.3.3.2.2.2.2">𝑚</ci><ci id="S2.SS3.p3.3.m3.3.3.2.2.2.3.cmml" xref="S2.SS3.p3.3.m3.3.3.2.2.2.3">𝑁</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.3.m3.3c">\mathbf{m}=[m_{1},\ldots,m_{N}]</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.3.m3.3d">bold_m = [ italic_m start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , italic_m start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ]</annotation></semantics></math> with its entry <math alttext="m_{n}\in[0,1]" class="ltx_Math" display="inline" id="S2.SS3.p3.4.m4.2"><semantics id="S2.SS3.p3.4.m4.2a"><mrow id="S2.SS3.p3.4.m4.2.3" xref="S2.SS3.p3.4.m4.2.3.cmml"><msub id="S2.SS3.p3.4.m4.2.3.2" xref="S2.SS3.p3.4.m4.2.3.2.cmml"><mi id="S2.SS3.p3.4.m4.2.3.2.2" xref="S2.SS3.p3.4.m4.2.3.2.2.cmml">m</mi><mi id="S2.SS3.p3.4.m4.2.3.2.3" xref="S2.SS3.p3.4.m4.2.3.2.3.cmml">n</mi></msub><mo id="S2.SS3.p3.4.m4.2.3.1" xref="S2.SS3.p3.4.m4.2.3.1.cmml">∈</mo><mrow id="S2.SS3.p3.4.m4.2.3.3.2" xref="S2.SS3.p3.4.m4.2.3.3.1.cmml"><mo id="S2.SS3.p3.4.m4.2.3.3.2.1" stretchy="false" xref="S2.SS3.p3.4.m4.2.3.3.1.cmml">[</mo><mn id="S2.SS3.p3.4.m4.1.1" xref="S2.SS3.p3.4.m4.1.1.cmml">0</mn><mo id="S2.SS3.p3.4.m4.2.3.3.2.2" xref="S2.SS3.p3.4.m4.2.3.3.1.cmml">,</mo><mn id="S2.SS3.p3.4.m4.2.2" xref="S2.SS3.p3.4.m4.2.2.cmml">1</mn><mo id="S2.SS3.p3.4.m4.2.3.3.2.3" stretchy="false" xref="S2.SS3.p3.4.m4.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.4.m4.2b"><apply id="S2.SS3.p3.4.m4.2.3.cmml" xref="S2.SS3.p3.4.m4.2.3"><in id="S2.SS3.p3.4.m4.2.3.1.cmml" xref="S2.SS3.p3.4.m4.2.3.1"></in><apply id="S2.SS3.p3.4.m4.2.3.2.cmml" xref="S2.SS3.p3.4.m4.2.3.2"><csymbol cd="ambiguous" id="S2.SS3.p3.4.m4.2.3.2.1.cmml" xref="S2.SS3.p3.4.m4.2.3.2">subscript</csymbol><ci id="S2.SS3.p3.4.m4.2.3.2.2.cmml" xref="S2.SS3.p3.4.m4.2.3.2.2">𝑚</ci><ci id="S2.SS3.p3.4.m4.2.3.2.3.cmml" xref="S2.SS3.p3.4.m4.2.3.2.3">𝑛</ci></apply><interval closure="closed" id="S2.SS3.p3.4.m4.2.3.3.1.cmml" xref="S2.SS3.p3.4.m4.2.3.3.2"><cn id="S2.SS3.p3.4.m4.1.1.cmml" type="integer" xref="S2.SS3.p3.4.m4.1.1">0</cn><cn id="S2.SS3.p3.4.m4.2.2.cmml" type="integer" xref="S2.SS3.p3.4.m4.2.2">1</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.4.m4.2c">m_{n}\in[0,1]</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.4.m4.2d">italic_m start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ∈ [ 0 , 1 ]</annotation></semantics></math> representing the proportion of model allocated to device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS3.p3.5.m5.1"><semantics id="S2.SS3.p3.5.m5.1a"><mi id="S2.SS3.p3.5.m5.1.1" xref="S2.SS3.p3.5.m5.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.5.m5.1b"><ci id="S2.SS3.p3.5.m5.1.1.cmml" xref="S2.SS3.p3.5.m5.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.5.m5.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.5.m5.1d">italic_n</annotation></semantics></math>. Consequently, the computation energy consumption for device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS3.p3.6.m6.1"><semantics id="S2.SS3.p3.6.m6.1a"><mi id="S2.SS3.p3.6.m6.1.1" xref="S2.SS3.p3.6.m6.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.6.m6.1b"><ci id="S2.SS3.p3.6.m6.1.1.cmml" xref="S2.SS3.p3.6.m6.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.6.m6.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.6.m6.1d">italic_n</annotation></semantics></math> is given by <math alttext="e_{n}m_{n}s^{\textup{tot}}" class="ltx_Math" display="inline" id="S2.SS3.p3.7.m7.1"><semantics id="S2.SS3.p3.7.m7.1a"><mrow id="S2.SS3.p3.7.m7.1.1" xref="S2.SS3.p3.7.m7.1.1.cmml"><msub id="S2.SS3.p3.7.m7.1.1.2" xref="S2.SS3.p3.7.m7.1.1.2.cmml"><mi id="S2.SS3.p3.7.m7.1.1.2.2" xref="S2.SS3.p3.7.m7.1.1.2.2.cmml">e</mi><mi id="S2.SS3.p3.7.m7.1.1.2.3" xref="S2.SS3.p3.7.m7.1.1.2.3.cmml">n</mi></msub><mo id="S2.SS3.p3.7.m7.1.1.1" xref="S2.SS3.p3.7.m7.1.1.1.cmml">⁢</mo><msub id="S2.SS3.p3.7.m7.1.1.3" xref="S2.SS3.p3.7.m7.1.1.3.cmml"><mi id="S2.SS3.p3.7.m7.1.1.3.2" xref="S2.SS3.p3.7.m7.1.1.3.2.cmml">m</mi><mi id="S2.SS3.p3.7.m7.1.1.3.3" xref="S2.SS3.p3.7.m7.1.1.3.3.cmml">n</mi></msub><mo id="S2.SS3.p3.7.m7.1.1.1a" xref="S2.SS3.p3.7.m7.1.1.1.cmml">⁢</mo><msup id="S2.SS3.p3.7.m7.1.1.4" xref="S2.SS3.p3.7.m7.1.1.4.cmml"><mi id="S2.SS3.p3.7.m7.1.1.4.2" xref="S2.SS3.p3.7.m7.1.1.4.2.cmml">s</mi><mtext id="S2.SS3.p3.7.m7.1.1.4.3" xref="S2.SS3.p3.7.m7.1.1.4.3a.cmml">tot</mtext></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.7.m7.1b"><apply id="S2.SS3.p3.7.m7.1.1.cmml" xref="S2.SS3.p3.7.m7.1.1"><times id="S2.SS3.p3.7.m7.1.1.1.cmml" xref="S2.SS3.p3.7.m7.1.1.1"></times><apply id="S2.SS3.p3.7.m7.1.1.2.cmml" xref="S2.SS3.p3.7.m7.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p3.7.m7.1.1.2.1.cmml" xref="S2.SS3.p3.7.m7.1.1.2">subscript</csymbol><ci id="S2.SS3.p3.7.m7.1.1.2.2.cmml" xref="S2.SS3.p3.7.m7.1.1.2.2">𝑒</ci><ci id="S2.SS3.p3.7.m7.1.1.2.3.cmml" xref="S2.SS3.p3.7.m7.1.1.2.3">𝑛</ci></apply><apply id="S2.SS3.p3.7.m7.1.1.3.cmml" xref="S2.SS3.p3.7.m7.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p3.7.m7.1.1.3.1.cmml" xref="S2.SS3.p3.7.m7.1.1.3">subscript</csymbol><ci id="S2.SS3.p3.7.m7.1.1.3.2.cmml" xref="S2.SS3.p3.7.m7.1.1.3.2">𝑚</ci><ci id="S2.SS3.p3.7.m7.1.1.3.3.cmml" xref="S2.SS3.p3.7.m7.1.1.3.3">𝑛</ci></apply><apply id="S2.SS3.p3.7.m7.1.1.4.cmml" xref="S2.SS3.p3.7.m7.1.1.4"><csymbol cd="ambiguous" id="S2.SS3.p3.7.m7.1.1.4.1.cmml" xref="S2.SS3.p3.7.m7.1.1.4">superscript</csymbol><ci id="S2.SS3.p3.7.m7.1.1.4.2.cmml" xref="S2.SS3.p3.7.m7.1.1.4.2">𝑠</ci><ci id="S2.SS3.p3.7.m7.1.1.4.3a.cmml" xref="S2.SS3.p3.7.m7.1.1.4.3"><mtext id="S2.SS3.p3.7.m7.1.1.4.3.cmml" mathsize="70%" xref="S2.SS3.p3.7.m7.1.1.4.3">tot</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.7.m7.1c">e_{n}m_{n}s^{\textup{tot}}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.7.m7.1d">italic_e start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_m start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_s start_POSTSUPERSCRIPT tot end_POSTSUPERSCRIPT</annotation></semantics></math>, where <math alttext="e_{n}" class="ltx_Math" display="inline" id="S2.SS3.p3.8.m8.1"><semantics id="S2.SS3.p3.8.m8.1a"><msub id="S2.SS3.p3.8.m8.1.1" xref="S2.SS3.p3.8.m8.1.1.cmml"><mi id="S2.SS3.p3.8.m8.1.1.2" xref="S2.SS3.p3.8.m8.1.1.2.cmml">e</mi><mi id="S2.SS3.p3.8.m8.1.1.3" xref="S2.SS3.p3.8.m8.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.8.m8.1b"><apply id="S2.SS3.p3.8.m8.1.1.cmml" xref="S2.SS3.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.8.m8.1.1.1.cmml" xref="S2.SS3.p3.8.m8.1.1">subscript</csymbol><ci id="S2.SS3.p3.8.m8.1.1.2.cmml" xref="S2.SS3.p3.8.m8.1.1.2">𝑒</ci><ci id="S2.SS3.p3.8.m8.1.1.3.cmml" xref="S2.SS3.p3.8.m8.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.8.m8.1c">e_{n}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.8.m8.1d">italic_e start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> denotes the device-specific energy coefficient that reflects the energy cost associated with accessing and processing each weight during computation, and <math alttext="s^{\textup{tot}}" class="ltx_Math" display="inline" id="S2.SS3.p3.9.m9.1"><semantics id="S2.SS3.p3.9.m9.1a"><msup id="S2.SS3.p3.9.m9.1.1" xref="S2.SS3.p3.9.m9.1.1.cmml"><mi id="S2.SS3.p3.9.m9.1.1.2" xref="S2.SS3.p3.9.m9.1.1.2.cmml">s</mi><mtext id="S2.SS3.p3.9.m9.1.1.3" xref="S2.SS3.p3.9.m9.1.1.3a.cmml">tot</mtext></msup><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.9.m9.1b"><apply id="S2.SS3.p3.9.m9.1.1.cmml" xref="S2.SS3.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.9.m9.1.1.1.cmml" xref="S2.SS3.p3.9.m9.1.1">superscript</csymbol><ci id="S2.SS3.p3.9.m9.1.1.2.cmml" xref="S2.SS3.p3.9.m9.1.1.2">𝑠</ci><ci id="S2.SS3.p3.9.m9.1.1.3a.cmml" xref="S2.SS3.p3.9.m9.1.1.3"><mtext id="S2.SS3.p3.9.m9.1.1.3.cmml" mathsize="70%" xref="S2.SS3.p3.9.m9.1.1.3">tot</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.9.m9.1c">s^{\textup{tot}}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.9.m9.1d">italic_s start_POSTSUPERSCRIPT tot end_POSTSUPERSCRIPT</annotation></semantics></math> is the number of parameters (weights) for each layer. The communication energy consumption of device <math alttext="n" class="ltx_Math" display="inline" id="S2.SS3.p3.10.m10.1"><semantics id="S2.SS3.p3.10.m10.1a"><mi id="S2.SS3.p3.10.m10.1.1" xref="S2.SS3.p3.10.m10.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.10.m10.1b"><ci id="S2.SS3.p3.10.m10.1.1.cmml" xref="S2.SS3.p3.10.m10.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.10.m10.1c">n</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.10.m10.1d">italic_n</annotation></semantics></math> can be derived as <math alttext="L_{0}\|b_{n}\|^{2}" class="ltx_Math" display="inline" id="S2.SS3.p3.11.m11.1"><semantics id="S2.SS3.p3.11.m11.1a"><mrow id="S2.SS3.p3.11.m11.1.1" xref="S2.SS3.p3.11.m11.1.1.cmml"><msub id="S2.SS3.p3.11.m11.1.1.3" xref="S2.SS3.p3.11.m11.1.1.3.cmml"><mi id="S2.SS3.p3.11.m11.1.1.3.2" xref="S2.SS3.p3.11.m11.1.1.3.2.cmml">L</mi><mn id="S2.SS3.p3.11.m11.1.1.3.3" xref="S2.SS3.p3.11.m11.1.1.3.3.cmml">0</mn></msub><mo id="S2.SS3.p3.11.m11.1.1.2" xref="S2.SS3.p3.11.m11.1.1.2.cmml">⁢</mo><msup id="S2.SS3.p3.11.m11.1.1.1" xref="S2.SS3.p3.11.m11.1.1.1.cmml"><mrow id="S2.SS3.p3.11.m11.1.1.1.1.1" xref="S2.SS3.p3.11.m11.1.1.1.1.2.cmml"><mo id="S2.SS3.p3.11.m11.1.1.1.1.1.2" stretchy="false" xref="S2.SS3.p3.11.m11.1.1.1.1.2.1.cmml">‖</mo><msub id="S2.SS3.p3.11.m11.1.1.1.1.1.1" xref="S2.SS3.p3.11.m11.1.1.1.1.1.1.cmml"><mi id="S2.SS3.p3.11.m11.1.1.1.1.1.1.2" xref="S2.SS3.p3.11.m11.1.1.1.1.1.1.2.cmml">b</mi><mi id="S2.SS3.p3.11.m11.1.1.1.1.1.1.3" xref="S2.SS3.p3.11.m11.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.SS3.p3.11.m11.1.1.1.1.1.3" stretchy="false" xref="S2.SS3.p3.11.m11.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.SS3.p3.11.m11.1.1.1.3" xref="S2.SS3.p3.11.m11.1.1.1.3.cmml">2</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.11.m11.1b"><apply id="S2.SS3.p3.11.m11.1.1.cmml" xref="S2.SS3.p3.11.m11.1.1"><times id="S2.SS3.p3.11.m11.1.1.2.cmml" xref="S2.SS3.p3.11.m11.1.1.2"></times><apply id="S2.SS3.p3.11.m11.1.1.3.cmml" xref="S2.SS3.p3.11.m11.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p3.11.m11.1.1.3.1.cmml" xref="S2.SS3.p3.11.m11.1.1.3">subscript</csymbol><ci id="S2.SS3.p3.11.m11.1.1.3.2.cmml" xref="S2.SS3.p3.11.m11.1.1.3.2">𝐿</ci><cn id="S2.SS3.p3.11.m11.1.1.3.3.cmml" type="integer" xref="S2.SS3.p3.11.m11.1.1.3.3">0</cn></apply><apply id="S2.SS3.p3.11.m11.1.1.1.cmml" xref="S2.SS3.p3.11.m11.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.11.m11.1.1.1.2.cmml" xref="S2.SS3.p3.11.m11.1.1.1">superscript</csymbol><apply id="S2.SS3.p3.11.m11.1.1.1.1.2.cmml" xref="S2.SS3.p3.11.m11.1.1.1.1.1"><csymbol cd="latexml" id="S2.SS3.p3.11.m11.1.1.1.1.2.1.cmml" xref="S2.SS3.p3.11.m11.1.1.1.1.1.2">norm</csymbol><apply id="S2.SS3.p3.11.m11.1.1.1.1.1.1.cmml" xref="S2.SS3.p3.11.m11.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.11.m11.1.1.1.1.1.1.1.cmml" xref="S2.SS3.p3.11.m11.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS3.p3.11.m11.1.1.1.1.1.1.2.cmml" xref="S2.SS3.p3.11.m11.1.1.1.1.1.1.2">𝑏</ci><ci id="S2.SS3.p3.11.m11.1.1.1.1.1.1.3.cmml" xref="S2.SS3.p3.11.m11.1.1.1.1.1.1.3">𝑛</ci></apply></apply><cn id="S2.SS3.p3.11.m11.1.1.1.3.cmml" type="integer" xref="S2.SS3.p3.11.m11.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.11.m11.1c">L_{0}\|b_{n}\|^{2}</annotation><annotation encoding="application/x-llamapun" id="S2.SS3.p3.11.m11.1d">italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥ italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ∥ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT</annotation></semantics></math>. Accordingly, the power constraint is given by <table class="ltx_equationgroup ltx_eqn_table" id="S2.E13"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E13X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle e_{n}m_{n}s^{\textup{tot}}+L_{0}\|b_{n}\|^{2}\leq P_{n}^{\textup% {max}},\forall n." class="ltx_Math" display="inline" id="S2.E13X.2.1.1.m1.1"><semantics id="S2.E13X.2.1.1.m1.1a"><mrow id="S2.E13X.2.1.1.m1.1.1.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E13X.2.1.1.m1.1.1.1.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E13X.2.1.1.m1.1.1.1.1.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.cmml"><mrow id="S2.E13X.2.1.1.m1.1.1.1.1.1.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.cmml"><msub id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.cmml"><mi id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.2.cmml">e</mi><mi id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.3.cmml">n</mi></msub><mo id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.1.cmml">⁢</mo><msub id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.cmml"><mi id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.2.cmml">m</mi><mi id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.3.cmml">n</mi></msub><mo id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.1a" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.1.cmml">⁢</mo><msup id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.cmml"><mi id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.2.cmml">s</mi><mtext id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.3a.cmml">tot</mtext></msup></mrow><mo id="S2.E13X.2.1.1.m1.1.1.1.1.1.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.2.cmml">+</mo><mrow id="S2.E13X.2.1.1.m1.1.1.1.1.1.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.cmml"><msub id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.2.cmml">L</mi><mn id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.3.cmml">0</mn></msub><mo id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.2.cmml">⁢</mo><msup id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.cmml"><mrow id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml"><mo id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.2" stretchy="false" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.2.1.cmml">‖</mo><msub id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.cmml">b</mi><mi id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.3" stretchy="false" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.3.cmml">2</mn></msup></mrow></mrow><mo id="S2.E13X.2.1.1.m1.1.1.1.1.4" xref="S2.E13X.2.1.1.m1.1.1.1.1.4.cmml">≤</mo><mrow id="S2.E13X.2.1.1.m1.1.1.1.1.3.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.3.cmml"><msubsup id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.cmml"><mi id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.2.cmml">P</mi><mi id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.3.cmml">n</mi><mtext id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.3a.cmml">max</mtext></msubsup><mo id="S2.E13X.2.1.1.m1.1.1.1.1.3.2.3" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.3.cmml">,</mo><mrow id="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.cmml"><mo id="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.1" rspace="0.167em" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.1.cmml">∀</mo><mi id="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.2" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.2.cmml">n</mi></mrow></mrow></mrow><mo id="S2.E13X.2.1.1.m1.1.1.1.2" lspace="0em" xref="S2.E13X.2.1.1.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E13X.2.1.1.m1.1b"><apply id="S2.E13X.2.1.1.m1.1.1.1.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1"><leq id="S2.E13X.2.1.1.m1.1.1.1.1.4.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.4"></leq><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1"><plus id="S2.E13X.2.1.1.m1.1.1.1.1.1.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.2"></plus><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3"><times id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.1"></times><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.2">𝑒</ci><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.3.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.2.3">𝑛</ci></apply><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.2">𝑚</ci><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.3.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.3.3">𝑛</ci></apply><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4">superscript</csymbol><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.2">𝑠</ci><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.3a.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.3"><mtext id="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.3.cmml" mathsize="70%" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.3.4.3">tot</mtext></ci></apply></apply><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1"><times id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.2"></times><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.2">𝐿</ci><cn id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.3.cmml" type="integer" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.3.3">0</cn></apply><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.2">norm</csymbol><apply id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2">𝑏</ci><ci id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><cn id="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.3.cmml" type="integer" xref="S2.E13X.2.1.1.m1.1.1.1.1.1.1.1.3">2</cn></apply></apply></apply><list id="S2.E13X.2.1.1.m1.1.1.1.1.3.3.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.2"><apply id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1">superscript</csymbol><apply id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1">subscript</csymbol><ci id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.2">𝑃</ci><ci id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.3.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.2.3">𝑛</ci></apply><ci id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.3a.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.3"><mtext id="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.3.cmml" mathsize="70%" xref="S2.E13X.2.1.1.m1.1.1.1.1.2.1.1.3">max</mtext></ci></apply><apply id="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2"><csymbol cd="latexml" id="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.1.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.1">for-all</csymbol><ci id="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.2.cmml" xref="S2.E13X.2.1.1.m1.1.1.1.1.3.2.2.2">𝑛</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E13X.2.1.1.m1.1c">\displaystyle e_{n}m_{n}s^{\textup{tot}}+L_{0}\|b_{n}\|^{2}\leq P_{n}^{\textup% {max}},\forall n.</annotation><annotation encoding="application/x-llamapun" id="S2.E13X.2.1.1.m1.1d">italic_e start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_m start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_s start_POSTSUPERSCRIPT tot end_POSTSUPERSCRIPT + italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥ italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ∥ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ≤ italic_P start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT max end_POSTSUPERSCRIPT , ∀ italic_n .</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(13)</td> </tr> </tbody> </table> </div> </section> <section class="ltx_subsection" id="S2.SS4"> <h3 class="ltx_title ltx_title_subsection"> II-D Problem Formulation </h3> <div class="ltx_para" id="S2.SS4.p1"> In the proposed distributed LLM inference system, the overall performance is determined by the model assignment policy <math alttext="\mathbf{m}" class="ltx_Math" display="inline" id="S2.SS4.p1.1.m1.1"><semantics id="S2.SS4.p1.1.m1.1a"><mi id="S2.SS4.p1.1.m1.1.1" xref="S2.SS4.p1.1.m1.1.1.cmml">𝐦</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.1.m1.1b"><ci id="S2.SS4.p1.1.m1.1.1.cmml" xref="S2.SS4.p1.1.m1.1.1">𝐦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.1.m1.1c">\mathbf{m}</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p1.1.m1.1d">bold_m</annotation></semantics></math> and the transceiver design <math alttext="\mathbf{a}" class="ltx_Math" display="inline" id="S2.SS4.p1.2.m2.1"><semantics id="S2.SS4.p1.2.m2.1a"><mi id="S2.SS4.p1.2.m2.1.1" xref="S2.SS4.p1.2.m2.1.1.cmml">𝐚</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.2.m2.1b"><ci id="S2.SS4.p1.2.m2.1.1.cmml" xref="S2.SS4.p1.2.m2.1.1">𝐚</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.2.m2.1c">\mathbf{a}</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p1.2.m2.1d">bold_a</annotation></semantics></math>, <math alttext="\left\{b_{n}\right\}" class="ltx_Math" display="inline" id="S2.SS4.p1.3.m3.1"><semantics id="S2.SS4.p1.3.m3.1a"><mrow id="S2.SS4.p1.3.m3.1.1.1" xref="S2.SS4.p1.3.m3.1.1.2.cmml"><mo id="S2.SS4.p1.3.m3.1.1.1.2" xref="S2.SS4.p1.3.m3.1.1.2.cmml">{</mo><msub id="S2.SS4.p1.3.m3.1.1.1.1" xref="S2.SS4.p1.3.m3.1.1.1.1.cmml"><mi id="S2.SS4.p1.3.m3.1.1.1.1.2" xref="S2.SS4.p1.3.m3.1.1.1.1.2.cmml">b</mi><mi id="S2.SS4.p1.3.m3.1.1.1.1.3" xref="S2.SS4.p1.3.m3.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.SS4.p1.3.m3.1.1.1.3" xref="S2.SS4.p1.3.m3.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.3.m3.1b"><set id="S2.SS4.p1.3.m3.1.1.2.cmml" xref="S2.SS4.p1.3.m3.1.1.1"><apply id="S2.SS4.p1.3.m3.1.1.1.1.cmml" xref="S2.SS4.p1.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.3.m3.1.1.1.1.1.cmml" xref="S2.SS4.p1.3.m3.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p1.3.m3.1.1.1.1.2.cmml" xref="S2.SS4.p1.3.m3.1.1.1.1.2">𝑏</ci><ci id="S2.SS4.p1.3.m3.1.1.1.1.3.cmml" xref="S2.SS4.p1.3.m3.1.1.1.1.3">𝑛</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.3.m3.1c">\left\{b_{n}\right\}</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p1.3.m3.1d">{ italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT }</annotation></semantics></math>. Optimal model assignment ensures that each device processes a suitable portion of the model based on its computational capability (e.g., memory size and compute power). Meanwhile, efficient transceiver optimization can reduce signal misalignment error and suppress channel noise, thereby improving inference accuracy. Thus, to improve inference performance, we formulate a joint model assignment and transceiver optimization problem that aims to minimize the average MSE, subject to the per-device power constraints. Importantly, the transceiver design can adapt dynamically to instantaneous CSI. In contrast, adapting the model assignment policy to instantaneous CSI in a real-time manner is impractical due to the significant latency caused by loading different model segments. Thus, model assignment should be finished before inference based on the long-term channel statistics. </div> <div class="ltx_para" id="S2.SS4.p2"> The resulting problem is therefore formulated as a mixed-timescale joint optimization of the short-term transceiver variables <math alttext="\mathbf{a}" class="ltx_Math" display="inline" id="S2.SS4.p2.1.m1.1"><semantics id="S2.SS4.p2.1.m1.1a"><mi id="S2.SS4.p2.1.m1.1.1" xref="S2.SS4.p2.1.m1.1.1.cmml">𝐚</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.1.m1.1b"><ci id="S2.SS4.p2.1.m1.1.1.cmml" xref="S2.SS4.p2.1.m1.1.1">𝐚</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.1.m1.1c">\mathbf{a}</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p2.1.m1.1d">bold_a</annotation></semantics></math>, <math alttext="\left\{b_{n}\right\}" class="ltx_Math" display="inline" id="S2.SS4.p2.2.m2.1"><semantics id="S2.SS4.p2.2.m2.1a"><mrow id="S2.SS4.p2.2.m2.1.1.1" xref="S2.SS4.p2.2.m2.1.1.2.cmml"><mo id="S2.SS4.p2.2.m2.1.1.1.2" xref="S2.SS4.p2.2.m2.1.1.2.cmml">{</mo><msub id="S2.SS4.p2.2.m2.1.1.1.1" xref="S2.SS4.p2.2.m2.1.1.1.1.cmml"><mi id="S2.SS4.p2.2.m2.1.1.1.1.2" xref="S2.SS4.p2.2.m2.1.1.1.1.2.cmml">b</mi><mi id="S2.SS4.p2.2.m2.1.1.1.1.3" xref="S2.SS4.p2.2.m2.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.SS4.p2.2.m2.1.1.1.3" xref="S2.SS4.p2.2.m2.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.2.m2.1b"><set id="S2.SS4.p2.2.m2.1.1.2.cmml" xref="S2.SS4.p2.2.m2.1.1.1"><apply id="S2.SS4.p2.2.m2.1.1.1.1.cmml" xref="S2.SS4.p2.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.2.m2.1.1.1.1.1.cmml" xref="S2.SS4.p2.2.m2.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p2.2.m2.1.1.1.1.2.cmml" xref="S2.SS4.p2.2.m2.1.1.1.1.2">𝑏</ci><ci id="S2.SS4.p2.2.m2.1.1.1.1.3.cmml" xref="S2.SS4.p2.2.m2.1.1.1.1.3">𝑛</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.2.m2.1c">\left\{b_{n}\right\}</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p2.2.m2.1d">{ italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT }</annotation></semantics></math> and the long-term model assignment policy <math alttext="\mathbf{m}" class="ltx_Math" display="inline" id="S2.SS4.p2.3.m3.1"><semantics id="S2.SS4.p2.3.m3.1a"><mi id="S2.SS4.p2.3.m3.1.1" xref="S2.SS4.p2.3.m3.1.1.cmml">𝐦</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.3.m3.1b"><ci id="S2.SS4.p2.3.m3.1.1.cmml" xref="S2.SS4.p2.3.m3.1.1">𝐦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.3.m3.1c">\mathbf{m}</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p2.3.m3.1d">bold_m</annotation></semantics></math> as follows, <table class="ltx_equationgroup ltx_eqn_table" id="S2.E14"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E14X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell"><math alttext="\displaystyle\mathcal{P}_{1}:~{}\min_{\mathbf{m}}" class="ltx_Math" display="inline" id="S2.E14X.2.1.1.m1.1"><semantics id="S2.E14X.2.1.1.m1.1a"><mrow id="S2.E14X.2.1.1.m1.1.1" xref="S2.E14X.2.1.1.m1.1.1.cmml"><msub id="S2.E14X.2.1.1.m1.1.1.2" xref="S2.E14X.2.1.1.m1.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E14X.2.1.1.m1.1.1.2.2" xref="S2.E14X.2.1.1.m1.1.1.2.2.cmml">𝒫</mi><mn id="S2.E14X.2.1.1.m1.1.1.2.3" xref="S2.E14X.2.1.1.m1.1.1.2.3.cmml">1</mn></msub><mo id="S2.E14X.2.1.1.m1.1.1.1" lspace="0.278em" rspace="0.608em" xref="S2.E14X.2.1.1.m1.1.1.1.cmml">:</mo><munder id="S2.E14X.2.1.1.m1.1.1.3" xref="S2.E14X.2.1.1.m1.1.1.3.cmml"><mi id="S2.E14X.2.1.1.m1.1.1.3.2" xref="S2.E14X.2.1.1.m1.1.1.3.2.cmml">min</mi><mi id="S2.E14X.2.1.1.m1.1.1.3.3" xref="S2.E14X.2.1.1.m1.1.1.3.3.cmml">𝐦</mi></munder></mrow><annotation-xml encoding="MathML-Content" id="S2.E14X.2.1.1.m1.1b"><apply id="S2.E14X.2.1.1.m1.1.1.cmml" xref="S2.E14X.2.1.1.m1.1.1"><ci id="S2.E14X.2.1.1.m1.1.1.1.cmml" xref="S2.E14X.2.1.1.m1.1.1.1">:</ci><apply id="S2.E14X.2.1.1.m1.1.1.2.cmml" xref="S2.E14X.2.1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.E14X.2.1.1.m1.1.1.2.1.cmml" xref="S2.E14X.2.1.1.m1.1.1.2">subscript</csymbol><ci id="S2.E14X.2.1.1.m1.1.1.2.2.cmml" xref="S2.E14X.2.1.1.m1.1.1.2.2">𝒫</ci><cn id="S2.E14X.2.1.1.m1.1.1.2.3.cmml" type="integer" xref="S2.E14X.2.1.1.m1.1.1.2.3">1</cn></apply><apply id="S2.E14X.2.1.1.m1.1.1.3.cmml" xref="S2.E14X.2.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.E14X.2.1.1.m1.1.1.3.1.cmml" xref="S2.E14X.2.1.1.m1.1.1.3">subscript</csymbol><min id="S2.E14X.2.1.1.m1.1.1.3.2.cmml" xref="S2.E14X.2.1.1.m1.1.1.3.2"></min><ci id="S2.E14X.2.1.1.m1.1.1.3.3.cmml" xref="S2.E14X.2.1.1.m1.1.1.3.3">𝐦</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E14X.2.1.1.m1.1c">\displaystyle\mathcal{P}_{1}:~{}\min_{\mathbf{m}}</annotation><annotation encoding="application/x-llamapun" id="S2.E14X.2.1.1.m1.1d">caligraphic_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT : roman_min start_POSTSUBSCRIPT bold_m end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle~{}~{}\mathbb{E}_{\mathbf{h}}\left[\min_{\mathbf{a},\left\{b_{n}% \right\}}\textup{MSE}(\mathbf{a},\left\{b_{n}\right\})\right]" class="ltx_Math" display="inline" id="S2.E14X.3.2.2.m1.4"><semantics id="S2.E14X.3.2.2.m1.4a"><mrow id="S2.E14X.3.2.2.m1.4.4" xref="S2.E14X.3.2.2.m1.4.4.cmml"><msub id="S2.E14X.3.2.2.m1.4.4.3" xref="S2.E14X.3.2.2.m1.4.4.3.cmml"><mi id="S2.E14X.3.2.2.m1.4.4.3.2" xref="S2.E14X.3.2.2.m1.4.4.3.2.cmml">𝔼</mi><mi id="S2.E14X.3.2.2.m1.4.4.3.3" xref="S2.E14X.3.2.2.m1.4.4.3.3.cmml">𝐡</mi></msub><mo id="S2.E14X.3.2.2.m1.4.4.2" xref="S2.E14X.3.2.2.m1.4.4.2.cmml">⁢</mo><mrow id="S2.E14X.3.2.2.m1.4.4.1.1" xref="S2.E14X.3.2.2.m1.4.4.1.2.cmml"><mo id="S2.E14X.3.2.2.m1.4.4.1.1.2" xref="S2.E14X.3.2.2.m1.4.4.1.2.1.cmml">[</mo><mrow id="S2.E14X.3.2.2.m1.4.4.1.1.1" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.cmml"><mrow id="S2.E14X.3.2.2.m1.4.4.1.1.1.3" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.cmml"><munder id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1.cmml"><mi id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1.2" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1.2.cmml">min</mi><mrow id="S2.E14X.3.2.2.m1.2.2.2.2" xref="S2.E14X.3.2.2.m1.2.2.2.3.cmml"><mi id="S2.E14X.3.2.2.m1.1.1.1.1" xref="S2.E14X.3.2.2.m1.1.1.1.1.cmml">𝐚</mi><mo id="S2.E14X.3.2.2.m1.2.2.2.2.2" xref="S2.E14X.3.2.2.m1.2.2.2.3.cmml">,</mo><mrow id="S2.E14X.3.2.2.m1.2.2.2.2.1.1" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.2.cmml"><mo id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.2" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.2.cmml">{</mo><msub id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.cmml"><mi id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.2" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.2.cmml">b</mi><mi id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.3" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.3.cmml">n</mi></msub><mo id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.3" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.2.cmml">}</mo></mrow></mrow></munder><mo id="S2.E14X.3.2.2.m1.4.4.1.1.1.3a" lspace="0.167em" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.cmml">⁡</mo><mtext id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.2" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.2a.cmml">MSE</mtext></mrow><mo id="S2.E14X.3.2.2.m1.4.4.1.1.1.2" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.2.cmml">⁢</mo><mrow id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.2.cmml"><mo id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.2" stretchy="false" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.2.cmml">(</mo><mi id="S2.E14X.3.2.2.m1.3.3" xref="S2.E14X.3.2.2.m1.3.3.cmml">𝐚</mi><mo id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.3" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.2.cmml">,</mo><mrow id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.2.cmml"><mo id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.2" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.2.cmml">{</mo><msub id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.2" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.2.cmml">b</mi><mi id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.3" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.3" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.2.cmml">}</mo></mrow><mo id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.4" stretchy="false" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S2.E14X.3.2.2.m1.4.4.1.1.3" xref="S2.E14X.3.2.2.m1.4.4.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E14X.3.2.2.m1.4b"><apply id="S2.E14X.3.2.2.m1.4.4.cmml" xref="S2.E14X.3.2.2.m1.4.4"><times id="S2.E14X.3.2.2.m1.4.4.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.2"></times><apply id="S2.E14X.3.2.2.m1.4.4.3.cmml" xref="S2.E14X.3.2.2.m1.4.4.3"><csymbol cd="ambiguous" id="S2.E14X.3.2.2.m1.4.4.3.1.cmml" xref="S2.E14X.3.2.2.m1.4.4.3">subscript</csymbol><ci id="S2.E14X.3.2.2.m1.4.4.3.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.3.2">𝔼</ci><ci id="S2.E14X.3.2.2.m1.4.4.3.3.cmml" xref="S2.E14X.3.2.2.m1.4.4.3.3">𝐡</ci></apply><apply id="S2.E14X.3.2.2.m1.4.4.1.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1"><csymbol cd="latexml" id="S2.E14X.3.2.2.m1.4.4.1.2.1.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.2">delimited-[]</csymbol><apply id="S2.E14X.3.2.2.m1.4.4.1.1.1.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1"><times id="S2.E14X.3.2.2.m1.4.4.1.1.1.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.2"></times><apply id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3"><apply id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1"><csymbol cd="ambiguous" id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1.1.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1">subscript</csymbol><min id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.1.2"></min><list id="S2.E14X.3.2.2.m1.2.2.2.3.cmml" xref="S2.E14X.3.2.2.m1.2.2.2.2"><ci id="S2.E14X.3.2.2.m1.1.1.1.1.cmml" xref="S2.E14X.3.2.2.m1.1.1.1.1">𝐚</ci><set id="S2.E14X.3.2.2.m1.2.2.2.2.1.2.cmml" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.1"><apply id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.cmml" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.1.cmml" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1">subscript</csymbol><ci id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.2.cmml" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.2">𝑏</ci><ci id="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.3.cmml" xref="S2.E14X.3.2.2.m1.2.2.2.2.1.1.1.3">𝑛</ci></apply></set></list></apply><ci id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.2a.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.2"><mtext id="S2.E14X.3.2.2.m1.4.4.1.1.1.3.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.3.2">MSE</mtext></ci></apply><interval closure="open" id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1"><ci id="S2.E14X.3.2.2.m1.3.3.cmml" xref="S2.E14X.3.2.2.m1.3.3">𝐚</ci><set id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1"><apply id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.2">𝑏</ci><ci id="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E14X.3.2.2.m1.4.4.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></set></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E14X.3.2.2.m1.4c">\displaystyle~{}~{}\mathbb{E}_{\mathbf{h}}\left[\min_{\mathbf{a},\left\{b_{n}% \right\}}\textup{MSE}(\mathbf{a},\left\{b_{n}\right\})\right]</annotation><annotation encoding="application/x-llamapun" id="S2.E14X.3.2.2.m1.4d">blackboard_E start_POSTSUBSCRIPT bold_h end_POSTSUBSCRIPT [ roman_min start_POSTSUBSCRIPT bold_a , { italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } end_POSTSUBSCRIPT MSE ( bold_a , { italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } ) ]</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="4">(14)</td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E14Xa"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_align_right ltx_eqn_cell">s.t.</td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle~{}~{}e_{n}m_{n}s^{\textup{tot}}+L_{0}\|b_{n}\|^{2}\leq P_{n}^{% \textup{max}},\forall n," class="ltx_Math" display="inline" id="S2.E14Xa.3.2.2.m1.1"><semantics id="S2.E14Xa.3.2.2.m1.1a"><mrow id="S2.E14Xa.3.2.2.m1.1.1.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.cmml"><mrow id="S2.E14Xa.3.2.2.m1.1.1.1.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.cmml"><mrow id="S2.E14Xa.3.2.2.m1.1.1.1.1.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.cmml"><mrow id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.cmml"><msub id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.cmml"><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.2.cmml">e</mi><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.3.cmml">n</mi></msub><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.1.cmml">⁢</mo><msub id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.cmml"><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.2.cmml">m</mi><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.3.cmml">n</mi></msub><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.1a" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.1.cmml">⁢</mo><msup id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.cmml"><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.2.cmml">s</mi><mtext id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.3a.cmml">tot</mtext></msup></mrow><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.2.cmml">+</mo><mrow id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.cmml"><msub id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.2.cmml">L</mi><mn id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.3.cmml">0</mn></msub><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.2.cmml">⁢</mo><msup id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.cmml"><mrow id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.2.cmml"><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.2" stretchy="false" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.2.1.cmml">‖</mo><msub id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.2.cmml">b</mi><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.3" stretchy="false" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.3.cmml">2</mn></msup></mrow></mrow><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.4" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.4.cmml">≤</mo><mrow id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.3.cmml"><msubsup id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.cmml"><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.2.cmml">P</mi><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.3.cmml">n</mi><mtext id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.3a.cmml">max</mtext></msubsup><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.3" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.3.cmml">,</mo><mrow id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.cmml"><mo id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.1" rspace="0.167em" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.1.cmml">∀</mo><mi id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.2.cmml">n</mi></mrow></mrow></mrow><mo id="S2.E14Xa.3.2.2.m1.1.1.1.2" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E14Xa.3.2.2.m1.1b"><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1"><leq id="S2.E14Xa.3.2.2.m1.1.1.1.1.4.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.4"></leq><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1"><plus id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.2"></plus><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3"><times id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.1"></times><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.2">𝑒</ci><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.3.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.2.3">𝑛</ci></apply><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.2">𝑚</ci><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.3.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.3.3">𝑛</ci></apply><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4">superscript</csymbol><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.2">𝑠</ci><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.3a.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.3"><mtext id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.3.cmml" mathsize="70%" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.3.4.3">tot</mtext></ci></apply></apply><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1"><times id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.2"></times><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.2">𝐿</ci><cn id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.3.cmml" type="integer" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.3.3">0</cn></apply><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.2">norm</csymbol><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.2">𝑏</ci><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><cn id="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.3.cmml" type="integer" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.1.1.1.3">2</cn></apply></apply></apply><list id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.3.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2"><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1">superscript</csymbol><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1">subscript</csymbol><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.2">𝑃</ci><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.3.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.2.3">𝑛</ci></apply><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.3a.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.3"><mtext id="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.3.cmml" mathsize="70%" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.2.1.1.3">max</mtext></ci></apply><apply id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2"><csymbol cd="latexml" id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.1.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.1">for-all</csymbol><ci id="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.2.cmml" xref="S2.E14Xa.3.2.2.m1.1.1.1.1.3.2.2.2">𝑛</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E14Xa.3.2.2.m1.1c">\displaystyle~{}~{}e_{n}m_{n}s^{\textup{tot}}+L_{0}\|b_{n}\|^{2}\leq P_{n}^{% \textup{max}},\forall n,</annotation><annotation encoding="application/x-llamapun" id="S2.E14Xa.3.2.2.m1.1d">italic_e start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_m start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT italic_s start_POSTSUPERSCRIPT tot end_POSTSUPERSCRIPT + italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥ italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ∥ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ≤ italic_P start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT max end_POSTSUPERSCRIPT , ∀ italic_n ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E14Xb"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle~{}~{}\sum_{n=1}^{N}m_{n}=1," class="ltx_Math" display="inline" id="S2.E14Xb.2.1.1.m1.1"><semantics id="S2.E14Xb.2.1.1.m1.1a"><mrow id="S2.E14Xb.2.1.1.m1.1.1.1" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E14Xb.2.1.1.m1.1.1.1.1" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.cmml"><mrow id="S2.E14Xb.2.1.1.m1.1.1.1.1.2" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.cmml"><mstyle displaystyle="true" id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.cmml"><munderover id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1a" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.cmml"><mo id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.2" movablelimits="false" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.2.cmml">∑</mo><mrow id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.cmml"><mi id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.2" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.2.cmml">n</mi><mo id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.1" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.1.cmml">=</mo><mn id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.3" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.3" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.3.cmml">N</mi></munderover></mstyle><msub id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.cmml"><mi id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.2" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.2.cmml">m</mi><mi id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.3" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.3.cmml">n</mi></msub></mrow><mo id="S2.E14Xb.2.1.1.m1.1.1.1.1.1" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.1.cmml">=</mo><mn id="S2.E14Xb.2.1.1.m1.1.1.1.1.3" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.3.cmml">1</mn></mrow><mo id="S2.E14Xb.2.1.1.m1.1.1.1.2" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E14Xb.2.1.1.m1.1b"><apply id="S2.E14Xb.2.1.1.m1.1.1.1.1.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1"><eq id="S2.E14Xb.2.1.1.m1.1.1.1.1.1.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.1"></eq><apply id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2"><apply id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1"><csymbol cd="ambiguous" id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.1.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1">superscript</csymbol><apply id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1"><csymbol cd="ambiguous" id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.1.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1">subscript</csymbol><sum id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.2.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.2"></sum><apply id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3"><eq id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.1.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.1"></eq><ci id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.2.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.2">𝑛</ci><cn id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.3.cmml" type="integer" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.2.3.3">1</cn></apply></apply><ci id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.3.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.1.3">𝑁</ci></apply><apply id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.1.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2">subscript</csymbol><ci id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.2.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.2">𝑚</ci><ci id="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.3.cmml" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.2.2.3">𝑛</ci></apply></apply><cn id="S2.E14Xb.2.1.1.m1.1.1.1.1.3.cmml" type="integer" xref="S2.E14Xb.2.1.1.m1.1.1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E14Xb.2.1.1.m1.1c">\displaystyle~{}~{}\sum_{n=1}^{N}m_{n}=1,</annotation><annotation encoding="application/x-llamapun" id="S2.E14Xb.2.1.1.m1.1d">∑ start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT italic_m start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT = 1 ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S2.E14Xc"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_td ltx_eqn_cell"></td> <td class="ltx_td ltx_align_left ltx_eqn_cell"><math alttext="\displaystyle~{}~{}0\leq m_{n}\leq 1,\forall n," class="ltx_Math" display="inline" id="S2.E14Xc.2.1.1.m1.1"><semantics id="S2.E14Xc.2.1.1.m1.1a"><mrow id="S2.E14Xc.2.1.1.m1.1.1.1"><mrow id="S2.E14Xc.2.1.1.m1.1.1.1.1.2" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.3.cmml"><mrow id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.cmml"><mn id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.2" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.2.cmml">0</mn><mo id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.3" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.3.cmml">≤</mo><msub id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.cmml"><mi id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.2" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.2.cmml">m</mi><mi id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.3" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.3.cmml">n</mi></msub><mo id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.5" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.5.cmml">≤</mo><mn id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.6" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.6.cmml">1</mn></mrow><mo id="S2.E14Xc.2.1.1.m1.1.1.1.1.2.3" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.3a.cmml">,</mo><mrow id="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.cmml"><mo id="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.1" rspace="0.167em" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.1.cmml">∀</mo><mi id="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.2" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.2.cmml">n</mi></mrow></mrow><mo id="S2.E14Xc.2.1.1.m1.1.1.1.2">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E14Xc.2.1.1.m1.1b"><apply id="S2.E14Xc.2.1.1.m1.1.1.1.1.3.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E14Xc.2.1.1.m1.1.1.1.1.3a.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1"><and id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1a.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1"></and><apply id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1b.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1"><leq id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.3"></leq><cn id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.2.cmml" type="integer" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.2">0</cn><apply id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.1.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.2.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.2">𝑚</ci><ci id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.3.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.3">𝑛</ci></apply></apply><apply id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1c.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1"><leq id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.5.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.5"></leq><share href="https://arxiv.org/html/2503.14882v1#S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.4.cmml" id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1d.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1"></share><cn id="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.6.cmml" type="integer" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.1.1.6">1</cn></apply></apply><apply id="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2"><csymbol cd="latexml" id="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.1.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.1">for-all</csymbol><ci id="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.2.cmml" xref="S2.E14Xc.2.1.1.m1.1.1.1.1.2.2.2">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E14Xc.2.1.1.m1.1c">\displaystyle~{}~{}0\leq m_{n}\leq 1,\forall n,</annotation><annotation encoding="application/x-llamapun" id="S2.E14Xc.2.1.1.m1.1d">0 ≤ italic_m start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ≤ 1 , ∀ italic_n ,</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr> </tbody> </table> where the expectation <math alttext="\mathbb{E}_{\mathbf{h}}\left[\cdot\right]" class="ltx_Math" display="inline" id="S2.SS4.p2.4.m1.1"><semantics id="S2.SS4.p2.4.m1.1a"><mrow id="S2.SS4.p2.4.m1.1.2" xref="S2.SS4.p2.4.m1.1.2.cmml"><msub id="S2.SS4.p2.4.m1.1.2.2" xref="S2.SS4.p2.4.m1.1.2.2.cmml"><mi id="S2.SS4.p2.4.m1.1.2.2.2" xref="S2.SS4.p2.4.m1.1.2.2.2.cmml">𝔼</mi><mi id="S2.SS4.p2.4.m1.1.2.2.3" xref="S2.SS4.p2.4.m1.1.2.2.3.cmml">𝐡</mi></msub><mo id="S2.SS4.p2.4.m1.1.2.1" xref="S2.SS4.p2.4.m1.1.2.1.cmml">⁢</mo><mrow id="S2.SS4.p2.4.m1.1.2.3.2" xref="S2.SS4.p2.4.m1.1.2.3.1.cmml"><mo id="S2.SS4.p2.4.m1.1.2.3.2.1" xref="S2.SS4.p2.4.m1.1.2.3.1.1.cmml">[</mo><mo id="S2.SS4.p2.4.m1.1.1" lspace="0em" rspace="0em" xref="S2.SS4.p2.4.m1.1.1.cmml">⋅</mo><mo id="S2.SS4.p2.4.m1.1.2.3.2.2" xref="S2.SS4.p2.4.m1.1.2.3.1.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.4.m1.1b"><apply id="S2.SS4.p2.4.m1.1.2.cmml" xref="S2.SS4.p2.4.m1.1.2"><times id="S2.SS4.p2.4.m1.1.2.1.cmml" xref="S2.SS4.p2.4.m1.1.2.1"></times><apply id="S2.SS4.p2.4.m1.1.2.2.cmml" xref="S2.SS4.p2.4.m1.1.2.2"><csymbol cd="ambiguous" id="S2.SS4.p2.4.m1.1.2.2.1.cmml" xref="S2.SS4.p2.4.m1.1.2.2">subscript</csymbol><ci id="S2.SS4.p2.4.m1.1.2.2.2.cmml" xref="S2.SS4.p2.4.m1.1.2.2.2">𝔼</ci><ci id="S2.SS4.p2.4.m1.1.2.2.3.cmml" xref="S2.SS4.p2.4.m1.1.2.2.3">𝐡</ci></apply><apply id="S2.SS4.p2.4.m1.1.2.3.1.cmml" xref="S2.SS4.p2.4.m1.1.2.3.2"><csymbol cd="latexml" id="S2.SS4.p2.4.m1.1.2.3.1.1.cmml" xref="S2.SS4.p2.4.m1.1.2.3.2.1">delimited-[]</csymbol><ci id="S2.SS4.p2.4.m1.1.1.cmml" xref="S2.SS4.p2.4.m1.1.1">⋅</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.4.m1.1c">\mathbb{E}_{\mathbf{h}}\left[\cdot\right]</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p2.4.m1.1d">blackboard_E start_POSTSUBSCRIPT bold_h end_POSTSUBSCRIPT [ ⋅ ]</annotation></semantics></math> is taken over all random channel realizations <math alttext="\mathbf{h}=\left\{\mathbf{h}_{n}\right\}_{n=1}^{N}" class="ltx_Math" display="inline" id="S2.SS4.p2.5.m2.1"><semantics id="S2.SS4.p2.5.m2.1a"><mrow id="S2.SS4.p2.5.m2.1.1" xref="S2.SS4.p2.5.m2.1.1.cmml"><mi id="S2.SS4.p2.5.m2.1.1.3" xref="S2.SS4.p2.5.m2.1.1.3.cmml">𝐡</mi><mo id="S2.SS4.p2.5.m2.1.1.2" xref="S2.SS4.p2.5.m2.1.1.2.cmml">=</mo><msubsup id="S2.SS4.p2.5.m2.1.1.1" xref="S2.SS4.p2.5.m2.1.1.1.cmml"><mrow id="S2.SS4.p2.5.m2.1.1.1.1.1.1" xref="S2.SS4.p2.5.m2.1.1.1.1.1.2.cmml"><mo id="S2.SS4.p2.5.m2.1.1.1.1.1.1.2" xref="S2.SS4.p2.5.m2.1.1.1.1.1.2.cmml">{</mo><msub id="S2.SS4.p2.5.m2.1.1.1.1.1.1.1" xref="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.cmml"><mi id="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.2" xref="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.2.cmml">𝐡</mi><mi id="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.3" xref="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.SS4.p2.5.m2.1.1.1.1.1.1.3" xref="S2.SS4.p2.5.m2.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS4.p2.5.m2.1.1.1.1.3" xref="S2.SS4.p2.5.m2.1.1.1.1.3.cmml"><mi id="S2.SS4.p2.5.m2.1.1.1.1.3.2" xref="S2.SS4.p2.5.m2.1.1.1.1.3.2.cmml">n</mi><mo id="S2.SS4.p2.5.m2.1.1.1.1.3.1" xref="S2.SS4.p2.5.m2.1.1.1.1.3.1.cmml">=</mo><mn id="S2.SS4.p2.5.m2.1.1.1.1.3.3" xref="S2.SS4.p2.5.m2.1.1.1.1.3.3.cmml">1</mn></mrow><mi id="S2.SS4.p2.5.m2.1.1.1.3" xref="S2.SS4.p2.5.m2.1.1.1.3.cmml">N</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.5.m2.1b"><apply id="S2.SS4.p2.5.m2.1.1.cmml" xref="S2.SS4.p2.5.m2.1.1"><eq id="S2.SS4.p2.5.m2.1.1.2.cmml" xref="S2.SS4.p2.5.m2.1.1.2"></eq><ci id="S2.SS4.p2.5.m2.1.1.3.cmml" xref="S2.SS4.p2.5.m2.1.1.3">𝐡</ci><apply id="S2.SS4.p2.5.m2.1.1.1.cmml" xref="S2.SS4.p2.5.m2.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.5.m2.1.1.1.2.cmml" xref="S2.SS4.p2.5.m2.1.1.1">superscript</csymbol><apply id="S2.SS4.p2.5.m2.1.1.1.1.cmml" xref="S2.SS4.p2.5.m2.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.5.m2.1.1.1.1.2.cmml" xref="S2.SS4.p2.5.m2.1.1.1">subscript</csymbol><set id="S2.SS4.p2.5.m2.1.1.1.1.1.2.cmml" xref="S2.SS4.p2.5.m2.1.1.1.1.1.1"><apply id="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS4.p2.5.m2.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.1.cmml" xref="S2.SS4.p2.5.m2.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.2.cmml" xref="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.2">𝐡</ci><ci id="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.3.cmml" xref="S2.SS4.p2.5.m2.1.1.1.1.1.1.1.3">𝑛</ci></apply></set><apply id="S2.SS4.p2.5.m2.1.1.1.1.3.cmml" xref="S2.SS4.p2.5.m2.1.1.1.1.3"><eq id="S2.SS4.p2.5.m2.1.1.1.1.3.1.cmml" xref="S2.SS4.p2.5.m2.1.1.1.1.3.1"></eq><ci id="S2.SS4.p2.5.m2.1.1.1.1.3.2.cmml" xref="S2.SS4.p2.5.m2.1.1.1.1.3.2">𝑛</ci><cn id="S2.SS4.p2.5.m2.1.1.1.1.3.3.cmml" type="integer" xref="S2.SS4.p2.5.m2.1.1.1.1.3.3">1</cn></apply></apply><ci id="S2.SS4.p2.5.m2.1.1.1.3.cmml" xref="S2.SS4.p2.5.m2.1.1.1.3">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.5.m2.1c">\mathbf{h}=\left\{\mathbf{h}_{n}\right\}_{n=1}^{N}</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p2.5.m2.1d">bold_h = { bold_h start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } start_POSTSUBSCRIPT italic_n = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT</annotation></semantics></math>. However, the problem <math alttext="\mathcal{P}_{1}" class="ltx_Math" display="inline" id="S2.SS4.p2.6.m3.1"><semantics id="S2.SS4.p2.6.m3.1a"><msub id="S2.SS4.p2.6.m3.1.1" xref="S2.SS4.p2.6.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.SS4.p2.6.m3.1.1.2" xref="S2.SS4.p2.6.m3.1.1.2.cmml">𝒫</mi><mn id="S2.SS4.p2.6.m3.1.1.3" xref="S2.SS4.p2.6.m3.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.6.m3.1b"><apply id="S2.SS4.p2.6.m3.1.1.cmml" xref="S2.SS4.p2.6.m3.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.6.m3.1.1.1.cmml" xref="S2.SS4.p2.6.m3.1.1">subscript</csymbol><ci id="S2.SS4.p2.6.m3.1.1.2.cmml" xref="S2.SS4.p2.6.m3.1.1.2">𝒫</ci><cn id="S2.SS4.p2.6.m3.1.1.3.cmml" type="integer" xref="S2.SS4.p2.6.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.6.m3.1c">\mathcal{P}_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.SS4.p2.6.m3.1d">caligraphic_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> is challenging to be solved due to the following three reasons. <ul class="ltx_itemize" id="S2.I2"> <li class="ltx_item" id="S2.I2.i1" style="list-style-type:none;"> • <div class="ltx_para" id="S2.I2.i1.p1"> Non-convexity: The objective function is inherently non-convex due to the coupling between the receiver aggregation beamformer <math alttext="\mathbf{a}" class="ltx_Math" display="inline" id="S2.I2.i1.p1.1.m1.1"><semantics id="S2.I2.i1.p1.1.m1.1a"><mi id="S2.I2.i1.p1.1.m1.1.1" xref="S2.I2.i1.p1.1.m1.1.1.cmml">𝐚</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.1.m1.1b"><ci id="S2.I2.i1.p1.1.m1.1.1.cmml" xref="S2.I2.i1.p1.1.m1.1.1">𝐚</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.1.m1.1c">\mathbf{a}</annotation><annotation encoding="application/x-llamapun" id="S2.I2.i1.p1.1.m1.1d">bold_a</annotation></semantics></math> and the transmitter scalars <math alttext="\left\{b_{n}\right\}" class="ltx_Math" display="inline" id="S2.I2.i1.p1.2.m2.1"><semantics id="S2.I2.i1.p1.2.m2.1a"><mrow id="S2.I2.i1.p1.2.m2.1.1.1" xref="S2.I2.i1.p1.2.m2.1.1.2.cmml"><mo id="S2.I2.i1.p1.2.m2.1.1.1.2" xref="S2.I2.i1.p1.2.m2.1.1.2.cmml">{</mo><msub id="S2.I2.i1.p1.2.m2.1.1.1.1" xref="S2.I2.i1.p1.2.m2.1.1.1.1.cmml"><mi id="S2.I2.i1.p1.2.m2.1.1.1.1.2" xref="S2.I2.i1.p1.2.m2.1.1.1.1.2.cmml">b</mi><mi id="S2.I2.i1.p1.2.m2.1.1.1.1.3" xref="S2.I2.i1.p1.2.m2.1.1.1.1.3.cmml">n</mi></msub><mo id="S2.I2.i1.p1.2.m2.1.1.1.3" xref="S2.I2.i1.p1.2.m2.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.2.m2.1b"><set id="S2.I2.i1.p1.2.m2.1.1.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1"><apply id="S2.I2.i1.p1.2.m2.1.1.1.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.I2.i1.p1.2.m2.1.1.1.1.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1.1">subscript</csymbol><ci id="S2.I2.i1.p1.2.m2.1.1.1.1.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1.1.2">𝑏</ci><ci id="S2.I2.i1.p1.2.m2.1.1.1.1.3.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1.1.3">𝑛</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.2.m2.1c">\left\{b_{n}\right\}</annotation><annotation encoding="application/x-llamapun" id="S2.I2.i1.p1.2.m2.1d">{ italic_b start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT }</annotation></semantics></math>. </div> </li> <li class="ltx_item" id="S2.I2.i2" style="list-style-type:none;"> • <div class="ltx_para" id="S2.I2.i2.p1"> Expectation over Random Channels: The objective involves an expectation over random CSI, which requires prior knowledge of channel statistics. </div> </li> <li class="ltx_item" id="S2.I2.i3" style="list-style-type:none;"> • <div class="ltx_para" id="S2.I2.i3.p1"> Interdependence of Timescales: The per-device power constraints link the short-term transceiver variables with the long-term model assignment policy, leading to a complex interplay between the two timescales. </div> </li> </ul> To address these challenges, we develop a two-stage algorithm that separately solves the short-term transceiver optimization and the long-term model assignment optimization in the following section. </div> </section> </section> <section class="ltx_section" id="S3"> <h2 class="ltx_title ltx_title_section"> III Algorithm Development </h2> <div class="ltx_para" id="S3.p1"> In this section, we develop an efficient two-stage algorithm to solve the joint model assignment and transceiver optimization problem <math alttext="\mathcal{P}_{1}" class="ltx_Math" display="inline" id="S3.p1.1.m1.1"><semantics id="S3.p1.1.m1.1a"><msub id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.1.m1.1.1.2" xref="S3.p1.1.m1.1.1.2.cmml">𝒫</mi><mn id="S3.p1.1.m1.1.1.3" xref="S3.p1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><apply id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.p1.1.m1.1.1.1.cmml" xref="S3.p1.1.m1.1.1">subscript</csymbol><ci id="S3.p1.1.m1.1.1.2.cmml" xref="S3.p1.1.m1.1.1.2">𝒫</ci><cn id="S3.p1.1.m1.1.1.3.cmml" type="integer" xref="S3.p1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">\mathcal{P}_{1}</annotation><annotation encoding="application/x-llamapun" id="S3.p1.1.m1.1d">caligraphic_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>. Then, we show that the proposed algorithm can converge to a stationary point of the original problem <math alttext="\mathcal{P}_{1}" class="ltx_Math" display="inline" id="S3.p1.2.m2.1"><semantics id="S3.p1.2.m2.1a"><msub id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.2.m2.1.1.2" xref="S3.p1.2.m2.1.1.2.cmml">𝒫</mi><mn id="S3.p1.2.m2.1.1.3" xref="S3.p1.2.m2.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><apply id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.cmml" xref="S3.p1.2.m2.1.1">subscript</csymbol><ci id="S3.p1.2.m2.1.1.2.cmml" xref="S3.p1.2.m2.1.1.2">𝒫</ci><cn id="S3.p1.2.m2.1.1.3.cmml" type="integer" xref="S3.p1.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">\mathcal{P}_{1}</annotation><annotation encoding="application/x-llamapun" id="S3.p1.2.m2.1d">caligraphic_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>. </div> <section class="ltx_subsection" id="S3.SS1"> <h3 class="ltx_title ltx_title_subsection"> III-A Problem Decomposition </h3> <div class="ltx_para" id="S3.SS1.p1"> We start by decomposing problem <math alttext="\mathcal{P}_{1}" class="ltx_Math" display="inline" id="S3.SS1.p1.1.m1.1"><semantics id="S3.SS1.p1.1.m1.1a"><msub id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.1.m1.1.1.2" xref="S3.SS1.p1.1.m1.1.1.2.cmml">𝒫</mi><mn id="S3.SS1.p1.1.m1.1.1.3" xref="S3.SS1.p1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><apply id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p1.1.m1.1.1.2.cmml" xref="S3.SS1.p1.1.m1.1.1.2">𝒫</ci><cn id="S3.SS1.p1.1.m1.1.1.3.cmml" type="integer" xref="S3.SS1.p1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">\mathcal{P}_{1}</annotation><annotation encoding="application/x-llamapun" id="S3.SS1.p1.1.m1.1d">caligraphic_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> into a family of short-term transceiver optimization problems and a long-term model assignment optimization problem as follows. </div> <section class="ltx_subsubsection" id="S3.SS1.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> III-A1 Short-term transceiver optimization for given model assignment policy <math alttext="\mathbf{m}" class="ltx_Math" display="inline" id="S3.SS1.SSS1.1.m1.1"><semantics id="S3.SS1.SSS1.1.m1.1b"><mi id="S3.SS1.SSS1.1.m1.1.1" xref="S3.SS1.SSS1.1.m1.1.1.cmml">𝐦</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.1.m1.1c"><ci id="S3.SS1.SSS1.1.m1.1.1.cmml" xref="S3.SS1.SSS1.1.m1.1.1">𝐦</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.1.m1.1d">\mathbf{m}</annotation><annotation encoding="application/x-llamapun" id="S3.SS1.SSS1.1.m1.1e">bold_m</annotation></semantics></math> and channel condition <math alttext="\mathbf{h}" class="ltx_Math" display="inline" id="S3.SS1.SSS1.2.m2.1"><semantics id="S3.SS1.SSS1.2.m2.1b"><mi id="S3.SS1.SSS1.2.m2.1.1" xref="S3.SS1.SSS1.2.m2.1.1.cmml">𝐡</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.2.m2.1c"><ci id="S3.SS1.SSS1.2.m2.1.1.cmml" xref="S3.SS1.SSS1.2.m2.1.1">𝐡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.2.m2.1d">\mathbf{h}</annotation><annotation encoding="application/x-llamapun" id="S3.SS1.SSS1.2.m2.1e">bold_h</annotation></semantics></math> </h4> <div class="ltx_para" id="S3.SS1.SSS1.p1"> <table class="ltx_equationgroup ltx_eqn_table" id="S3.E15"> <tbody> <tr class="ltx_equation ltx_eqn_row ltx_align_baseline" id="S3.E15X"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>