CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>Pairwise Judgment Formulation for Semantic Embedding Model in Web Search</title>  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <meta content="Semantic Embedding Model, Search Engine, Pairwise Judgment" lang="en" name="keywords"/> <base href="/html/2408.04197v2/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S1" title="In Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S2" title="In Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>Related Work</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3" title="In Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Semantic Embedding Model for Web Search</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S4" title="In Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Experimental Setup</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S5" title="In Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Atomic Strategies</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S6" title="In Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>Hybrid Strategy</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S7" title="In Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">7 </span>Conclusions</span></a></li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line ltx_leqno"> <h1 class="ltx_title ltx_title_document">Pairwise Judgment Formulation for Semantic Embedding Model in Web Search</h1> <div class="ltx_authors"> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Mengze Hong </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_orcid"><a class="ltx_ref" href="https://orcid.org/0009-0003-3188-4208" title="ORCID identifier">0009-0003-3188-4208</a></span> <span class="ltx_contact ltx_role_affiliation"><span class="ltx_text ltx_affiliation_institution" id="id1.1.id1">Hong Kong Polytechnic University</span><span class="ltx_text ltx_affiliation_city" id="id2.2.id2">Hong Kong</span><span class="ltx_text ltx_affiliation_country" id="id3.3.id3"></span> </span> <span class="ltx_contact ltx_role_email"><a href="mailto:mengze.hong@connect.polyu.hk">mengze.hong@connect.polyu.hk</a> </span></span></span> <span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Wailing Ng </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"><span class="ltx_text ltx_affiliation_institution" id="id4.1.id1">Hong Kong Polytechnic University</span><span class="ltx_text ltx_affiliation_city" id="id5.2.id2">Hong Kong</span><span class="ltx_text ltx_affiliation_country" id="id6.3.id3"></span> </span> <span class="ltx_contact ltx_role_email"><a href="mailto:wai-ling.ng@connect.polyu.hk">wai-ling.ng@connect.polyu.hk</a> </span></span></span> <span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Zichang Guo </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"><span class="ltx_text ltx_affiliation_institution" id="id7.1.id1">Hong Kong Polytechnic University</span><span class="ltx_text ltx_affiliation_city" id="id8.2.id2">Hong Kong</span><span class="ltx_text ltx_affiliation_country" id="id9.3.id3"></span> </span> <span class="ltx_contact ltx_role_email"><a href="mailto:zichang.guo@connect.polyu.hk">zichang.guo@connect.polyu.hk</a> </span></span></span> <span class="ltx_author_before"> and </span><span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Chen Jason Zhang </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_orcid"><a class="ltx_ref" href="https://orcid.org/0000-0002-3306-9317" title="ORCID identifier">0000-0002-3306-9317</a></span> <span class="ltx_contact ltx_role_affiliation"><span class="ltx_text ltx_affiliation_institution" id="id10.1.id1">Hong Kong Polytechnic University</span><span class="ltx_text ltx_affiliation_city" id="id11.2.id2">Hong Kong</span><span class="ltx_text ltx_affiliation_country" id="id12.3.id3"></span> </span> <span class="ltx_contact ltx_role_email"><a href="mailto:jason-c.zhang@polyu.edu.hk">jason-c.zhang@polyu.edu.hk</a> </span></span></span> </div> <div class="ltx_dates">(2018)</div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract.</h6> <p class="ltx_p" id="id13.id1">Semantic Embedding Model (SEM), a neural network-based Siamese architecture, is gaining momentum in information retrieval and natural language processing. In order to train SEM in a supervised fashion for Web search, the search engine query log is typically utilized to automatically formulate pairwise judgments as training data. Despite the growing application of semantic embeddings in the search engine industry, little work has been done on formulating effective pairwise judgments for training SEM. In this paper, we make the first in-depth investigation of a wide range of strategies for generating pairwise judgments for SEM. An interesting (perhaps surprising) discovery reveals that the conventional pairwise judgment formulation strategy wildly used in the field of pairwise Learning-to-Rank (LTR) is not necessarily effective for training SEM. Through a large-scale empirical study based on query logs and click-through activities from a major commercial search engine, we demonstrate the effective strategies for SEM and highlight the advantages of a hybrid heuristic (i.e., Clicked ¿ Non-Clicked) in comparison to the atomic heuristics (e.g., Clicked ¿ Skipped) in LTR. We conclude with best practices for training SEM and offer promising insights for future research.</p> </div> <div class="ltx_keywords">Semantic Embedding Model, Search Engine, Pairwise Judgment </div> <span class="ltx_note ltx_note_frontmatter ltx_role_copyright" id="id1"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">copyright: </span>acmlicensed</span></span></span><span class="ltx_note ltx_note_frontmatter ltx_role_journalyear" id="id2"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">journalyear: </span>2018</span></span></span><span class="ltx_note ltx_note_frontmatter ltx_role_doi" id="id3"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">doi: </span>XXXXXXX.XXXXXXX</span></span></span><span class="ltx_note ltx_note_frontmatter ltx_role_conference" id="id4"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">conference: </span>Preprint; November 2024; Hong Kong</span></span></span><span class="ltx_note ltx_note_frontmatter ltx_role_ccs" id="id5"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">ccs: </span>Information systems Web searching and information discovery</span></span></span><span class="ltx_note ltx_note_frontmatter ltx_role_ccs" id="id6"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">ccs: </span>Information systems Retrieval models and ranking</span></span></span> <section class="ltx_section" id="S1"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">1. </span>Introduction</h2> <div class="ltx_para" id="S1.p1"> <p class="ltx_p" id="S1.p1.9">With the increasing awareness of the importance of latent semantics within text <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib13" title="">2016b</a>; Leung et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib20" title="">2016</a>; Li et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib22" title="">2023</a>)</cite>, the Semantic Embedding Model (SEM) attracts lots of attention from the information retrieval and natural language processing <cite class="ltx_cite ltx_citemacro_citep">(Collobert et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib5" title="">2011</a>; Wu et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib27" title="">2014</a>)</cite>. For the purpose of Web information retrieval, SEM is trained based on the search engine query log, which consists of queries, search results, and the record of a variety of user activities <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib14" title="">2016c</a>)</cite>. In practice, the queries and search results (i.e., the titles of the retrieved Web pages) are utilized to compose pairwise training instances. Formally, for a query <math alttext="q" class="ltx_Math" display="inline" id="S1.p1.1.m1.1"><semantics id="S1.p1.1.m1.1a"><mi id="S1.p1.1.m1.1.1" xref="S1.p1.1.m1.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="S1.p1.1.m1.1b"><ci id="S1.p1.1.m1.1.1.cmml" xref="S1.p1.1.m1.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.1.m1.1c">q</annotation><annotation encoding="application/x-llamapun" id="S1.p1.1.m1.1d">italic_q</annotation></semantics></math>, if a title <math alttext="p_{q}" class="ltx_Math" display="inline" id="S1.p1.2.m2.1"><semantics id="S1.p1.2.m2.1a"><msub id="S1.p1.2.m2.1.1" xref="S1.p1.2.m2.1.1.cmml"><mi id="S1.p1.2.m2.1.1.2" xref="S1.p1.2.m2.1.1.2.cmml">p</mi><mi id="S1.p1.2.m2.1.1.3" xref="S1.p1.2.m2.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p1.2.m2.1b"><apply id="S1.p1.2.m2.1.1.cmml" xref="S1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S1.p1.2.m2.1.1.1.cmml" xref="S1.p1.2.m2.1.1">subscript</csymbol><ci id="S1.p1.2.m2.1.1.2.cmml" xref="S1.p1.2.m2.1.1.2">𝑝</ci><ci id="S1.p1.2.m2.1.1.3.cmml" xref="S1.p1.2.m2.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.2.m2.1c">p_{q}</annotation><annotation encoding="application/x-llamapun" id="S1.p1.2.m2.1d">italic_p start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT</annotation></semantics></math> is preferred to a title <math alttext="n_{q}" class="ltx_Math" display="inline" id="S1.p1.3.m3.1"><semantics id="S1.p1.3.m3.1a"><msub id="S1.p1.3.m3.1.1" xref="S1.p1.3.m3.1.1.cmml"><mi id="S1.p1.3.m3.1.1.2" xref="S1.p1.3.m3.1.1.2.cmml">n</mi><mi id="S1.p1.3.m3.1.1.3" xref="S1.p1.3.m3.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p1.3.m3.1b"><apply id="S1.p1.3.m3.1.1.cmml" xref="S1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S1.p1.3.m3.1.1.1.cmml" xref="S1.p1.3.m3.1.1">subscript</csymbol><ci id="S1.p1.3.m3.1.1.2.cmml" xref="S1.p1.3.m3.1.1.2">𝑛</ci><ci id="S1.p1.3.m3.1.1.3.cmml" xref="S1.p1.3.m3.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.3.m3.1c">n_{q}</annotation><annotation encoding="application/x-llamapun" id="S1.p1.3.m3.1d">italic_n start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT</annotation></semantics></math> by the user, they are formulated as a pairwise judgment <math alttext="p_{q}" class="ltx_Math" display="inline" id="S1.p1.4.m4.1"><semantics id="S1.p1.4.m4.1a"><msub id="S1.p1.4.m4.1.1" xref="S1.p1.4.m4.1.1.cmml"><mi id="S1.p1.4.m4.1.1.2" xref="S1.p1.4.m4.1.1.2.cmml">p</mi><mi id="S1.p1.4.m4.1.1.3" xref="S1.p1.4.m4.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p1.4.m4.1b"><apply id="S1.p1.4.m4.1.1.cmml" xref="S1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S1.p1.4.m4.1.1.1.cmml" xref="S1.p1.4.m4.1.1">subscript</csymbol><ci id="S1.p1.4.m4.1.1.2.cmml" xref="S1.p1.4.m4.1.1.2">𝑝</ci><ci id="S1.p1.4.m4.1.1.3.cmml" xref="S1.p1.4.m4.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.4.m4.1c">p_{q}</annotation><annotation encoding="application/x-llamapun" id="S1.p1.4.m4.1d">italic_p start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT</annotation></semantics></math> ¿ <math alttext="n_{q}" class="ltx_Math" display="inline" id="S1.p1.5.m5.1"><semantics id="S1.p1.5.m5.1a"><msub id="S1.p1.5.m5.1.1" xref="S1.p1.5.m5.1.1.cmml"><mi id="S1.p1.5.m5.1.1.2" xref="S1.p1.5.m5.1.1.2.cmml">n</mi><mi id="S1.p1.5.m5.1.1.3" xref="S1.p1.5.m5.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p1.5.m5.1b"><apply id="S1.p1.5.m5.1.1.cmml" xref="S1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S1.p1.5.m5.1.1.1.cmml" xref="S1.p1.5.m5.1.1">subscript</csymbol><ci id="S1.p1.5.m5.1.1.2.cmml" xref="S1.p1.5.m5.1.1.2">𝑛</ci><ci id="S1.p1.5.m5.1.1.3.cmml" xref="S1.p1.5.m5.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.5.m5.1c">n_{q}</annotation><annotation encoding="application/x-llamapun" id="S1.p1.5.m5.1d">italic_n start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT</annotation></semantics></math>, based on which SEM is trained in order to increase the similarity of (<math alttext="q" class="ltx_Math" display="inline" id="S1.p1.6.m6.1"><semantics id="S1.p1.6.m6.1a"><mi id="S1.p1.6.m6.1.1" xref="S1.p1.6.m6.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="S1.p1.6.m6.1b"><ci id="S1.p1.6.m6.1.1.cmml" xref="S1.p1.6.m6.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.6.m6.1c">q</annotation><annotation encoding="application/x-llamapun" id="S1.p1.6.m6.1d">italic_q</annotation></semantics></math>, <math alttext="p_{q}" class="ltx_Math" display="inline" id="S1.p1.7.m7.1"><semantics id="S1.p1.7.m7.1a"><msub id="S1.p1.7.m7.1.1" xref="S1.p1.7.m7.1.1.cmml"><mi id="S1.p1.7.m7.1.1.2" xref="S1.p1.7.m7.1.1.2.cmml">p</mi><mi id="S1.p1.7.m7.1.1.3" xref="S1.p1.7.m7.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p1.7.m7.1b"><apply id="S1.p1.7.m7.1.1.cmml" xref="S1.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S1.p1.7.m7.1.1.1.cmml" xref="S1.p1.7.m7.1.1">subscript</csymbol><ci id="S1.p1.7.m7.1.1.2.cmml" xref="S1.p1.7.m7.1.1.2">𝑝</ci><ci id="S1.p1.7.m7.1.1.3.cmml" xref="S1.p1.7.m7.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.7.m7.1c">p_{q}</annotation><annotation encoding="application/x-llamapun" id="S1.p1.7.m7.1d">italic_p start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT</annotation></semantics></math>) and the dissimilarity of (<math alttext="q" class="ltx_Math" display="inline" id="S1.p1.8.m8.1"><semantics id="S1.p1.8.m8.1a"><mi id="S1.p1.8.m8.1.1" xref="S1.p1.8.m8.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="S1.p1.8.m8.1b"><ci id="S1.p1.8.m8.1.1.cmml" xref="S1.p1.8.m8.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.8.m8.1c">q</annotation><annotation encoding="application/x-llamapun" id="S1.p1.8.m8.1d">italic_q</annotation></semantics></math>, <math alttext="n_{q}" class="ltx_Math" display="inline" id="S1.p1.9.m9.1"><semantics id="S1.p1.9.m9.1a"><msub id="S1.p1.9.m9.1.1" xref="S1.p1.9.m9.1.1.cmml"><mi id="S1.p1.9.m9.1.1.2" xref="S1.p1.9.m9.1.1.2.cmml">n</mi><mi id="S1.p1.9.m9.1.1.3" xref="S1.p1.9.m9.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p1.9.m9.1b"><apply id="S1.p1.9.m9.1.1.cmml" xref="S1.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S1.p1.9.m9.1.1.1.cmml" xref="S1.p1.9.m9.1.1">subscript</csymbol><ci id="S1.p1.9.m9.1.1.2.cmml" xref="S1.p1.9.m9.1.1.2">𝑛</ci><ci id="S1.p1.9.m9.1.1.3.cmml" xref="S1.p1.9.m9.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.9.m9.1c">n_{q}</annotation><annotation encoding="application/x-llamapun" id="S1.p1.9.m9.1d">italic_n start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT</annotation></semantics></math>).</p> </div> <div class="ltx_para" id="S1.p2"> <p class="ltx_p" id="S1.p2.1">A crucial issue of training SEM is to automatically formulate high-quality pairwise judgments, i.e., inferring the user’s implicit preference based on the query log. The same problem is intensely studied in the field of pairwise Learning-to-Rank (LTR) <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib8" title="">2011</a>; Joachims et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib19" title="">2007</a>; Radlinski and Joachims, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib23" title="">2005</a>)</cite>. Researchers propose an effective strategy for deriving pairwise judgment based on heuristics, which is inspired by the examination and click-through activities of users <cite class="ltx_cite ltx_citemacro_citep">(Chapelle and Zhang, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib3" title="">2009</a>)</cite>. The core of these heuristics is to eliminate noise and position bias from the query log. Specifically, the heuristics assume that a clicked document with the current query is preferred over an examined but not clicked document. With the intensive study of this topic in LTR, little work exists on pairwise judgment formulation for embedding-based models such as SEM. Hence, it is interesting to study whether the well-established heuristics in LTR can be successfully applied to SEM.</p> </div> <div class="ltx_para" id="S1.p3"> <p class="ltx_p" id="S1.p3.1">In this paper, we investigate this problem based on the query log of a major commercial search engine and propose a series of strategies to formulate pairwise judgments. Through extensive experiments, we quantitatively compare these strategies and demonstrate some effective strategies for training SEM. An important insight obtained is that the conventional pairwise judgment formulation heuristics in LTR are not necessarily effective for training SEM, which requires specialized strategies for deriving effective training data. These results reveal that exploring strategies for pairwise judgment formulation is a valuable direction for practitioners seeking to enhance the performance of embedding-based models. The contributions of this paper are summarized as follows:</p> </div> <div class="ltx_para" id="S1.p4"> <ol class="ltx_enumerate" id="S1.I1"> <li class="ltx_item" id="S1.I1.i1" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(1)</span> <div class="ltx_para" id="S1.I1.i1.p1"> <p class="ltx_p" id="S1.I1.i1.p1.1">We provide a detailed description of how to construct and apply the SEM model in Web search scenarios;</p> </div> </li> <li class="ltx_item" id="S1.I1.i2" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(2)</span> <div class="ltx_para" id="S1.I1.i2.p1"> <p class="ltx_p" id="S1.I1.i2.p1.1">We present the first in-depth study to propose and evaluate strategies (both atomic and hybrid) for formulating pairwise judgments to train embedding-based SEM model;</p> </div> </li> <li class="ltx_item" id="S1.I1.i3" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(3)</span> <div class="ltx_para" id="S1.I1.i3.p1"> <p class="ltx_p" id="S1.I1.i3.p1.1">Based on extensive experiments on the largest (to the best of our knowledge) industrial-level pairwise judgment dataset, we highlight the best practices for training SEM and explain the deviation from the training of LTR.</p> </div> </li> </ol> </div> <div class="ltx_para" id="S1.p5"> <p class="ltx_p" id="S1.p5.1">The remainder of this paper is organized as follows. In Section <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S2" title="2. Related Work ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2</span></a>, we review the most related work. In Section <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3" title="3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">3</span></a> and <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S4" title="4. Experimental Setup ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">4</span></a>, we describe the model architecture and experimental design. In Sections <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S5" title="5. Atomic Strategies ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">5</span></a> and <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S6" title="6. Hybrid Strategy ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">6</span></a>, we discuss the strategies for formulating pairwise judgments. Finally, we conclude this paper in Section <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S7" title="7. Conclusions ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">7</span></a>.</p> </div> </section> <section class="ltx_section" id="S2"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">2. </span>Related Work</h2> <div class="ltx_para" id="S2.p1"> <p class="ltx_p" id="S2.p1.1">Formulating pairwise judgments is well-studied in pairwise Learning-to-Rank (LTR). Radlinksi et al. <cite class="ltx_cite ltx_citemacro_citep">(Radlinski and Joachims, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib23" title="">2005</a>)</cite> proposed an approach to learning ranked retrieval functions by deriving pairwise judgment from a series of queries. Joachims et al. <cite class="ltx_cite ltx_citemacro_citep">(Joachims et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib19" title="">2007</a>)</cite> examined the reliability of implicit feedback from click-through data in Web search. They show that preferences derived from user clicks are reasonably accurate, particularly when comparing results from the same query. The heuristics proposed in <cite class="ltx_cite ltx_citemacro_citep">(Joachims et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib19" title="">2007</a>; Radlinski and Joachims, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib23" title="">2005</a>)</cite> are widely used in conventional LTR scenarios.</p> </div> <div class="ltx_para" id="S2.p2"> <p class="ltx_p" id="S2.p2.1">Search engine users’ browsing patterns are also intensively studied in the field of click models. Chapelle et al. <cite class="ltx_cite ltx_citemacro_citep">(Chapelle and Zhang, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib3" title="">2009</a>)</cite> consider click logs as an important source of implicit feedback. They proposed a Dynamic Bayesian Network that aims to provide an unbiased estimation of the relevance of the click logs. A personalized click model is proposed in <cite class="ltx_cite ltx_citemacro_citep">(Shen et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib24" title="">2012</a>)</cite> to describe the user-oriented click preferences, which applies and extends tensor factorization from the view of collaborative filtering. <cite class="ltx_cite ltx_citemacro_citep">(Chen et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib4" title="">2012</a>)</cite> proposed a Noise-aware Clicked Model by characterizing the noise degree of a click. With their individual differences, almost all click models assume that search engine users sequentially browse the results from top to bottom and click some results if they are perceived as relevant. This assumption is strictly aligned with those studied in <cite class="ltx_cite ltx_citemacro_citep">(Joachims et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib19" title="">2007</a>)</cite>.</p> </div> <div class="ltx_para" id="S2.p3"> <p class="ltx_p" id="S2.p3.1">In recent years, learning semantic embedding has attracted a lot of research attention. For example, Huang et al. <cite class="ltx_cite ltx_citemacro_citep">(Huang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib7" title="">2013</a>)</cite> developed deep structured semantic model that projects queries and documents into a common low-dimensional space where the relevance of a document given a query is readily computed as the distance between them. The proposed model is discriminatively trained by maximizing the conditional likelihood of the clicked documents given a query using the click-through data. Shen et al. <cite class="ltx_cite ltx_citemacro_citep">(Shen et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib25" title="">2014</a>)</cite> developed a variant of the deep structured semantic model by incorporating a convolutional-pooling structure over word sequences to learn low-dimensional semantic vector representations for search queries and Web documents.</p> </div> <div class="ltx_para" id="S2.p4"> <p class="ltx_p" id="S2.p4.1">Semantic Embedding Model (SEM) has been proposed in <cite class="ltx_cite ltx_citemacro_citep">(Wu et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib27" title="">2014</a>)</cite> and is considered to be a more efficient model for learning semantic embedding. The major difference between SEM and its counterparts is the use of hinge loss with a pairwise training paradigm rather than softmax-based loss functions. Hence, training SEM is typically more efficient since it does not need to conduct backpropagation for every training instance. With the success of SEM and its related models in the search engine industry <cite class="ltx_cite ltx_citemacro_citep">(Li et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib21" title="">2021</a>; Wang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib26" title="">2020</a>)</cite>, there is surprisingly little work on studying the best approach for deriving training data for them. To the best of our knowledge, this work presents the pioneering effort to study the strategies of formulating pairwise judgments for training SEM.</p> </div> <figure class="ltx_figure" id="S2.F1"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_2"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" height="524" id="S2.F1.g1" src="x1.png" width="747"/></div> </div> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span class="ltx_text" id="S2.F1.2.1.1" style="font-size:90%;">Figure 1</span>. </span><span class="ltx_text" id="S2.F1.3.2" style="font-size:90%;">Semantic Embedding Model Architecture</span></figcaption><div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_1"><span class="ltx_ERROR ltx_centering ltx_figure_panel undefined" id="S2.F1.4">\Description</span></div> <div class="ltx_flex_break"></div> <div class="ltx_flex_cell ltx_flex_size_1"> <p class="ltx_p ltx_figure_panel ltx_align_center" id="S2.F1.5">[Semantic Embedding Model Architecture]</p> </div> </div> </figure> </section> <section class="ltx_section" id="S3"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">3. </span>Semantic Embedding Model for Web Search</h2> <div class="ltx_para" id="S3.p1"> <p class="ltx_p" id="S3.p1.1">In this section, we first elaborate on the architecture of SEM for Web search, and then we detail how to optimize its parameters.</p> </div> <section class="ltx_subsection" id="S3.SSx1"> <h3 class="ltx_title ltx_title_subsection">Architecture</h3> <div class="ltx_para" id="S3.SSx1.p1"> <p class="ltx_p" id="S3.SSx1.p1.1">Inspired by the work in <cite class="ltx_cite ltx_citemacro_citep">(Wu et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib27" title="">2014</a>)</cite>, we describe how the Semantic Embedding Model (SEM) works for Web information retrieval scenarios. The architecture of SEM is shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S2.F1" title="Figure 1 ‣ 2. Related Work ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">1</span></a>. A hinge loss <math alttext="L" class="ltx_Math" display="inline" id="S3.SSx1.p1.1.m1.1"><semantics id="S3.SSx1.p1.1.m1.1a"><mi id="S3.SSx1.p1.1.m1.1.1" xref="S3.SSx1.p1.1.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p1.1.m1.1b"><ci id="S3.SSx1.p1.1.m1.1.1.cmml" xref="S3.SSx1.p1.1.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p1.1.m1.1c">L</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p1.1.m1.1d">italic_L</annotation></semantics></math> is used for training:</p> </div> <div class="ltx_para" id="S3.SSx1.p2"> <table class="ltx_equation ltx_eqn_table" id="S3.E1"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(1)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="L=\frac{1}{m}\sum_{i=1}^{m}\left(\cos\langle f(q_{i}),f(p_{q_{i}})\rangle-\cos% \langle f(q_{i}),f(n_{q_{i}})\rangle\right)" class="ltx_Math" display="block" id="S3.E1.m1.3"><semantics id="S3.E1.m1.3a"><mrow id="S3.E1.m1.3.3" xref="S3.E1.m1.3.3.cmml"><mi id="S3.E1.m1.3.3.3" xref="S3.E1.m1.3.3.3.cmml">L</mi><mo id="S3.E1.m1.3.3.2" xref="S3.E1.m1.3.3.2.cmml">=</mo><mrow id="S3.E1.m1.3.3.1" xref="S3.E1.m1.3.3.1.cmml"><mfrac id="S3.E1.m1.3.3.1.3" xref="S3.E1.m1.3.3.1.3.cmml"><mn id="S3.E1.m1.3.3.1.3.2" xref="S3.E1.m1.3.3.1.3.2.cmml">1</mn><mi id="S3.E1.m1.3.3.1.3.3" xref="S3.E1.m1.3.3.1.3.3.cmml">m</mi></mfrac><mo id="S3.E1.m1.3.3.1.2" xref="S3.E1.m1.3.3.1.2.cmml">⁢</mo><mrow id="S3.E1.m1.3.3.1.1" xref="S3.E1.m1.3.3.1.1.cmml"><munderover id="S3.E1.m1.3.3.1.1.2" xref="S3.E1.m1.3.3.1.1.2.cmml"><mo id="S3.E1.m1.3.3.1.1.2.2.2" movablelimits="false" rspace="0em" xref="S3.E1.m1.3.3.1.1.2.2.2.cmml">∑</mo><mrow id="S3.E1.m1.3.3.1.1.2.2.3" xref="S3.E1.m1.3.3.1.1.2.2.3.cmml"><mi id="S3.E1.m1.3.3.1.1.2.2.3.2" xref="S3.E1.m1.3.3.1.1.2.2.3.2.cmml">i</mi><mo id="S3.E1.m1.3.3.1.1.2.2.3.1" xref="S3.E1.m1.3.3.1.1.2.2.3.1.cmml">=</mo><mn id="S3.E1.m1.3.3.1.1.2.2.3.3" xref="S3.E1.m1.3.3.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S3.E1.m1.3.3.1.1.2.3" xref="S3.E1.m1.3.3.1.1.2.3.cmml">m</mi></munderover><mrow id="S3.E1.m1.3.3.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.cmml"><mo id="S3.E1.m1.3.3.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.cmml">(</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.cmml"><mrow id="S3.E1.m1.3.3.1.1.1.1.1.2.2" xref="S3.E1.m1.3.3.1.1.1.1.1.2.3.cmml"><mi id="S3.E1.m1.1.1" xref="S3.E1.m1.1.1.cmml">cos</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.2.2a" xref="S3.E1.m1.3.3.1.1.1.1.1.2.3.cmml">⁡</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2" xref="S3.E1.m1.3.3.1.1.1.1.1.2.3.cmml"><mo id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.3" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.2.3.cmml">⟨</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.cmml">f</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.2.cmml">⁢</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.2" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">q</mi><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.3" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.4" xref="S3.E1.m1.3.3.1.1.1.1.1.2.3.cmml">,</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.3" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.3.cmml">f</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.2" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.2.cmml">⁢</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.cmml"><mo id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.2" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.cmml">(</mo><msub id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.2.cmml">p</mi><msub id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.2" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.2.cmml">q</mi><mi id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.3" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.3.cmml">i</mi></msub></msub><mo id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.3" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.5" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.2.3.cmml">⟩</mo></mrow></mrow><mo id="S3.E1.m1.3.3.1.1.1.1.1.5" xref="S3.E1.m1.3.3.1.1.1.1.1.5.cmml">−</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.4.2" xref="S3.E1.m1.3.3.1.1.1.1.1.4.3.cmml"><mi id="S3.E1.m1.2.2" xref="S3.E1.m1.2.2.cmml">cos</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.4.2a" xref="S3.E1.m1.3.3.1.1.1.1.1.4.3.cmml">⁡</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2" xref="S3.E1.m1.3.3.1.1.1.1.1.4.3.cmml"><mo id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.3" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.4.3.cmml">⟨</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.3.cmml">f</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.2.cmml">⁢</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.cmml"><mo id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.2" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.cmml">(</mo><msub id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.2.cmml">q</mi><mi id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.3" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.4" xref="S3.E1.m1.3.3.1.1.1.1.1.4.3.cmml">,</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.3" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.3.cmml">f</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.2" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.2.cmml">⁢</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.cmml"><mo id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.2" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.cmml">(</mo><msub id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.2.cmml">n</mi><msub id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.2" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.2.cmml">q</mi><mi id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.3" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.3.cmml">i</mi></msub></msub><mo id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.3" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.5" stretchy="false" xref="S3.E1.m1.3.3.1.1.1.1.1.4.3.cmml">⟩</mo></mrow></mrow></mrow><mo id="S3.E1.m1.3.3.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.3b"><apply id="S3.E1.m1.3.3.cmml" xref="S3.E1.m1.3.3"><eq id="S3.E1.m1.3.3.2.cmml" xref="S3.E1.m1.3.3.2"></eq><ci id="S3.E1.m1.3.3.3.cmml" xref="S3.E1.m1.3.3.3">𝐿</ci><apply id="S3.E1.m1.3.3.1.cmml" xref="S3.E1.m1.3.3.1"><times id="S3.E1.m1.3.3.1.2.cmml" xref="S3.E1.m1.3.3.1.2"></times><apply id="S3.E1.m1.3.3.1.3.cmml" xref="S3.E1.m1.3.3.1.3"><divide id="S3.E1.m1.3.3.1.3.1.cmml" xref="S3.E1.m1.3.3.1.3"></divide><cn id="S3.E1.m1.3.3.1.3.2.cmml" type="integer" xref="S3.E1.m1.3.3.1.3.2">1</cn><ci id="S3.E1.m1.3.3.1.3.3.cmml" xref="S3.E1.m1.3.3.1.3.3">𝑚</ci></apply><apply id="S3.E1.m1.3.3.1.1.cmml" xref="S3.E1.m1.3.3.1.1"><apply id="S3.E1.m1.3.3.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.2.1.cmml" xref="S3.E1.m1.3.3.1.1.2">superscript</csymbol><apply id="S3.E1.m1.3.3.1.1.2.2.cmml" xref="S3.E1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.2.2.1.cmml" xref="S3.E1.m1.3.3.1.1.2">subscript</csymbol><sum id="S3.E1.m1.3.3.1.1.2.2.2.cmml" xref="S3.E1.m1.3.3.1.1.2.2.2"></sum><apply id="S3.E1.m1.3.3.1.1.2.2.3.cmml" xref="S3.E1.m1.3.3.1.1.2.2.3"><eq id="S3.E1.m1.3.3.1.1.2.2.3.1.cmml" xref="S3.E1.m1.3.3.1.1.2.2.3.1"></eq><ci id="S3.E1.m1.3.3.1.1.2.2.3.2.cmml" xref="S3.E1.m1.3.3.1.1.2.2.3.2">𝑖</ci><cn id="S3.E1.m1.3.3.1.1.2.2.3.3.cmml" type="integer" xref="S3.E1.m1.3.3.1.1.2.2.3.3">1</cn></apply></apply><ci id="S3.E1.m1.3.3.1.1.2.3.cmml" xref="S3.E1.m1.3.3.1.1.2.3">𝑚</ci></apply><apply id="S3.E1.m1.3.3.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1"><minus id="S3.E1.m1.3.3.1.1.1.1.1.5.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.5"></minus><apply id="S3.E1.m1.3.3.1.1.1.1.1.2.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2"><cos id="S3.E1.m1.1.1.cmml" xref="S3.E1.m1.1.1"></cos><apply id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1"><times id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.2"></times><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3">𝑓</ci><apply id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2"><times id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.2"></times><ci id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.3">𝑓</ci><apply id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.2">𝑝</ci><apply id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.2">𝑞</ci><ci id="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.2.2.2.2.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply><apply id="S3.E1.m1.3.3.1.1.1.1.1.4.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2"><cos id="S3.E1.m1.2.2.cmml" xref="S3.E1.m1.2.2"></cos><apply id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1"><times id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.2"></times><ci id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.3">𝑓</ci><apply id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.3.1.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2"><times id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.2"></times><ci id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.3">𝑓</ci><apply id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.2">𝑛</ci><apply id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.2">𝑞</ci><ci id="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.4.2.2.2.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.3c">L=\frac{1}{m}\sum_{i=1}^{m}\left(\cos\langle f(q_{i}),f(p_{q_{i}})\rangle-\cos% \langle f(q_{i}),f(n_{q_{i}})\rangle\right)</annotation><annotation encoding="application/x-llamapun" id="S3.E1.m1.3d">italic_L = divide start_ARG 1 end_ARG start_ARG italic_m end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT ( roman_cos ⟨ italic_f ( italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) , italic_f ( italic_p start_POSTSUBSCRIPT italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) ⟩ - roman_cos ⟨ italic_f ( italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) , italic_f ( italic_n start_POSTSUBSCRIPT italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) ⟩ )</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx1.p3"> <p class="ltx_p" id="S3.SSx1.p3.3">where <math alttext="m" class="ltx_Math" display="inline" id="S3.SSx1.p3.1.m1.1"><semantics id="S3.SSx1.p3.1.m1.1a"><mi id="S3.SSx1.p3.1.m1.1.1" xref="S3.SSx1.p3.1.m1.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p3.1.m1.1b"><ci id="S3.SSx1.p3.1.m1.1.1.cmml" xref="S3.SSx1.p3.1.m1.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p3.1.m1.1c">m</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p3.1.m1.1d">italic_m</annotation></semantics></math> is the amount of training instances, <math alttext="\cos" class="ltx_Math" display="inline" id="S3.SSx1.p3.2.m2.1"><semantics id="S3.SSx1.p3.2.m2.1a"><mi id="S3.SSx1.p3.2.m2.1.1" xref="S3.SSx1.p3.2.m2.1.1.cmml">cos</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p3.2.m2.1b"><cos id="S3.SSx1.p3.2.m2.1.1.cmml" xref="S3.SSx1.p3.2.m2.1.1"></cos></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p3.2.m2.1c">\cos</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p3.2.m2.1d">roman_cos</annotation></semantics></math> indicates cosine similarity and the function <math alttext="f(\cdot)" class="ltx_Math" display="inline" id="S3.SSx1.p3.3.m3.1"><semantics id="S3.SSx1.p3.3.m3.1a"><mrow id="S3.SSx1.p3.3.m3.1.2" xref="S3.SSx1.p3.3.m3.1.2.cmml"><mi id="S3.SSx1.p3.3.m3.1.2.2" xref="S3.SSx1.p3.3.m3.1.2.2.cmml">f</mi><mo id="S3.SSx1.p3.3.m3.1.2.1" xref="S3.SSx1.p3.3.m3.1.2.1.cmml">⁢</mo><mrow id="S3.SSx1.p3.3.m3.1.2.3.2" xref="S3.SSx1.p3.3.m3.1.2.cmml"><mo id="S3.SSx1.p3.3.m3.1.2.3.2.1" stretchy="false" xref="S3.SSx1.p3.3.m3.1.2.cmml">(</mo><mo id="S3.SSx1.p3.3.m3.1.1" lspace="0em" rspace="0em" xref="S3.SSx1.p3.3.m3.1.1.cmml">⋅</mo><mo id="S3.SSx1.p3.3.m3.1.2.3.2.2" stretchy="false" xref="S3.SSx1.p3.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx1.p3.3.m3.1b"><apply id="S3.SSx1.p3.3.m3.1.2.cmml" xref="S3.SSx1.p3.3.m3.1.2"><times id="S3.SSx1.p3.3.m3.1.2.1.cmml" xref="S3.SSx1.p3.3.m3.1.2.1"></times><ci id="S3.SSx1.p3.3.m3.1.2.2.cmml" xref="S3.SSx1.p3.3.m3.1.2.2">𝑓</ci><ci id="S3.SSx1.p3.3.m3.1.1.cmml" xref="S3.SSx1.p3.3.m3.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p3.3.m3.1c">f(\cdot)</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p3.3.m3.1d">italic_f ( ⋅ )</annotation></semantics></math> indicates the mapping from a query or a Web page title to a semantic embedding.</p> </div> <div class="ltx_para" id="S3.SSx1.p4"> <p class="ltx_p" id="S3.SSx1.p4.5">As shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S2.F1" title="Figure 1 ‣ 2. Related Work ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">1</span></a>, the bottom layer are the word embeddings. Through adding the embedding of words in query in an element-wise fashion, we obtain an intermediate representation of the query. If we denote <math alttext="x" class="ltx_Math" display="inline" id="S3.SSx1.p4.1.m1.1"><semantics id="S3.SSx1.p4.1.m1.1a"><mi id="S3.SSx1.p4.1.m1.1.1" xref="S3.SSx1.p4.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p4.1.m1.1b"><ci id="S3.SSx1.p4.1.m1.1.1.cmml" xref="S3.SSx1.p4.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p4.1.m1.1c">x</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p4.1.m1.1d">italic_x</annotation></semantics></math> as the input word embedding, <math alttext="j=1,\dots,N" class="ltx_Math" display="inline" id="S3.SSx1.p4.2.m2.3"><semantics id="S3.SSx1.p4.2.m2.3a"><mrow id="S3.SSx1.p4.2.m2.3.4" xref="S3.SSx1.p4.2.m2.3.4.cmml"><mi id="S3.SSx1.p4.2.m2.3.4.2" xref="S3.SSx1.p4.2.m2.3.4.2.cmml">j</mi><mo id="S3.SSx1.p4.2.m2.3.4.1" xref="S3.SSx1.p4.2.m2.3.4.1.cmml">=</mo><mrow id="S3.SSx1.p4.2.m2.3.4.3.2" xref="S3.SSx1.p4.2.m2.3.4.3.1.cmml"><mn id="S3.SSx1.p4.2.m2.1.1" xref="S3.SSx1.p4.2.m2.1.1.cmml">1</mn><mo id="S3.SSx1.p4.2.m2.3.4.3.2.1" xref="S3.SSx1.p4.2.m2.3.4.3.1.cmml">,</mo><mi id="S3.SSx1.p4.2.m2.2.2" mathvariant="normal" xref="S3.SSx1.p4.2.m2.2.2.cmml">…</mi><mo id="S3.SSx1.p4.2.m2.3.4.3.2.2" xref="S3.SSx1.p4.2.m2.3.4.3.1.cmml">,</mo><mi id="S3.SSx1.p4.2.m2.3.3" xref="S3.SSx1.p4.2.m2.3.3.cmml">N</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx1.p4.2.m2.3b"><apply id="S3.SSx1.p4.2.m2.3.4.cmml" xref="S3.SSx1.p4.2.m2.3.4"><eq id="S3.SSx1.p4.2.m2.3.4.1.cmml" xref="S3.SSx1.p4.2.m2.3.4.1"></eq><ci id="S3.SSx1.p4.2.m2.3.4.2.cmml" xref="S3.SSx1.p4.2.m2.3.4.2">𝑗</ci><list id="S3.SSx1.p4.2.m2.3.4.3.1.cmml" xref="S3.SSx1.p4.2.m2.3.4.3.2"><cn id="S3.SSx1.p4.2.m2.1.1.cmml" type="integer" xref="S3.SSx1.p4.2.m2.1.1">1</cn><ci id="S3.SSx1.p4.2.m2.2.2.cmml" xref="S3.SSx1.p4.2.m2.2.2">…</ci><ci id="S3.SSx1.p4.2.m2.3.3.cmml" xref="S3.SSx1.p4.2.m2.3.3">𝑁</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p4.2.m2.3c">j=1,\dots,N</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p4.2.m2.3d">italic_j = 1 , … , italic_N</annotation></semantics></math> as the term index of query, <math alttext="h" class="ltx_Math" display="inline" id="S3.SSx1.p4.3.m3.1"><semantics id="S3.SSx1.p4.3.m3.1a"><mi id="S3.SSx1.p4.3.m3.1.1" xref="S3.SSx1.p4.3.m3.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p4.3.m3.1b"><ci id="S3.SSx1.p4.3.m3.1.1.cmml" xref="S3.SSx1.p4.3.m3.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p4.3.m3.1c">h</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p4.3.m3.1d">italic_h</annotation></semantics></math> as the intermediate representation of query, and <math alttext="i" class="ltx_Math" display="inline" id="S3.SSx1.p4.4.m4.1"><semantics id="S3.SSx1.p4.4.m4.1a"><mi id="S3.SSx1.p4.4.m4.1.1" xref="S3.SSx1.p4.4.m4.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p4.4.m4.1b"><ci id="S3.SSx1.p4.4.m4.1.1.cmml" xref="S3.SSx1.p4.4.m4.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p4.4.m4.1c">i</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p4.4.m4.1d">italic_i</annotation></semantics></math> as the element id of <math alttext="h" class="ltx_Math" display="inline" id="S3.SSx1.p4.5.m5.1"><semantics id="S3.SSx1.p4.5.m5.1a"><mi id="S3.SSx1.p4.5.m5.1.1" xref="S3.SSx1.p4.5.m5.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p4.5.m5.1b"><ci id="S3.SSx1.p4.5.m5.1.1.cmml" xref="S3.SSx1.p4.5.m5.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p4.5.m5.1c">h</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p4.5.m5.1d">italic_h</annotation></semantics></math>, then we have</p> </div> <div class="ltx_para" id="S3.SSx1.p5"> <table class="ltx_equation ltx_eqn_table" id="S3.E2"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(2)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="h_{i}=\sum_{j=0}^{N}x_{i}" class="ltx_Math" display="block" id="S3.E2.m1.1"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1.1" xref="S3.E2.m1.1.1.cmml"><msub id="S3.E2.m1.1.1.2" xref="S3.E2.m1.1.1.2.cmml"><mi id="S3.E2.m1.1.1.2.2" xref="S3.E2.m1.1.1.2.2.cmml">h</mi><mi id="S3.E2.m1.1.1.2.3" xref="S3.E2.m1.1.1.2.3.cmml">i</mi></msub><mo id="S3.E2.m1.1.1.1" rspace="0.111em" xref="S3.E2.m1.1.1.1.cmml">=</mo><mrow id="S3.E2.m1.1.1.3" xref="S3.E2.m1.1.1.3.cmml"><munderover id="S3.E2.m1.1.1.3.1" xref="S3.E2.m1.1.1.3.1.cmml"><mo id="S3.E2.m1.1.1.3.1.2.2" movablelimits="false" xref="S3.E2.m1.1.1.3.1.2.2.cmml">∑</mo><mrow id="S3.E2.m1.1.1.3.1.2.3" xref="S3.E2.m1.1.1.3.1.2.3.cmml"><mi id="S3.E2.m1.1.1.3.1.2.3.2" xref="S3.E2.m1.1.1.3.1.2.3.2.cmml">j</mi><mo id="S3.E2.m1.1.1.3.1.2.3.1" xref="S3.E2.m1.1.1.3.1.2.3.1.cmml">=</mo><mn id="S3.E2.m1.1.1.3.1.2.3.3" xref="S3.E2.m1.1.1.3.1.2.3.3.cmml">0</mn></mrow><mi id="S3.E2.m1.1.1.3.1.3" xref="S3.E2.m1.1.1.3.1.3.cmml">N</mi></munderover><msub id="S3.E2.m1.1.1.3.2" xref="S3.E2.m1.1.1.3.2.cmml"><mi id="S3.E2.m1.1.1.3.2.2" xref="S3.E2.m1.1.1.3.2.2.cmml">x</mi><mi id="S3.E2.m1.1.1.3.2.3" xref="S3.E2.m1.1.1.3.2.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.1b"><apply id="S3.E2.m1.1.1.cmml" xref="S3.E2.m1.1.1"><eq id="S3.E2.m1.1.1.1.cmml" xref="S3.E2.m1.1.1.1"></eq><apply id="S3.E2.m1.1.1.2.cmml" xref="S3.E2.m1.1.1.2"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.2.1.cmml" xref="S3.E2.m1.1.1.2">subscript</csymbol><ci id="S3.E2.m1.1.1.2.2.cmml" xref="S3.E2.m1.1.1.2.2">ℎ</ci><ci id="S3.E2.m1.1.1.2.3.cmml" xref="S3.E2.m1.1.1.2.3">𝑖</ci></apply><apply id="S3.E2.m1.1.1.3.cmml" xref="S3.E2.m1.1.1.3"><apply id="S3.E2.m1.1.1.3.1.cmml" xref="S3.E2.m1.1.1.3.1"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.3.1.1.cmml" xref="S3.E2.m1.1.1.3.1">superscript</csymbol><apply id="S3.E2.m1.1.1.3.1.2.cmml" xref="S3.E2.m1.1.1.3.1"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.3.1.2.1.cmml" xref="S3.E2.m1.1.1.3.1">subscript</csymbol><sum id="S3.E2.m1.1.1.3.1.2.2.cmml" xref="S3.E2.m1.1.1.3.1.2.2"></sum><apply id="S3.E2.m1.1.1.3.1.2.3.cmml" xref="S3.E2.m1.1.1.3.1.2.3"><eq id="S3.E2.m1.1.1.3.1.2.3.1.cmml" xref="S3.E2.m1.1.1.3.1.2.3.1"></eq><ci id="S3.E2.m1.1.1.3.1.2.3.2.cmml" xref="S3.E2.m1.1.1.3.1.2.3.2">𝑗</ci><cn id="S3.E2.m1.1.1.3.1.2.3.3.cmml" type="integer" xref="S3.E2.m1.1.1.3.1.2.3.3">0</cn></apply></apply><ci id="S3.E2.m1.1.1.3.1.3.cmml" xref="S3.E2.m1.1.1.3.1.3">𝑁</ci></apply><apply id="S3.E2.m1.1.1.3.2.cmml" xref="S3.E2.m1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.3.2.1.cmml" xref="S3.E2.m1.1.1.3.2">subscript</csymbol><ci id="S3.E2.m1.1.1.3.2.2.cmml" xref="S3.E2.m1.1.1.3.2.2">𝑥</ci><ci id="S3.E2.m1.1.1.3.2.3.cmml" xref="S3.E2.m1.1.1.3.2.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.1c">h_{i}=\sum_{j=0}^{N}x_{i}</annotation><annotation encoding="application/x-llamapun" id="S3.E2.m1.1d">italic_h start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_j = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx1.p6"> <table class="ltx_equation ltx_eqn_table" id="S3.E3"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(3)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="g_{i}=softsign(h_{i})=\frac{h_{i}}{1+|h_{i}|}" class="ltx_Math" display="block" id="S3.E3.m1.2"><semantics id="S3.E3.m1.2a"><mrow id="S3.E3.m1.2.2" xref="S3.E3.m1.2.2.cmml"><msub id="S3.E3.m1.2.2.3" xref="S3.E3.m1.2.2.3.cmml"><mi id="S3.E3.m1.2.2.3.2" xref="S3.E3.m1.2.2.3.2.cmml">g</mi><mi id="S3.E3.m1.2.2.3.3" xref="S3.E3.m1.2.2.3.3.cmml">i</mi></msub><mo id="S3.E3.m1.2.2.4" xref="S3.E3.m1.2.2.4.cmml">=</mo><mrow id="S3.E3.m1.2.2.1" xref="S3.E3.m1.2.2.1.cmml"><mi id="S3.E3.m1.2.2.1.3" xref="S3.E3.m1.2.2.1.3.cmml">s</mi><mo id="S3.E3.m1.2.2.1.2" xref="S3.E3.m1.2.2.1.2.cmml">⁢</mo><mi id="S3.E3.m1.2.2.1.4" xref="S3.E3.m1.2.2.1.4.cmml">o</mi><mo id="S3.E3.m1.2.2.1.2a" xref="S3.E3.m1.2.2.1.2.cmml">⁢</mo><mi id="S3.E3.m1.2.2.1.5" xref="S3.E3.m1.2.2.1.5.cmml">f</mi><mo id="S3.E3.m1.2.2.1.2b" xref="S3.E3.m1.2.2.1.2.cmml">⁢</mo><mi id="S3.E3.m1.2.2.1.6" xref="S3.E3.m1.2.2.1.6.cmml">t</mi><mo id="S3.E3.m1.2.2.1.2c" xref="S3.E3.m1.2.2.1.2.cmml">⁢</mo><mi id="S3.E3.m1.2.2.1.7" xref="S3.E3.m1.2.2.1.7.cmml">s</mi><mo id="S3.E3.m1.2.2.1.2d" xref="S3.E3.m1.2.2.1.2.cmml">⁢</mo><mi id="S3.E3.m1.2.2.1.8" xref="S3.E3.m1.2.2.1.8.cmml">i</mi><mo id="S3.E3.m1.2.2.1.2e" xref="S3.E3.m1.2.2.1.2.cmml">⁢</mo><mi id="S3.E3.m1.2.2.1.9" xref="S3.E3.m1.2.2.1.9.cmml">g</mi><mo id="S3.E3.m1.2.2.1.2f" xref="S3.E3.m1.2.2.1.2.cmml">⁢</mo><mi id="S3.E3.m1.2.2.1.10" xref="S3.E3.m1.2.2.1.10.cmml">n</mi><mo id="S3.E3.m1.2.2.1.2g" xref="S3.E3.m1.2.2.1.2.cmml">⁢</mo><mrow id="S3.E3.m1.2.2.1.1.1" xref="S3.E3.m1.2.2.1.1.1.1.cmml"><mo id="S3.E3.m1.2.2.1.1.1.2" stretchy="false" xref="S3.E3.m1.2.2.1.1.1.1.cmml">(</mo><msub id="S3.E3.m1.2.2.1.1.1.1" xref="S3.E3.m1.2.2.1.1.1.1.cmml"><mi id="S3.E3.m1.2.2.1.1.1.1.2" xref="S3.E3.m1.2.2.1.1.1.1.2.cmml">h</mi><mi id="S3.E3.m1.2.2.1.1.1.1.3" xref="S3.E3.m1.2.2.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.E3.m1.2.2.1.1.1.3" stretchy="false" xref="S3.E3.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E3.m1.2.2.5" xref="S3.E3.m1.2.2.5.cmml">=</mo><mfrac id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml"><msub id="S3.E3.m1.1.1.3" xref="S3.E3.m1.1.1.3.cmml"><mi id="S3.E3.m1.1.1.3.2" xref="S3.E3.m1.1.1.3.2.cmml">h</mi><mi id="S3.E3.m1.1.1.3.3" xref="S3.E3.m1.1.1.3.3.cmml">i</mi></msub><mrow id="S3.E3.m1.1.1.1" xref="S3.E3.m1.1.1.1.cmml"><mn id="S3.E3.m1.1.1.1.3" xref="S3.E3.m1.1.1.1.3.cmml">1</mn><mo id="S3.E3.m1.1.1.1.2" xref="S3.E3.m1.1.1.1.2.cmml">+</mo><mrow id="S3.E3.m1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.2.cmml"><mo id="S3.E3.m1.1.1.1.1.1.2" stretchy="false" xref="S3.E3.m1.1.1.1.1.2.1.cmml">|</mo><msub id="S3.E3.m1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.2.cmml">h</mi><mi id="S3.E3.m1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.E3.m1.1.1.1.1.1.3" stretchy="false" xref="S3.E3.m1.1.1.1.1.2.1.cmml">|</mo></mrow></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.2b"><apply id="S3.E3.m1.2.2.cmml" xref="S3.E3.m1.2.2"><and id="S3.E3.m1.2.2a.cmml" xref="S3.E3.m1.2.2"></and><apply id="S3.E3.m1.2.2b.cmml" xref="S3.E3.m1.2.2"><eq id="S3.E3.m1.2.2.4.cmml" xref="S3.E3.m1.2.2.4"></eq><apply id="S3.E3.m1.2.2.3.cmml" xref="S3.E3.m1.2.2.3"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.3.1.cmml" xref="S3.E3.m1.2.2.3">subscript</csymbol><ci id="S3.E3.m1.2.2.3.2.cmml" xref="S3.E3.m1.2.2.3.2">𝑔</ci><ci id="S3.E3.m1.2.2.3.3.cmml" xref="S3.E3.m1.2.2.3.3">𝑖</ci></apply><apply id="S3.E3.m1.2.2.1.cmml" xref="S3.E3.m1.2.2.1"><times id="S3.E3.m1.2.2.1.2.cmml" xref="S3.E3.m1.2.2.1.2"></times><ci id="S3.E3.m1.2.2.1.3.cmml" xref="S3.E3.m1.2.2.1.3">𝑠</ci><ci id="S3.E3.m1.2.2.1.4.cmml" xref="S3.E3.m1.2.2.1.4">𝑜</ci><ci id="S3.E3.m1.2.2.1.5.cmml" xref="S3.E3.m1.2.2.1.5">𝑓</ci><ci id="S3.E3.m1.2.2.1.6.cmml" xref="S3.E3.m1.2.2.1.6">𝑡</ci><ci id="S3.E3.m1.2.2.1.7.cmml" xref="S3.E3.m1.2.2.1.7">𝑠</ci><ci id="S3.E3.m1.2.2.1.8.cmml" xref="S3.E3.m1.2.2.1.8">𝑖</ci><ci id="S3.E3.m1.2.2.1.9.cmml" xref="S3.E3.m1.2.2.1.9">𝑔</ci><ci id="S3.E3.m1.2.2.1.10.cmml" xref="S3.E3.m1.2.2.1.10">𝑛</ci><apply id="S3.E3.m1.2.2.1.1.1.1.cmml" xref="S3.E3.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.1.1.1.cmml" xref="S3.E3.m1.2.2.1.1.1">subscript</csymbol><ci id="S3.E3.m1.2.2.1.1.1.1.2.cmml" xref="S3.E3.m1.2.2.1.1.1.1.2">ℎ</ci><ci id="S3.E3.m1.2.2.1.1.1.1.3.cmml" xref="S3.E3.m1.2.2.1.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="S3.E3.m1.2.2c.cmml" xref="S3.E3.m1.2.2"><eq id="S3.E3.m1.2.2.5.cmml" xref="S3.E3.m1.2.2.5"></eq><share href="https://arxiv.org/html/2408.04197v2#S3.E3.m1.2.2.1.cmml" id="S3.E3.m1.2.2d.cmml" xref="S3.E3.m1.2.2"></share><apply id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1"><divide id="S3.E3.m1.1.1.2.cmml" xref="S3.E3.m1.1.1"></divide><apply id="S3.E3.m1.1.1.3.cmml" xref="S3.E3.m1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.3.1.cmml" xref="S3.E3.m1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.1.1.3.2.cmml" xref="S3.E3.m1.1.1.3.2">ℎ</ci><ci id="S3.E3.m1.1.1.3.3.cmml" xref="S3.E3.m1.1.1.3.3">𝑖</ci></apply><apply id="S3.E3.m1.1.1.1.cmml" xref="S3.E3.m1.1.1.1"><plus id="S3.E3.m1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.2"></plus><cn id="S3.E3.m1.1.1.1.3.cmml" type="integer" xref="S3.E3.m1.1.1.1.3">1</cn><apply id="S3.E3.m1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1"><abs id="S3.E3.m1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.1.1.1.1.1.2"></abs><apply id="S3.E3.m1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2">ℎ</ci><ci id="S3.E3.m1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.2c">g_{i}=softsign(h_{i})=\frac{h_{i}}{1+|h_{i}|}</annotation><annotation encoding="application/x-llamapun" id="S3.E3.m1.2d">italic_g start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = italic_s italic_o italic_f italic_t italic_s italic_i italic_g italic_n ( italic_h start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) = divide start_ARG italic_h start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG 1 + | italic_h start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | end_ARG</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx1.p7"> <p class="ltx_p" id="S3.SSx1.p7.3">Then, the intermediate representation goes through fully connected neural network layers and is mapped to a final embedding. We denote <math alttext="O" class="ltx_Math" display="inline" id="S3.SSx1.p7.1.m1.1"><semantics id="S3.SSx1.p7.1.m1.1a"><mi id="S3.SSx1.p7.1.m1.1.1" xref="S3.SSx1.p7.1.m1.1.1.cmml">O</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p7.1.m1.1b"><ci id="S3.SSx1.p7.1.m1.1.1.cmml" xref="S3.SSx1.p7.1.m1.1.1">𝑂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p7.1.m1.1c">O</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p7.1.m1.1d">italic_O</annotation></semantics></math> as the final embedding, <math alttext="W" class="ltx_Math" display="inline" id="S3.SSx1.p7.2.m2.1"><semantics id="S3.SSx1.p7.2.m2.1a"><mi id="S3.SSx1.p7.2.m2.1.1" xref="S3.SSx1.p7.2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p7.2.m2.1b"><ci id="S3.SSx1.p7.2.m2.1.1.cmml" xref="S3.SSx1.p7.2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p7.2.m2.1c">W</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p7.2.m2.1d">italic_W</annotation></semantics></math> as the fully connected neural network layer’s weight matrix, and <math alttext="b" class="ltx_Math" display="inline" id="S3.SSx1.p7.3.m3.1"><semantics id="S3.SSx1.p7.3.m3.1a"><mi id="S3.SSx1.p7.3.m3.1.1" xref="S3.SSx1.p7.3.m3.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S3.SSx1.p7.3.m3.1b"><ci id="S3.SSx1.p7.3.m3.1.1.cmml" xref="S3.SSx1.p7.3.m3.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p7.3.m3.1c">b</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p7.3.m3.1d">italic_b</annotation></semantics></math> as the bias, where</p> </div> <div class="ltx_para" id="S3.SSx1.p8"> <table class="ltx_equation ltx_eqn_table" id="S3.E4"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(4)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="O=Wh+b" class="ltx_Math" display="block" id="S3.E4.m1.1"><semantics id="S3.E4.m1.1a"><mrow id="S3.E4.m1.1.1" xref="S3.E4.m1.1.1.cmml"><mi id="S3.E4.m1.1.1.2" xref="S3.E4.m1.1.1.2.cmml">O</mi><mo id="S3.E4.m1.1.1.1" xref="S3.E4.m1.1.1.1.cmml">=</mo><mrow id="S3.E4.m1.1.1.3" xref="S3.E4.m1.1.1.3.cmml"><mrow id="S3.E4.m1.1.1.3.2" xref="S3.E4.m1.1.1.3.2.cmml"><mi id="S3.E4.m1.1.1.3.2.2" xref="S3.E4.m1.1.1.3.2.2.cmml">W</mi><mo id="S3.E4.m1.1.1.3.2.1" xref="S3.E4.m1.1.1.3.2.1.cmml">⁢</mo><mi id="S3.E4.m1.1.1.3.2.3" xref="S3.E4.m1.1.1.3.2.3.cmml">h</mi></mrow><mo id="S3.E4.m1.1.1.3.1" xref="S3.E4.m1.1.1.3.1.cmml">+</mo><mi id="S3.E4.m1.1.1.3.3" xref="S3.E4.m1.1.1.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.1b"><apply id="S3.E4.m1.1.1.cmml" xref="S3.E4.m1.1.1"><eq id="S3.E4.m1.1.1.1.cmml" xref="S3.E4.m1.1.1.1"></eq><ci id="S3.E4.m1.1.1.2.cmml" xref="S3.E4.m1.1.1.2">𝑂</ci><apply id="S3.E4.m1.1.1.3.cmml" xref="S3.E4.m1.1.1.3"><plus id="S3.E4.m1.1.1.3.1.cmml" xref="S3.E4.m1.1.1.3.1"></plus><apply id="S3.E4.m1.1.1.3.2.cmml" xref="S3.E4.m1.1.1.3.2"><times id="S3.E4.m1.1.1.3.2.1.cmml" xref="S3.E4.m1.1.1.3.2.1"></times><ci id="S3.E4.m1.1.1.3.2.2.cmml" xref="S3.E4.m1.1.1.3.2.2">𝑊</ci><ci id="S3.E4.m1.1.1.3.2.3.cmml" xref="S3.E4.m1.1.1.3.2.3">ℎ</ci></apply><ci id="S3.E4.m1.1.1.3.3.cmml" xref="S3.E4.m1.1.1.3.3">𝑏</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.1c">O=Wh+b</annotation><annotation encoding="application/x-llamapun" id="S3.E4.m1.1d">italic_O = italic_W italic_h + italic_b</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx1.p9"> <p class="ltx_p" id="S3.SSx1.p9.2">The two titles in a pairwise judgment are processed in an analogous approach. Based on the final embeddings of the query and title that are denoted as <math alttext="O_{q}" class="ltx_Math" display="inline" id="S3.SSx1.p9.1.m1.1"><semantics id="S3.SSx1.p9.1.m1.1a"><msub id="S3.SSx1.p9.1.m1.1.1" xref="S3.SSx1.p9.1.m1.1.1.cmml"><mi id="S3.SSx1.p9.1.m1.1.1.2" xref="S3.SSx1.p9.1.m1.1.1.2.cmml">O</mi><mi id="S3.SSx1.p9.1.m1.1.1.3" xref="S3.SSx1.p9.1.m1.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SSx1.p9.1.m1.1b"><apply id="S3.SSx1.p9.1.m1.1.1.cmml" xref="S3.SSx1.p9.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SSx1.p9.1.m1.1.1.1.cmml" xref="S3.SSx1.p9.1.m1.1.1">subscript</csymbol><ci id="S3.SSx1.p9.1.m1.1.1.2.cmml" xref="S3.SSx1.p9.1.m1.1.1.2">𝑂</ci><ci id="S3.SSx1.p9.1.m1.1.1.3.cmml" xref="S3.SSx1.p9.1.m1.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p9.1.m1.1c">O_{q}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p9.1.m1.1d">italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT</annotation></semantics></math> and <math alttext="O_{d}" class="ltx_Math" display="inline" id="S3.SSx1.p9.2.m2.1"><semantics id="S3.SSx1.p9.2.m2.1a"><msub id="S3.SSx1.p9.2.m2.1.1" xref="S3.SSx1.p9.2.m2.1.1.cmml"><mi id="S3.SSx1.p9.2.m2.1.1.2" xref="S3.SSx1.p9.2.m2.1.1.2.cmml">O</mi><mi id="S3.SSx1.p9.2.m2.1.1.3" xref="S3.SSx1.p9.2.m2.1.1.3.cmml">d</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SSx1.p9.2.m2.1b"><apply id="S3.SSx1.p9.2.m2.1.1.cmml" xref="S3.SSx1.p9.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SSx1.p9.2.m2.1.1.1.cmml" xref="S3.SSx1.p9.2.m2.1.1">subscript</csymbol><ci id="S3.SSx1.p9.2.m2.1.1.2.cmml" xref="S3.SSx1.p9.2.m2.1.1.2">𝑂</ci><ci id="S3.SSx1.p9.2.m2.1.1.3.cmml" xref="S3.SSx1.p9.2.m2.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p9.2.m2.1c">O_{d}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p9.2.m2.1d">italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT</annotation></semantics></math>, we calculate their cosine similarity as follows:</p> </div> <div class="ltx_para" id="S3.SSx1.p10"> <table class="ltx_equation ltx_eqn_table" id="S3.E5"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(5)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="\cos({O}_{q},{O}_{d})={\frac{{O}_{q}^{T}{O}_{d}}{\|{O}_{q}^{T}\||{O}_{d}\|}}" class="ltx_Math" display="block" id="S3.E5.m1.5"><semantics id="S3.E5.m1.5a"><mrow id="S3.E5.m1.5.5" xref="S3.E5.m1.5.5.cmml"><mrow id="S3.E5.m1.5.5.2.2" xref="S3.E5.m1.5.5.2.3.cmml"><mi id="S3.E5.m1.3.3" xref="S3.E5.m1.3.3.cmml">cos</mi><mo id="S3.E5.m1.5.5.2.2a" xref="S3.E5.m1.5.5.2.3.cmml">⁡</mo><mrow id="S3.E5.m1.5.5.2.2.2" xref="S3.E5.m1.5.5.2.3.cmml"><mo id="S3.E5.m1.5.5.2.2.2.3" stretchy="false" xref="S3.E5.m1.5.5.2.3.cmml">(</mo><msub id="S3.E5.m1.4.4.1.1.1.1" xref="S3.E5.m1.4.4.1.1.1.1.cmml"><mi id="S3.E5.m1.4.4.1.1.1.1.2" xref="S3.E5.m1.4.4.1.1.1.1.2.cmml">O</mi><mi id="S3.E5.m1.4.4.1.1.1.1.3" xref="S3.E5.m1.4.4.1.1.1.1.3.cmml">q</mi></msub><mo id="S3.E5.m1.5.5.2.2.2.4" xref="S3.E5.m1.5.5.2.3.cmml">,</mo><msub id="S3.E5.m1.5.5.2.2.2.2" xref="S3.E5.m1.5.5.2.2.2.2.cmml"><mi id="S3.E5.m1.5.5.2.2.2.2.2" xref="S3.E5.m1.5.5.2.2.2.2.2.cmml">O</mi><mi id="S3.E5.m1.5.5.2.2.2.2.3" xref="S3.E5.m1.5.5.2.2.2.2.3.cmml">d</mi></msub><mo id="S3.E5.m1.5.5.2.2.2.5" stretchy="false" xref="S3.E5.m1.5.5.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E5.m1.5.5.3" xref="S3.E5.m1.5.5.3.cmml">=</mo><mfrac id="S3.E5.m1.2.2" xref="S3.E5.m1.2.2.cmml"><mrow id="S3.E5.m1.2.2.4" xref="S3.E5.m1.2.2.4.cmml"><msubsup id="S3.E5.m1.2.2.4.2" xref="S3.E5.m1.2.2.4.2.cmml"><mi id="S3.E5.m1.2.2.4.2.2.2" xref="S3.E5.m1.2.2.4.2.2.2.cmml">O</mi><mi id="S3.E5.m1.2.2.4.2.2.3" xref="S3.E5.m1.2.2.4.2.2.3.cmml">q</mi><mi id="S3.E5.m1.2.2.4.2.3" xref="S3.E5.m1.2.2.4.2.3.cmml">T</mi></msubsup><mo id="S3.E5.m1.2.2.4.1" xref="S3.E5.m1.2.2.4.1.cmml">⁢</mo><msub id="S3.E5.m1.2.2.4.3" xref="S3.E5.m1.2.2.4.3.cmml"><mi id="S3.E5.m1.2.2.4.3.2" xref="S3.E5.m1.2.2.4.3.2.cmml">O</mi><mi id="S3.E5.m1.2.2.4.3.3" xref="S3.E5.m1.2.2.4.3.3.cmml">d</mi></msub></mrow><mrow id="S3.E5.m1.2.2.2" xref="S3.E5.m1.2.2.2.cmml"><mrow id="S3.E5.m1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.2.cmml"><mo id="S3.E5.m1.1.1.1.1.1.2" stretchy="false" xref="S3.E5.m1.1.1.1.1.2.1.cmml">‖</mo><msubsup id="S3.E5.m1.1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.1.1.cmml"><mi id="S3.E5.m1.1.1.1.1.1.1.2.2" xref="S3.E5.m1.1.1.1.1.1.1.2.2.cmml">O</mi><mi id="S3.E5.m1.1.1.1.1.1.1.2.3" xref="S3.E5.m1.1.1.1.1.1.1.2.3.cmml">q</mi><mi id="S3.E5.m1.1.1.1.1.1.1.3" xref="S3.E5.m1.1.1.1.1.1.1.3.cmml">T</mi></msubsup><mo id="S3.E5.m1.1.1.1.1.1.3" stretchy="false" xref="S3.E5.m1.1.1.1.1.2.1.cmml">‖</mo></mrow><mo id="S3.E5.m1.2.2.2.3" xref="S3.E5.m1.2.2.2.3.cmml">⁢</mo><mrow id="S3.E5.m1.2.2.2.2.1" xref="S3.E5.m1.2.2.2.2.2.cmml"><mo id="S3.E5.m1.2.2.2.2.1.2" stretchy="false" xref="S3.E5.m1.2.2.2.2.2.1.cmml">|</mo><msub id="S3.E5.m1.2.2.2.2.1.1" xref="S3.E5.m1.2.2.2.2.1.1.cmml"><mi id="S3.E5.m1.2.2.2.2.1.1.2" xref="S3.E5.m1.2.2.2.2.1.1.2.cmml">O</mi><mi id="S3.E5.m1.2.2.2.2.1.1.3" xref="S3.E5.m1.2.2.2.2.1.1.3.cmml">d</mi></msub><mo id="S3.E5.m1.2.2.2.2.1.3" stretchy="false" xref="S3.E5.m1.2.2.2.2.2.1.cmml">‖</mo></mrow></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S3.E5.m1.5b"><apply id="S3.E5.m1.5.5.cmml" xref="S3.E5.m1.5.5"><eq id="S3.E5.m1.5.5.3.cmml" xref="S3.E5.m1.5.5.3"></eq><apply id="S3.E5.m1.5.5.2.3.cmml" xref="S3.E5.m1.5.5.2.2"><cos id="S3.E5.m1.3.3.cmml" xref="S3.E5.m1.3.3"></cos><apply id="S3.E5.m1.4.4.1.1.1.1.cmml" xref="S3.E5.m1.4.4.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.4.4.1.1.1.1.1.cmml" xref="S3.E5.m1.4.4.1.1.1.1">subscript</csymbol><ci id="S3.E5.m1.4.4.1.1.1.1.2.cmml" xref="S3.E5.m1.4.4.1.1.1.1.2">𝑂</ci><ci id="S3.E5.m1.4.4.1.1.1.1.3.cmml" xref="S3.E5.m1.4.4.1.1.1.1.3">𝑞</ci></apply><apply id="S3.E5.m1.5.5.2.2.2.2.cmml" xref="S3.E5.m1.5.5.2.2.2.2"><csymbol cd="ambiguous" id="S3.E5.m1.5.5.2.2.2.2.1.cmml" xref="S3.E5.m1.5.5.2.2.2.2">subscript</csymbol><ci id="S3.E5.m1.5.5.2.2.2.2.2.cmml" xref="S3.E5.m1.5.5.2.2.2.2.2">𝑂</ci><ci id="S3.E5.m1.5.5.2.2.2.2.3.cmml" xref="S3.E5.m1.5.5.2.2.2.2.3">𝑑</ci></apply></apply><apply id="S3.E5.m1.2.2.cmml" xref="S3.E5.m1.2.2"><divide id="S3.E5.m1.2.2.3.cmml" xref="S3.E5.m1.2.2"></divide><apply id="S3.E5.m1.2.2.4.cmml" xref="S3.E5.m1.2.2.4"><times id="S3.E5.m1.2.2.4.1.cmml" xref="S3.E5.m1.2.2.4.1"></times><apply id="S3.E5.m1.2.2.4.2.cmml" xref="S3.E5.m1.2.2.4.2"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.4.2.1.cmml" xref="S3.E5.m1.2.2.4.2">superscript</csymbol><apply id="S3.E5.m1.2.2.4.2.2.cmml" xref="S3.E5.m1.2.2.4.2"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.4.2.2.1.cmml" xref="S3.E5.m1.2.2.4.2">subscript</csymbol><ci id="S3.E5.m1.2.2.4.2.2.2.cmml" xref="S3.E5.m1.2.2.4.2.2.2">𝑂</ci><ci id="S3.E5.m1.2.2.4.2.2.3.cmml" xref="S3.E5.m1.2.2.4.2.2.3">𝑞</ci></apply><ci id="S3.E5.m1.2.2.4.2.3.cmml" xref="S3.E5.m1.2.2.4.2.3">𝑇</ci></apply><apply id="S3.E5.m1.2.2.4.3.cmml" xref="S3.E5.m1.2.2.4.3"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.4.3.1.cmml" xref="S3.E5.m1.2.2.4.3">subscript</csymbol><ci id="S3.E5.m1.2.2.4.3.2.cmml" xref="S3.E5.m1.2.2.4.3.2">𝑂</ci><ci id="S3.E5.m1.2.2.4.3.3.cmml" xref="S3.E5.m1.2.2.4.3.3">𝑑</ci></apply></apply><apply id="S3.E5.m1.2.2.2.cmml" xref="S3.E5.m1.2.2.2"><times id="S3.E5.m1.2.2.2.3.cmml" xref="S3.E5.m1.2.2.2.3"></times><apply id="S3.E5.m1.1.1.1.1.2.cmml" xref="S3.E5.m1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E5.m1.1.1.1.1.2.1.cmml" xref="S3.E5.m1.1.1.1.1.1.2">norm</csymbol><apply id="S3.E5.m1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.E5.m1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.E5.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E5.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.E5.m1.1.1.1.1.1.1.2.2">𝑂</ci><ci id="S3.E5.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E5.m1.1.1.1.1.1.1.2.3">𝑞</ci></apply><ci id="S3.E5.m1.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.1.1.1.1.1.1.3">𝑇</ci></apply></apply><apply id="S3.E5.m1.2.2.2.2.2.cmml" xref="S3.E5.m1.2.2.2.2.1"><csymbol cd="latexml" id="S3.E5.m1.2.2.2.2.2.1.cmml" xref="S3.E5.m1.2.2.2.2.1.2">delimited-|‖</csymbol><apply id="S3.E5.m1.2.2.2.2.1.1.cmml" xref="S3.E5.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.2.2.1.1.1.cmml" xref="S3.E5.m1.2.2.2.2.1.1">subscript</csymbol><ci id="S3.E5.m1.2.2.2.2.1.1.2.cmml" xref="S3.E5.m1.2.2.2.2.1.1.2">𝑂</ci><ci id="S3.E5.m1.2.2.2.2.1.1.3.cmml" xref="S3.E5.m1.2.2.2.2.1.1.3">𝑑</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5.m1.5c">\cos({O}_{q},{O}_{d})={\frac{{O}_{q}^{T}{O}_{d}}{\|{O}_{q}^{T}\||{O}_{d}\|}}</annotation><annotation encoding="application/x-llamapun" id="S3.E5.m1.5d">roman_cos ( italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT , italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ) = divide start_ARG italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_ARG start_ARG ∥ italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT ∥ | italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ∥ end_ARG</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx1.p11"> <p class="ltx_p" id="S3.SSx1.p11.4">When a query <math alttext="q^{\prime}" class="ltx_Math" display="inline" id="S3.SSx1.p11.1.m1.1"><semantics id="S3.SSx1.p11.1.m1.1a"><msup id="S3.SSx1.p11.1.m1.1.1" xref="S3.SSx1.p11.1.m1.1.1.cmml"><mi id="S3.SSx1.p11.1.m1.1.1.2" xref="S3.SSx1.p11.1.m1.1.1.2.cmml">q</mi><mo id="S3.SSx1.p11.1.m1.1.1.3" xref="S3.SSx1.p11.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SSx1.p11.1.m1.1b"><apply id="S3.SSx1.p11.1.m1.1.1.cmml" xref="S3.SSx1.p11.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SSx1.p11.1.m1.1.1.1.cmml" xref="S3.SSx1.p11.1.m1.1.1">superscript</csymbol><ci id="S3.SSx1.p11.1.m1.1.1.2.cmml" xref="S3.SSx1.p11.1.m1.1.1.2">𝑞</ci><ci id="S3.SSx1.p11.1.m1.1.1.3.cmml" xref="S3.SSx1.p11.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p11.1.m1.1c">q^{\prime}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p11.1.m1.1d">italic_q start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT</annotation></semantics></math> and its corresponding Web page titles come, we can calculate the semantic relevance between <math alttext="q^{\prime}" class="ltx_Math" display="inline" id="S3.SSx1.p11.2.m2.1"><semantics id="S3.SSx1.p11.2.m2.1a"><msup id="S3.SSx1.p11.2.m2.1.1" xref="S3.SSx1.p11.2.m2.1.1.cmml"><mi id="S3.SSx1.p11.2.m2.1.1.2" xref="S3.SSx1.p11.2.m2.1.1.2.cmml">q</mi><mo id="S3.SSx1.p11.2.m2.1.1.3" xref="S3.SSx1.p11.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SSx1.p11.2.m2.1b"><apply id="S3.SSx1.p11.2.m2.1.1.cmml" xref="S3.SSx1.p11.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SSx1.p11.2.m2.1.1.1.cmml" xref="S3.SSx1.p11.2.m2.1.1">superscript</csymbol><ci id="S3.SSx1.p11.2.m2.1.1.2.cmml" xref="S3.SSx1.p11.2.m2.1.1.2">𝑞</ci><ci id="S3.SSx1.p11.2.m2.1.1.3.cmml" xref="S3.SSx1.p11.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p11.2.m2.1c">q^{\prime}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p11.2.m2.1d">italic_q start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT</annotation></semantics></math> and the title of a Web page title <math alttext="t^{\prime}" class="ltx_Math" display="inline" id="S3.SSx1.p11.3.m3.1"><semantics id="S3.SSx1.p11.3.m3.1a"><msup id="S3.SSx1.p11.3.m3.1.1" xref="S3.SSx1.p11.3.m3.1.1.cmml"><mi id="S3.SSx1.p11.3.m3.1.1.2" xref="S3.SSx1.p11.3.m3.1.1.2.cmml">t</mi><mo id="S3.SSx1.p11.3.m3.1.1.3" xref="S3.SSx1.p11.3.m3.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SSx1.p11.3.m3.1b"><apply id="S3.SSx1.p11.3.m3.1.1.cmml" xref="S3.SSx1.p11.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SSx1.p11.3.m3.1.1.1.cmml" xref="S3.SSx1.p11.3.m3.1.1">superscript</csymbol><ci id="S3.SSx1.p11.3.m3.1.1.2.cmml" xref="S3.SSx1.p11.3.m3.1.1.2">𝑡</ci><ci id="S3.SSx1.p11.3.m3.1.1.3.cmml" xref="S3.SSx1.p11.3.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p11.3.m3.1c">t^{\prime}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p11.3.m3.1d">italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT</annotation></semantics></math> by <math alttext="\cos\langle f(q^{\prime}),f(t^{\prime})\rangle" class="ltx_Math" display="inline" id="S3.SSx1.p11.4.m4.3"><semantics id="S3.SSx1.p11.4.m4.3a"><mrow id="S3.SSx1.p11.4.m4.3.3.2" xref="S3.SSx1.p11.4.m4.3.3.3.cmml"><mi id="S3.SSx1.p11.4.m4.1.1" xref="S3.SSx1.p11.4.m4.1.1.cmml">cos</mi><mo id="S3.SSx1.p11.4.m4.3.3.2a" xref="S3.SSx1.p11.4.m4.3.3.3.cmml">⁡</mo><mrow id="S3.SSx1.p11.4.m4.3.3.2.2" xref="S3.SSx1.p11.4.m4.3.3.3.cmml"><mo id="S3.SSx1.p11.4.m4.3.3.2.2.3" stretchy="false" xref="S3.SSx1.p11.4.m4.3.3.3.cmml">⟨</mo><mrow id="S3.SSx1.p11.4.m4.2.2.1.1.1" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.cmml"><mi id="S3.SSx1.p11.4.m4.2.2.1.1.1.3" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.3.cmml">f</mi><mo id="S3.SSx1.p11.4.m4.2.2.1.1.1.2" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.2.cmml">⁢</mo><mrow id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.cmml"><mo id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.2" stretchy="false" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.cmml">(</mo><msup id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.cmml"><mi id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.2" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.2.cmml">q</mi><mo id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.3" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.3.cmml">′</mo></msup><mo id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.3" stretchy="false" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SSx1.p11.4.m4.3.3.2.2.4" xref="S3.SSx1.p11.4.m4.3.3.3.cmml">,</mo><mrow id="S3.SSx1.p11.4.m4.3.3.2.2.2" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.cmml"><mi id="S3.SSx1.p11.4.m4.3.3.2.2.2.3" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.3.cmml">f</mi><mo id="S3.SSx1.p11.4.m4.3.3.2.2.2.2" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.2.cmml">⁢</mo><mrow id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.cmml"><mo id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.2" stretchy="false" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.cmml">(</mo><msup id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.cmml"><mi id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.2" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.2.cmml">t</mi><mo id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.3" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.3.cmml">′</mo></msup><mo id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.3" stretchy="false" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SSx1.p11.4.m4.3.3.2.2.5" stretchy="false" xref="S3.SSx1.p11.4.m4.3.3.3.cmml">⟩</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx1.p11.4.m4.3b"><apply id="S3.SSx1.p11.4.m4.3.3.3.cmml" xref="S3.SSx1.p11.4.m4.3.3.2"><cos id="S3.SSx1.p11.4.m4.1.1.cmml" xref="S3.SSx1.p11.4.m4.1.1"></cos><apply id="S3.SSx1.p11.4.m4.2.2.1.1.1.cmml" xref="S3.SSx1.p11.4.m4.2.2.1.1.1"><times id="S3.SSx1.p11.4.m4.2.2.1.1.1.2.cmml" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.2"></times><ci id="S3.SSx1.p11.4.m4.2.2.1.1.1.3.cmml" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.3">𝑓</ci><apply id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.cmml" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.1.cmml" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1">superscript</csymbol><ci id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.2.cmml" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.3.cmml" xref="S3.SSx1.p11.4.m4.2.2.1.1.1.1.1.1.3">′</ci></apply></apply><apply id="S3.SSx1.p11.4.m4.3.3.2.2.2.cmml" xref="S3.SSx1.p11.4.m4.3.3.2.2.2"><times id="S3.SSx1.p11.4.m4.3.3.2.2.2.2.cmml" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.2"></times><ci id="S3.SSx1.p11.4.m4.3.3.2.2.2.3.cmml" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.3">𝑓</ci><apply id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.cmml" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.1.cmml" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1">superscript</csymbol><ci id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.2.cmml" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.2">𝑡</ci><ci id="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.3.cmml" xref="S3.SSx1.p11.4.m4.3.3.2.2.2.1.1.1.3">′</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx1.p11.4.m4.3c">\cos\langle f(q^{\prime}),f(t^{\prime})\rangle</annotation><annotation encoding="application/x-llamapun" id="S3.SSx1.p11.4.m4.3d">roman_cos ⟨ italic_f ( italic_q start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) , italic_f ( italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) ⟩</annotation></semantics></math>, which can be directly utilized for ranking or work as a feature for sophisticated ranking algorithms.</p> </div> </section> <section class="ltx_subsection" id="S3.SSx2"> <h3 class="ltx_title ltx_title_subsection">Optimization</h3> <div class="ltx_para" id="S3.SSx2.p1"> <p class="ltx_p" id="S3.SSx2.p1.2">The neural network parameters and the word embeddings are updated by conventional backpropagation. The SEM model is trained using stochastic gradient descent. Let <math alttext="\Lambda" class="ltx_Math" display="inline" id="S3.SSx2.p1.1.m1.1"><semantics id="S3.SSx2.p1.1.m1.1a"><mi id="S3.SSx2.p1.1.m1.1.1" mathvariant="normal" xref="S3.SSx2.p1.1.m1.1.1.cmml">Λ</mi><annotation-xml encoding="MathML-Content" id="S3.SSx2.p1.1.m1.1b"><ci id="S3.SSx2.p1.1.m1.1.1.cmml" xref="S3.SSx2.p1.1.m1.1.1">Λ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p1.1.m1.1c">\Lambda</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p1.1.m1.1d">roman_Λ</annotation></semantics></math> be the parameters and <math alttext="\Delta=\cos\langle f(q_{i}),f(p_{q_{i}})\rangle" class="ltx_Math" display="inline" id="S3.SSx2.p1.2.m2.3"><semantics id="S3.SSx2.p1.2.m2.3a"><mrow id="S3.SSx2.p1.2.m2.3.3" xref="S3.SSx2.p1.2.m2.3.3.cmml"><mi id="S3.SSx2.p1.2.m2.3.3.4" mathvariant="normal" xref="S3.SSx2.p1.2.m2.3.3.4.cmml">Δ</mi><mo id="S3.SSx2.p1.2.m2.3.3.3" xref="S3.SSx2.p1.2.m2.3.3.3.cmml">=</mo><mrow id="S3.SSx2.p1.2.m2.3.3.2.2" xref="S3.SSx2.p1.2.m2.3.3.2.3.cmml"><mi id="S3.SSx2.p1.2.m2.1.1" xref="S3.SSx2.p1.2.m2.1.1.cmml">cos</mi><mo id="S3.SSx2.p1.2.m2.3.3.2.2a" xref="S3.SSx2.p1.2.m2.3.3.2.3.cmml">⁡</mo><mrow id="S3.SSx2.p1.2.m2.3.3.2.2.2" xref="S3.SSx2.p1.2.m2.3.3.2.3.cmml"><mo id="S3.SSx2.p1.2.m2.3.3.2.2.2.3" stretchy="false" xref="S3.SSx2.p1.2.m2.3.3.2.3.cmml">⟨</mo><mrow id="S3.SSx2.p1.2.m2.2.2.1.1.1.1" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.cmml"><mi id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.3" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.3.cmml">f</mi><mo id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.2" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.2.cmml">⁢</mo><mrow id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.cmml"><mo id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.2" stretchy="false" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.cmml"><mi id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.2" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.2.cmml">q</mi><mi id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.3" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.3" stretchy="false" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SSx2.p1.2.m2.3.3.2.2.2.4" xref="S3.SSx2.p1.2.m2.3.3.2.3.cmml">,</mo><mrow id="S3.SSx2.p1.2.m2.3.3.2.2.2.2" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.cmml"><mi id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.3" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.3.cmml">f</mi><mo id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.2" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.2.cmml">⁢</mo><mrow id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.cmml"><mo id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.2" stretchy="false" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.cmml">(</mo><msub id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.cmml"><mi id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.2" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.2.cmml">p</mi><msub id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.cmml"><mi id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.2" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.2.cmml">q</mi><mi id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.3" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.3.cmml">i</mi></msub></msub><mo id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.3" stretchy="false" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SSx2.p1.2.m2.3.3.2.2.2.5" stretchy="false" xref="S3.SSx2.p1.2.m2.3.3.2.3.cmml">⟩</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx2.p1.2.m2.3b"><apply id="S3.SSx2.p1.2.m2.3.3.cmml" xref="S3.SSx2.p1.2.m2.3.3"><eq id="S3.SSx2.p1.2.m2.3.3.3.cmml" xref="S3.SSx2.p1.2.m2.3.3.3"></eq><ci id="S3.SSx2.p1.2.m2.3.3.4.cmml" xref="S3.SSx2.p1.2.m2.3.3.4">Δ</ci><apply id="S3.SSx2.p1.2.m2.3.3.2.3.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2"><cos id="S3.SSx2.p1.2.m2.1.1.cmml" xref="S3.SSx2.p1.2.m2.1.1"></cos><apply id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.cmml" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1"><times id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.2.cmml" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.2"></times><ci id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.3.cmml" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.3">𝑓</ci><apply id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.cmml" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.2.cmml" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.3.cmml" xref="S3.SSx2.p1.2.m2.2.2.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2"><times id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.2.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.2"></times><ci id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.3.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.3">𝑓</ci><apply id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.1.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1">subscript</csymbol><ci id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.2.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.2">𝑝</ci><apply id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.1.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3">subscript</csymbol><ci id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.2.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.2">𝑞</ci><ci id="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.3.cmml" xref="S3.SSx2.p1.2.m2.3.3.2.2.2.2.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p1.2.m2.3c">\Delta=\cos\langle f(q_{i}),f(p_{q_{i}})\rangle</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p1.2.m2.3d">roman_Δ = roman_cos ⟨ italic_f ( italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) , italic_f ( italic_p start_POSTSUBSCRIPT italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) ⟩</annotation></semantics></math>, they are updated as follows:</p> </div> <div class="ltx_para" id="S3.SSx2.p2"> <table class="ltx_equation ltx_eqn_table" id="S3.E6"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(6)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="\Lambda_{t}=\Lambda_{t-1}-\gamma_{t}\frac{\partial\Delta}{\partial\Lambda_{t-1}}" class="ltx_Math" display="block" id="S3.E6.m1.1"><semantics id="S3.E6.m1.1a"><mrow id="S3.E6.m1.1.1" xref="S3.E6.m1.1.1.cmml"><msub id="S3.E6.m1.1.1.2" xref="S3.E6.m1.1.1.2.cmml"><mi id="S3.E6.m1.1.1.2.2" mathvariant="normal" xref="S3.E6.m1.1.1.2.2.cmml">Λ</mi><mi id="S3.E6.m1.1.1.2.3" xref="S3.E6.m1.1.1.2.3.cmml">t</mi></msub><mo id="S3.E6.m1.1.1.1" xref="S3.E6.m1.1.1.1.cmml">=</mo><mrow id="S3.E6.m1.1.1.3" xref="S3.E6.m1.1.1.3.cmml"><msub id="S3.E6.m1.1.1.3.2" xref="S3.E6.m1.1.1.3.2.cmml"><mi id="S3.E6.m1.1.1.3.2.2" mathvariant="normal" xref="S3.E6.m1.1.1.3.2.2.cmml">Λ</mi><mrow id="S3.E6.m1.1.1.3.2.3" xref="S3.E6.m1.1.1.3.2.3.cmml"><mi id="S3.E6.m1.1.1.3.2.3.2" xref="S3.E6.m1.1.1.3.2.3.2.cmml">t</mi><mo id="S3.E6.m1.1.1.3.2.3.1" xref="S3.E6.m1.1.1.3.2.3.1.cmml">−</mo><mn id="S3.E6.m1.1.1.3.2.3.3" xref="S3.E6.m1.1.1.3.2.3.3.cmml">1</mn></mrow></msub><mo id="S3.E6.m1.1.1.3.1" xref="S3.E6.m1.1.1.3.1.cmml">−</mo><mrow id="S3.E6.m1.1.1.3.3" xref="S3.E6.m1.1.1.3.3.cmml"><msub id="S3.E6.m1.1.1.3.3.2" xref="S3.E6.m1.1.1.3.3.2.cmml"><mi id="S3.E6.m1.1.1.3.3.2.2" xref="S3.E6.m1.1.1.3.3.2.2.cmml">γ</mi><mi id="S3.E6.m1.1.1.3.3.2.3" xref="S3.E6.m1.1.1.3.3.2.3.cmml">t</mi></msub><mo id="S3.E6.m1.1.1.3.3.1" xref="S3.E6.m1.1.1.3.3.1.cmml">⁢</mo><mfrac id="S3.E6.m1.1.1.3.3.3" xref="S3.E6.m1.1.1.3.3.3.cmml"><mrow id="S3.E6.m1.1.1.3.3.3.2" xref="S3.E6.m1.1.1.3.3.3.2.cmml"><mo id="S3.E6.m1.1.1.3.3.3.2.1" rspace="0em" xref="S3.E6.m1.1.1.3.3.3.2.1.cmml">∂</mo><mi id="S3.E6.m1.1.1.3.3.3.2.2" mathvariant="normal" xref="S3.E6.m1.1.1.3.3.3.2.2.cmml">Δ</mi></mrow><mrow id="S3.E6.m1.1.1.3.3.3.3" xref="S3.E6.m1.1.1.3.3.3.3.cmml"><mo id="S3.E6.m1.1.1.3.3.3.3.1" rspace="0em" xref="S3.E6.m1.1.1.3.3.3.3.1.cmml">∂</mo><msub id="S3.E6.m1.1.1.3.3.3.3.2" xref="S3.E6.m1.1.1.3.3.3.3.2.cmml"><mi id="S3.E6.m1.1.1.3.3.3.3.2.2" mathvariant="normal" xref="S3.E6.m1.1.1.3.3.3.3.2.2.cmml">Λ</mi><mrow id="S3.E6.m1.1.1.3.3.3.3.2.3" xref="S3.E6.m1.1.1.3.3.3.3.2.3.cmml"><mi id="S3.E6.m1.1.1.3.3.3.3.2.3.2" xref="S3.E6.m1.1.1.3.3.3.3.2.3.2.cmml">t</mi><mo id="S3.E6.m1.1.1.3.3.3.3.2.3.1" xref="S3.E6.m1.1.1.3.3.3.3.2.3.1.cmml">−</mo><mn id="S3.E6.m1.1.1.3.3.3.3.2.3.3" xref="S3.E6.m1.1.1.3.3.3.3.2.3.3.cmml">1</mn></mrow></msub></mrow></mfrac></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E6.m1.1b"><apply id="S3.E6.m1.1.1.cmml" xref="S3.E6.m1.1.1"><eq id="S3.E6.m1.1.1.1.cmml" xref="S3.E6.m1.1.1.1"></eq><apply id="S3.E6.m1.1.1.2.cmml" xref="S3.E6.m1.1.1.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.2.1.cmml" xref="S3.E6.m1.1.1.2">subscript</csymbol><ci id="S3.E6.m1.1.1.2.2.cmml" xref="S3.E6.m1.1.1.2.2">Λ</ci><ci id="S3.E6.m1.1.1.2.3.cmml" xref="S3.E6.m1.1.1.2.3">𝑡</ci></apply><apply id="S3.E6.m1.1.1.3.cmml" xref="S3.E6.m1.1.1.3"><minus id="S3.E6.m1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.3.1"></minus><apply id="S3.E6.m1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.3.2.1.cmml" xref="S3.E6.m1.1.1.3.2">subscript</csymbol><ci id="S3.E6.m1.1.1.3.2.2.cmml" xref="S3.E6.m1.1.1.3.2.2">Λ</ci><apply id="S3.E6.m1.1.1.3.2.3.cmml" xref="S3.E6.m1.1.1.3.2.3"><minus id="S3.E6.m1.1.1.3.2.3.1.cmml" xref="S3.E6.m1.1.1.3.2.3.1"></minus><ci id="S3.E6.m1.1.1.3.2.3.2.cmml" xref="S3.E6.m1.1.1.3.2.3.2">𝑡</ci><cn id="S3.E6.m1.1.1.3.2.3.3.cmml" type="integer" xref="S3.E6.m1.1.1.3.2.3.3">1</cn></apply></apply><apply id="S3.E6.m1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.3.3"><times id="S3.E6.m1.1.1.3.3.1.cmml" xref="S3.E6.m1.1.1.3.3.1"></times><apply id="S3.E6.m1.1.1.3.3.2.cmml" xref="S3.E6.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.3.3.2.1.cmml" xref="S3.E6.m1.1.1.3.3.2">subscript</csymbol><ci id="S3.E6.m1.1.1.3.3.2.2.cmml" xref="S3.E6.m1.1.1.3.3.2.2">𝛾</ci><ci id="S3.E6.m1.1.1.3.3.2.3.cmml" xref="S3.E6.m1.1.1.3.3.2.3">𝑡</ci></apply><apply id="S3.E6.m1.1.1.3.3.3.cmml" xref="S3.E6.m1.1.1.3.3.3"><divide id="S3.E6.m1.1.1.3.3.3.1.cmml" xref="S3.E6.m1.1.1.3.3.3"></divide><apply id="S3.E6.m1.1.1.3.3.3.2.cmml" xref="S3.E6.m1.1.1.3.3.3.2"><partialdiff id="S3.E6.m1.1.1.3.3.3.2.1.cmml" xref="S3.E6.m1.1.1.3.3.3.2.1"></partialdiff><ci id="S3.E6.m1.1.1.3.3.3.2.2.cmml" xref="S3.E6.m1.1.1.3.3.3.2.2">Δ</ci></apply><apply id="S3.E6.m1.1.1.3.3.3.3.cmml" xref="S3.E6.m1.1.1.3.3.3.3"><partialdiff id="S3.E6.m1.1.1.3.3.3.3.1.cmml" xref="S3.E6.m1.1.1.3.3.3.3.1"></partialdiff><apply id="S3.E6.m1.1.1.3.3.3.3.2.cmml" xref="S3.E6.m1.1.1.3.3.3.3.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.3.3.3.3.2.1.cmml" xref="S3.E6.m1.1.1.3.3.3.3.2">subscript</csymbol><ci id="S3.E6.m1.1.1.3.3.3.3.2.2.cmml" xref="S3.E6.m1.1.1.3.3.3.3.2.2">Λ</ci><apply id="S3.E6.m1.1.1.3.3.3.3.2.3.cmml" xref="S3.E6.m1.1.1.3.3.3.3.2.3"><minus id="S3.E6.m1.1.1.3.3.3.3.2.3.1.cmml" xref="S3.E6.m1.1.1.3.3.3.3.2.3.1"></minus><ci id="S3.E6.m1.1.1.3.3.3.3.2.3.2.cmml" xref="S3.E6.m1.1.1.3.3.3.3.2.3.2">𝑡</ci><cn id="S3.E6.m1.1.1.3.3.3.3.2.3.3.cmml" type="integer" xref="S3.E6.m1.1.1.3.3.3.3.2.3.3">1</cn></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6.m1.1c">\Lambda_{t}=\Lambda_{t-1}-\gamma_{t}\frac{\partial\Delta}{\partial\Lambda_{t-1}}</annotation><annotation encoding="application/x-llamapun" id="S3.E6.m1.1d">roman_Λ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = roman_Λ start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT - italic_γ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT divide start_ARG ∂ roman_Δ end_ARG start_ARG ∂ roman_Λ start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT end_ARG</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx2.p3"> <p class="ltx_p" id="S3.SSx2.p3.6">where <math alttext="\Lambda_{t}" class="ltx_Math" display="inline" id="S3.SSx2.p3.1.m1.1"><semantics id="S3.SSx2.p3.1.m1.1a"><msub id="S3.SSx2.p3.1.m1.1.1" xref="S3.SSx2.p3.1.m1.1.1.cmml"><mi id="S3.SSx2.p3.1.m1.1.1.2" mathvariant="normal" xref="S3.SSx2.p3.1.m1.1.1.2.cmml">Λ</mi><mi id="S3.SSx2.p3.1.m1.1.1.3" xref="S3.SSx2.p3.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SSx2.p3.1.m1.1b"><apply id="S3.SSx2.p3.1.m1.1.1.cmml" xref="S3.SSx2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p3.1.m1.1.1.1.cmml" xref="S3.SSx2.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SSx2.p3.1.m1.1.1.2.cmml" xref="S3.SSx2.p3.1.m1.1.1.2">Λ</ci><ci id="S3.SSx2.p3.1.m1.1.1.3.cmml" xref="S3.SSx2.p3.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p3.1.m1.1c">\Lambda_{t}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p3.1.m1.1d">roman_Λ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT</annotation></semantics></math> and <math alttext="\Lambda_{t-1}" class="ltx_Math" display="inline" id="S3.SSx2.p3.2.m2.1"><semantics id="S3.SSx2.p3.2.m2.1a"><msub id="S3.SSx2.p3.2.m2.1.1" xref="S3.SSx2.p3.2.m2.1.1.cmml"><mi id="S3.SSx2.p3.2.m2.1.1.2" mathvariant="normal" xref="S3.SSx2.p3.2.m2.1.1.2.cmml">Λ</mi><mrow id="S3.SSx2.p3.2.m2.1.1.3" xref="S3.SSx2.p3.2.m2.1.1.3.cmml"><mi id="S3.SSx2.p3.2.m2.1.1.3.2" xref="S3.SSx2.p3.2.m2.1.1.3.2.cmml">t</mi><mo id="S3.SSx2.p3.2.m2.1.1.3.1" xref="S3.SSx2.p3.2.m2.1.1.3.1.cmml">−</mo><mn id="S3.SSx2.p3.2.m2.1.1.3.3" xref="S3.SSx2.p3.2.m2.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SSx2.p3.2.m2.1b"><apply id="S3.SSx2.p3.2.m2.1.1.cmml" xref="S3.SSx2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p3.2.m2.1.1.1.cmml" xref="S3.SSx2.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SSx2.p3.2.m2.1.1.2.cmml" xref="S3.SSx2.p3.2.m2.1.1.2">Λ</ci><apply id="S3.SSx2.p3.2.m2.1.1.3.cmml" xref="S3.SSx2.p3.2.m2.1.1.3"><minus id="S3.SSx2.p3.2.m2.1.1.3.1.cmml" xref="S3.SSx2.p3.2.m2.1.1.3.1"></minus><ci id="S3.SSx2.p3.2.m2.1.1.3.2.cmml" xref="S3.SSx2.p3.2.m2.1.1.3.2">𝑡</ci><cn id="S3.SSx2.p3.2.m2.1.1.3.3.cmml" type="integer" xref="S3.SSx2.p3.2.m2.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p3.2.m2.1c">\Lambda_{t-1}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p3.2.m2.1d">roman_Λ start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT</annotation></semantics></math> are the model parameters at <math alttext="t^{th}" class="ltx_Math" display="inline" id="S3.SSx2.p3.3.m3.1"><semantics id="S3.SSx2.p3.3.m3.1a"><msup id="S3.SSx2.p3.3.m3.1.1" xref="S3.SSx2.p3.3.m3.1.1.cmml"><mi id="S3.SSx2.p3.3.m3.1.1.2" xref="S3.SSx2.p3.3.m3.1.1.2.cmml">t</mi><mrow id="S3.SSx2.p3.3.m3.1.1.3" xref="S3.SSx2.p3.3.m3.1.1.3.cmml"><mi id="S3.SSx2.p3.3.m3.1.1.3.2" xref="S3.SSx2.p3.3.m3.1.1.3.2.cmml">t</mi><mo id="S3.SSx2.p3.3.m3.1.1.3.1" xref="S3.SSx2.p3.3.m3.1.1.3.1.cmml">⁢</mo><mi id="S3.SSx2.p3.3.m3.1.1.3.3" xref="S3.SSx2.p3.3.m3.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SSx2.p3.3.m3.1b"><apply id="S3.SSx2.p3.3.m3.1.1.cmml" xref="S3.SSx2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p3.3.m3.1.1.1.cmml" xref="S3.SSx2.p3.3.m3.1.1">superscript</csymbol><ci id="S3.SSx2.p3.3.m3.1.1.2.cmml" xref="S3.SSx2.p3.3.m3.1.1.2">𝑡</ci><apply id="S3.SSx2.p3.3.m3.1.1.3.cmml" xref="S3.SSx2.p3.3.m3.1.1.3"><times id="S3.SSx2.p3.3.m3.1.1.3.1.cmml" xref="S3.SSx2.p3.3.m3.1.1.3.1"></times><ci id="S3.SSx2.p3.3.m3.1.1.3.2.cmml" xref="S3.SSx2.p3.3.m3.1.1.3.2">𝑡</ci><ci id="S3.SSx2.p3.3.m3.1.1.3.3.cmml" xref="S3.SSx2.p3.3.m3.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p3.3.m3.1c">t^{th}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p3.3.m3.1d">italic_t start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT</annotation></semantics></math> iteration and <math alttext="(t-1)^{th}" class="ltx_Math" display="inline" id="S3.SSx2.p3.4.m4.1"><semantics id="S3.SSx2.p3.4.m4.1a"><msup id="S3.SSx2.p3.4.m4.1.1" xref="S3.SSx2.p3.4.m4.1.1.cmml"><mrow id="S3.SSx2.p3.4.m4.1.1.1.1" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.cmml"><mo id="S3.SSx2.p3.4.m4.1.1.1.1.2" stretchy="false" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.cmml">(</mo><mrow id="S3.SSx2.p3.4.m4.1.1.1.1.1" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.cmml"><mi id="S3.SSx2.p3.4.m4.1.1.1.1.1.2" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.2.cmml">t</mi><mo id="S3.SSx2.p3.4.m4.1.1.1.1.1.1" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.1.cmml">−</mo><mn id="S3.SSx2.p3.4.m4.1.1.1.1.1.3" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.3.cmml">1</mn></mrow><mo id="S3.SSx2.p3.4.m4.1.1.1.1.3" stretchy="false" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.cmml">)</mo></mrow><mrow id="S3.SSx2.p3.4.m4.1.1.3" xref="S3.SSx2.p3.4.m4.1.1.3.cmml"><mi id="S3.SSx2.p3.4.m4.1.1.3.2" xref="S3.SSx2.p3.4.m4.1.1.3.2.cmml">t</mi><mo id="S3.SSx2.p3.4.m4.1.1.3.1" xref="S3.SSx2.p3.4.m4.1.1.3.1.cmml">⁢</mo><mi id="S3.SSx2.p3.4.m4.1.1.3.3" xref="S3.SSx2.p3.4.m4.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SSx2.p3.4.m4.1b"><apply id="S3.SSx2.p3.4.m4.1.1.cmml" xref="S3.SSx2.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p3.4.m4.1.1.2.cmml" xref="S3.SSx2.p3.4.m4.1.1">superscript</csymbol><apply id="S3.SSx2.p3.4.m4.1.1.1.1.1.cmml" xref="S3.SSx2.p3.4.m4.1.1.1.1"><minus id="S3.SSx2.p3.4.m4.1.1.1.1.1.1.cmml" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.1"></minus><ci id="S3.SSx2.p3.4.m4.1.1.1.1.1.2.cmml" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.2">𝑡</ci><cn id="S3.SSx2.p3.4.m4.1.1.1.1.1.3.cmml" type="integer" xref="S3.SSx2.p3.4.m4.1.1.1.1.1.3">1</cn></apply><apply id="S3.SSx2.p3.4.m4.1.1.3.cmml" xref="S3.SSx2.p3.4.m4.1.1.3"><times id="S3.SSx2.p3.4.m4.1.1.3.1.cmml" xref="S3.SSx2.p3.4.m4.1.1.3.1"></times><ci id="S3.SSx2.p3.4.m4.1.1.3.2.cmml" xref="S3.SSx2.p3.4.m4.1.1.3.2">𝑡</ci><ci id="S3.SSx2.p3.4.m4.1.1.3.3.cmml" xref="S3.SSx2.p3.4.m4.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p3.4.m4.1c">(t-1)^{th}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p3.4.m4.1d">( italic_t - 1 ) start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT</annotation></semantics></math> iteration respectively. And <math alttext="\gamma_{t}" class="ltx_Math" display="inline" id="S3.SSx2.p3.5.m5.1"><semantics id="S3.SSx2.p3.5.m5.1a"><msub id="S3.SSx2.p3.5.m5.1.1" xref="S3.SSx2.p3.5.m5.1.1.cmml"><mi id="S3.SSx2.p3.5.m5.1.1.2" xref="S3.SSx2.p3.5.m5.1.1.2.cmml">γ</mi><mi id="S3.SSx2.p3.5.m5.1.1.3" xref="S3.SSx2.p3.5.m5.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SSx2.p3.5.m5.1b"><apply id="S3.SSx2.p3.5.m5.1.1.cmml" xref="S3.SSx2.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p3.5.m5.1.1.1.cmml" xref="S3.SSx2.p3.5.m5.1.1">subscript</csymbol><ci id="S3.SSx2.p3.5.m5.1.1.2.cmml" xref="S3.SSx2.p3.5.m5.1.1.2">𝛾</ci><ci id="S3.SSx2.p3.5.m5.1.1.3.cmml" xref="S3.SSx2.p3.5.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p3.5.m5.1c">\gamma_{t}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p3.5.m5.1d">italic_γ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT</annotation></semantics></math> is the learning rate at <math alttext="t^{th}" class="ltx_Math" display="inline" id="S3.SSx2.p3.6.m6.1"><semantics id="S3.SSx2.p3.6.m6.1a"><msup id="S3.SSx2.p3.6.m6.1.1" xref="S3.SSx2.p3.6.m6.1.1.cmml"><mi id="S3.SSx2.p3.6.m6.1.1.2" xref="S3.SSx2.p3.6.m6.1.1.2.cmml">t</mi><mrow id="S3.SSx2.p3.6.m6.1.1.3" xref="S3.SSx2.p3.6.m6.1.1.3.cmml"><mi id="S3.SSx2.p3.6.m6.1.1.3.2" xref="S3.SSx2.p3.6.m6.1.1.3.2.cmml">t</mi><mo id="S3.SSx2.p3.6.m6.1.1.3.1" xref="S3.SSx2.p3.6.m6.1.1.3.1.cmml">⁢</mo><mi id="S3.SSx2.p3.6.m6.1.1.3.3" xref="S3.SSx2.p3.6.m6.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SSx2.p3.6.m6.1b"><apply id="S3.SSx2.p3.6.m6.1.1.cmml" xref="S3.SSx2.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p3.6.m6.1.1.1.cmml" xref="S3.SSx2.p3.6.m6.1.1">superscript</csymbol><ci id="S3.SSx2.p3.6.m6.1.1.2.cmml" xref="S3.SSx2.p3.6.m6.1.1.2">𝑡</ci><apply id="S3.SSx2.p3.6.m6.1.1.3.cmml" xref="S3.SSx2.p3.6.m6.1.1.3"><times id="S3.SSx2.p3.6.m6.1.1.3.1.cmml" xref="S3.SSx2.p3.6.m6.1.1.3.1"></times><ci id="S3.SSx2.p3.6.m6.1.1.3.2.cmml" xref="S3.SSx2.p3.6.m6.1.1.3.2">𝑡</ci><ci id="S3.SSx2.p3.6.m6.1.1.3.3.cmml" xref="S3.SSx2.p3.6.m6.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p3.6.m6.1c">t^{th}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p3.6.m6.1d">italic_t start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT</annotation></semantics></math> iteration. This process is conducted for all training instances and repeated for several iterations until convergence is achieved. We derive the model parameters gradient as follows:</p> </div> <div class="ltx_para" id="S3.SSx2.p4"> <table class="ltx_equation ltx_eqn_table" id="S3.E7"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(7)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="\frac{\partial\Delta}{\partial\Lambda_{t-1}}=\frac{\partial\cos(O_{q},O_{d+})}% {\partial\Lambda_{t-1}}-\frac{\partial\cos(O_{q},O_{d-})}{\partial\Lambda_{t-1}}" class="ltx_Math" display="block" id="S3.E7.m1.6"><semantics id="S3.E7.m1.6a"><mrow id="S3.E7.m1.6.7" xref="S3.E7.m1.6.7.cmml"><mfrac id="S3.E7.m1.6.7.2" xref="S3.E7.m1.6.7.2.cmml"><mrow id="S3.E7.m1.6.7.2.2" xref="S3.E7.m1.6.7.2.2.cmml"><mo id="S3.E7.m1.6.7.2.2.1" rspace="0em" xref="S3.E7.m1.6.7.2.2.1.cmml">∂</mo><mi id="S3.E7.m1.6.7.2.2.2" mathvariant="normal" xref="S3.E7.m1.6.7.2.2.2.cmml">Δ</mi></mrow><mrow id="S3.E7.m1.6.7.2.3" xref="S3.E7.m1.6.7.2.3.cmml"><mo id="S3.E7.m1.6.7.2.3.1" rspace="0em" xref="S3.E7.m1.6.7.2.3.1.cmml">∂</mo><msub id="S3.E7.m1.6.7.2.3.2" xref="S3.E7.m1.6.7.2.3.2.cmml"><mi id="S3.E7.m1.6.7.2.3.2.2" mathvariant="normal" xref="S3.E7.m1.6.7.2.3.2.2.cmml">Λ</mi><mrow id="S3.E7.m1.6.7.2.3.2.3" xref="S3.E7.m1.6.7.2.3.2.3.cmml"><mi id="S3.E7.m1.6.7.2.3.2.3.2" xref="S3.E7.m1.6.7.2.3.2.3.2.cmml">t</mi><mo id="S3.E7.m1.6.7.2.3.2.3.1" xref="S3.E7.m1.6.7.2.3.2.3.1.cmml">−</mo><mn id="S3.E7.m1.6.7.2.3.2.3.3" xref="S3.E7.m1.6.7.2.3.2.3.3.cmml">1</mn></mrow></msub></mrow></mfrac><mo id="S3.E7.m1.6.7.1" xref="S3.E7.m1.6.7.1.cmml">=</mo><mrow id="S3.E7.m1.6.7.3" xref="S3.E7.m1.6.7.3.cmml"><mfrac id="S3.E7.m1.3.3" xref="S3.E7.m1.3.3.cmml"><mrow id="S3.E7.m1.3.3.3" xref="S3.E7.m1.3.3.3.cmml"><mo id="S3.E7.m1.3.3.3.4" rspace="0.167em" xref="S3.E7.m1.3.3.3.4.cmml">∂</mo><mrow id="S3.E7.m1.3.3.3.3.2" xref="S3.E7.m1.3.3.3.3.3.cmml"><mi id="S3.E7.m1.1.1.1.1" xref="S3.E7.m1.1.1.1.1.cmml">cos</mi><mo id="S3.E7.m1.3.3.3.3.2a" xref="S3.E7.m1.3.3.3.3.3.cmml">⁡</mo><mrow id="S3.E7.m1.3.3.3.3.2.2" xref="S3.E7.m1.3.3.3.3.3.cmml"><mo id="S3.E7.m1.3.3.3.3.2.2.3" stretchy="false" xref="S3.E7.m1.3.3.3.3.3.cmml">(</mo><msub id="S3.E7.m1.2.2.2.2.1.1.1" xref="S3.E7.m1.2.2.2.2.1.1.1.cmml"><mi id="S3.E7.m1.2.2.2.2.1.1.1.2" xref="S3.E7.m1.2.2.2.2.1.1.1.2.cmml">O</mi><mi id="S3.E7.m1.2.2.2.2.1.1.1.3" xref="S3.E7.m1.2.2.2.2.1.1.1.3.cmml">q</mi></msub><mo id="S3.E7.m1.3.3.3.3.2.2.4" xref="S3.E7.m1.3.3.3.3.3.cmml">,</mo><msub id="S3.E7.m1.3.3.3.3.2.2.2" xref="S3.E7.m1.3.3.3.3.2.2.2.cmml"><mi id="S3.E7.m1.3.3.3.3.2.2.2.2" xref="S3.E7.m1.3.3.3.3.2.2.2.2.cmml">O</mi><mrow id="S3.E7.m1.3.3.3.3.2.2.2.3" xref="S3.E7.m1.3.3.3.3.2.2.2.3.cmml"><mi id="S3.E7.m1.3.3.3.3.2.2.2.3.2" xref="S3.E7.m1.3.3.3.3.2.2.2.3.2.cmml">d</mi><mo id="S3.E7.m1.3.3.3.3.2.2.2.3.3" xref="S3.E7.m1.3.3.3.3.2.2.2.3.3.cmml">+</mo></mrow></msub><mo id="S3.E7.m1.3.3.3.3.2.2.5" stretchy="false" xref="S3.E7.m1.3.3.3.3.3.cmml">)</mo></mrow></mrow></mrow><mrow id="S3.E7.m1.3.3.5" xref="S3.E7.m1.3.3.5.cmml"><mo id="S3.E7.m1.3.3.5.1" rspace="0em" xref="S3.E7.m1.3.3.5.1.cmml">∂</mo><msub id="S3.E7.m1.3.3.5.2" xref="S3.E7.m1.3.3.5.2.cmml"><mi id="S3.E7.m1.3.3.5.2.2" mathvariant="normal" xref="S3.E7.m1.3.3.5.2.2.cmml">Λ</mi><mrow id="S3.E7.m1.3.3.5.2.3" xref="S3.E7.m1.3.3.5.2.3.cmml"><mi id="S3.E7.m1.3.3.5.2.3.2" xref="S3.E7.m1.3.3.5.2.3.2.cmml">t</mi><mo id="S3.E7.m1.3.3.5.2.3.1" xref="S3.E7.m1.3.3.5.2.3.1.cmml">−</mo><mn id="S3.E7.m1.3.3.5.2.3.3" xref="S3.E7.m1.3.3.5.2.3.3.cmml">1</mn></mrow></msub></mrow></mfrac><mo id="S3.E7.m1.6.7.3.1" xref="S3.E7.m1.6.7.3.1.cmml">−</mo><mfrac id="S3.E7.m1.6.6" xref="S3.E7.m1.6.6.cmml"><mrow id="S3.E7.m1.6.6.3" xref="S3.E7.m1.6.6.3.cmml"><mo id="S3.E7.m1.6.6.3.4" rspace="0.167em" xref="S3.E7.m1.6.6.3.4.cmml">∂</mo><mrow id="S3.E7.m1.6.6.3.3.2" xref="S3.E7.m1.6.6.3.3.3.cmml"><mi id="S3.E7.m1.4.4.1.1" xref="S3.E7.m1.4.4.1.1.cmml">cos</mi><mo id="S3.E7.m1.6.6.3.3.2a" xref="S3.E7.m1.6.6.3.3.3.cmml">⁡</mo><mrow id="S3.E7.m1.6.6.3.3.2.2" xref="S3.E7.m1.6.6.3.3.3.cmml"><mo id="S3.E7.m1.6.6.3.3.2.2.3" stretchy="false" xref="S3.E7.m1.6.6.3.3.3.cmml">(</mo><msub id="S3.E7.m1.5.5.2.2.1.1.1" xref="S3.E7.m1.5.5.2.2.1.1.1.cmml"><mi id="S3.E7.m1.5.5.2.2.1.1.1.2" xref="S3.E7.m1.5.5.2.2.1.1.1.2.cmml">O</mi><mi id="S3.E7.m1.5.5.2.2.1.1.1.3" xref="S3.E7.m1.5.5.2.2.1.1.1.3.cmml">q</mi></msub><mo id="S3.E7.m1.6.6.3.3.2.2.4" xref="S3.E7.m1.6.6.3.3.3.cmml">,</mo><msub id="S3.E7.m1.6.6.3.3.2.2.2" xref="S3.E7.m1.6.6.3.3.2.2.2.cmml"><mi id="S3.E7.m1.6.6.3.3.2.2.2.2" xref="S3.E7.m1.6.6.3.3.2.2.2.2.cmml">O</mi><mrow id="S3.E7.m1.6.6.3.3.2.2.2.3" xref="S3.E7.m1.6.6.3.3.2.2.2.3.cmml"><mi id="S3.E7.m1.6.6.3.3.2.2.2.3.2" xref="S3.E7.m1.6.6.3.3.2.2.2.3.2.cmml">d</mi><mo id="S3.E7.m1.6.6.3.3.2.2.2.3.3" xref="S3.E7.m1.6.6.3.3.2.2.2.3.3.cmml">−</mo></mrow></msub><mo id="S3.E7.m1.6.6.3.3.2.2.5" stretchy="false" xref="S3.E7.m1.6.6.3.3.3.cmml">)</mo></mrow></mrow></mrow><mrow id="S3.E7.m1.6.6.5" xref="S3.E7.m1.6.6.5.cmml"><mo id="S3.E7.m1.6.6.5.1" rspace="0em" xref="S3.E7.m1.6.6.5.1.cmml">∂</mo><msub id="S3.E7.m1.6.6.5.2" xref="S3.E7.m1.6.6.5.2.cmml"><mi id="S3.E7.m1.6.6.5.2.2" mathvariant="normal" xref="S3.E7.m1.6.6.5.2.2.cmml">Λ</mi><mrow id="S3.E7.m1.6.6.5.2.3" xref="S3.E7.m1.6.6.5.2.3.cmml"><mi id="S3.E7.m1.6.6.5.2.3.2" xref="S3.E7.m1.6.6.5.2.3.2.cmml">t</mi><mo id="S3.E7.m1.6.6.5.2.3.1" xref="S3.E7.m1.6.6.5.2.3.1.cmml">−</mo><mn id="S3.E7.m1.6.6.5.2.3.3" xref="S3.E7.m1.6.6.5.2.3.3.cmml">1</mn></mrow></msub></mrow></mfrac></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E7.m1.6b"><apply id="S3.E7.m1.6.7.cmml" xref="S3.E7.m1.6.7"><eq id="S3.E7.m1.6.7.1.cmml" xref="S3.E7.m1.6.7.1"></eq><apply id="S3.E7.m1.6.7.2.cmml" xref="S3.E7.m1.6.7.2"><divide id="S3.E7.m1.6.7.2.1.cmml" xref="S3.E7.m1.6.7.2"></divide><apply id="S3.E7.m1.6.7.2.2.cmml" xref="S3.E7.m1.6.7.2.2"><partialdiff id="S3.E7.m1.6.7.2.2.1.cmml" xref="S3.E7.m1.6.7.2.2.1"></partialdiff><ci id="S3.E7.m1.6.7.2.2.2.cmml" xref="S3.E7.m1.6.7.2.2.2">Δ</ci></apply><apply id="S3.E7.m1.6.7.2.3.cmml" xref="S3.E7.m1.6.7.2.3"><partialdiff id="S3.E7.m1.6.7.2.3.1.cmml" xref="S3.E7.m1.6.7.2.3.1"></partialdiff><apply id="S3.E7.m1.6.7.2.3.2.cmml" xref="S3.E7.m1.6.7.2.3.2"><csymbol cd="ambiguous" id="S3.E7.m1.6.7.2.3.2.1.cmml" xref="S3.E7.m1.6.7.2.3.2">subscript</csymbol><ci id="S3.E7.m1.6.7.2.3.2.2.cmml" xref="S3.E7.m1.6.7.2.3.2.2">Λ</ci><apply id="S3.E7.m1.6.7.2.3.2.3.cmml" xref="S3.E7.m1.6.7.2.3.2.3"><minus id="S3.E7.m1.6.7.2.3.2.3.1.cmml" xref="S3.E7.m1.6.7.2.3.2.3.1"></minus><ci id="S3.E7.m1.6.7.2.3.2.3.2.cmml" xref="S3.E7.m1.6.7.2.3.2.3.2">𝑡</ci><cn id="S3.E7.m1.6.7.2.3.2.3.3.cmml" type="integer" xref="S3.E7.m1.6.7.2.3.2.3.3">1</cn></apply></apply></apply></apply><apply id="S3.E7.m1.6.7.3.cmml" xref="S3.E7.m1.6.7.3"><minus id="S3.E7.m1.6.7.3.1.cmml" xref="S3.E7.m1.6.7.3.1"></minus><apply id="S3.E7.m1.3.3.cmml" xref="S3.E7.m1.3.3"><divide id="S3.E7.m1.3.3.4.cmml" xref="S3.E7.m1.3.3"></divide><apply id="S3.E7.m1.3.3.3.cmml" xref="S3.E7.m1.3.3.3"><partialdiff id="S3.E7.m1.3.3.3.4.cmml" xref="S3.E7.m1.3.3.3.4"></partialdiff><apply id="S3.E7.m1.3.3.3.3.3.cmml" xref="S3.E7.m1.3.3.3.3.2"><cos id="S3.E7.m1.1.1.1.1.cmml" xref="S3.E7.m1.1.1.1.1"></cos><apply id="S3.E7.m1.2.2.2.2.1.1.1.cmml" xref="S3.E7.m1.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.E7.m1.2.2.2.2.1.1.1.1.cmml" xref="S3.E7.m1.2.2.2.2.1.1.1">subscript</csymbol><ci id="S3.E7.m1.2.2.2.2.1.1.1.2.cmml" xref="S3.E7.m1.2.2.2.2.1.1.1.2">𝑂</ci><ci id="S3.E7.m1.2.2.2.2.1.1.1.3.cmml" xref="S3.E7.m1.2.2.2.2.1.1.1.3">𝑞</ci></apply><apply id="S3.E7.m1.3.3.3.3.2.2.2.cmml" xref="S3.E7.m1.3.3.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.E7.m1.3.3.3.3.2.2.2.1.cmml" xref="S3.E7.m1.3.3.3.3.2.2.2">subscript</csymbol><ci id="S3.E7.m1.3.3.3.3.2.2.2.2.cmml" xref="S3.E7.m1.3.3.3.3.2.2.2.2">𝑂</ci><apply id="S3.E7.m1.3.3.3.3.2.2.2.3.cmml" xref="S3.E7.m1.3.3.3.3.2.2.2.3"><csymbol cd="latexml" id="S3.E7.m1.3.3.3.3.2.2.2.3.1.cmml" xref="S3.E7.m1.3.3.3.3.2.2.2.3">limit-from</csymbol><ci id="S3.E7.m1.3.3.3.3.2.2.2.3.2.cmml" xref="S3.E7.m1.3.3.3.3.2.2.2.3.2">𝑑</ci><plus id="S3.E7.m1.3.3.3.3.2.2.2.3.3.cmml" xref="S3.E7.m1.3.3.3.3.2.2.2.3.3"></plus></apply></apply></apply></apply><apply id="S3.E7.m1.3.3.5.cmml" xref="S3.E7.m1.3.3.5"><partialdiff id="S3.E7.m1.3.3.5.1.cmml" xref="S3.E7.m1.3.3.5.1"></partialdiff><apply id="S3.E7.m1.3.3.5.2.cmml" xref="S3.E7.m1.3.3.5.2"><csymbol cd="ambiguous" id="S3.E7.m1.3.3.5.2.1.cmml" xref="S3.E7.m1.3.3.5.2">subscript</csymbol><ci id="S3.E7.m1.3.3.5.2.2.cmml" xref="S3.E7.m1.3.3.5.2.2">Λ</ci><apply id="S3.E7.m1.3.3.5.2.3.cmml" xref="S3.E7.m1.3.3.5.2.3"><minus id="S3.E7.m1.3.3.5.2.3.1.cmml" xref="S3.E7.m1.3.3.5.2.3.1"></minus><ci id="S3.E7.m1.3.3.5.2.3.2.cmml" xref="S3.E7.m1.3.3.5.2.3.2">𝑡</ci><cn id="S3.E7.m1.3.3.5.2.3.3.cmml" type="integer" xref="S3.E7.m1.3.3.5.2.3.3">1</cn></apply></apply></apply></apply><apply id="S3.E7.m1.6.6.cmml" xref="S3.E7.m1.6.6"><divide id="S3.E7.m1.6.6.4.cmml" xref="S3.E7.m1.6.6"></divide><apply id="S3.E7.m1.6.6.3.cmml" xref="S3.E7.m1.6.6.3"><partialdiff id="S3.E7.m1.6.6.3.4.cmml" xref="S3.E7.m1.6.6.3.4"></partialdiff><apply id="S3.E7.m1.6.6.3.3.3.cmml" xref="S3.E7.m1.6.6.3.3.2"><cos id="S3.E7.m1.4.4.1.1.cmml" xref="S3.E7.m1.4.4.1.1"></cos><apply id="S3.E7.m1.5.5.2.2.1.1.1.cmml" xref="S3.E7.m1.5.5.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.E7.m1.5.5.2.2.1.1.1.1.cmml" xref="S3.E7.m1.5.5.2.2.1.1.1">subscript</csymbol><ci id="S3.E7.m1.5.5.2.2.1.1.1.2.cmml" xref="S3.E7.m1.5.5.2.2.1.1.1.2">𝑂</ci><ci id="S3.E7.m1.5.5.2.2.1.1.1.3.cmml" xref="S3.E7.m1.5.5.2.2.1.1.1.3">𝑞</ci></apply><apply id="S3.E7.m1.6.6.3.3.2.2.2.cmml" xref="S3.E7.m1.6.6.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.E7.m1.6.6.3.3.2.2.2.1.cmml" xref="S3.E7.m1.6.6.3.3.2.2.2">subscript</csymbol><ci id="S3.E7.m1.6.6.3.3.2.2.2.2.cmml" xref="S3.E7.m1.6.6.3.3.2.2.2.2">𝑂</ci><apply id="S3.E7.m1.6.6.3.3.2.2.2.3.cmml" xref="S3.E7.m1.6.6.3.3.2.2.2.3"><csymbol cd="latexml" id="S3.E7.m1.6.6.3.3.2.2.2.3.1.cmml" xref="S3.E7.m1.6.6.3.3.2.2.2.3">limit-from</csymbol><ci id="S3.E7.m1.6.6.3.3.2.2.2.3.2.cmml" xref="S3.E7.m1.6.6.3.3.2.2.2.3.2">𝑑</ci><minus id="S3.E7.m1.6.6.3.3.2.2.2.3.3.cmml" xref="S3.E7.m1.6.6.3.3.2.2.2.3.3"></minus></apply></apply></apply></apply><apply id="S3.E7.m1.6.6.5.cmml" xref="S3.E7.m1.6.6.5"><partialdiff id="S3.E7.m1.6.6.5.1.cmml" xref="S3.E7.m1.6.6.5.1"></partialdiff><apply id="S3.E7.m1.6.6.5.2.cmml" xref="S3.E7.m1.6.6.5.2"><csymbol cd="ambiguous" id="S3.E7.m1.6.6.5.2.1.cmml" xref="S3.E7.m1.6.6.5.2">subscript</csymbol><ci id="S3.E7.m1.6.6.5.2.2.cmml" xref="S3.E7.m1.6.6.5.2.2">Λ</ci><apply id="S3.E7.m1.6.6.5.2.3.cmml" xref="S3.E7.m1.6.6.5.2.3"><minus id="S3.E7.m1.6.6.5.2.3.1.cmml" xref="S3.E7.m1.6.6.5.2.3.1"></minus><ci id="S3.E7.m1.6.6.5.2.3.2.cmml" xref="S3.E7.m1.6.6.5.2.3.2">𝑡</ci><cn id="S3.E7.m1.6.6.5.2.3.3.cmml" type="integer" xref="S3.E7.m1.6.6.5.2.3.3">1</cn></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E7.m1.6c">\frac{\partial\Delta}{\partial\Lambda_{t-1}}=\frac{\partial\cos(O_{q},O_{d+})}% {\partial\Lambda_{t-1}}-\frac{\partial\cos(O_{q},O_{d-})}{\partial\Lambda_{t-1}}</annotation><annotation encoding="application/x-llamapun" id="S3.E7.m1.6d">divide start_ARG ∂ roman_Δ end_ARG start_ARG ∂ roman_Λ start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT end_ARG = divide start_ARG ∂ roman_cos ( italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT , italic_O start_POSTSUBSCRIPT italic_d + end_POSTSUBSCRIPT ) end_ARG start_ARG ∂ roman_Λ start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT end_ARG - divide start_ARG ∂ roman_cos ( italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT , italic_O start_POSTSUBSCRIPT italic_d - end_POSTSUBSCRIPT ) end_ARG start_ARG ∂ roman_Λ start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT end_ARG</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx2.p5"> <p class="ltx_p" id="S3.SSx2.p5.12">To simplify the notation of calculating the derivatives of <math alttext="W" class="ltx_Math" display="inline" id="S3.SSx2.p5.1.m1.1"><semantics id="S3.SSx2.p5.1.m1.1a"><mi id="S3.SSx2.p5.1.m1.1.1" xref="S3.SSx2.p5.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.1.m1.1b"><ci id="S3.SSx2.p5.1.m1.1.1.cmml" xref="S3.SSx2.p5.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.1.m1.1c">W</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.1.m1.1d">italic_W</annotation></semantics></math>, we let <math alttext="d" class="ltx_Math" display="inline" id="S3.SSx2.p5.2.m2.1"><semantics id="S3.SSx2.p5.2.m2.1a"><mi id="S3.SSx2.p5.2.m2.1.1" xref="S3.SSx2.p5.2.m2.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.2.m2.1b"><ci id="S3.SSx2.p5.2.m2.1.1.cmml" xref="S3.SSx2.p5.2.m2.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.2.m2.1c">d</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.2.m2.1d">italic_d</annotation></semantics></math> denote <math alttext="d^{+}" class="ltx_Math" display="inline" id="S3.SSx2.p5.3.m3.1"><semantics id="S3.SSx2.p5.3.m3.1a"><msup id="S3.SSx2.p5.3.m3.1.1" xref="S3.SSx2.p5.3.m3.1.1.cmml"><mi id="S3.SSx2.p5.3.m3.1.1.2" xref="S3.SSx2.p5.3.m3.1.1.2.cmml">d</mi><mo id="S3.SSx2.p5.3.m3.1.1.3" xref="S3.SSx2.p5.3.m3.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.3.m3.1b"><apply id="S3.SSx2.p5.3.m3.1.1.cmml" xref="S3.SSx2.p5.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p5.3.m3.1.1.1.cmml" xref="S3.SSx2.p5.3.m3.1.1">superscript</csymbol><ci id="S3.SSx2.p5.3.m3.1.1.2.cmml" xref="S3.SSx2.p5.3.m3.1.1.2">𝑑</ci><plus id="S3.SSx2.p5.3.m3.1.1.3.cmml" xref="S3.SSx2.p5.3.m3.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.3.m3.1c">d^{+}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.3.m3.1d">italic_d start_POSTSUPERSCRIPT + end_POSTSUPERSCRIPT</annotation></semantics></math> and <math alttext="d^{-}" class="ltx_Math" display="inline" id="S3.SSx2.p5.4.m4.1"><semantics id="S3.SSx2.p5.4.m4.1a"><msup id="S3.SSx2.p5.4.m4.1.1" xref="S3.SSx2.p5.4.m4.1.1.cmml"><mi id="S3.SSx2.p5.4.m4.1.1.2" xref="S3.SSx2.p5.4.m4.1.1.2.cmml">d</mi><mo id="S3.SSx2.p5.4.m4.1.1.3" xref="S3.SSx2.p5.4.m4.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.4.m4.1b"><apply id="S3.SSx2.p5.4.m4.1.1.cmml" xref="S3.SSx2.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p5.4.m4.1.1.1.cmml" xref="S3.SSx2.p5.4.m4.1.1">superscript</csymbol><ci id="S3.SSx2.p5.4.m4.1.1.2.cmml" xref="S3.SSx2.p5.4.m4.1.1.2">𝑑</ci><minus id="S3.SSx2.p5.4.m4.1.1.3.cmml" xref="S3.SSx2.p5.4.m4.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.4.m4.1c">d^{-}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.4.m4.1d">italic_d start_POSTSUPERSCRIPT - end_POSTSUPERSCRIPT</annotation></semantics></math>, and we let <math alttext="a" class="ltx_Math" display="inline" id="S3.SSx2.p5.5.m5.1"><semantics id="S3.SSx2.p5.5.m5.1a"><mi id="S3.SSx2.p5.5.m5.1.1" xref="S3.SSx2.p5.5.m5.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.5.m5.1b"><ci id="S3.SSx2.p5.5.m5.1.1.cmml" xref="S3.SSx2.p5.5.m5.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.5.m5.1c">a</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.5.m5.1d">italic_a</annotation></semantics></math>, <math alttext="b" class="ltx_Math" display="inline" id="S3.SSx2.p5.6.m6.1"><semantics id="S3.SSx2.p5.6.m6.1a"><mi id="S3.SSx2.p5.6.m6.1.1" xref="S3.SSx2.p5.6.m6.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.6.m6.1b"><ci id="S3.SSx2.p5.6.m6.1.1.cmml" xref="S3.SSx2.p5.6.m6.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.6.m6.1c">b</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.6.m6.1d">italic_b</annotation></semantics></math>, <math alttext="c" class="ltx_Math" display="inline" id="S3.SSx2.p5.7.m7.1"><semantics id="S3.SSx2.p5.7.m7.1a"><mi id="S3.SSx2.p5.7.m7.1.1" xref="S3.SSx2.p5.7.m7.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.7.m7.1b"><ci id="S3.SSx2.p5.7.m7.1.1.cmml" xref="S3.SSx2.p5.7.m7.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.7.m7.1c">c</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.7.m7.1d">italic_c</annotation></semantics></math> be <math alttext="O^{T}_{q}O_{d}" class="ltx_Math" display="inline" id="S3.SSx2.p5.8.m8.1"><semantics id="S3.SSx2.p5.8.m8.1a"><mrow id="S3.SSx2.p5.8.m8.1.1" xref="S3.SSx2.p5.8.m8.1.1.cmml"><msubsup id="S3.SSx2.p5.8.m8.1.1.2" xref="S3.SSx2.p5.8.m8.1.1.2.cmml"><mi id="S3.SSx2.p5.8.m8.1.1.2.2.2" xref="S3.SSx2.p5.8.m8.1.1.2.2.2.cmml">O</mi><mi id="S3.SSx2.p5.8.m8.1.1.2.3" xref="S3.SSx2.p5.8.m8.1.1.2.3.cmml">q</mi><mi id="S3.SSx2.p5.8.m8.1.1.2.2.3" xref="S3.SSx2.p5.8.m8.1.1.2.2.3.cmml">T</mi></msubsup><mo id="S3.SSx2.p5.8.m8.1.1.1" xref="S3.SSx2.p5.8.m8.1.1.1.cmml">⁢</mo><msub id="S3.SSx2.p5.8.m8.1.1.3" xref="S3.SSx2.p5.8.m8.1.1.3.cmml"><mi id="S3.SSx2.p5.8.m8.1.1.3.2" xref="S3.SSx2.p5.8.m8.1.1.3.2.cmml">O</mi><mi id="S3.SSx2.p5.8.m8.1.1.3.3" xref="S3.SSx2.p5.8.m8.1.1.3.3.cmml">d</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.8.m8.1b"><apply id="S3.SSx2.p5.8.m8.1.1.cmml" xref="S3.SSx2.p5.8.m8.1.1"><times id="S3.SSx2.p5.8.m8.1.1.1.cmml" xref="S3.SSx2.p5.8.m8.1.1.1"></times><apply id="S3.SSx2.p5.8.m8.1.1.2.cmml" xref="S3.SSx2.p5.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.SSx2.p5.8.m8.1.1.2.1.cmml" xref="S3.SSx2.p5.8.m8.1.1.2">subscript</csymbol><apply id="S3.SSx2.p5.8.m8.1.1.2.2.cmml" xref="S3.SSx2.p5.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.SSx2.p5.8.m8.1.1.2.2.1.cmml" xref="S3.SSx2.p5.8.m8.1.1.2">superscript</csymbol><ci id="S3.SSx2.p5.8.m8.1.1.2.2.2.cmml" xref="S3.SSx2.p5.8.m8.1.1.2.2.2">𝑂</ci><ci id="S3.SSx2.p5.8.m8.1.1.2.2.3.cmml" xref="S3.SSx2.p5.8.m8.1.1.2.2.3">𝑇</ci></apply><ci id="S3.SSx2.p5.8.m8.1.1.2.3.cmml" xref="S3.SSx2.p5.8.m8.1.1.2.3">𝑞</ci></apply><apply id="S3.SSx2.p5.8.m8.1.1.3.cmml" xref="S3.SSx2.p5.8.m8.1.1.3"><csymbol cd="ambiguous" id="S3.SSx2.p5.8.m8.1.1.3.1.cmml" xref="S3.SSx2.p5.8.m8.1.1.3">subscript</csymbol><ci id="S3.SSx2.p5.8.m8.1.1.3.2.cmml" xref="S3.SSx2.p5.8.m8.1.1.3.2">𝑂</ci><ci id="S3.SSx2.p5.8.m8.1.1.3.3.cmml" xref="S3.SSx2.p5.8.m8.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.8.m8.1c">O^{T}_{q}O_{d}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.8.m8.1d">italic_O start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT</annotation></semantics></math>, <math alttext="\frac{1}{\|Q_{q}\|}" class="ltx_Math" display="inline" id="S3.SSx2.p5.9.m9.1"><semantics id="S3.SSx2.p5.9.m9.1a"><mfrac id="S3.SSx2.p5.9.m9.1.1" xref="S3.SSx2.p5.9.m9.1.1.cmml"><mn id="S3.SSx2.p5.9.m9.1.1.3" xref="S3.SSx2.p5.9.m9.1.1.3.cmml">1</mn><mrow id="S3.SSx2.p5.9.m9.1.1.1.1" xref="S3.SSx2.p5.9.m9.1.1.1.2.cmml"><mo id="S3.SSx2.p5.9.m9.1.1.1.1.2" stretchy="false" xref="S3.SSx2.p5.9.m9.1.1.1.2.1.cmml">‖</mo><msub id="S3.SSx2.p5.9.m9.1.1.1.1.1" xref="S3.SSx2.p5.9.m9.1.1.1.1.1.cmml"><mi id="S3.SSx2.p5.9.m9.1.1.1.1.1.2" xref="S3.SSx2.p5.9.m9.1.1.1.1.1.2.cmml">Q</mi><mi id="S3.SSx2.p5.9.m9.1.1.1.1.1.3" xref="S3.SSx2.p5.9.m9.1.1.1.1.1.3.cmml">q</mi></msub><mo id="S3.SSx2.p5.9.m9.1.1.1.1.3" stretchy="false" xref="S3.SSx2.p5.9.m9.1.1.1.2.1.cmml">‖</mo></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.9.m9.1b"><apply id="S3.SSx2.p5.9.m9.1.1.cmml" xref="S3.SSx2.p5.9.m9.1.1"><divide id="S3.SSx2.p5.9.m9.1.1.2.cmml" xref="S3.SSx2.p5.9.m9.1.1"></divide><cn id="S3.SSx2.p5.9.m9.1.1.3.cmml" type="integer" xref="S3.SSx2.p5.9.m9.1.1.3">1</cn><apply id="S3.SSx2.p5.9.m9.1.1.1.2.cmml" xref="S3.SSx2.p5.9.m9.1.1.1.1"><csymbol cd="latexml" id="S3.SSx2.p5.9.m9.1.1.1.2.1.cmml" xref="S3.SSx2.p5.9.m9.1.1.1.1.2">norm</csymbol><apply id="S3.SSx2.p5.9.m9.1.1.1.1.1.cmml" xref="S3.SSx2.p5.9.m9.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p5.9.m9.1.1.1.1.1.1.cmml" xref="S3.SSx2.p5.9.m9.1.1.1.1.1">subscript</csymbol><ci id="S3.SSx2.p5.9.m9.1.1.1.1.1.2.cmml" xref="S3.SSx2.p5.9.m9.1.1.1.1.1.2">𝑄</ci><ci id="S3.SSx2.p5.9.m9.1.1.1.1.1.3.cmml" xref="S3.SSx2.p5.9.m9.1.1.1.1.1.3">𝑞</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.9.m9.1c">\frac{1}{\|Q_{q}\|}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.9.m9.1d">divide start_ARG 1 end_ARG start_ARG ∥ italic_Q start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT ∥ end_ARG</annotation></semantics></math>, and <math alttext="\frac{1}{\|Q_{d}\|}" class="ltx_Math" display="inline" id="S3.SSx2.p5.10.m10.1"><semantics id="S3.SSx2.p5.10.m10.1a"><mfrac id="S3.SSx2.p5.10.m10.1.1" xref="S3.SSx2.p5.10.m10.1.1.cmml"><mn id="S3.SSx2.p5.10.m10.1.1.3" xref="S3.SSx2.p5.10.m10.1.1.3.cmml">1</mn><mrow id="S3.SSx2.p5.10.m10.1.1.1.1" xref="S3.SSx2.p5.10.m10.1.1.1.2.cmml"><mo id="S3.SSx2.p5.10.m10.1.1.1.1.2" stretchy="false" xref="S3.SSx2.p5.10.m10.1.1.1.2.1.cmml">‖</mo><msub id="S3.SSx2.p5.10.m10.1.1.1.1.1" xref="S3.SSx2.p5.10.m10.1.1.1.1.1.cmml"><mi id="S3.SSx2.p5.10.m10.1.1.1.1.1.2" xref="S3.SSx2.p5.10.m10.1.1.1.1.1.2.cmml">Q</mi><mi id="S3.SSx2.p5.10.m10.1.1.1.1.1.3" xref="S3.SSx2.p5.10.m10.1.1.1.1.1.3.cmml">d</mi></msub><mo id="S3.SSx2.p5.10.m10.1.1.1.1.3" stretchy="false" xref="S3.SSx2.p5.10.m10.1.1.1.2.1.cmml">‖</mo></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.10.m10.1b"><apply id="S3.SSx2.p5.10.m10.1.1.cmml" xref="S3.SSx2.p5.10.m10.1.1"><divide id="S3.SSx2.p5.10.m10.1.1.2.cmml" xref="S3.SSx2.p5.10.m10.1.1"></divide><cn id="S3.SSx2.p5.10.m10.1.1.3.cmml" type="integer" xref="S3.SSx2.p5.10.m10.1.1.3">1</cn><apply id="S3.SSx2.p5.10.m10.1.1.1.2.cmml" xref="S3.SSx2.p5.10.m10.1.1.1.1"><csymbol cd="latexml" id="S3.SSx2.p5.10.m10.1.1.1.2.1.cmml" xref="S3.SSx2.p5.10.m10.1.1.1.1.2">norm</csymbol><apply id="S3.SSx2.p5.10.m10.1.1.1.1.1.cmml" xref="S3.SSx2.p5.10.m10.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SSx2.p5.10.m10.1.1.1.1.1.1.cmml" xref="S3.SSx2.p5.10.m10.1.1.1.1.1">subscript</csymbol><ci id="S3.SSx2.p5.10.m10.1.1.1.1.1.2.cmml" xref="S3.SSx2.p5.10.m10.1.1.1.1.1.2">𝑄</ci><ci id="S3.SSx2.p5.10.m10.1.1.1.1.1.3.cmml" xref="S3.SSx2.p5.10.m10.1.1.1.1.1.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.10.m10.1c">\frac{1}{\|Q_{d}\|}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.10.m10.1d">divide start_ARG 1 end_ARG start_ARG ∥ italic_Q start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ∥ end_ARG</annotation></semantics></math>, respectively. Then, we can compute <math alttext="\frac{\partial\Delta}{\partial W_{q}}" class="ltx_Math" display="inline" id="S3.SSx2.p5.11.m11.1"><semantics id="S3.SSx2.p5.11.m11.1a"><mfrac id="S3.SSx2.p5.11.m11.1.1" xref="S3.SSx2.p5.11.m11.1.1.cmml"><mrow id="S3.SSx2.p5.11.m11.1.1.2" xref="S3.SSx2.p5.11.m11.1.1.2.cmml"><mo id="S3.SSx2.p5.11.m11.1.1.2.1" rspace="0em" xref="S3.SSx2.p5.11.m11.1.1.2.1.cmml">∂</mo><mi id="S3.SSx2.p5.11.m11.1.1.2.2" mathvariant="normal" xref="S3.SSx2.p5.11.m11.1.1.2.2.cmml">Δ</mi></mrow><mrow id="S3.SSx2.p5.11.m11.1.1.3" xref="S3.SSx2.p5.11.m11.1.1.3.cmml"><mo id="S3.SSx2.p5.11.m11.1.1.3.1" rspace="0em" xref="S3.SSx2.p5.11.m11.1.1.3.1.cmml">∂</mo><msub id="S3.SSx2.p5.11.m11.1.1.3.2" xref="S3.SSx2.p5.11.m11.1.1.3.2.cmml"><mi id="S3.SSx2.p5.11.m11.1.1.3.2.2" xref="S3.SSx2.p5.11.m11.1.1.3.2.2.cmml">W</mi><mi id="S3.SSx2.p5.11.m11.1.1.3.2.3" xref="S3.SSx2.p5.11.m11.1.1.3.2.3.cmml">q</mi></msub></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.11.m11.1b"><apply id="S3.SSx2.p5.11.m11.1.1.cmml" xref="S3.SSx2.p5.11.m11.1.1"><divide id="S3.SSx2.p5.11.m11.1.1.1.cmml" xref="S3.SSx2.p5.11.m11.1.1"></divide><apply id="S3.SSx2.p5.11.m11.1.1.2.cmml" xref="S3.SSx2.p5.11.m11.1.1.2"><partialdiff id="S3.SSx2.p5.11.m11.1.1.2.1.cmml" xref="S3.SSx2.p5.11.m11.1.1.2.1"></partialdiff><ci id="S3.SSx2.p5.11.m11.1.1.2.2.cmml" xref="S3.SSx2.p5.11.m11.1.1.2.2">Δ</ci></apply><apply id="S3.SSx2.p5.11.m11.1.1.3.cmml" xref="S3.SSx2.p5.11.m11.1.1.3"><partialdiff id="S3.SSx2.p5.11.m11.1.1.3.1.cmml" xref="S3.SSx2.p5.11.m11.1.1.3.1"></partialdiff><apply id="S3.SSx2.p5.11.m11.1.1.3.2.cmml" xref="S3.SSx2.p5.11.m11.1.1.3.2"><csymbol cd="ambiguous" id="S3.SSx2.p5.11.m11.1.1.3.2.1.cmml" xref="S3.SSx2.p5.11.m11.1.1.3.2">subscript</csymbol><ci id="S3.SSx2.p5.11.m11.1.1.3.2.2.cmml" xref="S3.SSx2.p5.11.m11.1.1.3.2.2">𝑊</ci><ci id="S3.SSx2.p5.11.m11.1.1.3.2.3.cmml" xref="S3.SSx2.p5.11.m11.1.1.3.2.3">𝑞</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.11.m11.1c">\frac{\partial\Delta}{\partial W_{q}}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.11.m11.1d">divide start_ARG ∂ roman_Δ end_ARG start_ARG ∂ italic_W start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT end_ARG</annotation></semantics></math> and <math alttext="\frac{\partial\Delta}{\partial W_{d}}" class="ltx_Math" display="inline" id="S3.SSx2.p5.12.m12.1"><semantics id="S3.SSx2.p5.12.m12.1a"><mfrac id="S3.SSx2.p5.12.m12.1.1" xref="S3.SSx2.p5.12.m12.1.1.cmml"><mrow id="S3.SSx2.p5.12.m12.1.1.2" xref="S3.SSx2.p5.12.m12.1.1.2.cmml"><mo id="S3.SSx2.p5.12.m12.1.1.2.1" rspace="0em" xref="S3.SSx2.p5.12.m12.1.1.2.1.cmml">∂</mo><mi id="S3.SSx2.p5.12.m12.1.1.2.2" mathvariant="normal" xref="S3.SSx2.p5.12.m12.1.1.2.2.cmml">Δ</mi></mrow><mrow id="S3.SSx2.p5.12.m12.1.1.3" xref="S3.SSx2.p5.12.m12.1.1.3.cmml"><mo id="S3.SSx2.p5.12.m12.1.1.3.1" rspace="0em" xref="S3.SSx2.p5.12.m12.1.1.3.1.cmml">∂</mo><msub id="S3.SSx2.p5.12.m12.1.1.3.2" xref="S3.SSx2.p5.12.m12.1.1.3.2.cmml"><mi id="S3.SSx2.p5.12.m12.1.1.3.2.2" xref="S3.SSx2.p5.12.m12.1.1.3.2.2.cmml">W</mi><mi id="S3.SSx2.p5.12.m12.1.1.3.2.3" xref="S3.SSx2.p5.12.m12.1.1.3.2.3.cmml">d</mi></msub></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S3.SSx2.p5.12.m12.1b"><apply id="S3.SSx2.p5.12.m12.1.1.cmml" xref="S3.SSx2.p5.12.m12.1.1"><divide id="S3.SSx2.p5.12.m12.1.1.1.cmml" xref="S3.SSx2.p5.12.m12.1.1"></divide><apply id="S3.SSx2.p5.12.m12.1.1.2.cmml" xref="S3.SSx2.p5.12.m12.1.1.2"><partialdiff id="S3.SSx2.p5.12.m12.1.1.2.1.cmml" xref="S3.SSx2.p5.12.m12.1.1.2.1"></partialdiff><ci id="S3.SSx2.p5.12.m12.1.1.2.2.cmml" xref="S3.SSx2.p5.12.m12.1.1.2.2">Δ</ci></apply><apply id="S3.SSx2.p5.12.m12.1.1.3.cmml" xref="S3.SSx2.p5.12.m12.1.1.3"><partialdiff id="S3.SSx2.p5.12.m12.1.1.3.1.cmml" xref="S3.SSx2.p5.12.m12.1.1.3.1"></partialdiff><apply id="S3.SSx2.p5.12.m12.1.1.3.2.cmml" xref="S3.SSx2.p5.12.m12.1.1.3.2"><csymbol cd="ambiguous" id="S3.SSx2.p5.12.m12.1.1.3.2.1.cmml" xref="S3.SSx2.p5.12.m12.1.1.3.2">subscript</csymbol><ci id="S3.SSx2.p5.12.m12.1.1.3.2.2.cmml" xref="S3.SSx2.p5.12.m12.1.1.3.2.2">𝑊</ci><ci id="S3.SSx2.p5.12.m12.1.1.3.2.3.cmml" xref="S3.SSx2.p5.12.m12.1.1.3.2.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p5.12.m12.1c">\frac{\partial\Delta}{\partial W_{d}}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p5.12.m12.1d">divide start_ARG ∂ roman_Δ end_ARG start_ARG ∂ italic_W start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_ARG</annotation></semantics></math> by using the following formulas:</p> </div> <div class="ltx_para" id="S3.SSx2.p6"> <table class="ltx_equation ltx_eqn_table" id="S3.E8"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(8)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="{\frac{\partial\mathrm{cos}({O}_{q},{O}_{d})}{\partial W_{q}}}={\frac{\partial% }{\partial W_{q}}}{\frac{{O}_{q}^{T}{O}_{d}}{\|{O}_{q}\|\|{O}_{d}\|}}=\delta_{% {O}_{q}}^{(q,d)}h_{d}^{T}" class="ltx_Math" display="block" id="S3.E8.m1.6"><semantics id="S3.E8.m1.6a"><mrow id="S3.E8.m1.6.7" xref="S3.E8.m1.6.7.cmml"><mfrac id="S3.E8.m1.2.2" xref="S3.E8.m1.2.2.cmml"><mrow id="S3.E8.m1.2.2.2" xref="S3.E8.m1.2.2.2.cmml"><mo id="S3.E8.m1.2.2.2.3" rspace="0em" xref="S3.E8.m1.2.2.2.3.cmml">∂</mo><mrow id="S3.E8.m1.2.2.2.2" xref="S3.E8.m1.2.2.2.2.cmml"><mi id="S3.E8.m1.2.2.2.2.4" xref="S3.E8.m1.2.2.2.2.4.cmml">cos</mi><mo id="S3.E8.m1.2.2.2.2.3" xref="S3.E8.m1.2.2.2.2.3.cmml">⁢</mo><mrow id="S3.E8.m1.2.2.2.2.2.2" xref="S3.E8.m1.2.2.2.2.2.3.cmml"><mo id="S3.E8.m1.2.2.2.2.2.2.3" stretchy="false" xref="S3.E8.m1.2.2.2.2.2.3.cmml">(</mo><msub id="S3.E8.m1.1.1.1.1.1.1.1" xref="S3.E8.m1.1.1.1.1.1.1.1.cmml"><mi id="S3.E8.m1.1.1.1.1.1.1.1.2" xref="S3.E8.m1.1.1.1.1.1.1.1.2.cmml">O</mi><mi id="S3.E8.m1.1.1.1.1.1.1.1.3" xref="S3.E8.m1.1.1.1.1.1.1.1.3.cmml">q</mi></msub><mo id="S3.E8.m1.2.2.2.2.2.2.4" xref="S3.E8.m1.2.2.2.2.2.3.cmml">,</mo><msub id="S3.E8.m1.2.2.2.2.2.2.2" xref="S3.E8.m1.2.2.2.2.2.2.2.cmml"><mi id="S3.E8.m1.2.2.2.2.2.2.2.2" xref="S3.E8.m1.2.2.2.2.2.2.2.2.cmml">O</mi><mi id="S3.E8.m1.2.2.2.2.2.2.2.3" xref="S3.E8.m1.2.2.2.2.2.2.2.3.cmml">d</mi></msub><mo id="S3.E8.m1.2.2.2.2.2.2.5" stretchy="false" xref="S3.E8.m1.2.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><mrow id="S3.E8.m1.2.2.4" xref="S3.E8.m1.2.2.4.cmml"><mo id="S3.E8.m1.2.2.4.1" rspace="0em" xref="S3.E8.m1.2.2.4.1.cmml">∂</mo><msub id="S3.E8.m1.2.2.4.2" xref="S3.E8.m1.2.2.4.2.cmml"><mi id="S3.E8.m1.2.2.4.2.2" xref="S3.E8.m1.2.2.4.2.2.cmml">W</mi><mi id="S3.E8.m1.2.2.4.2.3" xref="S3.E8.m1.2.2.4.2.3.cmml">q</mi></msub></mrow></mfrac><mo id="S3.E8.m1.6.7.2" xref="S3.E8.m1.6.7.2.cmml">=</mo><mrow id="S3.E8.m1.6.7.3" xref="S3.E8.m1.6.7.3.cmml"><mfrac id="S3.E8.m1.6.7.3.2" xref="S3.E8.m1.6.7.3.2.cmml"><mo id="S3.E8.m1.6.7.3.2.2" xref="S3.E8.m1.6.7.3.2.2.cmml">∂</mo><mrow id="S3.E8.m1.6.7.3.2.3" xref="S3.E8.m1.6.7.3.2.3.cmml"><mo id="S3.E8.m1.6.7.3.2.3.1" rspace="0em" xref="S3.E8.m1.6.7.3.2.3.1.cmml">∂</mo><msub id="S3.E8.m1.6.7.3.2.3.2" xref="S3.E8.m1.6.7.3.2.3.2.cmml"><mi id="S3.E8.m1.6.7.3.2.3.2.2" xref="S3.E8.m1.6.7.3.2.3.2.2.cmml">W</mi><mi id="S3.E8.m1.6.7.3.2.3.2.3" xref="S3.E8.m1.6.7.3.2.3.2.3.cmml">q</mi></msub></mrow></mfrac><mo id="S3.E8.m1.6.7.3.1" xref="S3.E8.m1.6.7.3.1.cmml">⁢</mo><mfrac id="S3.E8.m1.4.4" xref="S3.E8.m1.4.4.cmml"><mrow id="S3.E8.m1.4.4.4" xref="S3.E8.m1.4.4.4.cmml"><msubsup id="S3.E8.m1.4.4.4.2" xref="S3.E8.m1.4.4.4.2.cmml"><mi id="S3.E8.m1.4.4.4.2.2.2" xref="S3.E8.m1.4.4.4.2.2.2.cmml">O</mi><mi id="S3.E8.m1.4.4.4.2.2.3" xref="S3.E8.m1.4.4.4.2.2.3.cmml">q</mi><mi id="S3.E8.m1.4.4.4.2.3" xref="S3.E8.m1.4.4.4.2.3.cmml">T</mi></msubsup><mo id="S3.E8.m1.4.4.4.1" xref="S3.E8.m1.4.4.4.1.cmml">⁢</mo><msub id="S3.E8.m1.4.4.4.3" xref="S3.E8.m1.4.4.4.3.cmml"><mi id="S3.E8.m1.4.4.4.3.2" xref="S3.E8.m1.4.4.4.3.2.cmml">O</mi><mi id="S3.E8.m1.4.4.4.3.3" xref="S3.E8.m1.4.4.4.3.3.cmml">d</mi></msub></mrow><mrow id="S3.E8.m1.4.4.2" xref="S3.E8.m1.4.4.2.cmml"><mrow id="S3.E8.m1.3.3.1.1.1" xref="S3.E8.m1.3.3.1.1.2.cmml"><mo id="S3.E8.m1.3.3.1.1.1.2" stretchy="false" xref="S3.E8.m1.3.3.1.1.2.1.cmml">‖</mo><msub id="S3.E8.m1.3.3.1.1.1.1" xref="S3.E8.m1.3.3.1.1.1.1.cmml"><mi id="S3.E8.m1.3.3.1.1.1.1.2" xref="S3.E8.m1.3.3.1.1.1.1.2.cmml">O</mi><mi id="S3.E8.m1.3.3.1.1.1.1.3" xref="S3.E8.m1.3.3.1.1.1.1.3.cmml">q</mi></msub><mo id="S3.E8.m1.3.3.1.1.1.3" stretchy="false" xref="S3.E8.m1.3.3.1.1.2.1.cmml">‖</mo></mrow><mo id="S3.E8.m1.4.4.2.3" xref="S3.E8.m1.4.4.2.3.cmml">⁢</mo><mrow id="S3.E8.m1.4.4.2.2.1" xref="S3.E8.m1.4.4.2.2.2.cmml"><mo id="S3.E8.m1.4.4.2.2.1.2" stretchy="false" xref="S3.E8.m1.4.4.2.2.2.1.cmml">‖</mo><msub id="S3.E8.m1.4.4.2.2.1.1" xref="S3.E8.m1.4.4.2.2.1.1.cmml"><mi id="S3.E8.m1.4.4.2.2.1.1.2" xref="S3.E8.m1.4.4.2.2.1.1.2.cmml">O</mi><mi id="S3.E8.m1.4.4.2.2.1.1.3" xref="S3.E8.m1.4.4.2.2.1.1.3.cmml">d</mi></msub><mo id="S3.E8.m1.4.4.2.2.1.3" stretchy="false" xref="S3.E8.m1.4.4.2.2.2.1.cmml">‖</mo></mrow></mrow></mfrac></mrow><mo id="S3.E8.m1.6.7.4" xref="S3.E8.m1.6.7.4.cmml">=</mo><mrow id="S3.E8.m1.6.7.5" xref="S3.E8.m1.6.7.5.cmml"><msubsup id="S3.E8.m1.6.7.5.2" xref="S3.E8.m1.6.7.5.2.cmml"><mi id="S3.E8.m1.6.7.5.2.2.2" xref="S3.E8.m1.6.7.5.2.2.2.cmml">δ</mi><msub id="S3.E8.m1.6.7.5.2.2.3" xref="S3.E8.m1.6.7.5.2.2.3.cmml"><mi id="S3.E8.m1.6.7.5.2.2.3.2" xref="S3.E8.m1.6.7.5.2.2.3.2.cmml">O</mi><mi id="S3.E8.m1.6.7.5.2.2.3.3" xref="S3.E8.m1.6.7.5.2.2.3.3.cmml">q</mi></msub><mrow id="S3.E8.m1.6.6.2.4" xref="S3.E8.m1.6.6.2.3.cmml"><mo id="S3.E8.m1.6.6.2.4.1" stretchy="false" xref="S3.E8.m1.6.6.2.3.cmml">(</mo><mi id="S3.E8.m1.5.5.1.1" xref="S3.E8.m1.5.5.1.1.cmml">q</mi><mo id="S3.E8.m1.6.6.2.4.2" xref="S3.E8.m1.6.6.2.3.cmml">,</mo><mi id="S3.E8.m1.6.6.2.2" xref="S3.E8.m1.6.6.2.2.cmml">d</mi><mo id="S3.E8.m1.6.6.2.4.3" stretchy="false" xref="S3.E8.m1.6.6.2.3.cmml">)</mo></mrow></msubsup><mo id="S3.E8.m1.6.7.5.1" xref="S3.E8.m1.6.7.5.1.cmml">⁢</mo><msubsup id="S3.E8.m1.6.7.5.3" xref="S3.E8.m1.6.7.5.3.cmml"><mi id="S3.E8.m1.6.7.5.3.2.2" xref="S3.E8.m1.6.7.5.3.2.2.cmml">h</mi><mi id="S3.E8.m1.6.7.5.3.2.3" xref="S3.E8.m1.6.7.5.3.2.3.cmml">d</mi><mi id="S3.E8.m1.6.7.5.3.3" xref="S3.E8.m1.6.7.5.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E8.m1.6b"><apply id="S3.E8.m1.6.7.cmml" xref="S3.E8.m1.6.7"><and id="S3.E8.m1.6.7a.cmml" xref="S3.E8.m1.6.7"></and><apply id="S3.E8.m1.6.7b.cmml" xref="S3.E8.m1.6.7"><eq id="S3.E8.m1.6.7.2.cmml" xref="S3.E8.m1.6.7.2"></eq><apply id="S3.E8.m1.2.2.cmml" xref="S3.E8.m1.2.2"><divide id="S3.E8.m1.2.2.3.cmml" xref="S3.E8.m1.2.2"></divide><apply id="S3.E8.m1.2.2.2.cmml" xref="S3.E8.m1.2.2.2"><partialdiff id="S3.E8.m1.2.2.2.3.cmml" xref="S3.E8.m1.2.2.2.3"></partialdiff><apply id="S3.E8.m1.2.2.2.2.cmml" xref="S3.E8.m1.2.2.2.2"><times id="S3.E8.m1.2.2.2.2.3.cmml" xref="S3.E8.m1.2.2.2.2.3"></times><ci id="S3.E8.m1.2.2.2.2.4.cmml" xref="S3.E8.m1.2.2.2.2.4">cos</ci><interval closure="open" id="S3.E8.m1.2.2.2.2.2.3.cmml" xref="S3.E8.m1.2.2.2.2.2.2"><apply id="S3.E8.m1.1.1.1.1.1.1.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E8.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E8.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.2">𝑂</ci><ci id="S3.E8.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.3">𝑞</ci></apply><apply id="S3.E8.m1.2.2.2.2.2.2.2.cmml" xref="S3.E8.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.E8.m1.2.2.2.2.2.2.2.1.cmml" xref="S3.E8.m1.2.2.2.2.2.2.2">subscript</csymbol><ci id="S3.E8.m1.2.2.2.2.2.2.2.2.cmml" xref="S3.E8.m1.2.2.2.2.2.2.2.2">𝑂</ci><ci id="S3.E8.m1.2.2.2.2.2.2.2.3.cmml" xref="S3.E8.m1.2.2.2.2.2.2.2.3">𝑑</ci></apply></interval></apply></apply><apply id="S3.E8.m1.2.2.4.cmml" xref="S3.E8.m1.2.2.4"><partialdiff id="S3.E8.m1.2.2.4.1.cmml" xref="S3.E8.m1.2.2.4.1"></partialdiff><apply id="S3.E8.m1.2.2.4.2.cmml" xref="S3.E8.m1.2.2.4.2"><csymbol cd="ambiguous" id="S3.E8.m1.2.2.4.2.1.cmml" xref="S3.E8.m1.2.2.4.2">subscript</csymbol><ci id="S3.E8.m1.2.2.4.2.2.cmml" xref="S3.E8.m1.2.2.4.2.2">𝑊</ci><ci id="S3.E8.m1.2.2.4.2.3.cmml" xref="S3.E8.m1.2.2.4.2.3">𝑞</ci></apply></apply></apply><apply id="S3.E8.m1.6.7.3.cmml" xref="S3.E8.m1.6.7.3"><times id="S3.E8.m1.6.7.3.1.cmml" xref="S3.E8.m1.6.7.3.1"></times><apply id="S3.E8.m1.6.7.3.2.cmml" xref="S3.E8.m1.6.7.3.2"><divide id="S3.E8.m1.6.7.3.2.1.cmml" xref="S3.E8.m1.6.7.3.2"></divide><partialdiff id="S3.E8.m1.6.7.3.2.2.cmml" xref="S3.E8.m1.6.7.3.2.2"></partialdiff><apply id="S3.E8.m1.6.7.3.2.3.cmml" xref="S3.E8.m1.6.7.3.2.3"><partialdiff id="S3.E8.m1.6.7.3.2.3.1.cmml" xref="S3.E8.m1.6.7.3.2.3.1"></partialdiff><apply id="S3.E8.m1.6.7.3.2.3.2.cmml" xref="S3.E8.m1.6.7.3.2.3.2"><csymbol cd="ambiguous" id="S3.E8.m1.6.7.3.2.3.2.1.cmml" xref="S3.E8.m1.6.7.3.2.3.2">subscript</csymbol><ci id="S3.E8.m1.6.7.3.2.3.2.2.cmml" xref="S3.E8.m1.6.7.3.2.3.2.2">𝑊</ci><ci id="S3.E8.m1.6.7.3.2.3.2.3.cmml" xref="S3.E8.m1.6.7.3.2.3.2.3">𝑞</ci></apply></apply></apply><apply id="S3.E8.m1.4.4.cmml" xref="S3.E8.m1.4.4"><divide id="S3.E8.m1.4.4.3.cmml" xref="S3.E8.m1.4.4"></divide><apply id="S3.E8.m1.4.4.4.cmml" xref="S3.E8.m1.4.4.4"><times id="S3.E8.m1.4.4.4.1.cmml" xref="S3.E8.m1.4.4.4.1"></times><apply id="S3.E8.m1.4.4.4.2.cmml" xref="S3.E8.m1.4.4.4.2"><csymbol cd="ambiguous" id="S3.E8.m1.4.4.4.2.1.cmml" xref="S3.E8.m1.4.4.4.2">superscript</csymbol><apply id="S3.E8.m1.4.4.4.2.2.cmml" xref="S3.E8.m1.4.4.4.2"><csymbol cd="ambiguous" id="S3.E8.m1.4.4.4.2.2.1.cmml" xref="S3.E8.m1.4.4.4.2">subscript</csymbol><ci id="S3.E8.m1.4.4.4.2.2.2.cmml" xref="S3.E8.m1.4.4.4.2.2.2">𝑂</ci><ci id="S3.E8.m1.4.4.4.2.2.3.cmml" xref="S3.E8.m1.4.4.4.2.2.3">𝑞</ci></apply><ci id="S3.E8.m1.4.4.4.2.3.cmml" xref="S3.E8.m1.4.4.4.2.3">𝑇</ci></apply><apply id="S3.E8.m1.4.4.4.3.cmml" xref="S3.E8.m1.4.4.4.3"><csymbol cd="ambiguous" id="S3.E8.m1.4.4.4.3.1.cmml" xref="S3.E8.m1.4.4.4.3">subscript</csymbol><ci id="S3.E8.m1.4.4.4.3.2.cmml" xref="S3.E8.m1.4.4.4.3.2">𝑂</ci><ci id="S3.E8.m1.4.4.4.3.3.cmml" xref="S3.E8.m1.4.4.4.3.3">𝑑</ci></apply></apply><apply id="S3.E8.m1.4.4.2.cmml" xref="S3.E8.m1.4.4.2"><times id="S3.E8.m1.4.4.2.3.cmml" xref="S3.E8.m1.4.4.2.3"></times><apply id="S3.E8.m1.3.3.1.1.2.cmml" xref="S3.E8.m1.3.3.1.1.1"><csymbol cd="latexml" id="S3.E8.m1.3.3.1.1.2.1.cmml" xref="S3.E8.m1.3.3.1.1.1.2">norm</csymbol><apply id="S3.E8.m1.3.3.1.1.1.1.cmml" xref="S3.E8.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S3.E8.m1.3.3.1.1.1.1.1.cmml" xref="S3.E8.m1.3.3.1.1.1.1">subscript</csymbol><ci id="S3.E8.m1.3.3.1.1.1.1.2.cmml" xref="S3.E8.m1.3.3.1.1.1.1.2">𝑂</ci><ci id="S3.E8.m1.3.3.1.1.1.1.3.cmml" xref="S3.E8.m1.3.3.1.1.1.1.3">𝑞</ci></apply></apply><apply id="S3.E8.m1.4.4.2.2.2.cmml" xref="S3.E8.m1.4.4.2.2.1"><csymbol cd="latexml" id="S3.E8.m1.4.4.2.2.2.1.cmml" xref="S3.E8.m1.4.4.2.2.1.2">norm</csymbol><apply id="S3.E8.m1.4.4.2.2.1.1.cmml" xref="S3.E8.m1.4.4.2.2.1.1"><csymbol cd="ambiguous" id="S3.E8.m1.4.4.2.2.1.1.1.cmml" xref="S3.E8.m1.4.4.2.2.1.1">subscript</csymbol><ci id="S3.E8.m1.4.4.2.2.1.1.2.cmml" xref="S3.E8.m1.4.4.2.2.1.1.2">𝑂</ci><ci id="S3.E8.m1.4.4.2.2.1.1.3.cmml" xref="S3.E8.m1.4.4.2.2.1.1.3">𝑑</ci></apply></apply></apply></apply></apply></apply><apply id="S3.E8.m1.6.7c.cmml" xref="S3.E8.m1.6.7"><eq id="S3.E8.m1.6.7.4.cmml" xref="S3.E8.m1.6.7.4"></eq><share href="https://arxiv.org/html/2408.04197v2#S3.E8.m1.6.7.3.cmml" id="S3.E8.m1.6.7d.cmml" xref="S3.E8.m1.6.7"></share><apply id="S3.E8.m1.6.7.5.cmml" xref="S3.E8.m1.6.7.5"><times id="S3.E8.m1.6.7.5.1.cmml" xref="S3.E8.m1.6.7.5.1"></times><apply id="S3.E8.m1.6.7.5.2.cmml" xref="S3.E8.m1.6.7.5.2"><csymbol cd="ambiguous" id="S3.E8.m1.6.7.5.2.1.cmml" xref="S3.E8.m1.6.7.5.2">superscript</csymbol><apply id="S3.E8.m1.6.7.5.2.2.cmml" xref="S3.E8.m1.6.7.5.2"><csymbol cd="ambiguous" id="S3.E8.m1.6.7.5.2.2.1.cmml" xref="S3.E8.m1.6.7.5.2">subscript</csymbol><ci id="S3.E8.m1.6.7.5.2.2.2.cmml" xref="S3.E8.m1.6.7.5.2.2.2">𝛿</ci><apply id="S3.E8.m1.6.7.5.2.2.3.cmml" xref="S3.E8.m1.6.7.5.2.2.3"><csymbol cd="ambiguous" id="S3.E8.m1.6.7.5.2.2.3.1.cmml" xref="S3.E8.m1.6.7.5.2.2.3">subscript</csymbol><ci id="S3.E8.m1.6.7.5.2.2.3.2.cmml" xref="S3.E8.m1.6.7.5.2.2.3.2">𝑂</ci><ci id="S3.E8.m1.6.7.5.2.2.3.3.cmml" xref="S3.E8.m1.6.7.5.2.2.3.3">𝑞</ci></apply></apply><interval closure="open" id="S3.E8.m1.6.6.2.3.cmml" xref="S3.E8.m1.6.6.2.4"><ci id="S3.E8.m1.5.5.1.1.cmml" xref="S3.E8.m1.5.5.1.1">𝑞</ci><ci id="S3.E8.m1.6.6.2.2.cmml" xref="S3.E8.m1.6.6.2.2">𝑑</ci></interval></apply><apply id="S3.E8.m1.6.7.5.3.cmml" xref="S3.E8.m1.6.7.5.3"><csymbol cd="ambiguous" id="S3.E8.m1.6.7.5.3.1.cmml" xref="S3.E8.m1.6.7.5.3">superscript</csymbol><apply id="S3.E8.m1.6.7.5.3.2.cmml" xref="S3.E8.m1.6.7.5.3"><csymbol cd="ambiguous" id="S3.E8.m1.6.7.5.3.2.1.cmml" xref="S3.E8.m1.6.7.5.3">subscript</csymbol><ci id="S3.E8.m1.6.7.5.3.2.2.cmml" xref="S3.E8.m1.6.7.5.3.2.2">ℎ</ci><ci id="S3.E8.m1.6.7.5.3.2.3.cmml" xref="S3.E8.m1.6.7.5.3.2.3">𝑑</ci></apply><ci id="S3.E8.m1.6.7.5.3.3.cmml" xref="S3.E8.m1.6.7.5.3.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E8.m1.6c">{\frac{\partial\mathrm{cos}({O}_{q},{O}_{d})}{\partial W_{q}}}={\frac{\partial% }{\partial W_{q}}}{\frac{{O}_{q}^{T}{O}_{d}}{\|{O}_{q}\|\|{O}_{d}\|}}=\delta_{% {O}_{q}}^{(q,d)}h_{d}^{T}</annotation><annotation encoding="application/x-llamapun" id="S3.E8.m1.6d">divide start_ARG ∂ roman_cos ( italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT , italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ) end_ARG start_ARG ∂ italic_W start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT end_ARG = divide start_ARG ∂ end_ARG start_ARG ∂ italic_W start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT end_ARG divide start_ARG italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_ARG start_ARG ∥ italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT ∥ ∥ italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ∥ end_ARG = italic_δ start_POSTSUBSCRIPT italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT italic_h start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx2.p7"> <table class="ltx_equation ltx_eqn_table" id="S3.E9"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(9)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="{\frac{\partial\mathrm{cos}({O}_{q},{O}_{d})}{\partial W_{d}}}={\frac{\partial% }{\partial W_{d}}}{\frac{{O}_{q}^{T}{O}_{d}}{\|{O}_{q}\|\|{O}_{d}\|}}=\delta_{% {O}_{d}}^{(q,d)}h_{d}^{T}" class="ltx_Math" display="block" id="S3.E9.m1.6"><semantics id="S3.E9.m1.6a"><mrow id="S3.E9.m1.6.7" xref="S3.E9.m1.6.7.cmml"><mfrac id="S3.E9.m1.2.2" xref="S3.E9.m1.2.2.cmml"><mrow id="S3.E9.m1.2.2.2" xref="S3.E9.m1.2.2.2.cmml"><mo id="S3.E9.m1.2.2.2.3" rspace="0em" xref="S3.E9.m1.2.2.2.3.cmml">∂</mo><mrow id="S3.E9.m1.2.2.2.2" xref="S3.E9.m1.2.2.2.2.cmml"><mi id="S3.E9.m1.2.2.2.2.4" xref="S3.E9.m1.2.2.2.2.4.cmml">cos</mi><mo id="S3.E9.m1.2.2.2.2.3" xref="S3.E9.m1.2.2.2.2.3.cmml">⁢</mo><mrow id="S3.E9.m1.2.2.2.2.2.2" xref="S3.E9.m1.2.2.2.2.2.3.cmml"><mo id="S3.E9.m1.2.2.2.2.2.2.3" stretchy="false" xref="S3.E9.m1.2.2.2.2.2.3.cmml">(</mo><msub id="S3.E9.m1.1.1.1.1.1.1.1" xref="S3.E9.m1.1.1.1.1.1.1.1.cmml"><mi id="S3.E9.m1.1.1.1.1.1.1.1.2" xref="S3.E9.m1.1.1.1.1.1.1.1.2.cmml">O</mi><mi id="S3.E9.m1.1.1.1.1.1.1.1.3" xref="S3.E9.m1.1.1.1.1.1.1.1.3.cmml">q</mi></msub><mo id="S3.E9.m1.2.2.2.2.2.2.4" xref="S3.E9.m1.2.2.2.2.2.3.cmml">,</mo><msub id="S3.E9.m1.2.2.2.2.2.2.2" xref="S3.E9.m1.2.2.2.2.2.2.2.cmml"><mi id="S3.E9.m1.2.2.2.2.2.2.2.2" xref="S3.E9.m1.2.2.2.2.2.2.2.2.cmml">O</mi><mi id="S3.E9.m1.2.2.2.2.2.2.2.3" xref="S3.E9.m1.2.2.2.2.2.2.2.3.cmml">d</mi></msub><mo id="S3.E9.m1.2.2.2.2.2.2.5" stretchy="false" xref="S3.E9.m1.2.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><mrow id="S3.E9.m1.2.2.4" xref="S3.E9.m1.2.2.4.cmml"><mo id="S3.E9.m1.2.2.4.1" rspace="0em" xref="S3.E9.m1.2.2.4.1.cmml">∂</mo><msub id="S3.E9.m1.2.2.4.2" xref="S3.E9.m1.2.2.4.2.cmml"><mi id="S3.E9.m1.2.2.4.2.2" xref="S3.E9.m1.2.2.4.2.2.cmml">W</mi><mi id="S3.E9.m1.2.2.4.2.3" xref="S3.E9.m1.2.2.4.2.3.cmml">d</mi></msub></mrow></mfrac><mo id="S3.E9.m1.6.7.2" xref="S3.E9.m1.6.7.2.cmml">=</mo><mrow id="S3.E9.m1.6.7.3" xref="S3.E9.m1.6.7.3.cmml"><mfrac id="S3.E9.m1.6.7.3.2" xref="S3.E9.m1.6.7.3.2.cmml"><mo id="S3.E9.m1.6.7.3.2.2" xref="S3.E9.m1.6.7.3.2.2.cmml">∂</mo><mrow id="S3.E9.m1.6.7.3.2.3" xref="S3.E9.m1.6.7.3.2.3.cmml"><mo id="S3.E9.m1.6.7.3.2.3.1" rspace="0em" xref="S3.E9.m1.6.7.3.2.3.1.cmml">∂</mo><msub id="S3.E9.m1.6.7.3.2.3.2" xref="S3.E9.m1.6.7.3.2.3.2.cmml"><mi id="S3.E9.m1.6.7.3.2.3.2.2" xref="S3.E9.m1.6.7.3.2.3.2.2.cmml">W</mi><mi id="S3.E9.m1.6.7.3.2.3.2.3" xref="S3.E9.m1.6.7.3.2.3.2.3.cmml">d</mi></msub></mrow></mfrac><mo id="S3.E9.m1.6.7.3.1" xref="S3.E9.m1.6.7.3.1.cmml">⁢</mo><mfrac id="S3.E9.m1.4.4" xref="S3.E9.m1.4.4.cmml"><mrow id="S3.E9.m1.4.4.4" xref="S3.E9.m1.4.4.4.cmml"><msubsup id="S3.E9.m1.4.4.4.2" xref="S3.E9.m1.4.4.4.2.cmml"><mi id="S3.E9.m1.4.4.4.2.2.2" xref="S3.E9.m1.4.4.4.2.2.2.cmml">O</mi><mi id="S3.E9.m1.4.4.4.2.2.3" xref="S3.E9.m1.4.4.4.2.2.3.cmml">q</mi><mi id="S3.E9.m1.4.4.4.2.3" xref="S3.E9.m1.4.4.4.2.3.cmml">T</mi></msubsup><mo id="S3.E9.m1.4.4.4.1" xref="S3.E9.m1.4.4.4.1.cmml">⁢</mo><msub id="S3.E9.m1.4.4.4.3" xref="S3.E9.m1.4.4.4.3.cmml"><mi id="S3.E9.m1.4.4.4.3.2" xref="S3.E9.m1.4.4.4.3.2.cmml">O</mi><mi id="S3.E9.m1.4.4.4.3.3" xref="S3.E9.m1.4.4.4.3.3.cmml">d</mi></msub></mrow><mrow id="S3.E9.m1.4.4.2" xref="S3.E9.m1.4.4.2.cmml"><mrow id="S3.E9.m1.3.3.1.1.1" xref="S3.E9.m1.3.3.1.1.2.cmml"><mo id="S3.E9.m1.3.3.1.1.1.2" stretchy="false" xref="S3.E9.m1.3.3.1.1.2.1.cmml">‖</mo><msub id="S3.E9.m1.3.3.1.1.1.1" xref="S3.E9.m1.3.3.1.1.1.1.cmml"><mi id="S3.E9.m1.3.3.1.1.1.1.2" xref="S3.E9.m1.3.3.1.1.1.1.2.cmml">O</mi><mi id="S3.E9.m1.3.3.1.1.1.1.3" xref="S3.E9.m1.3.3.1.1.1.1.3.cmml">q</mi></msub><mo id="S3.E9.m1.3.3.1.1.1.3" stretchy="false" xref="S3.E9.m1.3.3.1.1.2.1.cmml">‖</mo></mrow><mo id="S3.E9.m1.4.4.2.3" xref="S3.E9.m1.4.4.2.3.cmml">⁢</mo><mrow id="S3.E9.m1.4.4.2.2.1" xref="S3.E9.m1.4.4.2.2.2.cmml"><mo id="S3.E9.m1.4.4.2.2.1.2" stretchy="false" xref="S3.E9.m1.4.4.2.2.2.1.cmml">‖</mo><msub id="S3.E9.m1.4.4.2.2.1.1" xref="S3.E9.m1.4.4.2.2.1.1.cmml"><mi id="S3.E9.m1.4.4.2.2.1.1.2" xref="S3.E9.m1.4.4.2.2.1.1.2.cmml">O</mi><mi id="S3.E9.m1.4.4.2.2.1.1.3" xref="S3.E9.m1.4.4.2.2.1.1.3.cmml">d</mi></msub><mo id="S3.E9.m1.4.4.2.2.1.3" stretchy="false" xref="S3.E9.m1.4.4.2.2.2.1.cmml">‖</mo></mrow></mrow></mfrac></mrow><mo id="S3.E9.m1.6.7.4" xref="S3.E9.m1.6.7.4.cmml">=</mo><mrow id="S3.E9.m1.6.7.5" xref="S3.E9.m1.6.7.5.cmml"><msubsup id="S3.E9.m1.6.7.5.2" xref="S3.E9.m1.6.7.5.2.cmml"><mi id="S3.E9.m1.6.7.5.2.2.2" xref="S3.E9.m1.6.7.5.2.2.2.cmml">δ</mi><msub id="S3.E9.m1.6.7.5.2.2.3" xref="S3.E9.m1.6.7.5.2.2.3.cmml"><mi id="S3.E9.m1.6.7.5.2.2.3.2" xref="S3.E9.m1.6.7.5.2.2.3.2.cmml">O</mi><mi id="S3.E9.m1.6.7.5.2.2.3.3" xref="S3.E9.m1.6.7.5.2.2.3.3.cmml">d</mi></msub><mrow id="S3.E9.m1.6.6.2.4" xref="S3.E9.m1.6.6.2.3.cmml"><mo id="S3.E9.m1.6.6.2.4.1" stretchy="false" xref="S3.E9.m1.6.6.2.3.cmml">(</mo><mi id="S3.E9.m1.5.5.1.1" xref="S3.E9.m1.5.5.1.1.cmml">q</mi><mo id="S3.E9.m1.6.6.2.4.2" xref="S3.E9.m1.6.6.2.3.cmml">,</mo><mi id="S3.E9.m1.6.6.2.2" xref="S3.E9.m1.6.6.2.2.cmml">d</mi><mo id="S3.E9.m1.6.6.2.4.3" stretchy="false" xref="S3.E9.m1.6.6.2.3.cmml">)</mo></mrow></msubsup><mo id="S3.E9.m1.6.7.5.1" xref="S3.E9.m1.6.7.5.1.cmml">⁢</mo><msubsup id="S3.E9.m1.6.7.5.3" xref="S3.E9.m1.6.7.5.3.cmml"><mi id="S3.E9.m1.6.7.5.3.2.2" xref="S3.E9.m1.6.7.5.3.2.2.cmml">h</mi><mi id="S3.E9.m1.6.7.5.3.2.3" xref="S3.E9.m1.6.7.5.3.2.3.cmml">d</mi><mi id="S3.E9.m1.6.7.5.3.3" xref="S3.E9.m1.6.7.5.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E9.m1.6b"><apply id="S3.E9.m1.6.7.cmml" xref="S3.E9.m1.6.7"><and id="S3.E9.m1.6.7a.cmml" xref="S3.E9.m1.6.7"></and><apply id="S3.E9.m1.6.7b.cmml" xref="S3.E9.m1.6.7"><eq id="S3.E9.m1.6.7.2.cmml" xref="S3.E9.m1.6.7.2"></eq><apply id="S3.E9.m1.2.2.cmml" xref="S3.E9.m1.2.2"><divide id="S3.E9.m1.2.2.3.cmml" xref="S3.E9.m1.2.2"></divide><apply id="S3.E9.m1.2.2.2.cmml" xref="S3.E9.m1.2.2.2"><partialdiff id="S3.E9.m1.2.2.2.3.cmml" xref="S3.E9.m1.2.2.2.3"></partialdiff><apply id="S3.E9.m1.2.2.2.2.cmml" xref="S3.E9.m1.2.2.2.2"><times id="S3.E9.m1.2.2.2.2.3.cmml" xref="S3.E9.m1.2.2.2.2.3"></times><ci id="S3.E9.m1.2.2.2.2.4.cmml" xref="S3.E9.m1.2.2.2.2.4">cos</ci><interval closure="open" id="S3.E9.m1.2.2.2.2.2.3.cmml" xref="S3.E9.m1.2.2.2.2.2.2"><apply id="S3.E9.m1.1.1.1.1.1.1.1.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E9.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E9.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.2">𝑂</ci><ci id="S3.E9.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.3">𝑞</ci></apply><apply id="S3.E9.m1.2.2.2.2.2.2.2.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.E9.m1.2.2.2.2.2.2.2.1.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2">subscript</csymbol><ci id="S3.E9.m1.2.2.2.2.2.2.2.2.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2">𝑂</ci><ci id="S3.E9.m1.2.2.2.2.2.2.2.3.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.3">𝑑</ci></apply></interval></apply></apply><apply id="S3.E9.m1.2.2.4.cmml" xref="S3.E9.m1.2.2.4"><partialdiff id="S3.E9.m1.2.2.4.1.cmml" xref="S3.E9.m1.2.2.4.1"></partialdiff><apply id="S3.E9.m1.2.2.4.2.cmml" xref="S3.E9.m1.2.2.4.2"><csymbol cd="ambiguous" id="S3.E9.m1.2.2.4.2.1.cmml" xref="S3.E9.m1.2.2.4.2">subscript</csymbol><ci id="S3.E9.m1.2.2.4.2.2.cmml" xref="S3.E9.m1.2.2.4.2.2">𝑊</ci><ci id="S3.E9.m1.2.2.4.2.3.cmml" xref="S3.E9.m1.2.2.4.2.3">𝑑</ci></apply></apply></apply><apply id="S3.E9.m1.6.7.3.cmml" xref="S3.E9.m1.6.7.3"><times id="S3.E9.m1.6.7.3.1.cmml" xref="S3.E9.m1.6.7.3.1"></times><apply id="S3.E9.m1.6.7.3.2.cmml" xref="S3.E9.m1.6.7.3.2"><divide id="S3.E9.m1.6.7.3.2.1.cmml" xref="S3.E9.m1.6.7.3.2"></divide><partialdiff id="S3.E9.m1.6.7.3.2.2.cmml" xref="S3.E9.m1.6.7.3.2.2"></partialdiff><apply id="S3.E9.m1.6.7.3.2.3.cmml" xref="S3.E9.m1.6.7.3.2.3"><partialdiff id="S3.E9.m1.6.7.3.2.3.1.cmml" xref="S3.E9.m1.6.7.3.2.3.1"></partialdiff><apply id="S3.E9.m1.6.7.3.2.3.2.cmml" xref="S3.E9.m1.6.7.3.2.3.2"><csymbol cd="ambiguous" id="S3.E9.m1.6.7.3.2.3.2.1.cmml" xref="S3.E9.m1.6.7.3.2.3.2">subscript</csymbol><ci id="S3.E9.m1.6.7.3.2.3.2.2.cmml" xref="S3.E9.m1.6.7.3.2.3.2.2">𝑊</ci><ci id="S3.E9.m1.6.7.3.2.3.2.3.cmml" xref="S3.E9.m1.6.7.3.2.3.2.3">𝑑</ci></apply></apply></apply><apply id="S3.E9.m1.4.4.cmml" xref="S3.E9.m1.4.4"><divide id="S3.E9.m1.4.4.3.cmml" xref="S3.E9.m1.4.4"></divide><apply id="S3.E9.m1.4.4.4.cmml" xref="S3.E9.m1.4.4.4"><times id="S3.E9.m1.4.4.4.1.cmml" xref="S3.E9.m1.4.4.4.1"></times><apply id="S3.E9.m1.4.4.4.2.cmml" xref="S3.E9.m1.4.4.4.2"><csymbol cd="ambiguous" id="S3.E9.m1.4.4.4.2.1.cmml" xref="S3.E9.m1.4.4.4.2">superscript</csymbol><apply id="S3.E9.m1.4.4.4.2.2.cmml" xref="S3.E9.m1.4.4.4.2"><csymbol cd="ambiguous" id="S3.E9.m1.4.4.4.2.2.1.cmml" xref="S3.E9.m1.4.4.4.2">subscript</csymbol><ci id="S3.E9.m1.4.4.4.2.2.2.cmml" xref="S3.E9.m1.4.4.4.2.2.2">𝑂</ci><ci id="S3.E9.m1.4.4.4.2.2.3.cmml" xref="S3.E9.m1.4.4.4.2.2.3">𝑞</ci></apply><ci id="S3.E9.m1.4.4.4.2.3.cmml" xref="S3.E9.m1.4.4.4.2.3">𝑇</ci></apply><apply id="S3.E9.m1.4.4.4.3.cmml" xref="S3.E9.m1.4.4.4.3"><csymbol cd="ambiguous" id="S3.E9.m1.4.4.4.3.1.cmml" xref="S3.E9.m1.4.4.4.3">subscript</csymbol><ci id="S3.E9.m1.4.4.4.3.2.cmml" xref="S3.E9.m1.4.4.4.3.2">𝑂</ci><ci id="S3.E9.m1.4.4.4.3.3.cmml" xref="S3.E9.m1.4.4.4.3.3">𝑑</ci></apply></apply><apply id="S3.E9.m1.4.4.2.cmml" xref="S3.E9.m1.4.4.2"><times id="S3.E9.m1.4.4.2.3.cmml" xref="S3.E9.m1.4.4.2.3"></times><apply id="S3.E9.m1.3.3.1.1.2.cmml" xref="S3.E9.m1.3.3.1.1.1"><csymbol cd="latexml" id="S3.E9.m1.3.3.1.1.2.1.cmml" xref="S3.E9.m1.3.3.1.1.1.2">norm</csymbol><apply id="S3.E9.m1.3.3.1.1.1.1.cmml" xref="S3.E9.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S3.E9.m1.3.3.1.1.1.1.1.cmml" xref="S3.E9.m1.3.3.1.1.1.1">subscript</csymbol><ci id="S3.E9.m1.3.3.1.1.1.1.2.cmml" xref="S3.E9.m1.3.3.1.1.1.1.2">𝑂</ci><ci id="S3.E9.m1.3.3.1.1.1.1.3.cmml" xref="S3.E9.m1.3.3.1.1.1.1.3">𝑞</ci></apply></apply><apply id="S3.E9.m1.4.4.2.2.2.cmml" xref="S3.E9.m1.4.4.2.2.1"><csymbol cd="latexml" id="S3.E9.m1.4.4.2.2.2.1.cmml" xref="S3.E9.m1.4.4.2.2.1.2">norm</csymbol><apply id="S3.E9.m1.4.4.2.2.1.1.cmml" xref="S3.E9.m1.4.4.2.2.1.1"><csymbol cd="ambiguous" id="S3.E9.m1.4.4.2.2.1.1.1.cmml" xref="S3.E9.m1.4.4.2.2.1.1">subscript</csymbol><ci id="S3.E9.m1.4.4.2.2.1.1.2.cmml" xref="S3.E9.m1.4.4.2.2.1.1.2">𝑂</ci><ci id="S3.E9.m1.4.4.2.2.1.1.3.cmml" xref="S3.E9.m1.4.4.2.2.1.1.3">𝑑</ci></apply></apply></apply></apply></apply></apply><apply id="S3.E9.m1.6.7c.cmml" xref="S3.E9.m1.6.7"><eq id="S3.E9.m1.6.7.4.cmml" xref="S3.E9.m1.6.7.4"></eq><share href="https://arxiv.org/html/2408.04197v2#S3.E9.m1.6.7.3.cmml" id="S3.E9.m1.6.7d.cmml" xref="S3.E9.m1.6.7"></share><apply id="S3.E9.m1.6.7.5.cmml" xref="S3.E9.m1.6.7.5"><times id="S3.E9.m1.6.7.5.1.cmml" xref="S3.E9.m1.6.7.5.1"></times><apply id="S3.E9.m1.6.7.5.2.cmml" xref="S3.E9.m1.6.7.5.2"><csymbol cd="ambiguous" id="S3.E9.m1.6.7.5.2.1.cmml" xref="S3.E9.m1.6.7.5.2">superscript</csymbol><apply id="S3.E9.m1.6.7.5.2.2.cmml" xref="S3.E9.m1.6.7.5.2"><csymbol cd="ambiguous" id="S3.E9.m1.6.7.5.2.2.1.cmml" xref="S3.E9.m1.6.7.5.2">subscript</csymbol><ci id="S3.E9.m1.6.7.5.2.2.2.cmml" xref="S3.E9.m1.6.7.5.2.2.2">𝛿</ci><apply id="S3.E9.m1.6.7.5.2.2.3.cmml" xref="S3.E9.m1.6.7.5.2.2.3"><csymbol cd="ambiguous" id="S3.E9.m1.6.7.5.2.2.3.1.cmml" xref="S3.E9.m1.6.7.5.2.2.3">subscript</csymbol><ci id="S3.E9.m1.6.7.5.2.2.3.2.cmml" xref="S3.E9.m1.6.7.5.2.2.3.2">𝑂</ci><ci id="S3.E9.m1.6.7.5.2.2.3.3.cmml" xref="S3.E9.m1.6.7.5.2.2.3.3">𝑑</ci></apply></apply><interval closure="open" id="S3.E9.m1.6.6.2.3.cmml" xref="S3.E9.m1.6.6.2.4"><ci id="S3.E9.m1.5.5.1.1.cmml" xref="S3.E9.m1.5.5.1.1">𝑞</ci><ci id="S3.E9.m1.6.6.2.2.cmml" xref="S3.E9.m1.6.6.2.2">𝑑</ci></interval></apply><apply id="S3.E9.m1.6.7.5.3.cmml" xref="S3.E9.m1.6.7.5.3"><csymbol cd="ambiguous" id="S3.E9.m1.6.7.5.3.1.cmml" xref="S3.E9.m1.6.7.5.3">superscript</csymbol><apply id="S3.E9.m1.6.7.5.3.2.cmml" xref="S3.E9.m1.6.7.5.3"><csymbol cd="ambiguous" id="S3.E9.m1.6.7.5.3.2.1.cmml" xref="S3.E9.m1.6.7.5.3">subscript</csymbol><ci id="S3.E9.m1.6.7.5.3.2.2.cmml" xref="S3.E9.m1.6.7.5.3.2.2">ℎ</ci><ci id="S3.E9.m1.6.7.5.3.2.3.cmml" xref="S3.E9.m1.6.7.5.3.2.3">𝑑</ci></apply><ci id="S3.E9.m1.6.7.5.3.3.cmml" xref="S3.E9.m1.6.7.5.3.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E9.m1.6c">{\frac{\partial\mathrm{cos}({O}_{q},{O}_{d})}{\partial W_{d}}}={\frac{\partial% }{\partial W_{d}}}{\frac{{O}_{q}^{T}{O}_{d}}{\|{O}_{q}\|\|{O}_{d}\|}}=\delta_{% {O}_{d}}^{(q,d)}h_{d}^{T}</annotation><annotation encoding="application/x-llamapun" id="S3.E9.m1.6d">divide start_ARG ∂ roman_cos ( italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT , italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ) end_ARG start_ARG ∂ italic_W start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_ARG = divide start_ARG ∂ end_ARG start_ARG ∂ italic_W start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_ARG divide start_ARG italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_ARG start_ARG ∥ italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT ∥ ∥ italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ∥ end_ARG = italic_δ start_POSTSUBSCRIPT italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT italic_h start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx2.p8"> <p class="ltx_p" id="S3.SSx2.p8.2">where <math alttext="\delta_{{O}_{q}}^{(q,d)}=bcO_{d}-acb^{3}O_{q}" class="ltx_Math" display="inline" id="S3.SSx2.p8.1.m1.2"><semantics id="S3.SSx2.p8.1.m1.2a"><mrow id="S3.SSx2.p8.1.m1.2.3" xref="S3.SSx2.p8.1.m1.2.3.cmml"><msubsup id="S3.SSx2.p8.1.m1.2.3.2" xref="S3.SSx2.p8.1.m1.2.3.2.cmml"><mi id="S3.SSx2.p8.1.m1.2.3.2.2.2" xref="S3.SSx2.p8.1.m1.2.3.2.2.2.cmml">δ</mi><msub id="S3.SSx2.p8.1.m1.2.3.2.2.3" xref="S3.SSx2.p8.1.m1.2.3.2.2.3.cmml"><mi id="S3.SSx2.p8.1.m1.2.3.2.2.3.2" xref="S3.SSx2.p8.1.m1.2.3.2.2.3.2.cmml">O</mi><mi id="S3.SSx2.p8.1.m1.2.3.2.2.3.3" xref="S3.SSx2.p8.1.m1.2.3.2.2.3.3.cmml">q</mi></msub><mrow id="S3.SSx2.p8.1.m1.2.2.2.4" xref="S3.SSx2.p8.1.m1.2.2.2.3.cmml"><mo id="S3.SSx2.p8.1.m1.2.2.2.4.1" stretchy="false" xref="S3.SSx2.p8.1.m1.2.2.2.3.cmml">(</mo><mi id="S3.SSx2.p8.1.m1.1.1.1.1" xref="S3.SSx2.p8.1.m1.1.1.1.1.cmml">q</mi><mo id="S3.SSx2.p8.1.m1.2.2.2.4.2" xref="S3.SSx2.p8.1.m1.2.2.2.3.cmml">,</mo><mi id="S3.SSx2.p8.1.m1.2.2.2.2" xref="S3.SSx2.p8.1.m1.2.2.2.2.cmml">d</mi><mo id="S3.SSx2.p8.1.m1.2.2.2.4.3" stretchy="false" xref="S3.SSx2.p8.1.m1.2.2.2.3.cmml">)</mo></mrow></msubsup><mo id="S3.SSx2.p8.1.m1.2.3.1" xref="S3.SSx2.p8.1.m1.2.3.1.cmml">=</mo><mrow id="S3.SSx2.p8.1.m1.2.3.3" xref="S3.SSx2.p8.1.m1.2.3.3.cmml"><mrow id="S3.SSx2.p8.1.m1.2.3.3.2" xref="S3.SSx2.p8.1.m1.2.3.3.2.cmml"><mi id="S3.SSx2.p8.1.m1.2.3.3.2.2" xref="S3.SSx2.p8.1.m1.2.3.3.2.2.cmml">b</mi><mo id="S3.SSx2.p8.1.m1.2.3.3.2.1" xref="S3.SSx2.p8.1.m1.2.3.3.2.1.cmml">⁢</mo><mi id="S3.SSx2.p8.1.m1.2.3.3.2.3" xref="S3.SSx2.p8.1.m1.2.3.3.2.3.cmml">c</mi><mo id="S3.SSx2.p8.1.m1.2.3.3.2.1a" xref="S3.SSx2.p8.1.m1.2.3.3.2.1.cmml">⁢</mo><msub id="S3.SSx2.p8.1.m1.2.3.3.2.4" xref="S3.SSx2.p8.1.m1.2.3.3.2.4.cmml"><mi id="S3.SSx2.p8.1.m1.2.3.3.2.4.2" xref="S3.SSx2.p8.1.m1.2.3.3.2.4.2.cmml">O</mi><mi id="S3.SSx2.p8.1.m1.2.3.3.2.4.3" xref="S3.SSx2.p8.1.m1.2.3.3.2.4.3.cmml">d</mi></msub></mrow><mo id="S3.SSx2.p8.1.m1.2.3.3.1" xref="S3.SSx2.p8.1.m1.2.3.3.1.cmml">−</mo><mrow id="S3.SSx2.p8.1.m1.2.3.3.3" xref="S3.SSx2.p8.1.m1.2.3.3.3.cmml"><mi id="S3.SSx2.p8.1.m1.2.3.3.3.2" xref="S3.SSx2.p8.1.m1.2.3.3.3.2.cmml">a</mi><mo id="S3.SSx2.p8.1.m1.2.3.3.3.1" xref="S3.SSx2.p8.1.m1.2.3.3.3.1.cmml">⁢</mo><mi id="S3.SSx2.p8.1.m1.2.3.3.3.3" xref="S3.SSx2.p8.1.m1.2.3.3.3.3.cmml">c</mi><mo id="S3.SSx2.p8.1.m1.2.3.3.3.1a" xref="S3.SSx2.p8.1.m1.2.3.3.3.1.cmml">⁢</mo><msup id="S3.SSx2.p8.1.m1.2.3.3.3.4" xref="S3.SSx2.p8.1.m1.2.3.3.3.4.cmml"><mi id="S3.SSx2.p8.1.m1.2.3.3.3.4.2" xref="S3.SSx2.p8.1.m1.2.3.3.3.4.2.cmml">b</mi><mn id="S3.SSx2.p8.1.m1.2.3.3.3.4.3" xref="S3.SSx2.p8.1.m1.2.3.3.3.4.3.cmml">3</mn></msup><mo id="S3.SSx2.p8.1.m1.2.3.3.3.1b" xref="S3.SSx2.p8.1.m1.2.3.3.3.1.cmml">⁢</mo><msub id="S3.SSx2.p8.1.m1.2.3.3.3.5" xref="S3.SSx2.p8.1.m1.2.3.3.3.5.cmml"><mi id="S3.SSx2.p8.1.m1.2.3.3.3.5.2" xref="S3.SSx2.p8.1.m1.2.3.3.3.5.2.cmml">O</mi><mi id="S3.SSx2.p8.1.m1.2.3.3.3.5.3" xref="S3.SSx2.p8.1.m1.2.3.3.3.5.3.cmml">q</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx2.p8.1.m1.2b"><apply id="S3.SSx2.p8.1.m1.2.3.cmml" xref="S3.SSx2.p8.1.m1.2.3"><eq id="S3.SSx2.p8.1.m1.2.3.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.1"></eq><apply id="S3.SSx2.p8.1.m1.2.3.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.2"><csymbol cd="ambiguous" id="S3.SSx2.p8.1.m1.2.3.2.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.2">superscript</csymbol><apply id="S3.SSx2.p8.1.m1.2.3.2.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.2"><csymbol cd="ambiguous" id="S3.SSx2.p8.1.m1.2.3.2.2.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.2">subscript</csymbol><ci id="S3.SSx2.p8.1.m1.2.3.2.2.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.2.2.2">𝛿</ci><apply id="S3.SSx2.p8.1.m1.2.3.2.2.3.cmml" xref="S3.SSx2.p8.1.m1.2.3.2.2.3"><csymbol cd="ambiguous" id="S3.SSx2.p8.1.m1.2.3.2.2.3.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.2.2.3">subscript</csymbol><ci id="S3.SSx2.p8.1.m1.2.3.2.2.3.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.2.2.3.2">𝑂</ci><ci id="S3.SSx2.p8.1.m1.2.3.2.2.3.3.cmml" xref="S3.SSx2.p8.1.m1.2.3.2.2.3.3">𝑞</ci></apply></apply><interval closure="open" id="S3.SSx2.p8.1.m1.2.2.2.3.cmml" xref="S3.SSx2.p8.1.m1.2.2.2.4"><ci id="S3.SSx2.p8.1.m1.1.1.1.1.cmml" xref="S3.SSx2.p8.1.m1.1.1.1.1">𝑞</ci><ci id="S3.SSx2.p8.1.m1.2.2.2.2.cmml" xref="S3.SSx2.p8.1.m1.2.2.2.2">𝑑</ci></interval></apply><apply id="S3.SSx2.p8.1.m1.2.3.3.cmml" xref="S3.SSx2.p8.1.m1.2.3.3"><minus id="S3.SSx2.p8.1.m1.2.3.3.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.1"></minus><apply id="S3.SSx2.p8.1.m1.2.3.3.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.2"><times id="S3.SSx2.p8.1.m1.2.3.3.2.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.2.1"></times><ci id="S3.SSx2.p8.1.m1.2.3.3.2.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.2.2">𝑏</ci><ci id="S3.SSx2.p8.1.m1.2.3.3.2.3.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.2.3">𝑐</ci><apply id="S3.SSx2.p8.1.m1.2.3.3.2.4.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.2.4"><csymbol cd="ambiguous" id="S3.SSx2.p8.1.m1.2.3.3.2.4.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.2.4">subscript</csymbol><ci id="S3.SSx2.p8.1.m1.2.3.3.2.4.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.2.4.2">𝑂</ci><ci id="S3.SSx2.p8.1.m1.2.3.3.2.4.3.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.2.4.3">𝑑</ci></apply></apply><apply id="S3.SSx2.p8.1.m1.2.3.3.3.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3"><times id="S3.SSx2.p8.1.m1.2.3.3.3.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.1"></times><ci id="S3.SSx2.p8.1.m1.2.3.3.3.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.2">𝑎</ci><ci id="S3.SSx2.p8.1.m1.2.3.3.3.3.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.3">𝑐</ci><apply id="S3.SSx2.p8.1.m1.2.3.3.3.4.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.4"><csymbol cd="ambiguous" id="S3.SSx2.p8.1.m1.2.3.3.3.4.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.4">superscript</csymbol><ci id="S3.SSx2.p8.1.m1.2.3.3.3.4.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.4.2">𝑏</ci><cn id="S3.SSx2.p8.1.m1.2.3.3.3.4.3.cmml" type="integer" xref="S3.SSx2.p8.1.m1.2.3.3.3.4.3">3</cn></apply><apply id="S3.SSx2.p8.1.m1.2.3.3.3.5.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.5"><csymbol cd="ambiguous" id="S3.SSx2.p8.1.m1.2.3.3.3.5.1.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.5">subscript</csymbol><ci id="S3.SSx2.p8.1.m1.2.3.3.3.5.2.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.5.2">𝑂</ci><ci id="S3.SSx2.p8.1.m1.2.3.3.3.5.3.cmml" xref="S3.SSx2.p8.1.m1.2.3.3.3.5.3">𝑞</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p8.1.m1.2c">\delta_{{O}_{q}}^{(q,d)}=bcO_{d}-acb^{3}O_{q}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p8.1.m1.2d">italic_δ start_POSTSUBSCRIPT italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT = italic_b italic_c italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT - italic_a italic_c italic_b start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT</annotation></semantics></math> and <math alttext="\delta_{{O}_{d}}^{(q,d)}=bcO_{q}-acb^{3}O_{d}" class="ltx_Math" display="inline" id="S3.SSx2.p8.2.m2.2"><semantics id="S3.SSx2.p8.2.m2.2a"><mrow id="S3.SSx2.p8.2.m2.2.3" xref="S3.SSx2.p8.2.m2.2.3.cmml"><msubsup id="S3.SSx2.p8.2.m2.2.3.2" xref="S3.SSx2.p8.2.m2.2.3.2.cmml"><mi id="S3.SSx2.p8.2.m2.2.3.2.2.2" xref="S3.SSx2.p8.2.m2.2.3.2.2.2.cmml">δ</mi><msub id="S3.SSx2.p8.2.m2.2.3.2.2.3" xref="S3.SSx2.p8.2.m2.2.3.2.2.3.cmml"><mi id="S3.SSx2.p8.2.m2.2.3.2.2.3.2" xref="S3.SSx2.p8.2.m2.2.3.2.2.3.2.cmml">O</mi><mi id="S3.SSx2.p8.2.m2.2.3.2.2.3.3" xref="S3.SSx2.p8.2.m2.2.3.2.2.3.3.cmml">d</mi></msub><mrow id="S3.SSx2.p8.2.m2.2.2.2.4" xref="S3.SSx2.p8.2.m2.2.2.2.3.cmml"><mo id="S3.SSx2.p8.2.m2.2.2.2.4.1" stretchy="false" xref="S3.SSx2.p8.2.m2.2.2.2.3.cmml">(</mo><mi id="S3.SSx2.p8.2.m2.1.1.1.1" xref="S3.SSx2.p8.2.m2.1.1.1.1.cmml">q</mi><mo id="S3.SSx2.p8.2.m2.2.2.2.4.2" xref="S3.SSx2.p8.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.SSx2.p8.2.m2.2.2.2.2" xref="S3.SSx2.p8.2.m2.2.2.2.2.cmml">d</mi><mo id="S3.SSx2.p8.2.m2.2.2.2.4.3" stretchy="false" xref="S3.SSx2.p8.2.m2.2.2.2.3.cmml">)</mo></mrow></msubsup><mo id="S3.SSx2.p8.2.m2.2.3.1" xref="S3.SSx2.p8.2.m2.2.3.1.cmml">=</mo><mrow id="S3.SSx2.p8.2.m2.2.3.3" xref="S3.SSx2.p8.2.m2.2.3.3.cmml"><mrow id="S3.SSx2.p8.2.m2.2.3.3.2" xref="S3.SSx2.p8.2.m2.2.3.3.2.cmml"><mi id="S3.SSx2.p8.2.m2.2.3.3.2.2" xref="S3.SSx2.p8.2.m2.2.3.3.2.2.cmml">b</mi><mo id="S3.SSx2.p8.2.m2.2.3.3.2.1" xref="S3.SSx2.p8.2.m2.2.3.3.2.1.cmml">⁢</mo><mi id="S3.SSx2.p8.2.m2.2.3.3.2.3" xref="S3.SSx2.p8.2.m2.2.3.3.2.3.cmml">c</mi><mo id="S3.SSx2.p8.2.m2.2.3.3.2.1a" xref="S3.SSx2.p8.2.m2.2.3.3.2.1.cmml">⁢</mo><msub id="S3.SSx2.p8.2.m2.2.3.3.2.4" xref="S3.SSx2.p8.2.m2.2.3.3.2.4.cmml"><mi id="S3.SSx2.p8.2.m2.2.3.3.2.4.2" xref="S3.SSx2.p8.2.m2.2.3.3.2.4.2.cmml">O</mi><mi id="S3.SSx2.p8.2.m2.2.3.3.2.4.3" xref="S3.SSx2.p8.2.m2.2.3.3.2.4.3.cmml">q</mi></msub></mrow><mo id="S3.SSx2.p8.2.m2.2.3.3.1" xref="S3.SSx2.p8.2.m2.2.3.3.1.cmml">−</mo><mrow id="S3.SSx2.p8.2.m2.2.3.3.3" xref="S3.SSx2.p8.2.m2.2.3.3.3.cmml"><mi id="S3.SSx2.p8.2.m2.2.3.3.3.2" xref="S3.SSx2.p8.2.m2.2.3.3.3.2.cmml">a</mi><mo id="S3.SSx2.p8.2.m2.2.3.3.3.1" xref="S3.SSx2.p8.2.m2.2.3.3.3.1.cmml">⁢</mo><mi id="S3.SSx2.p8.2.m2.2.3.3.3.3" xref="S3.SSx2.p8.2.m2.2.3.3.3.3.cmml">c</mi><mo id="S3.SSx2.p8.2.m2.2.3.3.3.1a" xref="S3.SSx2.p8.2.m2.2.3.3.3.1.cmml">⁢</mo><msup id="S3.SSx2.p8.2.m2.2.3.3.3.4" xref="S3.SSx2.p8.2.m2.2.3.3.3.4.cmml"><mi id="S3.SSx2.p8.2.m2.2.3.3.3.4.2" xref="S3.SSx2.p8.2.m2.2.3.3.3.4.2.cmml">b</mi><mn id="S3.SSx2.p8.2.m2.2.3.3.3.4.3" xref="S3.SSx2.p8.2.m2.2.3.3.3.4.3.cmml">3</mn></msup><mo id="S3.SSx2.p8.2.m2.2.3.3.3.1b" xref="S3.SSx2.p8.2.m2.2.3.3.3.1.cmml">⁢</mo><msub id="S3.SSx2.p8.2.m2.2.3.3.3.5" xref="S3.SSx2.p8.2.m2.2.3.3.3.5.cmml"><mi id="S3.SSx2.p8.2.m2.2.3.3.3.5.2" xref="S3.SSx2.p8.2.m2.2.3.3.3.5.2.cmml">O</mi><mi id="S3.SSx2.p8.2.m2.2.3.3.3.5.3" xref="S3.SSx2.p8.2.m2.2.3.3.3.5.3.cmml">d</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx2.p8.2.m2.2b"><apply id="S3.SSx2.p8.2.m2.2.3.cmml" xref="S3.SSx2.p8.2.m2.2.3"><eq id="S3.SSx2.p8.2.m2.2.3.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.1"></eq><apply id="S3.SSx2.p8.2.m2.2.3.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.2"><csymbol cd="ambiguous" id="S3.SSx2.p8.2.m2.2.3.2.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.2">superscript</csymbol><apply id="S3.SSx2.p8.2.m2.2.3.2.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.2"><csymbol cd="ambiguous" id="S3.SSx2.p8.2.m2.2.3.2.2.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.2">subscript</csymbol><ci id="S3.SSx2.p8.2.m2.2.3.2.2.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.2.2.2">𝛿</ci><apply id="S3.SSx2.p8.2.m2.2.3.2.2.3.cmml" xref="S3.SSx2.p8.2.m2.2.3.2.2.3"><csymbol cd="ambiguous" id="S3.SSx2.p8.2.m2.2.3.2.2.3.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.2.2.3">subscript</csymbol><ci id="S3.SSx2.p8.2.m2.2.3.2.2.3.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.2.2.3.2">𝑂</ci><ci id="S3.SSx2.p8.2.m2.2.3.2.2.3.3.cmml" xref="S3.SSx2.p8.2.m2.2.3.2.2.3.3">𝑑</ci></apply></apply><interval closure="open" id="S3.SSx2.p8.2.m2.2.2.2.3.cmml" xref="S3.SSx2.p8.2.m2.2.2.2.4"><ci id="S3.SSx2.p8.2.m2.1.1.1.1.cmml" xref="S3.SSx2.p8.2.m2.1.1.1.1">𝑞</ci><ci id="S3.SSx2.p8.2.m2.2.2.2.2.cmml" xref="S3.SSx2.p8.2.m2.2.2.2.2">𝑑</ci></interval></apply><apply id="S3.SSx2.p8.2.m2.2.3.3.cmml" xref="S3.SSx2.p8.2.m2.2.3.3"><minus id="S3.SSx2.p8.2.m2.2.3.3.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.1"></minus><apply id="S3.SSx2.p8.2.m2.2.3.3.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.2"><times id="S3.SSx2.p8.2.m2.2.3.3.2.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.2.1"></times><ci id="S3.SSx2.p8.2.m2.2.3.3.2.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.2.2">𝑏</ci><ci id="S3.SSx2.p8.2.m2.2.3.3.2.3.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.2.3">𝑐</ci><apply id="S3.SSx2.p8.2.m2.2.3.3.2.4.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.2.4"><csymbol cd="ambiguous" id="S3.SSx2.p8.2.m2.2.3.3.2.4.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.2.4">subscript</csymbol><ci id="S3.SSx2.p8.2.m2.2.3.3.2.4.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.2.4.2">𝑂</ci><ci id="S3.SSx2.p8.2.m2.2.3.3.2.4.3.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.2.4.3">𝑞</ci></apply></apply><apply id="S3.SSx2.p8.2.m2.2.3.3.3.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3"><times id="S3.SSx2.p8.2.m2.2.3.3.3.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.1"></times><ci id="S3.SSx2.p8.2.m2.2.3.3.3.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.2">𝑎</ci><ci id="S3.SSx2.p8.2.m2.2.3.3.3.3.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.3">𝑐</ci><apply id="S3.SSx2.p8.2.m2.2.3.3.3.4.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.4"><csymbol cd="ambiguous" id="S3.SSx2.p8.2.m2.2.3.3.3.4.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.4">superscript</csymbol><ci id="S3.SSx2.p8.2.m2.2.3.3.3.4.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.4.2">𝑏</ci><cn id="S3.SSx2.p8.2.m2.2.3.3.3.4.3.cmml" type="integer" xref="S3.SSx2.p8.2.m2.2.3.3.3.4.3">3</cn></apply><apply id="S3.SSx2.p8.2.m2.2.3.3.3.5.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.5"><csymbol cd="ambiguous" id="S3.SSx2.p8.2.m2.2.3.3.3.5.1.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.5">subscript</csymbol><ci id="S3.SSx2.p8.2.m2.2.3.3.3.5.2.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.5.2">𝑂</ci><ci id="S3.SSx2.p8.2.m2.2.3.3.3.5.3.cmml" xref="S3.SSx2.p8.2.m2.2.3.3.3.5.3">𝑑</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p8.2.m2.2c">\delta_{{O}_{d}}^{(q,d)}=bcO_{q}-acb^{3}O_{d}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p8.2.m2.2d">italic_δ start_POSTSUBSCRIPT italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT = italic_b italic_c italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT - italic_a italic_c italic_b start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT</annotation></semantics></math></p> </div> <figure class="ltx_figure" id="S3.F2"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_4"> <figure class="ltx_figure ltx_figure_panel ltx_align_center" id="S3.F2.sf1"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="464" id="S3.F2.sf1.g1" src="extracted/6016158/figures/Figure_2.png" width="598"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span class="ltx_text" id="S3.F2.sf1.2.1.1" style="font-size:90%;">(a)</span> </span><span class="ltx_text" id="S3.F2.sf1.3.2" style="font-size:90%;">Atomic Strategies: Test-1</span></figcaption> </figure> </div> <div class="ltx_flex_cell ltx_flex_size_4"> <figure class="ltx_figure ltx_figure_panel ltx_align_center" id="S3.F2.sf2"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="464" id="S3.F2.sf2.g1" src="extracted/6016158/figures/Figure_3.png" width="598"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span class="ltx_text" id="S3.F2.sf2.2.1.1" style="font-size:90%;">(b)</span> </span><span class="ltx_text" id="S3.F2.sf2.3.2" style="font-size:90%;">Atomic Strategies: Test-2</span></figcaption> </figure> </div> <div class="ltx_flex_cell ltx_flex_size_4"> <figure class="ltx_figure ltx_figure_panel ltx_align_center" id="S3.F2.sf3"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="464" id="S3.F2.sf3.g1" src="extracted/6016158/figures/Figure_4.png" width="598"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span class="ltx_text" id="S3.F2.sf3.2.1.1" style="font-size:90%;">(c)</span> </span><span class="ltx_text" id="S3.F2.sf3.3.2" style="font-size:90%;">Hybrid Strategy: Test-1 and Test-2</span></figcaption> </figure> </div> </div> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span class="ltx_text" id="S3.F2.2.1.1" style="font-size:90%;">Figure 2</span>. </span><span class="ltx_text" id="S3.F2.3.2" style="font-size:90%;">Performance of Pairwise Judgments Formed by Various Strategies for Training Semantic Embedding Models</span></figcaption><div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_1"><span class="ltx_ERROR ltx_centering ltx_figure_panel undefined" id="S3.F2.4">\Description</span></div> <div class="ltx_flex_break"></div> <div class="ltx_flex_cell ltx_flex_size_1"> <p class="ltx_p ltx_figure_panel ltx_align_center" id="S3.F2.5">[Performance of Pairwise Judgments Formed by Various Strategies for Training Semantic Embedding Models]</p> </div> </div> </figure> <div class="ltx_para" id="S3.SSx2.p9"> <p class="ltx_p" id="S3.SSx2.p9.3">Similarly, we can compute the gradient of intermediate representation <math alttext="\partial\Delta/\partial h" class="ltx_Math" display="inline" id="S3.SSx2.p9.1.m1.1"><semantics id="S3.SSx2.p9.1.m1.1a"><mrow id="S3.SSx2.p9.1.m1.1.1" xref="S3.SSx2.p9.1.m1.1.1.cmml"><mo id="S3.SSx2.p9.1.m1.1.1.1" rspace="0em" xref="S3.SSx2.p9.1.m1.1.1.1.cmml">∂</mo><mrow id="S3.SSx2.p9.1.m1.1.1.2" xref="S3.SSx2.p9.1.m1.1.1.2.cmml"><mi id="S3.SSx2.p9.1.m1.1.1.2.2" mathvariant="normal" xref="S3.SSx2.p9.1.m1.1.1.2.2.cmml">Δ</mi><mo id="S3.SSx2.p9.1.m1.1.1.2.1" xref="S3.SSx2.p9.1.m1.1.1.2.1.cmml">/</mo><mrow id="S3.SSx2.p9.1.m1.1.1.2.3" xref="S3.SSx2.p9.1.m1.1.1.2.3.cmml"><mo id="S3.SSx2.p9.1.m1.1.1.2.3.1" lspace="0em" rspace="0em" xref="S3.SSx2.p9.1.m1.1.1.2.3.1.cmml">∂</mo><mi id="S3.SSx2.p9.1.m1.1.1.2.3.2" xref="S3.SSx2.p9.1.m1.1.1.2.3.2.cmml">h</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx2.p9.1.m1.1b"><apply id="S3.SSx2.p9.1.m1.1.1.cmml" xref="S3.SSx2.p9.1.m1.1.1"><partialdiff id="S3.SSx2.p9.1.m1.1.1.1.cmml" xref="S3.SSx2.p9.1.m1.1.1.1"></partialdiff><apply id="S3.SSx2.p9.1.m1.1.1.2.cmml" xref="S3.SSx2.p9.1.m1.1.1.2"><divide id="S3.SSx2.p9.1.m1.1.1.2.1.cmml" xref="S3.SSx2.p9.1.m1.1.1.2.1"></divide><ci id="S3.SSx2.p9.1.m1.1.1.2.2.cmml" xref="S3.SSx2.p9.1.m1.1.1.2.2">Δ</ci><apply id="S3.SSx2.p9.1.m1.1.1.2.3.cmml" xref="S3.SSx2.p9.1.m1.1.1.2.3"><partialdiff id="S3.SSx2.p9.1.m1.1.1.2.3.1.cmml" xref="S3.SSx2.p9.1.m1.1.1.2.3.1"></partialdiff><ci id="S3.SSx2.p9.1.m1.1.1.2.3.2.cmml" xref="S3.SSx2.p9.1.m1.1.1.2.3.2">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p9.1.m1.1c">\partial\Delta/\partial h</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p9.1.m1.1d">∂ roman_Δ / ∂ italic_h</annotation></semantics></math>, then get the gradient of element-wise adding result <math alttext="\partial\Delta/\partial v" class="ltx_Math" display="inline" id="S3.SSx2.p9.2.m2.1"><semantics id="S3.SSx2.p9.2.m2.1a"><mrow id="S3.SSx2.p9.2.m2.1.1" xref="S3.SSx2.p9.2.m2.1.1.cmml"><mo id="S3.SSx2.p9.2.m2.1.1.1" rspace="0em" xref="S3.SSx2.p9.2.m2.1.1.1.cmml">∂</mo><mrow id="S3.SSx2.p9.2.m2.1.1.2" xref="S3.SSx2.p9.2.m2.1.1.2.cmml"><mi id="S3.SSx2.p9.2.m2.1.1.2.2" mathvariant="normal" xref="S3.SSx2.p9.2.m2.1.1.2.2.cmml">Δ</mi><mo id="S3.SSx2.p9.2.m2.1.1.2.1" xref="S3.SSx2.p9.2.m2.1.1.2.1.cmml">/</mo><mrow id="S3.SSx2.p9.2.m2.1.1.2.3" xref="S3.SSx2.p9.2.m2.1.1.2.3.cmml"><mo id="S3.SSx2.p9.2.m2.1.1.2.3.1" lspace="0em" rspace="0em" xref="S3.SSx2.p9.2.m2.1.1.2.3.1.cmml">∂</mo><mi id="S3.SSx2.p9.2.m2.1.1.2.3.2" xref="S3.SSx2.p9.2.m2.1.1.2.3.2.cmml">v</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SSx2.p9.2.m2.1b"><apply id="S3.SSx2.p9.2.m2.1.1.cmml" xref="S3.SSx2.p9.2.m2.1.1"><partialdiff id="S3.SSx2.p9.2.m2.1.1.1.cmml" xref="S3.SSx2.p9.2.m2.1.1.1"></partialdiff><apply id="S3.SSx2.p9.2.m2.1.1.2.cmml" xref="S3.SSx2.p9.2.m2.1.1.2"><divide id="S3.SSx2.p9.2.m2.1.1.2.1.cmml" xref="S3.SSx2.p9.2.m2.1.1.2.1"></divide><ci id="S3.SSx2.p9.2.m2.1.1.2.2.cmml" xref="S3.SSx2.p9.2.m2.1.1.2.2">Δ</ci><apply id="S3.SSx2.p9.2.m2.1.1.2.3.cmml" xref="S3.SSx2.p9.2.m2.1.1.2.3"><partialdiff id="S3.SSx2.p9.2.m2.1.1.2.3.1.cmml" xref="S3.SSx2.p9.2.m2.1.1.2.3.1"></partialdiff><ci id="S3.SSx2.p9.2.m2.1.1.2.3.2.cmml" xref="S3.SSx2.p9.2.m2.1.1.2.3.2">𝑣</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p9.2.m2.1c">\partial\Delta/\partial v</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p9.2.m2.1d">∂ roman_Δ / ∂ italic_v</annotation></semantics></math>. With softsign function in our model, each <math alttext="\delta" class="ltx_Math" display="inline" id="S3.SSx2.p9.3.m3.1"><semantics id="S3.SSx2.p9.3.m3.1a"><mi id="S3.SSx2.p9.3.m3.1.1" xref="S3.SSx2.p9.3.m3.1.1.cmml">δ</mi><annotation-xml encoding="MathML-Content" id="S3.SSx2.p9.3.m3.1b"><ci id="S3.SSx2.p9.3.m3.1.1.cmml" xref="S3.SSx2.p9.3.m3.1.1">𝛿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p9.3.m3.1c">\delta</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p9.3.m3.1d">italic_δ</annotation></semantics></math> in the element-wise adding result can be calculated by:</p> </div> <div class="ltx_para" id="S3.SSx2.p10"> <table class="ltx_equation ltx_eqn_table" id="S3.E10"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(10)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="\delta_{v_{q}}^{(q,d)}=\frac{1}{(1+|V_{q}|)^{2}}\circ W_{q}^{T}\delta_{{O}_{q}% }^{(q,d)}" class="ltx_Math" display="block" id="S3.E10.m1.5"><semantics id="S3.E10.m1.5a"><mrow id="S3.E10.m1.5.6" xref="S3.E10.m1.5.6.cmml"><msubsup id="S3.E10.m1.5.6.2" xref="S3.E10.m1.5.6.2.cmml"><mi id="S3.E10.m1.5.6.2.2.2" xref="S3.E10.m1.5.6.2.2.2.cmml">δ</mi><msub id="S3.E10.m1.5.6.2.2.3" xref="S3.E10.m1.5.6.2.2.3.cmml"><mi id="S3.E10.m1.5.6.2.2.3.2" xref="S3.E10.m1.5.6.2.2.3.2.cmml">v</mi><mi id="S3.E10.m1.5.6.2.2.3.3" xref="S3.E10.m1.5.6.2.2.3.3.cmml">q</mi></msub><mrow id="S3.E10.m1.2.2.2.4" xref="S3.E10.m1.2.2.2.3.cmml"><mo id="S3.E10.m1.2.2.2.4.1" stretchy="false" xref="S3.E10.m1.2.2.2.3.cmml">(</mo><mi id="S3.E10.m1.1.1.1.1" xref="S3.E10.m1.1.1.1.1.cmml">q</mi><mo id="S3.E10.m1.2.2.2.4.2" xref="S3.E10.m1.2.2.2.3.cmml">,</mo><mi id="S3.E10.m1.2.2.2.2" xref="S3.E10.m1.2.2.2.2.cmml">d</mi><mo id="S3.E10.m1.2.2.2.4.3" stretchy="false" xref="S3.E10.m1.2.2.2.3.cmml">)</mo></mrow></msubsup><mo id="S3.E10.m1.5.6.1" xref="S3.E10.m1.5.6.1.cmml">=</mo><mrow id="S3.E10.m1.5.6.3" xref="S3.E10.m1.5.6.3.cmml"><mrow id="S3.E10.m1.5.6.3.2" xref="S3.E10.m1.5.6.3.2.cmml"><mfrac id="S3.E10.m1.3.3" xref="S3.E10.m1.3.3.cmml"><mn id="S3.E10.m1.3.3.3" xref="S3.E10.m1.3.3.3.cmml">1</mn><msup id="S3.E10.m1.3.3.1" xref="S3.E10.m1.3.3.1.cmml"><mrow id="S3.E10.m1.3.3.1.1.1" xref="S3.E10.m1.3.3.1.1.1.1.cmml"><mo id="S3.E10.m1.3.3.1.1.1.2" stretchy="false" xref="S3.E10.m1.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.E10.m1.3.3.1.1.1.1" xref="S3.E10.m1.3.3.1.1.1.1.cmml"><mn id="S3.E10.m1.3.3.1.1.1.1.3" xref="S3.E10.m1.3.3.1.1.1.1.3.cmml">1</mn><mo id="S3.E10.m1.3.3.1.1.1.1.2" xref="S3.E10.m1.3.3.1.1.1.1.2.cmml">+</mo><mrow id="S3.E10.m1.3.3.1.1.1.1.1.1" xref="S3.E10.m1.3.3.1.1.1.1.1.2.cmml"><mo id="S3.E10.m1.3.3.1.1.1.1.1.1.2" stretchy="false" xref="S3.E10.m1.3.3.1.1.1.1.1.2.1.cmml">|</mo><msub id="S3.E10.m1.3.3.1.1.1.1.1.1.1" xref="S3.E10.m1.3.3.1.1.1.1.1.1.1.cmml"><mi id="S3.E10.m1.3.3.1.1.1.1.1.1.1.2" xref="S3.E10.m1.3.3.1.1.1.1.1.1.1.2.cmml">V</mi><mi id="S3.E10.m1.3.3.1.1.1.1.1.1.1.3" xref="S3.E10.m1.3.3.1.1.1.1.1.1.1.3.cmml">q</mi></msub><mo id="S3.E10.m1.3.3.1.1.1.1.1.1.3" stretchy="false" xref="S3.E10.m1.3.3.1.1.1.1.1.2.1.cmml">|</mo></mrow></mrow><mo id="S3.E10.m1.3.3.1.1.1.3" stretchy="false" xref="S3.E10.m1.3.3.1.1.1.1.cmml">)</mo></mrow><mn id="S3.E10.m1.3.3.1.3" xref="S3.E10.m1.3.3.1.3.cmml">2</mn></msup></mfrac><mo id="S3.E10.m1.5.6.3.2.1" lspace="0.222em" rspace="0.222em" xref="S3.E10.m1.5.6.3.2.1.cmml">∘</mo><msubsup id="S3.E10.m1.5.6.3.2.2" xref="S3.E10.m1.5.6.3.2.2.cmml"><mi id="S3.E10.m1.5.6.3.2.2.2.2" xref="S3.E10.m1.5.6.3.2.2.2.2.cmml">W</mi><mi id="S3.E10.m1.5.6.3.2.2.2.3" xref="S3.E10.m1.5.6.3.2.2.2.3.cmml">q</mi><mi id="S3.E10.m1.5.6.3.2.2.3" xref="S3.E10.m1.5.6.3.2.2.3.cmml">T</mi></msubsup></mrow><mo id="S3.E10.m1.5.6.3.1" xref="S3.E10.m1.5.6.3.1.cmml">⁢</mo><msubsup id="S3.E10.m1.5.6.3.3" xref="S3.E10.m1.5.6.3.3.cmml"><mi id="S3.E10.m1.5.6.3.3.2.2" xref="S3.E10.m1.5.6.3.3.2.2.cmml">δ</mi><msub id="S3.E10.m1.5.6.3.3.2.3" xref="S3.E10.m1.5.6.3.3.2.3.cmml"><mi id="S3.E10.m1.5.6.3.3.2.3.2" xref="S3.E10.m1.5.6.3.3.2.3.2.cmml">O</mi><mi id="S3.E10.m1.5.6.3.3.2.3.3" xref="S3.E10.m1.5.6.3.3.2.3.3.cmml">q</mi></msub><mrow id="S3.E10.m1.5.5.2.4" xref="S3.E10.m1.5.5.2.3.cmml"><mo id="S3.E10.m1.5.5.2.4.1" stretchy="false" xref="S3.E10.m1.5.5.2.3.cmml">(</mo><mi id="S3.E10.m1.4.4.1.1" xref="S3.E10.m1.4.4.1.1.cmml">q</mi><mo id="S3.E10.m1.5.5.2.4.2" xref="S3.E10.m1.5.5.2.3.cmml">,</mo><mi id="S3.E10.m1.5.5.2.2" xref="S3.E10.m1.5.5.2.2.cmml">d</mi><mo id="S3.E10.m1.5.5.2.4.3" stretchy="false" xref="S3.E10.m1.5.5.2.3.cmml">)</mo></mrow></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E10.m1.5b"><apply id="S3.E10.m1.5.6.cmml" xref="S3.E10.m1.5.6"><eq id="S3.E10.m1.5.6.1.cmml" xref="S3.E10.m1.5.6.1"></eq><apply id="S3.E10.m1.5.6.2.cmml" xref="S3.E10.m1.5.6.2"><csymbol cd="ambiguous" id="S3.E10.m1.5.6.2.1.cmml" xref="S3.E10.m1.5.6.2">superscript</csymbol><apply id="S3.E10.m1.5.6.2.2.cmml" xref="S3.E10.m1.5.6.2"><csymbol cd="ambiguous" id="S3.E10.m1.5.6.2.2.1.cmml" xref="S3.E10.m1.5.6.2">subscript</csymbol><ci id="S3.E10.m1.5.6.2.2.2.cmml" xref="S3.E10.m1.5.6.2.2.2">𝛿</ci><apply id="S3.E10.m1.5.6.2.2.3.cmml" xref="S3.E10.m1.5.6.2.2.3"><csymbol cd="ambiguous" id="S3.E10.m1.5.6.2.2.3.1.cmml" xref="S3.E10.m1.5.6.2.2.3">subscript</csymbol><ci id="S3.E10.m1.5.6.2.2.3.2.cmml" xref="S3.E10.m1.5.6.2.2.3.2">𝑣</ci><ci id="S3.E10.m1.5.6.2.2.3.3.cmml" xref="S3.E10.m1.5.6.2.2.3.3">𝑞</ci></apply></apply><interval closure="open" id="S3.E10.m1.2.2.2.3.cmml" xref="S3.E10.m1.2.2.2.4"><ci id="S3.E10.m1.1.1.1.1.cmml" xref="S3.E10.m1.1.1.1.1">𝑞</ci><ci id="S3.E10.m1.2.2.2.2.cmml" xref="S3.E10.m1.2.2.2.2">𝑑</ci></interval></apply><apply id="S3.E10.m1.5.6.3.cmml" xref="S3.E10.m1.5.6.3"><times id="S3.E10.m1.5.6.3.1.cmml" xref="S3.E10.m1.5.6.3.1"></times><apply id="S3.E10.m1.5.6.3.2.cmml" xref="S3.E10.m1.5.6.3.2"><compose id="S3.E10.m1.5.6.3.2.1.cmml" xref="S3.E10.m1.5.6.3.2.1"></compose><apply id="S3.E10.m1.3.3.cmml" xref="S3.E10.m1.3.3"><divide id="S3.E10.m1.3.3.2.cmml" xref="S3.E10.m1.3.3"></divide><cn id="S3.E10.m1.3.3.3.cmml" type="integer" xref="S3.E10.m1.3.3.3">1</cn><apply id="S3.E10.m1.3.3.1.cmml" xref="S3.E10.m1.3.3.1"><csymbol cd="ambiguous" id="S3.E10.m1.3.3.1.2.cmml" xref="S3.E10.m1.3.3.1">superscript</csymbol><apply id="S3.E10.m1.3.3.1.1.1.1.cmml" xref="S3.E10.m1.3.3.1.1.1"><plus id="S3.E10.m1.3.3.1.1.1.1.2.cmml" xref="S3.E10.m1.3.3.1.1.1.1.2"></plus><cn id="S3.E10.m1.3.3.1.1.1.1.3.cmml" type="integer" xref="S3.E10.m1.3.3.1.1.1.1.3">1</cn><apply id="S3.E10.m1.3.3.1.1.1.1.1.2.cmml" xref="S3.E10.m1.3.3.1.1.1.1.1.1"><abs id="S3.E10.m1.3.3.1.1.1.1.1.2.1.cmml" xref="S3.E10.m1.3.3.1.1.1.1.1.1.2"></abs><apply id="S3.E10.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S3.E10.m1.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E10.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="S3.E10.m1.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E10.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S3.E10.m1.3.3.1.1.1.1.1.1.1.2">𝑉</ci><ci id="S3.E10.m1.3.3.1.1.1.1.1.1.1.3.cmml" xref="S3.E10.m1.3.3.1.1.1.1.1.1.1.3">𝑞</ci></apply></apply></apply><cn id="S3.E10.m1.3.3.1.3.cmml" type="integer" xref="S3.E10.m1.3.3.1.3">2</cn></apply></apply><apply id="S3.E10.m1.5.6.3.2.2.cmml" xref="S3.E10.m1.5.6.3.2.2"><csymbol cd="ambiguous" id="S3.E10.m1.5.6.3.2.2.1.cmml" xref="S3.E10.m1.5.6.3.2.2">superscript</csymbol><apply id="S3.E10.m1.5.6.3.2.2.2.cmml" xref="S3.E10.m1.5.6.3.2.2"><csymbol cd="ambiguous" id="S3.E10.m1.5.6.3.2.2.2.1.cmml" xref="S3.E10.m1.5.6.3.2.2">subscript</csymbol><ci id="S3.E10.m1.5.6.3.2.2.2.2.cmml" xref="S3.E10.m1.5.6.3.2.2.2.2">𝑊</ci><ci id="S3.E10.m1.5.6.3.2.2.2.3.cmml" xref="S3.E10.m1.5.6.3.2.2.2.3">𝑞</ci></apply><ci id="S3.E10.m1.5.6.3.2.2.3.cmml" xref="S3.E10.m1.5.6.3.2.2.3">𝑇</ci></apply></apply><apply id="S3.E10.m1.5.6.3.3.cmml" xref="S3.E10.m1.5.6.3.3"><csymbol cd="ambiguous" id="S3.E10.m1.5.6.3.3.1.cmml" xref="S3.E10.m1.5.6.3.3">superscript</csymbol><apply id="S3.E10.m1.5.6.3.3.2.cmml" xref="S3.E10.m1.5.6.3.3"><csymbol cd="ambiguous" id="S3.E10.m1.5.6.3.3.2.1.cmml" xref="S3.E10.m1.5.6.3.3">subscript</csymbol><ci id="S3.E10.m1.5.6.3.3.2.2.cmml" xref="S3.E10.m1.5.6.3.3.2.2">𝛿</ci><apply id="S3.E10.m1.5.6.3.3.2.3.cmml" xref="S3.E10.m1.5.6.3.3.2.3"><csymbol cd="ambiguous" id="S3.E10.m1.5.6.3.3.2.3.1.cmml" xref="S3.E10.m1.5.6.3.3.2.3">subscript</csymbol><ci id="S3.E10.m1.5.6.3.3.2.3.2.cmml" xref="S3.E10.m1.5.6.3.3.2.3.2">𝑂</ci><ci id="S3.E10.m1.5.6.3.3.2.3.3.cmml" xref="S3.E10.m1.5.6.3.3.2.3.3">𝑞</ci></apply></apply><interval closure="open" id="S3.E10.m1.5.5.2.3.cmml" xref="S3.E10.m1.5.5.2.4"><ci id="S3.E10.m1.4.4.1.1.cmml" xref="S3.E10.m1.4.4.1.1">𝑞</ci><ci id="S3.E10.m1.5.5.2.2.cmml" xref="S3.E10.m1.5.5.2.2">𝑑</ci></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E10.m1.5c">\delta_{v_{q}}^{(q,d)}=\frac{1}{(1+|V_{q}|)^{2}}\circ W_{q}^{T}\delta_{{O}_{q}% }^{(q,d)}</annotation><annotation encoding="application/x-llamapun" id="S3.E10.m1.5d">italic_δ start_POSTSUBSCRIPT italic_v start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT = divide start_ARG 1 end_ARG start_ARG ( 1 + | italic_V start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT | ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ∘ italic_W start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_δ start_POSTSUBSCRIPT italic_O start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx2.p11"> <table class="ltx_equation ltx_eqn_table" id="S3.E11"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_left">(11)</span></td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="\delta_{v_{d}}^{(q,d)}=\frac{1}{(1+|V_{d}|)^{2}}\circ W_{d}^{T}\delta_{O_{d}}^% {(q,d)}" class="ltx_Math" display="block" id="S3.E11.m1.5"><semantics id="S3.E11.m1.5a"><mrow id="S3.E11.m1.5.6" xref="S3.E11.m1.5.6.cmml"><msubsup id="S3.E11.m1.5.6.2" xref="S3.E11.m1.5.6.2.cmml"><mi id="S3.E11.m1.5.6.2.2.2" xref="S3.E11.m1.5.6.2.2.2.cmml">δ</mi><msub id="S3.E11.m1.5.6.2.2.3" xref="S3.E11.m1.5.6.2.2.3.cmml"><mi id="S3.E11.m1.5.6.2.2.3.2" xref="S3.E11.m1.5.6.2.2.3.2.cmml">v</mi><mi id="S3.E11.m1.5.6.2.2.3.3" xref="S3.E11.m1.5.6.2.2.3.3.cmml">d</mi></msub><mrow id="S3.E11.m1.2.2.2.4" xref="S3.E11.m1.2.2.2.3.cmml"><mo id="S3.E11.m1.2.2.2.4.1" stretchy="false" xref="S3.E11.m1.2.2.2.3.cmml">(</mo><mi id="S3.E11.m1.1.1.1.1" xref="S3.E11.m1.1.1.1.1.cmml">q</mi><mo id="S3.E11.m1.2.2.2.4.2" xref="S3.E11.m1.2.2.2.3.cmml">,</mo><mi id="S3.E11.m1.2.2.2.2" xref="S3.E11.m1.2.2.2.2.cmml">d</mi><mo id="S3.E11.m1.2.2.2.4.3" stretchy="false" xref="S3.E11.m1.2.2.2.3.cmml">)</mo></mrow></msubsup><mo id="S3.E11.m1.5.6.1" xref="S3.E11.m1.5.6.1.cmml">=</mo><mrow id="S3.E11.m1.5.6.3" xref="S3.E11.m1.5.6.3.cmml"><mrow id="S3.E11.m1.5.6.3.2" xref="S3.E11.m1.5.6.3.2.cmml"><mfrac id="S3.E11.m1.3.3" xref="S3.E11.m1.3.3.cmml"><mn id="S3.E11.m1.3.3.3" xref="S3.E11.m1.3.3.3.cmml">1</mn><msup id="S3.E11.m1.3.3.1" xref="S3.E11.m1.3.3.1.cmml"><mrow id="S3.E11.m1.3.3.1.1.1" xref="S3.E11.m1.3.3.1.1.1.1.cmml"><mo id="S3.E11.m1.3.3.1.1.1.2" stretchy="false" xref="S3.E11.m1.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.E11.m1.3.3.1.1.1.1" xref="S3.E11.m1.3.3.1.1.1.1.cmml"><mn id="S3.E11.m1.3.3.1.1.1.1.3" xref="S3.E11.m1.3.3.1.1.1.1.3.cmml">1</mn><mo id="S3.E11.m1.3.3.1.1.1.1.2" xref="S3.E11.m1.3.3.1.1.1.1.2.cmml">+</mo><mrow id="S3.E11.m1.3.3.1.1.1.1.1.1" xref="S3.E11.m1.3.3.1.1.1.1.1.2.cmml"><mo id="S3.E11.m1.3.3.1.1.1.1.1.1.2" stretchy="false" xref="S3.E11.m1.3.3.1.1.1.1.1.2.1.cmml">|</mo><msub id="S3.E11.m1.3.3.1.1.1.1.1.1.1" xref="S3.E11.m1.3.3.1.1.1.1.1.1.1.cmml"><mi id="S3.E11.m1.3.3.1.1.1.1.1.1.1.2" xref="S3.E11.m1.3.3.1.1.1.1.1.1.1.2.cmml">V</mi><mi id="S3.E11.m1.3.3.1.1.1.1.1.1.1.3" xref="S3.E11.m1.3.3.1.1.1.1.1.1.1.3.cmml">d</mi></msub><mo id="S3.E11.m1.3.3.1.1.1.1.1.1.3" stretchy="false" xref="S3.E11.m1.3.3.1.1.1.1.1.2.1.cmml">|</mo></mrow></mrow><mo id="S3.E11.m1.3.3.1.1.1.3" stretchy="false" xref="S3.E11.m1.3.3.1.1.1.1.cmml">)</mo></mrow><mn id="S3.E11.m1.3.3.1.3" xref="S3.E11.m1.3.3.1.3.cmml">2</mn></msup></mfrac><mo id="S3.E11.m1.5.6.3.2.1" lspace="0.222em" rspace="0.222em" xref="S3.E11.m1.5.6.3.2.1.cmml">∘</mo><msubsup id="S3.E11.m1.5.6.3.2.2" xref="S3.E11.m1.5.6.3.2.2.cmml"><mi id="S3.E11.m1.5.6.3.2.2.2.2" xref="S3.E11.m1.5.6.3.2.2.2.2.cmml">W</mi><mi id="S3.E11.m1.5.6.3.2.2.2.3" xref="S3.E11.m1.5.6.3.2.2.2.3.cmml">d</mi><mi id="S3.E11.m1.5.6.3.2.2.3" xref="S3.E11.m1.5.6.3.2.2.3.cmml">T</mi></msubsup></mrow><mo id="S3.E11.m1.5.6.3.1" xref="S3.E11.m1.5.6.3.1.cmml">⁢</mo><msubsup id="S3.E11.m1.5.6.3.3" xref="S3.E11.m1.5.6.3.3.cmml"><mi id="S3.E11.m1.5.6.3.3.2.2" xref="S3.E11.m1.5.6.3.3.2.2.cmml">δ</mi><msub id="S3.E11.m1.5.6.3.3.2.3" xref="S3.E11.m1.5.6.3.3.2.3.cmml"><mi id="S3.E11.m1.5.6.3.3.2.3.2" xref="S3.E11.m1.5.6.3.3.2.3.2.cmml">O</mi><mi id="S3.E11.m1.5.6.3.3.2.3.3" xref="S3.E11.m1.5.6.3.3.2.3.3.cmml">d</mi></msub><mrow id="S3.E11.m1.5.5.2.4" xref="S3.E11.m1.5.5.2.3.cmml"><mo id="S3.E11.m1.5.5.2.4.1" stretchy="false" xref="S3.E11.m1.5.5.2.3.cmml">(</mo><mi id="S3.E11.m1.4.4.1.1" xref="S3.E11.m1.4.4.1.1.cmml">q</mi><mo id="S3.E11.m1.5.5.2.4.2" xref="S3.E11.m1.5.5.2.3.cmml">,</mo><mi id="S3.E11.m1.5.5.2.2" xref="S3.E11.m1.5.5.2.2.cmml">d</mi><mo id="S3.E11.m1.5.5.2.4.3" stretchy="false" xref="S3.E11.m1.5.5.2.3.cmml">)</mo></mrow></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E11.m1.5b"><apply id="S3.E11.m1.5.6.cmml" xref="S3.E11.m1.5.6"><eq id="S3.E11.m1.5.6.1.cmml" xref="S3.E11.m1.5.6.1"></eq><apply id="S3.E11.m1.5.6.2.cmml" xref="S3.E11.m1.5.6.2"><csymbol cd="ambiguous" id="S3.E11.m1.5.6.2.1.cmml" xref="S3.E11.m1.5.6.2">superscript</csymbol><apply id="S3.E11.m1.5.6.2.2.cmml" xref="S3.E11.m1.5.6.2"><csymbol cd="ambiguous" id="S3.E11.m1.5.6.2.2.1.cmml" xref="S3.E11.m1.5.6.2">subscript</csymbol><ci id="S3.E11.m1.5.6.2.2.2.cmml" xref="S3.E11.m1.5.6.2.2.2">𝛿</ci><apply id="S3.E11.m1.5.6.2.2.3.cmml" xref="S3.E11.m1.5.6.2.2.3"><csymbol cd="ambiguous" id="S3.E11.m1.5.6.2.2.3.1.cmml" xref="S3.E11.m1.5.6.2.2.3">subscript</csymbol><ci id="S3.E11.m1.5.6.2.2.3.2.cmml" xref="S3.E11.m1.5.6.2.2.3.2">𝑣</ci><ci id="S3.E11.m1.5.6.2.2.3.3.cmml" xref="S3.E11.m1.5.6.2.2.3.3">𝑑</ci></apply></apply><interval closure="open" id="S3.E11.m1.2.2.2.3.cmml" xref="S3.E11.m1.2.2.2.4"><ci id="S3.E11.m1.1.1.1.1.cmml" xref="S3.E11.m1.1.1.1.1">𝑞</ci><ci id="S3.E11.m1.2.2.2.2.cmml" xref="S3.E11.m1.2.2.2.2">𝑑</ci></interval></apply><apply id="S3.E11.m1.5.6.3.cmml" xref="S3.E11.m1.5.6.3"><times id="S3.E11.m1.5.6.3.1.cmml" xref="S3.E11.m1.5.6.3.1"></times><apply id="S3.E11.m1.5.6.3.2.cmml" xref="S3.E11.m1.5.6.3.2"><compose id="S3.E11.m1.5.6.3.2.1.cmml" xref="S3.E11.m1.5.6.3.2.1"></compose><apply id="S3.E11.m1.3.3.cmml" xref="S3.E11.m1.3.3"><divide id="S3.E11.m1.3.3.2.cmml" xref="S3.E11.m1.3.3"></divide><cn id="S3.E11.m1.3.3.3.cmml" type="integer" xref="S3.E11.m1.3.3.3">1</cn><apply id="S3.E11.m1.3.3.1.cmml" xref="S3.E11.m1.3.3.1"><csymbol cd="ambiguous" id="S3.E11.m1.3.3.1.2.cmml" xref="S3.E11.m1.3.3.1">superscript</csymbol><apply id="S3.E11.m1.3.3.1.1.1.1.cmml" xref="S3.E11.m1.3.3.1.1.1"><plus id="S3.E11.m1.3.3.1.1.1.1.2.cmml" xref="S3.E11.m1.3.3.1.1.1.1.2"></plus><cn id="S3.E11.m1.3.3.1.1.1.1.3.cmml" type="integer" xref="S3.E11.m1.3.3.1.1.1.1.3">1</cn><apply id="S3.E11.m1.3.3.1.1.1.1.1.2.cmml" xref="S3.E11.m1.3.3.1.1.1.1.1.1"><abs id="S3.E11.m1.3.3.1.1.1.1.1.2.1.cmml" xref="S3.E11.m1.3.3.1.1.1.1.1.1.2"></abs><apply id="S3.E11.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S3.E11.m1.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E11.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="S3.E11.m1.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E11.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S3.E11.m1.3.3.1.1.1.1.1.1.1.2">𝑉</ci><ci id="S3.E11.m1.3.3.1.1.1.1.1.1.1.3.cmml" xref="S3.E11.m1.3.3.1.1.1.1.1.1.1.3">𝑑</ci></apply></apply></apply><cn id="S3.E11.m1.3.3.1.3.cmml" type="integer" xref="S3.E11.m1.3.3.1.3">2</cn></apply></apply><apply id="S3.E11.m1.5.6.3.2.2.cmml" xref="S3.E11.m1.5.6.3.2.2"><csymbol cd="ambiguous" id="S3.E11.m1.5.6.3.2.2.1.cmml" xref="S3.E11.m1.5.6.3.2.2">superscript</csymbol><apply id="S3.E11.m1.5.6.3.2.2.2.cmml" xref="S3.E11.m1.5.6.3.2.2"><csymbol cd="ambiguous" id="S3.E11.m1.5.6.3.2.2.2.1.cmml" xref="S3.E11.m1.5.6.3.2.2">subscript</csymbol><ci id="S3.E11.m1.5.6.3.2.2.2.2.cmml" xref="S3.E11.m1.5.6.3.2.2.2.2">𝑊</ci><ci id="S3.E11.m1.5.6.3.2.2.2.3.cmml" xref="S3.E11.m1.5.6.3.2.2.2.3">𝑑</ci></apply><ci id="S3.E11.m1.5.6.3.2.2.3.cmml" xref="S3.E11.m1.5.6.3.2.2.3">𝑇</ci></apply></apply><apply id="S3.E11.m1.5.6.3.3.cmml" xref="S3.E11.m1.5.6.3.3"><csymbol cd="ambiguous" id="S3.E11.m1.5.6.3.3.1.cmml" xref="S3.E11.m1.5.6.3.3">superscript</csymbol><apply id="S3.E11.m1.5.6.3.3.2.cmml" xref="S3.E11.m1.5.6.3.3"><csymbol cd="ambiguous" id="S3.E11.m1.5.6.3.3.2.1.cmml" xref="S3.E11.m1.5.6.3.3">subscript</csymbol><ci id="S3.E11.m1.5.6.3.3.2.2.cmml" xref="S3.E11.m1.5.6.3.3.2.2">𝛿</ci><apply id="S3.E11.m1.5.6.3.3.2.3.cmml" xref="S3.E11.m1.5.6.3.3.2.3"><csymbol cd="ambiguous" id="S3.E11.m1.5.6.3.3.2.3.1.cmml" xref="S3.E11.m1.5.6.3.3.2.3">subscript</csymbol><ci id="S3.E11.m1.5.6.3.3.2.3.2.cmml" xref="S3.E11.m1.5.6.3.3.2.3.2">𝑂</ci><ci id="S3.E11.m1.5.6.3.3.2.3.3.cmml" xref="S3.E11.m1.5.6.3.3.2.3.3">𝑑</ci></apply></apply><interval closure="open" id="S3.E11.m1.5.5.2.3.cmml" xref="S3.E11.m1.5.5.2.4"><ci id="S3.E11.m1.4.4.1.1.cmml" xref="S3.E11.m1.4.4.1.1">𝑞</ci><ci id="S3.E11.m1.5.5.2.2.cmml" xref="S3.E11.m1.5.5.2.2">𝑑</ci></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E11.m1.5c">\delta_{v_{d}}^{(q,d)}=\frac{1}{(1+|V_{d}|)^{2}}\circ W_{d}^{T}\delta_{O_{d}}^% {(q,d)}</annotation><annotation encoding="application/x-llamapun" id="S3.E11.m1.5d">italic_δ start_POSTSUBSCRIPT italic_v start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT = divide start_ARG 1 end_ARG start_ARG ( 1 + | italic_V start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT | ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ∘ italic_W start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_δ start_POSTSUBSCRIPT italic_O start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S3.SSx2.p12"> <p class="ltx_p" id="S3.SSx2.p12.3">The operator <math alttext="\circ" class="ltx_Math" display="inline" id="S3.SSx2.p12.1.m1.1"><semantics id="S3.SSx2.p12.1.m1.1a"><mo id="S3.SSx2.p12.1.m1.1.1" xref="S3.SSx2.p12.1.m1.1.1.cmml">∘</mo><annotation-xml encoding="MathML-Content" id="S3.SSx2.p12.1.m1.1b"><compose id="S3.SSx2.p12.1.m1.1.1.cmml" xref="S3.SSx2.p12.1.m1.1.1"></compose></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p12.1.m1.1c">\circ</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p12.1.m1.1d">∘</annotation></semantics></math> is the element-wise multiplication in the above two formulas. Finally, we backpropagate the gradient of element-wise adding result <math alttext="\delta_{v_{q}}^{(q,d)}" class="ltx_Math" display="inline" id="S3.SSx2.p12.2.m2.2"><semantics id="S3.SSx2.p12.2.m2.2a"><msubsup id="S3.SSx2.p12.2.m2.2.3" xref="S3.SSx2.p12.2.m2.2.3.cmml"><mi id="S3.SSx2.p12.2.m2.2.3.2.2" xref="S3.SSx2.p12.2.m2.2.3.2.2.cmml">δ</mi><msub id="S3.SSx2.p12.2.m2.2.3.2.3" xref="S3.SSx2.p12.2.m2.2.3.2.3.cmml"><mi id="S3.SSx2.p12.2.m2.2.3.2.3.2" xref="S3.SSx2.p12.2.m2.2.3.2.3.2.cmml">v</mi><mi id="S3.SSx2.p12.2.m2.2.3.2.3.3" xref="S3.SSx2.p12.2.m2.2.3.2.3.3.cmml">q</mi></msub><mrow id="S3.SSx2.p12.2.m2.2.2.2.4" xref="S3.SSx2.p12.2.m2.2.2.2.3.cmml"><mo id="S3.SSx2.p12.2.m2.2.2.2.4.1" stretchy="false" xref="S3.SSx2.p12.2.m2.2.2.2.3.cmml">(</mo><mi id="S3.SSx2.p12.2.m2.1.1.1.1" xref="S3.SSx2.p12.2.m2.1.1.1.1.cmml">q</mi><mo id="S3.SSx2.p12.2.m2.2.2.2.4.2" xref="S3.SSx2.p12.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.SSx2.p12.2.m2.2.2.2.2" xref="S3.SSx2.p12.2.m2.2.2.2.2.cmml">d</mi><mo id="S3.SSx2.p12.2.m2.2.2.2.4.3" stretchy="false" xref="S3.SSx2.p12.2.m2.2.2.2.3.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SSx2.p12.2.m2.2b"><apply id="S3.SSx2.p12.2.m2.2.3.cmml" xref="S3.SSx2.p12.2.m2.2.3"><csymbol cd="ambiguous" id="S3.SSx2.p12.2.m2.2.3.1.cmml" xref="S3.SSx2.p12.2.m2.2.3">superscript</csymbol><apply id="S3.SSx2.p12.2.m2.2.3.2.cmml" xref="S3.SSx2.p12.2.m2.2.3"><csymbol cd="ambiguous" id="S3.SSx2.p12.2.m2.2.3.2.1.cmml" xref="S3.SSx2.p12.2.m2.2.3">subscript</csymbol><ci id="S3.SSx2.p12.2.m2.2.3.2.2.cmml" xref="S3.SSx2.p12.2.m2.2.3.2.2">𝛿</ci><apply id="S3.SSx2.p12.2.m2.2.3.2.3.cmml" xref="S3.SSx2.p12.2.m2.2.3.2.3"><csymbol cd="ambiguous" id="S3.SSx2.p12.2.m2.2.3.2.3.1.cmml" xref="S3.SSx2.p12.2.m2.2.3.2.3">subscript</csymbol><ci id="S3.SSx2.p12.2.m2.2.3.2.3.2.cmml" xref="S3.SSx2.p12.2.m2.2.3.2.3.2">𝑣</ci><ci id="S3.SSx2.p12.2.m2.2.3.2.3.3.cmml" xref="S3.SSx2.p12.2.m2.2.3.2.3.3">𝑞</ci></apply></apply><interval closure="open" id="S3.SSx2.p12.2.m2.2.2.2.3.cmml" xref="S3.SSx2.p12.2.m2.2.2.2.4"><ci id="S3.SSx2.p12.2.m2.1.1.1.1.cmml" xref="S3.SSx2.p12.2.m2.1.1.1.1">𝑞</ci><ci id="S3.SSx2.p12.2.m2.2.2.2.2.cmml" xref="S3.SSx2.p12.2.m2.2.2.2.2">𝑑</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p12.2.m2.2c">\delta_{v_{q}}^{(q,d)}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p12.2.m2.2d">italic_δ start_POSTSUBSCRIPT italic_v start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT</annotation></semantics></math> and <math alttext="\delta_{v_{d}}^{(q,d)}" class="ltx_Math" display="inline" id="S3.SSx2.p12.3.m3.2"><semantics id="S3.SSx2.p12.3.m3.2a"><msubsup id="S3.SSx2.p12.3.m3.2.3" xref="S3.SSx2.p12.3.m3.2.3.cmml"><mi id="S3.SSx2.p12.3.m3.2.3.2.2" xref="S3.SSx2.p12.3.m3.2.3.2.2.cmml">δ</mi><msub id="S3.SSx2.p12.3.m3.2.3.2.3" xref="S3.SSx2.p12.3.m3.2.3.2.3.cmml"><mi id="S3.SSx2.p12.3.m3.2.3.2.3.2" xref="S3.SSx2.p12.3.m3.2.3.2.3.2.cmml">v</mi><mi id="S3.SSx2.p12.3.m3.2.3.2.3.3" xref="S3.SSx2.p12.3.m3.2.3.2.3.3.cmml">d</mi></msub><mrow id="S3.SSx2.p12.3.m3.2.2.2.4" xref="S3.SSx2.p12.3.m3.2.2.2.3.cmml"><mo id="S3.SSx2.p12.3.m3.2.2.2.4.1" stretchy="false" xref="S3.SSx2.p12.3.m3.2.2.2.3.cmml">(</mo><mi id="S3.SSx2.p12.3.m3.1.1.1.1" xref="S3.SSx2.p12.3.m3.1.1.1.1.cmml">q</mi><mo id="S3.SSx2.p12.3.m3.2.2.2.4.2" xref="S3.SSx2.p12.3.m3.2.2.2.3.cmml">,</mo><mi id="S3.SSx2.p12.3.m3.2.2.2.2" xref="S3.SSx2.p12.3.m3.2.2.2.2.cmml">d</mi><mo id="S3.SSx2.p12.3.m3.2.2.2.4.3" stretchy="false" xref="S3.SSx2.p12.3.m3.2.2.2.3.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SSx2.p12.3.m3.2b"><apply id="S3.SSx2.p12.3.m3.2.3.cmml" xref="S3.SSx2.p12.3.m3.2.3"><csymbol cd="ambiguous" id="S3.SSx2.p12.3.m3.2.3.1.cmml" xref="S3.SSx2.p12.3.m3.2.3">superscript</csymbol><apply id="S3.SSx2.p12.3.m3.2.3.2.cmml" xref="S3.SSx2.p12.3.m3.2.3"><csymbol cd="ambiguous" id="S3.SSx2.p12.3.m3.2.3.2.1.cmml" xref="S3.SSx2.p12.3.m3.2.3">subscript</csymbol><ci id="S3.SSx2.p12.3.m3.2.3.2.2.cmml" xref="S3.SSx2.p12.3.m3.2.3.2.2">𝛿</ci><apply id="S3.SSx2.p12.3.m3.2.3.2.3.cmml" xref="S3.SSx2.p12.3.m3.2.3.2.3"><csymbol cd="ambiguous" id="S3.SSx2.p12.3.m3.2.3.2.3.1.cmml" xref="S3.SSx2.p12.3.m3.2.3.2.3">subscript</csymbol><ci id="S3.SSx2.p12.3.m3.2.3.2.3.2.cmml" xref="S3.SSx2.p12.3.m3.2.3.2.3.2">𝑣</ci><ci id="S3.SSx2.p12.3.m3.2.3.2.3.3.cmml" xref="S3.SSx2.p12.3.m3.2.3.2.3.3">𝑑</ci></apply></apply><interval closure="open" id="S3.SSx2.p12.3.m3.2.2.2.3.cmml" xref="S3.SSx2.p12.3.m3.2.2.2.4"><ci id="S3.SSx2.p12.3.m3.1.1.1.1.cmml" xref="S3.SSx2.p12.3.m3.1.1.1.1">𝑞</ci><ci id="S3.SSx2.p12.3.m3.2.2.2.2.cmml" xref="S3.SSx2.p12.3.m3.2.2.2.2">𝑑</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SSx2.p12.3.m3.2c">\delta_{v_{d}}^{(q,d)}</annotation><annotation encoding="application/x-llamapun" id="S3.SSx2.p12.3.m3.2d">italic_δ start_POSTSUBSCRIPT italic_v start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_q , italic_d ) end_POSTSUPERSCRIPT</annotation></semantics></math> to each word embedding of query and documents.</p> </div> </section> </section> <section class="ltx_section" id="S4"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">4. </span>Experimental Setup</h2> <div class="ltx_para" id="S4.p1"> <p class="ltx_p" id="S4.p1.1">The Web search results of a query can be categorized into three groups based on the user’s click signals:</p> </div> <div class="ltx_para" id="S4.p2"> <ol class="ltx_enumerate" id="S4.I1"> <li class="ltx_item" id="S4.I1.i1" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(1)</span> <div class="ltx_para" id="S4.I1.i1.p1"> <p class="ltx_p" id="S4.I1.i1.p1.1"><span class="ltx_text ltx_font_italic" id="S4.I1.i1.p1.1.1">Clicked</span>: The title of the result that is clicked by the users.</p> </div> </li> <li class="ltx_item" id="S4.I1.i2" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(2)</span> <div class="ltx_para" id="S4.I1.i2.p1"> <p class="ltx_p" id="S4.I1.i2.p1.1"><span class="ltx_text ltx_font_italic" id="S4.I1.i2.p1.1.1">Skipped</span>: The title of the result that is ranked above a clicked one, i.e., the result is examined by the user but not clicked.</p> </div> </li> <li class="ltx_item" id="S4.I1.i3" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(3)</span> <div class="ltx_para" id="S4.I1.i3.p1"> <p class="ltx_p" id="S4.I1.i3.p1.1"><span class="ltx_text ltx_font_italic" id="S4.I1.i3.p1.1.1">Non-Examined</span>: The title of the result that is ranked below all clicked ones.</p> </div> </li> </ol> </div> <div class="ltx_para" id="S4.p3"> <p class="ltx_p" id="S4.p3.1">In our experiments, we measure the performance by two metrics. The first metric is whether the model can effectively predict future clicking by scoring clicked results higher than non-clicked results. To achieve this goal, we prepare a testing dataset named Test-1 by deriving 23,000,000 pairwise judgments from the holdout query log. For a specific query and its top ten results, the pairwise judgment in Test-1 contains a randomly chosen clicked result and a randomly chosen non-clicked result. The second metric aims to test whether the model can generate results aligned with human judgment. The second testing dataset, named Test-2, is manually prepared by human experts and contains 530,000 pairwise judgments.</p> </div> </section> <section class="ltx_section" id="S5"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">5. </span>Atomic Strategies</h2> <div class="ltx_para" id="S5.p1"> <p class="ltx_p" id="S5.p1.1">In LTR, it is well believed that the relative preferences of clicked documents over skipped ones are reasonably accurate <cite class="ltx_cite ltx_citemacro_citep">(Agichtein et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib2" title="">2006</a>; Joachims et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib18" title="">2005</a>)</cite>. Therefore, based on a user’s ranking preference reflected in his/her clicks, we propose several strategies to derive pairwise judgments. These strategies are mutually exclusive and can be used as the basic building blocks for deriving more complicated pairwise judgments.</p> </div> <div class="ltx_para" id="S5.p2"> <ol class="ltx_enumerate" id="S5.I1"> <li class="ltx_item" id="S5.I1.i1" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(1)</span> <div class="ltx_para" id="S5.I1.i1.p1"> <p class="ltx_p" id="S5.I1.i1.p1.1"><span class="ltx_text ltx_font_italic" id="S5.I1.i1.p1.1.1">Clicked</span> ¿ <span class="ltx_text ltx_font_italic" id="S5.I1.i1.p1.1.2">Skipped</span>: This strategy is widely utilized in LTR applications. It assumes that the clicked results should be preferred to the skipped results.</p> </div> </li> <li class="ltx_item" id="S5.I1.i2" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(2)</span> <div class="ltx_para" id="S5.I1.i2.p1"> <p class="ltx_p" id="S5.I1.i2.p1.1"><span class="ltx_text ltx_font_italic" id="S5.I1.i2.p1.1.1">Clicked</span> ¿ <span class="ltx_text ltx_font_italic" id="S5.I1.i2.p1.1.2">Clicked</span>: This strategy differentiates clicked results by their click-through rate (CTR) and assumes that a result with a higher CTR is preferred to that with a lower CTR.</p> </div> </li> <li class="ltx_item" id="S5.I1.i3" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(3)</span> <div class="ltx_para" id="S5.I1.i3.p1"> <p class="ltx_p" id="S5.I1.i3.p1.1"><span class="ltx_text ltx_font_italic" id="S5.I1.i3.p1.1.1">Clicked</span> ¿ <span class="ltx_text ltx_font_italic" id="S5.I1.i3.p1.1.2">Non-Examined</span>: This strategy assumes that the clicked results are preferred to those that are not examined.</p> </div> </li> <li class="ltx_item" id="S5.I1.i4" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(4)</span> <div class="ltx_para" id="S5.I1.i4.p1"> <p class="ltx_p" id="S5.I1.i4.p1.1"><span class="ltx_text ltx_font_italic" id="S5.I1.i4.p1.1.1">Skipped</span> ¿ <span class="ltx_text ltx_font_italic" id="S5.I1.i4.p1.1.2">Non-Examined</span>: This strategy is rarely applied in LTR since no click information is utilized. We include this strategy in our experiment for completeness.</p> </div> </li> </ol> </div> <div class="ltx_para" id="S5.p3"> <p class="ltx_p" id="S5.p3.1">Empirically, SEM usually takes several iterations to converge, and we find that 50 iterations are typically enough to obtain a stable model. The experimental result on Test-1 is shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3.F2.sf1" title="In Figure 2 ‣ Optimization ‣ 3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2(a)</span></a>. We observe that <span class="ltx_text ltx_font_italic" id="S5.p3.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p3.1.2">Non-Examined</span> achieves the highest precision, indicating that the pairwise judgments formed by this strategy are of the best quality for training SEM. Pairwise judgments formed by <span class="ltx_text ltx_font_italic" id="S5.p3.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p3.1.4">Skipped</span> and <span class="ltx_text ltx_font_italic" id="S5.p3.1.5">Skipped</span>¿<span class="ltx_text ltx_font_italic" id="S5.p3.1.6">Non-Examined</span> are of lower quality, and those formed by <span class="ltx_text ltx_font_italic" id="S5.p3.1.7">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p3.1.8">Clicked</span> are of the lowest quality.</p> </div> <div class="ltx_para" id="S5.p4"> <p class="ltx_p" id="S5.p4.1">We find that the result is quite different from the experience accumulated from conventional LTR. In LTR, the best pairwise judgment is typically formulated between skipped results and clicked results, i.e., the strategy <span class="ltx_text ltx_font_italic" id="S5.p4.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p4.1.2">Skipped</span>. However, for training SEM, the best strategy is <span class="ltx_text ltx_font_italic" id="S5.p4.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p4.1.4">Non-Examined</span> and the performance discrepancy between <span class="ltx_text ltx_font_italic" id="S5.p4.1.5">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p4.1.6">Skipped</span> and <span class="ltx_text ltx_font_italic" id="S5.p4.1.7">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p4.1.8">Non-Examined</span> is large. We further conduct an evaluation on Test-2, and the result is shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3.F2.sf2" title="In Figure 2 ‣ Optimization ‣ 3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2(b)</span></a>. We can see that the precision of the strategies on Test-2 is lower than those of Test-1, attributed to the increasing difficulty of the testing data in Test-2. However, the performance trends of different strategies and their relative positions shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3.F2.sf2" title="In Figure 2 ‣ Optimization ‣ 3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2(b)</span></a> are closely consistent with those in Figure <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3.F2.sf1" title="In Figure 2 ‣ Optimization ‣ 3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2(a)</span></a>.</p> </div> <div class="ltx_para" id="S5.p5"> <p class="ltx_p" id="S5.p5.1">From Figures <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3.F2.sf1" title="In Figure 2 ‣ Optimization ‣ 3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2(a)</span></a> and <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3.F2.sf2" title="In Figure 2 ‣ Optimization ‣ 3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2(b)</span></a>, <span class="ltx_text ltx_font_italic" id="S5.p5.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p5.1.2">Clicked</span> is identified as the only strategy leads to a decrease in precision as more iterations are conducted, indicating that the pairwise judgments derived by this strategies are nearly redundant. Another important observation is the fluctuation of the precision: <span class="ltx_text ltx_font_italic" id="S5.p5.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p5.1.4">Non-Examined</span> has the least fluctuation, that of <span class="ltx_text ltx_font_italic" id="S5.p5.1.5">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S5.p5.1.6">Skipped</span> is larger and that of <span class="ltx_text ltx_font_italic" id="S5.p5.1.7">Skipped</span>¿<span class="ltx_text ltx_font_italic" id="S5.p5.1.8">Non-Examined</span> is the largest. The scale of fluctuation reveals the extent of the training distorted by occasionally encountering bad training instances.</p> </div> </section> <section class="ltx_section" id="S6"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">6. </span>Hybrid Strategy</h2> <div class="ltx_para" id="S6.p1"> <p class="ltx_p" id="S6.p1.1">In the previous section, the atomic strategies are investigated, and the <span class="ltx_text ltx_font_italic" id="S6.p1.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p1.1.2">Non-Examined</span> is identified to provide the best performance in both tasks. A large performance discrepancy between any two atomic strategies is observed. Intuitively, combining these atomic strategies will not bring better results than <span class="ltx_text ltx_font_italic" id="S6.p1.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p1.1.4">Non-Examined</span> since the low-quality training instances will “contaminate” the result. However, through extensive empirical evaluation, we find that the intuition holds but with one exception, which results in the following hybrid strategy: <span class="ltx_text ltx_font_bold ltx_font_italic" id="S6.p1.1.5">Clicked<span class="ltx_text ltx_font_upright" id="S6.p1.1.5.1">¿</span>Non-Clicked</span>, a combination of <span class="ltx_text ltx_font_italic" id="S6.p1.1.6">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p1.1.7">Skipped</span> and <span class="ltx_text ltx_font_italic" id="S6.p1.1.8">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p1.1.9">Non-Examined</span>.</p> </div> <figure class="ltx_table" id="S6.T1"> <figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span class="ltx_text" id="S6.T1.6.1.1" style="font-size:90%;">Table 1</span>. </span><span class="ltx_text" id="S6.T1.7.2" style="font-size:90%;">Distribution of Pairwise Judgment Dataset</span></figcaption> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="S6.T1.4"> <thead class="ltx_thead"> <tr class="ltx_tr" id="S6.T1.4.5.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" id="S6.T1.4.5.1.1"><span class="ltx_text ltx_font_bold" id="S6.T1.4.5.1.1.1">Strategy</span></th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="S6.T1.4.5.1.2"><span class="ltx_text ltx_font_bold" id="S6.T1.4.5.1.2.1">Percentage</span></th> </tr> </thead> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="S6.T1.1.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" id="S6.T1.1.1.1">Clicked <math alttext=">" class="ltx_Math" display="inline" id="S6.T1.1.1.1.m1.1"><semantics id="S6.T1.1.1.1.m1.1a"><mo id="S6.T1.1.1.1.m1.1.1" xref="S6.T1.1.1.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S6.T1.1.1.1.m1.1b"><gt id="S6.T1.1.1.1.m1.1.1.cmml" xref="S6.T1.1.1.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.1.1.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S6.T1.1.1.1.m1.1d">></annotation></semantics></math> Clicked</th> <td class="ltx_td ltx_align_center ltx_border_t" id="S6.T1.1.1.2">5.96%</td> </tr> <tr class="ltx_tr" id="S6.T1.2.2"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row" id="S6.T1.2.2.1">Clicked <math alttext=">" class="ltx_Math" display="inline" id="S6.T1.2.2.1.m1.1"><semantics id="S6.T1.2.2.1.m1.1a"><mo id="S6.T1.2.2.1.m1.1.1" xref="S6.T1.2.2.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S6.T1.2.2.1.m1.1b"><gt id="S6.T1.2.2.1.m1.1.1.cmml" xref="S6.T1.2.2.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.2.2.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S6.T1.2.2.1.m1.1d">></annotation></semantics></math> Skipped</th> <td class="ltx_td ltx_align_center" id="S6.T1.2.2.2">22.25%</td> </tr> <tr class="ltx_tr" id="S6.T1.3.3"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row" id="S6.T1.3.3.1">Skipped <math alttext=">" class="ltx_Math" display="inline" id="S6.T1.3.3.1.m1.1"><semantics id="S6.T1.3.3.1.m1.1a"><mo id="S6.T1.3.3.1.m1.1.1" xref="S6.T1.3.3.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S6.T1.3.3.1.m1.1b"><gt id="S6.T1.3.3.1.m1.1.1.cmml" xref="S6.T1.3.3.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.3.3.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S6.T1.3.3.1.m1.1d">></annotation></semantics></math> Non-Examined</th> <td class="ltx_td ltx_align_center" id="S6.T1.3.3.2">32.92%</td> </tr> <tr class="ltx_tr" id="S6.T1.4.4"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" id="S6.T1.4.4.1">Clicked <math alttext=">" class="ltx_Math" display="inline" id="S6.T1.4.4.1.m1.1"><semantics id="S6.T1.4.4.1.m1.1a"><mo id="S6.T1.4.4.1.m1.1.1" xref="S6.T1.4.4.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S6.T1.4.4.1.m1.1b"><gt id="S6.T1.4.4.1.m1.1.1.cmml" xref="S6.T1.4.4.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.4.4.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S6.T1.4.4.1.m1.1d">></annotation></semantics></math> Non-Examined</th> <td class="ltx_td ltx_align_center ltx_border_bb" id="S6.T1.4.4.2">38.87%</td> </tr> </tbody> </table> </figure> <div class="ltx_para" id="S6.p2"> <p class="ltx_p" id="S6.p2.1">Based on the experimental results shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3.F2.sf3" title="In Figure 2 ‣ Optimization ‣ 3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2(c)</span></a>, we can see that <span class="ltx_text ltx_font_italic" id="S6.p2.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p2.1.2">Non-Clicked</span> slightly outperforms the best atomic strategy <span class="ltx_text ltx_font_italic" id="S6.p2.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p2.1.4">Non-Examined</span> on Test-1. In order to reveal the underlying reason, we present the distribution of the four atomic strategies in Table <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S6.T1" title="Table 1 ‣ 6. Hybrid Strategy ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">1</span></a> and the statistics suggest that no single strategy covers a majority of the pairs. <span class="ltx_text ltx_font_italic" id="S6.p2.1.5">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p2.1.6">Skipped</span> and <span class="ltx_text ltx_font_italic" id="S6.p2.1.7">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p2.1.8">Non-Examined</span> are responsible for 22.25% and 38.87% of the potential training data. Hence, relying on a single strategy such as <span class="ltx_text ltx_font_italic" id="S6.p2.1.9">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p2.1.10">Non-Examined</span> rules out many “fairly good” training instances and misses the chance of updating the embedding of many words. In the long run, the model using <span class="ltx_text ltx_font_italic" id="S6.p2.1.11">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p2.1.12">Non-Clicked</span> strategy sees more training instances, thus achieving slightly better performance than the atomic strategies. Another possible explanation is that the testing data in Test-1 are essentially derived by <span class="ltx_text ltx_font_italic" id="S6.p2.1.13">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p2.1.14">Non-Clicked</span> strategy, hence resulting in a better chance of getting a model with a good fit for the testing data.</p> </div> <div class="ltx_para" id="S6.p3"> <p class="ltx_p" id="S6.p3.1">In order to see whether the performance of <span class="ltx_text ltx_font_italic" id="S6.p3.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p3.1.2">Non-Clicked</span> is exaggerated by Test-1, we also present the experimental result of Test-2 in Figure <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#S3.F2.sf3" title="In Figure 2 ‣ Optimization ‣ 3. Semantic Embedding Model for Web Search ‣ Pairwise Judgment Formulation for Semantic Embedding Model in Web Search"><span class="ltx_text ltx_ref_tag">2(c)</span></a> and show that the results of Test-2 are different from those of Test-1. In the beginning, <span class="ltx_text ltx_font_italic" id="S6.p3.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p3.1.4">Non-Examined</span> performs better than <span class="ltx_text ltx_font_italic" id="S6.p3.1.5">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.p3.1.6">Non-Clicked</span>, indicating that high-quality pairwise judgments give a better start for training SEM. In the long run, the performance discrepancy of the two strategies diminishes and results in similar performance when convergence is achieved.</p> </div> <div class="ltx_para" id="S6.p4"> <p class="ltx_p" id="S6.p4.1">Based on the observations from experimental results, the insights and best practices for pairwise judgment formulation are summarized as follows:</p> <ol class="ltx_enumerate" id="S6.I1"> <li class="ltx_item" id="S6.I1.i1" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(1)</span> <div class="ltx_para" id="S6.I1.i1.p1"> <p class="ltx_p" id="S6.I1.i1.p1.1">The conventional LTR strategy for pairwise judgment formulation is not suitable for training SEM. LTR aims to learn the weights of features in a ranking function. In contrast, the goal of SEM is to learn good representations of the features themselves.</p> </div> </li> <li class="ltx_item" id="S6.I1.i2" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(2)</span> <div class="ltx_para" id="S6.I1.i2.p1"> <p class="ltx_p" id="S6.I1.i2.p1.1">When <span class="ltx_text ltx_font_italic" id="S6.I1.i2.p1.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.I1.i2.p1.1.2">Non-Examined</span> is applied, adding pairwise judgments derived from <span class="ltx_text ltx_font_italic" id="S6.I1.i2.p1.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.I1.i2.p1.1.4">Skipped</span> to form a hybrid heuristic may slightly improve its performance by extending the diversity of training instances.</p> </div> </li> <li class="ltx_item" id="S6.I1.i3" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">(3)</span> <div class="ltx_para" id="S6.I1.i3.p1"> <p class="ltx_p" id="S6.I1.i3.p1.1">The strategy <span class="ltx_text ltx_font_italic" id="S6.I1.i3.p1.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.I1.i3.p1.1.2">Non-Examined</span>, which is rarely used in LTR, derives the best training data for SEM. Solely relying on this strategy leads to a significantly smaller amount of training data (i.e., 38.87%) while maintaining the performance that is nearly as good as that achieved using the hybrid strategy <span class="ltx_text ltx_font_italic" id="S6.I1.i3.p1.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S6.I1.i3.p1.1.4">Non-Clicked</span>, which derives much more data (i.e., 61.10% as a combination of the two atomic strategies) for the training process.</p> </div> </li> </ol> </div> </section> <section class="ltx_section" id="S7"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">7. </span>Conclusions</h2> <div class="ltx_para" id="S7.p1"> <p class="ltx_p" id="S7.p1.1">In this paper, we investigate the formulation of pairwise judgments for the Semantic Embedding Model (SEM) through an in-depth study of data from the query log of a major search engine. By conducting large-scale experiments, we quantitatively compare various strategies for pairwise judgment formulation and demonstrate the effectiveness of our proposed approaches. Notably, our findings reveal significant differences between the effective strategies for neural network-based SEM and traditional pairwise learning-to-rank (LTR) methods. Specifically, the strategy <span class="ltx_text ltx_font_italic" id="S7.p1.1.1">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S7.p1.1.2">Non-Examined</span>, although rarely utilized in LTR, proves to yield the highest quality training data for SEM. The hybrid strategy <span class="ltx_text ltx_font_italic" id="S7.p1.1.3">Clicked</span>¿<span class="ltx_text ltx_font_italic" id="S7.p1.1.4">Non-Clicked</span>, which encompasses a broader range of pairwise judgments, demonstrates slightly superior performance by providing a more extensive dataset for the model to learn from. Future work includes introducing more signals into the procedure of pairwise judgment formulation and investigating formulation strategies for SEM variants. The formulated pairwise judgment can also inspire downstream applications including topic discovery <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib9" title="">2014</a>; Jiang and Ng, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib12" title="">2013</a>; Jiang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib15" title="">2015b</a>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib17" title="">2023</a>)</cite>, intent mining <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib10" title="">2016a</a>; Jiang and Yang, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib16" title="">2016</a>; Hong et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib6" title="">2024</a>)</cite>, and user personalization <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib11" title="">2015a</a>; Zhang et al<span class="ltx_text">.</span>, <a class="ltx_ref" href="https://arxiv.org/html/2408.04197v2#bib.bib28" title="">2023</a>)</cite>.</p> </div> </section> <section class="ltx_bibliography" id="bib"> <h2 class="ltx_title ltx_title_bibliography">References</h2> <ul class="ltx_biblist"> <li class="ltx_bibitem" id="bib.bib1"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(1)</span> <span class="ltx_bibblock"> </span> </li> <li class="ltx_bibitem" id="bib.bib2"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Agichtein et al<span class="ltx_text" id="bib.bib2.2.2.1">.</span> (2006)</span> <span class="ltx_bibblock"> Eugene Agichtein, Eric Brill, Susan Dumais, and Robert Ragno. 2006. </span> <span class="ltx_bibblock">Learning user interaction models for predicting web search result preferences. In <em class="ltx_emph ltx_font_italic" id="bib.bib2.3.1">Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</em> (Seattle, Washington, USA) <em class="ltx_emph ltx_font_italic" id="bib.bib2.4.2">(SIGIR ’06)</em>. Association for Computing Machinery, New York, NY, USA, 3–10. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/1148170.1148175" title="">https://doi.org/10.1145/1148170.1148175</a> </span> </li> <li class="ltx_bibitem" id="bib.bib3"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chapelle and Zhang (2009)</span> <span class="ltx_bibblock"> Olivier Chapelle and Ya Zhang. 2009. </span> <span class="ltx_bibblock">A dynamic bayesian network click model for web search ranking. In <em class="ltx_emph ltx_font_italic" id="bib.bib3.1.1">Proceedings of the 18th International Conference on World Wide Web</em> (Madrid, Spain) <em class="ltx_emph ltx_font_italic" id="bib.bib3.2.2">(WWW ’09)</em>. Association for Computing Machinery, New York, NY, USA, 1–10. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/1526709.1526711" title="">https://doi.org/10.1145/1526709.1526711</a> </span> </li> <li class="ltx_bibitem" id="bib.bib4"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al<span class="ltx_text" id="bib.bib4.2.2.1">.</span> (2012)</span> <span class="ltx_bibblock"> Weizhu Chen, Dong Wang, Yuchen Zhang, Zheng Chen, Adish Singla, and Qiang Yang. 2012. </span> <span class="ltx_bibblock">A noise-aware click model for web search. In <em class="ltx_emph ltx_font_italic" id="bib.bib4.3.1">Proceedings of the fifth ACM international conference on Web search and data mining</em> (Seattle, Washington, USA) <em class="ltx_emph ltx_font_italic" id="bib.bib4.4.2">(WSDM ’12)</em>. Association for Computing Machinery, New York, NY, USA, 313–322. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2124295.2124335" title="">https://doi.org/10.1145/2124295.2124335</a> </span> </li> <li class="ltx_bibitem" id="bib.bib5"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Collobert et al<span class="ltx_text" id="bib.bib5.2.2.1">.</span> (2011)</span> <span class="ltx_bibblock"> Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. </span> <span class="ltx_bibblock">Natural Language Processing (Almost) from Scratch. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib5.3.1">J. Mach. Learn. Res.</em> 12, null (11 2011), 2493–2537. </span> <span class="ltx_bibblock"> </span> </li> <li class="ltx_bibitem" id="bib.bib6"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hong et al<span class="ltx_text" id="bib.bib6.2.2.1">.</span> (2024)</span> <span class="ltx_bibblock"> Mengze Hong, Di Jiang, Yuanfeng Song, and Chen Jason Zhang. 2024. </span> <span class="ltx_bibblock">Neural-Bayesian Program Learning for Few-shot Dialogue Intent Parsing. </span> <span class="ltx_bibblock"> </span> <span class="ltx_bibblock">arXiv:2410.06190 [cs.CL] <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://arxiv.org/abs/2410.06190" title="">https://arxiv.org/abs/2410.06190</a> </span> </li> <li class="ltx_bibitem" id="bib.bib7"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al<span class="ltx_text" id="bib.bib7.2.2.1">.</span> (2013)</span> <span class="ltx_bibblock"> Po-Sen Huang, Xiaodong He, Jianfeng Gao, li Deng, Alex Acero, and Larry Heck. 2013. </span> <span class="ltx_bibblock">Learning deep structured semantic models for web search using clickthrough data. In <em class="ltx_emph ltx_font_italic" id="bib.bib7.3.1">Proceedings of the 22nd ACM international conference on Conference on information & knowledge management</em>. 2333–2338. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2505515.2505665" title="">https://doi.org/10.1145/2505515.2505665</a> </span> </li> <li class="ltx_bibitem" id="bib.bib8"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al<span class="ltx_text" id="bib.bib8.2.2.1">.</span> (2011)</span> <span class="ltx_bibblock"> Di Jiang, Kenneth Wai-Ting Leung, and Wilfred Ng. 2011. </span> <span class="ltx_bibblock">Context-aware search personalization with concept preference. In <em class="ltx_emph ltx_font_italic" id="bib.bib8.3.1">Proceedings of the 20th ACM International Conference on Information and Knowledge Management</em> (Glasgow, Scotland, UK) <em class="ltx_emph ltx_font_italic" id="bib.bib8.4.2">(CIKM ’11)</em>. Association for Computing Machinery, New York, NY, USA, 563–572. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2063576.2063661" title="">https://doi.org/10.1145/2063576.2063661</a> </span> </li> <li class="ltx_bibitem" id="bib.bib9"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al<span class="ltx_text" id="bib.bib9.2.2.1">.</span> (2014)</span> <span class="ltx_bibblock"> Di Jiang, Kenneth Wai-Ting Leung, and Wilfred Ng. 2014. </span> <span class="ltx_bibblock">Fast topic discovery from web search streams. In <em class="ltx_emph ltx_font_italic" id="bib.bib9.3.1">Proceedings of the 23rd International Conference on World Wide Web</em> (Seoul, Korea) <em class="ltx_emph ltx_font_italic" id="bib.bib9.4.2">(WWW ’14)</em>. Association for Computing Machinery, New York, NY, USA, 949–960. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2566486.2567965" title="">https://doi.org/10.1145/2566486.2567965</a> </span> </li> <li class="ltx_bibitem" id="bib.bib10"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al<span class="ltx_text" id="bib.bib10.2.2.1">.</span> (2016a)</span> <span class="ltx_bibblock"> Di Jiang, Kenneth Wai-Ting Leung, and Wilfred Ng. 2016a. </span> <span class="ltx_bibblock">Query intent mining with multiple dimensions of web search data. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib10.3.1">World Wide Web</em> 19, 3 (5 2016), 475–497. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1007/s11280-015-0336-2" title="">https://doi.org/10.1007/s11280-015-0336-2</a> </span> </li> <li class="ltx_bibitem" id="bib.bib11"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al<span class="ltx_text" id="bib.bib11.2.2.1">.</span> (2015a)</span> <span class="ltx_bibblock"> Di Jiang, Kenneth Wai-Ting Leung, Lingxiao Yang, and Wilfred Ng. 2015a. </span> <span class="ltx_bibblock">Query suggestion with diversification and personalization. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib11.3.1">Knowledge-Based Systems</em> 89 (2015), 553–568. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1016/j.knosys.2015.09.003" title="">https://doi.org/10.1016/j.knosys.2015.09.003</a> </span> </li> <li class="ltx_bibitem" id="bib.bib12"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang and Ng (2013)</span> <span class="ltx_bibblock"> Di Jiang and Wilfred Ng. 2013. </span> <span class="ltx_bibblock">Mining web search topics with diverse spatiotemporal patterns. In <em class="ltx_emph ltx_font_italic" id="bib.bib12.1.1">Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval</em> (Dublin, Ireland) <em class="ltx_emph ltx_font_italic" id="bib.bib12.2.2">(SIGIR ’13)</em>. Association for Computing Machinery, New York, NY, USA, 881–884. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2484028.2484124" title="">https://doi.org/10.1145/2484028.2484124</a> </span> </li> <li class="ltx_bibitem" id="bib.bib13"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al<span class="ltx_text" id="bib.bib13.2.2.1">.</span> (2016b)</span> <span class="ltx_bibblock"> Di Jiang, Lei Shi, Rongzhong Lian, and Hua Wu. 2016b. </span> <span class="ltx_bibblock">Latent Topic Embedding. In <em class="ltx_emph ltx_font_italic" id="bib.bib13.3.1">Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers</em>, Yuji Matsumoto and Rashmi Prasad (Eds.). The COLING 2016 Organizing Committee, Osaka, Japan, 2689–2698. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://aclanthology.org/C16-1253" title="">https://aclanthology.org/C16-1253</a> </span> </li> <li class="ltx_bibitem" id="bib.bib14"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al<span class="ltx_text" id="bib.bib14.2.2.1">.</span> (2016c)</span> <span class="ltx_bibblock"> Di Jiang, Yongxin Tong, and Yuanfeng Song. 2016c. </span> <span class="ltx_bibblock">Cross-Lingual Topic Discovery From Multilingual Search Engine Query Log. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib14.3.1">ACM Trans. Inf. Syst.</em> 35, 2, Article 9 (9 2016), 28 pages. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2956235" title="">https://doi.org/10.1145/2956235</a> </span> </li> <li class="ltx_bibitem" id="bib.bib15"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al<span class="ltx_text" id="bib.bib15.2.2.1">.</span> (2015b)</span> <span class="ltx_bibblock"> Di Jiang, Jan Vosecky, Kenneth Wai-Ting Leung, Lingxiao Yang, and Wilfred Ng. 2015b. </span> <span class="ltx_bibblock">SG-WSTD: A framework for scalable geographic web search topic discovery. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib15.3.1">Knowledge-Based Systems</em> 84 (2015), 18–33. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1016/j.knosys.2015.03.020" title="">https://doi.org/10.1016/j.knosys.2015.03.020</a> </span> </li> <li class="ltx_bibitem" id="bib.bib16"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang and Yang (2016)</span> <span class="ltx_bibblock"> Di Jiang and Lingxiao Yang. 2016. </span> <span class="ltx_bibblock">Query intent inference via search engine log. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib16.1.1">Knowl. Inf. Syst.</em> 49, 2 (11 2016), 661–685. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1007/s10115-015-0915-7" title="">https://doi.org/10.1007/s10115-015-0915-7</a> </span> </li> <li class="ltx_bibitem" id="bib.bib17"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al<span class="ltx_text" id="bib.bib17.2.2.1">.</span> (2023)</span> <span class="ltx_bibblock"> Di Jiang, Chen Zhang, and Yuanfeng Song. 2023. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib17.3.1">Probabilistic Topic Models - Foundation and Application</em>. </span> <span class="ltx_bibblock">Springer. 1–149 pages. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1007/978-981-99-2431-8" title="">https://doi.org/10.1007/978-981-99-2431-8</a> </span> </li> <li class="ltx_bibitem" id="bib.bib18"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Joachims et al<span class="ltx_text" id="bib.bib18.2.2.1">.</span> (2005)</span> <span class="ltx_bibblock"> Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. </span> <span class="ltx_bibblock">Accurately interpreting clickthrough data as implicit feedback. In <em class="ltx_emph ltx_font_italic" id="bib.bib18.3.1">Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</em> (Salvador, Brazil) <em class="ltx_emph ltx_font_italic" id="bib.bib18.4.2">(SIGIR ’05)</em>. Association for Computing Machinery, New York, NY, USA, 154–161. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/1076034.1076063" title="">https://doi.org/10.1145/1076034.1076063</a> </span> </li> <li class="ltx_bibitem" id="bib.bib19"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Joachims et al<span class="ltx_text" id="bib.bib19.2.2.1">.</span> (2007)</span> <span class="ltx_bibblock"> Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007. </span> <span class="ltx_bibblock">Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib19.3.1">ACM Trans. Inf. Syst.</em> 25, 2 (4 2007), 7–es. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/1229179.1229181" title="">https://doi.org/10.1145/1229179.1229181</a> </span> </li> <li class="ltx_bibitem" id="bib.bib20"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Leung et al<span class="ltx_text" id="bib.bib20.2.2.1">.</span> (2016)</span> <span class="ltx_bibblock"> Kenneth Wai-Ting Leung, Di Jiang, Dik Lun Lee, and Wilfred Ng. 2016. </span> <span class="ltx_bibblock">Constructing Maintainable Semantic Relation Network from Ambiguous Concepts in Web Content. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib20.3.1">ACM Transactions on Internet Technology</em> 16, 1, Article 6 (2 2016), 23 pages. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2814568" title="">https://doi.org/10.1145/2814568</a> </span> </li> <li class="ltx_bibitem" id="bib.bib21"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al<span class="ltx_text" id="bib.bib21.2.2.1">.</span> (2021)</span> <span class="ltx_bibblock"> Chen Li, Di Jiang, Xinyu Wang, WEI Yibin, Pu Wang, and HE Jingzhou. 2021. </span> <span class="ltx_bibblock">Method, apparatus, server, and storage medium for recalling for search. </span> <span class="ltx_bibblock"> </span> <span class="ltx_bibblock"> </span> <span class="ltx_bibblock">US Patent 11,182,445. </span> </li> <li class="ltx_bibitem" id="bib.bib22"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al<span class="ltx_text" id="bib.bib22.2.2.1">.</span> (2023)</span> <span class="ltx_bibblock"> Yawen Li, Di Jiang, Rongzhong Lian, Xueyang Wu, Conghui Tan, Yi Xu, and Zhiyang Su. 2023. </span> <span class="ltx_bibblock">Heterogeneous Latent Topic Discovery for Semantic Text Mining. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib22.3.1">IEEE Transactions on Knowledge and Data Engineering</em> 35, 1 (2023), 533–544. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1109/TKDE.2021.3077025" title="">https://doi.org/10.1109/TKDE.2021.3077025</a> </span> </li> <li class="ltx_bibitem" id="bib.bib23"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radlinski and Joachims (2005)</span> <span class="ltx_bibblock"> Filip Radlinski and Thorsten Joachims. 2005. </span> <span class="ltx_bibblock">Query chains: learning to rank from implicit feedback. In <em class="ltx_emph ltx_font_italic" id="bib.bib23.1.1">Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining</em> (Chicago, Illinois, USA) <em class="ltx_emph ltx_font_italic" id="bib.bib23.2.2">(KDD ’05)</em>. Association for Computing Machinery, New York, NY, USA, 239–248. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/1081870.1081899" title="">https://doi.org/10.1145/1081870.1081899</a> </span> </li> <li class="ltx_bibitem" id="bib.bib24"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shen et al<span class="ltx_text" id="bib.bib24.2.2.1">.</span> (2012)</span> <span class="ltx_bibblock"> Si Shen, Botao Hu, Weizhu Chen, and Qiang Yang. 2012. </span> <span class="ltx_bibblock">Personalized click model through collaborative filtering. In <em class="ltx_emph ltx_font_italic" id="bib.bib24.3.1">Proceedings of the Fifth ACM International Conference on Web Search and Data Mining</em> (Seattle, Washington, USA) <em class="ltx_emph ltx_font_italic" id="bib.bib24.4.2">(WSDM ’12)</em>. Association for Computing Machinery, New York, NY, USA, 323–332. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2124295.2124336" title="">https://doi.org/10.1145/2124295.2124336</a> </span> </li> <li class="ltx_bibitem" id="bib.bib25"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shen et al<span class="ltx_text" id="bib.bib25.2.2.1">.</span> (2014)</span> <span class="ltx_bibblock"> Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. </span> <span class="ltx_bibblock">A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval. In <em class="ltx_emph ltx_font_italic" id="bib.bib25.3.1">Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management</em> (Shanghai, China) <em class="ltx_emph ltx_font_italic" id="bib.bib25.4.2">(CIKM ’14)</em>. Association for Computing Machinery, New York, NY, USA, 101–110. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1145/2661829.2661935" title="">https://doi.org/10.1145/2661829.2661935</a> </span> </li> <li class="ltx_bibitem" id="bib.bib26"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al<span class="ltx_text" id="bib.bib26.2.2.1">.</span> (2020)</span> <span class="ltx_bibblock"> Xinyu Wang, Di Jiang, Lei Shi, Chen Li, LIAO Meng, and HE Jingzhou. 2020. </span> <span class="ltx_bibblock">Searching method and searching apparatus based on neural network and search engine. </span> <span class="ltx_bibblock"> </span> <span class="ltx_bibblock"> </span> <span class="ltx_bibblock">US Patent 10,762,150. </span> </li> <li class="ltx_bibitem" id="bib.bib27"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al<span class="ltx_text" id="bib.bib27.2.2.1">.</span> (2014)</span> <span class="ltx_bibblock"> Haiyang Wu, Daxiang Dong, Xiaoguang Hu, Dianhai Yu, Wei He, Hua Wu, Haifeng Wang, and Ting Liu. 2014. </span> <span class="ltx_bibblock">Improve Statistical Machine Translation with Context-Sensitive Bilingual Semantic Embedding Model. In <em class="ltx_emph ltx_font_italic" id="bib.bib27.3.1">Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)</em>, Alessandro Moschitti, Bo Pang, and Walter Daelemans (Eds.). Association for Computational Linguistics, Doha, Qatar, 142–146. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.3115/v1/D14-1015" title="">https://doi.org/10.3115/v1/D14-1015</a> </span> </li> <li class="ltx_bibitem" id="bib.bib28"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al<span class="ltx_text" id="bib.bib28.2.2.1">.</span> (2023)</span> <span class="ltx_bibblock"> Chen Zhang, Haodi Zhang, Qifan Li, Kaishun Wu, Di Jiang, Yuanfeng Song, Peiguang Lin, and Lei Chen. 2023. </span> <span class="ltx_bibblock">Burstiness-Aware Web Search Analysis on Different Levels of Evidences. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib28.3.1">IEEE Transactions on Knowledge and Data Engineering</em> 35, 3 (2023), 2341–2352. </span> <span class="ltx_bibblock"> <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://doi.org/10.1109/TKDE.2021.3109304" title="">https://doi.org/10.1109/TKDE.2021.3109304</a> </span> </li> </ul> </section> </article> </div> <footer class="ltx_page_footer"> <div class="ltx_page_logo">Generated on Thu Nov 21 16:41:08 2024 by <a class="ltx_LaTeXML_logo" href="http://dlmf.nist.gov/LaTeXML/"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img alt="Mascot Sammy" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg=="/></a> </div></footer> </div> </body> </html>