CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation</title>  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <base href="/html/2501.09292v3/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S1" title="In To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S2" title="In To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>Related Work</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S3" title="In To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Tasks and Datasets</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S4" title="In To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Approach</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S4.SS1" title="In 4 Approach ‣ To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.1 </span>Uncertainty Evaluation of Future Sentence</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S4.SS2" title="In 4 Approach ‣ To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2 </span>Sequence Level Uncertainty Evaluation Measures</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S4.SS3" title="In 4 Approach ‣ To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.3 </span>Subquery Generation for Retrieval</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S5" title="In To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Setup</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S6" title="In To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>Results</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S7" title="In To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">7 </span>Conclusion</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S8" title="In To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">8 </span>Ethical Considerations</span></a></li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line"> <h1 class="ltx_title ltx_title_document">To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation</h1> <div class="ltx_authors"> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Kaustubh D. Dhole <br class="ltx_break"/>Department of Computer Science <br class="ltx_break"/>Emory University <br class="ltx_break"/>Atlanta, GA 30307, USA <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id1.1.id1">kdhole@emory.edu</span> <br class="ltx_break"/> </span></span> </div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract</h6> <p class="ltx_p" id="id2.id1">Retrieval-Augmented Generation equips large language models with the capability to retrieve external knowledge, thereby mitigating hallucinations by incorporating information beyond the model’s intrinsic abilities. However, most prior works have focused on invoking retrieval deterministically, which makes it unsuitable for tasks such as long-form question answering. Instead, dynamically performing retrieval by invoking it only when the underlying LLM lacks the required knowledge can be more efficient. In this context, we delve deeper into the question, “To Retrieve or Not to Retrieve?” by exploring multiple uncertainty detection methods. We evaluate these methods for the task of long-form question answering, employing dynamic retrieval, and present our comparisons. Our findings suggest that uncertainty detection metrics, such as Degree Matrix Jaccard and Eccentricity, can reduce the number of retrieval calls by almost half, with only a slight reduction in question-answering accuracy.</p> </div> <section class="ltx_section" id="S1"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2> <div class="ltx_para ltx_noindent" id="S1.p1"> <p class="ltx_p" id="S1.p1.1">Recently, Large Language Models (LLMs) like ChatGPT <cite class="ltx_cite ltx_citemacro_cite">OpenAI (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib16" title="">2023</a>)</cite>, Gemini <cite class="ltx_cite ltx_citemacro_cite">Team et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib21" title="">2023</a>)</cite>, and others are showing impressive strides in tasks across numerous benchmarks <cite class="ltx_cite ltx_citemacro_cite">Srivastava et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib20" title="">2023</a>)</cite>. This success has been largely owed to their exposure to massive training data and successive fine-tuning of instruction datasets. To increase the helpfulness and decrease the harmfulness of the models, they are being further fine-tuned over preference collections <cite class="ltx_cite ltx_citemacro_cite">Bai et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib1" title="">2022</a>); Ouyang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib17" title="">2022</a>); Rafailov et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib18" title="">2024</a>)</cite>.</p> </div> <div class="ltx_para ltx_noindent" id="S1.p2"> <p class="ltx_p" id="S1.p2.1">Further, Retrieval Augmented Generation (RAG) <cite class="ltx_cite ltx_citemacro_cite">Lewis et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib12" title="">2020</a>); Dhole (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib4" title="">2024a</a>); Dhole et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib6" title="">2024</a>)</cite>, in the effort to mitigate hallucinations, enriches these models with domain-specific information and tackles scenarios where the intrinsic knowledge of the base model falls short. By integrating externally retrieved content during the generation phase, RAG enhances the model’s ability to produce less hallucinatory and domain-conditioned responses. This approach has been particularly valuable in complex applications such as long-form generation like multi-hop question answering, which often requires multiple retrievals to address a query comprehensively.</p> </div> <div class="ltx_para ltx_noindent" id="S1.p3"> <p class="ltx_p" id="S1.p3.1">However, to optimize the efficiency of RAG, retrieval should only be invoked when necessary — also referred to as conditional retrieval. Previous conditional RAG setups have explored multiple paradigms like low token probabilities <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib9" title="">2023</a>)</cite>, external classifiers <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib23" title="">2023</a>)</cite>, or low entity popularity <cite class="ltx_cite ltx_citemacro_cite">Mallen et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib15" title="">2023</a>)</cite> as indicators of the LLMs’ knowledge gaps. However, most of these methods fall short in either approximating knowledge gaps of the LLMs or lacking the ability to invoke retrieval dynamically.</p> </div> <div class="ltx_para ltx_noindent" id="S1.p4"> <p class="ltx_p" id="S1.p4.1">On the other hand, with the potential of LLMs to hallucinate, there has been an increasing interest in <span class="ltx_text ltx_font_bold" id="S1.p4.1.1">uncertainty detection</span> methods to gauge LLMs’ confidence in their outputs <cite class="ltx_cite ltx_citemacro_cite">Fadeeva et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib7" title="">2023</a>)</cite>. Unlike traditional methods that rely on rigid heuristics or external classifiers, uncertainty detection leverages the inherent variability in LLM-generated responses to estimate confidence dynamically.</p> </div> <div class="ltx_para ltx_noindent" id="S1.p5"> <p class="ltx_p" id="S1.p5.1">For instance, semantic sets-based UD approaches <cite class="ltx_cite ltx_citemacro_cite">Lin et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib13" title="">2023</a>)</cite> group responses based on meaning, and use the number of clusters to directly reflect the level of uncertainty — with greater variability signaling higher uncertainty. Similarly, spectral methods using eigenvalue Laplacians quantify response diversity by identifying strong or weak clustering patterns in pairwise similarity graphs. These approaches align with the probabilistic nature of LLMs as well as adaptively gauge uncertainty based on output coherence, making them more robust against adversarial or ambiguous inputs.</p> </div> <div class="ltx_para ltx_noindent" id="S1.p6"> <p class="ltx_p" id="S1.p6.1">In this work, we evaluate if such uncertainty detection methods can indeed enhance the reliability of conditionally invoking retrieval, by measuring its impact on a downstream task of multi-hop question answering.</p> </div> <div class="ltx_para ltx_noindent" id="S1.p7"> <p class="ltx_p" id="S1.p7.1">In that regard, we resort to a conditional RAG system and employ numerous uncertainty detection metrics to test the need for invoking retrieval. Our RAG system performs forward-looking active retrieval in the style of <cite class="ltx_cite ltx_citemacro_citet">Jiang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib9" title="">2023</a>)</cite>.</p> </div> <div class="ltx_para ltx_noindent" id="S1.p8"> <p class="ltx_p" id="S1.p8.1">Specifically, we contribute the following:</p> <ul class="ltx_itemize" id="S1.I1"> <li class="ltx_item" id="S1.I1.i1" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S1.I1.i1.p1"> <p class="ltx_p" id="S1.I1.i1.p1.1">We design a method that performs retrieval augmented generation with dynamic retrieval through uncertainty detection</p> </div> </li> <li class="ltx_item" id="S1.I1.i2" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S1.I1.i2.p1"> <p class="ltx_p" id="S1.I1.i2.p1.1">We perform an exhaustive analysis of various conditions from the “uncertainty quantification” literature to gauge the best strategy to dynamically retrieve during generation</p> </div> </li> <li class="ltx_item" id="S1.I1.i3" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para ltx_noindent" id="S1.I1.i3.p1"> <p class="ltx_p" id="S1.I1.i3.p1.1">Based on the results, we present insights for future research</p> </div> </li> </ul> </div> <div class="ltx_para ltx_noindent" id="S1.p9"> <p class="ltx_p" id="S1.p9.1">Our insights are useful to gauge whether uncertainty detection methods can help improve the efficiency of RAG.</p> </div> </section> <section class="ltx_section" id="S2"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2> <div class="ltx_para ltx_noindent" id="S2.p1"> <p class="ltx_p" id="S2.p1.1">Here, we summarise some of the related work on uncertainty quantification and some active RAG efforts.</p> </div> <div class="ltx_para ltx_noindent" id="S2.p2"> <p class="ltx_p" id="S2.p2.1">There has been a lot of recent work on uncertainty quantification of white box and black box NLG models. <cite class="ltx_cite ltx_citemacro_citet">Lin et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib13" title="">2023</a>)</cite> showed that along with their generations, GPT-3 can output a verbalized form of the uncertainty, viz. “high confidence” or “85% confidence”. <cite class="ltx_cite ltx_citemacro_citet">Kadavath et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib10" title="">2022</a>)</cite> show that models can be made to sample answers and then made to self-evaluate the probability of P(True). <cite class="ltx_cite ltx_citemacro_citet">Kuhn et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib11" title="">2023</a>)</cite> recently proposed to compute the semantic entropy by considering the equivalence relationships amongst generated responses.</p> </div> <div class="ltx_para ltx_noindent" id="S2.p3"> <p class="ltx_p" id="S2.p3.1"><cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib22" title="">2024</a>)</cite> proposed Self-DC that tackled compositional questions via iterative divide-and-conquer based on LLM certainty. <cite class="ltx_cite ltx_citemacro_cite">Yao et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib25" title="">2024</a>)</cite> propose utilising the model’s internal states to estimate uncertainty and deciding whether to retrieve or not.</p> </div> <div class="ltx_para ltx_noindent" id="S2.p4"> <p class="ltx_p" id="S2.p4.1">We now describe the tasks and datasets used in our analysis along with the UD approaches employed.</p> </div> </section> <section class="ltx_section" id="S3"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">3 </span>Tasks and Datasets</h2> <div class="ltx_para ltx_noindent" id="S3.p1"> <p class="ltx_p" id="S3.p1.1">We conduct experiments on the 2WikiMultihopQA dataset <cite class="ltx_cite ltx_citemacro_cite">Ho et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib8" title="">2020</a>)</cite>, a multi-hop open domain question answering (QA) dataset that tests the reasoning and inference skills of question-answering models. Questions in this dataset generally require two steps of reasoning to deduce the final answer, and the information for each step of reasoning can be obtained through referencing external information viz., Wikipedia passages.</p> </div> </section> <section class="ltx_section" id="S4"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">4 </span>Approach</h2> <div class="ltx_para ltx_noindent" id="S4.p1"> <p class="ltx_p" id="S4.p1.1">We now describe our uncertainty-aware, retrieval-augmented generation in the following two subsections.</p> </div> <section class="ltx_subsection" id="S4.SS1"> <h3 class="ltx_title ltx_title_subsection"> <span class="ltx_tag ltx_tag_subsection">4.1 </span>Uncertainty Evaluation of Future Sentence</h3> <div class="ltx_para ltx_noindent" id="S4.SS1.p1"> <p class="ltx_p" id="S4.SS1.p1.7">Given a query <math alttext="\mathbf{q}" class="ltx_Math" display="inline" id="S4.SS1.p1.1.m1.1"><semantics id="S4.SS1.p1.1.m1.1a"><mi id="S4.SS1.p1.1.m1.1.1" xref="S4.SS1.p1.1.m1.1.1.cmml">𝐪</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.1.m1.1b"><ci id="S4.SS1.p1.1.m1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1">𝐪</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.1.m1.1c">\mathbf{q}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.1.m1.1d">bold_q</annotation></semantics></math>, a retriever <math alttext="\mathbf{R}" class="ltx_Math" display="inline" id="S4.SS1.p1.2.m2.1"><semantics id="S4.SS1.p1.2.m2.1a"><mi id="S4.SS1.p1.2.m2.1.1" xref="S4.SS1.p1.2.m2.1.1.cmml">𝐑</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.2.m2.1b"><ci id="S4.SS1.p1.2.m2.1.1.cmml" xref="S4.SS1.p1.2.m2.1.1">𝐑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.2.m2.1c">\mathbf{R}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.2.m2.1d">bold_R</annotation></semantics></math>, a text generator <math alttext="\mathbf{G}" class="ltx_Math" display="inline" id="S4.SS1.p1.3.m3.1"><semantics id="S4.SS1.p1.3.m3.1a"><mi id="S4.SS1.p1.3.m3.1.1" xref="S4.SS1.p1.3.m3.1.1.cmml">𝐆</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.3.m3.1b"><ci id="S4.SS1.p1.3.m3.1.1.cmml" xref="S4.SS1.p1.3.m3.1.1">𝐆</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.3.m3.1c">\mathbf{G}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.3.m3.1d">bold_G</annotation></semantics></math>, and a black box uncertainty estimation function <math alttext="\mathbf{U}" class="ltx_Math" display="inline" id="S4.SS1.p1.4.m4.1"><semantics id="S4.SS1.p1.4.m4.1a"><mi id="S4.SS1.p1.4.m4.1.1" xref="S4.SS1.p1.4.m4.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.4.m4.1b"><ci id="S4.SS1.p1.4.m4.1.1.cmml" xref="S4.SS1.p1.4.m4.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.4.m4.1c">\mathbf{U}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.4.m4.1d">bold_U</annotation></semantics></math>, and partially generated sequence <math alttext="t_{<i}" class="ltx_Math" display="inline" id="S4.SS1.p1.5.m5.1"><semantics id="S4.SS1.p1.5.m5.1a"><msub id="S4.SS1.p1.5.m5.1.1" xref="S4.SS1.p1.5.m5.1.1.cmml"><mi id="S4.SS1.p1.5.m5.1.1.2" xref="S4.SS1.p1.5.m5.1.1.2.cmml">t</mi><mrow id="S4.SS1.p1.5.m5.1.1.3" xref="S4.SS1.p1.5.m5.1.1.3.cmml"><mi id="S4.SS1.p1.5.m5.1.1.3.2" xref="S4.SS1.p1.5.m5.1.1.3.2.cmml"></mi><mo id="S4.SS1.p1.5.m5.1.1.3.1" xref="S4.SS1.p1.5.m5.1.1.3.1.cmml"><</mo><mi id="S4.SS1.p1.5.m5.1.1.3.3" xref="S4.SS1.p1.5.m5.1.1.3.3.cmml">i</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.5.m5.1b"><apply id="S4.SS1.p1.5.m5.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.5.m5.1.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1">subscript</csymbol><ci id="S4.SS1.p1.5.m5.1.1.2.cmml" xref="S4.SS1.p1.5.m5.1.1.2">𝑡</ci><apply id="S4.SS1.p1.5.m5.1.1.3.cmml" xref="S4.SS1.p1.5.m5.1.1.3"><lt id="S4.SS1.p1.5.m5.1.1.3.1.cmml" xref="S4.SS1.p1.5.m5.1.1.3.1"></lt><csymbol cd="latexml" id="S4.SS1.p1.5.m5.1.1.3.2.cmml" xref="S4.SS1.p1.5.m5.1.1.3.2">absent</csymbol><ci id="S4.SS1.p1.5.m5.1.1.3.3.cmml" xref="S4.SS1.p1.5.m5.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.5.m5.1c">t_{<i}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.5.m5.1d">italic_t start_POSTSUBSCRIPT < italic_i end_POSTSUBSCRIPT</annotation></semantics></math> until time step <math alttext="i" class="ltx_Math" display="inline" id="S4.SS1.p1.6.m6.1"><semantics id="S4.SS1.p1.6.m6.1a"><mi id="S4.SS1.p1.6.m6.1.1" xref="S4.SS1.p1.6.m6.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.6.m6.1b"><ci id="S4.SS1.p1.6.m6.1.1.cmml" xref="S4.SS1.p1.6.m6.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.6.m6.1c">i</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.6.m6.1d">italic_i</annotation></semantics></math>, – we first generate a temporary sentence <math alttext="t_{n}" class="ltx_Math" display="inline" id="S4.SS1.p1.7.m7.1"><semantics id="S4.SS1.p1.7.m7.1a"><msub id="S4.SS1.p1.7.m7.1.1" xref="S4.SS1.p1.7.m7.1.1.cmml"><mi id="S4.SS1.p1.7.m7.1.1.2" xref="S4.SS1.p1.7.m7.1.1.2.cmml">t</mi><mi id="S4.SS1.p1.7.m7.1.1.3" xref="S4.SS1.p1.7.m7.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.7.m7.1b"><apply id="S4.SS1.p1.7.m7.1.1.cmml" xref="S4.SS1.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.7.m7.1.1.1.cmml" xref="S4.SS1.p1.7.m7.1.1">subscript</csymbol><ci id="S4.SS1.p1.7.m7.1.1.2.cmml" xref="S4.SS1.p1.7.m7.1.1.2">𝑡</ci><ci id="S4.SS1.p1.7.m7.1.1.3.cmml" xref="S4.SS1.p1.7.m7.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.7.m7.1c">t_{n}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p1.7.m7.1d">italic_t start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> in the style of FLARE <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib9" title="">2023</a>)</cite>.</p> </div> <div class="ltx_para ltx_noindent" id="S4.SS1.p2"> <p class="ltx_p" id="S4.SS1.p2.7">We use a prompt template <math alttext="\mathbf{P}" class="ltx_Math" display="inline" id="S4.SS1.p2.1.m1.1"><semantics id="S4.SS1.p2.1.m1.1a"><mi id="S4.SS1.p2.1.m1.1.1" xref="S4.SS1.p2.1.m1.1.1.cmml">𝐏</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.1.m1.1b"><ci id="S4.SS1.p2.1.m1.1.1.cmml" xref="S4.SS1.p2.1.m1.1.1">𝐏</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.1.m1.1c">\mathbf{P}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.1.m1.1d">bold_P</annotation></semantics></math>, which could take the form of a zero-shot or a few-shot instruction. This instruction takes as input the query, zero or more retrieved documents <math alttext="d_{1}\ldots d_{k}" class="ltx_Math" display="inline" id="S4.SS1.p2.2.m2.1"><semantics id="S4.SS1.p2.2.m2.1a"><mrow id="S4.SS1.p2.2.m2.1.1" xref="S4.SS1.p2.2.m2.1.1.cmml"><msub id="S4.SS1.p2.2.m2.1.1.2" xref="S4.SS1.p2.2.m2.1.1.2.cmml"><mi id="S4.SS1.p2.2.m2.1.1.2.2" xref="S4.SS1.p2.2.m2.1.1.2.2.cmml">d</mi><mn id="S4.SS1.p2.2.m2.1.1.2.3" xref="S4.SS1.p2.2.m2.1.1.2.3.cmml">1</mn></msub><mo id="S4.SS1.p2.2.m2.1.1.1" xref="S4.SS1.p2.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.SS1.p2.2.m2.1.1.3" mathvariant="normal" xref="S4.SS1.p2.2.m2.1.1.3.cmml">…</mi><mo id="S4.SS1.p2.2.m2.1.1.1a" xref="S4.SS1.p2.2.m2.1.1.1.cmml">⁢</mo><msub id="S4.SS1.p2.2.m2.1.1.4" xref="S4.SS1.p2.2.m2.1.1.4.cmml"><mi id="S4.SS1.p2.2.m2.1.1.4.2" xref="S4.SS1.p2.2.m2.1.1.4.2.cmml">d</mi><mi id="S4.SS1.p2.2.m2.1.1.4.3" xref="S4.SS1.p2.2.m2.1.1.4.3.cmml">k</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.2.m2.1b"><apply id="S4.SS1.p2.2.m2.1.1.cmml" xref="S4.SS1.p2.2.m2.1.1"><times id="S4.SS1.p2.2.m2.1.1.1.cmml" xref="S4.SS1.p2.2.m2.1.1.1"></times><apply id="S4.SS1.p2.2.m2.1.1.2.cmml" xref="S4.SS1.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="S4.SS1.p2.2.m2.1.1.2.1.cmml" xref="S4.SS1.p2.2.m2.1.1.2">subscript</csymbol><ci id="S4.SS1.p2.2.m2.1.1.2.2.cmml" xref="S4.SS1.p2.2.m2.1.1.2.2">𝑑</ci><cn id="S4.SS1.p2.2.m2.1.1.2.3.cmml" type="integer" xref="S4.SS1.p2.2.m2.1.1.2.3">1</cn></apply><ci id="S4.SS1.p2.2.m2.1.1.3.cmml" xref="S4.SS1.p2.2.m2.1.1.3">…</ci><apply id="S4.SS1.p2.2.m2.1.1.4.cmml" xref="S4.SS1.p2.2.m2.1.1.4"><csymbol cd="ambiguous" id="S4.SS1.p2.2.m2.1.1.4.1.cmml" xref="S4.SS1.p2.2.m2.1.1.4">subscript</csymbol><ci id="S4.SS1.p2.2.m2.1.1.4.2.cmml" xref="S4.SS1.p2.2.m2.1.1.4.2">𝑑</ci><ci id="S4.SS1.p2.2.m2.1.1.4.3.cmml" xref="S4.SS1.p2.2.m2.1.1.4.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.2.m2.1c">d_{1}\ldots d_{k}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.2.m2.1d">italic_d start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT … italic_d start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math>, and the answer tokens generated until now. Here, we use <math alttext="t_{i}" class="ltx_Math" display="inline" id="S4.SS1.p2.3.m3.1"><semantics id="S4.SS1.p2.3.m3.1a"><msub id="S4.SS1.p2.3.m3.1.1" xref="S4.SS1.p2.3.m3.1.1.cmml"><mi id="S4.SS1.p2.3.m3.1.1.2" xref="S4.SS1.p2.3.m3.1.1.2.cmml">t</mi><mi id="S4.SS1.p2.3.m3.1.1.3" xref="S4.SS1.p2.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.3.m3.1b"><apply id="S4.SS1.p2.3.m3.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.3.m3.1.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1">subscript</csymbol><ci id="S4.SS1.p2.3.m3.1.1.2.cmml" xref="S4.SS1.p2.3.m3.1.1.2">𝑡</ci><ci id="S4.SS1.p2.3.m3.1.1.3.cmml" xref="S4.SS1.p2.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.3.m3.1c">t_{i}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.3.m3.1d">italic_t start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT</annotation></semantics></math> to represent the <math alttext="i^{th}" class="ltx_Math" display="inline" id="S4.SS1.p2.4.m4.1"><semantics id="S4.SS1.p2.4.m4.1a"><msup id="S4.SS1.p2.4.m4.1.1" xref="S4.SS1.p2.4.m4.1.1.cmml"><mi id="S4.SS1.p2.4.m4.1.1.2" xref="S4.SS1.p2.4.m4.1.1.2.cmml">i</mi><mrow id="S4.SS1.p2.4.m4.1.1.3" xref="S4.SS1.p2.4.m4.1.1.3.cmml"><mi id="S4.SS1.p2.4.m4.1.1.3.2" xref="S4.SS1.p2.4.m4.1.1.3.2.cmml">t</mi><mo id="S4.SS1.p2.4.m4.1.1.3.1" xref="S4.SS1.p2.4.m4.1.1.3.1.cmml">⁢</mo><mi id="S4.SS1.p2.4.m4.1.1.3.3" xref="S4.SS1.p2.4.m4.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.4.m4.1b"><apply id="S4.SS1.p2.4.m4.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.4.m4.1.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1">superscript</csymbol><ci id="S4.SS1.p2.4.m4.1.1.2.cmml" xref="S4.SS1.p2.4.m4.1.1.2">𝑖</ci><apply id="S4.SS1.p2.4.m4.1.1.3.cmml" xref="S4.SS1.p2.4.m4.1.1.3"><times id="S4.SS1.p2.4.m4.1.1.3.1.cmml" xref="S4.SS1.p2.4.m4.1.1.3.1"></times><ci id="S4.SS1.p2.4.m4.1.1.3.2.cmml" xref="S4.SS1.p2.4.m4.1.1.3.2">𝑡</ci><ci id="S4.SS1.p2.4.m4.1.1.3.3.cmml" xref="S4.SS1.p2.4.m4.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.4.m4.1c">i^{th}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.4.m4.1d">italic_i start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT</annotation></semantics></math> temporary sentence and <math alttext="y_{<i}" class="ltx_Math" display="inline" id="S4.SS1.p2.5.m5.1"><semantics id="S4.SS1.p2.5.m5.1a"><msub id="S4.SS1.p2.5.m5.1.1" xref="S4.SS1.p2.5.m5.1.1.cmml"><mi id="S4.SS1.p2.5.m5.1.1.2" xref="S4.SS1.p2.5.m5.1.1.2.cmml">y</mi><mrow id="S4.SS1.p2.5.m5.1.1.3" xref="S4.SS1.p2.5.m5.1.1.3.cmml"><mi id="S4.SS1.p2.5.m5.1.1.3.2" xref="S4.SS1.p2.5.m5.1.1.3.2.cmml"></mi><mo id="S4.SS1.p2.5.m5.1.1.3.1" xref="S4.SS1.p2.5.m5.1.1.3.1.cmml"><</mo><mi id="S4.SS1.p2.5.m5.1.1.3.3" xref="S4.SS1.p2.5.m5.1.1.3.3.cmml">i</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.5.m5.1b"><apply id="S4.SS1.p2.5.m5.1.1.cmml" xref="S4.SS1.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.5.m5.1.1.1.cmml" xref="S4.SS1.p2.5.m5.1.1">subscript</csymbol><ci id="S4.SS1.p2.5.m5.1.1.2.cmml" xref="S4.SS1.p2.5.m5.1.1.2">𝑦</ci><apply id="S4.SS1.p2.5.m5.1.1.3.cmml" xref="S4.SS1.p2.5.m5.1.1.3"><lt id="S4.SS1.p2.5.m5.1.1.3.1.cmml" xref="S4.SS1.p2.5.m5.1.1.3.1"></lt><csymbol cd="latexml" id="S4.SS1.p2.5.m5.1.1.3.2.cmml" xref="S4.SS1.p2.5.m5.1.1.3.2">absent</csymbol><ci id="S4.SS1.p2.5.m5.1.1.3.3.cmml" xref="S4.SS1.p2.5.m5.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.5.m5.1c">y_{<i}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.5.m5.1d">italic_y start_POSTSUBSCRIPT < italic_i end_POSTSUBSCRIPT</annotation></semantics></math> to represent all the initialised and generated sentences <math alttext="\{0\ldots(i-1)\}" class="ltx_Math" display="inline" id="S4.SS1.p2.6.m6.1"><semantics id="S4.SS1.p2.6.m6.1a"><mrow id="S4.SS1.p2.6.m6.1.1.1" xref="S4.SS1.p2.6.m6.1.1.2.cmml"><mo id="S4.SS1.p2.6.m6.1.1.1.2" stretchy="false" xref="S4.SS1.p2.6.m6.1.1.2.cmml">{</mo><mrow id="S4.SS1.p2.6.m6.1.1.1.1" xref="S4.SS1.p2.6.m6.1.1.1.1.cmml"><mn id="S4.SS1.p2.6.m6.1.1.1.1.3" xref="S4.SS1.p2.6.m6.1.1.1.1.3.cmml">0</mn><mo id="S4.SS1.p2.6.m6.1.1.1.1.2" xref="S4.SS1.p2.6.m6.1.1.1.1.2.cmml">⁢</mo><mi id="S4.SS1.p2.6.m6.1.1.1.1.4" mathvariant="normal" xref="S4.SS1.p2.6.m6.1.1.1.1.4.cmml">…</mi><mo id="S4.SS1.p2.6.m6.1.1.1.1.2a" xref="S4.SS1.p2.6.m6.1.1.1.1.2.cmml">⁢</mo><mrow id="S4.SS1.p2.6.m6.1.1.1.1.1.1" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.cmml"><mo id="S4.SS1.p2.6.m6.1.1.1.1.1.1.2" stretchy="false" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS1.p2.6.m6.1.1.1.1.1.1.1" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.cmml"><mi id="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.2" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.2.cmml">i</mi><mo id="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.1" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.3" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo id="S4.SS1.p2.6.m6.1.1.1.1.1.1.3" stretchy="false" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.SS1.p2.6.m6.1.1.1.3" stretchy="false" xref="S4.SS1.p2.6.m6.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.6.m6.1b"><set id="S4.SS1.p2.6.m6.1.1.2.cmml" xref="S4.SS1.p2.6.m6.1.1.1"><apply id="S4.SS1.p2.6.m6.1.1.1.1.cmml" xref="S4.SS1.p2.6.m6.1.1.1.1"><times id="S4.SS1.p2.6.m6.1.1.1.1.2.cmml" xref="S4.SS1.p2.6.m6.1.1.1.1.2"></times><cn id="S4.SS1.p2.6.m6.1.1.1.1.3.cmml" type="integer" xref="S4.SS1.p2.6.m6.1.1.1.1.3">0</cn><ci id="S4.SS1.p2.6.m6.1.1.1.1.4.cmml" xref="S4.SS1.p2.6.m6.1.1.1.1.4">…</ci><apply id="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.cmml" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1"><minus id="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.1.cmml" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.1"></minus><ci id="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.2.cmml" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.2">𝑖</ci><cn id="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.3.cmml" type="integer" xref="S4.SS1.p2.6.m6.1.1.1.1.1.1.1.3">1</cn></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.6.m6.1c">\{0\ldots(i-1)\}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.6.m6.1d">{ 0 … ( italic_i - 1 ) }</annotation></semantics></math>. <math alttext="t_{i}" class="ltx_Math" display="inline" id="S4.SS1.p2.7.m7.1"><semantics id="S4.SS1.p2.7.m7.1a"><msub id="S4.SS1.p2.7.m7.1.1" xref="S4.SS1.p2.7.m7.1.1.cmml"><mi id="S4.SS1.p2.7.m7.1.1.2" xref="S4.SS1.p2.7.m7.1.1.2.cmml">t</mi><mi id="S4.SS1.p2.7.m7.1.1.3" xref="S4.SS1.p2.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.7.m7.1b"><apply id="S4.SS1.p2.7.m7.1.1.cmml" xref="S4.SS1.p2.7.m7.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.7.m7.1.1.1.cmml" xref="S4.SS1.p2.7.m7.1.1">subscript</csymbol><ci id="S4.SS1.p2.7.m7.1.1.2.cmml" xref="S4.SS1.p2.7.m7.1.1.2">𝑡</ci><ci id="S4.SS1.p2.7.m7.1.1.3.cmml" xref="S4.SS1.p2.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.7.m7.1c">t_{i}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p2.7.m7.1d">italic_t start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT</annotation></semantics></math> is first obtained without performing retrieval:</p> <table class="ltx_equation ltx_eqn_table" id="S4.E1"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="t_{i}=\mathbf{G}(\mathbf{P}\{\mathbf{q},\ldots,y_{i-1}\})" class="ltx_Math" display="block" id="S4.E1.m1.3"><semantics id="S4.E1.m1.3a"><mrow id="S4.E1.m1.3.3" xref="S4.E1.m1.3.3.cmml"><msub id="S4.E1.m1.3.3.3" xref="S4.E1.m1.3.3.3.cmml"><mi id="S4.E1.m1.3.3.3.2" xref="S4.E1.m1.3.3.3.2.cmml">t</mi><mi id="S4.E1.m1.3.3.3.3" xref="S4.E1.m1.3.3.3.3.cmml">i</mi></msub><mo id="S4.E1.m1.3.3.2" xref="S4.E1.m1.3.3.2.cmml">=</mo><mrow id="S4.E1.m1.3.3.1" xref="S4.E1.m1.3.3.1.cmml"><mi id="S4.E1.m1.3.3.1.3" xref="S4.E1.m1.3.3.1.3.cmml">𝐆</mi><mo id="S4.E1.m1.3.3.1.2" xref="S4.E1.m1.3.3.1.2.cmml">⁢</mo><mrow id="S4.E1.m1.3.3.1.1.1" xref="S4.E1.m1.3.3.1.1.1.1.cmml"><mo id="S4.E1.m1.3.3.1.1.1.2" stretchy="false" xref="S4.E1.m1.3.3.1.1.1.1.cmml">(</mo><mrow id="S4.E1.m1.3.3.1.1.1.1" xref="S4.E1.m1.3.3.1.1.1.1.cmml"><mi id="S4.E1.m1.3.3.1.1.1.1.3" xref="S4.E1.m1.3.3.1.1.1.1.3.cmml">𝐏</mi><mo id="S4.E1.m1.3.3.1.1.1.1.2" xref="S4.E1.m1.3.3.1.1.1.1.2.cmml">⁢</mo><mrow id="S4.E1.m1.3.3.1.1.1.1.1.1" xref="S4.E1.m1.3.3.1.1.1.1.1.2.cmml"><mo id="S4.E1.m1.3.3.1.1.1.1.1.1.2" stretchy="false" xref="S4.E1.m1.3.3.1.1.1.1.1.2.cmml">{</mo><mi id="S4.E1.m1.1.1" xref="S4.E1.m1.1.1.cmml">𝐪</mi><mo id="S4.E1.m1.3.3.1.1.1.1.1.1.3" xref="S4.E1.m1.3.3.1.1.1.1.1.2.cmml">,</mo><mi id="S4.E1.m1.2.2" mathvariant="normal" xref="S4.E1.m1.2.2.cmml">…</mi><mo id="S4.E1.m1.3.3.1.1.1.1.1.1.4" xref="S4.E1.m1.3.3.1.1.1.1.1.2.cmml">,</mo><msub id="S4.E1.m1.3.3.1.1.1.1.1.1.1" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.cmml"><mi id="S4.E1.m1.3.3.1.1.1.1.1.1.1.2" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.2.cmml">y</mi><mrow id="S4.E1.m1.3.3.1.1.1.1.1.1.1.3" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.2" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.2.cmml">i</mi><mo id="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.1" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.1.cmml">−</mo><mn id="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.3" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S4.E1.m1.3.3.1.1.1.1.1.1.5" stretchy="false" xref="S4.E1.m1.3.3.1.1.1.1.1.2.cmml">}</mo></mrow></mrow><mo id="S4.E1.m1.3.3.1.1.1.3" stretchy="false" xref="S4.E1.m1.3.3.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E1.m1.3b"><apply id="S4.E1.m1.3.3.cmml" xref="S4.E1.m1.3.3"><eq id="S4.E1.m1.3.3.2.cmml" xref="S4.E1.m1.3.3.2"></eq><apply id="S4.E1.m1.3.3.3.cmml" xref="S4.E1.m1.3.3.3"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.3.1.cmml" xref="S4.E1.m1.3.3.3">subscript</csymbol><ci id="S4.E1.m1.3.3.3.2.cmml" xref="S4.E1.m1.3.3.3.2">𝑡</ci><ci id="S4.E1.m1.3.3.3.3.cmml" xref="S4.E1.m1.3.3.3.3">𝑖</ci></apply><apply id="S4.E1.m1.3.3.1.cmml" xref="S4.E1.m1.3.3.1"><times id="S4.E1.m1.3.3.1.2.cmml" xref="S4.E1.m1.3.3.1.2"></times><ci id="S4.E1.m1.3.3.1.3.cmml" xref="S4.E1.m1.3.3.1.3">𝐆</ci><apply id="S4.E1.m1.3.3.1.1.1.1.cmml" xref="S4.E1.m1.3.3.1.1.1"><times id="S4.E1.m1.3.3.1.1.1.1.2.cmml" xref="S4.E1.m1.3.3.1.1.1.1.2"></times><ci id="S4.E1.m1.3.3.1.1.1.1.3.cmml" xref="S4.E1.m1.3.3.1.1.1.1.3">𝐏</ci><set id="S4.E1.m1.3.3.1.1.1.1.1.2.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.1"><ci id="S4.E1.m1.1.1.cmml" xref="S4.E1.m1.1.1">𝐪</ci><ci id="S4.E1.m1.2.2.cmml" xref="S4.E1.m1.2.2">…</ci><apply id="S4.E1.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S4.E1.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.2">𝑦</ci><apply id="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.3"><minus id="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.1"></minus><ci id="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.2">𝑖</ci><cn id="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.3.cmml" type="integer" xref="S4.E1.m1.3.3.1.1.1.1.1.1.1.3.3">1</cn></apply></apply></set></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E1.m1.3c">t_{i}=\mathbf{G}(\mathbf{P}\{\mathbf{q},\ldots,y_{i-1}\})</annotation><annotation encoding="application/x-llamapun" id="S4.E1.m1.3d">italic_t start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = bold_G ( bold_P { bold_q , … , italic_y start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT } )</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td> </tr></tbody> </table> </div> <div class="ltx_para ltx_noindent" id="S4.SS1.p3"> <p class="ltx_p" id="S4.SS1.p3.4">During generation, we evaluate the uncertainty of this temporary sentence <math alttext="t_{n}" class="ltx_Math" display="inline" id="S4.SS1.p3.1.m1.1"><semantics id="S4.SS1.p3.1.m1.1a"><msub id="S4.SS1.p3.1.m1.1.1" xref="S4.SS1.p3.1.m1.1.1.cmml"><mi id="S4.SS1.p3.1.m1.1.1.2" xref="S4.SS1.p3.1.m1.1.1.2.cmml">t</mi><mi id="S4.SS1.p3.1.m1.1.1.3" xref="S4.SS1.p3.1.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.1.m1.1b"><apply id="S4.SS1.p3.1.m1.1.1.cmml" xref="S4.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p3.1.m1.1.1.1.cmml" xref="S4.SS1.p3.1.m1.1.1">subscript</csymbol><ci id="S4.SS1.p3.1.m1.1.1.2.cmml" xref="S4.SS1.p3.1.m1.1.1.2">𝑡</ci><ci id="S4.SS1.p3.1.m1.1.1.3.cmml" xref="S4.SS1.p3.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.1.m1.1c">t_{n}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p3.1.m1.1d">italic_t start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT</annotation></semantics></math> to gauge if the generator needs more information. If the uncertainty <math alttext="\mathbf{U}(t_{n})" class="ltx_Math" display="inline" id="S4.SS1.p3.2.m2.1"><semantics id="S4.SS1.p3.2.m2.1a"><mrow id="S4.SS1.p3.2.m2.1.1" xref="S4.SS1.p3.2.m2.1.1.cmml"><mi id="S4.SS1.p3.2.m2.1.1.3" xref="S4.SS1.p3.2.m2.1.1.3.cmml">𝐔</mi><mo id="S4.SS1.p3.2.m2.1.1.2" xref="S4.SS1.p3.2.m2.1.1.2.cmml">⁢</mo><mrow id="S4.SS1.p3.2.m2.1.1.1.1" xref="S4.SS1.p3.2.m2.1.1.1.1.1.cmml"><mo id="S4.SS1.p3.2.m2.1.1.1.1.2" stretchy="false" xref="S4.SS1.p3.2.m2.1.1.1.1.1.cmml">(</mo><msub id="S4.SS1.p3.2.m2.1.1.1.1.1" xref="S4.SS1.p3.2.m2.1.1.1.1.1.cmml"><mi id="S4.SS1.p3.2.m2.1.1.1.1.1.2" xref="S4.SS1.p3.2.m2.1.1.1.1.1.2.cmml">t</mi><mi id="S4.SS1.p3.2.m2.1.1.1.1.1.3" xref="S4.SS1.p3.2.m2.1.1.1.1.1.3.cmml">n</mi></msub><mo id="S4.SS1.p3.2.m2.1.1.1.1.3" stretchy="false" xref="S4.SS1.p3.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.2.m2.1b"><apply id="S4.SS1.p3.2.m2.1.1.cmml" xref="S4.SS1.p3.2.m2.1.1"><times id="S4.SS1.p3.2.m2.1.1.2.cmml" xref="S4.SS1.p3.2.m2.1.1.2"></times><ci id="S4.SS1.p3.2.m2.1.1.3.cmml" xref="S4.SS1.p3.2.m2.1.1.3">𝐔</ci><apply id="S4.SS1.p3.2.m2.1.1.1.1.1.cmml" xref="S4.SS1.p3.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p3.2.m2.1.1.1.1.1.1.cmml" xref="S4.SS1.p3.2.m2.1.1.1.1">subscript</csymbol><ci id="S4.SS1.p3.2.m2.1.1.1.1.1.2.cmml" xref="S4.SS1.p3.2.m2.1.1.1.1.1.2">𝑡</ci><ci id="S4.SS1.p3.2.m2.1.1.1.1.1.3.cmml" xref="S4.SS1.p3.2.m2.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.2.m2.1c">\mathbf{U}(t_{n})</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p3.2.m2.1d">bold_U ( italic_t start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT )</annotation></semantics></math> exceeds a threshold <math alttext="\theta_{\mathbf{U}}" class="ltx_Math" display="inline" id="S4.SS1.p3.3.m3.1"><semantics id="S4.SS1.p3.3.m3.1a"><msub id="S4.SS1.p3.3.m3.1.1" xref="S4.SS1.p3.3.m3.1.1.cmml"><mi id="S4.SS1.p3.3.m3.1.1.2" xref="S4.SS1.p3.3.m3.1.1.2.cmml">θ</mi><mi id="S4.SS1.p3.3.m3.1.1.3" xref="S4.SS1.p3.3.m3.1.1.3.cmml">𝐔</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.3.m3.1b"><apply id="S4.SS1.p3.3.m3.1.1.cmml" xref="S4.SS1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS1.p3.3.m3.1.1.1.cmml" xref="S4.SS1.p3.3.m3.1.1">subscript</csymbol><ci id="S4.SS1.p3.3.m3.1.1.2.cmml" xref="S4.SS1.p3.3.m3.1.1.2">𝜃</ci><ci id="S4.SS1.p3.3.m3.1.1.3.cmml" xref="S4.SS1.p3.3.m3.1.1.3">𝐔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.3.m3.1c">\theta_{\mathbf{U}}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p3.3.m3.1d">italic_θ start_POSTSUBSCRIPT bold_U end_POSTSUBSCRIPT</annotation></semantics></math>, the model is not certain and may lack the necessary knowledge to provide an accurate answer. The next sentence <math alttext="y_{i}" class="ltx_Math" display="inline" id="S4.SS1.p3.4.m4.1"><semantics id="S4.SS1.p3.4.m4.1a"><msub id="S4.SS1.p3.4.m4.1.1" xref="S4.SS1.p3.4.m4.1.1.cmml"><mi id="S4.SS1.p3.4.m4.1.1.2" xref="S4.SS1.p3.4.m4.1.1.2.cmml">y</mi><mi id="S4.SS1.p3.4.m4.1.1.3" xref="S4.SS1.p3.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.4.m4.1b"><apply id="S4.SS1.p3.4.m4.1.1.cmml" xref="S4.SS1.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.p3.4.m4.1.1.1.cmml" xref="S4.SS1.p3.4.m4.1.1">subscript</csymbol><ci id="S4.SS1.p3.4.m4.1.1.2.cmml" xref="S4.SS1.p3.4.m4.1.1.2">𝑦</ci><ci id="S4.SS1.p3.4.m4.1.1.3.cmml" xref="S4.SS1.p3.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.4.m4.1c">y_{i}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p3.4.m4.1d">italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT</annotation></semantics></math> is then computed by appending retrieved information to the model context:</p> <table class="ltx_equation ltx_eqn_table" id="S4.E2"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="y_{i}=\begin{cases}\mathbf{G}(\mathbf{P}\{d_{1},\ldots,d_{k},\mathbf{q},\ldots% ,y_{i-1}\})&\text{if}~{}\mathbf{U}(t_{i})>\theta_{\mathbf{U}}\\ \mathbf{G}(\mathbf{P}\{\mathbf{q},\ldots,y_{i-1}\})&otherwise\end{cases}" class="ltx_Math" display="block" id="S4.E2.m1.4"><semantics id="S4.E2.m1.4a"><mrow id="S4.E2.m1.4.5" xref="S4.E2.m1.4.5.cmml"><msub id="S4.E2.m1.4.5.2" xref="S4.E2.m1.4.5.2.cmml"><mi id="S4.E2.m1.4.5.2.2" xref="S4.E2.m1.4.5.2.2.cmml">y</mi><mi id="S4.E2.m1.4.5.2.3" xref="S4.E2.m1.4.5.2.3.cmml">i</mi></msub><mo id="S4.E2.m1.4.5.1" xref="S4.E2.m1.4.5.1.cmml">=</mo><mrow id="S4.E2.m1.4.4" xref="S4.E2.m1.4.5.3.1.cmml"><mo id="S4.E2.m1.4.4.5" xref="S4.E2.m1.4.5.3.1.1.cmml">{</mo><mtable columnspacing="5pt" displaystyle="true" id="S4.E2.m1.4.4.4" rowspacing="0pt" xref="S4.E2.m1.4.5.3.1.cmml"><mtr id="S4.E2.m1.4.4.4a" xref="S4.E2.m1.4.5.3.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="S4.E2.m1.4.4.4b" xref="S4.E2.m1.4.5.3.1.cmml"><mrow id="S4.E2.m1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.6" xref="S4.E2.m1.1.1.1.1.1.1.6.cmml">𝐆</mi><mo id="S4.E2.m1.1.1.1.1.1.1.5" xref="S4.E2.m1.1.1.1.1.1.1.5.cmml">⁢</mo><mrow id="S4.E2.m1.1.1.1.1.1.1.4.1" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.cmml"><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.2" stretchy="false" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.cmml">(</mo><mrow id="S4.E2.m1.1.1.1.1.1.1.4.1.1" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.4.1.1.5" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.5.cmml">𝐏</mi><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.4" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.4.cmml">⁢</mo><mrow id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml"><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.4" stretchy="false" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml">{</mo><msub id="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.2.cmml">d</mi><mn id="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.3" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.5" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml">,</mo><mi id="S4.E2.m1.1.1.1.1.1.1.1" mathvariant="normal" xref="S4.E2.m1.1.1.1.1.1.1.1.cmml">…</mi><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.6" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml">,</mo><msub id="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.2" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.2.cmml">d</mi><mi id="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.3" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.3.cmml">k</mi></msub><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.7" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml">,</mo><mi id="S4.E2.m1.1.1.1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.1.1.2.cmml">𝐪</mi><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.8" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml">,</mo><mi id="S4.E2.m1.1.1.1.1.1.1.3" mathvariant="normal" xref="S4.E2.m1.1.1.1.1.1.1.3.cmml">…</mi><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.9" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml">,</mo><msub id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.2" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.2.cmml">y</mi><mrow id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.2" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.2.cmml">i</mi><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.1" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.1.cmml">−</mo><mn id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.3" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.3.cmml">1</mn></mrow></msub><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.10" stretchy="false" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml">}</mo></mrow></mrow><mo id="S4.E2.m1.1.1.1.1.1.1.4.1.3" stretchy="false" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E2.m1.4.4.4c" xref="S4.E2.m1.4.5.3.1.cmml"><mrow id="S4.E2.m1.2.2.2.2.2.1" xref="S4.E2.m1.2.2.2.2.2.1.cmml"><mrow id="S4.E2.m1.2.2.2.2.2.1.1" xref="S4.E2.m1.2.2.2.2.2.1.1.cmml"><mtext id="S4.E2.m1.2.2.2.2.2.1.1.3" xref="S4.E2.m1.2.2.2.2.2.1.1.3a.cmml">if</mtext><mo id="S4.E2.m1.2.2.2.2.2.1.1.2" lspace="0.330em" xref="S4.E2.m1.2.2.2.2.2.1.1.2.cmml">⁢</mo><mi id="S4.E2.m1.2.2.2.2.2.1.1.4" xref="S4.E2.m1.2.2.2.2.2.1.1.4.cmml">𝐔</mi><mo id="S4.E2.m1.2.2.2.2.2.1.1.2a" xref="S4.E2.m1.2.2.2.2.2.1.1.2.cmml">⁢</mo><mrow id="S4.E2.m1.2.2.2.2.2.1.1.1.1" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.cmml"><mo id="S4.E2.m1.2.2.2.2.2.1.1.1.1.2" stretchy="false" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.cmml">(</mo><msub id="S4.E2.m1.2.2.2.2.2.1.1.1.1.1" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.cmml"><mi id="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.2" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.2.cmml">t</mi><mi id="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.3" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S4.E2.m1.2.2.2.2.2.1.1.1.1.3" stretchy="false" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E2.m1.2.2.2.2.2.1.2" xref="S4.E2.m1.2.2.2.2.2.1.2.cmml">></mo><msub id="S4.E2.m1.2.2.2.2.2.1.3" xref="S4.E2.m1.2.2.2.2.2.1.3.cmml"><mi id="S4.E2.m1.2.2.2.2.2.1.3.2" xref="S4.E2.m1.2.2.2.2.2.1.3.2.cmml">θ</mi><mi id="S4.E2.m1.2.2.2.2.2.1.3.3" xref="S4.E2.m1.2.2.2.2.2.1.3.3.cmml">𝐔</mi></msub></mrow></mtd></mtr><mtr id="S4.E2.m1.4.4.4d" xref="S4.E2.m1.4.5.3.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="S4.E2.m1.4.4.4e" xref="S4.E2.m1.4.5.3.1.cmml"><mrow id="S4.E2.m1.3.3.3.3.1.1" xref="S4.E2.m1.3.3.3.3.1.1.cmml"><mi id="S4.E2.m1.3.3.3.3.1.1.5" xref="S4.E2.m1.3.3.3.3.1.1.5.cmml">𝐆</mi><mo id="S4.E2.m1.3.3.3.3.1.1.4" xref="S4.E2.m1.3.3.3.3.1.1.4.cmml">⁢</mo><mrow id="S4.E2.m1.3.3.3.3.1.1.3.1" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.cmml"><mo id="S4.E2.m1.3.3.3.3.1.1.3.1.2" stretchy="false" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.cmml">(</mo><mrow id="S4.E2.m1.3.3.3.3.1.1.3.1.1" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.cmml"><mi id="S4.E2.m1.3.3.3.3.1.1.3.1.1.3" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.3.cmml">𝐏</mi><mo id="S4.E2.m1.3.3.3.3.1.1.3.1.1.2" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.2.cmml">⁢</mo><mrow id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.2.cmml"><mo id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.2" stretchy="false" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.2.cmml">{</mo><mi id="S4.E2.m1.3.3.3.3.1.1.1" xref="S4.E2.m1.3.3.3.3.1.1.1.cmml">𝐪</mi><mo id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.3" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.2.cmml">,</mo><mi id="S4.E2.m1.3.3.3.3.1.1.2" mathvariant="normal" xref="S4.E2.m1.3.3.3.3.1.1.2.cmml">…</mi><mo id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.4" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.2.cmml">,</mo><msub id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.cmml"><mi id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.2" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.2.cmml">y</mi><mrow id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.cmml"><mi id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.2" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.2.cmml">i</mi><mo id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.1" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.1.cmml">−</mo><mn id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.3" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.5" stretchy="false" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.2.cmml">}</mo></mrow></mrow><mo id="S4.E2.m1.3.3.3.3.1.1.3.1.3" stretchy="false" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E2.m1.4.4.4f" xref="S4.E2.m1.4.5.3.1.cmml"><mrow id="S4.E2.m1.4.4.4.4.2.1" xref="S4.E2.m1.4.4.4.4.2.1.cmml"><mi id="S4.E2.m1.4.4.4.4.2.1.2" xref="S4.E2.m1.4.4.4.4.2.1.2.cmml">o</mi><mo id="S4.E2.m1.4.4.4.4.2.1.1" xref="S4.E2.m1.4.4.4.4.2.1.1.cmml">⁢</mo><mi id="S4.E2.m1.4.4.4.4.2.1.3" xref="S4.E2.m1.4.4.4.4.2.1.3.cmml">t</mi><mo id="S4.E2.m1.4.4.4.4.2.1.1a" xref="S4.E2.m1.4.4.4.4.2.1.1.cmml">⁢</mo><mi id="S4.E2.m1.4.4.4.4.2.1.4" xref="S4.E2.m1.4.4.4.4.2.1.4.cmml">h</mi><mo id="S4.E2.m1.4.4.4.4.2.1.1b" xref="S4.E2.m1.4.4.4.4.2.1.1.cmml">⁢</mo><mi id="S4.E2.m1.4.4.4.4.2.1.5" xref="S4.E2.m1.4.4.4.4.2.1.5.cmml">e</mi><mo id="S4.E2.m1.4.4.4.4.2.1.1c" xref="S4.E2.m1.4.4.4.4.2.1.1.cmml">⁢</mo><mi id="S4.E2.m1.4.4.4.4.2.1.6" xref="S4.E2.m1.4.4.4.4.2.1.6.cmml">r</mi><mo id="S4.E2.m1.4.4.4.4.2.1.1d" xref="S4.E2.m1.4.4.4.4.2.1.1.cmml">⁢</mo><mi id="S4.E2.m1.4.4.4.4.2.1.7" xref="S4.E2.m1.4.4.4.4.2.1.7.cmml">w</mi><mo id="S4.E2.m1.4.4.4.4.2.1.1e" xref="S4.E2.m1.4.4.4.4.2.1.1.cmml">⁢</mo><mi id="S4.E2.m1.4.4.4.4.2.1.8" xref="S4.E2.m1.4.4.4.4.2.1.8.cmml">i</mi><mo id="S4.E2.m1.4.4.4.4.2.1.1f" xref="S4.E2.m1.4.4.4.4.2.1.1.cmml">⁢</mo><mi id="S4.E2.m1.4.4.4.4.2.1.9" xref="S4.E2.m1.4.4.4.4.2.1.9.cmml">s</mi><mo id="S4.E2.m1.4.4.4.4.2.1.1g" xref="S4.E2.m1.4.4.4.4.2.1.1.cmml">⁢</mo><mi id="S4.E2.m1.4.4.4.4.2.1.10" xref="S4.E2.m1.4.4.4.4.2.1.10.cmml">e</mi></mrow></mtd></mtr></mtable></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E2.m1.4b"><apply id="S4.E2.m1.4.5.cmml" xref="S4.E2.m1.4.5"><eq id="S4.E2.m1.4.5.1.cmml" xref="S4.E2.m1.4.5.1"></eq><apply id="S4.E2.m1.4.5.2.cmml" xref="S4.E2.m1.4.5.2"><csymbol cd="ambiguous" id="S4.E2.m1.4.5.2.1.cmml" xref="S4.E2.m1.4.5.2">subscript</csymbol><ci id="S4.E2.m1.4.5.2.2.cmml" xref="S4.E2.m1.4.5.2.2">𝑦</ci><ci id="S4.E2.m1.4.5.2.3.cmml" xref="S4.E2.m1.4.5.2.3">𝑖</ci></apply><apply id="S4.E2.m1.4.5.3.1.cmml" xref="S4.E2.m1.4.4"><csymbol cd="latexml" id="S4.E2.m1.4.5.3.1.1.cmml" xref="S4.E2.m1.4.4.5">cases</csymbol><apply id="S4.E2.m1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1"><times id="S4.E2.m1.1.1.1.1.1.1.5.cmml" xref="S4.E2.m1.1.1.1.1.1.1.5"></times><ci id="S4.E2.m1.1.1.1.1.1.1.6.cmml" xref="S4.E2.m1.1.1.1.1.1.1.6">𝐆</ci><apply id="S4.E2.m1.1.1.1.1.1.1.4.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1"><times id="S4.E2.m1.1.1.1.1.1.1.4.1.1.4.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.4"></times><ci id="S4.E2.m1.1.1.1.1.1.1.4.1.1.5.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.5">𝐏</ci><set id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.4.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3"><apply id="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.2">𝑑</ci><cn id="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.3.cmml" type="integer" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.1.1.1.3">1</cn></apply><ci id="S4.E2.m1.1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1">…</ci><apply id="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.2">𝑑</ci><ci id="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.2.2.2.3">𝑘</ci></apply><ci id="S4.E2.m1.1.1.1.1.1.1.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.2">𝐪</ci><ci id="S4.E2.m1.1.1.1.1.1.1.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3">…</ci><apply id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.2">𝑦</ci><apply id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3"><minus id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.1"></minus><ci id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.2">𝑖</ci><cn id="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.3.cmml" type="integer" xref="S4.E2.m1.1.1.1.1.1.1.4.1.1.3.3.3.3.3">1</cn></apply></apply></set></apply></apply><apply id="S4.E2.m1.2.2.2.2.2.1.cmml" xref="S4.E2.m1.2.2.2.2.2.1"><gt id="S4.E2.m1.2.2.2.2.2.1.2.cmml" xref="S4.E2.m1.2.2.2.2.2.1.2"></gt><apply id="S4.E2.m1.2.2.2.2.2.1.1.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1"><times id="S4.E2.m1.2.2.2.2.2.1.1.2.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1.2"></times><ci id="S4.E2.m1.2.2.2.2.2.1.1.3a.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1.3"><mtext id="S4.E2.m1.2.2.2.2.2.1.1.3.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1.3">if</mtext></ci><ci id="S4.E2.m1.2.2.2.2.2.1.1.4.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1.4">𝐔</ci><apply id="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.1.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1">subscript</csymbol><ci id="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.2.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.2">𝑡</ci><ci id="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.3.cmml" xref="S4.E2.m1.2.2.2.2.2.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S4.E2.m1.2.2.2.2.2.1.3.cmml" xref="S4.E2.m1.2.2.2.2.2.1.3"><csymbol cd="ambiguous" id="S4.E2.m1.2.2.2.2.2.1.3.1.cmml" xref="S4.E2.m1.2.2.2.2.2.1.3">subscript</csymbol><ci id="S4.E2.m1.2.2.2.2.2.1.3.2.cmml" xref="S4.E2.m1.2.2.2.2.2.1.3.2">𝜃</ci><ci id="S4.E2.m1.2.2.2.2.2.1.3.3.cmml" xref="S4.E2.m1.2.2.2.2.2.1.3.3">𝐔</ci></apply></apply><apply id="S4.E2.m1.3.3.3.3.1.1.cmml" xref="S4.E2.m1.3.3.3.3.1.1"><times id="S4.E2.m1.3.3.3.3.1.1.4.cmml" xref="S4.E2.m1.3.3.3.3.1.1.4"></times><ci id="S4.E2.m1.3.3.3.3.1.1.5.cmml" xref="S4.E2.m1.3.3.3.3.1.1.5">𝐆</ci><apply id="S4.E2.m1.3.3.3.3.1.1.3.1.1.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1"><times id="S4.E2.m1.3.3.3.3.1.1.3.1.1.2.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.2"></times><ci id="S4.E2.m1.3.3.3.3.1.1.3.1.1.3.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.3">𝐏</ci><set id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.2.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1"><ci id="S4.E2.m1.3.3.3.3.1.1.1.cmml" xref="S4.E2.m1.3.3.3.3.1.1.1">𝐪</ci><ci id="S4.E2.m1.3.3.3.3.1.1.2.cmml" xref="S4.E2.m1.3.3.3.3.1.1.2">…</ci><apply id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.1.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1">subscript</csymbol><ci id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.2.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.2">𝑦</ci><apply id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3"><minus id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.1.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.1"></minus><ci id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.2.cmml" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.2">𝑖</ci><cn id="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.3.cmml" type="integer" xref="S4.E2.m1.3.3.3.3.1.1.3.1.1.1.1.1.3.3">1</cn></apply></apply></set></apply></apply><apply id="S4.E2.m1.4.4.4.4.2.1.cmml" xref="S4.E2.m1.4.4.4.4.2.1"><times id="S4.E2.m1.4.4.4.4.2.1.1.cmml" xref="S4.E2.m1.4.4.4.4.2.1.1"></times><ci id="S4.E2.m1.4.4.4.4.2.1.2.cmml" xref="S4.E2.m1.4.4.4.4.2.1.2">𝑜</ci><ci id="S4.E2.m1.4.4.4.4.2.1.3.cmml" xref="S4.E2.m1.4.4.4.4.2.1.3">𝑡</ci><ci id="S4.E2.m1.4.4.4.4.2.1.4.cmml" xref="S4.E2.m1.4.4.4.4.2.1.4">ℎ</ci><ci id="S4.E2.m1.4.4.4.4.2.1.5.cmml" xref="S4.E2.m1.4.4.4.4.2.1.5">𝑒</ci><ci id="S4.E2.m1.4.4.4.4.2.1.6.cmml" xref="S4.E2.m1.4.4.4.4.2.1.6">𝑟</ci><ci id="S4.E2.m1.4.4.4.4.2.1.7.cmml" xref="S4.E2.m1.4.4.4.4.2.1.7">𝑤</ci><ci id="S4.E2.m1.4.4.4.4.2.1.8.cmml" xref="S4.E2.m1.4.4.4.4.2.1.8">𝑖</ci><ci id="S4.E2.m1.4.4.4.4.2.1.9.cmml" xref="S4.E2.m1.4.4.4.4.2.1.9">𝑠</ci><ci id="S4.E2.m1.4.4.4.4.2.1.10.cmml" xref="S4.E2.m1.4.4.4.4.2.1.10">𝑒</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E2.m1.4c">y_{i}=\begin{cases}\mathbf{G}(\mathbf{P}\{d_{1},\ldots,d_{k},\mathbf{q},\ldots% ,y_{i-1}\})&\text{if}~{}\mathbf{U}(t_{i})>\theta_{\mathbf{U}}\\ \mathbf{G}(\mathbf{P}\{\mathbf{q},\ldots,y_{i-1}\})&otherwise\end{cases}</annotation><annotation encoding="application/x-llamapun" id="S4.E2.m1.4d">italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = { start_ROW start_CELL bold_G ( bold_P { italic_d start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , italic_d start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT , bold_q , … , italic_y start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT } ) end_CELL start_CELL if bold_U ( italic_t start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) > italic_θ start_POSTSUBSCRIPT bold_U end_POSTSUBSCRIPT end_CELL end_ROW start_ROW start_CELL bold_G ( bold_P { bold_q , … , italic_y start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT } ) end_CELL start_CELL italic_o italic_t italic_h italic_e italic_r italic_w italic_i italic_s italic_e end_CELL end_ROW</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td> </tr></tbody> </table> <p class="ltx_p" id="S4.SS1.p3.6">where <math alttext="d_{1}\ldots d_{k}" class="ltx_Math" display="inline" id="S4.SS1.p3.5.m1.1"><semantics id="S4.SS1.p3.5.m1.1a"><mrow id="S4.SS1.p3.5.m1.1.1" xref="S4.SS1.p3.5.m1.1.1.cmml"><msub id="S4.SS1.p3.5.m1.1.1.2" xref="S4.SS1.p3.5.m1.1.1.2.cmml"><mi id="S4.SS1.p3.5.m1.1.1.2.2" xref="S4.SS1.p3.5.m1.1.1.2.2.cmml">d</mi><mn id="S4.SS1.p3.5.m1.1.1.2.3" xref="S4.SS1.p3.5.m1.1.1.2.3.cmml">1</mn></msub><mo id="S4.SS1.p3.5.m1.1.1.1" xref="S4.SS1.p3.5.m1.1.1.1.cmml">⁢</mo><mi id="S4.SS1.p3.5.m1.1.1.3" mathvariant="normal" xref="S4.SS1.p3.5.m1.1.1.3.cmml">…</mi><mo id="S4.SS1.p3.5.m1.1.1.1a" xref="S4.SS1.p3.5.m1.1.1.1.cmml">⁢</mo><msub id="S4.SS1.p3.5.m1.1.1.4" xref="S4.SS1.p3.5.m1.1.1.4.cmml"><mi id="S4.SS1.p3.5.m1.1.1.4.2" xref="S4.SS1.p3.5.m1.1.1.4.2.cmml">d</mi><mi id="S4.SS1.p3.5.m1.1.1.4.3" xref="S4.SS1.p3.5.m1.1.1.4.3.cmml">k</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.5.m1.1b"><apply id="S4.SS1.p3.5.m1.1.1.cmml" xref="S4.SS1.p3.5.m1.1.1"><times id="S4.SS1.p3.5.m1.1.1.1.cmml" xref="S4.SS1.p3.5.m1.1.1.1"></times><apply id="S4.SS1.p3.5.m1.1.1.2.cmml" xref="S4.SS1.p3.5.m1.1.1.2"><csymbol cd="ambiguous" id="S4.SS1.p3.5.m1.1.1.2.1.cmml" xref="S4.SS1.p3.5.m1.1.1.2">subscript</csymbol><ci id="S4.SS1.p3.5.m1.1.1.2.2.cmml" xref="S4.SS1.p3.5.m1.1.1.2.2">𝑑</ci><cn id="S4.SS1.p3.5.m1.1.1.2.3.cmml" type="integer" xref="S4.SS1.p3.5.m1.1.1.2.3">1</cn></apply><ci id="S4.SS1.p3.5.m1.1.1.3.cmml" xref="S4.SS1.p3.5.m1.1.1.3">…</ci><apply id="S4.SS1.p3.5.m1.1.1.4.cmml" xref="S4.SS1.p3.5.m1.1.1.4"><csymbol cd="ambiguous" id="S4.SS1.p3.5.m1.1.1.4.1.cmml" xref="S4.SS1.p3.5.m1.1.1.4">subscript</csymbol><ci id="S4.SS1.p3.5.m1.1.1.4.2.cmml" xref="S4.SS1.p3.5.m1.1.1.4.2">𝑑</ci><ci id="S4.SS1.p3.5.m1.1.1.4.3.cmml" xref="S4.SS1.p3.5.m1.1.1.4.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.5.m1.1c">d_{1}\ldots d_{k}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p3.5.m1.1d">italic_d start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT … italic_d start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT</annotation></semantics></math> are obtained from a retrieval system <math alttext="\mathbf{R}" class="ltx_Math" display="inline" id="S4.SS1.p3.6.m2.1"><semantics id="S4.SS1.p3.6.m2.1a"><mi id="S4.SS1.p3.6.m2.1.1" xref="S4.SS1.p3.6.m2.1.1.cmml">𝐑</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.6.m2.1b"><ci id="S4.SS1.p3.6.m2.1.1.cmml" xref="S4.SS1.p3.6.m2.1.1">𝐑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.6.m2.1c">\mathbf{R}</annotation><annotation encoding="application/x-llamapun" id="S4.SS1.p3.6.m2.1d">bold_R</annotation></semantics></math>.</p> <table class="ltx_equation ltx_eqn_table" id="S4.E3"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="d_{1}\ldots d_{k}:=\mathbf{R}(\mathbf{q})" class="ltx_Math" display="block" id="S4.E3.m1.1"><semantics id="S4.E3.m1.1a"><mrow id="S4.E3.m1.1.2" xref="S4.E3.m1.1.2.cmml"><mrow id="S4.E3.m1.1.2.2" xref="S4.E3.m1.1.2.2.cmml"><msub id="S4.E3.m1.1.2.2.2" xref="S4.E3.m1.1.2.2.2.cmml"><mi id="S4.E3.m1.1.2.2.2.2" xref="S4.E3.m1.1.2.2.2.2.cmml">d</mi><mn id="S4.E3.m1.1.2.2.2.3" xref="S4.E3.m1.1.2.2.2.3.cmml">1</mn></msub><mo id="S4.E3.m1.1.2.2.1" xref="S4.E3.m1.1.2.2.1.cmml">⁢</mo><mi id="S4.E3.m1.1.2.2.3" mathvariant="normal" xref="S4.E3.m1.1.2.2.3.cmml">…</mi><mo id="S4.E3.m1.1.2.2.1a" xref="S4.E3.m1.1.2.2.1.cmml">⁢</mo><msub id="S4.E3.m1.1.2.2.4" xref="S4.E3.m1.1.2.2.4.cmml"><mi id="S4.E3.m1.1.2.2.4.2" xref="S4.E3.m1.1.2.2.4.2.cmml">d</mi><mi id="S4.E3.m1.1.2.2.4.3" xref="S4.E3.m1.1.2.2.4.3.cmml">k</mi></msub></mrow><mo id="S4.E3.m1.1.2.1" lspace="0.278em" rspace="0.278em" xref="S4.E3.m1.1.2.1.cmml">:=</mo><mrow id="S4.E3.m1.1.2.3" xref="S4.E3.m1.1.2.3.cmml"><mi id="S4.E3.m1.1.2.3.2" xref="S4.E3.m1.1.2.3.2.cmml">𝐑</mi><mo id="S4.E3.m1.1.2.3.1" xref="S4.E3.m1.1.2.3.1.cmml">⁢</mo><mrow id="S4.E3.m1.1.2.3.3.2" xref="S4.E3.m1.1.2.3.cmml"><mo id="S4.E3.m1.1.2.3.3.2.1" stretchy="false" xref="S4.E3.m1.1.2.3.cmml">(</mo><mi id="S4.E3.m1.1.1" xref="S4.E3.m1.1.1.cmml">𝐪</mi><mo id="S4.E3.m1.1.2.3.3.2.2" stretchy="false" xref="S4.E3.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E3.m1.1b"><apply id="S4.E3.m1.1.2.cmml" xref="S4.E3.m1.1.2"><csymbol cd="latexml" id="S4.E3.m1.1.2.1.cmml" xref="S4.E3.m1.1.2.1">assign</csymbol><apply id="S4.E3.m1.1.2.2.cmml" xref="S4.E3.m1.1.2.2"><times id="S4.E3.m1.1.2.2.1.cmml" xref="S4.E3.m1.1.2.2.1"></times><apply id="S4.E3.m1.1.2.2.2.cmml" xref="S4.E3.m1.1.2.2.2"><csymbol cd="ambiguous" id="S4.E3.m1.1.2.2.2.1.cmml" xref="S4.E3.m1.1.2.2.2">subscript</csymbol><ci id="S4.E3.m1.1.2.2.2.2.cmml" xref="S4.E3.m1.1.2.2.2.2">𝑑</ci><cn id="S4.E3.m1.1.2.2.2.3.cmml" type="integer" xref="S4.E3.m1.1.2.2.2.3">1</cn></apply><ci id="S4.E3.m1.1.2.2.3.cmml" xref="S4.E3.m1.1.2.2.3">…</ci><apply id="S4.E3.m1.1.2.2.4.cmml" xref="S4.E3.m1.1.2.2.4"><csymbol cd="ambiguous" id="S4.E3.m1.1.2.2.4.1.cmml" xref="S4.E3.m1.1.2.2.4">subscript</csymbol><ci id="S4.E3.m1.1.2.2.4.2.cmml" xref="S4.E3.m1.1.2.2.4.2">𝑑</ci><ci id="S4.E3.m1.1.2.2.4.3.cmml" xref="S4.E3.m1.1.2.2.4.3">𝑘</ci></apply></apply><apply id="S4.E3.m1.1.2.3.cmml" xref="S4.E3.m1.1.2.3"><times id="S4.E3.m1.1.2.3.1.cmml" xref="S4.E3.m1.1.2.3.1"></times><ci id="S4.E3.m1.1.2.3.2.cmml" xref="S4.E3.m1.1.2.3.2">𝐑</ci><ci id="S4.E3.m1.1.1.cmml" xref="S4.E3.m1.1.1">𝐪</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E3.m1.1c">d_{1}\ldots d_{k}:=\mathbf{R}(\mathbf{q})</annotation><annotation encoding="application/x-llamapun" id="S4.E3.m1.1d">italic_d start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT … italic_d start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT := bold_R ( bold_q )</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td> </tr></tbody> </table> </div> </section> <section class="ltx_subsection" id="S4.SS2"> <h3 class="ltx_title ltx_title_subsection"> <span class="ltx_tag ltx_tag_subsection">4.2 </span>Sequence Level Uncertainty Evaluation Measures</h3> <div class="ltx_para ltx_noindent" id="S4.SS2.p1"> <p class="ltx_p" id="S4.SS2.p1.1">We resort to 5 recently introduced sequence-level uncertainty evaluation measures. Each of them work in a black box manner without requiring information regarding the model parameters.</p> </div> <div class="ltx_para" id="S4.SS2.p2"> <p class="ltx_p" id="S4.SS2.p2.5">The high-level strategy of all the methods is the same. Given an input <math alttext="x" class="ltx_Math" display="inline" id="S4.SS2.p2.1.m1.1"><semantics id="S4.SS2.p2.1.m1.1a"><mi id="S4.SS2.p2.1.m1.1.1" xref="S4.SS2.p2.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.1.m1.1b"><ci id="S4.SS2.p2.1.m1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.1.m1.1c">x</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.p2.1.m1.1d">italic_x</annotation></semantics></math>, first generate <math alttext="n" class="ltx_Math" display="inline" id="S4.SS2.p2.2.m2.1"><semantics id="S4.SS2.p2.2.m2.1a"><mi id="S4.SS2.p2.2.m2.1.1" xref="S4.SS2.p2.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.2.m2.1b"><ci id="S4.SS2.p2.2.m2.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.2.m2.1c">n</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.p2.2.m2.1d">italic_n</annotation></semantics></math> responses through some generator <math alttext="G" class="ltx_Math" display="inline" id="S4.SS2.p2.3.m3.1"><semantics id="S4.SS2.p2.3.m3.1a"><mi id="S4.SS2.p2.3.m3.1.1" xref="S4.SS2.p2.3.m3.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.3.m3.1b"><ci id="S4.SS2.p2.3.m3.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.3.m3.1c">G</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.p2.3.m3.1d">italic_G</annotation></semantics></math> and then compute pairwise similarity scores of each of the <math alttext="n" class="ltx_Math" display="inline" id="S4.SS2.p2.4.m4.1"><semantics id="S4.SS2.p2.4.m4.1a"><mi id="S4.SS2.p2.4.m4.1.1" xref="S4.SS2.p2.4.m4.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.4.m4.1b"><ci id="S4.SS2.p2.4.m4.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.4.m4.1c">n</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.p2.4.m4.1d">italic_n</annotation></semantics></math> responses with each other. Using these similarity values, compute an uncertainty estimate <math alttext="U(x)" class="ltx_Math" display="inline" id="S4.SS2.p2.5.m5.1"><semantics id="S4.SS2.p2.5.m5.1a"><mrow id="S4.SS2.p2.5.m5.1.2" xref="S4.SS2.p2.5.m5.1.2.cmml"><mi id="S4.SS2.p2.5.m5.1.2.2" xref="S4.SS2.p2.5.m5.1.2.2.cmml">U</mi><mo id="S4.SS2.p2.5.m5.1.2.1" xref="S4.SS2.p2.5.m5.1.2.1.cmml">⁢</mo><mrow id="S4.SS2.p2.5.m5.1.2.3.2" xref="S4.SS2.p2.5.m5.1.2.cmml"><mo id="S4.SS2.p2.5.m5.1.2.3.2.1" stretchy="false" xref="S4.SS2.p2.5.m5.1.2.cmml">(</mo><mi id="S4.SS2.p2.5.m5.1.1" xref="S4.SS2.p2.5.m5.1.1.cmml">x</mi><mo id="S4.SS2.p2.5.m5.1.2.3.2.2" stretchy="false" xref="S4.SS2.p2.5.m5.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.5.m5.1b"><apply id="S4.SS2.p2.5.m5.1.2.cmml" xref="S4.SS2.p2.5.m5.1.2"><times id="S4.SS2.p2.5.m5.1.2.1.cmml" xref="S4.SS2.p2.5.m5.1.2.1"></times><ci id="S4.SS2.p2.5.m5.1.2.2.cmml" xref="S4.SS2.p2.5.m5.1.2.2">𝑈</ci><ci id="S4.SS2.p2.5.m5.1.1.cmml" xref="S4.SS2.p2.5.m5.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.5.m5.1c">U(x)</annotation><annotation encoding="application/x-llamapun" id="S4.SS2.p2.5.m5.1d">italic_U ( italic_x )</annotation></semantics></math> or a confidence score.</p> <ul class="ltx_itemize" id="S4.I1"> <li class="ltx_item" id="S4.I1.i1" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S4.I1.i1.p1"> <p class="ltx_p" id="S4.I1.i1.p1.1"><span class="ltx_text ltx_font_bold" id="S4.I1.i1.p1.1.1">Semantic Sets</span>: In the black-box approach of <cite class="ltx_cite ltx_citemacro_cite">Kuhn et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib11" title="">2023</a>)</cite>, the authors propose to compute semantic sets i.e. groups of responses that are close together in meaning. These semantic sets of equivalence subsets are computed using a Natural Language Inference (NLI) classifier. Here, the number of semantic sets can be regarded as an uncertainty estimate as when the responses differ in meaning, the number of groups increases.</p> </div> </li> <li class="ltx_item" id="S4.I1.i2" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S4.I1.i2.p1"> <p class="ltx_p" id="S4.I1.i2.p1.1"><span class="ltx_text ltx_font_bold" id="S4.I1.i2.p1.1.1">Eigen Value Laplacian</span>: defines the uncertainty estimate by capturing the essence of spectral clustering. First, an adjacency matrix is created from the pairwise similarities of responses. Then the matrix is partitioned into clusters, where each cluster corresponds to a distinct “meaning” or category within the responses. The eigenvalues close to one indicate strong cluster formations, thus contributing less to the uncertainty estimate, while those further from one suggest weaker clustering or more diffuse distributions of responses, hence increasing the uncertainty estimate. <br class="ltx_break"/>The degree matrix of the adjacency graph is also used to compute the uncertainty estimate <cite class="ltx_cite ltx_citemacro_cite">Lin et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib13" title="">2023</a>)</cite>. A node that is well-connected to other nodes, might be less uncertain. We use two similarity metrics for computing the degree matrix.</p> </div> </li> <li class="ltx_item" id="S4.I1.i3" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para" id="S4.I1.i3.p1"> <p class="ltx_p" id="S4.I1.i3.p1.1"><span class="ltx_text ltx_font_bold" id="S4.I1.i3.p1.1.1">Degree Matrix (Jaccard Index)</span>: The Jaccard similarity is a light-weight metric where sentences or passages are treated as sets of words, and similarity between responses is computed by taking the fraction of the intersection of the two sets and the union of the two sets.</p> </div> </li> <li class="ltx_item" id="S4.I1.i4" style="list-style-type:none;"> <span class="ltx_tag ltx_tag_item">•</span> <div class="ltx_para ltx_noindent" id="S4.I1.i4.p1"> <p class="ltx_p" id="S4.I1.i4.p1.1"><span class="ltx_text ltx_font_bold" id="S4.I1.i4.p1.1.1">Degree Matrix (NLI)</span>: Here, the similarity between responses is computed through classifying entailment relations amongst them. A classifier predicts whether a pair of responses contradict, entail, or are neutral to each other.</p> </div> </li> </ul> </div> <figure class="ltx_table" id="S4.T1"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S4.T1.5" style="width:397.5pt;height:110.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-59.5pt,16.6pt) scale(0.76949518994168,0.76949518994168) ;"> <table class="ltx_tabular ltx_align_middle" id="S4.T1.5.5"> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="S4.T1.5.5.6.1"> <td class="ltx_td ltx_align_left ltx_border_tt" id="S4.T1.5.5.6.1.1"><span class="ltx_text ltx_font_bold" id="S4.T1.5.5.6.1.1.1">Uncertainty Estimator</span></td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" id="S4.T1.5.5.6.1.2"><span class="ltx_text ltx_font_bold" id="S4.T1.5.5.6.1.2.1">Trigger Retrieval When</span></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S4.T1.5.5.6.1.3"><span class="ltx_text ltx_font_bold" id="S4.T1.5.5.6.1.3.1">Retrieval Query</span></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S4.T1.5.5.6.1.4"><span class="ltx_text ltx_font_bold" id="S4.T1.5.5.6.1.4.1">#examples</span></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S4.T1.5.5.6.1.5"><span class="ltx_text ltx_font_bold" id="S4.T1.5.5.6.1.5.1">#search</span></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S4.T1.5.5.6.1.6"><span class="ltx_text ltx_font_bold" id="S4.T1.5.5.6.1.6.1">#steps</span></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S4.T1.5.5.6.1.7"><span class="ltx_text ltx_font_bold" id="S4.T1.5.5.6.1.7.1">f1</span></td> </tr> <tr class="ltx_tr" id="S4.T1.1.1.1"> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T1.1.1.1.2">Always Retrieve</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S4.T1.1.1.1.1">U <math alttext="\geq" class="ltx_Math" display="inline" id="S4.T1.1.1.1.1.m1.1"><semantics id="S4.T1.1.1.1.1.m1.1a"><mo id="S4.T1.1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.1.m1.1.1.cmml">≥</mo><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.1.m1.1b"><geq id="S4.T1.1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.1.m1.1.1"></geq></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.1.m1.1c">\geq</annotation><annotation encoding="application/x-llamapun" id="S4.T1.1.1.1.1.m1.1d">≥</annotation></semantics></math> 0</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.1.1.1.3">Temporary Sentence</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.1.1.1.4">25</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.1.1.1.5">4.60</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.1.1.1.6">3.60</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.1.1.1.7">0.552</td> </tr> <tr class="ltx_tr" id="S4.T1.5.5.7.2"> <td class="ltx_td ltx_align_left" id="S4.T1.5.5.7.2.1">Always Retrieve</td> <td class="ltx_td ltx_border_r" id="S4.T1.5.5.7.2.2"></td> <td class="ltx_td ltx_align_center" id="S4.T1.5.5.7.2.3">Sub-Query</td> <td class="ltx_td ltx_align_center" id="S4.T1.5.5.7.2.4">25</td> <td class="ltx_td ltx_align_center" id="S4.T1.5.5.7.2.5">5.00</td> <td class="ltx_td ltx_align_center" id="S4.T1.5.5.7.2.6">4.00</td> <td class="ltx_td ltx_align_center" id="S4.T1.5.5.7.2.7">0.538</td> </tr> <tr class="ltx_tr" id="S4.T1.5.5.8.3"> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T1.5.5.8.3.1">FLARE-Instruct</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S4.T1.5.5.8.3.2">“…[Search”</td> <td class="ltx_td ltx_border_t" id="S4.T1.5.5.8.3.3"></td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.5.5.8.3.4">25</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.5.5.8.3.5">4.80</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.5.5.8.3.6">3.80</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.5.5.8.3.7">0.531</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.2"> <td class="ltx_td ltx_align_left" id="S4.T1.2.2.2.2">Degree Matrix Jaccard</td> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T1.2.2.2.1">U <math alttext=">" class="ltx_Math" display="inline" id="S4.T1.2.2.2.1.m1.1"><semantics id="S4.T1.2.2.2.1.m1.1a"><mo id="S4.T1.2.2.2.1.m1.1.1" xref="S4.T1.2.2.2.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.2.1.m1.1b"><gt id="S4.T1.2.2.2.1.m1.1.1.cmml" xref="S4.T1.2.2.2.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.2.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S4.T1.2.2.2.1.m1.1d">></annotation></semantics></math> 0.4</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.2.3">Sub-Query</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.2.4">24</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.2.5">1.46</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.2.6">3.67</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.2.7">0.593</td> </tr> <tr class="ltx_tr" id="S4.T1.3.3.3"> <td class="ltx_td ltx_align_left" id="S4.T1.3.3.3.2">Eccentricity</td> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T1.3.3.3.1">U <math alttext=">" class="ltx_Math" display="inline" id="S4.T1.3.3.3.1.m1.1"><semantics id="S4.T1.3.3.3.1.m1.1a"><mo id="S4.T1.3.3.3.1.m1.1.1" xref="S4.T1.3.3.3.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S4.T1.3.3.3.1.m1.1b"><gt id="S4.T1.3.3.3.1.m1.1.1.cmml" xref="S4.T1.3.3.3.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.3.3.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S4.T1.3.3.3.1.m1.1d">></annotation></semantics></math> 2</td> <td class="ltx_td ltx_align_center" id="S4.T1.3.3.3.3">Sub-Query</td> <td class="ltx_td ltx_align_center" id="S4.T1.3.3.3.4">22</td> <td class="ltx_td ltx_align_center" id="S4.T1.3.3.3.5">2.23</td> <td class="ltx_td ltx_align_center" id="S4.T1.3.3.3.6">4.05</td> <td class="ltx_td ltx_align_center" id="S4.T1.3.3.3.7"> <span class="ltx_text ltx_font_bold" id="S4.T1.3.3.3.7.1">0.605</span> </td> </tr> <tr class="ltx_tr" id="S4.T1.4.4.4"> <td class="ltx_td ltx_align_left" id="S4.T1.4.4.4.2">Semantic Sets</td> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T1.4.4.4.1">U <math alttext=">" class="ltx_Math" display="inline" id="S4.T1.4.4.4.1.m1.1"><semantics id="S4.T1.4.4.4.1.m1.1a"><mo id="S4.T1.4.4.4.1.m1.1.1" xref="S4.T1.4.4.4.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S4.T1.4.4.4.1.m1.1b"><gt id="S4.T1.4.4.4.1.m1.1.1.cmml" xref="S4.T1.4.4.4.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.4.4.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S4.T1.4.4.4.1.m1.1d">></annotation></semantics></math> 2</td> <td class="ltx_td ltx_align_center" id="S4.T1.4.4.4.3">Sub-Query</td> <td class="ltx_td ltx_align_center" id="S4.T1.4.4.4.4">23</td> <td class="ltx_td ltx_align_center" id="S4.T1.4.4.4.5">2.52</td> <td class="ltx_td ltx_align_center" id="S4.T1.4.4.4.6">4.09</td> <td class="ltx_td ltx_align_center" id="S4.T1.4.4.4.7">0.411</td> </tr> <tr class="ltx_tr" id="S4.T1.5.5.5"> <td class="ltx_td ltx_align_left ltx_border_bb" id="S4.T1.5.5.5.2">Degree Matrix NLI</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" id="S4.T1.5.5.5.1">U <math alttext=">" class="ltx_Math" display="inline" id="S4.T1.5.5.5.1.m1.1"><semantics id="S4.T1.5.5.5.1.m1.1a"><mo id="S4.T1.5.5.5.1.m1.1.1" xref="S4.T1.5.5.5.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S4.T1.5.5.5.1.m1.1b"><gt id="S4.T1.5.5.5.1.m1.1.1.cmml" xref="S4.T1.5.5.5.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.5.5.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S4.T1.5.5.5.1.m1.1d">></annotation></semantics></math> 0.5</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.5.5.5.3">Sub-Query</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.5.5.5.4">24</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.5.5.5.5">2.25</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.5.5.5.6">4.00</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.5.5.5.7">0.535</td> </tr> </tbody> </table> </span></div> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Performance Metrics over a smaller seed set</figcaption> </figure> <figure class="ltx_table" id="S4.T2"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S4.T2.2" style="width:397.5pt;height:189pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-47.4pt,22.5pt) scale(0.807530485392001,0.807530485392001) ;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="S4.T2.2.2"> <thead class="ltx_thead"> <tr class="ltx_tr" id="S4.T2.2.2.3.1"> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" id="S4.T2.2.2.3.1.1"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.3.1.1.1">Uncertainty Estimator</span></th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" id="S4.T2.2.2.3.1.2"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.3.1.2.1">Trigger Retrieval When</span></th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="S4.T2.2.2.3.1.3"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.3.1.3.1">#search</span></th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="S4.T2.2.2.3.1.4"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.3.1.4.1">#steps</span></th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="S4.T2.2.2.3.1.5"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.3.1.5.1">ret ratio</span></th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="S4.T2.2.2.3.1.6"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.3.1.6.1">correct</span></th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="S4.T2.2.2.3.1.7"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.3.1.7.1">incorrect</span></th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="S4.T2.2.2.3.1.8"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.3.1.8.1">f1</span></th> </tr> </thead> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="S4.T2.2.2.4.1"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" id="S4.T2.2.2.4.1.1">Always Retrieve</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" id="S4.T2.2.2.4.1.2">Always</th> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.4.1.3">4.63</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.4.1.4">3.63</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.4.1.5">1.32</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.4.1.6">0.493</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.4.1.7">0.493</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.4.1.8">0.578</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.5.2"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T2.2.2.5.2.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_r" id="S4.T2.2.2.5.2.2"></th> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.5.2.3">4.61</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.5.2.4">3.61</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.5.2.5">1.33</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.5.2.6">0.52</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.5.2.7">0.467</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.5.2.8">0.594</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.6.3"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T2.2.2.6.3.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_r" id="S4.T2.2.2.6.3.2"></th> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.6.3.3">4.61</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.6.3.4">3.61</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.6.3.5">1.33</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.6.3.6">0.493</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.6.3.7">0.493</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.6.3.8">0.571</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.7.4"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T2.2.2.7.4.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_r" id="S4.T2.2.2.7.4.2"></th> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.7.4.3"></td> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.7.4.4"></td> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.7.4.5"></td> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.7.4.6"></td> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.7.4.7"></td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.7.4.8"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.7.4.8.1">0.581</span></td> </tr> <tr class="ltx_tr" id="S4.T2.1.1.1"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" id="S4.T2.1.1.1.2">Degree Matrix Jaccard</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" id="S4.T2.1.1.1.1">U <math alttext=">" class="ltx_Math" display="inline" id="S4.T2.1.1.1.1.m1.1"><semantics id="S4.T2.1.1.1.1.m1.1a"><mo id="S4.T2.1.1.1.1.m1.1.1" xref="S4.T2.1.1.1.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S4.T2.1.1.1.1.m1.1b"><gt id="S4.T2.1.1.1.1.m1.1.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.1.1.1.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S4.T2.1.1.1.1.m1.1d">></annotation></semantics></math> 0.4</th> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.1.1.1.3">1.80</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.1.1.1.4">3.61</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.1.1.1.5">0.57</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.1.1.1.6">0.453</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.1.1.1.7">0.533</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.1.1.1.8">0.538</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.8.5"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T2.2.2.8.5.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_r" id="S4.T2.2.2.8.5.2"></th> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.8.5.3">1.92</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.8.5.4">3.60</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.8.5.5">0.61</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.8.5.6">0.44</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.8.5.7">0.547</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.8.5.8">0.525</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.9.6"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T2.2.2.9.6.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_r" id="S4.T2.2.2.9.6.2"></th> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.9.6.3">1.85</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.9.6.4">3.61</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.9.6.5">0.57</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.9.6.6">0.419</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.9.6.7">0.568</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.9.6.8">0.508</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.10.7"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T2.2.2.10.7.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_r" id="S4.T2.2.2.10.7.2"></th> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.10.7.3"></td> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.10.7.4"></td> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.10.7.5"></td> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.10.7.6"></td> <td class="ltx_td ltx_border_t" id="S4.T2.2.2.10.7.7"></td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.10.7.8"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.10.7.8.1">0.524</span></td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.2"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" id="S4.T2.2.2.2.2">Eccentricity</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" id="S4.T2.2.2.2.1">U <math alttext=">" class="ltx_Math" display="inline" id="S4.T2.2.2.2.1.m1.1"><semantics id="S4.T2.2.2.2.1.m1.1a"><mo id="S4.T2.2.2.2.1.m1.1.1" xref="S4.T2.2.2.2.1.m1.1.1.cmml">></mo><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.2.1.m1.1b"><gt id="S4.T2.2.2.2.1.m1.1.1.cmml" xref="S4.T2.2.2.2.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.2.1.m1.1c">></annotation><annotation encoding="application/x-llamapun" id="S4.T2.2.2.2.1.m1.1d">></annotation></semantics></math> 2</th> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.2.3">2.17</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.2.4">3.60</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.2.5">0.64</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.2.6">0.44</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.2.7">0.547</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.2.2.2.8">0.525</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.11.8"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T2.2.2.11.8.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_r" id="S4.T2.2.2.11.8.2"></th> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.11.8.3">2.25</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.11.8.4">3.63</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.11.8.5">0.67</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.11.8.6">0.467</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.11.8.7">0.533</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.11.8.8">0.565</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.12.9"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T2.2.2.12.9.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_r" id="S4.T2.2.2.12.9.2"></th> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.12.9.3">2.23</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.12.9.4">3.63</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.12.9.5">0.64</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.12.9.6">0.507</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.12.9.7">0.493</td> <td class="ltx_td ltx_align_center" id="S4.T2.2.2.12.9.8">0.594</td> </tr> <tr class="ltx_tr" id="S4.T2.2.2.13.10"> <th class="ltx_td ltx_th ltx_th_row ltx_border_bb" id="S4.T2.2.2.13.10.1"></th> <th class="ltx_td ltx_th ltx_th_row ltx_border_bb ltx_border_r" id="S4.T2.2.2.13.10.2"></th> <td class="ltx_td ltx_border_bb ltx_border_t" id="S4.T2.2.2.13.10.3"></td> <td class="ltx_td ltx_border_bb ltx_border_t" id="S4.T2.2.2.13.10.4"></td> <td class="ltx_td ltx_border_bb ltx_border_t" id="S4.T2.2.2.13.10.5"></td> <td class="ltx_td ltx_border_bb ltx_border_t" id="S4.T2.2.2.13.10.6"></td> <td class="ltx_td ltx_border_bb ltx_border_t" id="S4.T2.2.2.13.10.7"></td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S4.T2.2.2.13.10.8"><span class="ltx_text ltx_font_bold" id="S4.T2.2.2.13.10.8.1">0.561</span></td> </tr> </tbody> </table> </span></div> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Performance Metrics for Different Uncertainty Estimators for 75 examples.</figcaption> </figure> </section> <section class="ltx_subsection" id="S4.SS3"> <h3 class="ltx_title ltx_title_subsection"> <span class="ltx_tag ltx_tag_subsection">4.3 </span>Subquery Generation for Retrieval</h3> <div class="ltx_para ltx_noindent" id="S4.SS3.p1"> <p class="ltx_p" id="S4.SS3.p1.1">We resort to retrieving relevant knowledge to account for the information that the model is lacking to answer the question. FLARE <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib9" title="">2023</a>)</cite> generates a retrieval query for the missing entity in the temporary sentence by using the sentence with the low probability token removed or by prompting an external question generator to generate a question for the missing entity as the answer. We generalize this by instead prompting the model to generate a subquery to figure out the missing information needed to answer the user query in an open-ended manner.</p> </div> <div class="ltx_para ltx_noindent" id="S4.SS3.p2"> <p class="ltx_p" id="S4.SS3.p2.4">We define a subquery generator <math alttext="\mathbf{S_{Q}}" class="ltx_Math" display="inline" id="S4.SS3.p2.1.m1.1"><semantics id="S4.SS3.p2.1.m1.1a"><msub id="S4.SS3.p2.1.m1.1.1" xref="S4.SS3.p2.1.m1.1.1.cmml"><mi id="S4.SS3.p2.1.m1.1.1.2" xref="S4.SS3.p2.1.m1.1.1.2.cmml">𝐒</mi><mi id="S4.SS3.p2.1.m1.1.1.3" xref="S4.SS3.p2.1.m1.1.1.3.cmml">𝐐</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.1.m1.1b"><apply id="S4.SS3.p2.1.m1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS3.p2.1.m1.1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1">subscript</csymbol><ci id="S4.SS3.p2.1.m1.1.1.2.cmml" xref="S4.SS3.p2.1.m1.1.1.2">𝐒</ci><ci id="S4.SS3.p2.1.m1.1.1.3.cmml" xref="S4.SS3.p2.1.m1.1.1.3">𝐐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.1.m1.1c">\mathbf{S_{Q}}</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.p2.1.m1.1d">bold_S start_POSTSUBSCRIPT bold_Q end_POSTSUBSCRIPT</annotation></semantics></math> which takes in as input few-shot exemplars of subqueries, the current user query <math alttext="\mathbf{q}" class="ltx_Math" display="inline" id="S4.SS3.p2.2.m2.1"><semantics id="S4.SS3.p2.2.m2.1a"><mi id="S4.SS3.p2.2.m2.1.1" xref="S4.SS3.p2.2.m2.1.1.cmml">𝐪</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.2.m2.1b"><ci id="S4.SS3.p2.2.m2.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1">𝐪</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.2.m2.1c">\mathbf{q}</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.p2.2.m2.1d">bold_q</annotation></semantics></math>, and the current partial answer sentences uttered in chain-of-thought <cite class="ltx_cite ltx_citemacro_cite">Wei et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib24" title="">2022</a>)</cite> fashion. It seeks to generate subqueries to get a specific piece of information not generated in the partial answer sentences but is needed to answer <math alttext="\mathbf{q}" class="ltx_Math" display="inline" id="S4.SS3.p2.3.m3.1"><semantics id="S4.SS3.p2.3.m3.1a"><mi id="S4.SS3.p2.3.m3.1.1" xref="S4.SS3.p2.3.m3.1.1.cmml">𝐪</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.3.m3.1b"><ci id="S4.SS3.p2.3.m3.1.1.cmml" xref="S4.SS3.p2.3.m3.1.1">𝐪</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.3.m3.1c">\mathbf{q}</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.p2.3.m3.1d">bold_q</annotation></semantics></math>. Once this subquery is generated, we use this subquery to retrieve additional passages from the external retriever <math alttext="\mathbf{R}" class="ltx_Math" display="inline" id="S4.SS3.p2.4.m4.1"><semantics id="S4.SS3.p2.4.m4.1a"><mi id="S4.SS3.p2.4.m4.1.1" xref="S4.SS3.p2.4.m4.1.1.cmml">𝐑</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.4.m4.1b"><ci id="S4.SS3.p2.4.m4.1.1.cmml" xref="S4.SS3.p2.4.m4.1.1">𝐑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.4.m4.1c">\mathbf{R}</annotation><annotation encoding="application/x-llamapun" id="S4.SS3.p2.4.m4.1d">bold_R</annotation></semantics></math>. These passages are then appended to the user input, and the generation continues.</p> </div> <div class="ltx_para ltx_noindent" id="S4.SS3.p3"> <p class="ltx_p" id="S4.SS3.p3.1">For instance, for the question, “Which film has the director who died first, Promised Heaven or Fire Over England?”, and the partially generated answer, “The film Promised Heaven was directed by Eldar Ryazanov. Fire Over England was directed by William K. Howard. Eldar Ryazanov died on November 30, 2015.”, we expect the model to generate a subquery, “When did William K. Howard die?”.</p> </div> </section> </section> <section class="ltx_section" id="S5"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">5 </span>Setup</h2> <div class="ltx_para ltx_noindent" id="S5.p1"> <p class="ltx_p" id="S5.p1.1">The generator used in all experiments was GPT-3 (davinci-002) <cite class="ltx_cite ltx_citemacro_cite">Brown et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib2" title="">2020</a>)</cite>, and the retriever employed was BM25 through PyTerrier <cite class="ltx_cite ltx_citemacro_cite">Macdonald et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib14" title="">2021</a>); Dhole (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib5" title="">2024b</a>)</cite>. The base code used for conducting the experiments and computing the metrics presented in the tables was obtained from the active RAG setup by <cite class="ltx_cite ltx_citemacro_citet">Jiang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib9" title="">2023</a>)</cite>. For uncertainty detection, we resort to the <cite class="ltx_cite ltx_citemacro_citet">Fadeeva et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib7" title="">2023</a>)</cite>’s LM-Polygraph library.</p> </div> <div class="ltx_para ltx_noindent" id="S5.p2"> <p class="ltx_p" id="S5.p2.1">Since running GPT-3 (davinci-002) along with many of the uncertainty detection metrics could be expensive to run (due to making multiple calls), we first perform a run for a small seed set of 25 queries across all metrics and then choose the 3 best metrics for a rerun across a larger set of 75 examples. We perform each run three times.</p> </div> </section> <section class="ltx_section" id="S6"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">6 </span>Results</h2> <div class="ltx_para ltx_noindent" id="S6.p1"> <p class="ltx_p" id="S6.p1.1">We now present the results in Tables <a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S4.T1" title="Table 1 ‣ 4.2 Sequence Level Uncertainty Evaluation Measures ‣ 4 Approach ‣ To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_tag">1</span></a> and <a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S4.T2" title="Table 2 ‣ 4.2 Sequence Level Uncertainty Evaluation Measures ‣ 4 Approach ‣ To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_tag">2</span></a> for the smaller and the larger sets respectively.</p> </div> <div class="ltx_para ltx_noindent" id="S6.p2"> <p class="ltx_p" id="S6.p2.1">The baseline method where retrieval was always invoked yielded an F1 score of <span class="ltx_text ltx_font_bold" id="S6.p2.1.1">0.552</span> when using temporary sentences as retrieval queries and <span class="ltx_text ltx_font_bold" id="S6.p2.1.2">0.538</span> when subqueries were generated for retrieval but required most number of retrieval operations.</p> </div> <div class="ltx_para ltx_noindent" id="S6.p3"> <p class="ltx_p" id="S6.p3.1">Triggering retrieval, when uncertainty computed through <span class="ltx_text ltx_font_bold" id="S6.p3.1.1">Eccentricity</span> i.e. <math alttext="U>2" class="ltx_Math" display="inline" id="S6.p3.1.m1.1"><semantics id="S6.p3.1.m1.1a"><mrow id="S6.p3.1.m1.1.1" xref="S6.p3.1.m1.1.1.cmml"><mi id="S6.p3.1.m1.1.1.2" xref="S6.p3.1.m1.1.1.2.cmml">U</mi><mo id="S6.p3.1.m1.1.1.1" xref="S6.p3.1.m1.1.1.1.cmml">></mo><mn id="S6.p3.1.m1.1.1.3" xref="S6.p3.1.m1.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.p3.1.m1.1b"><apply id="S6.p3.1.m1.1.1.cmml" xref="S6.p3.1.m1.1.1"><gt id="S6.p3.1.m1.1.1.1.cmml" xref="S6.p3.1.m1.1.1.1"></gt><ci id="S6.p3.1.m1.1.1.2.cmml" xref="S6.p3.1.m1.1.1.2">𝑈</ci><cn id="S6.p3.1.m1.1.1.3.cmml" type="integer" xref="S6.p3.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.p3.1.m1.1c">U>2</annotation><annotation encoding="application/x-llamapun" id="S6.p3.1.m1.1d">italic_U > 2</annotation></semantics></math>, led to the highest F1 score of <span class="ltx_text ltx_font_bold" id="S6.p3.1.2">0.605</span>, with a lesser number of search operations. This approach balanced retrieval efficiency and task performance better than other methods. It required half the number of search operations than an Always Retrieve approach. <span class="ltx_text ltx_font_bold" id="S6.p3.1.3">Semantic Sets’</span> innovative clustering approach performed poorly, with an F1 score of <span class="ltx_text ltx_font_bold" id="S6.p3.1.4">0.411</span>. Using entailment-based similarity to compute uncertainty via the <span class="ltx_text ltx_font_bold" id="S6.p3.1.5">Degree Matrix NLI</span> measure achieved an F1 score of <span class="ltx_text ltx_font_bold" id="S6.p3.1.6">0.535</span>, comparable to the baseline. The lightweight <span class="ltx_text ltx_font_bold" id="S6.p3.1.7">Degree Matrix (Jaccard)</span> necessitated the least number of retrieval operations to perform better than an Always Retrieve baseline.</p> </div> <div class="ltx_para ltx_noindent" id="S6.p4"> <p class="ltx_p" id="S6.p4.1">Table <a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#S4.T2" title="Table 2 ‣ 4.2 Sequence Level Uncertainty Evaluation Measures ‣ 4 Approach ‣ To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation"><span class="ltx_text ltx_ref_tag">2</span></a> presents additional performance metrics over a larger set of 75 examples. Notably, the <span class="ltx_text ltx_font_bold" id="S6.p4.1.1">Eccentricity</span> method consistently demonstrated the best balance between retrieval efficiency and performance, achieving an average F1 score of <span class="ltx_text ltx_font_bold" id="S6.p4.1.2">0.561</span> across different experimental runs, while reducing unnecessary retrievals compared to the baseline.</p> </div> <div class="ltx_para ltx_noindent" id="S6.p5"> <p class="ltx_p" id="S6.p5.1"><span class="ltx_text ltx_font_bold" id="S6.p5.1.1">Degree Matrix (Jaccard)</span> performed slightly worse in F1 score (<span class="ltx_text ltx_font_bold" id="S6.p5.1.2">0.524</span>) but depended on retrieval the least indicating its potential for applications where minimizing retrieval costs is crucial.</p> </div> <div class="ltx_para ltx_noindent" id="S6.p6"> <p class="ltx_p" id="S6.p6.1">In contrast, the <span class="ltx_text ltx_font_bold" id="S6.p6.1.1">Always Retrieve</span> approach performed better than both conditional retrieval approaches but necessitated almost twice the number of retrieval calls.</p> </div> </section> <section class="ltx_section" id="S7"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">7 </span>Conclusion</h2> <div class="ltx_para ltx_noindent" id="S7.p1"> <p class="ltx_p" id="S7.p1.1">Our experiments demonstrate that dynamic retrieval, guided by uncertainty detection, improves the efficiency of retrieval-augmented generation systems, making it useful where retrieval can be expensive to compute. Among the methods tested, <span class="ltx_text ltx_font_bold" id="S7.p1.1.1">Eccentricity-based uncertainty detection</span> emerged as the best-performing approach, offering the highest F1 score with a moderate number of retrieval steps and searches. This method effectively balances retrieval efficiency with task performance.</p> </div> <div class="ltx_para ltx_noindent" id="S7.p2"> <p class="ltx_p" id="S7.p2.1">The <span class="ltx_text ltx_font_bold" id="S7.p2.1.1">Degree Matrix (Jaccard)</span> method also showed promising results, particularly in reducing retrieval costs while maintaining reasonable performance. Conversely, methods such as <span class="ltx_text ltx_font_bold" id="S7.p2.1.2">Semantic Sets</span> and <span class="ltx_text ltx_font_bold" id="S7.p2.1.3">FLARE-Instruct</span> underperformed, highlighting the need for more reliable uncertainty estimators.</p> </div> <div class="ltx_para ltx_noindent" id="S7.p3"> <p class="ltx_p" id="S7.p3.1">Although some black-box uncertainty detection methods require multiple runs of generation, which can be costly, always retrieving may be preferable in RAG applications where lightweight retrieval methods like BM25 suffice. This is also evident from the results on the larger set.</p> </div> <div class="ltx_para ltx_noindent" id="S7.p4"> <p class="ltx_p" id="S7.p4.1">Besides, we feel that uncertainty detection might become more mainstream as the propensity for hallucination in LLMs increases, and as end applications demand more confidence and interpretability <cite class="ltx_cite ltx_citemacro_cite">Dhole et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib6" title="">2024</a>)</cite> in their outputs making uncertainty detection a necessity. Our work focuses on exploiting uncertainty detection for RAG, especially where retrieval can be expensive like the usage of heavy and composite retrieval systems employing numerous components like reformulation, dense retrieval <cite class="ltx_cite ltx_citemacro_cite">Santhanam et al. (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib19" title="">2021</a>)</cite>, reranking, etc.</p> </div> </section> <section class="ltx_section" id="S8"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">8 </span>Ethical Considerations</h2> <div class="ltx_para ltx_noindent" id="S8.p1"> <p class="ltx_p" id="S8.p1.1">When evaluating large language models (LLMs), it is essential to adopt a sociotechnical perspective <cite class="ltx_cite ltx_citemacro_cite">Dhole (<a class="ltx_ref" href="https://arxiv.org/html/2501.09292v3#bib.bib3" title="">2023</a>)</cite>, acknowledging that their outputs are influenced by both social contexts and technical design choices. Proper safeguards should be in place to mitigate biases and prevent the generation of harmful or toxic content. Furthermore, the uncertainty detection approaches we employed rely on estimations derived from various neural network computations, which are inherently shaped by the data on which the models are trained. Consequently, it is critical to thoroughly test uncertainty detection methods to ensure they meet the requirements of the intended applications.</p> </div> <div class="ltx_para ltx_noindent" id="S8.p2"> <p class="ltx_p" id="S8.p2.1">Despite these precautions, there remains a possibility that some approaches may misrepresent the level of certainty, as no method is flawless. Therefore, ongoing evaluation and refinement of uncertainty detection mechanisms are necessary to minimize inaccuracies and potential misinterpretations.</p> </div> </section> <section class="ltx_section" id="Sx1"> <h2 class="ltx_title ltx_title_section">Acknowledgements</h2> <div class="ltx_para ltx_noindent" id="Sx1.p1"> <p class="ltx_p" id="Sx1.p1.1">The author would like to thank Eugene Agichtein for insightful discussions and the anonymous reviewers for their useful feedback. The author would also like to thank Microsoft for providing OpenAI credits through the Microsoft Accelerating Foundation Models Research Award.</p> </div> </section> <section class="ltx_bibliography" id="bib"> <h2 class="ltx_title ltx_title_bibliography">References</h2> <ul class="ltx_biblist"> <li class="ltx_bibitem" id="bib.bib1"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. (2022)</span> <span class="ltx_bibblock"> Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. </span> <span class="ltx_bibblock">Training a helpful and harmless assistant with reinforcement learning from human feedback. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib1.1.1">arXiv preprint arXiv:2204.05862</em>, 2022. </span> </li> <li class="ltx_bibitem" id="bib.bib2"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span> <span class="ltx_bibblock"> Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. </span> <span class="ltx_bibblock">Language models are few-shot learners, 2020. </span> </li> <li class="ltx_bibitem" id="bib.bib3"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dhole (2023)</span> <span class="ltx_bibblock"> Kaustubh Dhole. </span> <span class="ltx_bibblock">Large language models as sociotechnical systems. </span> <span class="ltx_bibblock">In <em class="ltx_emph ltx_font_italic" id="bib.bib3.1.1">Proceedings of the Big Picture Workshop</em>, pp. 66–79, 2023. </span> </li> <li class="ltx_bibitem" id="bib.bib4"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dhole (2024a)</span> <span class="ltx_bibblock"> Kaustubh Dhole. </span> <span class="ltx_bibblock">Kaucus-knowledgeable user simulators for training large language models. </span> <span class="ltx_bibblock">In <em class="ltx_emph ltx_font_italic" id="bib.bib4.1.1">Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024)</em>, pp. 53–65, 2024a. </span> </li> <li class="ltx_bibitem" id="bib.bib5"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dhole (2024b)</span> <span class="ltx_bibblock"> Kaustubh D Dhole. </span> <span class="ltx_bibblock">Pyterrier-genrank: The pyterrier plugin for reranking with large language models. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib5.1.1">arXiv preprint arXiv:2412.05339</em>, 2024b. </span> </li> <li class="ltx_bibitem" id="bib.bib6"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dhole et al. (2024)</span> <span class="ltx_bibblock"> Kaustubh D. Dhole, Kai Shu, and Eugene Agichtein. </span> <span class="ltx_bibblock">Conqret: Benchmarking fine-grained evaluation of retrieval augmented argumentation with llm judges, 2024. </span> </li> <li class="ltx_bibitem" id="bib.bib7"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fadeeva et al. (2023)</span> <span class="ltx_bibblock"> Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. </span> <span class="ltx_bibblock">LM-polygraph: Uncertainty estimation for language models. </span> <span class="ltx_bibblock">In Yansong Feng and Els Lefever (eds.), <em class="ltx_emph ltx_font_italic" id="bib.bib7.1.1">Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations</em>, pp. 446–461, Singapore, December 2023. Association for Computational Linguistics. </span> <span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.emnlp-demo.41</span>. </span> <span class="ltx_bibblock">URL <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://aclanthology.org/2023.emnlp-demo.41" title="">https://aclanthology.org/2023.emnlp-demo.41</a>. </span> </li> <li class="ltx_bibitem" id="bib.bib8"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ho et al. (2020)</span> <span class="ltx_bibblock"> Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. </span> <span class="ltx_bibblock">Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. </span> <span class="ltx_bibblock">In <em class="ltx_emph ltx_font_italic" id="bib.bib8.1.1">Proceedings of the 28th International Conference on Computational Linguistics</em>, pp. 6609–6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. </span> <span class="ltx_bibblock">URL <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.aclweb.org/anthology/2020.coling-main.580" title="">https://www.aclweb.org/anthology/2020.coling-main.580</a>. </span> </li> <li class="ltx_bibitem" id="bib.bib9"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023)</span> <span class="ltx_bibblock"> Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. </span> <span class="ltx_bibblock">Active retrieval augmented generation. </span> <span class="ltx_bibblock">In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), <em class="ltx_emph ltx_font_italic" id="bib.bib9.1.1">Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</em>, pp. 7969–7992, Singapore, December 2023. Association for Computational Linguistics. </span> <span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.emnlp-main.495</span>. </span> <span class="ltx_bibblock">URL <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://aclanthology.org/2023.emnlp-main.495" title="">https://aclanthology.org/2023.emnlp-main.495</a>. </span> </li> <li class="ltx_bibitem" id="bib.bib10"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kadavath et al. (2022)</span> <span class="ltx_bibblock"> Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. </span> <span class="ltx_bibblock">Language models (mostly) know what they know. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib10.1.1">arXiv preprint arXiv:2207.05221</em>, 2022. </span> </li> <li class="ltx_bibitem" id="bib.bib11"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kuhn et al. (2023)</span> <span class="ltx_bibblock"> Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. </span> <span class="ltx_bibblock">Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. </span> <span class="ltx_bibblock">In <em class="ltx_emph ltx_font_italic" id="bib.bib11.1.1">The Eleventh International Conference on Learning Representations</em>, 2023. </span> <span class="ltx_bibblock">URL <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://openreview.net/forum?id=VD-AYtP0dve" title="">https://openreview.net/forum?id=VD-AYtP0dve</a>. </span> </li> <li class="ltx_bibitem" id="bib.bib12"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lewis et al. (2020)</span> <span class="ltx_bibblock"> Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. </span> <span class="ltx_bibblock">Retrieval-augmented generation for knowledge-intensive nlp tasks. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib12.1.1">Advances in Neural Information Processing Systems</em>, 33:9459–9474, 2020. </span> </li> <li class="ltx_bibitem" id="bib.bib13"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. (2023)</span> <span class="ltx_bibblock"> Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. </span> <span class="ltx_bibblock">Generating with confidence: Uncertainty quantification for black-box large language models. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib13.1.1">Transactions on Machine Learning Research</em>, 2023. </span> </li> <li class="ltx_bibitem" id="bib.bib14"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Macdonald et al. (2021)</span> <span class="ltx_bibblock"> Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. </span> <span class="ltx_bibblock">Pyterrier: Declarative experimentation in python from bm25 to dense retrieval. </span> <span class="ltx_bibblock">In <em class="ltx_emph ltx_font_italic" id="bib.bib14.1.1">Proceedings of the 30th acm international conference on information & knowledge management</em>, pp. 4526–4533, 2021. </span> </li> <li class="ltx_bibitem" id="bib.bib15"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mallen et al. (2023)</span> <span class="ltx_bibblock"> Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. </span> <span class="ltx_bibblock">When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. </span> <span class="ltx_bibblock">In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), <em class="ltx_emph ltx_font_italic" id="bib.bib15.1.1">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pp. 9802–9822, Toronto, Canada, July 2023. Association for Computational Linguistics. </span> <span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.acl-long.546</span>. </span> <span class="ltx_bibblock">URL <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://aclanthology.org/2023.acl-long.546" title="">https://aclanthology.org/2023.acl-long.546</a>. </span> </li> <li class="ltx_bibitem" id="bib.bib16"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span> <span class="ltx_bibblock"> OpenAI. </span> <span class="ltx_bibblock">Gpt-4 technical report, 2023. </span> </li> <li class="ltx_bibitem" id="bib.bib17"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. (2022)</span> <span class="ltx_bibblock"> Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. </span> <span class="ltx_bibblock">Training language models to follow instructions with human feedback. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib17.1.1">Advances in neural information processing systems</em>, 35:27730–27744, 2022. </span> </li> <li class="ltx_bibitem" id="bib.bib18"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rafailov et al. (2024)</span> <span class="ltx_bibblock"> Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. </span> <span class="ltx_bibblock">Direct preference optimization: Your language model is secretly a reward model. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib18.1.1">Advances in Neural Information Processing Systems</em>, 36, 2024. </span> </li> <li class="ltx_bibitem" id="bib.bib19"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Santhanam et al. (2021)</span> <span class="ltx_bibblock"> Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. </span> <span class="ltx_bibblock">Colbertv2: Effective and efficient retrieval via lightweight late interaction. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib19.1.1">arXiv preprint arXiv:2112.01488</em>, 2021. </span> </li> <li class="ltx_bibitem" id="bib.bib20"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Srivastava et al. (2023)</span> <span class="ltx_bibblock"> Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. </span> <span class="ltx_bibblock">Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib20.1.1">Transactions on Machine Learning Research</em>, 2023. </span> </li> <li class="ltx_bibitem" id="bib.bib21"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Team et al. (2023)</span> <span class="ltx_bibblock"> Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. </span> <span class="ltx_bibblock">Gemini: a family of highly capable multimodal models. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib21.1.1">arXiv preprint arXiv:2312.11805</em>, 2023. </span> </li> <li class="ltx_bibitem" id="bib.bib22"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2024)</span> <span class="ltx_bibblock"> Hongru Wang, Boyang Xue, Baohang Zhou, Tianhua Zhang, Cunxiang Wang, Huimin Wang, Guanhua Chen, and Kam-fai Wong. </span> <span class="ltx_bibblock">Self-dc: When to reason and when to act? self divide-and-conquer for compositional unknown questions. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib22.1.1">arXiv preprint arXiv:2402.13514</em>, 2024. </span> </li> <li class="ltx_bibitem" id="bib.bib23"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023)</span> <span class="ltx_bibblock"> Yile Wang, Peng Li, Maosong Sun, and Yang Liu. </span> <span class="ltx_bibblock">Self-knowledge guided retrieval augmentation for large language models. </span> <span class="ltx_bibblock">In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), <em class="ltx_emph ltx_font_italic" id="bib.bib23.1.1">Findings of the Association for Computational Linguistics: EMNLP 2023</em>, pp. 10303–10315, Singapore, December 2023. Association for Computational Linguistics. </span> <span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.findings-emnlp.691</span>. </span> <span class="ltx_bibblock">URL <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://aclanthology.org/2023.findings-emnlp.691" title="">https://aclanthology.org/2023.findings-emnlp.691</a>. </span> </li> <li class="ltx_bibitem" id="bib.bib24"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022)</span> <span class="ltx_bibblock"> Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. </span> <span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language models. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib24.1.1">Advances in neural information processing systems</em>, 35:24824–24837, 2022. </span> </li> <li class="ltx_bibitem" id="bib.bib25"> <span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2024)</span> <span class="ltx_bibblock"> Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Weichuan Liu, Lei Hou, and Juanzi Li. </span> <span class="ltx_bibblock">Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation. </span> <span class="ltx_bibblock"><em class="ltx_emph ltx_font_italic" id="bib.bib25.1.1">arXiv preprint arXiv:2406.19215</em>, 2024. </span> </li> </ul> </section> </article> </div> <footer class="ltx_page_footer"> <div class="ltx_page_logo">Generated on Tue Mar 18 16:39:21 2025 by <a class="ltx_LaTeXML_logo" href="http://dlmf.nist.gov/LaTeXML/"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img alt="Mascot Sammy" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg=="/></a> </div></footer> </div> </body> </html>