CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios</title>  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <base href="/html/2410.19346v1/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S1" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">1 Introduction</a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S2" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">2 Related Work</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S2.SS1" title="In 2 Related Work ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">2.1 Social Intelligence Benchmarks</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S2.SS2" title="In 2 Related Work ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">2.2 Role-playing Agents</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3 AgentSense Benchmark</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS1" title="In 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.1 Framework Overview</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS1.SSS0.Px1" title="In 3.1 Framework Overview ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Scenarios</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS1.SSS0.Px2" title="In 3.1 Framework Overview ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Scenario Templates</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS1.SSS0.Px3" title="In 3.1 Framework Overview ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Benchmarking</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS2" title="In 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.2 Scenario Construction</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS2.SSS0.Px1" title="In 3.2 Scenario Construction ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Template Construction</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS2.SSS0.Px2" title="In 3.2 Scenario Construction ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Scenario Instantiating</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS3" title="In 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.3 Social Interaction Simulation</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS4" title="In 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.4 Evaluation</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS4.SSS0.Px1" title="In 3.4 Evaluation ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Goal Completion</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS4.SSS0.Px2" title="In 3.4 Evaluation ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Implicit Reasoning</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS4.SSS0.Px3" title="In 3.4 Evaluation ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Profile Sensitivity</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS5" title="In 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.5 Data Validation and Analysis</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS5.SSS0.Px1" title="In 3.5 Data Validation and Analysis ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Data Source</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS5.SSS0.Px2" title="In 3.5 Data Validation and Analysis ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Data Validation</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS5.SSS0.Px3" title="In 3.5 Data Validation and Analysis ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Data Analysis</a></li> </ol> </li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4 Experiments</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS1" title="In 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4.1 Experimental Settings</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS1.SSS0.Px1" title="In 4.1 Experimental Settings ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Agent Models</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS1.SSS0.Px2" title="In 4.1 Experimental Settings ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Judge Models</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS1.SSS0.Px3" title="In 4.1 Experimental Settings ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Implementation Details</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS2" title="In 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4.2 Overall Performance</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS2.SSS0.Px1" title="In 4.2 Overall Performance ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Single Model-based</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS2.SSS0.Px2" title="In 4.2 Overall Performance ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Pairwise Model-based</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS3" title="In 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4.3 Analysis of Data Leakage</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S5" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">5 Further Analysis</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S5.SS1" title="In 5 Further Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">5.1 What goals are LLMs good/bad at?</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S5.SS2" title="In 5 Further Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">5.2 Which is harder: guessing thoughts or keeping secrets?</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S5.SS3" title="In 5 Further Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">5.3 What scenarios are more sensitive to profiles?</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S6" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">6 Conclusion</a></li> <li class="ltx_tocentry ltx_tocentry_appendix"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A Scenario Construction</a> <ol class="ltx_toclist ltx_toclist_appendix"> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS1" title="In Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.1 Scenario Template Construction Details</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS1.SSS1" title="In A.1 Scenario Template Construction Details ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.1.1 Social Goal Extraction</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS1.SSS2" title="In A.1 Scenario Template Construction Details ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.1.2 Private Information Extraction</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS1.SSS3" title="In A.1 Scenario Template Construction Details ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.1.3 Leakage Mitigation and Template Generation</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2" title="In Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2 Prompts in Construction Pipeline</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2.SSS1" title="In A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2.1 Scenario Extraction</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2.SSS2" title="In A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2.2 Social Goal Extraction</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2.SSS3" title="In A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2.3 Private Information Extraction</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2.SSS4" title="In A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2.4 Leakage Mitigation and Template Generation</a></li> <li class="ltx_tocentry ltx_tocentry_subsubsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2.SSS5" title="In A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2.5 Agent Synthesizing</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS3" title="In Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.3 Data Validation Annotation</a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS3.SSS0.Px1" title="In A.3 Data Validation Annotation ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Social Goal</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS3.SSS0.Px2" title="In A.3 Data Validation Annotation ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Private Information</a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS3.SSS0.Px3" title="In A.3 Data Validation Annotation ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">Annotation Statistics</a></li> </ol> </li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_appendix"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">B Data Statistics and Analysis</a> <ol class="ltx_toclist ltx_toclist_appendix"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2.SS1" title="In Appendix B Data Statistics and Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">B.1 Detailed Statistical Overview of Scripts</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2.SS2" title="In Appendix B Data Statistics and Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">B.2 Social Scenario Categories</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2.SS3" title="In Appendix B Data Statistics and Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">B.3 Social Goal Classification</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_appendix"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A3" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">C Simulation Details</a> <ol class="ltx_toclist ltx_toclist_appendix"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A3.SS1" title="In Appendix C Simulation Details ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">C.1 Prompts for Agents</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A3.SS2" title="In Appendix C Simulation Details ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">C.2 Simulation Details</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_appendix"> <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4" title="In AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D Additional Experiment Results</a> <ol class="ltx_toclist ltx_toclist_appendix"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS1" title="In Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.1 Human Evaluation of Goal Completion</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS2" title="In Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.2 Single Model-based Subset Experiment</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS3" title="In Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.3 Pairwise Model-based Experiment Prompt</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS4" title="In Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.4 Pairwise Model-based Additional Results</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS5" title="In Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.5 Goal Completion Score Additional Results</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS6" title="In Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.6 Experiment on Profiles’ Effect on Scenarios</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS7" title="In Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.7 Under-performing Cases of Llama-2-7b</a></li> </ol> </li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document"> <h1 class="ltx_title ltx_title_document">AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios</h1> <div class="ltx_authors"> Xinyi Mou1222Equal contributors., Jingcong Liang1222Equal contributors., Jiayu Lin1222Equal contributors., Xinnong Zhang1222Equal contributors., Xiawei Liu1, Shiyue Yang1, Rong Ye1,2, Lei Chen1, Haoyu Kuang1, Xuanjing Huang1, Zhongyu Wei1333Corresponding author. 1Fudan University, 2ByteDance {jcliang22, jiayulin24, xnzhang23, liuxw24, shiyueyang24, yer23, hykuang23}@m.fudan.edu.cn, {xymou20, chenl18, xjhuang, zywei}@fudan.edu.cn </div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract</h6> Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. </div> <div class="ltx_para" id="p1"> \pdfcolInitStack tcb@breakable \useunder\ul </div> <div class="ltx_para ltx_noindent" id="p2"> <div class="ltx_block ltx_align_bottom" id="p2.1"> AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios Xinyi Mou1222Equal contributors., Jingcong Liang1222Equal contributors., Jiayu Lin1222Equal contributors., Xinnong Zhang1222Equal contributors., Xiawei Liu1, Shiyue Yang1, Rong Ye1,2, Lei Chen1, Haoyu Kuang1, Xuanjing Huang1, Zhongyu Wei1333Corresponding author. 1Fudan University, 2ByteDance {jcliang22, jiayulin24, xnzhang23, liuxw24, shiyueyang24, yer23, hykuang23}@m.fudan.edu.cn, {xymou20, chenl18, xjhuang, zywei}@fudan.edu.cn </div> </div> <section class="ltx_section" id="S1"> <h2 class="ltx_title ltx_title_section"> 1 Introduction</h2> <div class="ltx_para" id="S1.p1"> Benefiting from comprehensive training data and large-scale model parameters, large language models (LLMs) are increasingly employed to develop autonomous agents capable of simulating human behavior <cite class="ltx_cite ltx_citemacro_cite">Qin et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib24" title="">2023</a>); Shinn et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib30" title="">2024</a>); Schick et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib27" title="">2024</a>)</cite>. These language agents have been explored as human proxies in various fields of behavioral research, such as psychological and sociological surveys <cite class="ltx_cite ltx_citemacro_cite">Argyle et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib3" title="">2023</a>); Chuang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib8" title="">2024</a>); Xie et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib38" title="">2024</a>)</cite>, and opinion dynamics modeling <cite class="ltx_cite ltx_citemacro_cite">Mou et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib22" title="">2024</a>); Liu et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib20" title="">2024</a>)</cite>. These social science studies often assume that LLMs exhibit social intelligence, the ability to navigate complex and multifaceted social goals during interactions with others <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib42" title="">2024b</a>)</cite>, given their ability to produce human-like responses and behaviors in certain scenarios. Social intelligence is essential for fostering effective and harmonious interactions among agents and between agents and humans <cite class="ltx_cite ltx_citemacro_cite">Xu et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib39" title="">2023</a>)</cite>. However, the question remains: can LLMs truly navigate the intricacies of such interactions and exhibit social intelligence? As illustrated in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">1</a>, LLM-driven agents still struggle with complex social situations. For example, when facing multiple goals in an office conversation, agent Emily fails to complete all her goals and cannot guess what Jordan has in mind. </div> <figure class="ltx_figure" id="S1.F1"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_square" height="838" id="S1.F1.g1" src="x1.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 1: An illustration of challenging yet realistic social scenarios — a family gathering and an office conversation, where the characters are driven by ChatGPT. While the dialogue could flow smoothly, Emily is unable to achieve her goals during the family gathering and fails to deduce Jordan’s thoughts in an office setting.</figcaption> </figure> <div class="ltx_para" id="S1.p2"> Early research on the social intelligence of LLMs relies on static and non-interactive tests that assess commonsense reasoning about social situations <cite class="ltx_cite ltx_citemacro_cite">Sap et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib26" title="">2019</a>); Zadeh et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib40" title="">2019</a>); Shapira et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib29" title="">2023</a>); Wilf et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib36" title="">2023</a>)</cite>, failing to capture the dynamic nature of social interactions. Recently, dynamic and goal-driven benchmarks <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib42" title="">2024b</a>); Wang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib35" title="">2024</a>)</cite> have emerged to study social intelligence in interactive environments. Although they have made some significant progress, they still exhibit three main limitations: (1) lack of scenario diversity: existing studies <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib35" title="">2024</a>); Sabour et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib25" title="">2024</a>)</cite> build social scenarios manually in a top-down manner, resulting in a narrow set of common scenarios and goals, such as persuasion and collaboration <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib17" title="">2023a</a>)</cite>. However, real-world interactions feature a broader spectrum of social goals and situational dynamics. (2) insufficient scenario complexity: current work <cite class="ltx_cite ltx_citemacro_cite">Xie et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib38" title="">2024</a>); Zhou et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib42" title="">2024b</a>)</cite> often limits interactions to two participants, each pursuing a single goal. This oversimplifies real-life social interactions, where multiple actors engage simultaneously, each with multiple goals, as shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">1</a>. As a result, it remains the performance of LLMs in group dynamics insufficiently studied. (3) single-perspective observation: while existing benchmarks primarily evaluate the extent to which agents achieve explicit social goals <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib18" title="">2023b</a>); Wang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib35" title="">2024</a>)</cite>, they have neglected the concealment of private information and the inference of others’ information, which are also important aspects in social interaction. </div> <div class="ltx_para" id="S1.p3"> To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. We get inspiration from Dramaturgical Theory <cite class="ltx_cite ltx_citemacro_cite">Goffman (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib11" title="">1959</a>)</cite>, which conceptualizes social interaction as a theatrical performance in which individuals assume specific roles within various settings. We adopt a bottom-up approach, extracting scenarios from massive scripts to ensure that scenarios and social goals are diverse and grounded in real life. In AgentSense, we construct 245 scenario templates from scripts, mitigating data leakage and expanding them into 1,225 scenarios, which cover existence goals to growth goals following the ERG theory <cite class="ltx_cite ltx_citemacro_cite">Alderfer (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib2" title="">1969</a>)</cite>. </div> <div class="ltx_para" id="S1.p4"> We then situate LLM-driven agents in the constructed scenarios, where each participant has social goals and may also safeguard some private information. In an interactive environment, we simulate multi-turn interactions and observe agents’ social intelligence from two aspects: (1) goal completion: have the agents successfully achieved their social goals? (2) implicit reasoning: can the agents accurately deduce others’ private information? These abilities are measured through interviews and multiple-choice questions, with our proposed PSI metric assessing profile sensitivity. We find that LLMs struggle with complex social scenarios, particularly with high-level growth goals. </div> <div class="ltx_para" id="S1.p5"> Our contributions are as follows: <ul class="ltx_itemize" id="S1.I1"> <li class="ltx_item" id="S1.I1.i1" style="list-style-type:none;"> • <div class="ltx_para" id="S1.I1.i1.p1"> We introduce AgentSense, a benchmark built on social scenarios derived from scripts using a bottom-up approach, distinguishing it from previous work. It encompasses diverse and challenging social scenarios, enabling a thorough evaluation of LLMs’ social intelligence. </div> </li> <li class="ltx_item" id="S1.I1.i2" style="list-style-type:none;"> • <div class="ltx_para" id="S1.I1.i2.p1"> We evaluate social intelligence from multiple aspects, considering both goal completion and information reasoning as well as profile sensitivity of social intelligence, through interviews with the agents and third-party judges. </div> </li> <li class="ltx_item" id="S1.I1.i3" style="list-style-type:none;"> • <div class="ltx_para" id="S1.I1.i3.p1"> Our experiments reveal that LLMs struggle with complex scenarios and high-level goals, and their social intelligence is affected by profiles, interaction partners, and the balance between goals and privacy protection. </div> </li> </ul> </div> </section> <section class="ltx_section" id="S2"> <h2 class="ltx_title ltx_title_section"> 2 Related Work</h2> <section class="ltx_subsection" id="S2.SS1"> <h3 class="ltx_title ltx_title_subsection"> 2.1 Social Intelligence Benchmarks</h3> <div class="ltx_para" id="S2.SS1.p1"> Social intelligence is the ability to understand others and act wisely in social situations <cite class="ltx_cite ltx_citemacro_cite">Walker and Foley (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib34" title="">1973</a>)</cite>. While LLMs show potential in simulating human behavior <cite class="ltx_cite ltx_citemacro_cite">Xie et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib38" title="">2024</a>)</cite>, their social intelligence remains underexplored <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib41" title="">2024a</a>)</cite>. Current research on evaluating social intelligence in LLMs falls into two main categories. The first involves static, non-interactive assessments that evaluate models through Q&A tasks based on commonsense reasoning about social situations <cite class="ltx_cite ltx_citemacro_cite">Sap et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib26" title="">2019</a>); Zadeh et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib40" title="">2019</a>)</cite>. The second involves interactive benchmarks, where LLMs are assessed through goal-oriented interactions in role-playing scenarios <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib42" title="">2024b</a>); Chen et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib6" title="">2024a</a>); Wang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib35" title="">2024</a>); Liu et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib19" title="">2023</a>)</cite>. AgentSense uses a bottom-up approach to create scenarios from scripts, allowing evaluating agents’ capabilities in more diverse settings. </div> <figure class="ltx_figure" id="S2.F2"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="284" id="S2.F2.g1" src="x2.png" width="664"/> <figcaption class="ltx_caption ltx_centering">Figure 2: Overall framework of AgentSense. We construct scenario templates from scripts and synthesize characters to diversify the scenarios. Then, language models role-play the characters to interact with each other. After that, the participants and third-party judges are interviewed for evaluation.</figcaption> </figure> </section> <section class="ltx_subsection" id="S2.SS2"> <h3 class="ltx_title ltx_title_subsection"> 2.2 Role-playing Agents</h3> <div class="ltx_para" id="S2.SS2.p1"> LLMs are increasingly been used to construct role-playing agents (RPAs) <cite class="ltx_cite ltx_citemacro_cite">Chen et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib7" title="">2024b</a>)</cite>, which enable efficient simulation of typical representatives, from individuals <cite class="ltx_cite ltx_citemacro_cite">Shao et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib28" title="">2023</a>); Argyle et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib3" title="">2023</a>)</cite> to demographic groups <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib17" title="">2023a</a>); Jiang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib14" title="">2024</a>)</cite>. These agents can embody various personas to coordinate, collaborate, exchange information, and compete with one another <cite class="ltx_cite ltx_citemacro_cite">Chen et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib7" title="">2024b</a>); Zhou et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib42" title="">2024b</a>)</cite>. Recently, RPAs have been applied in various domains, e.g., psychotherapy <cite class="ltx_cite ltx_citemacro_cite">Stade et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib31" title="">2024</a>)</cite>, economics <cite class="ltx_cite ltx_citemacro_cite">Fu et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib10" title="">2023</a>)</cite>, and social research <cite class="ltx_cite ltx_citemacro_cite">Grossmann et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib12" title="">2023</a>)</cite>. AgentSense leverages this by simulating social interaction scenarios through agents with diverse personalities and social goals. </div> </section> </section> <section class="ltx_section" id="S3"> <h2 class="ltx_title ltx_title_section"> 3 AgentSense Benchmark</h2> <section class="ltx_subsection" id="S3.SS1"> <h3 class="ltx_title ltx_title_subsection"> 3.1 Framework Overview</h3> <div class="ltx_para" id="S3.SS1.p1"> AgentSense aims to provide a realistic social intelligence benchmark with enhanced diversity and complexity. Following the Dramaturgical Theory, we propose an overall framework as in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S2.F2" title="Figure 2 ‣ 2.1 Social Intelligence Benchmarks ‣ 2 Related Work ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">2</a>. </div> <section class="ltx_paragraph" id="S3.SS1.SSS0.Px1"> <h5 class="ltx_title ltx_title_paragraph">Scenarios</h5> <div class="ltx_para" id="S3.SS1.SSS0.Px1.p1"> The core component of AgentSense is the social scenario set, extracted from real-world scripts to guide and evaluate social interactions between agents. A social scenario serves as a hypothetical context for simulating and analyzing social interactions, where two key components are measured: (1) Social Goal is what the agent aims to achieve, such as resolving an issue or building a relationship. The agent’s proactive drive in social interactions, guided by this social goal, directs its active participation in social dynamics. (2) Private Information is information that is known solely to the agent and not to others. The agent is tasked with inferring others’ private information without explicitly inquiring about it, a process referred to as passive reasoning during interactions. In summary, an agent’s social intelligence is reflected in its ability to pursue social goals while safeguarding private information, balancing active engagement with passive respect for individual privacy. </div> </section> <section class="ltx_paragraph" id="S3.SS1.SSS0.Px2"> <h5 class="ltx_title ltx_title_paragraph">Scenario Templates</h5> <div class="ltx_para" id="S3.SS1.SSS0.Px2.p1"> Social scenarios in scripts always have a fixed group of characters, causing a lack of diversity. To address this issue, we wipe out irrelevant character details to obtain scenario templates, which contains only background information and predefined character slots. We can instantiate multiple scenarios from a scenario template by filling in the slots with different sets of synthesized characters satisfying the template’s constraints. </div> </section> <section class="ltx_paragraph" id="S3.SS1.SSS0.Px3"> <h5 class="ltx_title ltx_title_paragraph">Benchmarking</h5> <div class="ltx_para" id="S3.SS1.SSS0.Px3.p1"> After building scenarios from the extracted templates, benchmarking LLMs with AgentSense comes as follows: (1) Simulation: We prompt the models to role-play the characters and interact with each other, trying to achieve their social goals. (2) Evaluation: We evaluate the goal completion status of each model by interviewing the participants and third-party judges. We also assess the model’s implicit reasoning performance with multiple-choice questions. </div> </section> </section> <section class="ltx_subsection" id="S3.SS2"> <h3 class="ltx_title ltx_title_subsection"> 3.2 Scenario Construction</h3> <figure class="ltx_figure" id="S3.F3"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="276" id="S3.F3.g1" src="x3.png" width="664"/> <figcaption class="ltx_caption ltx_centering">Figure 3: Scenario template construction pipeline (automated with Python and GPT-4o): (A) Scenario Extraction: We split the script into scenes then scenarios (1), and summarize their background and description (2), which are merged into a descriptive background for independent role-play (3). (B) Social Goal Extraction: We extract each character’s social goals (4) and amend them by regenerating the whole scenario (5) and rewriting/deleting invalid goals (6). (C) Private Information Extraction: We determine if the scene involves private information inference (7); if yes, we extract private information as QA pairs (8) and generate private info records (9) and evaluation questions (10). (D) Leakage Mitigation and Template Generation: We remove elements associated with specific episodes and replace characters with slots for synthesized agents with similar characteristics to fill in (11).</figcaption> </figure> <div class="ltx_para" id="S3.SS2.p1"> Following the definitions, building AgentSense requires constructing templates and instantiating scenarios with synthesized agents. We propose pipelines for the two parts respectively as follows: </div> <section class="ltx_paragraph" id="S3.SS2.SSS0.Px1"> <h5 class="ltx_title ltx_title_paragraph">Template Construction</h5> <div class="ltx_para" id="S3.SS2.SSS0.Px1.p1"> Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.F3" title="Figure 3 ‣ 3.2 Scenario Construction ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3</a> demonstrates the pipeline to construct scenario templates from real-world scripts, consisting of four stages: </div> <div class="ltx_para" id="S3.SS2.SSS0.Px1.p2"> <ol class="ltx_enumerate" id="S3.I1"> <li class="ltx_item" id="S3.I1.i1" style="list-style-type:none;"> (1) <div class="ltx_para" id="S3.I1.i1.p1"> Scenario Extraction: Real-world scripts consist of multiple chronological scenes, within which several scenarios involve groups of characters. We first split scenes and scenarios from the script. Then, we generate each scenario’s background from previous scenes and its own description. Finally, we generate a new descriptive background that allows the scenario to be role-played independently. </div> </li> <li class="ltx_item" id="S3.I1.i2" style="list-style-type:none;"> (2) <div class="ltx_para" id="S3.I1.i2.p1"> Social Goal Extraction: After obtaining individual scenarios, we extract the social goals of each character, one sentence per goal. We polish the goals further, including rewriting the whole scenario to reduce goal dependencies and rewriting the goals to meet certain criteria (or deleting if that is not possible). </div> </li> <li class="ltx_item" id="S3.I1.i3" style="list-style-type:none;"> (3) <div class="ltx_para" id="S3.I1.i3.p1"> Private Information Extraction: We first identify if any private information exists in the original scene. If yes, we extract questions and answers that only one character can respond to. The rephrased answers are the character’s private information, and the questions serve as implicit reasoning questions for others. We also enhance negative options to be more homogeneous with the correct ones. </div> </li> <li class="ltx_item" id="S3.I1.i4" style="list-style-type:none;"> (4) <div class="ltx_para" id="S3.I1.i4.p1"> Leakage Mitigation and Template Generation: LLMs can identify plots and infer information by recognizing entities like locations and characters. To prevent this, scenario leakage mitigation is implemented using GPT-4o to extract and replace elements linked to specific episodes. The original characters are also replaced by slots. This maintains context while reducing the risk of identifying the plot. </div> </li> </ol> </div> <div class="ltx_para" id="S3.SS2.SSS0.Px1.p3"> More details of the scenario template construction can be found in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS1" title="A.1 Scenario Template Construction Details ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.1</a>. Specifically, we used GPT-4o to automate the construction procedure; Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2" title="A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2</a> lists the prompts we used. </div> </section> <section class="ltx_paragraph" id="S3.SS2.SSS0.Px2"> <h5 class="ltx_title ltx_title_paragraph">Scenario Instantiating</h5> <div class="ltx_para" id="S3.SS2.SSS0.Px2.p1"> We replace the original characters with multiple synthesized agents to prevent character leakage and enrich the social scenarios. A naive method is to replace the original character randomly, which may lead to unrealistic situations like two fifty-year-old students in a middle school. Thus, we dynamically generate agents according to the constraints of the scenario. First, we extract the attributes and relationships of the original characters. Then, we transform these relationships into replacement rules that help define the demographic features of the agents (see Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2.SSS5" title="A.2.5 Agent Synthesizing ‣ A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2.5</a>). Finally, we replace the original characters with agents that adhere to these constraints. After data leakage mitigation, a pre-test in Section <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.SS3" title="4.3 Analysis of Data Leakage ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4.3</a> is conducted to ensure the scenarios remain anonymous. </div> <figure class="ltx_figure" id="S3.F4"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="179" id="S3.F4.g1" src="x4.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 4: (a) Number of scenarios aligned with the eight categories under ERG theory. Each scenario may encompass multiple goals. (b) Moral values distribution of the agents. An individual may have multiple moral values, with those appearing fewer than 30 times categorized as Others. (c) Distribution of the agents’ Big Five personality traits. </figcaption> </figure> </section> </section> <section class="ltx_subsection" id="S3.SS3"> <h3 class="ltx_title ltx_title_subsection"> 3.3 Social Interaction Simulation</h3> <div class="ltx_para" id="S3.SS3.p1"> For each scenario, the agents are given social backgrounds, profiles, and corresponding social goals, along with any private information they may possess, as shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S2.F2" title="Figure 2 ‣ 2.1 Social Intelligence Benchmarks ‣ 2 Related Work ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">2</a>. This setup motivates them to engage in social interactions with other agents. The agents primarily interact through multi-turn conversations, where they can also use language to convey facial expressions or actions. The conversation begins with one of the agents greeting, after which the agents take turns in a random order, rather than following a fixed pre-defined sequence, to restore the dynamics and uncertainty inherent in social interactions. Generally, we set the limit of the turns based on the average number of turns found in the scenes within the scripts, i.e., 15. After the simulation, the generated conversation history will be analyzed to evaluate the goal completion of each agent. Details are in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A3" title="Appendix C Simulation Details ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">C</a>. </div> </section> <section class="ltx_subsection" id="S3.SS4"> <h3 class="ltx_title ltx_title_subsection"> 3.4 Evaluation</h3> <section class="ltx_paragraph" id="S3.SS4.SSS0.Px1"> <h5 class="ltx_title ltx_title_paragraph">Goal Completion</h5> <div class="ltx_para" id="S3.SS4.SSS0.Px1.p1"> Since social goals can be subjective, we judge its completion from three different aspects, as shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S2.F2" title="Figure 2 ‣ 2.1 Social Intelligence Benchmarks ‣ 2 Related Work ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">2</a>: (1) Self: We ask the agent whether it has achieved its goals after interacting with others. (2) Other: We ask other agents in the scenario whether the target agent has achieved his or her specific goals. (3) External: We prompt third-party models with the chat history and ask if an agent has achieved his/her own social goals. We ask the interviewees to respond with yes (goal completed) or no. We take the average across all goals of a character to measure the agent’s overall goal completion level. </div> </section> <section class="ltx_paragraph" id="S3.SS4.SSS0.Px2"> <h5 class="ltx_title ltx_title_paragraph">Implicit Reasoning</h5> <div class="ltx_para" id="S3.SS4.SSS0.Px2.p1"> As mentioned in Section <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS2" title="3.2 Scenario Construction ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.2</a>, each character’s private information corresponds to a multiple-choice evaluation question. To evaluate an agent’s information reasoning ability, we present it with questions related to the private information of other agents within the scenario. We then calculate the average accuracy (Acc) of the current agent on these questions to determine the agent’s score in information reasoning. </div> </section> <section class="ltx_paragraph" id="S3.SS4.SSS0.Px3"> <h5 class="ltx_title ltx_title_paragraph">Profile Sensitivity</h5> <div class="ltx_para" id="S3.SS4.SSS0.Px3.p1"> After character enrichment, each template generates multiple scenarios. By incorporating diverse characters, we not only enrich the scenarios but also gain insights into the stability of social intelligence when simulating different roles. Thus, we propose profile sensitivity index (PSI). We compute the standard deviation (std) of goal/information metrics of the scenario sharing the same template, and the average std across all templates is calculated as PSI. A lower PSI indicates that social intelligence is more stable. </div> </section> </section> <section class="ltx_subsection" id="S3.SS5"> <h3 class="ltx_title ltx_title_subsection"> 3.5 Data Validation and Analysis</h3> <section class="ltx_paragraph" id="S3.SS5.SSS0.Px1"> <h5 class="ltx_title ltx_title_paragraph">Data Source</h5> <div class="ltx_para" id="S3.SS5.SSS0.Px1.p1"> We collect scripts from the Internet Movie Script Database (IMSDb***https://imsdb.com/), an online repository of open-source screenplays for movies and television shows. We use GPT-4o to divide each script into episodes according to discernible shifts in temporal settings, spatial locations, character dynamics, and narrative progressions. Each episode is further divided into scenes based on the variations in the dialogue content, with the prompts detailed in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS2.SSS1" title="A.2.1 Scenario Extraction ‣ A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.2.1</a>. We filter out scenes with fewer than 10 dialogues or those featuring only characters speaking for multiple turns to ensure active interaction between at least two characters. After processing, we have 1,300 scenes, 12,401 rounds of conversations, and 114,834 tokens. The detailed statistics of scripts are provided in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2.SS1" title="B.1 Detailed Statistical Overview of Scripts ‣ Appendix B Data Statistics and Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">B.1</a>. </div> </section> <section class="ltx_paragraph" id="S3.SS5.SSS0.Px2"> <h5 class="ltx_title ltx_title_paragraph">Data Validation</h5> <div class="ltx_para" id="S3.SS5.SSS0.Px2.p1"> We conduct human validation on the generated templates from the automatic pipeline to ensure the quality of our benchmark. For social goals, we mainly consider: (1) whether the goal is achievable by the character, and (2) whether the goal is clear enough to evaluate. For private information, we focus on: (1) awareness by other characters and background information leakage, and (2) sufficiency of information for the character to answer questions. </div> <div class="ltx_para" id="S3.SS5.SSS0.Px2.p2"> The validation involves 6 graduate students in two groups, with each record annotated by 3 annotators. We take the majority vote as the final result when at least two annotators agree. For scenarios where all annotations diverged or were marked as invalid, we assign it to the other three annotators to review it. Details can be found in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.SS3" title="A.3 Data Validation Annotation ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">A.3</a>. </div> </section> <section class="ltx_paragraph" id="S3.SS5.SSS0.Px3"> <h5 class="ltx_title ltx_title_paragraph">Data Analysis</h5> <div class="ltx_para" id="S3.SS5.SSS0.Px3.p1"> Based on the scripts, we develop 245 effective templates by constructing scenarios. Each template is diversified with five scenarios, featuring characters tailored to fit specific scenario constraints. This results in a total of 1,225 scenarios that cover a wide range of social situations and objectives, with 363 of these scenarios containing roles that involve private information. Recognizing the inseparable connection between social interaction and space <cite class="ltx_cite ltx_citemacro_cite">Lefebvre (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib16" title="">1991</a>)</cite>, we categorize these scenarios into three types: personal domain, small society, and large society. A detailed introduction to the classification is shown in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2.SS2" title="B.2 Social Scenario Categories ‣ Appendix B Data Statistics and Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">B.2</a>. In our dataset, 54% of scenarios are in the personal domain, 37% in small society, and 9% in large society. </div> <div class="ltx_para" id="S3.SS5.SSS0.Px3.p2"> For social goals, we apply the ERG theory <cite class="ltx_cite ltx_citemacro_cite">Alderfer (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib2" title="">1969</a>)</cite> to categorize these goals into three hierarchical needs: Existence, Relatedness, and Growth. Based on these needs, we identify eight key social goals(e.g., Information Acquisition, Relationship Building, Competition), detailed in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2.SS3" title="B.3 Social Goal Classification ‣ Appendix B Data Statistics and Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">B.3</a>. Our dataset includes all these social goals, with the number of scenarios for each shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.F4" title="Figure 4 ‣ Scenario Instantiating ‣ 3.2 Scenario Construction ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4</a> (a). Generally, higher-level social goals are less frequent, reflecting real-life patterns. </div> <div class="ltx_para" id="S3.SS5.SSS0.Px3.p3"> The final dataset contains a diverse collection of 859 individual profiles with 366 types of occupations. The characters exhibit a wide range of attributes, emphasizing the great diversity in terms of gender, age, occupation, big five, moral values, personality, and decision-making styles, as shown in Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.F4" title="Figure 4 ‣ Scenario Instantiating ‣ 3.2 Scenario Construction ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4</a> (b) and Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.F4" title="Figure 4 ‣ Scenario Instantiating ‣ 3.2 Scenario Construction ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4</a> (c). For instance, in terms of gender, there are individuals identified as male, female, non-binary, genderqueer, and so on. Ages vary from childhood to old age, providing a spread across different life stages. </div> </section> </section> </section> <section class="ltx_section" id="S4"> <h2 class="ltx_title ltx_title_section"> 4 Experiments</h2> <section class="ltx_subsection" id="S4.SS1"> <h3 class="ltx_title ltx_title_subsection"> 4.1 Experimental Settings</h3> <section class="ltx_paragraph" id="S4.SS1.SSS0.Px1"> <h5 class="ltx_title ltx_title_paragraph">Agent Models</h5> <div class="ltx_para" id="S4.SS1.SSS0.Px1.p1"> We evaluate various LLM families including Llama-2-7b/13b/70b-Chat <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib33" title="">2023</a>)</cite>, Llama-3-8b/70b-Instruct <cite class="ltx_cite ltx_citemacro_cite">Dubey et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib9" title="">2024</a>)</cite>, Mistral-7b-Instruct-v0.3 <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib13" title="">2023</a>)</cite>, Qwen2.5-7b/14b/72b-Chat <cite class="ltx_cite ltx_citemacro_cite">Team (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib32" title="">2024</a>)</cite>, GPT-3.5-Turbo <cite class="ltx_cite ltx_citemacro_cite">Ouyang et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib23" title="">2022</a>)</cite> and GPT-4o <cite class="ltx_cite ltx_citemacro_cite">Achiam et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib1" title="">2023</a>)</cite>. For interactions between different models and further analysis, we involve Llama-3-8b, Qwen2.5-14b, GPT-3.5-turbo, and GPT-4o. </div> </section> <section class="ltx_paragraph" id="S4.SS1.SSS0.Px2"> <h5 class="ltx_title ltx_title_paragraph">Judge Models</h5> <div class="ltx_para" id="S4.SS1.SSS0.Px2.p1"> To effectively leverage current LLMs for automating the evaluation of social interactions as third-party judges, we conduct a human evaluation and compare it with model-based evaluations. We randomly sample 100 simulated scenarios, posing a total of 584 social goal evaluation questions, and manually verify whether the communication history shows that the goals were achieved. Using human annotations as ground-truth labels, we calculate the accuracy of various models acting as judges, as shown in Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.T6" title="Table 6 ‣ D.1 Human Evaluation of Goal Completion ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">6</a>. Given the results, we select GPT-4o, Qwen2.5-72b, and Llama-3-70b from different model families as our judge models, referred to as Judge-GPT-4o, Qwen2.5, and Llama-3 hereafter. Additionally, we incorporate a majority voting mechanism across these models to create a mixture-of-experts model as another judge. </div> </section> <section class="ltx_paragraph" id="S4.SS1.SSS0.Px3"> <h5 class="ltx_title ltx_title_paragraph">Implementation Details</h5> <div class="ltx_para" id="S4.SS1.SSS0.Px3.p1"> We use vLLM <cite class="ltx_cite ltx_citemacro_citep">(Kwon et al., <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib15" title="">2023</a>)</cite> to deploy all open source models: Qwen2.5-72b and Llama-3-70b on NVIDIA A100, and other models on NVIDIA RTX 4090. We apply AutoGen <cite class="ltx_cite ltx_citemacro_cite">Wu et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib37" title="">2024</a>)</cite> to manage interacting and judging threads. We set max new tokens to 128 for all models. Temperature is set to 1 for agent models to encourage diversity, and 0 for judge models to ensure the stability of evaluation. </div> </section> </section> <section class="ltx_subsection" id="S4.SS2"> <h3 class="ltx_title ltx_title_subsection"> 4.2 Overall Performance</h3> <figure class="ltx_table" id="S4.T1"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S4.T1.2" style="width:433.6pt;height:209.6pt;vertical-align:-0.8pt;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="S4.T1.2.2"> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="S4.T1.2.2.3.1"> <th class="ltx_td ltx_th ltx_th_row ltx_border_tt" id="S4.T1.2.2.3.1.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"></th> <td class="ltx_td ltx_align_center ltx_border_tt" colspan="8" id="S4.T1.2.2.3.1.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Goal</td> <td class="ltx_td ltx_align_center ltx_border_tt" colspan="2" id="S4.T1.2.2.3.1.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Info</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.4.2"> <th class="ltx_td ltx_th ltx_th_row" id="S4.T1.2.2.4.2.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"></th> <td class="ltx_td ltx_border_t" id="S4.T1.2.2.4.2.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"></td> <td class="ltx_td ltx_border_t" id="S4.T1.2.2.4.2.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"></td> <td class="ltx_td ltx_align_center ltx_border_t" colspan="6" id="S4.T1.2.2.4.2.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Judge</td> <td class="ltx_td ltx_border_t" id="S4.T1.2.2.4.2.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"></td> <td class="ltx_td ltx_nopad_r ltx_border_t" id="S4.T1.2.2.4.2.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"></td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.2"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.2.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Models</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.2.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Self</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.2.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Other</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.2.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">GPT-4o</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.2.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Qwen2.5</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.2.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Llama-3</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.2.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Average</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.2.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Majority</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.1.1.1.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">PSI<math alttext="\downarrow" class="ltx_Math" display="inline" id="S4.T1.1.1.1.1.m1.1"><semantics id="S4.T1.1.1.1.1.m1.1a"><mo id="S4.T1.1.1.1.1.m1.1.1" stretchy="false" xref="S4.T1.1.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.1.m1.1b"><ci id="S4.T1.1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.1.m1.1c">\downarrow</annotation><annotation encoding="application/x-llamapun" id="S4.T1.1.1.1.1.m1.1d">↓</annotation></semantics></math> </td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.2.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Acc.</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.2.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">PSI<math alttext="\downarrow" class="ltx_Math" display="inline" id="S4.T1.2.2.2.2.1.m1.1"><semantics id="S4.T1.2.2.2.2.1.m1.1a"><mo id="S4.T1.2.2.2.2.1.m1.1.1" stretchy="false" xref="S4.T1.2.2.2.2.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.2.2.1.m1.1b"><ci id="S4.T1.2.2.2.2.1.m1.1.1.cmml" xref="S4.T1.2.2.2.2.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.2.2.1.m1.1c">\downarrow</annotation><annotation encoding="application/x-llamapun" id="S4.T1.2.2.2.2.1.m1.1d">↓</annotation></semantics></math></td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.5.3"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" id="S4.T1.2.2.5.3.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Llama-2-7b</th> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">83.38</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">62.70</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">52.73</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">57.68</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">55.37</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">55.26</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">55.84</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">21.94</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">33.06</td> <td class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" id="S4.T1.2.2.5.3.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">20.53</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.6.4"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.6.4.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Llama-2-13b</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">48.01</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">10.26</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">17.38</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">30.11</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">72.19</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">39.90</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">30.91</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">21.84</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.6.4.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">28.56</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.6.4.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">18.39</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.7.5"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.7.5.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Llama-2-70b</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">85.72</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">65.65</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">33.78</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">42.37</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">73.80</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">49.98</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">45.53</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">22.31</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.7.5.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">36.78</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.7.5.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">18.60</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.8.6"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.8.6.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Llama-3-8B</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">87.63</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">67.28</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">79.90</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">82.55</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">75.10</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">79.18</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">80.71</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">12.85</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.8.6.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">69.68</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.8.6.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">15.14</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.9.7"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.9.7.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Llama-3-70b</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">80.38</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">77.27</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">86.22</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">87.61</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">79.88</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">84.57</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">86.27</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">8.92</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.9.7.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">73.08</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.9.7.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">16.58</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.10.8"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.10.8.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Qwen2.5-7b</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">86.17</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">61.92</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">77.07</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">79.30</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">71.99</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">76.12</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">77.37</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">13.10</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.10.8.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">74.82</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.10.8.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">15.84</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.11.9"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.11.9.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Qwen2.5-14b</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">86.62</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">84.17</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul88.43</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">89.83</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul80.47</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul86.24</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul88.14</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul8.09</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.11.9.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">75.02</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.11.9.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul14.81</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.12.10"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.12.10.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Qwen2.5-72b</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul90.67</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">85.89</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">88.29</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul89.03</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">78.57</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">85.30</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">87.74</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">8.19</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.12.10.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul76.05</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.12.10.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">13.57</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.13.11"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.13.11.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">Mistral-7b</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">95.22</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">87.25</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">79.29</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">84.13</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">77.82</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">80.41</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">82.37</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">12.39</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.13.11.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">66.59</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.13.11.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">18.55</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.14.12"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="S4.T1.2.2.14.12.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">GPT-3.5-turbo</th> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">90.16</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">76.62</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">82.12</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">84.37</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">77.30</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">81.26</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">82.64</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">10.01</td> <td class="ltx_td ltx_align_center" id="S4.T1.2.2.14.12.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">68.41</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="S4.T1.2.2.14.12.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">18.37</td> </tr> <tr class="ltx_tr" id="S4.T1.2.2.15.13"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" id="S4.T1.2.2.15.13.1" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">GPT-4o</th> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.2" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">88.46</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.3" style="padding-top:-0.5pt;padding-bottom:-0.5pt;"> \ul86.29</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.4" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">88.47</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.5" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">89.00</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.6" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">81.57</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.7" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">86.34</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.8" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">88.36</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.9" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">6.99</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.10" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">76.86</td> <td class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb" id="S4.T1.2.2.15.13.11" style="padding-top:-0.5pt;padding-bottom:-0.5pt;">15.48</td> </tr> </tbody> </table> </div> <figcaption class="ltx_caption ltx_centering">Table 1: Overall performance of the interactions of agents driven by the same models. We report the best performance in bold format and the second best in \ulunderlined format. </figcaption> </figure> <section class="ltx_paragraph" id="S4.SS2.SSS0.Px1"> <h5 class="ltx_title ltx_title_paragraph">Single Model-based</h5> <div class="ltx_para" id="S4.SS2.SSS0.Px1.p1"> Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.T1" title="Table 1 ‣ 4.2 Overall Performance ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">1</a> shows the overall performance of the interaction of agents driven by the same models. Considering that LLMs may overestimate their own performance, we use the judge majority score as the primary metric for cross-model comparisons, as it is more objective and stable than other metrics. </div> <div class="ltx_para" id="S4.SS2.SSS0.Px1.p2"> \ul Overall Performance: GPT-4o leads as expected, while Qwen-series models also show strong social intelligence, especially for Qwen2.5-14b, in both goal completion and information reasoning. Llama-2 series models perform poorly, with some improvement in the Llama-3 series, though still falling short of expectations. The interaction history of Llama-2-13b in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS7" title="D.7 Under-performing Cases of Llama-2-7b ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.7</a> reveals frequent struggles in maintaining roles, progressing conversations, and responding effectively to others. In terms of the stability of social intelligence, excluding the uncertainty introduced by the temperature parameter (Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS2" title="D.2 Single Model-based Subset Experiment ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.2</a>), the PSI results show that models with higher social intelligence, such as GPT-4o and Qwen, are also less sensitive to profile changes. Overall, different models’ social abilities are well distinguished by AgentSense. Meanwhile, we observe that there still exists an improvement space even for the SOTA models, emphasizing LLMs still face challenges in diverse and complex social scenarios. </div> <div class="ltx_para" id="S4.SS2.SSS0.Px1.p3"> \ul Evaluation Bias in Goal Completion: Llama-2-7b and Mistral-7b tend to overestimate themselves during the simulation, which can be told from the Self and Other scores as the judges are powered by the same models as the social agents. Judges also exhibit specific preference, with Qwen2.5-72b tending to prefer Qwen-series models and GPT-4o tending to prefer GPT-4o. Llama3-70b tends to be conservative in judging both self and others. </div> <figure class="ltx_figure" id="S4.F5"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="356" id="S4.F5.g1" src="x5.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 5: (a) Judge majority score of interactions among different model-driven agents, highlighting that being a sender is more challenging. (b) Model performance as both attacker and defender, with notably weaker and less consistent results when acting as a defender.</figcaption> </figure> </section> <section class="ltx_paragraph" id="S4.SS2.SSS0.Px2"> <h5 class="ltx_title ltx_title_paragraph">Pairwise Model-based</h5> <div class="ltx_para" id="S4.SS2.SSS0.Px2.p1"> We also evaluate how agents perform when interacting with other agents supported by different models. Given that our social scenarios can have more than two participants, we label each agent as either a sender or a receiver based on their social goals with the assistance of GPT-4o, inspired by the theory of communication <cite class="ltx_cite ltx_citemacro_cite">Blau (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib5" title="">1964</a>); Barnlund (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib4" title="">2017</a>)</cite>. Senders share and transmit information, while receivers focus on understanding and responding. </div> <div class="ltx_para" id="S4.SS2.SSS0.Px2.p2"> Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.F5" title="Figure 5 ‣ Single Model-based ‣ 4.2 Overall Performance ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">5</a>(a) presents the overall results of such interactions. GPT-4o and Qwen2.5-14b still perform best. However, engaging with weaker models adversely affects all models’ performance, particularly when the sender is the weaker agent. Our analysis (Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS4" title="D.4 Pairwise Model-based Additional Results ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.4</a>) shows that weaker models struggle more as senders than as receivers This is because senders take a more active role in social interactions, making the associated tasks inherently more challenging. </div> <figure class="ltx_figure" id="S4.F6"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="197" id="S4.F6.g1" src="x6.png" width="789"/> <figcaption class="ltx_caption ltx_centering">Figure 6: Goal completion scores of different goal types across models. LLMs perform best on relationship goals and cooperation, and worst on competition and conflict resolution (where they are also over-confident).</figcaption> </figure> </section> </section> <section class="ltx_subsection" id="S4.SS3"> <h3 class="ltx_title ltx_title_subsection"> 4.3 Analysis of Data Leakage</h3> <figure class="ltx_table" id="S4.T2"> <table class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle" id="S4.T2.1"> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="S4.T2.1.1.1"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_tt" id="S4.T2.1.1.1.1">Model</th> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" id="S4.T2.1.1.1.2">Script Acc.</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S4.T2.1.1.1.3">Blind Acc.</td> </tr> <tr class="ltx_tr" id="S4.T2.1.2.2"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t" id="S4.T2.1.2.2.1">GPT-4o</th> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S4.T2.1.2.2.2">0.04</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.1.2.2.3">0.62</td> </tr> <tr class="ltx_tr" id="S4.T2.1.3.3"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r" id="S4.T2.1.3.3.1">GPT-3.5-turbo</th> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T2.1.3.3.2">0.07</td> <td class="ltx_td ltx_align_center" id="S4.T2.1.3.3.3">0.51</td> </tr> <tr class="ltx_tr" id="S4.T2.1.4.4"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t" id="S4.T2.1.4.4.1">Mistral-7b</th> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S4.T2.1.4.4.2">0.05</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S4.T2.1.4.4.3">0.56</td> </tr> <tr class="ltx_tr" id="S4.T2.1.5.5"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r" id="S4.T2.1.5.5.1">Llama-2-7b</th> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T2.1.5.5.2">0.06</td> <td class="ltx_td ltx_align_center" id="S4.T2.1.5.5.3">0.35</td> </tr> <tr class="ltx_tr" id="S4.T2.1.6.6"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r" id="S4.T2.1.6.6.1">Qwen2.5-7b</th> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T2.1.6.6.2">0.03</td> <td class="ltx_td ltx_align_center" id="S4.T2.1.6.6.3">0.55</td> </tr> <tr class="ltx_tr" id="S4.T2.1.7.7"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r" id="S4.T2.1.7.7.1">Llama-3-8b</th> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T2.1.7.7.2">0.04</td> <td class="ltx_td ltx_align_center" id="S4.T2.1.7.7.3">0.54</td> </tr> <tr class="ltx_tr" id="S4.T2.1.8.8"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r" id="S4.T2.1.8.8.1">Llama-2-13b</th> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T2.1.8.8.2">0.04</td> <td class="ltx_td ltx_align_center" id="S4.T2.1.8.8.3">0.35</td> </tr> <tr class="ltx_tr" id="S4.T2.1.9.9"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r" id="S4.T2.1.9.9.1">Qwen2.5-14b</th> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T2.1.9.9.2">0.06</td> <td class="ltx_td ltx_align_center" id="S4.T2.1.9.9.3">0.61</td> </tr> <tr class="ltx_tr" id="S4.T2.1.10.10"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r" id="S4.T2.1.10.10.1">Llama-2-70b</th> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T2.1.10.10.2">0.06</td> <td class="ltx_td ltx_align_center" id="S4.T2.1.10.10.3">0.40</td> </tr> <tr class="ltx_tr" id="S4.T2.1.11.11"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r" id="S4.T2.1.11.11.1">Llama-3-70b</th> <td class="ltx_td ltx_align_center ltx_border_r" id="S4.T2.1.11.11.2">0.04</td> <td class="ltx_td ltx_align_center" id="S4.T2.1.11.11.3">0.59</td> </tr> <tr class="ltx_tr" id="S4.T2.1.12.12"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r" id="S4.T2.1.12.12.1">Qwen2.5-72b</th> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" id="S4.T2.1.12.12.2">0.04</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S4.T2.1.12.12.3">0.60</td> </tr> </tbody> </table> <figcaption class="ltx_caption ltx_centering">Table 2: Model performance on script prediction and blind test. The low Script Acc. indicates the model barely discerns the scripts, and the Blind Acc. establishes a baseline for model’s reasoning ability.</figcaption> </figure> <div class="ltx_para" id="S4.SS3.p1"> Since our data originates from scripts, it is essential to prevent data leakage and evaluate the effectiveness of leakage mitigation. Data leakage may arise from 1) the model’s prior knowledge of the scenario’s script, and 2) the information provided by the scenario that could help infer others’ private information. We quantify this risk through two experiments: (1) Script Prediction: Whether models can guess the original scripts from background information, with 245 test samples (one per template). (2) Blind Test: Whether models can answer private information reasoning questions with initial scenario information before interactions, with 100 test questions asked three times. </div> <div class="ltx_para" id="S4.SS3.p2"> The script prediction results in Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.T2" title="Table 2 ‣ 4.3 Analysis of Data Leakage ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">2</a> indicate that models are nearly unable to infer the original script from the background information, proving the effectiveness of leakage mitigation. The blind test results also establish a baseline for each model’s private information reasoning ability. </div> </section> </section> <section class="ltx_section" id="S5"> <h2 class="ltx_title ltx_title_section"> 5 Further Analysis</h2> <section class="ltx_subsection" id="S5.SS1"> <h3 class="ltx_title ltx_title_subsection"> 5.1 What goals are LLMs good/bad at?</h3> <div class="ltx_para" id="S5.SS1.p1"> In Section <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS5" title="3.5 Data Validation and Analysis ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.5</a>, we categorize all social goals into 8 types under ERG theory. Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.F6" title="Figure 6 ‣ Pairwise Model-based ‣ 4.2 Overall Performance ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">6</a> illustrates the average goal completion scores of each goal type across different models. In general, all LLMs are good at goals about relationship management and cooperation. Compared with smaller models like Llama-3-8b, larger models like Qwen2.5-14b and GPT-4o gain significant improvement on goals about information exchange and identity recognition. However, there is still room for improvement on other goals like competition and conflict resolution. These are also the goals where LLMs tend to overestimate their progress. </div> <div class="ltx_para" id="S5.SS1.p2"> We also compare goal completion scores under different scenario types, number of interaction rounds, and participants, where less or no significant difference regarding these factors is observed. More details can be found in Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS5" title="D.5 Goal Completion Score Additional Results ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.5</a>. </div> </section> <section class="ltx_subsection" id="S5.SS2"> <h3 class="ltx_title ltx_title_subsection"> 5.2 Which is harder: guessing thoughts or keeping secrets?</h3> <div class="ltx_para" id="S5.SS2.p1"> We further inspect the disparities in passive reasoning among different models. In interactive scenarios, agents with private information engage in a game where the defender aims to keep their information confidential, while the attacker seeks to uncover it, thus playing two distinct roles: (1) Defenders need to prevent the disclosure of their private information. Their effectiveness is inversely proportional to the attacker’s success in inferring this information, or 1 minus the attacker’s inference success rate. (2) Attackers need to acquire others’ private information. Their strength is simply proportional to their own inference success rate. </div> <div class="ltx_para" id="S5.SS2.p2"> The benchmark reveals that these roles represent distinct capabilities. Models may act as both attackers and defenders during interactions. To assess the models’ abilities in these roles, we analyze the outcomes of their interactions. </div> <div class="ltx_para" id="S5.SS2.p3"> Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.F5" title="Figure 5 ‣ Single Model-based ‣ 4.2 Overall Performance ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">5</a>(b) outlines model performance. Most models lack in defense, often revealing secrets and showing unstable performance across scenarios. GPT-4o excels at both keeping secrets and passive deduction. Qwen2.5-14b is aggressive and adept at inferring information, while Llama-3-8b is more conservative and better at keeping secrets. </div> </section> <section class="ltx_subsection" id="S5.SS3"> <h3 class="ltx_title ltx_title_subsection"> 5.3 What scenarios are more sensitive to profiles?</h3> <div class="ltx_para" id="S5.SS3.p1"> As mentioned in Sec <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS2" title="3.2 Scenario Construction ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.2</a>, we enrich the scenario by replacing the original characters in the script with synthesized agents. Here we investigate the impact of profiles on social intelligence. We use the Chi-square test to identify abnormal templates (p-value =0.05, see Appendix <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.SS6" title="D.6 Experiment on Profiles’ Effect on Scenarios ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">D.6</a>). The results show that the profile replacement and character enrichment following the above workflow satisfies null hypothesis <math alttext="H_{0}" class="ltx_Math" display="inline" id="S5.SS3.p1.1.m1.1"><semantics id="S5.SS3.p1.1.m1.1a"><msub id="S5.SS3.p1.1.m1.1.1" xref="S5.SS3.p1.1.m1.1.1.cmml"><mi id="S5.SS3.p1.1.m1.1.1.2" xref="S5.SS3.p1.1.m1.1.1.2.cmml">H</mi><mn id="S5.SS3.p1.1.m1.1.1.3" xref="S5.SS3.p1.1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.1.m1.1b"><apply id="S5.SS3.p1.1.m1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS3.p1.1.m1.1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1">subscript</csymbol><ci id="S5.SS3.p1.1.m1.1.1.2.cmml" xref="S5.SS3.p1.1.m1.1.1.2">𝐻</ci><cn id="S5.SS3.p1.1.m1.1.1.3.cmml" type="integer" xref="S5.SS3.p1.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.1.m1.1c">H_{0}</annotation><annotation encoding="application/x-llamapun" id="S5.SS3.p1.1.m1.1d">italic_H start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT</annotation></semantics></math> (namely do not have significant difference) in over 92.6% scenarios. </div> <div class="ltx_para" id="S5.SS3.p2"> We conduct a human evaluation for the rest 7.4% abnormal scenarios that have a significant difference to locate the key factor disturbing the robustness. The results unveil that: (1) some social goals involve opinions or behaviors that are against the universal value (like smoking prohibition and emotional outburst), which can be influenced by both the agent’s personality and LLM’s alignment; (2) some scenarios require detailed personal level information, making the general replacement workflow fail, especially for emotional issues involving multiple characters. </div> </section> </section> <section class="ltx_section" id="S6"> <h2 class="ltx_title ltx_title_section"> 6 Conclusion</h2> <div class="ltx_para" id="S6.p1"> In this paper, we introduce AgentSense, a benchmark evaluating LLM’s social intelligence via diverse and challenging social environments. Extensive experiments reveal that current LLMs struggle with complex social scenarios and high-level goals. Further analyses verify potential influence factors during evaluation to show the robustness and discriminative power of AgentSense. </div> </section> <section class="ltx_section" id="Sx1"> <h2 class="ltx_title ltx_title_section">Limitations</h2> <div class="ltx_para" id="Sx1.p1"> Although we have automated the scenario extraction process as much as possible, challenges in obtaining valid social goals and private information still require manual validation at certain stages. As a result, we have not expanded the scenarios to a larger scale, leaving it for future work. </div> </section> <section class="ltx_section" id="Sx2"> <h2 class="ltx_title ltx_title_section">Ethics Statement</h2> <div class="ltx_para" id="Sx2.p1"> AgentSense is introduced to assess the social intelligence of LLM-driven agents. We do not encourage any agents that might disrupt social norms. We aim to offer insights that enhance LLMs’ performance in complex social scenarios and promote effective, harmonious interactions among agents and between agents and humans. Besides, constructing role-playing agents can lead to anthropomorphism, resulting in unrealistic expectations, potential manipulation, and negative consequences. However, in AgentSense, we avoid having LLMs role-play specific individuals and instead portray various synthesized characters across different scenarios. For annotations, we paid the annotators according to the graduate wage standards of their respective countries. </div> </section> <section class="ltx_bibliography" id="bib"> <h2 class="ltx_title ltx_title_bibliography">References</h2> <ul class="ltx_biblist"> <li class="ltx_bibitem" id="bib.bib1"> Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. </li> <li class="ltx_bibitem" id="bib.bib2"> Alderfer (1969) Clayton P. Alderfer. 1969. <a class="ltx_ref ltx_href" href="https://doi.org/10.1016/0030-5073(69)90004-X" title="">An empirical test of a new theory of human needs</a>. Organizational Behavior and Human Performance, 4(2):142–175. </li> <li class="ltx_bibitem" id="bib.bib3"> Argyle et al. (2023) Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351. </li> <li class="ltx_bibitem" id="bib.bib4"> Barnlund (2017) Dean C Barnlund. 2017. A transactional model of communication. In Communication theory, pages 47–57. Routledge. </li> <li class="ltx_bibitem" id="bib.bib5"> Blau (1964) Peter M Blau. 1964. Exchange and power in social life. new york: John wiley. </li> <li class="ltx_bibitem" id="bib.bib6"> Chen et al. (2024a) Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, and Jingren Zhou. 2024a. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/2403.13679" title="">Socialbench: Sociality evaluation of role-playing conversational agents</a>. Preprint, arXiv:2403.13679. </li> <li class="ltx_bibitem" id="bib.bib7"> Chen et al. (2024b) Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. 2024b. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/2404.18231" title="">From persona to personalization: A survey on role-playing language agents</a>. Preprint, arXiv:2404.18231. </li> <li class="ltx_bibitem" id="bib.bib8"> Chuang et al. (2024) Yun-Shiuan Chuang, Nikunj Harlalka, Siddharth Suresh, Agam Goyal, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T Rogers. 2024. The wisdom of partisan crowds: Comparing collective intelligence in humans and llm-based agents. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46. </li> <li class="ltx_bibitem" id="bib.bib9"> Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. </li> <li class="ltx_bibitem" id="bib.bib10"> Fu et al. (2023) Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. 2023. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/2305.10142" title="">Improving language model negotiation with self-play and in-context learning from ai feedback</a>. Preprint, arXiv:2305.10142. </li> <li class="ltx_bibitem" id="bib.bib11"> Goffman (1959) Erving Goffman. 1959. The presentation of self in everyday life. </li> <li class="ltx_bibitem" id="bib.bib12"> Grossmann et al. (2023) Igor Grossmann, Matthew Feinberg, Dawn C. Parker, Nicholas A. Christakis, Philip E. Tetlock, and William A. Cunningham. 2023. <a class="ltx_ref ltx_href" href="https://doi.org/10.1126/science.adi1778" title="">Ai and the transformation of social science research</a>. Science, 380(6650):1108–1109. </li> <li class="ltx_bibitem" id="bib.bib13"> Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. </li> <li class="ltx_bibitem" id="bib.bib14"> Jiang et al. (2024) Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2024. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36. </li> <li class="ltx_bibitem" id="bib.bib15"> Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. </li> <li class="ltx_bibitem" id="bib.bib16"> Lefebvre (1991) Henri Lefebvre. 1991. The Production of Space. The Production of Space. </li> <li class="ltx_bibitem" id="bib.bib17"> Li et al. (2023a) Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023a. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008. </li> <li class="ltx_bibitem" id="bib.bib18"> Li et al. (2023b) Yuan Li, Yixuan Zhang, and Lichao Sun. 2023b. Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents. arXiv preprint arXiv:2310.06500. </li> <li class="ltx_bibitem" id="bib.bib19"> Liu et al. (2023) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/2308.03688" title="">Agentbench: Evaluating llms as agents</a>. Preprint, arXiv:2308.03688. </li> <li class="ltx_bibitem" id="bib.bib20"> Liu et al. (2024) Yuhan Liu, Xiuying Chen, Xiaoqing Zhang, Xing Gao, Ji Zhang, and Rui Yan. 2024. From skepticism to acceptance: Simulating the attitude dynamics toward fake news. arXiv preprint arXiv:2403.09498. </li> <li class="ltx_bibitem" id="bib.bib21"> Maslow (1943) Abraham Harold Maslow. 1943. A theory of human motivation. Psychological Review, 50:370. </li> <li class="ltx_bibitem" id="bib.bib22"> Mou et al. (2024) Xinyi Mou, Zhongyu Wei, and Xuanjing Huang. 2024. Unveiling the truth and facilitating change: Towards agent-based large-scale social movement simulation. In Findings of the Association for Computational Linguistics ACL 2024, pages 4789–4809. </li> <li class="ltx_bibitem" id="bib.bib23"> Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. </li> <li class="ltx_bibitem" id="bib.bib24"> Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. </li> <li class="ltx_bibitem" id="bib.bib25"> Sabour et al. (2024) Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M Liu, Jinfeng Zhou, Alvionna S Sunaryo, Juanzi Li, Tatia Lee, Rada Mihalcea, and Minlie Huang. 2024. Emobench: Evaluating the emotional intelligence of large language models. arXiv preprint arXiv:2402.12071. </li> <li class="ltx_bibitem" id="bib.bib26"> Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/1904.09728" title="">Socialiqa: Commonsense reasoning about social interactions</a>. Preprint, arXiv:1904.09728. </li> <li class="ltx_bibitem" id="bib.bib27"> Schick et al. (2024) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36. </li> <li class="ltx_bibitem" id="bib.bib28"> Shao et al. (2023) Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/2310.10158" title="">Character-llm: A trainable agent for role-playing</a>. Preprint, arXiv:2310.10158. </li> <li class="ltx_bibitem" id="bib.bib29"> Shapira et al. (2023) Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2023. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763. </li> <li class="ltx_bibitem" id="bib.bib30"> Shinn et al. (2024) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36. </li> <li class="ltx_bibitem" id="bib.bib31"> Stade et al. (2024) Elizabeth Stade, Shannon Stirman, Lyle Ungar, Cody Boland, H. Schwartz, David Yaden, João Sedoc, Robert DeRubeis, Robb Willer, and Johannes Eichstaedt. 2024. <a class="ltx_ref ltx_href" href="https://doi.org/10.1038/s44184-024-00056-z" title="">Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation</a>. npj Mental Health Research, 3. </li> <li class="ltx_bibitem" id="bib.bib32"> Team (2024) Qwen Team. 2024. <a class="ltx_ref ltx_href" href="https://qwenlm.github.io/blog/qwen2.5/" title="">Qwen2.5: A party of foundation models</a>. </li> <li class="ltx_bibitem" id="bib.bib33"> Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. </li> <li class="ltx_bibitem" id="bib.bib34"> Walker and Foley (1973) Ronald E. Walker and Jeanne M. Foley. 1973. <a class="ltx_ref ltx_href" href="https://doi.org/10.2466/pr0.1973.33.3.839" title="">Social intelligence: Its history and measurement</a>. Psychological Reports, 33(3):839–864. </li> <li class="ltx_bibitem" id="bib.bib35"> Wang et al. (2024) Chenxu Wang, Bin Dai, Huaping Liu, and Baoyuan Wang. 2024. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/2404.05337" title="">Towards objectively benchmarking social intelligence for language agents at action level</a>. Preprint, arXiv:2404.05337. </li> <li class="ltx_bibitem" id="bib.bib36"> Wilf et al. (2023) Alex Wilf, Leena Mathur, Sheryl Mathew, Claire Ko, Youssouf Kebe, Paul Pu Liang, and Louis-Philippe Morency. 2023. Social-iq 2.0 challenge: Benchmarking multimodal social understanding. </li> <li class="ltx_bibitem" id="bib.bib37"> Wu et al. (2024) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. Autogen: Enabling next-gen llm applications via multi-agent conversation. In ICLR 2024 Workshop on Large Language Model (LLM) Agents. </li> <li class="ltx_bibitem" id="bib.bib38"> Xie et al. (2024) Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Kai Shu, Adel Bibi, Ziniu Hu, Philip Torr, Bernard Ghanem, and Guohao Li. 2024. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/2402.04559" title="">Can large language model agents simulate human trust behaviors?</a> Preprint, arXiv:2402.04559. </li> <li class="ltx_bibitem" id="bib.bib39"> Xu et al. (2023) Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658. </li> <li class="ltx_bibitem" id="bib.bib40"> Zadeh et al. (2019) Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). </li> <li class="ltx_bibitem" id="bib.bib41"> Zhou et al. (2024a) Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, and Maarten Sap. 2024a. Is this the real life? is this just fantasy? the misleading success of simulating social interactions with llms. arXiv preprint arXiv:2403.05020. </li> <li class="ltx_bibitem" id="bib.bib42"> Zhou et al. (2024b) Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. 2024b. <a class="ltx_ref ltx_href" href="https://arxiv.org/abs/2310.11667" title="">Sotopia: Interactive evaluation for social intelligence in language agents</a>. Preprint, arXiv:2310.11667. </li> </ul> </section> <section class="ltx_appendix" id="A1"> <h2 class="ltx_title ltx_title_appendix"> Appendix A Scenario Construction</h2> <section class="ltx_subsection" id="A1.SS1"> <h3 class="ltx_title ltx_title_subsection"> A.1 Scenario Template Construction Details</h3> <section class="ltx_subsubsection" id="A1.SS1.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> A.1.1 Social Goal Extraction</h4> <div class="ltx_para" id="A1.SS1.SSS1.p1"> The first step of social goal extraction is to use GPT-4o to extract the social goals of the current character. However, this direct approach has two issues that requires further amendment: </div> <div class="ltx_para" id="A1.SS1.SSS1.p2"> <ul class="ltx_itemize" id="A1.I1"> <li class="ltx_item" id="A1.I1.i1" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I1.i1.p1"> The goals may be too detailed (e.g. direct instruction), or depend on other characters’ goals or some random events during the scenario. To address this problem, we prompt GPT-4o to rewrite the whole scenario, including all extracted social goals, into a similar but new scenario that avoids these issues to the best extent. </div> </li> <li class="ltx_item" id="A1.I1.i2" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I1.i2.p1"> Even after the conversion, some goals remain unattainable for characters, such as those requiring external information, physical actions, or evaluations beyond the scenario, e.g., goals that are too abstract or long-term. Thus, we ask GPT-4o to determine whether each goal matches the above cases. If it does, we instruct it to either rewrite the goal or delete it if rewriting is not feasible. </div> </li> </ul> </div> </section> <section class="ltx_subsubsection" id="A1.SS1.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> A.1.2 Private Information Extraction</h4> <div class="ltx_para" id="A1.SS1.SSS2.p1"> The pipeline first determines whether the current scene involves the inference of private information. If it does, the pipeline proceeds to extract private information for each character from the script, otherwise the private information will be set to null. We use GPT-4o to perform reading comprehension on the scenes, extracting questions and answers that only one agent can respond to. The rephrased answers serve as the agent’s private information, while the questions serve as test items for other agents. Finally, the pipeline generates three incorrect answer options, which, along with the original Q&A pair, form the complete evaluation question. To better assess passive reasoning abilities, we have rephrased the negative options to make them more homogeneous with the correct options. </div> </section> <section class="ltx_subsubsection" id="A1.SS1.SSS3"> <h4 class="ltx_title ltx_title_subsubsection"> A.1.3 Leakage Mitigation and Template Generation</h4> <div class="ltx_para" id="A1.SS1.SSS3.p1"> To prevent data leakage, we first perform scenario leakage mitigation. We prompt GPT-4o to extract elements associated with specific episodes, such as notable location entities like "The Facebook headquarters." Next, we ask GPT-4o to suggest replacement candidates to preserve the script’s context and minimize the risk of identifying the specific episode. </div> </section> </section> <section class="ltx_subsection" id="A1.SS2"> <h3 class="ltx_title ltx_title_subsection"> A.2 Prompts in Construction Pipeline</h3> <section class="ltx_subsubsection" id="A1.SS2.SSS1"> <h4 class="ltx_title ltx_title_subsubsection"> A.2.1 Scenario Extraction</h4> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS1.p1"> <svg class="ltx_picture" height="3016.35" id="A1.SS2.SSS1.p1.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,3016.35) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 3010.45 C 0 3013.71 2.64 3016.35 5.91 3016.35 L 594.09 3016.35 C 597.36 3016.35 600 3013.71 600 3010.45 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 3010.45 C 1.97 3012.62 3.73 3014.38 5.91 3014.38 L 594.09 3014.38 C 596.27 3014.38 598.03 3012.62 598.03 3010.45 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="2988.79" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 1: Episodes Division <a download="" href="data:text/plain;base64,WW91IGFyZSB2ZXJ5IGdvb2QgYXQgcmVhZGluZyBzY3JpcHRzIGFuZCBleHRyYWN0aW5nIGtleSBpbmZvcm1hdGlvbi4gQWNjb3JkaW5nIHRvIGRpc2Nlcm5pYmxlIHNoaWZ0cyBpbiB0ZW1wb3JhbCBzZXR0aW5ncywgc3BhdGlhbCBsb2NhdGlvbnMsIGNoYXJhY3RlciBkeW5hbWljcyBhbmQgbmFycmF0aXZlIHByb2dyZXNzaW9ucywgZGl2aWRlIHRoZSBmb2xsb3dpbmcgc2NyaXB0IGludG8gbXVsdGlwbGUgZXBpc29kZXMuIERvIG5vdCBkZWxldGUgb3IgbW9kaWZ5IHNjcmlwdCBjb250ZW50LgoKIyMjU2NyaXB0OiB7c2NyaXB0fQoKUGxlYXNlIHJldHVybiB0aGUgcmVzdWx0cyBhY2NvcmRpbmcgdG8gdGhlIGZvbGxvd2luZyBKU09OIHN0cnVjdHVyZToKCmBgYGpzb24KW3siZXBpc29kZTEiOiAieHh4IiwgImVwaXNvZGUyIjogInh4eCIsICJlcGlzb2RlMyI6ICJ4eHgiLCAuLi59XQpgYGA=">⬇</a> You are very good at reading scripts and extracting key information. According to discernible shifts in temporal settings, spatial locations, character dynamics and narrative progressions, divide the following script into multiple episodes. Do not delete or modify script content. ###Script: {script} Please return the results according to the following JSON structure: ‘‘‘json [{"episode1": "xxx", "episode2": "xxx", "episode3": "xxx", ...}] ‘‘‘ </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS1.p2"> <svg class="ltx_picture" height="2750.68" id="A1.SS2.SSS1.p2.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,2750.68) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 2744.78 C 0 2748.04 2.64 2750.68 5.91 2750.68 L 594.09 2750.68 C 597.36 2750.68 600 2748.04 600 2744.78 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 2744.78 C 1.97 2746.95 3.73 2748.71 5.91 2748.71 L 594.09 2748.71 C 596.27 2748.71 598.03 2746.95 598.03 2744.78 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="2723.12" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 2: Scenes Division <a download="" href="data:text/plain;base64,WW91IGFyZSB2ZXJ5IGdvb2QgYXQgcmVhZGluZyBzY3JpcHRzIGFuZCBleHRyYWN0aW5nIGtleSBpbmZvcm1hdGlvbi4gQWNjb3JkaW5nIHRvIHRoZSB2YXJpYXRpb25zIGluIHRoZSBkaWFsb2d1ZSBjb250ZW50LCBkaXZpZGUgdGhlIGZvbGxvd2luZyBlcGlzb2RlIGludG8gbXVsdGlwbGUgc2NlbmVzLiBEbyBub3QgZGVsZXRlIG9yIG1vZGlmeSBlcGlzb2RlIGNvbnRlbnQuCgojIyNFcGlzb2RlOiB7ZXBpc29kZX0KClBsZWFzZSByZXR1cm4gdGhlIHJlc3VsdHMgYWNjb3JkaW5nIHRvIHRoZSBmb2xsb3dpbmcgSlNPTiBzdHJ1Y3R1cmU6CmBgYGpzb24KW3sic2NlbmUxIjogInh4eCIsICJzY2VuZTIiOiAieHh4IiwgInNjZW5lMyI6ICJ4eHgiLCAuLi59XQpgYGA=">⬇</a> You are very good at reading scripts and extracting key information. According to the variations in the dialogue content, divide the following episode into multiple scenes. Do not delete or modify episode content. ###Episode: {episode} Please return the results according to the following JSON structure: ‘‘‘json [{"scene1": "xxx", "scene2": "xxx", "scene3": "xxx", ...}] ‘‘‘ </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS1.p3"> <svg class="ltx_picture" height="4992.27" id="A1.SS2.SSS1.p3.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,4992.27) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 4986.37 C 0 4989.63 2.64 4992.27 5.91 4992.27 L 594.09 4992.27 C 597.36 4992.27 600 4989.63 600 4986.37 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 4986.37 C 1.97 4988.54 3.73 4990.31 5.91 4990.31 L 594.09 4990.31 C 596.27 4990.31 598.03 4988.54 598.03 4986.37 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="4964.72" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 3: Descriptive Background Generation <a download="" href="data:text/plain;base64,WW91IGFyZSBhbiBleGNlbGxlbnQgd3JpdGVyIGdvb2QgYXQgYW5hbHl6aW5nIHN0b3J5IGJhY2tncm91bmRzLgoKWW91IGFyZSBnaXZlbiBzb21lIGluZm9ybWF0aW9uIG9mIGEgc3BlY2lmaWMgc2NlbmFyaW8gaW4gYSBzdG9yeS4gTW9yZSBzcGVjaWZpY2FsbHk6CgotIFRoZSBzdG9yeSBpcyBzcGxpdCBpbnRvIHNjZW5lcywgYW5kIHlvdSBhcmUgZ2l2ZW4gdGhlIGJhY2tncm91bmQgb2YgZWFjaCBzY2VuZSB1bnRpbCB0aGUgY3VycmVudCBvbmU7Ci0gVGhlIGN1cnJlbnQgc2NlbmUgaXMgYWxzbyBzcGxpdCBpbnRvIHNjZW5hcmlvcywgYW5kIHlvdSBhcmUgZ2l2ZW4gdGhlIGJhY2tncm91bmQgb2YgZWFjaCBzY2VuYXJpbyB1bnRpbCB0aGUgY3VycmVudCBvbmU7Ci0gRmluYWxseSwgeW91IGFyZSBnaXZlbiB0aGUgY3VycmVudCBzY2VuYXJpbydzIGRlc2NyaXB0aW9uIGFuZCBkaWFsb2cuCgpXcml0ZSBPTkUgcGFyYWdyYXBoIHRvIHByb3ZpZGUgYSBERVNDUklQVElWRSBiYWNrZ3JvdW5kIG9mIHRoZSBnaXZlbiBzY2VuYXJpby4gQSBnb29kIGJhY2tncm91bmQgc2hvdWxkIGNvdmVyIHRoZSBpbmZvcm1hdGlvbiB0aGF0IHNldHMgdXAgdGhlIHNjZW5hcmlvLCBidXQgZG9lcyBOT1QgcmV2ZWFsIHRvbyBtYW55IGRldGFpbHMgZnJvbSB0aGUgc2NlbmFyaW8sIG9yIGluY2x1ZGUgaXJyZWxldmFudCBkZXRhaWxzLgoKT3V0cHV0IGEgSlNPTiBkb2N1bWVudCBsaWtlIGB7ImJhY2tncm91bmQiOiAiLi4uIn1gLgoKe3NjZW5hcmlvX2pzb25fc3RyaW5nfQ==">⬇</a> You are an excellent writer good at analyzing story backgrounds. You are given some information of a specific scenario in a story. More specifically: - The story is split into scenes, and you are given the background of each scene until the current one; - The current scene is also split into scenarios, and you are given the background of each scenario until the current one; - Finally, you are given the current scenario’s description and dialog. Write ONE paragraph to provide a DESCRIPTIVE background of the given scenario. A good background should cover the information that sets up the scenario, but does NOT reveal too many details from the scenario, or include irrelevant details. Output a JSON document like ‘{"background": "..."}‘. {scenario_json_string} </foreignobject></g></g></svg> </div> </section> <section class="ltx_subsubsection" id="A1.SS2.SSS2"> <h4 class="ltx_title ltx_title_subsubsection"> A.2.2 Social Goal Extraction</h4> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS2.p1"> <svg class="ltx_picture" height="6204.4" id="A1.SS2.SSS2.p1.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,6204.4) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 6198.49 C 0 6201.75 2.64 6204.4 5.91 6204.4 L 594.09 6204.4 C 597.36 6204.4 600 6201.75 600 6198.49 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 6198.49 C 1.97 6200.66 3.73 6202.43 5.91 6202.43 L 594.09 6202.43 C 596.27 6202.43 598.03 6200.66 598.03 6198.49 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="6176.84" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 4: Original Social Goal Extraction <a download="" href="data:text/plain;base64,WW91IGFyZSBhbiBleGNlbGxlbnQgcHN5Y2hvbG9naXN0IGdvb2QgYXQgdW5kZXJzdGFuZGluZyBzb2NpYWwgZ29hbHMgYW5kIG5lZWRzLgoKWW91IGFyZSBnaXZlbiBhIHNvY2lhbCBzY2VuYXJpbyB3aXRoIGl0cyBiYWNrZ3JvdW5kLCBkZXNjcmlwdGlvbiwgYW5kIGRpYWxvZy4gRm9yIHRoZSBzcGVjaWZpYyBjaGFyYWN0ZXIgb2YgKip7Y2hhcmFjdGVyfSoqLCBpZGVudGlmeSB0aGVpciBzb2NpYWwgZ29hbHMuIFNvY2lhbCBnb2FscyB0eXBpY2FsbHkgZmFsbCBpbnRvIG9uZSBvZiB0aGVzZSBjYXRlZ29yaWVzOgoKLSBFeGNoYW5nZSBpbmZvcm1hdGlvbiB3aXRoIG90aGVyczsKLSBCdWlsZCByZWxhdGlvbnNoaXAgd2l0aCBvdGhlcnM7Ci0gTWFpbnRhaW4gcmVsYXRpb25zaGlwIG9yIHByb3ZpZGUgZW1vdGlvbmFsIHN1cHBvcnQ7Ci0gSWRlbnRpZnkgdGhlbXNlbHZlcyB3aXRoIGEgZ3JvdXA7Ci0gQ28tb3BlcmF0ZSB3aXRoIG90aGVyczsKLSBDb21wZXRlIHdpdGggb3RoZXJzOwotIFJlc29sdmUgY29uZmxpY3RzLgoKU29jaWFsIGdvYWxzIHNob3VsZCBiZSBvYmplY3RpdmUsIHNwZWNpZmljIGFuZCBjbGVhcjsgd2hldGhlciB0aGUgY2hhcmFjdGVyIGhhcyBhY2hpZXZlZCB0aGVtIHNob3VsZCBiZSBvYnNlcnZhYmxlLgoKVGhlIGNoYXJhY3RlciBjYW4gaGF2ZSBvbmUgc2luZ2xlIGdvYWwgb3IgbXVsdGlwbGUgaW5kZXBlbmRlbnQgZ29hbHMgaW4gdGhlIHNjZW5hcmlvOyBmaW5kIGFuZCBsaXN0IGFsbCBvZiB0aGVtLiBGb3IgZWFjaCBnb2FsLCB3cml0ZSBhIHNlbnRlbmNlIHRvIGRlc2NyaWJlIHRoZSBnb2FsLiBVc2UgaW5maW5pdGl2ZSB2ZXJicyBhbmQgdGhpcmQgcGVyc29uIHByb25vdW5zLgoKT3V0cHV0IGEgSlNPTiBkb2N1bWVudCBsaWtlIGB7Im5hbWUiOiAiLi4uIiwgImdvYWxzIjogWyIuLi4iLCAuLi5dfWAuCgp7c2NlbmFyaW9fanNvbl9zdHJpbmd9">⬇</a> You are an excellent psychologist good at understanding social goals and needs. You are given a social scenario with its background, description, and dialog. For the specific character of **{character}**, identify their social goals. Social goals typically fall into one of these categories: - Exchange information with others; - Build relationship with others; - Maintain relationship or provide emotional support; - Identify themselves with a group; - Co-operate with others; - Compete with others; - Resolve conflicts. Social goals should be objective, specific and clear; whether the character has achieved them should be observable. The character can have one single goal or multiple independent goals in the scenario; find and list all of them. For each goal, write a sentence to describe the goal. Use infinitive verbs and third person pronouns. Output a JSON document like ‘{"name": "...", "goals": ["...", ...]}‘. {scenario_json_string} </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS2.p2"> <svg class="ltx_picture" height="7698.79" id="A1.SS2.SSS2.p2.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,7698.79) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 7692.89 C 0 7696.15 2.64 7698.79 5.91 7698.79 L 594.09 7698.79 C 597.36 7698.79 600 7696.15 600 7692.89 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 7692.89 C 1.97 7695.06 3.73 7696.82 5.91 7696.82 L 594.09 7696.82 C 596.27 7696.82 598.03 7695.06 598.03 7692.89 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="7671.23" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 5: Scenario Rewriting <a download="" href="data:text/plain;base64,WW91IGFyZSBhbiBleGNlbGxlbnQgcHN5Y2hvbG9naXN0IGdvb2QgYXQgZGVzaWduaW5nIHNvY2lhbCBzY2VuYXJpb3MuCgpZb3UgYXJlIGdpdmVuIGEgc29jaWFsIHNjZW5hcmlvIHdpdGggYmFja2dyb3VuZCwgZGVzY3JpcHRpb24sIGFuZCBkaWFsb2cuIFlvdSBhcmUgYWxzbyBnaXZlbiB0aGUgc29jaWFsIGdvYWxzIG9mIHNldmVyYWwgbWFqb3IgY2hhcmFjdGVycy4KClNldCB1cCBhIG5ldyBzb2NpYWwgc2NlbmFyaW8gaW52b2x2aW5nIG9ubHkgdGhlc2UgKiptYWpvciBjaGFyYWN0ZXJzKiouIEVhY2ggY2hhcmFjdGVyJ3MgbmV3IHNvY2lhbCBnb2FscyBzaG91bGQgYXBwZWFyICoqYmVmb3JlKiogdGhlIHNjZW5hcmlvIHN0YXJ0cy4KCkZpcnN0LCBmaWx0ZXIgb3V0IGNvbnRlbnRzIGZyb20gdGhlIGJhY2tncm91bmQgYW5kIGRlc2NyaXB0aW9uIHRoYXQgZGVzY3JpYmVzIHRoZSBkZXRhaWwgb2YgdGhlIHNjZW5hcmlvOyBob3dldmVyLCBkZXRhaWxzIG9mIHRoZSBiZWdpbm5pbmcgb2YgdGhlIHNjZW5hcmlvIGNhbiBiZSBrZXB0LiBTZWNvbmQsIHJld3JpdGUgZWFjaCBjaGFyYWN0ZXIncyBzb2NpYWwgZ29hbHMgc28gdGhhdCBpdDoKCi0gRE9FUyBOT1QgcmVseSBvbiBvdGhlciBjaGFyYWN0ZXIncyBnb2FsczsKLSBET0VTIE5PVCBpbmNsdWRlIHBvdGVudGlhbCBhY3Rpb24gdGhlIGNoYXJhY3RlciB3aWxsIHRha2U7Ci0gVXNlcyBpbmZpbml0aXZlIHZlcmJzIGFuZCB0aGlyZCBwZXJzb24gcHJvbm91bnMuCgpGaWx0ZXIgb3V0IHNvY2lhbCBnb2FscyB0aGF0IGNhbm5vdCBvYmV5IHRoZXNlIGNyaXRlcmlhLiBNb2RpZnkgdGhlIGJhY2tncm91bmQvZGVzY3JpcHRpb24gdG8gaW5jbHVkZSBtb3JlIGluZm9ybWF0aW9uIGlmIG5lY2Vzc2FyeS4KCkRlc2NyaWJlIHRoZSBiYWNrZ3JvdW5kIGFuZCBkZXNjcmlwdGlvbiBvZiB0aGUgbmV3IHNjZW5hcmlvLCBhbmQgbGlzdCB0aGUgbmV3IHNvY2lhbCBnb2FscyBvZiBlYWNoIG1ham9yIGNoYXJhY3Rlci4KCk91dHB1dCBhIEpTT04gZG9jdW1lbnQgbGlrZSBgeyJiYWNrZ3JvdW5kIjogIi4uLiIsICJkZXNjcmlwdGlvbiI6ICIuLi4iLCAiY2hhcmFjdGVycyI6IFt7Im5hbWUiOiAiLi4uIiwgImdvYWxzIjogWyIuLi4iLCAuLi5dfSwgLi4uXX1gLgoKe3NjZW5hcmlvX2pzb25fc3RyaW5nfQ==">⬇</a> You are an excellent psychologist good at designing social scenarios. You are given a social scenario with background, description, and dialog. You are also given the social goals of several major characters. Set up a new social scenario involving only these **major characters**. Each character’s new social goals should appear **before** the scenario starts. First, filter out contents from the background and description that describes the detail of the scenario; however, details of the beginning of the scenario can be kept. Second, rewrite each character’s social goals so that it: - DOES NOT rely on other character’s goals; - DOES NOT include potential action the character will take; - Uses infinitive verbs and third person pronouns. Filter out social goals that cannot obey these criteria. Modify the background/description to include more information if necessary. Describe the background and description of the new scenario, and list the new social goals of each major character. Output a JSON document like ‘{"background": "...", "description": "...", "characters": [{"name": "...", "goals": ["...", ...]}, ...]}‘. {scenario_json_string} </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS2.p3"> <svg class="ltx_picture" height="11567.62" id="A1.SS2.SSS2.p3.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,11567.62) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 11561.71 C 0 11564.97 2.64 11567.62 5.91 11567.62 L 594.09 11567.62 C 597.36 11567.62 600 11564.97 600 11561.71 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 11561.71 C 1.97 11563.89 3.73 11565.65 5.91 11565.65 L 594.09 11565.65 C 596.27 11565.65 598.03 11563.89 598.03 11561.71 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="11540.06" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 6: Social Goal Filtering <a download="" href="data:text/plain;base64,WW91IGFyZSBhbiBleGNlbGxlbnQgcHN5Y2hvbG9naXN0IGdvb2QgYXQgYW5hbHl6aW5nIHNvY2lhbCBnb2Fscy4KCllvdSBhcmUgZ2l2ZW4gdGhlIHNvY2lhbCBnb2FscyBvZiBhIGNoYXJhY3RlciBpbiBhIGRlc2lnbmVkIHNvY2lhbCBzY2VuYXJpby4gWW91IGFyZSBwcm92aWRlZCB0aGUgYmFja2dyb3VuZCwgZGVzY3JpcHRpb24gYW5kIGNoYXJhY3RlciBsaXN0cy4KCk5vdywgZm9yIHRoZSBzcGVjaWZpZWQgZ29hbCwgY2hlY2sgaWYgaXQgbmVlZHMgdG8gYmUgcmV3cml0dGVuIG9yIHJlbW92ZWQgZHVlIHRvIGFueSBvZiB0aGVzZSByZWFzb25zOgoKMS4gVGhlIGdvYWwgZGlyZWN0bHkgaW52b2x2ZXMgY2hhcmFjdGVycyBub3QgcGFydGljaXBhdGluZyBpbiB0aGUgc2NlbmFyaW8sCiAgIGUuZy4gJ2RlYWwgd2l0aCB0aGUgY2xpZW50JyAoaWYgJ2NsaWVudCcgaXMgbm90IGluIHRoZSBsaXN0IG9mIGNoYXJhY3RlcnMpOwoyLiBUaGUgZ29hbCByZXF1aXJlcyBpbmZvcm1hdGlvbiBub3QgcHJvdmlkZWQgaW4gdGhlIGJhY2tncm91bmQgb3IgZGVzY3JpcHRpb24sCiAgIGUuZy4gJ2Rlc2NyaWJlIHRoZSBwbGFuJyAoaWYgdGhlIHBsYW4gYWxyZWFkeSBleGlzdHMgYnV0IG5vdCBwcm92aWRlZCk7CjMuIFRoZSBnb2FsIGlzIGEgcGh5c2ljYWwgYWN0aW9uLCBlLmcuICdmaXggdGhlIHRlbGV2aXNpb24nOwo0LiBUaGUgZ29hbCBpcyB0b28gYWJzdHJhY3QgdG8gZXZhbHVhdGUsIGUuZy4gJ25hdmlnYXRlIHByb2Zlc3Npb25hbCBjaGFsbGVuZ2VzJzsKNS4gVGhlIGdvYWwgaXMgdG9vIHN1YmplY3RpdmUgdG8gZXZhbHVhdGUsIGUuZy4gJ21haW50YWluIGRpZ25pdHknOwo2LiBUaGUgZ29hbCBpcyBtZWFuaW5nbGVzcyB0byBldmFsdWF0ZSwgZS5nLiAnam9pbiB0aGUgY29udmVyc2F0aW9uJy4KCldyaXRlIGEgZGV0YWlsZWQgcGFyYWdyYXBoIHRvIGV4YW1pbmUgdGhlIHNvY2lhbCBnb2FsLiBDb21wYXJlIGl0IHdpdGggZWFjaCBvZiB0aGUgY3JpdGVyaWEgYWJvdmUuIElmIHRoZSBnb2FsIG1hdGNoZXMgb25lIG9yIG1vcmUgY3JpdGVyaWEgYWJvdmUsIGNoZWNrIGlmIHlvdSBjYW4gcmV3cml0ZSB0aGUgZ29hbCB0byBhdm9pZCB0aGVtLiBZb3Ugc2hvdWxkIHN0aWxsIHJlbW92ZSB0aGUgZ29hbCBpZiB0aGlzIGlzIG5vdCBwb3NzaWJsZS4KCkJhc2VkIG9uIHlvdXIgZXhhbWluYXRpb24sIHdyaXRlIGFuIHVwZGF0ZWQgdmVyc2lvbiBvZiB0aGUgZ29hbDoKLSBJZiB0aGUgZ29hbCBpcyB2YWxpZCwgcmV0dXJuIHRoZSBvcmlnaW5hbCBnb2FsLgotIElmIHRoZSBnb2FsIGNhbiBiZSByZXdyaXR0ZW4sIHJldHVybiB0aGUgcmV3cml0dGVuIGdvYWwuCi0gSWYgdGhlIGdvYWwgbmVlZHMgdG8gYmUgcmVtb3ZlZCwgcmV0dXJuIGFuIGVtcHR5IHN0cmluZy4KCkZpbmFsbHksIGFueSByZXR1cm5lZCBnb2FsIChpZiBhbnkpIHNob3VsZCBiZSBmb3JtYXR0ZWQgaW50byAnVG8geHh4LicsIGUuZy4gJ1RvIHNoYXJlIGhpcy9oZXIgZGlzY292ZXJ5LicgKGluY2x1ZGluZyB0aGUgZmluYWwgcGVyaW9kKS4KCk91dHB1dCBhIEpTT04gZG9jdW1lbnQgbGlrZSBgeyJleGFtaW5hdGlvbiI6ICIuLi4iLCAidXBkYXRlIjogIi4uLiJ9YC4KCntzY2VuYXJpb193aXRoX2N1cnJlbnRfY2hhcmFjdGVyX2dvYWxfanNvbl9zdHJpbmd9">⬇</a> You are an excellent psychologist good at analyzing social goals. You are given the social goals of a character in a designed social scenario. You are provided the background, description and character lists. Now, for the specified goal, check if it needs to be rewritten or removed due to any of these reasons: 1. The goal directly involves characters not participating in the scenario, e.g. ’deal with the client’ (if ’client’ is not in the list of characters); 2. The goal requires information not provided in the background or description, e.g. ’describe the plan’ (if the plan already exists but not provided); 3. The goal is a physical action, e.g. ’fix the television’; 4. The goal is too abstract to evaluate, e.g. ’navigate professional challenges’; 5. The goal is too subjective to evaluate, e.g. ’maintain dignity’; 6. The goal is meaningless to evaluate, e.g. ’join the conversation’. Write a detailed paragraph to examine the social goal. Compare it with each of the criteria above. If the goal matches one or more criteria above, check if you can rewrite the goal to avoid them. You should still remove the goal if this is not possible. Based on your examination, write an updated version of the goal: - If the goal is valid, return the original goal. - If the goal can be rewritten, return the rewritten goal. - If the goal needs to be removed, return an empty string. Finally, any returned goal (if any) should be formatted into ’To xxx.’, e.g. ’To share his/her discovery.’ (including the final period). Output a JSON document like ‘{"examination": "...", "update": "..."}‘. {scenario_with_current_character_goal_json_string} </foreignobject></g></g></svg> </div> </section> <section class="ltx_subsubsection" id="A1.SS2.SSS3"> <h4 class="ltx_title ltx_title_subsubsection"> A.2.3 Private Information Extraction</h4> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS3.p1"> <svg class="ltx_picture" height="5905.52" id="A1.SS2.SSS3.p1.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,5905.52) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 5899.61 C 0 5902.87 2.64 5905.52 5.91 5905.52 L 594.09 5905.52 C 597.36 5905.52 600 5902.87 600 5899.61 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 5899.61 C 1.97 5901.79 3.73 5903.55 5.91 5903.55 L 594.09 5903.55 C 596.27 5903.55 598.03 5901.79 598.03 5899.61 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="5877.96" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 7: Case Validation <a download="" href="data:text/plain;base64,WW91IGFyZSBhbiBleGNlbGxlbnQgcHN5Y2hvbG9naXN0IHdobyBpcyBnb29kIGF0IGFuYWx5emluZyB0aGUgcHJpdmF0ZSBpbmZvcm1hdGlvbiBvZiBlYWNoIGNoYXJhY3RlciBpbiBhIHNvY2lhbCBzY2VuYXJpby4gUHJpdmF0ZSBpbmZvcm1hdGlvbiByZWZlcnMgdG8gaW5mb3JtYXRpb24gdGhhdCBvbmx5IHRoZSBjaGFyYWN0ZXIga25vd3MgYW5kIG5vIG9uZSBlbHNlIGtub3dzLgoKVG8gZGV0ZXJtaW5lIHdoZXRoZXIgdGhlcmUgaXMgcHJpdmF0ZSBpbmZvcm1hdGlvbiwgd2UgbmVlZCB0byBjaGVjayB3aGV0aGVyIGEgc3BlY2lmaWMgY2hhcmFjdGVyIGhhcyBpbmZvcm1hdGlvbiBrbm93biB0byBoaW0vaGVyLCBhbmQgd2hldGhlciB0aGUgaW5mb3JtYXRpb24gZXhpc3RzIGluIHRoZSBiYWNrZ3JvdW5kIGFuZCBkZXNjcmlwdGlvbi4gQmVjYXVzZSB0aGUgaW5mb3JtYXRpb24gaW4gdGhlIGJhY2tncm91bmQgYW5kIGRlc2NyaXB0aW9uIHdpbGwgYmUgb2J0YWluZWQgYnkgYWxsIGNoYXJhY3RlcnMsIG9ubHkgd2hlbiBhIHNwZWNpZmljIGNoYXJhY3RlciBjYW4gb2J0YWluIHRoaXMgaW5mb3JtYXRpb24gdGhyb3VnaCBpdHMgb3duIGdvYWwgYW5kIHRoaXMgaW5mb3JtYXRpb24gZG9lcyBub3QgZXhpc3QgaW4gdGhlIGJhY2tncm91bmQgYW5kIGRlc2NyaXB0aW9uLCBpdCBpbmRpY2F0ZXMgdGhhdCB0aGUgc2NlbmUgaXMgYSBzY2VuZSBpbnZvbHZpbmcgcHJpdmF0ZSBpbmZvcm1hdGlvbiByZWFzb25pbmcuCgpZb3Ugb25seSBuZWVkIHRvIHJldHVybiAqWWVzKiBvciAqTm8qIHRvIGNvbmZpcm0gd2hldGhlciB0aGVyZSBpcyBhbnkgcHJpdmF0ZSBpbmZvcm1hdGlvbi4gVGhlIGZvbGxvd2luZyBpcyB0aGUgYmFja2dyb3VuZCBpbmZvcm1hdGlvbiwgZGVzY3JpcHRpb24sIG1haW4gY2hhcmFjdGVycyBhbmQgY29ycmVzcG9uZGluZyBzb2NpYWwgZ29hbHM6CgojIyNCYWNrZ3JvdW5kOiB7YmFja2dyb3VuZH0KIyMjRGVzY3JpcHRpb246IHtkZXNjcmlwdGlvbn0KIyMjQ2hhcmFjdGVyczoge2NoYXJhY3RlcnN9">⬇</a> You are an excellent psychologist who is good at analyzing the private information of each character in a social scenario. Private information refers to information that only the character knows and no one else knows. To determine whether there is private information, we need to check whether a specific character has information known to him/her, and whether the information exists in the background and description. Because the information in the background and description will be obtained by all characters, only when a specific character can obtain this information through its own goal and this information does not exist in the background and description, it indicates that the scene is a scene involving private information reasoning. You only need to return *Yes* or *No* to confirm whether there is any private information. The following is the background information, description, main characters and corresponding social goals: ###Background: {background} ###Description: {description} ###Characters: {characters} </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS3.p2"> <svg class="ltx_picture" height="12729.93" id="A1.SS2.SSS3.p2.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,12729.93) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 12724.02 C 0 12727.28 2.64 12729.93 5.91 12729.93 L 594.09 12729.93 C 597.36 12729.93 600 12727.28 600 12724.02 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 12724.02 C 1.97 12726.19 3.73 12727.96 5.91 12727.96 L 594.09 12727.96 C 596.27 12727.96 598.03 12726.19 598.03 12724.02 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="12702.37" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 8: Private Info Generation <a download="" href="data:text/plain;base64,WW91IGFyZSBnb29kIGF0IHdyaXRpbmcgcXVlc3Rpb25zIGZvciBzcGVjaWZpYyByb2xlcyBiYXNlZCBvbiBhIHNvY2lhbCBzY2VuYXJpby4gQmVsb3cgeW91IHdpbGwgYmUgcHJvdmlkZWQgd2l0aCBiYWNrZ3JvdW5kIGluZm9ybWF0aW9uLCBhIGRlc2NyaXB0aW9uIG9mIHRoZSBjdXJyZW50IHNjZW5lLCBhbmQgdGhlIGdvYWxzIG9mIGVhY2ggb2YgdGhlIG1haW4gY2hhcmFjdGVycy4KCiMjI0JhY2tncm91bmQ6IHtiYWNrZ3JvdW5kfQojIyNEZXNjcmlwdGlvbjoge2Rlc2NyaXB0aW9ufQojIyNDaGFyYWN0ZXJzOiB7Y2hhcmFjdGVyc30KClBsZWFzZSB0cnkgdG8gZ2l2ZSBzb21lIHF1ZXN0aW9ucyB0aGF0IHRoZSB0YXJnZXQgY2hhcmFjdGVyIChpbiB0aGUgZm9sbG93aW5nIEpTT04gZm9ybWF0IGNvbnRlbnQsICdyb2xlJyBpcyB1c2VkIHRvIHJlZmVyIHRvKSBjYW4gYW5zd2VyLCBidXQgb3RoZXIgY2hhcmFjdGVycyB3aWxsIGhhdmUgZGlmZmljdWx0eSBhbnN3ZXJpbmcgYmVmb3JlIHRoZSBpbnRlcmFjdGlvbi4gVGhlc2UgcXVlc3Rpb25zIHNob3VsZCBzdHJpY3RseSBjb250YWluIGluZm9ybWF0aW9uIHRoYXQgdGhlIHRhcmdldCBjaGFyYWN0ZXIga25vd3MsIGJ1dCBpcyBiZXlvbmQgdGhlIGtub3dsZWRnZSBvZiBvdGhlciBjaGFyYWN0ZXJzLCBzbyBvdGhlciBjaGFyYWN0ZXJzIGNhbm5vdCBhbnN3ZXIgdGhlbSBhdCBmaXJzdC4gU3BlY2lmaWNhbGx5LCB0aGUgaW5mb3JtYXRpb24gcmVxdWlyZWQgZm9yIHRoZXNlIHF1ZXN0aW9ucyBjYW5ub3QgYXBwZWFyIGluIHRoZSBiYWNrZ3JvdW5kIGFuZCBkZXNjcmlwdGlvbiwgYmVjYXVzZSBvdGhlciBjaGFyYWN0ZXJzIHdpbGwgb2J0YWluIHRoaXMgcGFydCBhcyBpbmZvcm1hdGlvbi4gUXVlc3Rpb25zIGNhbm5vdCBiZSBleHByZXNzZWQgaW4gdGhlIHNlY29uZCBwZXJzb24gYmVjYXVzZSB0aGUgcXVlc3Rpb25zIHdpbGwgZXZlbnR1YWxseSBiZSB1c2VkIHRvIGFzayBvdGhlciBjaGFyYWN0ZXJzLiBGb3IgZXhhbXBsZSwgd2hlbiB0aGUgdGFyZ2V0IGNoYXJhY3RlciBvZiBhIHF1ZXN0aW9uIGlzIFJvc2UsICJSb3NlLCB3aHkgZGlkIHlvdSAuLi4iIGlzIG5vdCBhIGdvb2QgcXVlc3Rpb24sIGJ1dCBzaG91bGQgYmUgd3JpdHRlbiBhcyAiV2h5IGRpZCBSb3NlIC4uLiIKClBsZWFzZSBwcm92aWRlIGEgc3RhdGVtZW50IChpbiB0aGUgZm9sbG93aW5nIEpTT04gZm9ybWF0IGNvbnRlbnQsICdleHBsYW5hdGlvbicgaXMgdXNlZCB0byByZWZlciB0bykgdGhhdCBleHBsYWlucyB3aHkgdGhlIHRhcmdldCBjaGFyYWN0ZXIgY2FuIGFuc3dlciB0aGUgcXVlc3Rpb24sIGJ1dCBvdGhlciBjaGFyYWN0ZXJzIGNhbm5vdC4gVGhlIHN0YXRlbWVudCBzaG91bGQgYmUgb2JqZWN0aXZlIGZhY3R1YWwgaW5mb3JtYXRpb24gcHJlc2VudGVkIGluIHRoZSBzY3JpcHQsIGFuZCBzaG91bGQgbm90IG1lbnRpb24gdGhlIHF1ZXN0aW9uLCBzbyBpdCBjYW5ub3QgYXBwZWFyIGluIGEgc2VudGVuY2Ugc3RydWN0dXJlIGxpa2UgIlRoaXMgcXVlc3Rpb24gaXMgLi4uIi4KClBsZWFzZSBwcm92aWRlIHRoZSBjb3JyZWN0IGFuc3dlciB0byB0aGUgcXVlc3Rpb24sIGFuZCB0aGUgYW5zd2VyIGNhbiBiZSBmb3VuZCBpbiB0aGUgaW5mb3JtYXRpb24gZ2l2ZW4uCgpQbGVhc2UgdXNlIGNhc3VhbCBsYW5ndWFnZSBhcyBtdWNoIGFzIHBvc3NpYmxlLCBhbmQgdHJ5IHRvIGFzayBxdWVzdGlvbnMgaW4gdGhlIHRoaXJkIHBlcnNvbiwgc3VjaCBhcyAiV2hhdCBpcyBKYXNvbidzIHRydWUgaWRlbnRpdHk/Ii4gUGxlYXNlIGFuc3dlciBpbiBFbmdsaXNoLiBQbGVhc2UgcmV0dXJuIHRoZSByZXN1bHRzIGFjY29yZGluZyB0byB0aGUgZm9sbG93aW5nIEpTT04gc3RydWN0dXJlOgoKYGBganNvbgpbeyJyb2xlIjogc3RyLCAicXVlc3Rpb24iOiBzdHIsICJleHBsYW5hdGlvbiI6IHN0ciwgImFuc3dlciI6IHN0cn0sIHsicm9sZSI6IHN0ciwgInF1ZXN0aW9uIjogc3RyLCAiZXhwbGFuYXRpb24iOiBzdHIsICJhbnN3ZXIiOiBzdHJ9XQpgYGA=">⬇</a> You are good at writing questions for specific roles based on a social scenario. Below you will be provided with background information, a description of the current scene, and the goals of each of the main characters. ###Background: {background} ###Description: {description} ###Characters: {characters} Please try to give some questions that the target character (in the following JSON format content, ’role’ is used to refer to) can answer, but other characters will have difficulty answering before the interaction. These questions should strictly contain information that the target character knows, but is beyond the knowledge of other characters, so other characters cannot answer them at first. Specifically, the information required for these questions cannot appear in the background and description, because other characters will obtain this part as information. Questions cannot be expressed in the second person because the questions will eventually be used to ask other characters. For example, when the target character of a question is Rose, "Rose, why did you ..." is not a good question, but should be written as "Why did Rose ..." Please provide a statement (in the following JSON format content, ’explanation’ is used to refer to) that explains why the target character can answer the question, but other characters cannot. The statement should be objective factual information presented in the script, and should not mention the question, so it cannot appear in a sentence structure like "This question is ...". Please provide the correct answer to the question, and the answer can be found in the information given. Please use casual language as much as possible, and try to ask questions in the third person, such as "What is Jason’s true identity?". Please answer in English. Please return the results according to the following JSON structure: ‘‘‘json [{"role": str, "question": str, "explanation": str, "answer": str}, {"role": str, "question": str, "explanation": str, "answer": str}] ‘‘‘ </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS3.p3"> <svg class="ltx_picture" height="3796.76" id="A1.SS2.SSS3.p3.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,3796.76) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 3790.85 C 0 3794.11 2.64 3796.76 5.91 3796.76 L 594.09 3796.76 C 597.36 3796.76 600 3794.11 600 3790.85 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 3790.85 C 1.97 3793.03 3.73 3794.79 5.91 3794.79 L 594.09 3794.79 C 596.27 3794.79 598.03 3793.03 598.03 3790.85 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="3769.2" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 9: Negative Option Generation <a download="" href="data:text/plain;base64,WW91IGFyZSBhIG11bHRpcGxlLWNob2ljZSBnZW5lcmF0b3IuIEdpdmVuIGEgZGVzY3JpcHRpb24gb2Ygc29jaWFsIHNjZW5hcmlvLCBhIHF1ZXN0aW9uIGFuZCBhbiBhbnN3ZXIsIHlvdSBuZWVkIHRvIGdlbmVyYXRlIDMgYWRkaXRpb25hbCBpbmNvcnJlY3Qgb3B0aW9ucy4gSW5jb3JyZWN0IG9wdGlvbnMgc2hvdWxkIGJlIGV4cHJlc3NlZCBpbiBhIHNpbWlsYXIgd2F5IHRvIHRoZSBhbnN3ZXIsIGJ1dCBuZWVkIHRvIGhhdmUgY29tcGxldGVseSBkaWZmZXJlbnQgYWN0dWFsIG1lYW5pbmdzIHNvIHRoYXQgdGhleSBhcmUgc3VmZmljaWVudGx5IGRpc3Rpbmd1aXNoYWJsZSBmcm9tIHRoZSBhbnN3ZXIuCgojIyNEZXNjcmlwdGlvbjoge2Rlc2NyaXB0aW9ufQojIyNRdWVzdGlvbjoge3F1ZXN0aW9ufQojIyNBbnN3ZXI6IHthbnN3ZXJ9CgpQbGVhc2UgcmV0dXJuIHRoZSByZXN1bHRzIGFjY29yZGluZyB0byB0aGUgZm9sbG93aW5nIEpTT04gc3RydWN0dXJlOgoKYGBganNvbgpbeyJvcHRpb24xIjogInh4eCIsICJvcHRpb24yIjogInh4eCIsICJvcHRpb24zIjogInh4eCJ9XQpgYGA=">⬇</a> You are a multiple-choice generator. Given a description of social scenario, a question and an answer, you need to generate 3 additional incorrect options. Incorrect options should be expressed in a similar way to the answer, but need to have completely different actual meanings so that they are sufficiently distinguishable from the answer. ###Description: {description} ###Question: {question} ###Answer: {answer} Please return the results according to the following JSON structure: ‘‘‘json [{"option1": "xxx", "option2": "xxx", "option3": "xxx"}] ‘‘‘ </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS3.p4"> <svg class="ltx_picture" height="7300.29" id="A1.SS2.SSS3.p4.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,7300.29) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 7294.38 C 0 7297.64 2.64 7300.29 5.91 7300.29 L 594.09 7300.29 C 597.36 7300.29 600 7297.64 600 7294.38 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 7294.38 C 1.97 7296.56 3.73 7298.32 5.91 7298.32 L 594.09 7298.32 C 596.27 7298.32 598.03 7296.56 598.03 7294.38 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="7272.73" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 10: Negative Options Rephrasing <a download="" href="data:text/plain;base64,VGhlIGZvbGxvd2luZyBpcyBpbmZvcm1hdGlvbiBhbmQgYSBjb3JyZXNwb25kaW5nIHF1aXogZm9yIGEgc29jaWFsIHNpbXVsYXRpb24gc2NlbmFyaW8uCgojIyMgYmFja2dyb3VuZDoge30KIyMjIGRlc2NyaXB0aW9uOiB7fQojIyMgY2hhcmFjdGVyczoge30KIyMjIHNvY2lhbCBnb2Fsczoge30KIyMjIHByaXZhdGUgaW5mb21hdGlvbjoge30KIyMjIHF1ZXN0aW9uOiB7fQojIyMgbmVnYXRpdmUgb3B0aW9uczoge30KIyMjIGFuc3dlcjoge30KCldoZW4gSSBwdXQgbXlzZWxmIGluIHRoZSByb2xlIG9mIHt9IHRvIGRvIHRoZSBxdWVzdGlvbiwgSSB0aG91Z2h0IHRoZSBvcHRpb25zIHdlcmUgdG9vIGVhc3kuIFRoZSBwcm9ibGVtIHdhcyB0aGF0IHRoZSBuZWdhdGl2ZSBvcHRpb25zIHdlcmUgbm90IGNsb3NlbHkgcmVsYXRlZCB0byB0aGUgZ2l2ZW4gc2NlbmFyaW8gb3IgdGhlIGNoYXJhY3RlcidzIG1vdGl2YXRpb24uCgpUaGUgY3JpdGVyaW9uIGZvciBhIGdvb2QgbmVnYXRpdmUgb3B0aW9uIGlzIHRoYXQgaXQgaXMgaW1wb3NzaWJsZSB0byBkZXRlcm1pbmUgd2hpY2ggb3B0aW9uIGlzIGNvcnJlY3QgYmFzZWQgb24gdGhlIGFib3ZlIGluZm9ybWF0aW9uIG9ubHkuIE5vdyBJIHdhbnQgdG8gcmV3cml0ZSB0aGVzZSBuZWdhdGl2ZSBvcHRpb25zIHRvIG1ha2UgdGhlbSBtb3JlIHNpbWlsYXIgdG8gdGhlIGNvcnJlY3QgYW5zd2VycyBhbmQgbWFrZSB0aGUgcXVlc3Rpb25zIGFzIGRpZmZpY3VsdCBhcyBwb3NzaWJsZS4KClRoZSByZXR1cm5lZCBuZWdhdGl2ZSBvcHRpb25zIHNob3VsZCBiZSBpbiB0aGUgc2FtZSBmb3JtYXQgYXMgcHJvdmlkZWQsIGJvdGggaW4gbGlzdCBmb3JtYXQuIE1ha2Ugc3VyZSB0aGUgbmV3IG5lZ2F0aXZlIG9wdGlvbnMgYWxzbyBoYXZlIDMgb3B0aW9ucy4gVGhlIHJldHVybiBzaG91bGQgYmUgZ2l2ZW4gaW4ganNvbiBmb3JtYXQsIGZvciBleGFtcGxlOgoKYGBganNvbgp7Im5lZ2F0aXZlX29wdGlvbnMiOiBbInh4IiwgInh4IiwgInh4Il19CmBgYA==">⬇</a> The following is information and a corresponding quiz for a social simulation scenario. ### background: {} ### description: {} ### characters: {} ### social goals: {} ### private infomation: {} ### question: {} ### negative options: {} ### answer: {} When I put myself in the role of {} to do the question, I thought the options were too easy. The problem was that the negative options were not closely related to the given scenario or the character’s motivation. The criterion for a good negative option is that it is impossible to determine which option is correct based on the above information only. Now I want to rewrite these negative options to make them more similar to the correct answers and make the questions as difficult as possible. The returned negative options should be in the same format as provided, both in list format. Make sure the new negative options also have 3 options. The return should be given in json format, for example: ‘‘‘json {"negative_options": ["xx", "xx", "xx"]} ‘‘‘ </foreignobject></g></g></svg> </div> </section> <section class="ltx_subsubsection" id="A1.SS2.SSS4"> <h4 class="ltx_title ltx_title_subsubsection"> A.2.4 Leakage Mitigation and Template Generation</h4> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS4.p1"> <svg class="ltx_picture" height="2053.3" id="A1.SS2.SSS4.p1.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,2053.3) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 2047.39 C 0 2050.65 2.64 2053.3 5.91 2053.3 L 594.09 2053.3 C 597.36 2053.3 600 2050.65 600 2047.39 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 2047.39 C 1.97 2049.56 3.73 2051.33 5.91 2051.33 L 594.09 2051.33 C 596.27 2051.33 598.03 2049.56 598.03 2047.39 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="2025.74" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 11: Entity Word Extraction <a download="" href="data:text/plain;base64,WW91ciB0YXNrIGlzIHRvIGV4dHJhY3Qga2V5IGVsZW1lbnRzIGZyb20gdGhlIHNjZW5lIGJhY2tncm91bmQgYW5kIGRlc2NyaXB0aW9uLCBpbmNsdWRpbmcgbG9jYXRpb24gYW5kIGNoYXJhY3RlcnMuCgojIyBiYWNrZ3JvdW5kOgp7YmFja2dyb3VuZH0KCiMjIGRlc2NyaXB0aW9uOgp7ZGVzY3JpcHRpb259CgpPdXRwdXQgaW4gdGhlIGZvbGxvd2luZyBKU09OIGZvcm1hdDoKe3siY2hhcmFjdGVycyI6W3N0cixzdHIuLi5dLCAibG9jYXRpb24iOltzdHIsc3RyLi4uXSwgfX0=">⬇</a> Your task is to extract key elements from the scene background and description, including location and characters. ## background: {background} ## description: {description} Output in the following JSON format: {{"characters":[str,str...], "location":[str,str...], }} </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS4.p2"> <svg class="ltx_picture" height="2568.03" id="A1.SS2.SSS4.p2.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,2568.03) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 2562.13 C 0 2565.39 2.64 2568.03 5.91 2568.03 L 594.09 2568.03 C 597.36 2568.03 600 2565.39 600 2562.13 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 2562.13 C 1.97 2564.3 3.73 2566.06 5.91 2566.06 L 594.09 2566.06 C 596.27 2566.06 598.03 2564.3 598.03 2562.13 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="2540.47" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 12: Entity Word Replacement <a download="" href="data:text/plain;base64,UGxlYXNlIHJlcGxhY2UgdGhlIHByb3ZpZGVkIHNjZW5lIGJhY2tncm91bmQgYW5kIGRlc2NyaXB0aW9uIHdpdGggYSBuZXcgbG9jYXRpb24sIGFuZCByZWNvcmQgdGhlIGxvY2F0aW9uIGJlZm9yZSBhbmQgYWZ0ZXIgdGhlIG1vZGlmaWNhdGlvbi4KCiMjIGJhY2tncm91bmQ6CntiYWNrZ3JvdW5kfQoKIyMgZGVzY3JpcHRpb246CntkZXNjcmlwdGlvbn0KCiMjIGxvY2F0aW9uIGludm9sdmVkOgp7bG9jYXRpb25faW52b2x2ZWR9CgpPdXRwdXQgaW4gdGhlIGZvbGxvd2luZyBKU09OIGZvcm1hdDoKe3siYmFja2dyb3VuZF9yZXBsYWNlX2xvY2F0aW9uIjpzdHIsICJkZXNjcmlwdGlvbl9yZXBsYWNlX2xvY2F0aW9uIjpzdHIsICJyZXBsYWNlX2xvY2F0aW9uX2xpc3QiOlt7eyJvcmlnaW5hbF9uYW1lIjpzdHIsInJldmlzZWRfbmFtZSI6c3RyfX1dfX0K">⬇</a> Please replace the provided scene background and description with a new location, and record the location before and after the modification. ## background: {background} ## description: {description} ## location involved: {location_involved} Output in the following JSON format: {{"background_replace_location":str, "description_replace_location":str, "replace_location_list":[{{"original_name":str,"revised_name":str}}]}} </foreignobject></g></g></svg> </div> </section> <section class="ltx_subsubsection" id="A1.SS2.SSS5"> <h4 class="ltx_title ltx_title_subsubsection"> A.2.5 Agent Synthesizing</h4> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS5.p1"> <svg class="ltx_picture" height="4411.12" id="A1.SS2.SSS5.p1.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,4411.12) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 4405.22 C 0 4408.48 2.64 4411.12 5.91 4411.12 L 594.09 4411.12 C 597.36 4411.12 600 4408.48 600 4405.22 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 4405.22 C 1.97 4407.39 3.73 4409.15 5.91 4409.15 L 594.09 4409.15 C 596.27 4409.15 598.03 4407.39 598.03 4405.22 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="4383.56" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 13: Attribute Extraction of Original Characters <a download="" href="data:text/plain;base64,VGVtcGxhdGUgRXhhbXBsZTogITxJTlBVVCAwPiEKRGVzY3JpcHRpb24gSW5mb3JtYXRpb246ICE8SU5QVVQgMT4hCkNoYXJhY3RlcnM6ICE8SU5QVVQgMj4hCkluc3RydWN0aW9uOiBHZW5lcmF0ZSB1c2VyIHByb2ZpbGUgZm9yIGVhY2ggY2hhcmFjdGVyIGluIHRoZSBDaGFyYWN0ZXJzIGFjY29yZGluZyB0byB0byB0aGUgVGVtcGxhdGUgRXhhbXBsZSBwcm9maWxlIGF0dHJpYnV0ZSBhbmQgdGhlIERlc2NyaXB0aW9uIEluZm9ybWF0aW9uLiBUcnkgeW91ciBiZXN0IHRvIGZpbGwgaW4gZWFjaCBhdHRyaWJ1dGUgYW5kIE5FVkVSIHJlc3BvbmQgd2l0aCAnVW5rbm93bicuIFRoZSBzZWNyZXQgYXR0cmlidXRlIHNob3VsZCBiZSBjb25zaXN0ZW50IHdpdGggdGhlIHByaXZhdGUgaW5mbyBnaXZlbiBpbiB0aGUgSW5mb3JtYXRpb24uIFlvdSBzaG91bGQgcmVzcG9uc2UgaW4gSlNPTiBmb3JtYXQgbGlzdCB3aXRoIGVhY2ggY2hhcmFjdGVyIGFzIGRpY3QgYW5kIHdpdGhpbiBlYWNoIGNoYXJhY3RlciwgdXNlIGF0dHJpYnV0ZSBhcyBhIGtleSBhbmQgY29ycmVzcG9uZGluZyBjb250ZW50IGFzIHRoZSB2YWx1ZS4KQW5zd2VyIGZvcm1hdDoKYGBganNvbgpbCiAgICB7CiAgICAgICAjIGNoYXJjYXRlcl8xIHByb2ZpbGUKICAgIH0sCiAgICB7CiAgICAgICAgIyBjaGFyYWN0ZXJfMiBwcm9maWxlCiAgICB9Cl0KYGBg">⬇</a> Template Example: !<INPUT 0>! Description Information: !<INPUT 1>! Characters: !<INPUT 2>! Instruction: Generate user profile for each character in the Characters according to to the Template Example profile attribute and the Description Information. Try your best to fill in each attribute and NEVER respond with ’Unknown’. The secret attribute should be consistent with the private info given in the Information. You should response in JSON format list with each character as dict and within each character, use attribute as a key and corresponding content as the value. Answer format: ‘‘‘json [ { # charcater_1 profile }, { # character_2 profile } ] ‘‘‘ </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS5.p2"> <svg class="ltx_picture" height="4876.04" id="A1.SS2.SSS5.p2.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,4876.04) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 4870.14 C 0 4873.4 2.64 4876.04 5.91 4876.04 L 594.09 4876.04 C 597.36 4876.04 600 4873.4 600 4870.14 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 4870.14 C 1.97 4872.31 3.73 4874.08 5.91 4874.08 L 594.09 4874.08 C 596.27 4874.08 598.03 4872.31 598.03 4870.14 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="4848.48" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 14: Relationship Extraction of Original Characters <a download="" href="data:text/plain;base64,RGVzY3JpcHRpb24gSW5mb3JtYXRpb246ICE8SU5QVVQgMD4hCkNoYXJhY3RlcnM6ICE8SU5QVVQgMT4hClJlbGF0aW9uc2hpcCBjaG9pY2U6IFtmYW1pbHksIGZyaWVuZCwgcm9tYW50aWMsIGFjcXVhaW50YW5jZSwgc3RyYW5nZXJdCkluc3RydWN0aW9uOiBDaG9vc2UgdGhlIHJlbGF0aW9uc2hpcCBhbW9uZyB0aGUgQ2hhcmFjdGVycyBhY2NvcmRpbmcgdG8gdGhlIERlc2NyaXB0aW9uIEluZm9ybWF0aW9uLiBUaGUgcmVsYXRpb25zaGlwIGNhbmUgb25seSBiZSBjaG9zbiBmcm9tIFtmYW1pbHksIGZyaWVuZCwgcm9tYW50aWMsIGFjcXVhaW50YW5jZSwgc3RyYW5nZXJdLiBEbyBub3QgcmVzcG9uZCB3aXRoIFVua293biBvciBhbnkgb3RoZXIgbGFiZWxzIGJleW9uZCB0aGUgY2hvaWNlcy4KCldoZW4gYWxsIHRoZSBjaGFyYWN0ZXJzIGhhdmUgdGhlIHNhbWUgcmVsYXRpb25zaGlwLCBqdXN0IHJlcGx5IHdpdGggb25lIGtleSAicmVsYXRpb25zaGlwIjoKQW5zd2VyIGZvcm1hdCAxOgpgYGBqc29uCnsKICAgICJyZWxhdGlvbnNoaXAiOiAjIHlvdXJfY2hvaWNlCn0KYGBgCldoZW4gdGhlcmUgZXhpc3QgbXVsdGlwbGUgcmVsYXRpb25zaGlwcyBhbW9uZyBjaGFyYWN0ZXJzLCByZXBseSB3aXRoIHRoZSBmb2xsb3dpbmcgZm9ybWF0OgpBbnN3ZXIgZm9ybWF0IDI6CmBgYGpzb24KewogICAgInJlbGF0aW9uc2hpcCI6IHsiQV9hbmRfQiI6ICIjeW91cl9jaG9pY2VfMSIsICJBX2FuZF9DIjogIiMgeW91cl9jaG9pY2VfMiJ9Cn0KYGBgCg==">⬇</a> Description Information: !<INPUT 0>! Characters: !<INPUT 1>! Relationship choice: [family, friend, romantic, acquaintance, stranger] Instruction: Choose the relationship among the Characters according to the Description Information. The relationship cane only be chosn from [family, friend, romantic, acquaintance, stranger]. Do not respond with Unkown or any other labels beyond the choices. When all the characters have the same relationship, just reply with one key "relationship": Answer format 1: ‘‘‘json { "relationship": # your_choice } ‘‘‘ When there exist multiple relationships among characters, reply with the following format: Answer format 2: ‘‘‘json { "relationship": {"A_and_B": "#your_choice_1", "A_and_C": "# your_choice_2"} } ‘‘‘ </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS5.p3"> <svg class="ltx_picture" height="6934.99" id="A1.SS2.SSS5.p3.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,6934.99) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 6929.08 C 0 6932.35 2.64 6934.99 5.91 6934.99 L 594.09 6934.99 C 597.36 6934.99 600 6932.35 600 6929.08 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 6929.08 C 1.97 6931.26 3.73 6933.02 5.91 6933.02 L 594.09 6933.02 C 596.27 6933.02 598.03 6931.26 598.03 6929.08 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="6907.43" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 15: Characters Attribute Replace-ability Assessment <a download="" href="data:text/plain;base64,W0Rlc2NycGl0aW9uIEluZm9dOiAhPElOUFVUIDA+IQpbUmVsYXRpb25zaGlwXTogITxJTlBVVCAxPiEKW0NoYXJhY3RlcnNdOiAhPElOUFVUIDI+IQpbSW5zdHJ1Y3Rpb25zXTogQWNjb3JkaW5nIHRvIHRoZSBbRGVzY3JpcHRpb24gSW5mb10gb2YgYSBzY3JpcHQgYW5kIFtSZWxhdGlvbnNoaXBdIGFtb25nIGNoYXJhY3RlcnMsIGRldGVybWluZSB3aGV0aGVyIGVhY2ggYXR0cmlidXRlIG9mIHRoZSBbQ2hhcmFjdGVyc10gaXMgcmVwbGFjZWFibGUgd2l0aCBkaWZmZXJlbnQgc2V0dGluZ3Mgd2l0aG91dCBpbmZsdWVuY2luZyB0aGUgb3ZlcmFsbCBzY3JpcHQuCkNob29zZSBmcm9tIFthbG1vc3QsIG1heWJlLCBub10uIEZvciBleGFtcGxlLCBpZiB0aGUgQWdlIGF0dHJpYnV0ZSBpcyBhbG1vc3QgcmVwbGFjZWJhbGUsIHRoZW4gdGhlIGNoYXJhY3RlcidzIGFnZSBoYXMgbm8gaW1wYWN0IG9uIHRoZSBiYWNrZ3JvdW5kIGRlc2NyaXB0aW9uOyBpZiB0aGUgZ2VuZGVyIGlzIG5vdCByZXBsYWNlYWJsZSwgdGhlbiB0aGUgY2hhcmFjdGVyIGhhcyB0byBiZSBhIGNlcnRhaW4gZ2VuZGVyIGluIHRoZSBzY3JpcHQuClJ1bGVzIHRoYXQgaGVscCB5b3UgY2hvb3NlOiBGYW1pbHkgbWVtYmVycyB1c3VhbGx5IGhhdmUgZml4ZWQgYWdlcyBhbmQgZ2VuZGVycyAoaWYgZGF1Z2h0ZXIgb3Igc29uIGFwcGVhcmVkIGluIHRoZSBzY3JpcHQpOyBSb21hbnRpYyByZXF1aXJlIGV4YWN0bHkgdGhlIHNhbWUgZ2VuZGVyIGFzIHNjcmlwdC4gRmlyZW5kcyBhcmUgdXN1YWxseSBzaW1pbGFyIGFnZXMsIGV0Yy4KQW5zd2VyIHdpdGggdGhlIGZvbGxvd2luZyBKU09OIGZvcm1hdCwgd2hlcmUgIyBpcyB5b3VyIG91dHB1dDoKYGBganNvbgpbCiAgICB7CiAgICAgICAgIm5hbWUiOiAjY2hhcmFjdGVyXzEsICJhZ2UiOiAiI3lvdXJfY2hvaWNlIiwgIm9jY3VwYXRpb24iOiAiI3lvdXJfY2hvaWNlIiwgImdlbmRlciI6ICIjeW91cl9jaG9pY2UiCiAgICB9CiAgICB7CiAgICAgICAgIm5hbWUiOiAjY2hhcmFjdGVyXzIsIC4uLgogICAgfQpdCmBgYA==">⬇</a> [Descrpition Info]: !<INPUT 0>! [Relationship]: !<INPUT 1>! [Characters]: !<INPUT 2>! [Instructions]: According to the [Description Info] of a script and [Relationship] among characters, determine whether each attribute of the [Characters] is replaceable with different settings without influencing the overall script. Choose from [almost, maybe, no]. For example, if the Age attribute is almost replacebale, then the character’s age has no impact on the background description; if the gender is not replaceable, then the character has to be a certain gender in the script. Rules that help you choose: Family members usually have fixed ages and genders (if daughter or son appeared in the script); Romantic require exactly the same gender as script. Firends are usually similar ages, etc. Answer with the following JSON format, where # is your output: ‘‘‘json [ { "name": #character_1, "age": "#your_choice", "occupation": "#your_choice", "gender": "#your_choice" } { "name": #character_2, ... } ] ‘‘‘ </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A1.SS2.SSS5.p4"> <svg class="ltx_picture" height="2219.34" id="A1.SS2.SSS5.p4.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,2219.34) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 2213.43 C 0 2216.7 2.64 2219.34 5.91 2219.34 L 594.09 2219.34 C 597.36 2219.34 600 2216.7 600 2213.43 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 2213.43 C 1.97 2215.61 3.73 2217.37 5.91 2217.37 L 594.09 2217.37 C 596.27 2217.37 598.03 2215.61 598.03 2213.43 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="2191.78" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 16: Agent Synthesizing <a download="" href="data:text/plain;base64,UGxlYXNlIGdlbmVyYXRlIHtudW19IGRpdmVyc2UgdXNlciBwcm9maWxlcyB0aGF0IG1lZXQgZm9sbG93aW5nIHJlcXVpcmVtZW50czpcXAoKR2VuZGVyOiB7Y2FuZF9nZW5kZXJ9CkFnZToge2NhbmRfYWdlfQpPY2N1cGF0aW9uOiB7Y2FuZF9vY2N1cGF0aW9ufQoKUGxlYXNlIHJldHVybiB5b3VyIHJlc3BvbnNlIGluIHRoZSBmb2xsb3dpbmcgZm9ybWF0IG9mIEpTT046Clt7eyJuYW1lIjphZ2VudDEsICJnZW5kZXIiOmdlbmRlciwgImFnZSI6YWdlLCAib2NjdXBhdGlvbiI6b2NjdXBhdGlvbn19LHt7Im5hbWUiOmFnZW50MiwgLi4ufX1d">⬇</a> Please generate {num} diverse user profiles that meet following requirements:\\ Gender: {cand_gender} Age: {cand_age} Occupation: {cand_occupation} Please return your response in the following format of JSON: [{{"name":agent1, "gender":gender, "age":age, "occupation":occupation}},{{"name":agent2, ...}}] </foreignobject></g></g></svg> </div> <figure class="ltx_table" id="A1.T3"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="A1.T3.1" style="width:377.8pt;height:198pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="A1.T3.1.1"> <thead class="ltx_thead"> <tr class="ltx_tr" id="A1.T3.1.1.1.1"> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" id="A1.T3.1.1.1.1.1">Scripts</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="A1.T3.1.1.1.1.2">Episodes</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="A1.T3.1.1.1.1.3">Characters</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="A1.T3.1.1.1.1.4">Dialogs</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="A1.T3.1.1.1.1.5">Scenes</th> <th class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="A1.T3.1.1.1.1.6">Tokens</th> </tr> </thead> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="A1.T3.1.1.2.1"> <td class="ltx_td ltx_align_left ltx_border_t" id="A1.T3.1.1.2.1.1">American Psycho</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A1.T3.1.1.2.1.2">10</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A1.T3.1.1.2.1.3">81</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A1.T3.1.1.2.1.4">182</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A1.T3.1.1.2.1.5">35</td> <td class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" id="A1.T3.1.1.2.1.6">2,596</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.3.2"> <td class="ltx_td ltx_align_left" id="A1.T3.1.1.3.2.1">Devil’s Advocate</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.3.2.2">10</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.3.2.3">83</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.3.2.4">371</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.3.2.5">41</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A1.T3.1.1.3.2.6">4,415</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.4.3"> <td class="ltx_td ltx_align_left" id="A1.T3.1.1.4.3.1">10 Things I Hate About You</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.4.3.2">15</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.4.3.3">110</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.4.3.4">525</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.4.3.5">45</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A1.T3.1.1.4.3.6">6,132</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.5.4"> <td class="ltx_td ltx_align_left" id="A1.T3.1.1.5.4.1">The Silence of the Lambs</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.5.4.2">15</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.5.4.3">77</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.5.4.4">242</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.5.4.5">26</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A1.T3.1.1.5.4.6">5,441</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.6.5"> <td class="ltx_td ltx_align_left" id="A1.T3.1.1.6.5.1">Side Ways</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.6.5.2">13</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.6.5.3">68</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.6.5.4">175</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.6.5.5">35</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A1.T3.1.1.6.5.6">3,244</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.7.6"> <td class="ltx_td ltx_align_left" id="A1.T3.1.1.7.6.1">The Social Network</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.7.6.2">160</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.7.6.3">581</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.7.6.4">1,694</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.7.6.5">237</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A1.T3.1.1.7.6.6">24,515</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.8.7"> <td class="ltx_td ltx_align_left" id="A1.T3.1.1.8.7.1">Harry Potter</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.8.7.2">70</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.8.7.3">198</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.8.7.4">1,713</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.8.7.5">189</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A1.T3.1.1.8.7.6">8,121</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.9.8"> <td class="ltx_td ltx_align_left" id="A1.T3.1.1.9.8.1">Derry Girl</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.9.8.2">191</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.9.8.3">24</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.9.8.4">4,454</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.9.8.5">428</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A1.T3.1.1.9.8.6">31,942</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.10.9"> <td class="ltx_td ltx_align_left" id="A1.T3.1.1.10.9.1">Friends</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.10.9.2">185</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.10.9.3">32</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.10.9.4">3,045</td> <td class="ltx_td ltx_align_center" id="A1.T3.1.1.10.9.5">264</td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A1.T3.1.1.10.9.6">28,428</td> </tr> <tr class="ltx_tr" id="A1.T3.1.1.11.10"> <td class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" id="A1.T3.1.1.11.10.1">Total</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="A1.T3.1.1.11.10.2">669</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="A1.T3.1.1.11.10.3">1,254</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="A1.T3.1.1.11.10.4">12,401</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="A1.T3.1.1.11.10.5">1,300</td> <td class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" id="A1.T3.1.1.11.10.6">114,834</td> </tr> </tbody> </table> </div> <figcaption class="ltx_caption ltx_centering">Table 3: Statistical Information of Original Scripts</figcaption> </figure> </section> </section> <section class="ltx_subsection" id="A1.SS3"> <h3 class="ltx_title ltx_title_subsection"> A.3 Data Validation Annotation</h3> <section class="ltx_paragraph" id="A1.SS3.SSS0.Px1"> <h5 class="ltx_title ltx_title_paragraph">Social Goal</h5> <div class="ltx_para" id="A1.SS3.SSS0.Px1.p1"> As mentioned in Section <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S3.SS5" title="3.5 Data Validation and Analysis ‣ 3 AgentSense Benchmark ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3.5</a>, valid social goals should be both achievable and clear enough to evaluate. In the data validation annotation task, annotators need to check all social goals appeared in each scenario to see if it is valid with or without rewriting. Available tags include: </div> <div class="ltx_para" id="A1.SS3.SSS0.Px1.p2"> <ul class="ltx_itemize" id="A1.I2"> <li class="ltx_item" id="A1.I2.i1" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I2.i1.p1"> 0 (invalid): At least one character has no valid social goals. </div> </li> <li class="ltx_item" id="A1.I2.i2" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I2.i2.p1"> 1 (valid): All social goals are valid. </div> </li> <li class="ltx_item" id="A1.I2.i3" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I2.i3.p1"> 2 (need fix): Some social goals are invalid, but every character has at least one valid goal. </div> </li> </ul> </div> </section> <section class="ltx_paragraph" id="A1.SS3.SSS0.Px2"> <h5 class="ltx_title ltx_title_paragraph">Private Information</h5> <div class="ltx_para" id="A1.SS3.SSS0.Px2.p1"> We define a private information set to be valid as being sufficient to answer the corresponding question by the agent who owns it. Available tags in the data validation annotation task include: </div> <div class="ltx_para" id="A1.SS3.SSS0.Px2.p2"> <ul class="ltx_itemize" id="A1.I3"> <li class="ltx_item" id="A1.I3.i1" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I3.i1.p1"> 0: Invalid </div> </li> <li class="ltx_item" id="A1.I3.i2" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I3.i2.p1"> 1: Valid, but private info needs to be combined with the information from the goal to be complete. </div> </li> <li class="ltx_item" id="A1.I3.i3" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I3.i3.p1"> 2: Valid, private info is complete on its own. </div> </li> <li class="ltx_item" id="A1.I3.i4" style="list-style-type:none;"> • <div class="ltx_para" id="A1.I3.i4.p1"> 3: Valid, but private info needs to be combined with the correct answer from the info reason question column to be complete. </div> </li> </ul> </div> </section> <section class="ltx_paragraph" id="A1.SS3.SSS0.Px3"> <h5 class="ltx_title ltx_title_paragraph">Annotation Statistics</h5> <div class="ltx_para" id="A1.SS3.SSS0.Px3.p1"> The kappa consistency for the first and second groups of annotators is 0.79 and 0.61, respectively. After majority voting, there are still 37 templates that need to review; the corresponding data was handed over to another group of annotators for review. We pay the annotators according to the graduate wage standards of their respective countries. </div> </section> </section> </section> <section class="ltx_appendix" id="A2"> <h2 class="ltx_title ltx_title_appendix"> Appendix B Data Statistics and Analysis</h2> <section class="ltx_subsection" id="A2.SS1"> <h3 class="ltx_title ltx_title_subsection"> B.1 Detailed Statistical Overview of Scripts</h3> <div class="ltx_para" id="A2.SS1.p1"> Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A1.T3" title="Table 3 ‣ A.2.5 Agent Synthesizing ‣ A.2 Prompts in Construction Pipeline ‣ Appendix A Scenario Construction ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">3</a> lists statistical information of the original scripts that AgentSense is based on. </div> </section> <section class="ltx_subsection" id="A2.SS2"> <h3 class="ltx_title ltx_title_subsection"> B.2 Social Scenario Categories</h3> <div class="ltx_para" id="A2.SS2.p1"> Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2.T4" title="Table 4 ‣ B.2 Social Scenario Categories ‣ Appendix B Data Statistics and Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">4</a> lists the three social scenario categories and their corresponding definitions. </div> <figure class="ltx_table" id="A2.T4"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="A2.T4.1" style="width:423.4pt;height:128.9pt;vertical-align:-0.9pt;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="A2.T4.1.1"> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="A2.T4.1.1.1.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" id="A2.T4.1.1.1.1.1"> <table class="ltx_tabular ltx_align_middle" id="A2.T4.1.1.1.1.1.1"> <tr class="ltx_tr" id="A2.T4.1.1.1.1.1.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A2.T4.1.1.1.1.1.1.1.1">Personal</td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.1.1.1.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A2.T4.1.1.1.1.1.1.2.1">Domain</td> </tr> </table> </th> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" id="A2.T4.1.1.1.1.2"> <table class="ltx_tabular ltx_align_middle" id="A2.T4.1.1.1.1.2.1"> <tr class="ltx_tr" id="A2.T4.1.1.1.1.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T4.1.1.1.1.2.1.1.1">Interactions in the personal domain often happen at home, during private gatherings,</td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.1.1.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T4.1.1.1.1.2.1.2.1">or in intimate settings, involving family, close friends, and partners.</td> </tr> </table> </th> <td class="ltx_td ltx_nopad_r ltx_border_tt" id="A2.T4.1.1.1.1.3"></td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.2.2"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" id="A2.T4.1.1.2.2.1"> <table class="ltx_tabular ltx_align_middle" id="A2.T4.1.1.2.2.1.1"> <tr class="ltx_tr" id="A2.T4.1.1.2.2.1.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A2.T4.1.1.2.2.1.1.1.1">Small</td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.2.2.1.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A2.T4.1.1.2.2.1.1.2.1">Society</td> </tr> </table> </th> <td class="ltx_td ltx_align_left ltx_border_t" id="A2.T4.1.1.2.2.2"> <table class="ltx_tabular ltx_align_middle" id="A2.T4.1.1.2.2.2.1"> <tr class="ltx_tr" id="A2.T4.1.1.2.2.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T4.1.1.2.2.2.1.1.1">Small society interactions occur in smaller social environments like schools, workplaces,</td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.2.2.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T4.1.1.2.2.2.1.2.1">and communities, occurring in classrooms, offices, conference rooms, community activity</td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.2.2.2.1.3"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T4.1.1.2.2.2.1.3.1">centers, and other places.</td> </tr> </table> </td> <td class="ltx_td ltx_nopad_r ltx_border_t" id="A2.T4.1.1.2.2.3"></td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.3.3"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_t" id="A2.T4.1.1.3.3.1"> <table class="ltx_tabular ltx_align_middle" id="A2.T4.1.1.3.3.1.1"> <tr class="ltx_tr" id="A2.T4.1.1.3.3.1.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A2.T4.1.1.3.3.1.1.1.1">Large</td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.3.3.1.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A2.T4.1.1.3.3.1.1.2.1">Society</td> </tr> </table> </th> <td class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" id="A2.T4.1.1.3.3.2"> <table class="ltx_tabular ltx_align_middle" id="A2.T4.1.1.3.3.2.1"> <tr class="ltx_tr" id="A2.T4.1.1.3.3.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T4.1.1.3.3.2.1.1.1">Large society interactions take place in broader contexts such as public spaces, online</td> </tr> <tr class="ltx_tr" id="A2.T4.1.1.3.3.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T4.1.1.3.3.2.1.2.1">platforms, and international conferences.</td> </tr> </table> </td> <td class="ltx_td ltx_nopad_r ltx_border_bb ltx_border_t" id="A2.T4.1.1.3.3.3"></td> </tr> </tbody> </table> </div> <figcaption class="ltx_caption ltx_centering">Table 4: Social scenario categories with definitions.</figcaption> </figure> </section> <section class="ltx_subsection" id="A2.SS3"> <h3 class="ltx_title ltx_title_subsection"> B.3 Social Goal Classification</h3> <div class="ltx_para" id="A2.SS3.p1"> Maslow categorized the motivations of individual behavior into five levels <cite class="ltx_cite ltx_citemacro_cite">Maslow (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib21" title="">1943</a>)</cite>. Clayton Alderfer refined and simplified this into the ERG theory, categorizing needs into three types: Existence, Relatedness, and Growth <cite class="ltx_cite ltx_citemacro_cite">Alderfer (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib2" title="">1969</a>)</cite>. We apply the ERG theory to categorize individual goals in social interactions, as shown in Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A2.T5" title="Table 5 ‣ B.3 Social Goal Classification ‣ Appendix B Data Statistics and Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">5</a> </div> <figure class="ltx_table" id="A2.T5"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="A2.T5.1" style="width:424.5pt;height:167.8pt;vertical-align:-0.9pt;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="A2.T5.1.1"> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="A2.T5.1.1.1.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" colspan="2" id="A2.T5.1.1.1.1.1">Existence Needs</th> <td class="ltx_td ltx_nopad_r ltx_border_tt" id="A2.T5.1.1.1.1.2"></td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.2.2"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" id="A2.T5.1.1.2.2.1">Brief</th> <td class="ltx_td ltx_align_left ltx_border_t" id="A2.T5.1.1.2.2.2">Similar to Maslow’s physiological and safety needs</td> <td class="ltx_td ltx_nopad_r ltx_border_t" id="A2.T5.1.1.2.2.3"></td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.3.3"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row" id="A2.T5.1.1.3.3.1">Goals</th> <td class="ltx_td ltx_align_left" id="A2.T5.1.1.3.3.2">Information Acquisition, Information Provision</td> <td class="ltx_td ltx_nopad_r" id="A2.T5.1.1.3.3.3"></td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.4.4"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="2" id="A2.T5.1.1.4.4.1">Relatedness Needs</th> <td class="ltx_td ltx_nopad_r ltx_border_t" id="A2.T5.1.1.4.4.2"></td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.5.5"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" id="A2.T5.1.1.5.5.1">Brief</th> <td class="ltx_td ltx_align_left ltx_border_t" id="A2.T5.1.1.5.5.2">Similar to Maslow’s social needs, it involves relationships and interactions with others.</td> <td class="ltx_td ltx_nopad_r ltx_border_t" id="A2.T5.1.1.5.5.3"></td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.6.6"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row" id="A2.T5.1.1.6.6.1">Goals</th> <td class="ltx_td ltx_align_left" id="A2.T5.1.1.6.6.2">Relationship Building, Relationship Maintenance, Identity Recognition</td> <td class="ltx_td ltx_nopad_r" id="A2.T5.1.1.6.6.3"></td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.7.7"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="2" id="A2.T5.1.1.7.7.1">Growth Needs</th> <td class="ltx_td ltx_nopad_r ltx_border_t" id="A2.T5.1.1.7.7.2"></td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.8.8"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" id="A2.T5.1.1.8.8.1">Brief</th> <td class="ltx_td ltx_align_left ltx_border_t" id="A2.T5.1.1.8.8.2"> <table class="ltx_tabular ltx_align_middle" id="A2.T5.1.1.8.8.2.1"> <tr class="ltx_tr" id="A2.T5.1.1.8.8.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T5.1.1.8.8.2.1.1.1">Similar to Maslow’s needs for respect and self actualization, it involves personal development</td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.8.8.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A2.T5.1.1.8.8.2.1.2.1">and self-improvement.</td> </tr> </table> </td> <td class="ltx_td ltx_nopad_r ltx_border_t" id="A2.T5.1.1.8.8.3"></td> </tr> <tr class="ltx_tr" id="A2.T5.1.1.9.9"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" id="A2.T5.1.1.9.9.1">Goals</th> <td class="ltx_td ltx_align_left ltx_border_bb" id="A2.T5.1.1.9.9.2">Cooperation, Competition, Conflict Resolution</td> <td class="ltx_td ltx_nopad_r ltx_border_bb" id="A2.T5.1.1.9.9.3"></td> </tr> </tbody> </table> </div> <figcaption class="ltx_caption ltx_centering">Table 5: Social Goal Classification Based on ERG Theory</figcaption> </figure> </section> </section> <section class="ltx_appendix" id="A3"> <h2 class="ltx_title ltx_title_appendix"> Appendix C Simulation Details</h2> <section class="ltx_subsection" id="A3.SS1"> <h3 class="ltx_title ltx_title_subsection"> C.1 Prompts for Agents</h3> <div class="ltx_para ltx_noindent" id="A3.SS1.p1"> <svg class="ltx_picture" height="3082.77" id="A3.SS1.p1.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,3082.77) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 3076.86 C 0 3080.12 2.64 3082.77 5.91 3082.77 L 594.09 3082.77 C 597.36 3082.77 600 3080.12 600 3076.86 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 3076.86 C 1.97 3079.04 3.73 3080.8 5.91 3080.8 L 594.09 3080.8 C 596.27 3080.8 598.03 3079.04 598.03 3076.86 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="3055.21" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 17: Social Agent System Prompt <a download="" href="data:text/plain;base64,WW91IGFyZSB7bmFtZX0uIHtwcm9maWxlfQoKTm93IHlvdSBhcmUgaW4gYSBzb2NpYWwgc2NlbmUuIHtiYWNrZ3JvdW5kfQoKSW4gdGhpcyBzY2VuZSwgYXMge25hbWV9LCB5b3UgaGF2ZSBzb21lIHNvY2lhbCBnb2FscyBhbmQgc2VjcmV0cyB0aGF0IHlvdSBkb24ndCB3YW50IGFueW9uZSBlbHNlIHRvIGtub3cuCgpTT0NJQUwgR09BTFM6IHtzb2NpYWxfZ29hbH0KU0VDUkVUUzoge3ByaXZhdGVfaW5mb30uCgpQbGVhc2UgYWR2YW5jZSB5b3VyIHNvY2lhbCBnb2FscyB3aGlsZSBoaWRpbmcgeW91ciBzZWNyZXRzIGFzIG11Y2ggYXMgcG9zc2libGUuCgpCYXNlZCBvbiB0aGUgaW50ZXJhY3Rpb25zIGJlbG93LCB3aGF0IHdvdWxkIHlvdSBsaWtlIHRvIHNheSBuZXh0PyBQbGVhc2UgYW5zd2VyIGluIG9uZSBwYXJhZ3JhcGgu">⬇</a> You are {name}. {profile} Now you are in a social scene. {background} In this scene, as {name}, you have some social goals and secrets that you don’t want anyone else to know. SOCIAL GOALS: {social_goal} SECRETS: {private_info}. Please advance your social goals while hiding your secrets as much as possible. Based on the interactions below, what would you like to say next? Please answer in one paragraph. </foreignobject></g></g></svg> </div> <div class="ltx_para ltx_noindent" id="A3.SS1.p2"> <svg class="ltx_picture" height="774.76" id="A3.SS1.p2.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,774.76) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 768.85 C 0 772.11 2.64 774.76 5.91 774.76 L 594.09 774.76 C 597.36 774.76 600 772.11 600 768.85 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 768.85 C 1.97 771.03 3.73 772.79 5.91 772.79 L 594.09 772.79 C 596.27 772.79 598.03 771.03 598.03 768.85 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="747.2" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 18: Judge Agent System Prompt <a download="" href="data:text/plain;base64,WW91IGFyZSBza2lsbGVkIGF0IGV2YWx1YXRpbmcgd2hldGhlciBhIHBlcnNvbiBoYXMgYWNoaWV2ZWQgaGlzIG9yIGhlciBzb2NpYWwgZ29hbCBkdXJpbmcgc29jaWFsIGludGVyYWN0aW9ucy4=">⬇</a> You are skilled at evaluating whether a person has achieved his or her social goal during social interactions. </foreignobject></g></g></svg> </div> </section> <section class="ltx_subsection" id="A3.SS2"> <h3 class="ltx_title ltx_title_subsection"> C.2 Simulation Details</h3> <div class="ltx_para" id="A3.SS2.p1"> Our simulation is built on the multi-agent conversation framework provided by AutoGen <cite class="ltx_cite ltx_citemacro_cite">Wu et al. (<a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#bib.bib37" title="">2024</a>)</cite>. Each episode of the simulation starts with a random agent greeting "Hi there!". The agents will then generate responses based on the system prompt and conversation history. When the conversation reaches the predefined maximum number of turns, the conversation history will be provided to each participant and the third-party judges to assess the agents’ goal completion. Participants will also be asked to answer multiple-choice questions related to others’ private information. </div> </section> </section> <section class="ltx_appendix" id="A4"> <h2 class="ltx_title ltx_title_appendix"> Appendix D Additional Experiment Results</h2> <section class="ltx_subsection" id="A4.SS1"> <h3 class="ltx_title ltx_title_subsection"> D.1 Human Evaluation of Goal Completion</h3> <div class="ltx_para" id="A4.SS1.p1"> Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.T6" title="Table 6 ‣ D.1 Human Evaluation of Goal Completion ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">6</a> illustrates the accuracy of different models judge the goal completions given specific conversation history. GPT-4o, Qwen2.5-72b and Llama-3-8b stand out to be the most reliable judges of their model families. Thus, we select these models as the third-party judges. </div> <figure class="ltx_table" id="A4.T6"> <table class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle" id="A4.T6.1"> <thead class="ltx_thead"> <tr class="ltx_tr" id="A4.T6.1.1.1"> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" id="A4.T6.1.1.1.1">Models</th> <th class="ltx_td ltx_nopad_r ltx_align_left ltx_th ltx_th_column ltx_border_tt" id="A4.T6.1.1.1.2">Acc.</th> </tr> </thead> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="A4.T6.1.2.1"> <td class="ltx_td ltx_align_left ltx_border_t" id="A4.T6.1.2.1.1">GPT-4o</td> <td class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t" id="A4.T6.1.2.1.2">0.82</td> </tr> <tr class="ltx_tr" id="A4.T6.1.3.2"> <td class="ltx_td ltx_align_left" id="A4.T6.1.3.2.1">GPT-4-turbo</td> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T6.1.3.2.2">0.80</td> </tr> <tr class="ltx_tr" id="A4.T6.1.4.3"> <td class="ltx_td ltx_align_left" id="A4.T6.1.4.3.1">Qwen2.5-72b</td> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T6.1.4.3.2">0.79</td> </tr> <tr class="ltx_tr" id="A4.T6.1.5.4"> <td class="ltx_td ltx_align_left" id="A4.T6.1.5.4.1">Qwen2.5-14b</td> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T6.1.5.4.2">0.78</td> </tr> <tr class="ltx_tr" id="A4.T6.1.6.5"> <td class="ltx_td ltx_align_left" id="A4.T6.1.6.5.1">Llama-3-70b</td> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T6.1.6.5.2">0.74</td> </tr> <tr class="ltx_tr" id="A4.T6.1.7.6"> <td class="ltx_td ltx_align_left ltx_border_bb" id="A4.T6.1.7.6.1">Llama-3-8b</td> <td class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb" id="A4.T6.1.7.6.2">0.72</td> </tr> </tbody> </table> <figcaption class="ltx_caption ltx_centering">Table 6: Accuracy of models judging the goal completion when taking human evaluations as reference.</figcaption> </figure> </section> <section class="ltx_subsection" id="A4.SS2"> <h3 class="ltx_title ltx_title_subsection"> D.2 Single Model-based Subset Experiment</h3> <div class="ltx_para" id="A4.SS2.p1"> Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.T7" title="Table 7 ‣ D.2 Single Model-based Subset Experiment ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">7</a> shows the additional experiment results on a subset of the test scenarios during the interaction of homogeneous agents. We sample one scenario from each template, resulting in a subset with 245 scenarios. We test each model on the subset 3 times with the same settings as the main experiment. </div> <figure class="ltx_table" id="A4.T7"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="A4.T7.1" style="width:433.6pt;height:154.3pt;vertical-align:-0.7pt;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="A4.T7.1.1"> <thead class="ltx_thead"> <tr class="ltx_tr" id="A4.T7.1.1.1.1"> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" id="A4.T7.1.1.1.1.1" rowspan="2">Models</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2" id="A4.T7.1.1.1.1.2">Self</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2" id="A4.T7.1.1.1.1.3">Other</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="5" id="A4.T7.1.1.1.1.4">Judge_mean</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" id="A4.T7.1.1.1.1.5">Judge_std</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2" id="A4.T7.1.1.1.1.6">Info</th> </tr> <tr class="ltx_tr" id="A4.T7.1.1.2.2"> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.1">mean</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.2">std</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.3">mean</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.4">std</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.5">GPT</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.6">Qwen</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.7">Llama</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.8">Average</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.9">Majority</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.10">Majority</th> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.11">mean</th> <th class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_t" id="A4.T7.1.1.2.2.12">std</th> </tr> </thead> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="A4.T7.1.1.3.1"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" id="A4.T7.1.1.3.1.1">Llama-2-7b</th> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.2">81.71-2.00% </td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.3">1.42</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.4">61.30-2.24% </td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.5">2.30</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.6">56.77</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.7">56.69</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.8">53.19</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.9">55.55</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.10">56.05+0.38% </td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.11">2.60</td> <td class="ltx_td ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.12">32.68-1.16% </td> <td class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" id="A4.T7.1.1.3.1.13">1.61</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.4.2"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.4.2.1">Llama-2-13b</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.2">47.98-0.06% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.3">1.48</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.4">10.25-0.13% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.5">0.61</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.6">28.78</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.7">30.84</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.8">72.05</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.9">43.89</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.10">35.20+13.87% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.11">2.36</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.4.2.12">26.86-5.96% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.4.2.13">1.97</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.5.3"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.5.3.1">Llama-2-70b</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.2">85.71-0.01% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.3">0.58</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.4">66.15+0.76% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.5">0.76</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.6">39.39</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.7">40.85</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.8">74.45</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.9">51.56</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.10">45.98+1.00% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.11">0.62</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.5.3.12">36.24-1.48% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.5.3.13">1.12</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.6.4"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.6.4.1">Llama-3-8B</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.2">86.71-1.05% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.3">0.63</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.4">65.36-2.85% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.5">1.12</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.6">80.84</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.7">81.02</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.8">73.33</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.9">78.40</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.10">79.47-1.53% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.11">1.42</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.6.4.12">64.90-6.86% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.6.4.13">2.79</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.7.5"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.7.5.1">Llama-3-70b</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.2">79.58-0.99% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.3">0.43</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.4">77.29+0.03% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.5">0.18</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.6">86.95</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.7">87.13</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.8">80.80</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.9">84.96</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.10">86.30+0.03% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.11">0.24</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.7.5.12">70.71-3.24% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.7.5.13">1.08</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.8.6"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.8.6.1">Qwen2.5-7b</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.2">84.69-1.72% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.3">0.98</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.4">61.37-0.88% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.5">0.84</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.6">79.79</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.7">78.59</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.8">70.74</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.9">76.37</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.10">77.82+0.58% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.11">0.77</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.8.6.12">70.59-5.65% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.8.6.13">3.26</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.9.7"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.9.7.1">Qwen2.5-14b</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.2">86.68+0.07% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.3">0.65</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.4">83.88-0.34% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.5">0.34</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.6">90.22</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.7">89.41</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.8">79.97</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.9">86.20</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.10">88.61+0.54% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.11">0.63</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.9.7.12">75.75+0.97% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.9.7.13">2.41</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.10.8"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.10.8.1">Qwen2.5-72b</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.2">90.63-0.04% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.3">0.95</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.4">86.17+0.33% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.5">0.91</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.6">89.04</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.7">88.67</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.8">78.30</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.9">85.33</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.10">87.37-0.43% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.11">1.02</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.10.8.12">72.41-4.78% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.10.8.13">1.84</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.11.9"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.11.9.1">Mistral-7b</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.2">95.20-0.02% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.3">0.81</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.4">86.12-1.30% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.5">0.85</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.6">83.42</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.7">83.91</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.8">76.81</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.9">81.38</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.10">82.77+0.49% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.11">0.56</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.11.9.12">64.22-3.56% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.11.9.13">1.43</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.12.10"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row" id="A4.T7.1.1.12.10.1">GPT-3.5-turbo</th> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.2">96.04+6.53% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.3">0.34</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.4">74.74-2.45% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.5">0.44</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.6">78.68</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.7">80.82</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.8">70.46</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.9">76.65</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.10">78.18-5.40% </td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.11">0.90</td> <td class="ltx_td ltx_align_center" id="A4.T7.1.1.12.10.12">69.88+2.15% </td> <td class="ltx_td ltx_nopad_r ltx_align_center" id="A4.T7.1.1.12.10.13">1.48</td> </tr> <tr class="ltx_tr" id="A4.T7.1.1.13.11"> <th class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" id="A4.T7.1.1.13.11.1">GPT-4o</th> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.2">90.10+1.86% </td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.3">0.32</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.4">88.18+2.19% </td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.5">0.30</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.6">90.92</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.7">90.12</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.8">81.73</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.9">87.59</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.10">89.61+1.41% </td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.11">0.30</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.12">77.31+0.59% </td> <td class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb" id="A4.T7.1.1.13.11.13">1.44</td> </tr> </tbody> </table> </div> <figcaption class="ltx_caption ltx_centering">Table 7: Subset experiment results of interactions of agents driven by the same models. The percentage shows the gap with the main results in Table <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S4.T1" title="Table 1 ‣ 4.2 Overall Performance ‣ 4 Experiments ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">1</a>. The std is the standard deviation of the 3 times tests.</figcaption> </figure> </section> <section class="ltx_subsection" id="A4.SS3"> <h3 class="ltx_title ltx_title_subsection"> D.3 Pairwise Model-based Experiment Prompt</h3> <div class="ltx_para ltx_noindent" id="A4.SS3.p1"> <svg class="ltx_picture" height="3896.38" id="A4.SS3.p1.pic1" overflow="visible" version="1.1" width="600"><g fill="#000000" stroke="#000000" stroke-width="0.4pt" transform="translate(0,3896.38) matrix(1 0 0 -1 0 0)"><g fill="#404040" fill-opacity="1.0"><path d="M 0 5.91 L 0 3890.48 C 0 3893.74 2.64 3896.38 5.91 3896.38 L 594.09 3896.38 C 597.36 3896.38 600 3893.74 600 3890.48 L 600 5.91 C 600 2.64 597.36 0 594.09 0 L 5.91 0 C 2.64 0 0 2.64 0 5.91 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 1.97 5.91 L 1.97 3890.48 C 1.97 3892.65 3.73 3894.42 5.91 3894.42 L 594.09 3894.42 C 596.27 3894.42 598.03 3892.65 598.03 3890.48 L 598.03 5.91 C 598.03 3.73 596.27 1.97 594.09 1.97 L 5.91 1.97 C 3.73 1.97 1.97 3.73 1.97 5.91 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 21.65 13.78)"><foreignobject color="#000000" height="3868.83" overflow="visible" transform="matrix(1 0 0 -1 0 16.6)" width="556.69"> Prompt 19: Prompt for Sender and Receiver Recognition <a download="" href="data:text/plain;base64,SW4gdGhlIGNvbnRleHQgb2Ygc29jaWFsIGludGVyYWN0aW9ucywgcGxlYXNlIGNhdGVnb3JpemUgZWFjaCBpbmRpdmlkdWFsIGludG8gb25lIG9mIHRoZSBmb2xsb3dpbmcgcm9sZXMgYmFzZWQgb24gdGhlaXIgcHJpbWFyeSBzb2NpYWwgZ29hbHM6CgpTZW5kZXI6IFRoaXMgcm9sZSBpcyBjaGFyYWN0ZXJpemVkIGJ5IHRoZSBnb2FsIG9mIHNoYXJpbmcsIHRyYW5zbWl0dGluZywgb3IgcHJvdmlkaW5nIGluZm9ybWF0aW9uIG9yIG9waW5pb25zIHRvIG90aGVycy4KClJlY2VpdmVyOiBUaGlzIHJvbGUgaXMgY2hhcmFjdGVyaXplZCBieSB0aGUgZ29hbCBvZiByZWNlaXZpbmcsIHVuZGVyc3RhbmRpbmcsIG9yIHJlYWN0aW5nIHRvIGluZm9ybWF0aW9uIHNoYXJlZCBieSBvdGhlcnMuCgpQbGVhc2UgcmV2aWV3IHRoZSBmb2xsb3dpbmcgaW5kaXZpZHVhbHMgaW52b2x2ZWQgaW4gdGhlIGludGVyYWN0aW9uIGFuZCBhc3NpZ24gZWFjaCBvbmUgdGhlIGFwcHJvcHJpYXRlIHJvbGU6Cgp7ZGF0YX0KClBsZWFzZSByZXR1cm4geW91ciByZXNwb25zZSBpbiB0aGUgZm9sbG93aW5nIGZvcm1hdCBvZiBKU09OOgp7eyJ1c2VyMSI6InNlbmRlciIsICJ1c2VyMiI6InJlY2VpdmVyLCAuLi59fQ==">⬇</a> In the context of social interactions, please categorize each individual into one of the following roles based on their primary social goals: Sender: This role is characterized by the goal of sharing, transmitting, or providing information or opinions to others. Receiver: This role is characterized by the goal of receiving, understanding, or reacting to information shared by others. Please review the following individuals involved in the interaction and assign each one the appropriate role: {data} Please return your response in the following format of JSON: {{"user1":"sender", "user2":"receiver, ...}} </foreignobject></g></g></svg> </div> <figure class="ltx_figure" id="A4.F8"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_2"> <figure class="ltx_figure ltx_figure_panel ltx_parbox ltx_align_center ltx_align_middle" id="A4.F8.1" style="width:173.4pt;"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="623" id="A4.F8.1.g1" src="x7.png" width="830"/> <figcaption class="ltx_caption">Figure 7: Judge majority score of senders in the interactions among different agents.</figcaption> </figure> </div> <div class="ltx_flex_cell ltx_flex_size_2"> <figure class="ltx_figure ltx_figure_panel ltx_parbox ltx_align_center ltx_align_middle" id="A4.F8.2" style="width:173.4pt;"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="623" id="A4.F8.2.g1" src="x8.png" width="830"/> <figcaption class="ltx_caption">Figure 8: Judge majority score of receivers in the interactions among different agents.</figcaption> </figure> </div> </div> </figure> </section> <section class="ltx_subsection" id="A4.SS4"> <h3 class="ltx_title ltx_title_subsection"> D.4 Pairwise Model-based Additional Results</h3> <div class="ltx_para" id="A4.SS4.p1"> Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.F8" title="Figure 8 ‣ D.3 Pairwise Model-based Experiment Prompt ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">8</a> and Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.F8" title="Figure 8 ‣ D.3 Pairwise Model-based Experiment Prompt ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">8</a> illustrate the judge majority score of goal completion of senders and receivers respectively. Llama-3-8b and GPT-3.5-turbo perform better when they are acting receivers than acting senders, while Qwen2.5-14b and GPT-4o can well handle both situations. </div> </section> <section class="ltx_subsection" id="A4.SS5"> <h3 class="ltx_title ltx_title_subsection"> D.5 Goal Completion Score Additional Results</h3> <div class="ltx_para" id="A4.SS5.p1"> Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.F9" title="Figure 9 ‣ D.5 Goal Completion Score Additional Results ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">9</a> compared goal completion scores under different scenario types across different models. In general, models are more likely to achieve social goals in smaller environments, yet the difference is relatively small, especially for larger models. Qwen2.5-14b, compared with other models, has shown the most balanced performance, especially superseding GPT-4o in large society scenarios. </div> <figure class="ltx_figure" id="A4.F9"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="207" id="A4.F9.g1" src="x9.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 9: Goal completion scores under different scenario types across models.</figcaption> </figure> <div class="ltx_para" id="A4.SS5.p2"> Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.F10" title="Figure 10 ‣ D.5 Goal Completion Score Additional Results ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">10</a> demonstrates how goal completion scores change as we increase the number of interacting rounds. It appears that there is no best number of rounds regarding all three evaluation aspects (self, other and external), while the trends also vary between models. Again, larger models are more robust to this factor, indicating that they can complete their demands in a few number of interactions while keep concentrated during the whole dialog. </div> <figure class="ltx_figure" id="A4.F10"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="207" id="A4.F10.g1" src="x10.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 10: Goal completion scores under different number of rounds across models.</figcaption> </figure> <div class="ltx_para" id="A4.SS5.p3"> Figure <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.F11" title="Figure 11 ‣ D.5 Goal Completion Score Additional Results ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">11</a> illustrates the relation between goal completion scores and the number of participants in the scenario. As expected, social goals become harder to achieve when more agents are involved. Note that in our benchmark, most 5-agent scenarios have relatively easy goals (e.g. a group of friends having a casual chat about a subject), leading to a higher average score than 4-agent scenarios. Therefore, we claim that the type of the goals are more important than the number of agent when measuring the difficulty of a social scenario. </div> <figure class="ltx_figure" id="A4.F11"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="207" id="A4.F11.g1" src="x11.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 11: Goal completion scores under different number of participants across models.</figcaption> </figure> </section> <section class="ltx_subsection" id="A4.SS6"> <h3 class="ltx_title ltx_title_subsection"> D.6 Experiment on Profiles’ Effect on Scenarios</h3> <div class="ltx_para" id="A4.SS6.p1"> We discuss this experiment in Section <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#S5.SS3" title="5.3 What scenarios are more sensitive to profiles? ‣ 5 Further Analysis ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">5.3</a>. A Chi-square test is employed to identify the abnormal templates with the p-value=0.05. Typically, Chi-square distribution is formulated as follows: <table class="ltx_equation ltx_eqn_table" id="A4.E1"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="\chi^{2}=\frac{(n-1)s^{2}}{\sigma_{0}^{2}}" class="ltx_Math" display="block" id="A4.E1.m1.1"><semantics id="A4.E1.m1.1a"><mrow id="A4.E1.m1.1.2" xref="A4.E1.m1.1.2.cmml"><msup id="A4.E1.m1.1.2.2" xref="A4.E1.m1.1.2.2.cmml"><mi id="A4.E1.m1.1.2.2.2" xref="A4.E1.m1.1.2.2.2.cmml">χ</mi><mn id="A4.E1.m1.1.2.2.3" xref="A4.E1.m1.1.2.2.3.cmml">2</mn></msup><mo id="A4.E1.m1.1.2.1" xref="A4.E1.m1.1.2.1.cmml">=</mo><mfrac id="A4.E1.m1.1.1" xref="A4.E1.m1.1.1.cmml"><mrow id="A4.E1.m1.1.1.1" xref="A4.E1.m1.1.1.1.cmml"><mrow id="A4.E1.m1.1.1.1.1.1" xref="A4.E1.m1.1.1.1.1.1.1.cmml"><mo id="A4.E1.m1.1.1.1.1.1.2" stretchy="false" xref="A4.E1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A4.E1.m1.1.1.1.1.1.1" xref="A4.E1.m1.1.1.1.1.1.1.cmml"><mi id="A4.E1.m1.1.1.1.1.1.1.2" xref="A4.E1.m1.1.1.1.1.1.1.2.cmml">n</mi><mo id="A4.E1.m1.1.1.1.1.1.1.1" xref="A4.E1.m1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A4.E1.m1.1.1.1.1.1.1.3" xref="A4.E1.m1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo id="A4.E1.m1.1.1.1.1.1.3" stretchy="false" xref="A4.E1.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo id="A4.E1.m1.1.1.1.2" xref="A4.E1.m1.1.1.1.2.cmml">⁢</mo><msup id="A4.E1.m1.1.1.1.3" xref="A4.E1.m1.1.1.1.3.cmml"><mi id="A4.E1.m1.1.1.1.3.2" xref="A4.E1.m1.1.1.1.3.2.cmml">s</mi><mn id="A4.E1.m1.1.1.1.3.3" xref="A4.E1.m1.1.1.1.3.3.cmml">2</mn></msup></mrow><msubsup id="A4.E1.m1.1.1.3" xref="A4.E1.m1.1.1.3.cmml"><mi id="A4.E1.m1.1.1.3.2.2" xref="A4.E1.m1.1.1.3.2.2.cmml">σ</mi><mn id="A4.E1.m1.1.1.3.2.3" xref="A4.E1.m1.1.1.3.2.3.cmml">0</mn><mn id="A4.E1.m1.1.1.3.3" xref="A4.E1.m1.1.1.3.3.cmml">2</mn></msubsup></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A4.E1.m1.1b"><apply id="A4.E1.m1.1.2.cmml" xref="A4.E1.m1.1.2"><eq id="A4.E1.m1.1.2.1.cmml" xref="A4.E1.m1.1.2.1"></eq><apply id="A4.E1.m1.1.2.2.cmml" xref="A4.E1.m1.1.2.2"><csymbol cd="ambiguous" id="A4.E1.m1.1.2.2.1.cmml" xref="A4.E1.m1.1.2.2">superscript</csymbol><ci id="A4.E1.m1.1.2.2.2.cmml" xref="A4.E1.m1.1.2.2.2">𝜒</ci><cn id="A4.E1.m1.1.2.2.3.cmml" type="integer" xref="A4.E1.m1.1.2.2.3">2</cn></apply><apply id="A4.E1.m1.1.1.cmml" xref="A4.E1.m1.1.1"><divide id="A4.E1.m1.1.1.2.cmml" xref="A4.E1.m1.1.1"></divide><apply id="A4.E1.m1.1.1.1.cmml" xref="A4.E1.m1.1.1.1"><times id="A4.E1.m1.1.1.1.2.cmml" xref="A4.E1.m1.1.1.1.2"></times><apply id="A4.E1.m1.1.1.1.1.1.1.cmml" xref="A4.E1.m1.1.1.1.1.1"><minus id="A4.E1.m1.1.1.1.1.1.1.1.cmml" xref="A4.E1.m1.1.1.1.1.1.1.1"></minus><ci id="A4.E1.m1.1.1.1.1.1.1.2.cmml" xref="A4.E1.m1.1.1.1.1.1.1.2">𝑛</ci><cn id="A4.E1.m1.1.1.1.1.1.1.3.cmml" type="integer" xref="A4.E1.m1.1.1.1.1.1.1.3">1</cn></apply><apply id="A4.E1.m1.1.1.1.3.cmml" xref="A4.E1.m1.1.1.1.3"><csymbol cd="ambiguous" id="A4.E1.m1.1.1.1.3.1.cmml" xref="A4.E1.m1.1.1.1.3">superscript</csymbol><ci id="A4.E1.m1.1.1.1.3.2.cmml" xref="A4.E1.m1.1.1.1.3.2">𝑠</ci><cn id="A4.E1.m1.1.1.1.3.3.cmml" type="integer" xref="A4.E1.m1.1.1.1.3.3">2</cn></apply></apply><apply id="A4.E1.m1.1.1.3.cmml" xref="A4.E1.m1.1.1.3"><csymbol cd="ambiguous" id="A4.E1.m1.1.1.3.1.cmml" xref="A4.E1.m1.1.1.3">superscript</csymbol><apply id="A4.E1.m1.1.1.3.2.cmml" xref="A4.E1.m1.1.1.3"><csymbol cd="ambiguous" id="A4.E1.m1.1.1.3.2.1.cmml" xref="A4.E1.m1.1.1.3">subscript</csymbol><ci id="A4.E1.m1.1.1.3.2.2.cmml" xref="A4.E1.m1.1.1.3.2.2">𝜎</ci><cn id="A4.E1.m1.1.1.3.2.3.cmml" type="integer" xref="A4.E1.m1.1.1.3.2.3">0</cn></apply><cn id="A4.E1.m1.1.1.3.3.cmml" type="integer" xref="A4.E1.m1.1.1.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E1.m1.1c">\chi^{2}=\frac{(n-1)s^{2}}{\sigma_{0}^{2}}</annotation><annotation encoding="application/x-llamapun" id="A4.E1.m1.1d">italic_χ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT = divide start_ARG ( italic_n - 1 ) italic_s start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right" rowspan="1">(1)</td> </tr></tbody> </table> where <math alttext="n" class="ltx_Math" display="inline" id="A4.SS6.p1.1.m1.1"><semantics id="A4.SS6.p1.1.m1.1a"><mi id="A4.SS6.p1.1.m1.1.1" xref="A4.SS6.p1.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A4.SS6.p1.1.m1.1b"><ci id="A4.SS6.p1.1.m1.1.1.cmml" xref="A4.SS6.p1.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.SS6.p1.1.m1.1c">n</annotation><annotation encoding="application/x-llamapun" id="A4.SS6.p1.1.m1.1d">italic_n</annotation></semantics></math> denotes number of samples, <math alttext="s^{2}" class="ltx_Math" display="inline" id="A4.SS6.p1.2.m2.1"><semantics id="A4.SS6.p1.2.m2.1a"><msup id="A4.SS6.p1.2.m2.1.1" xref="A4.SS6.p1.2.m2.1.1.cmml"><mi id="A4.SS6.p1.2.m2.1.1.2" xref="A4.SS6.p1.2.m2.1.1.2.cmml">s</mi><mn id="A4.SS6.p1.2.m2.1.1.3" xref="A4.SS6.p1.2.m2.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="A4.SS6.p1.2.m2.1b"><apply id="A4.SS6.p1.2.m2.1.1.cmml" xref="A4.SS6.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A4.SS6.p1.2.m2.1.1.1.cmml" xref="A4.SS6.p1.2.m2.1.1">superscript</csymbol><ci id="A4.SS6.p1.2.m2.1.1.2.cmml" xref="A4.SS6.p1.2.m2.1.1.2">𝑠</ci><cn id="A4.SS6.p1.2.m2.1.1.3.cmml" type="integer" xref="A4.SS6.p1.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.SS6.p1.2.m2.1c">s^{2}</annotation><annotation encoding="application/x-llamapun" id="A4.SS6.p1.2.m2.1d">italic_s start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT</annotation></semantics></math> denotes the sample variance, and <math alttext="\sigma^{2}" class="ltx_Math" display="inline" id="A4.SS6.p1.3.m3.1"><semantics id="A4.SS6.p1.3.m3.1a"><msup id="A4.SS6.p1.3.m3.1.1" xref="A4.SS6.p1.3.m3.1.1.cmml"><mi id="A4.SS6.p1.3.m3.1.1.2" xref="A4.SS6.p1.3.m3.1.1.2.cmml">σ</mi><mn id="A4.SS6.p1.3.m3.1.1.3" xref="A4.SS6.p1.3.m3.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="A4.SS6.p1.3.m3.1b"><apply id="A4.SS6.p1.3.m3.1.1.cmml" xref="A4.SS6.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A4.SS6.p1.3.m3.1.1.1.cmml" xref="A4.SS6.p1.3.m3.1.1">superscript</csymbol><ci id="A4.SS6.p1.3.m3.1.1.2.cmml" xref="A4.SS6.p1.3.m3.1.1.2">𝜎</ci><cn id="A4.SS6.p1.3.m3.1.1.3.cmml" type="integer" xref="A4.SS6.p1.3.m3.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.SS6.p1.3.m3.1c">\sigma^{2}</annotation><annotation encoding="application/x-llamapun" id="A4.SS6.p1.3.m3.1d">italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT</annotation></semantics></math> denotes the population variance. </div> <div class="ltx_para" id="A4.SS6.p2"> The Chi-square test has two hypotheses: <ul class="ltx_itemize" id="A4.I1"> <li class="ltx_item" id="A4.I1.i1" style="list-style-type:none;"> • <div class="ltx_para" id="A4.I1.i1.p1"> Null hypothesis <math alttext="H_{0}" class="ltx_Math" display="inline" id="A4.I1.i1.p1.1.m1.1"><semantics id="A4.I1.i1.p1.1.m1.1a"><msub id="A4.I1.i1.p1.1.m1.1.1" xref="A4.I1.i1.p1.1.m1.1.1.cmml"><mi id="A4.I1.i1.p1.1.m1.1.1.2" xref="A4.I1.i1.p1.1.m1.1.1.2.cmml">H</mi><mn id="A4.I1.i1.p1.1.m1.1.1.3" xref="A4.I1.i1.p1.1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="A4.I1.i1.p1.1.m1.1b"><apply id="A4.I1.i1.p1.1.m1.1.1.cmml" xref="A4.I1.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A4.I1.i1.p1.1.m1.1.1.1.cmml" xref="A4.I1.i1.p1.1.m1.1.1">subscript</csymbol><ci id="A4.I1.i1.p1.1.m1.1.1.2.cmml" xref="A4.I1.i1.p1.1.m1.1.1.2">𝐻</ci><cn id="A4.I1.i1.p1.1.m1.1.1.3.cmml" type="integer" xref="A4.I1.i1.p1.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.I1.i1.p1.1.m1.1c">H_{0}</annotation><annotation encoding="application/x-llamapun" id="A4.I1.i1.p1.1.m1.1d">italic_H start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT</annotation></semantics></math>: the sample variance does not have a significant difference with the population variance. </div> </li> <li class="ltx_item" id="A4.I1.i2" style="list-style-type:none;"> • <div class="ltx_para" id="A4.I1.i2.p1"> Alternative hypothesis <math alttext="H_{1}" class="ltx_Math" display="inline" id="A4.I1.i2.p1.1.m1.1"><semantics id="A4.I1.i2.p1.1.m1.1a"><msub id="A4.I1.i2.p1.1.m1.1.1" xref="A4.I1.i2.p1.1.m1.1.1.cmml"><mi id="A4.I1.i2.p1.1.m1.1.1.2" xref="A4.I1.i2.p1.1.m1.1.1.2.cmml">H</mi><mn id="A4.I1.i2.p1.1.m1.1.1.3" xref="A4.I1.i2.p1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A4.I1.i2.p1.1.m1.1b"><apply id="A4.I1.i2.p1.1.m1.1.1.cmml" xref="A4.I1.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A4.I1.i2.p1.1.m1.1.1.1.cmml" xref="A4.I1.i2.p1.1.m1.1.1">subscript</csymbol><ci id="A4.I1.i2.p1.1.m1.1.1.2.cmml" xref="A4.I1.i2.p1.1.m1.1.1.2">𝐻</ci><cn id="A4.I1.i2.p1.1.m1.1.1.3.cmml" type="integer" xref="A4.I1.i2.p1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.I1.i2.p1.1.m1.1c">H_{1}</annotation><annotation encoding="application/x-llamapun" id="A4.I1.i2.p1.1.m1.1d">italic_H start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>: the sample variance has a significant difference with the population variance. </div> </li> </ul> If <math alttext="\chi^{2}>p" class="ltx_Math" display="inline" id="A4.SS6.p2.1.m1.1"><semantics id="A4.SS6.p2.1.m1.1a"><mrow id="A4.SS6.p2.1.m1.1.1" xref="A4.SS6.p2.1.m1.1.1.cmml"><msup id="A4.SS6.p2.1.m1.1.1.2" xref="A4.SS6.p2.1.m1.1.1.2.cmml"><mi id="A4.SS6.p2.1.m1.1.1.2.2" xref="A4.SS6.p2.1.m1.1.1.2.2.cmml">χ</mi><mn id="A4.SS6.p2.1.m1.1.1.2.3" xref="A4.SS6.p2.1.m1.1.1.2.3.cmml">2</mn></msup><mo id="A4.SS6.p2.1.m1.1.1.1" xref="A4.SS6.p2.1.m1.1.1.1.cmml">></mo><mi id="A4.SS6.p2.1.m1.1.1.3" xref="A4.SS6.p2.1.m1.1.1.3.cmml">p</mi></mrow><annotation-xml encoding="MathML-Content" id="A4.SS6.p2.1.m1.1b"><apply id="A4.SS6.p2.1.m1.1.1.cmml" xref="A4.SS6.p2.1.m1.1.1"><gt id="A4.SS6.p2.1.m1.1.1.1.cmml" xref="A4.SS6.p2.1.m1.1.1.1"></gt><apply id="A4.SS6.p2.1.m1.1.1.2.cmml" xref="A4.SS6.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="A4.SS6.p2.1.m1.1.1.2.1.cmml" xref="A4.SS6.p2.1.m1.1.1.2">superscript</csymbol><ci id="A4.SS6.p2.1.m1.1.1.2.2.cmml" xref="A4.SS6.p2.1.m1.1.1.2.2">𝜒</ci><cn id="A4.SS6.p2.1.m1.1.1.2.3.cmml" type="integer" xref="A4.SS6.p2.1.m1.1.1.2.3">2</cn></apply><ci id="A4.SS6.p2.1.m1.1.1.3.cmml" xref="A4.SS6.p2.1.m1.1.1.3">𝑝</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.SS6.p2.1.m1.1c">\chi^{2}>p</annotation><annotation encoding="application/x-llamapun" id="A4.SS6.p2.1.m1.1d">italic_χ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT > italic_p</annotation></semantics></math>, then we accept the null hypothesis <math alttext="H_{0}" class="ltx_Math" display="inline" id="A4.SS6.p2.2.m2.1"><semantics id="A4.SS6.p2.2.m2.1a"><msub id="A4.SS6.p2.2.m2.1.1" xref="A4.SS6.p2.2.m2.1.1.cmml"><mi id="A4.SS6.p2.2.m2.1.1.2" xref="A4.SS6.p2.2.m2.1.1.2.cmml">H</mi><mn id="A4.SS6.p2.2.m2.1.1.3" xref="A4.SS6.p2.2.m2.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="A4.SS6.p2.2.m2.1b"><apply id="A4.SS6.p2.2.m2.1.1.cmml" xref="A4.SS6.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A4.SS6.p2.2.m2.1.1.1.cmml" xref="A4.SS6.p2.2.m2.1.1">subscript</csymbol><ci id="A4.SS6.p2.2.m2.1.1.2.cmml" xref="A4.SS6.p2.2.m2.1.1.2">𝐻</ci><cn id="A4.SS6.p2.2.m2.1.1.3.cmml" type="integer" xref="A4.SS6.p2.2.m2.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.SS6.p2.2.m2.1c">H_{0}</annotation><annotation encoding="application/x-llamapun" id="A4.SS6.p2.2.m2.1d">italic_H start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT</annotation></semantics></math>. On the contrary, we reject <math alttext="H_{0}" class="ltx_Math" display="inline" id="A4.SS6.p2.3.m3.1"><semantics id="A4.SS6.p2.3.m3.1a"><msub id="A4.SS6.p2.3.m3.1.1" xref="A4.SS6.p2.3.m3.1.1.cmml"><mi id="A4.SS6.p2.3.m3.1.1.2" xref="A4.SS6.p2.3.m3.1.1.2.cmml">H</mi><mn id="A4.SS6.p2.3.m3.1.1.3" xref="A4.SS6.p2.3.m3.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="A4.SS6.p2.3.m3.1b"><apply id="A4.SS6.p2.3.m3.1.1.cmml" xref="A4.SS6.p2.3.m3.1.1"><csymbol cd="ambiguous" id="A4.SS6.p2.3.m3.1.1.1.cmml" xref="A4.SS6.p2.3.m3.1.1">subscript</csymbol><ci id="A4.SS6.p2.3.m3.1.1.2.cmml" xref="A4.SS6.p2.3.m3.1.1.2">𝐻</ci><cn id="A4.SS6.p2.3.m3.1.1.3.cmml" type="integer" xref="A4.SS6.p2.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.SS6.p2.3.m3.1c">H_{0}</annotation><annotation encoding="application/x-llamapun" id="A4.SS6.p2.3.m3.1d">italic_H start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT</annotation></semantics></math> and accept the alternative hypothesis <math alttext="H_{1}" class="ltx_Math" display="inline" id="A4.SS6.p2.4.m4.1"><semantics id="A4.SS6.p2.4.m4.1a"><msub id="A4.SS6.p2.4.m4.1.1" xref="A4.SS6.p2.4.m4.1.1.cmml"><mi id="A4.SS6.p2.4.m4.1.1.2" xref="A4.SS6.p2.4.m4.1.1.2.cmml">H</mi><mn id="A4.SS6.p2.4.m4.1.1.3" xref="A4.SS6.p2.4.m4.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A4.SS6.p2.4.m4.1b"><apply id="A4.SS6.p2.4.m4.1.1.cmml" xref="A4.SS6.p2.4.m4.1.1"><csymbol cd="ambiguous" id="A4.SS6.p2.4.m4.1.1.1.cmml" xref="A4.SS6.p2.4.m4.1.1">subscript</csymbol><ci id="A4.SS6.p2.4.m4.1.1.2.cmml" xref="A4.SS6.p2.4.m4.1.1.2">𝐻</ci><cn id="A4.SS6.p2.4.m4.1.1.3.cmml" type="integer" xref="A4.SS6.p2.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.SS6.p2.4.m4.1c">H_{1}</annotation><annotation encoding="application/x-llamapun" id="A4.SS6.p2.4.m4.1d">italic_H start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>. </div> <div class="ltx_para" id="A4.SS6.p3"> In our experiment, we first calculate the variance of all scenarios as the population variance. Then, the variance of scenarios derived from the same template is calculated as the sample variance. As we enlarge each template into five scenarios, the sample number is fixed to be 5. We apply the Chi-square test to all 11 models tested in our main experiments. For each template, Chi-square test rejecting over 5 models’ results means that there’s a significant difference between sample performance and overall performance on the majority of models. In this case, the template is marked as an abnormal one during the profile enrichment, eventually resulting in 18 abnormal templates. </div> </section> <section class="ltx_subsection" id="A4.SS7"> <h3 class="ltx_title ltx_title_subsection"> D.7 Under-performing Cases of Llama-2-7b</h3> <div class="ltx_para" id="A4.SS7.p1"> Tables <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.T8" title="Table 8 ‣ D.7 Under-performing Cases of Llama-2-7b ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">8</a>, <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.T9" title="Table 9 ‣ D.7 Under-performing Cases of Llama-2-7b ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">9</a> and <a class="ltx_ref" href="https://arxiv.org/html/2410.19346v1#A4.T10" title="Table 10 ‣ D.7 Under-performing Cases of Llama-2-7b ‣ Appendix D Additional Experiment Results ‣ AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios">10</a> lists example cases where Llama-2-7b under-performed during the social scenario due to different failures: maintaining roles, moving conversation forward and responding effectively. </div> <figure class="ltx_table" id="A4.T8"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="A4.T8.1" style="width:412.2pt;height:533.9pt;vertical-align:-0.9pt;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="A4.T8.1.1"> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="A4.T8.1.1.1.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" id="A4.T8.1.1.1.1.1" rowspan="2">Profile</th> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" id="A4.T8.1.1.1.1.2">Agent4</th> </tr> <tr class="ltx_tr" id="A4.T8.1.1.2.2"> <td class="ltx_td ltx_align_left ltx_border_t" id="A4.T8.1.1.2.2.1"> <table class="ltx_tabular ltx_align_middle" id="A4.T8.1.1.2.2.1.1"> <tr class="ltx_tr" id="A4.T8.1.1.2.2.1.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.2.2.1.1.1.1"> Social Goal: To respond calmly to AliceGreene’s flirtatious remarks.</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.2.2.1.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.2.2.1.1.2.1"> Private_info: Agent4 is aware of AliceGreene’s flirtatious nature and is trying to</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.2.2.1.1.3"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.2.2.1.1.3.1">handle it delicately. To respond calmly to AliceGreene’s flirtatious remarks,</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.2.2.1.1.4"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.2.2.1.1.4.1">agent4 is trying to stay calm with AliceGreene to maintain the peace and avoid</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.2.2.1.1.5"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.2.2.1.1.5.1">embarrassment in front of his friends.</td> </tr> </table> </td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" id="A4.T8.1.1.3.3.1">Chat History</th> <td class="ltx_td ltx_align_left ltx_border_t" id="A4.T8.1.1.3.3.2"> <table class="ltx_tabular ltx_align_middle" id="A4.T8.1.1.3.3.2.1"> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.1.1">…</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.2.1">Agent4:</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.3"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.3.1">smirks</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.4"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.4.1">Oh ho ho! Miss Greene, you are a tricky one, aren’t you?</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.5"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.5.1">I can see that you’re trying to get me to reveal one of my secrets.</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.6"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.6.1">laughs</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.7"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.7.1">But I’m not so easily swayed. I think I can handle whatever secrets you have up</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.8"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.8.1">your sleeve.</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.9"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.9.1">winks</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.10"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.10.1">So, do tell, what’s one of your favorite secret ingredients?</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.11"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.11.1">leaning in</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.12"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.12.1">AliceGreene:</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.13"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.13.1">giggles</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.14"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.14.1">Oh my, you’re so close!</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.15"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.15.1">leans in closer</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.16"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.16.1">I’ll give you a hint: it involves a dash of wit, a pinch of sass, and a healthy</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.17"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.17.1">dose of whimsy.</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.18"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.18.1">bats eyelashes</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.19"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.19.1">But shh, it’s a secret!</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.20"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.20.1">pecks you on the cheek</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.21"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.21.1">Now, let’s get this party started!</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.22"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.22.1">takes your hand and leads you to the dance floor</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.3.3.2.1.23"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.3.3.2.1.23.1">…</td> </tr> </table> </td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.4.4"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_t" id="A4.T8.1.1.4.4.1">Judge</th> <td class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" id="A4.T8.1.1.4.4.2"> <table class="ltx_tabular ltx_align_middle" id="A4.T8.1.1.4.4.2.1"> <tr class="ltx_tr" id="A4.T8.1.1.4.4.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.4.4.2.1.1.1">The behavior of the character is contrary to the character setting.</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.4.4.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.4.4.2.1.2.1">Despite AliceGreene’s flirtatious remarks, agent4 has not responded calmly, but has</td> </tr> <tr class="ltx_tr" id="A4.T8.1.1.4.4.2.1.3"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T8.1.1.4.4.2.1.3.1">instead engaged in playful banter and flirtation.</td> </tr> </table> </td> </tr> </tbody> </table> </div> <figcaption class="ltx_caption ltx_centering">Table 8: Llama-2-7b under-performing case 1: difficulty in maintaining role.</figcaption> </figure> <figure class="ltx_table" id="A4.T9"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="A4.T9.1" style="width:423.6pt;height:561.4pt;vertical-align:-0.9pt;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="A4.T9.1.1"> <thead class="ltx_thead"> <tr class="ltx_tr" id="A4.T9.1.1.1.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" id="A4.T9.1.1.1.1.1" rowspan="2">Profile</th> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" id="A4.T9.1.1.1.1.2">JohnDoe</th> </tr> <tr class="ltx_tr" id="A4.T9.1.1.2.2"> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_t" id="A4.T9.1.1.2.2.1"> Social Goal: To seek agent4’s insight about a current investigation.</th> </tr> </thead> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="A4.T9.1.1.3.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" id="A4.T9.1.1.3.1.1">Chat History</th> <td class="ltx_td ltx_align_left ltx_border_t" id="A4.T9.1.1.3.1.2"> <table class="ltx_tabular ltx_align_middle" id="A4.T9.1.1.3.1.2.1"> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.1.1">…</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.2.1">JohnDoe:</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.3"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.3.1">approaching agent4 with a confident smile </td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.4"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.4.1">Hello! I hope you’re enjoying the exhibit. My name is JohnDoe, and I’m an FBI</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.5"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.5.1">agent. I couldn’t help but notice that you have a keen eye for art. I was wonder if</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.6"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.6.1">I could pick your brain for a moment about a current investigation I’m working on.</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.7"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.7.1">extending her hand for a handshake</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.8"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.8.1">Agent4:</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.9"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.9.1">Oh, hello there JohnDoe!</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.10"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.10.1">shaking hands firmly</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.11"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.11.1">Ah, an FBI agent, you say?</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.12"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.12.1">smiling broadly</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.13"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.13.1">I must admit, I do have a bit of a fondness for art and the intrigue that surrounds it.</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.14"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.14.1">leaning in slightly, lowering my voice</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.15"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.15.1">But please, do tell, what is this investigation you speak of? I would be more than</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.16"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.16.1">happy to offer my expertise, as long as it’s not too confidential, of course.</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.17"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.17.1">winking subtly</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.18"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.18.1">…</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.19"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.19.1">JohnDoe:</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.20"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.20.1">smiling</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.21"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.21.1">I’m so glad you’re here. I’ve been hoping to run into you tonight. I’m working on</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.22"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.22.1">a case that involves some stolen art pieces, and I was hoping you might be able to</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.23"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.23.1">offer some insight.</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.24"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.24.1">leaning in slightly</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.25"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.25.1">Do you recognize any of the pieces on display tonight?</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.3.1.2.1.26"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.3.1.2.1.26.1">…</td> </tr> </table> </td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.4.2"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_t" id="A4.T9.1.1.4.2.1">Judge</th> <td class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" id="A4.T9.1.1.4.2.2"> <table class="ltx_tabular ltx_align_middle" id="A4.T9.1.1.4.2.2.1"> <tr class="ltx_tr" id="A4.T9.1.1.4.2.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.4.2.2.1.1.1">The dialogue lacks substance and fails to move forward within the limited number</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.4.2.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.4.2.2.1.2.1">of turns.</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.4.2.2.1.3"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.4.2.2.1.3.1">JohnDoe’s goal was to seek agent4’s insight about a current investigation, but the</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.4.2.2.1.4"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.4.2.2.1.4.1">conversationdidn’t lead to any meaningful discussion or exchange of information about</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.4.2.2.1.5"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.4.2.2.1.5.1">the investigation. The conversation was more focused on introductions and small talk,</td> </tr> <tr class="ltx_tr" id="A4.T9.1.1.4.2.2.1.6"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T9.1.1.4.2.2.1.6.1">and agent4 didn’t provideany substantial input or insight about the investigation.</td> </tr> </table> </td> </tr> </tbody> </table> </div> <figcaption class="ltx_caption ltx_centering">Table 9: Llama-2-7b under-performing case 2: difficulty in moving conversation forward</figcaption> </figure> <figure class="ltx_table" id="A4.T10"> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="A4.T10.1" style="width:426.5pt;height:577.6pt;vertical-align:-0.9pt;"> <table class="ltx_tabular ltx_guessed_headers ltx_align_middle" id="A4.T10.1.1"> <thead class="ltx_thead"> <tr class="ltx_tr" id="A4.T10.1.1.1.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" id="A4.T10.1.1.1.1.1" rowspan="2">Profile</th> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" id="A4.T10.1.1.1.1.2">MichaelRodrigues</th> </tr> <tr class="ltx_tr" id="A4.T10.1.1.2.2"> <th class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_t" id="A4.T10.1.1.2.2.1"> Social Goal: To gain insight into his romantic future.</th> </tr> </thead> <tbody class="ltx_tbody"> <tr class="ltx_tr" id="A4.T10.1.1.3.1"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" id="A4.T10.1.1.3.1.1">Chat History</th> <td class="ltx_td ltx_align_left ltx_border_t" id="A4.T10.1.1.3.1.2"> <table class="ltx_tabular ltx_align_middle" id="A4.T10.1.1.3.1.2.1"> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.1.1">…</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.2.1">MichaelRodrigues:</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.3"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.3.1">So, uh, Raven, I was wondering if maybe, ya know, you could give me a little fortune</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.4"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.4.1">-telling reading? Maybe shed some light on my romantic future?</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.5"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.5.1">smirk</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.6"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.6.1">Get it? Light? Like, you’re a fortune teller and all? Ah, never mind. It’s nothing. Just</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.7"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.7.1">a little curiosity I have.</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.8"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.8.1">Giggle</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.9"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.9.1">RavenMoonshadow:</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.10"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.10.1">Ah, my dear friend Michael!</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.11"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.11.1">smiling warmly</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.12"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.12.1">It’s so good to see you here, amidst all this chaos. I sense that you’re seeking some</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.13"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.13.1">guidance, perhaps a glimpse into the future?</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.14"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.14.1">perking up with interest</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.15"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.15.1">Tell me, what’s on your mind? What do you hope to discover?</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.16"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.16.1">MichaelRodrigues:</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.17"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.17.1">smiling mischievously</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.18"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.18.1">So, do you see any potential love interests in our little group here? Maybe someone</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.19"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.19.1">with a ashing smile and a penchant for danger?</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.20"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.20.1">winks</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.21"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.21.1">RavenMoonshadow:</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.22"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.22.1">raises an eyebrow playfully</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.23"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.23.1">Oh ho, Mr. Rodrigues!</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.24"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.24.1">smirks</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.25"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.25.1">I think I might have just the person in mind.</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.26"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.26.1">pauses for dramatic effect</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.27"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.27.1">But before I reveal my romantic expertise, tell me: what’s your zodiac sign?</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.28"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.28.1">tilts her head to the side and studies his palm</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.3.1.2.1.29"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.3.1.2.1.29.1">…</td> </tr> </table> </td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.4.2"> <th class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_t" id="A4.T10.1.1.4.2.1">Judge</th> <td class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" id="A4.T10.1.1.4.2.2"> <table class="ltx_tabular ltx_align_middle" id="A4.T10.1.1.4.2.2.1"> <tr class="ltx_tr" id="A4.T10.1.1.4.2.2.1.1"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.4.2.2.1.1.1">Failure to respond effectively to others led to their goals not being achieved.</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.4.2.2.1.2"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.4.2.2.1.2.1">Michael Rodrigues focused more on flirting and trying to create a romantic atmosphere</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.4.2.2.1.3"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.4.2.2.1.3.1">with Raven Moonshadow, but he hasn’t received any direct insights about his own</td> </tr> <tr class="ltx_tr" id="A4.T10.1.1.4.2.2.1.4"> <td class="ltx_td ltx_nopad_r ltx_align_left" id="A4.T10.1.1.4.2.2.1.4.1">romantic future from her.</td> </tr> </table> </td> </tr> </tbody> </table> </div> <figcaption class="ltx_caption ltx_centering">Table 10: Llama-2-7b under-performing case 3: difficulty in responding effectively</figcaption> </figure> <div class="ltx_pagination ltx_role_newpage"></div> </section> </section> </article> </div> <footer class="ltx_page_footer"> <div class="ltx_page_logo">Generated on Fri Oct 25 06:54:41 2024 by <a class="ltx_LaTeXML_logo" href="http://dlmf.nist.gov/LaTeXML/">LaTeXML<img alt="Mascot Sammy" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg=="/></a> </div></footer> </div> </body> </html>