CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models</title>  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <meta content="Failure characterization, LLM, failure-recovery, reliability, OpenAI, Anthropic, Character.AI, operational data analytics " lang="en" name="keywords"/> <base href="/html/2501.12469v2/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S1" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">1 Introduction</a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">2 Anatomy of an LLM-service Incident: Model and Example</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.SS1" title="In 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">2.1 Model and Real-World Example</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.SS2" title="In 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">2.2 LLM-Specific Terms and Metrics</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S3" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">3 Dataset Collection and Preparation</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S3.SS1" title="In 3. Dataset Collection and Preparation ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">3.1 Selection and Introduction of LLM Services</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S3.SS2" title="In 3. Dataset Collection and Preparation ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">3.2 Data Collection and Dataset Preparation</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">4 Failure-Recovery Analysis</a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S5" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">5 Failure Patterns Over Time</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S5.SS1" title="In 5. Failure Patterns Over Time ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">5.1 Temporal Distributions</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S5.SS2" title="In 5. Failure Patterns Over Time ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">5.2 Auto-correlations</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S5.SS3" title="In 5. Failure Patterns Over Time ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">5.3 Service Availability Over Time</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S6" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">6 Co-occurrence of Failures</a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S6.SS1" title="In 6. Co-occurrence of Failures ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">6.1 Co-occurrence of Outages</a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S6.SS2" title="In 6. Co-occurrence of Failures ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">6.2 Impact Range of Incidents</a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S7" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">7 Limitations and Validity</a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S8" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">8 Related Work</a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S9" title="In An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">9 Conclusion</a></li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line ltx_leqno"> <div class="ltx_para" id="p1"> \useunder \setcctypeby-nc </div> <h1 class="ltx_title ltx_title_document">An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models</h1> <div class="ltx_authors"> Xiaoyu Chu <a href="mailto:x.chu@vu.nl">x.chu@vu.nl</a> Vrije Universiteit AmsterdamThe Netherlands , Sacheendra Talluri <a href="mailto:s.talluri@vu.nl">s.talluri@vu.nl</a> Vrije Universiteit AmsterdamThe Netherlands , Qingxian Lu <a href="mailto:q.lu@student.vu.nl">q.lu@student.vu.nl</a> Vrije Universiteit AmsterdamThe Netherlands and Alexandru Iosup <a href="mailto:a.iosup@vu.nl">a.iosup@vu.nl</a> Vrije Universiteit AmsterdamThe Netherlands </div> <div class="ltx_dates">(2025)</div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract.</h6> People and businesses increasingly rely on public LLM services, such as ChatGPT, DALL·E, and Claude. Understanding their outages, and particularly measuring their failure-recovery processes, is becoming a stringent problem. However, only limited studies exist in this emerging area. Addressing this problem, in this work we conduct an empirical characterization of outages and failure-recovery in public LLM services. We collect and prepare datasets for 8 commonly used LLM services across 3 major LLM providers, including market-leads OpenAI and Anthropic. We conduct a detailed analysis of failure recovery statistical properties, temporal patterns, co-occurrence, and the impact range of outage-causing incidents. We make over 10 observations, among which: (1) Failures in OpenAI’s ChatGPT take longer to resolve but occur less frequently than those in Anthropic’s Claude; (2) OpenAI and Anthropic service failures exhibit strong weekly and monthly periodicity; and (3) OpenAI services offer better failure-isolation than Anthropic services. Our research explains LLM failure characteristics and thus enables optimization in building and using LLM systems. FAIR data and code are publicly available on <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://zenodo.org/records/14018219" title="">https://zenodo.org/records/14018219</a> and <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://github.com/atlarge-research/llm-service-analysis" title="">https://github.com/atlarge-research/llm-service-analysis</a>. </div> <div class="ltx_keywords">Failure characterization, LLM, failure-recovery, reliability, OpenAI, Anthropic, Character.AI, operational data analytics </div> ††copyright: cc††journalyear: 2025††copyright: cc††conference: Proceedings of the 16th ACM/SPEC International Conference on Performance Engineering; May 5–9, 2025; Toronto, ON, Canada††booktitle: Proceedings of the 16th ACM/SPEC International Conference on Performance Engineering (ICPE ’25), May 5–9, 2025, Toronto, ON, Canada††doi: 10.1145/3676151.3719372††isbn: 979-8-4007-1073-5/2025/05††ccs: Computer systems organization Reliability <section class="ltx_section" id="S1"> <h2 class="ltx_title ltx_title_section"> 1. Introduction</h2> <figure class="ltx_figure" id="S1.F1"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="475" id="S1.F1.g1" src="x1.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 1. Monthly website visits, outages, and incidents for ChatGPT. Vertical axis: (left) number of website visits in billions; (right) monthly outage and incident counts. Data: website visits <cite class="ltx_cite ltx_citemacro_citep">(Pro, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib45" title="">2024</a>)</cite>, outages (<a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.T3" title="In 2.2. LLM-Specific Terms and Metrics ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 3</a>), and incidents (<a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.T4" title="In 2.2. LLM-Specific Terms and Metrics ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 4</a>).</figcaption> </figure> <div class="ltx_para" id="S1.p1"> In the past 5 years, increased availability of data and computation enabled Large Language Models (LLMs) to support scientists, businesses, and general users in a wide range of applications, such as coding <cite class="ltx_cite ltx_citemacro_citep">(Vaithilingam et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib60" title="">2022</a>; Liu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib38" title="">2023</a>)</cite>, image generation <cite class="ltx_cite ltx_citemacro_citep">(Koh et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib33" title="">2023</a>; Lu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib39" title="">2023</a>)</cite>, and general problem-solving <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib67" title="">2024</a>; Wang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib62" title="">2024b</a>)</cite>. Hundreds of millions of users rely increasingly on public LLM services such as ChatGPT <cite class="ltx_cite ltx_citemacro_citep">(Reuters, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib49" title="">2024b</a>)</cite>, DALL·E <cite class="ltx_cite ltx_citemacro_citep">(Hub, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib26" title="">2022</a>)</cite>, and Claude <cite class="ltx_cite ltx_citemacro_citep">(Reuters, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib48" title="">2024a</a>)</cite>. Understanding service outages, and how incidents leading to them are addressed, is essential to enhancing the fault tolerance and quality of service (QoS) of LLM systems. However, relatively little data and no peer-reviewed studies exist in this rapidly emerging area. Addressing this problem, and complementing studies that focus on LLM resource utilization <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib25" title="">2024</a>; Wang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib63" title="">2024a</a>)</cite> and user satisfaction <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib62" title="">2024b</a>)</cite>, in this work we conduct the first data-driven, empirical characterization of outages and incidents in public LLM services. We conduct three classes of analysis on long-term datasets we collect from 8 public LLM services from OpenAI, Anthropic, and Character.AI. </div> <div class="ltx_para" id="S1.p2"> The reliability of public LLM services is becoming increasingly important, as service failures can severely erode user experience and cause substantial financial losses under the competitive market. Driven by demand and market strategy, public LLM providers compete intensely, investing over $ 40 billion in 2024 <cite class="ltx_cite ltx_citemacro_citep">(IDC, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib27" title="">2024</a>)</cite>. Failures quickly affect many users and become highly visible, as the user cohort has already reached a global scale. For example, the launch of ChatGPT marked a major breakthrough in LLM applications <cite class="ltx_cite ltx_citemacro_citep">(Roumeliotis and Tselikas, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib51" title="">2023</a>)</cite> and set a user-adoption record, with 100 million monthly active users within 2 months after its 2022 launch <cite class="ltx_cite ltx_citemacro_citep">(Reuters, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib47" title="">2023</a>)</cite>. </div> <div class="ltx_para" id="S1.p3"> Although service reliability is important, users still frequently encounter issues with LLM services. For example, users report to DownDetector many login failures, request errors, and high response latency when using ChatGPT <cite class="ltx_cite ltx_citemacro_citep">(DownDetector, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib16" title="">2024</a>)</cite>. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S1.F1" title="In 1. Introduction ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 1</a> shows the monthly website visits <cite class="ltx_cite ltx_citemacro_citep">(Pro, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib45" title="">2024</a>)</cite>, and outages and incidents for ChatGPT as reported by OpenAI. As ChatGPT’s monthly web visits grow dramatically, the number of its outages and especially incidents also exhibit an upward trend. Thus, significant LLM failures continuously occur, decreasing user satisfaction and potentially causing financial loss, making reliable LLM services a challenge. </div> <div class="ltx_para" id="S1.p4"> Understanding dependability aspects can help improve systems especially when the workload characteristics are also understood. Previous work already provides system-level workload characteristics for the workloads of machine learning <cite class="ltx_cite ltx_citemacro_citep">(Chu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib15" title="">2023</a>, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib14" title="">2024</a>; Versluis et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib61" title="">2023</a>)</cite>, deep learning <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib35" title="">2022</a>)</cite>, big data <cite class="ltx_cite ltx_citemacro_citep">(Talluri et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib54" title="">2019</a>)</cite>, and more general clouds <cite class="ltx_cite ltx_citemacro_citep">(Reiss et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib46" title="">2012</a>)</cite>. Recently, LLM workloads have received attention as well <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib25" title="">2024</a>; Wang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib63" title="">2024a</a>)</cite>. What remains unaddressed in characterizing the failures of LLM. </div> <div class="ltx_para" id="S1.p5"> We identify and address in this work two main challenges in understanding how public LLM services currently fail. First, no longitudinal service failure data currently exists. Ideally, the community would have access to a large number of similarly curated datasets that capture LLM-service failures, under the same failure model, over long periods of time. There are some efforts to provide available LLM workloads, such as BurstGPT <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib63" title="">2024a</a>)</cite> and AcmeTrace <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib25" title="">2024</a>)</cite>, but they each focus on one LLM service, and none provides service failure data for it. Second, no comprehensive analysis of failures in public LLM services currently exists. At this stage in the scientific area, such an analysis would ideally be data-driven, and include for example general characteristics of failures, such as Mean Time Between Failures (MTBF) and To Recovery (MTTR) from classical dependability analysis <cite class="ltx_cite ltx_citemacro_citep">(Avizienis et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib10" title="">2004</a>)</cite>, and also of the time spent in various stages of the recovery-process specific to LLM operations; a temporal analysis of failures; and an analysis of failure cascades (co-occurrences). Such kinds of analysis would enable future research into models, and future theoretical and practical studies of LLM systems. </div> <div class="ltx_para" id="S1.p6"> Addressing both main challenges, this research aims to provide a thorough empirical characterization of LLM service failures, using data from official outages and incident reports, which are the two types of information self-disclosed by LLM service providers when significant failures occur. Our contribution is manyfold: </div> <div class="ltx_para" id="S1.p7"> <ol class="ltx_enumerate" id="S1.I1"> <li class="ltx_item" id="S1.I1.i1" style="list-style-type:none;"> (i) <div class="ltx_para" id="S1.I1.i1.p1"> We summarize the de facto industry standard for modeling LLM-service outages and exemplify the anatomy of an outage (<a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2" title="2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 2</a>); </div> </li> <li class="ltx_item" id="S1.I1.i2" style="list-style-type:none;"> (ii) <div class="ltx_para" id="S1.I1.i2.p1"> We collect outage and incident data for 8 LLM services, and prepare the corresponding LLM-failure datasets (<a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S3" title="3. Dataset Collection and Preparation ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 3</a>). This study covers representative, commonly used LLM services, across 3 LLM-service providers; </div> </li> <li class="ltx_item" id="S1.I1.i3" style="list-style-type:none;"> (iii) <div class="ltx_para" id="S1.I1.i3.p1"> We analyze the failure characteristics of 8 LLM services (<a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4" title="4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 4</a>). We analyze the MTTR and MTBF by provider and by service, the time spent in various stages of the recovery process, and quantify empirically the model parameters; </div> </li> <li class="ltx_item" id="S1.I1.i4" style="list-style-type:none;"> (iv) <div class="ltx_para" id="S1.I1.i4.p1"> We analyze LLM service failures over time (<a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S5" title="5. Failure Patterns Over Time ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 5</a>). We explore service availability over hourly and daily intervals, identify various diurnal and weekly patterns, and investigate auto-correlations; </div> </li> <li class="ltx_item" id="S1.I1.i5" style="list-style-type:none;"> (v) <div class="ltx_para" id="S1.I1.i5.p1"> We analyze the co-occurrence of failures (<a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S6" title="6. Co-occurrence of Failures ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 6</a>). Specifically, we analyze the co-occurrence of failures per provider, and of pairs of services across providers; </div> </li> <li class="ltx_item" id="S1.I1.i6" style="list-style-type:none;"> (vi) <div class="ltx_para" id="S1.I1.i6.p1"> We follow the principles of open science and release the datasets and software as open, FAIR artifacts to enable reproducibility and further research. </div> </li> </ol> </div> </section> <section class="ltx_section" id="S2"> <h2 class="ltx_title ltx_title_section"> 2. Anatomy of an LLM-service Incident: Model and Example</h2> <div class="ltx_para" id="S2.p1"> We present in this section a model, coupled with an example, of how an LLM-service incident occurs, affects actual users, and is managed by the LLM-service provider. </div> <figure class="ltx_figure" id="S2.F2"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="515" id="S2.F2.g1" src="x2.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 2. Visualization of the failure-recovery model with user reports of a selected ChatGPT incident, UDT time.</figcaption> </figure> <section class="ltx_subsection" id="S2.SS1"> <h3 class="ltx_title ltx_title_subsection"> 2.1. Model and Real-World Example</h3> <div class="ltx_para" id="S2.SS1.p1"> A failure-recovery process not only leads to addressing a system failure, but also shows a complete story of how the system experienced the (cascading) failure and provides insights to track and improve the system and services affected by it <cite class="ltx_cite ltx_citemacro_citep">(Gamell et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib17" title="">2014</a>)</cite>. </div> <div class="ltx_para" id="S2.SS1.p2"> Industry leads, such as OpenAI and Anthropic, report availability data built around a de facto standard model of their failure-recovery process. Acting as a tutorial, this section summarizes this industry model and exemplifies it through an actual incident. The example considers both data self-reported by the LLM service provider, OpenAI, and data reported by users experiencing the incident; analyzing user-reported data across all incidents studied here is useful but outside the scope of this article. </div> <div class="ltx_para" id="S2.SS1.p3"> Selection of the incident. We selected the real-world incident in which a major outage happened with the ChatGPT LLM service on April 10, 2024; this is the first major incident since, on April 1, OpenAI enabled free-to-use access to ChatGPT without signup, effectively opening up ChatGPT trials for everyone <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib42" title="">2024b</a>)</cite>. OpenAI reported the April 10 incident with complete details about all the stages of its failure-recovery process <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib41" title="">2024a</a>)</cite>, which is only done for significant incidents that require many local resources to address. In parallel with OpenAI team’s efforts to identify and resolve the incident, we recorded two other data sources. First, users reported problems using the ChatGPT service on DownDetector <cite class="ltx_cite ltx_citemacro_citep">(Archive, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib8" title="">2024</a>)</cite>; user-reported failures of public services are increasingly used to check the truthfulness and completeness of self-reported failure reports <cite class="ltx_cite ltx_citemacro_citep">(Talluri et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib55" title="">2021</a>)</cite>, but so far they have not been used in peer-reviewed studies of LLM services. We also recorded reports about this outage from news media across the technical and political spectrum, such as Fox <cite class="ltx_cite ltx_citemacro_citep">(York, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib68" title="">2024</a>)</cite>. </div> <figure class="ltx_table" id="S2.T1"> <figcaption class="ltx_caption ltx_centering">Table 1. Parameters and, below the double line, output metrics of the failure-recovery model proposed in this work.</figcaption> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S2.T1.28" style="width:433.6pt;height:167.4pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_align_middle" id="S2.T1.28.28"> <tr class="ltx_tr" id="S2.T1.28.28.29"> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T1.28.28.29.1">ID</td> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T1.28.28.29.2">Name</td> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T1.28.28.29.3">Definition</td> </tr> <tr class="ltx_tr" id="S2.T1.1.1.1"> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T1.1.1.1.1"><math alttext="S_{1}" class="ltx_Math" display="inline" id="S2.T1.1.1.1.1.m1.1"><semantics id="S2.T1.1.1.1.1.m1.1a"><msub id="S2.T1.1.1.1.1.m1.1.1" xref="S2.T1.1.1.1.1.m1.1.1.cmml"><mi id="S2.T1.1.1.1.1.m1.1.1.2" xref="S2.T1.1.1.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.1.1.1.1.m1.1.1.3" xref="S2.T1.1.1.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.1.1.1.1.m1.1b"><apply id="S2.T1.1.1.1.1.m1.1.1.cmml" xref="S2.T1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.1.1.1.1.m1.1.1.1.cmml" xref="S2.T1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.1.1.1.1.m1.1.1.2.cmml" xref="S2.T1.1.1.1.1.m1.1.1.2">𝑆</ci><cn id="S2.T1.1.1.1.1.m1.1.1.3.cmml" type="integer" xref="S2.T1.1.1.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.1.1.1.1.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.1.1.1.1.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T1.1.1.1.2">Investigating Status</td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T1.1.1.1.3">The operational team has started investigating an incident.</td> </tr> <tr class="ltx_tr" id="S2.T1.2.2.2"> <td class="ltx_td ltx_align_left" id="S2.T1.2.2.2.1"><math alttext="S_{2}" class="ltx_Math" display="inline" id="S2.T1.2.2.2.1.m1.1"><semantics id="S2.T1.2.2.2.1.m1.1a"><msub id="S2.T1.2.2.2.1.m1.1.1" xref="S2.T1.2.2.2.1.m1.1.1.cmml"><mi id="S2.T1.2.2.2.1.m1.1.1.2" xref="S2.T1.2.2.2.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.2.2.2.1.m1.1.1.3" xref="S2.T1.2.2.2.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.2.2.2.1.m1.1b"><apply id="S2.T1.2.2.2.1.m1.1.1.cmml" xref="S2.T1.2.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.2.2.2.1.m1.1.1.1.cmml" xref="S2.T1.2.2.2.1.m1.1.1">subscript</csymbol><ci id="S2.T1.2.2.2.1.m1.1.1.2.cmml" xref="S2.T1.2.2.2.1.m1.1.1.2">𝑆</ci><cn id="S2.T1.2.2.2.1.m1.1.1.3.cmml" type="integer" xref="S2.T1.2.2.2.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.2.2.2.1.m1.1c">S_{2}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.2.2.2.1.m1.1d">italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T1.2.2.2.2">Identified Status</td> <td class="ltx_td ltx_align_left" id="S2.T1.2.2.2.3">The issues have been identified.</td> </tr> <tr class="ltx_tr" id="S2.T1.3.3.3"> <td class="ltx_td ltx_align_left" id="S2.T1.3.3.3.1"><math alttext="S_{3}" class="ltx_Math" display="inline" id="S2.T1.3.3.3.1.m1.1"><semantics id="S2.T1.3.3.3.1.m1.1a"><msub id="S2.T1.3.3.3.1.m1.1.1" xref="S2.T1.3.3.3.1.m1.1.1.cmml"><mi id="S2.T1.3.3.3.1.m1.1.1.2" xref="S2.T1.3.3.3.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.3.3.3.1.m1.1.1.3" xref="S2.T1.3.3.3.1.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.3.3.3.1.m1.1b"><apply id="S2.T1.3.3.3.1.m1.1.1.cmml" xref="S2.T1.3.3.3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.3.3.3.1.m1.1.1.1.cmml" xref="S2.T1.3.3.3.1.m1.1.1">subscript</csymbol><ci id="S2.T1.3.3.3.1.m1.1.1.2.cmml" xref="S2.T1.3.3.3.1.m1.1.1.2">𝑆</ci><cn id="S2.T1.3.3.3.1.m1.1.1.3.cmml" type="integer" xref="S2.T1.3.3.3.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.3.3.3.1.m1.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.3.3.3.1.m1.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T1.3.3.3.2">Monitoring Status</td> <td class="ltx_td ltx_align_left" id="S2.T1.3.3.3.3">A fix has been implemented and the operational team started monitoring the results.</td> </tr> <tr class="ltx_tr" id="S2.T1.4.4.4"> <td class="ltx_td ltx_align_left" id="S2.T1.4.4.4.1"><math alttext="S_{4}" class="ltx_Math" display="inline" id="S2.T1.4.4.4.1.m1.1"><semantics id="S2.T1.4.4.4.1.m1.1a"><msub id="S2.T1.4.4.4.1.m1.1.1" xref="S2.T1.4.4.4.1.m1.1.1.cmml"><mi id="S2.T1.4.4.4.1.m1.1.1.2" xref="S2.T1.4.4.4.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.4.4.4.1.m1.1.1.3" xref="S2.T1.4.4.4.1.m1.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.4.4.4.1.m1.1b"><apply id="S2.T1.4.4.4.1.m1.1.1.cmml" xref="S2.T1.4.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.4.4.4.1.m1.1.1.1.cmml" xref="S2.T1.4.4.4.1.m1.1.1">subscript</csymbol><ci id="S2.T1.4.4.4.1.m1.1.1.2.cmml" xref="S2.T1.4.4.4.1.m1.1.1.2">𝑆</ci><cn id="S2.T1.4.4.4.1.m1.1.1.3.cmml" type="integer" xref="S2.T1.4.4.4.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.4.4.4.1.m1.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.4.4.4.1.m1.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T1.4.4.4.2">Resolved Status</td> <td class="ltx_td ltx_align_left" id="S2.T1.4.4.4.3">The incident has been resolved.</td> </tr> <tr class="ltx_tr" id="S2.T1.5.5.5"> <td class="ltx_td ltx_align_left" id="S2.T1.5.5.5.1"><math alttext="S_{5}" class="ltx_Math" display="inline" id="S2.T1.5.5.5.1.m1.1"><semantics id="S2.T1.5.5.5.1.m1.1a"><msub id="S2.T1.5.5.5.1.m1.1.1" xref="S2.T1.5.5.5.1.m1.1.1.cmml"><mi id="S2.T1.5.5.5.1.m1.1.1.2" xref="S2.T1.5.5.5.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.5.5.5.1.m1.1.1.3" xref="S2.T1.5.5.5.1.m1.1.1.3.cmml">5</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.5.5.5.1.m1.1b"><apply id="S2.T1.5.5.5.1.m1.1.1.cmml" xref="S2.T1.5.5.5.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.5.5.5.1.m1.1.1.1.cmml" xref="S2.T1.5.5.5.1.m1.1.1">subscript</csymbol><ci id="S2.T1.5.5.5.1.m1.1.1.2.cmml" xref="S2.T1.5.5.5.1.m1.1.1.2">𝑆</ci><cn id="S2.T1.5.5.5.1.m1.1.1.3.cmml" type="integer" xref="S2.T1.5.5.5.1.m1.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.5.5.5.1.m1.1c">S_{5}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.5.5.5.1.m1.1d">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T1.5.5.5.2">Postmortem Status</td> <td class="ltx_td ltx_align_left" id="S2.T1.5.5.5.3">A summary of the incident after it has been resolved.</td> </tr> <tr class="ltx_tr" id="S2.T1.8.8.8"> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T1.6.6.6.1"><math alttext="P_{I}" class="ltx_Math" display="inline" id="S2.T1.6.6.6.1.m1.1"><semantics id="S2.T1.6.6.6.1.m1.1a"><msub id="S2.T1.6.6.6.1.m1.1.1" xref="S2.T1.6.6.6.1.m1.1.1.cmml"><mi id="S2.T1.6.6.6.1.m1.1.1.2" xref="S2.T1.6.6.6.1.m1.1.1.2.cmml">P</mi><mi id="S2.T1.6.6.6.1.m1.1.1.3" xref="S2.T1.6.6.6.1.m1.1.1.3.cmml">I</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.6.6.6.1.m1.1b"><apply id="S2.T1.6.6.6.1.m1.1.1.cmml" xref="S2.T1.6.6.6.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.6.6.6.1.m1.1.1.1.cmml" xref="S2.T1.6.6.6.1.m1.1.1">subscript</csymbol><ci id="S2.T1.6.6.6.1.m1.1.1.2.cmml" xref="S2.T1.6.6.6.1.m1.1.1.2">𝑃</ci><ci id="S2.T1.6.6.6.1.m1.1.1.3.cmml" xref="S2.T1.6.6.6.1.m1.1.1.3">𝐼</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.6.6.6.1.m1.1c">P_{I}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.6.6.6.1.m1.1d">italic_P start_POSTSUBSCRIPT italic_I end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T1.8.8.8.4">Investigating Period</td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T1.8.8.8.3">From <math alttext="S_{1}" class="ltx_Math" display="inline" id="S2.T1.7.7.7.2.m1.1"><semantics id="S2.T1.7.7.7.2.m1.1a"><msub id="S2.T1.7.7.7.2.m1.1.1" xref="S2.T1.7.7.7.2.m1.1.1.cmml"><mi id="S2.T1.7.7.7.2.m1.1.1.2" xref="S2.T1.7.7.7.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.7.7.7.2.m1.1.1.3" xref="S2.T1.7.7.7.2.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.7.7.7.2.m1.1b"><apply id="S2.T1.7.7.7.2.m1.1.1.cmml" xref="S2.T1.7.7.7.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.7.7.7.2.m1.1.1.1.cmml" xref="S2.T1.7.7.7.2.m1.1.1">subscript</csymbol><ci id="S2.T1.7.7.7.2.m1.1.1.2.cmml" xref="S2.T1.7.7.7.2.m1.1.1.2">𝑆</ci><cn id="S2.T1.7.7.7.2.m1.1.1.3.cmml" type="integer" xref="S2.T1.7.7.7.2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.7.7.7.2.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.7.7.7.2.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> to <math alttext="S_{2}" class="ltx_Math" display="inline" id="S2.T1.8.8.8.3.m2.1"><semantics id="S2.T1.8.8.8.3.m2.1a"><msub id="S2.T1.8.8.8.3.m2.1.1" xref="S2.T1.8.8.8.3.m2.1.1.cmml"><mi id="S2.T1.8.8.8.3.m2.1.1.2" xref="S2.T1.8.8.8.3.m2.1.1.2.cmml">S</mi><mn id="S2.T1.8.8.8.3.m2.1.1.3" xref="S2.T1.8.8.8.3.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.8.8.8.3.m2.1b"><apply id="S2.T1.8.8.8.3.m2.1.1.cmml" xref="S2.T1.8.8.8.3.m2.1.1"><csymbol cd="ambiguous" id="S2.T1.8.8.8.3.m2.1.1.1.cmml" xref="S2.T1.8.8.8.3.m2.1.1">subscript</csymbol><ci id="S2.T1.8.8.8.3.m2.1.1.2.cmml" xref="S2.T1.8.8.8.3.m2.1.1.2">𝑆</ci><cn id="S2.T1.8.8.8.3.m2.1.1.3.cmml" type="integer" xref="S2.T1.8.8.8.3.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.8.8.8.3.m2.1c">S_{2}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.8.8.8.3.m2.1d">italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math>, showing the time from observing to identifying the issues.</td> </tr> <tr class="ltx_tr" id="S2.T1.11.11.11"> <td class="ltx_td ltx_align_left" id="S2.T1.9.9.9.1"><math alttext="P_{R}" class="ltx_Math" display="inline" id="S2.T1.9.9.9.1.m1.1"><semantics id="S2.T1.9.9.9.1.m1.1a"><msub id="S2.T1.9.9.9.1.m1.1.1" xref="S2.T1.9.9.9.1.m1.1.1.cmml"><mi id="S2.T1.9.9.9.1.m1.1.1.2" xref="S2.T1.9.9.9.1.m1.1.1.2.cmml">P</mi><mi id="S2.T1.9.9.9.1.m1.1.1.3" xref="S2.T1.9.9.9.1.m1.1.1.3.cmml">R</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.9.9.9.1.m1.1b"><apply id="S2.T1.9.9.9.1.m1.1.1.cmml" xref="S2.T1.9.9.9.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.9.9.9.1.m1.1.1.1.cmml" xref="S2.T1.9.9.9.1.m1.1.1">subscript</csymbol><ci id="S2.T1.9.9.9.1.m1.1.1.2.cmml" xref="S2.T1.9.9.9.1.m1.1.1.2">𝑃</ci><ci id="S2.T1.9.9.9.1.m1.1.1.3.cmml" xref="S2.T1.9.9.9.1.m1.1.1.3">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.9.9.9.1.m1.1c">P_{R}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.9.9.9.1.m1.1d">italic_P start_POSTSUBSCRIPT italic_R end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T1.11.11.11.4">Repairing Period</td> <td class="ltx_td ltx_align_left" id="S2.T1.11.11.11.3">From <math alttext="S_{2}" class="ltx_Math" display="inline" id="S2.T1.10.10.10.2.m1.1"><semantics id="S2.T1.10.10.10.2.m1.1a"><msub id="S2.T1.10.10.10.2.m1.1.1" xref="S2.T1.10.10.10.2.m1.1.1.cmml"><mi id="S2.T1.10.10.10.2.m1.1.1.2" xref="S2.T1.10.10.10.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.10.10.10.2.m1.1.1.3" xref="S2.T1.10.10.10.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.10.10.10.2.m1.1b"><apply id="S2.T1.10.10.10.2.m1.1.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.10.10.10.2.m1.1.1.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1">subscript</csymbol><ci id="S2.T1.10.10.10.2.m1.1.1.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2">𝑆</ci><cn id="S2.T1.10.10.10.2.m1.1.1.3.cmml" type="integer" xref="S2.T1.10.10.10.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.10.10.10.2.m1.1c">S_{2}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.10.10.10.2.m1.1d">italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math> to <math alttext="S_{3}" class="ltx_Math" display="inline" id="S2.T1.11.11.11.3.m2.1"><semantics id="S2.T1.11.11.11.3.m2.1a"><msub id="S2.T1.11.11.11.3.m2.1.1" xref="S2.T1.11.11.11.3.m2.1.1.cmml"><mi id="S2.T1.11.11.11.3.m2.1.1.2" xref="S2.T1.11.11.11.3.m2.1.1.2.cmml">S</mi><mn id="S2.T1.11.11.11.3.m2.1.1.3" xref="S2.T1.11.11.11.3.m2.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.11.11.11.3.m2.1b"><apply id="S2.T1.11.11.11.3.m2.1.1.cmml" xref="S2.T1.11.11.11.3.m2.1.1"><csymbol cd="ambiguous" id="S2.T1.11.11.11.3.m2.1.1.1.cmml" xref="S2.T1.11.11.11.3.m2.1.1">subscript</csymbol><ci id="S2.T1.11.11.11.3.m2.1.1.2.cmml" xref="S2.T1.11.11.11.3.m2.1.1.2">𝑆</ci><cn id="S2.T1.11.11.11.3.m2.1.1.3.cmml" type="integer" xref="S2.T1.11.11.11.3.m2.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.11.11.11.3.m2.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.11.11.11.3.m2.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math>, showing the time to repair the issues.</td> </tr> <tr class="ltx_tr" id="S2.T1.14.14.14"> <td class="ltx_td ltx_align_left" id="S2.T1.12.12.12.1"><math alttext="P_{C}" class="ltx_Math" display="inline" id="S2.T1.12.12.12.1.m1.1"><semantics id="S2.T1.12.12.12.1.m1.1a"><msub id="S2.T1.12.12.12.1.m1.1.1" xref="S2.T1.12.12.12.1.m1.1.1.cmml"><mi id="S2.T1.12.12.12.1.m1.1.1.2" xref="S2.T1.12.12.12.1.m1.1.1.2.cmml">P</mi><mi id="S2.T1.12.12.12.1.m1.1.1.3" xref="S2.T1.12.12.12.1.m1.1.1.3.cmml">C</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.12.12.12.1.m1.1b"><apply id="S2.T1.12.12.12.1.m1.1.1.cmml" xref="S2.T1.12.12.12.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.12.12.12.1.m1.1.1.1.cmml" xref="S2.T1.12.12.12.1.m1.1.1">subscript</csymbol><ci id="S2.T1.12.12.12.1.m1.1.1.2.cmml" xref="S2.T1.12.12.12.1.m1.1.1.2">𝑃</ci><ci id="S2.T1.12.12.12.1.m1.1.1.3.cmml" xref="S2.T1.12.12.12.1.m1.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.12.12.12.1.m1.1c">P_{C}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.12.12.12.1.m1.1d">italic_P start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T1.14.14.14.4">Checking Period</td> <td class="ltx_td ltx_align_left" id="S2.T1.14.14.14.3">From <math alttext="S_{3}" class="ltx_Math" display="inline" id="S2.T1.13.13.13.2.m1.1"><semantics id="S2.T1.13.13.13.2.m1.1a"><msub id="S2.T1.13.13.13.2.m1.1.1" xref="S2.T1.13.13.13.2.m1.1.1.cmml"><mi id="S2.T1.13.13.13.2.m1.1.1.2" xref="S2.T1.13.13.13.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.13.13.13.2.m1.1.1.3" xref="S2.T1.13.13.13.2.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.13.13.13.2.m1.1b"><apply id="S2.T1.13.13.13.2.m1.1.1.cmml" xref="S2.T1.13.13.13.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.13.13.13.2.m1.1.1.1.cmml" xref="S2.T1.13.13.13.2.m1.1.1">subscript</csymbol><ci id="S2.T1.13.13.13.2.m1.1.1.2.cmml" xref="S2.T1.13.13.13.2.m1.1.1.2">𝑆</ci><cn id="S2.T1.13.13.13.2.m1.1.1.3.cmml" type="integer" xref="S2.T1.13.13.13.2.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.13.13.13.2.m1.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.13.13.13.2.m1.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math> to <math alttext="S_{4}" class="ltx_Math" display="inline" id="S2.T1.14.14.14.3.m2.1"><semantics id="S2.T1.14.14.14.3.m2.1a"><msub id="S2.T1.14.14.14.3.m2.1.1" xref="S2.T1.14.14.14.3.m2.1.1.cmml"><mi id="S2.T1.14.14.14.3.m2.1.1.2" xref="S2.T1.14.14.14.3.m2.1.1.2.cmml">S</mi><mn id="S2.T1.14.14.14.3.m2.1.1.3" xref="S2.T1.14.14.14.3.m2.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.14.14.14.3.m2.1b"><apply id="S2.T1.14.14.14.3.m2.1.1.cmml" xref="S2.T1.14.14.14.3.m2.1.1"><csymbol cd="ambiguous" id="S2.T1.14.14.14.3.m2.1.1.1.cmml" xref="S2.T1.14.14.14.3.m2.1.1">subscript</csymbol><ci id="S2.T1.14.14.14.3.m2.1.1.2.cmml" xref="S2.T1.14.14.14.3.m2.1.1.2">𝑆</ci><cn id="S2.T1.14.14.14.3.m2.1.1.3.cmml" type="integer" xref="S2.T1.14.14.14.3.m2.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.14.14.14.3.m2.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.14.14.14.3.m2.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math>, showing the time to confirm the fix is stable and effective.</td> </tr> <tr class="ltx_tr" id="S2.T1.17.17.17"> <td class="ltx_td ltx_align_left" id="S2.T1.15.15.15.1"><math alttext="P_{L}" class="ltx_Math" display="inline" id="S2.T1.15.15.15.1.m1.1"><semantics id="S2.T1.15.15.15.1.m1.1a"><msub id="S2.T1.15.15.15.1.m1.1.1" xref="S2.T1.15.15.15.1.m1.1.1.cmml"><mi id="S2.T1.15.15.15.1.m1.1.1.2" xref="S2.T1.15.15.15.1.m1.1.1.2.cmml">P</mi><mi id="S2.T1.15.15.15.1.m1.1.1.3" xref="S2.T1.15.15.15.1.m1.1.1.3.cmml">L</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.15.15.15.1.m1.1b"><apply id="S2.T1.15.15.15.1.m1.1.1.cmml" xref="S2.T1.15.15.15.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.15.15.15.1.m1.1.1.1.cmml" xref="S2.T1.15.15.15.1.m1.1.1">subscript</csymbol><ci id="S2.T1.15.15.15.1.m1.1.1.2.cmml" xref="S2.T1.15.15.15.1.m1.1.1.2">𝑃</ci><ci id="S2.T1.15.15.15.1.m1.1.1.3.cmml" xref="S2.T1.15.15.15.1.m1.1.1.3">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.15.15.15.1.m1.1c">P_{L}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.15.15.15.1.m1.1d">italic_P start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T1.17.17.17.4">Learning Period</td> <td class="ltx_td ltx_align_left" id="S2.T1.17.17.17.3">From <math alttext="S_{4}" class="ltx_Math" display="inline" id="S2.T1.16.16.16.2.m1.1"><semantics id="S2.T1.16.16.16.2.m1.1a"><msub id="S2.T1.16.16.16.2.m1.1.1" xref="S2.T1.16.16.16.2.m1.1.1.cmml"><mi id="S2.T1.16.16.16.2.m1.1.1.2" xref="S2.T1.16.16.16.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.16.16.16.2.m1.1.1.3" xref="S2.T1.16.16.16.2.m1.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.16.16.16.2.m1.1b"><apply id="S2.T1.16.16.16.2.m1.1.1.cmml" xref="S2.T1.16.16.16.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.16.16.16.2.m1.1.1.1.cmml" xref="S2.T1.16.16.16.2.m1.1.1">subscript</csymbol><ci id="S2.T1.16.16.16.2.m1.1.1.2.cmml" xref="S2.T1.16.16.16.2.m1.1.1.2">𝑆</ci><cn id="S2.T1.16.16.16.2.m1.1.1.3.cmml" type="integer" xref="S2.T1.16.16.16.2.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.16.16.16.2.m1.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.16.16.16.2.m1.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math> to <math alttext="S_{5}" class="ltx_Math" display="inline" id="S2.T1.17.17.17.3.m2.1"><semantics id="S2.T1.17.17.17.3.m2.1a"><msub id="S2.T1.17.17.17.3.m2.1.1" xref="S2.T1.17.17.17.3.m2.1.1.cmml"><mi id="S2.T1.17.17.17.3.m2.1.1.2" xref="S2.T1.17.17.17.3.m2.1.1.2.cmml">S</mi><mn id="S2.T1.17.17.17.3.m2.1.1.3" xref="S2.T1.17.17.17.3.m2.1.1.3.cmml">5</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.17.17.17.3.m2.1b"><apply id="S2.T1.17.17.17.3.m2.1.1.cmml" xref="S2.T1.17.17.17.3.m2.1.1"><csymbol cd="ambiguous" id="S2.T1.17.17.17.3.m2.1.1.1.cmml" xref="S2.T1.17.17.17.3.m2.1.1">subscript</csymbol><ci id="S2.T1.17.17.17.3.m2.1.1.2.cmml" xref="S2.T1.17.17.17.3.m2.1.1.2">𝑆</ci><cn id="S2.T1.17.17.17.3.m2.1.1.3.cmml" type="integer" xref="S2.T1.17.17.17.3.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.17.17.17.3.m2.1c">S_{5}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.17.17.17.3.m2.1d">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT</annotation></semantics></math>, showing the time to provide the incident’s root cause.</td> </tr> <tr class="ltx_tr" id="S2.T1.23.23.23"> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T1.18.18.18.1"><math alttext="MTTR" class="ltx_Math" display="inline" id="S2.T1.18.18.18.1.m1.1"><semantics id="S2.T1.18.18.18.1.m1.1a"><mrow id="S2.T1.18.18.18.1.m1.1.1" xref="S2.T1.18.18.18.1.m1.1.1.cmml"><mi id="S2.T1.18.18.18.1.m1.1.1.2" xref="S2.T1.18.18.18.1.m1.1.1.2.cmml">M</mi><mo id="S2.T1.18.18.18.1.m1.1.1.1" xref="S2.T1.18.18.18.1.m1.1.1.1.cmml">⁢</mo><mi id="S2.T1.18.18.18.1.m1.1.1.3" xref="S2.T1.18.18.18.1.m1.1.1.3.cmml">T</mi><mo id="S2.T1.18.18.18.1.m1.1.1.1a" xref="S2.T1.18.18.18.1.m1.1.1.1.cmml">⁢</mo><mi id="S2.T1.18.18.18.1.m1.1.1.4" xref="S2.T1.18.18.18.1.m1.1.1.4.cmml">T</mi><mo id="S2.T1.18.18.18.1.m1.1.1.1b" xref="S2.T1.18.18.18.1.m1.1.1.1.cmml">⁢</mo><mi id="S2.T1.18.18.18.1.m1.1.1.5" xref="S2.T1.18.18.18.1.m1.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.18.18.18.1.m1.1b"><apply id="S2.T1.18.18.18.1.m1.1.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1"><times id="S2.T1.18.18.18.1.m1.1.1.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1"></times><ci id="S2.T1.18.18.18.1.m1.1.1.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1.2">𝑀</ci><ci id="S2.T1.18.18.18.1.m1.1.1.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3">𝑇</ci><ci id="S2.T1.18.18.18.1.m1.1.1.4.cmml" xref="S2.T1.18.18.18.1.m1.1.1.4">𝑇</ci><ci id="S2.T1.18.18.18.1.m1.1.1.5.cmml" xref="S2.T1.18.18.18.1.m1.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.18.18.18.1.m1.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S2.T1.18.18.18.1.m1.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T1.23.23.23.7">Mean Time To Resolve</td> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T1.23.23.23.6">From <math alttext="S_{1}" class="ltx_Math" display="inline" id="S2.T1.19.19.19.2.m1.1"><semantics id="S2.T1.19.19.19.2.m1.1a"><msub id="S2.T1.19.19.19.2.m1.1.1" xref="S2.T1.19.19.19.2.m1.1.1.cmml"><mi id="S2.T1.19.19.19.2.m1.1.1.2" xref="S2.T1.19.19.19.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.19.19.19.2.m1.1.1.3" xref="S2.T1.19.19.19.2.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.19.19.19.2.m1.1b"><apply id="S2.T1.19.19.19.2.m1.1.1.cmml" xref="S2.T1.19.19.19.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.19.19.19.2.m1.1.1.1.cmml" xref="S2.T1.19.19.19.2.m1.1.1">subscript</csymbol><ci id="S2.T1.19.19.19.2.m1.1.1.2.cmml" xref="S2.T1.19.19.19.2.m1.1.1.2">𝑆</ci><cn id="S2.T1.19.19.19.2.m1.1.1.3.cmml" type="integer" xref="S2.T1.19.19.19.2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.19.19.19.2.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.19.19.19.2.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> to <math alttext="S_{4}" class="ltx_Math" display="inline" id="S2.T1.20.20.20.3.m2.1"><semantics id="S2.T1.20.20.20.3.m2.1a"><msub id="S2.T1.20.20.20.3.m2.1.1" xref="S2.T1.20.20.20.3.m2.1.1.cmml"><mi id="S2.T1.20.20.20.3.m2.1.1.2" xref="S2.T1.20.20.20.3.m2.1.1.2.cmml">S</mi><mn id="S2.T1.20.20.20.3.m2.1.1.3" xref="S2.T1.20.20.20.3.m2.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.20.20.20.3.m2.1b"><apply id="S2.T1.20.20.20.3.m2.1.1.cmml" xref="S2.T1.20.20.20.3.m2.1.1"><csymbol cd="ambiguous" id="S2.T1.20.20.20.3.m2.1.1.1.cmml" xref="S2.T1.20.20.20.3.m2.1.1">subscript</csymbol><ci id="S2.T1.20.20.20.3.m2.1.1.2.cmml" xref="S2.T1.20.20.20.3.m2.1.1.2">𝑆</ci><cn id="S2.T1.20.20.20.3.m2.1.1.3.cmml" type="integer" xref="S2.T1.20.20.20.3.m2.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.20.20.20.3.m2.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.20.20.20.3.m2.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math>, covering <math alttext="P_{I}" class="ltx_Math" display="inline" id="S2.T1.21.21.21.4.m3.1"><semantics id="S2.T1.21.21.21.4.m3.1a"><msub id="S2.T1.21.21.21.4.m3.1.1" xref="S2.T1.21.21.21.4.m3.1.1.cmml"><mi id="S2.T1.21.21.21.4.m3.1.1.2" xref="S2.T1.21.21.21.4.m3.1.1.2.cmml">P</mi><mi id="S2.T1.21.21.21.4.m3.1.1.3" xref="S2.T1.21.21.21.4.m3.1.1.3.cmml">I</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.21.21.21.4.m3.1b"><apply id="S2.T1.21.21.21.4.m3.1.1.cmml" xref="S2.T1.21.21.21.4.m3.1.1"><csymbol cd="ambiguous" id="S2.T1.21.21.21.4.m3.1.1.1.cmml" xref="S2.T1.21.21.21.4.m3.1.1">subscript</csymbol><ci id="S2.T1.21.21.21.4.m3.1.1.2.cmml" xref="S2.T1.21.21.21.4.m3.1.1.2">𝑃</ci><ci id="S2.T1.21.21.21.4.m3.1.1.3.cmml" xref="S2.T1.21.21.21.4.m3.1.1.3">𝐼</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.21.21.21.4.m3.1c">P_{I}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.21.21.21.4.m3.1d">italic_P start_POSTSUBSCRIPT italic_I end_POSTSUBSCRIPT</annotation></semantics></math>, <math alttext="P_{R}" class="ltx_Math" display="inline" id="S2.T1.22.22.22.5.m4.1"><semantics id="S2.T1.22.22.22.5.m4.1a"><msub id="S2.T1.22.22.22.5.m4.1.1" xref="S2.T1.22.22.22.5.m4.1.1.cmml"><mi id="S2.T1.22.22.22.5.m4.1.1.2" xref="S2.T1.22.22.22.5.m4.1.1.2.cmml">P</mi><mi id="S2.T1.22.22.22.5.m4.1.1.3" xref="S2.T1.22.22.22.5.m4.1.1.3.cmml">R</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.22.22.22.5.m4.1b"><apply id="S2.T1.22.22.22.5.m4.1.1.cmml" xref="S2.T1.22.22.22.5.m4.1.1"><csymbol cd="ambiguous" id="S2.T1.22.22.22.5.m4.1.1.1.cmml" xref="S2.T1.22.22.22.5.m4.1.1">subscript</csymbol><ci id="S2.T1.22.22.22.5.m4.1.1.2.cmml" xref="S2.T1.22.22.22.5.m4.1.1.2">𝑃</ci><ci id="S2.T1.22.22.22.5.m4.1.1.3.cmml" xref="S2.T1.22.22.22.5.m4.1.1.3">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.22.22.22.5.m4.1c">P_{R}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.22.22.22.5.m4.1d">italic_P start_POSTSUBSCRIPT italic_R end_POSTSUBSCRIPT</annotation></semantics></math>, <math alttext="P_{C}" class="ltx_Math" display="inline" id="S2.T1.23.23.23.6.m5.1"><semantics id="S2.T1.23.23.23.6.m5.1a"><msub id="S2.T1.23.23.23.6.m5.1.1" xref="S2.T1.23.23.23.6.m5.1.1.cmml"><mi id="S2.T1.23.23.23.6.m5.1.1.2" xref="S2.T1.23.23.23.6.m5.1.1.2.cmml">P</mi><mi id="S2.T1.23.23.23.6.m5.1.1.3" xref="S2.T1.23.23.23.6.m5.1.1.3.cmml">C</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.23.23.23.6.m5.1b"><apply id="S2.T1.23.23.23.6.m5.1.1.cmml" xref="S2.T1.23.23.23.6.m5.1.1"><csymbol cd="ambiguous" id="S2.T1.23.23.23.6.m5.1.1.1.cmml" xref="S2.T1.23.23.23.6.m5.1.1">subscript</csymbol><ci id="S2.T1.23.23.23.6.m5.1.1.2.cmml" xref="S2.T1.23.23.23.6.m5.1.1.2">𝑃</ci><ci id="S2.T1.23.23.23.6.m5.1.1.3.cmml" xref="S2.T1.23.23.23.6.m5.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.23.23.23.6.m5.1c">P_{C}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.23.23.23.6.m5.1d">italic_P start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT</annotation></semantics></math>, showing the full time of resolving issues.</td> </tr> <tr class="ltx_tr" id="S2.T1.25.25.25"> <td class="ltx_td ltx_align_left" id="S2.T1.24.24.24.1"><math alttext="MTBF" class="ltx_Math" display="inline" id="S2.T1.24.24.24.1.m1.1"><semantics id="S2.T1.24.24.24.1.m1.1a"><mrow id="S2.T1.24.24.24.1.m1.1.1" xref="S2.T1.24.24.24.1.m1.1.1.cmml"><mi id="S2.T1.24.24.24.1.m1.1.1.2" xref="S2.T1.24.24.24.1.m1.1.1.2.cmml">M</mi><mo id="S2.T1.24.24.24.1.m1.1.1.1" xref="S2.T1.24.24.24.1.m1.1.1.1.cmml">⁢</mo><mi id="S2.T1.24.24.24.1.m1.1.1.3" xref="S2.T1.24.24.24.1.m1.1.1.3.cmml">T</mi><mo id="S2.T1.24.24.24.1.m1.1.1.1a" xref="S2.T1.24.24.24.1.m1.1.1.1.cmml">⁢</mo><mi id="S2.T1.24.24.24.1.m1.1.1.4" xref="S2.T1.24.24.24.1.m1.1.1.4.cmml">B</mi><mo id="S2.T1.24.24.24.1.m1.1.1.1b" xref="S2.T1.24.24.24.1.m1.1.1.1.cmml">⁢</mo><mi id="S2.T1.24.24.24.1.m1.1.1.5" xref="S2.T1.24.24.24.1.m1.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.24.24.24.1.m1.1b"><apply id="S2.T1.24.24.24.1.m1.1.1.cmml" xref="S2.T1.24.24.24.1.m1.1.1"><times id="S2.T1.24.24.24.1.m1.1.1.1.cmml" xref="S2.T1.24.24.24.1.m1.1.1.1"></times><ci id="S2.T1.24.24.24.1.m1.1.1.2.cmml" xref="S2.T1.24.24.24.1.m1.1.1.2">𝑀</ci><ci id="S2.T1.24.24.24.1.m1.1.1.3.cmml" xref="S2.T1.24.24.24.1.m1.1.1.3">𝑇</ci><ci id="S2.T1.24.24.24.1.m1.1.1.4.cmml" xref="S2.T1.24.24.24.1.m1.1.1.4">𝐵</ci><ci id="S2.T1.24.24.24.1.m1.1.1.5.cmml" xref="S2.T1.24.24.24.1.m1.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.24.24.24.1.m1.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S2.T1.24.24.24.1.m1.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T1.25.25.25.3">Mean Time Between Failures</td> <td class="ltx_td ltx_align_left" id="S2.T1.25.25.25.2">From the <math alttext="S_{1}" class="ltx_Math" display="inline" id="S2.T1.25.25.25.2.m1.1"><semantics id="S2.T1.25.25.25.2.m1.1a"><msub id="S2.T1.25.25.25.2.m1.1.1" xref="S2.T1.25.25.25.2.m1.1.1.cmml"><mi id="S2.T1.25.25.25.2.m1.1.1.2" xref="S2.T1.25.25.25.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.25.25.25.2.m1.1.1.3" xref="S2.T1.25.25.25.2.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.25.25.25.2.m1.1b"><apply id="S2.T1.25.25.25.2.m1.1.1.cmml" xref="S2.T1.25.25.25.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.25.25.25.2.m1.1.1.1.cmml" xref="S2.T1.25.25.25.2.m1.1.1">subscript</csymbol><ci id="S2.T1.25.25.25.2.m1.1.1.2.cmml" xref="S2.T1.25.25.25.2.m1.1.1.2">𝑆</ci><cn id="S2.T1.25.25.25.2.m1.1.1.3.cmml" type="integer" xref="S2.T1.25.25.25.2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.25.25.25.2.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.25.25.25.2.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> of the current incident to the next, showing how frequently failures happen.</td> </tr> <tr class="ltx_tr" id="S2.T1.28.28.28"> <td class="ltx_td ltx_align_left ltx_border_bb" id="S2.T1.28.28.28.3"> <math alttext="T" class="ltx_Math" display="inline" id="S2.T1.26.26.26.1.m1.1"><semantics id="S2.T1.26.26.26.1.m1.1a"><mi id="S2.T1.26.26.26.1.m1.1.1" xref="S2.T1.26.26.26.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S2.T1.26.26.26.1.m1.1b"><ci id="S2.T1.26.26.26.1.m1.1.1.cmml" xref="S2.T1.26.26.26.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.26.26.26.1.m1.1c">T</annotation><annotation encoding="application/x-llamapun" id="S2.T1.26.26.26.1.m1.1d">italic_T</annotation></semantics></math>, <math alttext="T_{S}" class="ltx_Math" display="inline" id="S2.T1.27.27.27.2.m2.1"><semantics id="S2.T1.27.27.27.2.m2.1a"><msub id="S2.T1.27.27.27.2.m2.1.1" xref="S2.T1.27.27.27.2.m2.1.1.cmml"><mi id="S2.T1.27.27.27.2.m2.1.1.2" xref="S2.T1.27.27.27.2.m2.1.1.2.cmml">T</mi><mi id="S2.T1.27.27.27.2.m2.1.1.3" xref="S2.T1.27.27.27.2.m2.1.1.3.cmml">S</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.27.27.27.2.m2.1b"><apply id="S2.T1.27.27.27.2.m2.1.1.cmml" xref="S2.T1.27.27.27.2.m2.1.1"><csymbol cd="ambiguous" id="S2.T1.27.27.27.2.m2.1.1.1.cmml" xref="S2.T1.27.27.27.2.m2.1.1">subscript</csymbol><ci id="S2.T1.27.27.27.2.m2.1.1.2.cmml" xref="S2.T1.27.27.27.2.m2.1.1.2">𝑇</ci><ci id="S2.T1.27.27.27.2.m2.1.1.3.cmml" xref="S2.T1.27.27.27.2.m2.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.27.27.27.2.m2.1c">T_{S}</annotation><annotation encoding="application/x-llamapun" id="S2.T1.27.27.27.2.m2.1d">italic_T start_POSTSUBSCRIPT italic_S end_POSTSUBSCRIPT</annotation></semantics></math>, <math alttext="A" class="ltx_Math" display="inline" id="S2.T1.28.28.28.3.m3.1"><semantics id="S2.T1.28.28.28.3.m3.1a"><mi id="S2.T1.28.28.28.3.m3.1.1" xref="S2.T1.28.28.28.3.m3.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.T1.28.28.28.3.m3.1b"><ci id="S2.T1.28.28.28.3.m3.1.1.cmml" xref="S2.T1.28.28.28.3.m3.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.28.28.28.3.m3.1c">A</annotation><annotation encoding="application/x-llamapun" id="S2.T1.28.28.28.3.m3.1d">italic_A</annotation></semantics></math> </td> <td class="ltx_td ltx_align_left ltx_border_bb" id="S2.T1.28.28.28.4">Outage time, scaled, availability</td> <td class="ltx_td ltx_align_left ltx_border_bb" id="S2.T1.28.28.28.5">Definitions discussed in <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.SS2" title="2.2. LLM-Specific Terms and Metrics ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 2.2</a> </td> </tr> </table> </div> </figure> <figure class="ltx_table" id="S2.T2"> <figcaption class="ltx_caption ltx_centering">Table 2. Values of parameters for the selected incident <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib41" title="">2024a</a>)</cite>, UDT time. Status-markers <math alttext="S_{1}" class="ltx_Math" display="inline" id="S2.T2.3.1.m1.1"><semantics id="S2.T2.3.1.m1.1b"><msub id="S2.T2.3.1.m1.1.1" xref="S2.T2.3.1.m1.1.1.cmml"><mi id="S2.T2.3.1.m1.1.1.2" xref="S2.T2.3.1.m1.1.1.2.cmml">S</mi><mn id="S2.T2.3.1.m1.1.1.3" xref="S2.T2.3.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T2.3.1.m1.1c"><apply id="S2.T2.3.1.m1.1.1.cmml" xref="S2.T2.3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.3.1.m1.1.1.1.cmml" xref="S2.T2.3.1.m1.1.1">subscript</csymbol><ci id="S2.T2.3.1.m1.1.1.2.cmml" xref="S2.T2.3.1.m1.1.1.2">𝑆</ci><cn id="S2.T2.3.1.m1.1.1.3.cmml" type="integer" xref="S2.T2.3.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.3.1.m1.1d">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.3.1.m1.1e">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> through <math alttext="S_{4}" class="ltx_Math" display="inline" id="S2.T2.4.2.m2.1"><semantics id="S2.T2.4.2.m2.1b"><msub id="S2.T2.4.2.m2.1.1" xref="S2.T2.4.2.m2.1.1.cmml"><mi id="S2.T2.4.2.m2.1.1.2" xref="S2.T2.4.2.m2.1.1.2.cmml">S</mi><mn id="S2.T2.4.2.m2.1.1.3" xref="S2.T2.4.2.m2.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T2.4.2.m2.1c"><apply id="S2.T2.4.2.m2.1.1.cmml" xref="S2.T2.4.2.m2.1.1"><csymbol cd="ambiguous" id="S2.T2.4.2.m2.1.1.1.cmml" xref="S2.T2.4.2.m2.1.1">subscript</csymbol><ci id="S2.T2.4.2.m2.1.1.2.cmml" xref="S2.T2.4.2.m2.1.1.2">𝑆</ci><cn id="S2.T2.4.2.m2.1.1.3.cmml" type="integer" xref="S2.T2.4.2.m2.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.4.2.m2.1d">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.4.2.m2.1e">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math> occur on April 10, 2024.</figcaption> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S2.T2.13" style="width:385.9pt;height:24.9pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_align_middle" id="S2.T2.13.9"> <tr class="ltx_tr" id="S2.T2.13.9.9"> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.13.9.9.10">Incident ID</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.5.1.1.1"><math alttext="S_{1}" class="ltx_Math" display="inline" id="S2.T2.5.1.1.1.m1.1"><semantics id="S2.T2.5.1.1.1.m1.1a"><msub id="S2.T2.5.1.1.1.m1.1.1" xref="S2.T2.5.1.1.1.m1.1.1.cmml"><mi id="S2.T2.5.1.1.1.m1.1.1.2" xref="S2.T2.5.1.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T2.5.1.1.1.m1.1.1.3" xref="S2.T2.5.1.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T2.5.1.1.1.m1.1b"><apply id="S2.T2.5.1.1.1.m1.1.1.cmml" xref="S2.T2.5.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.5.1.1.1.m1.1.1.1.cmml" xref="S2.T2.5.1.1.1.m1.1.1">subscript</csymbol><ci id="S2.T2.5.1.1.1.m1.1.1.2.cmml" xref="S2.T2.5.1.1.1.m1.1.1.2">𝑆</ci><cn id="S2.T2.5.1.1.1.m1.1.1.3.cmml" type="integer" xref="S2.T2.5.1.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.5.1.1.1.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.5.1.1.1.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.6.2.2.2"><math alttext="S_{2}" class="ltx_Math" display="inline" id="S2.T2.6.2.2.2.m1.1"><semantics id="S2.T2.6.2.2.2.m1.1a"><msub id="S2.T2.6.2.2.2.m1.1.1" xref="S2.T2.6.2.2.2.m1.1.1.cmml"><mi id="S2.T2.6.2.2.2.m1.1.1.2" xref="S2.T2.6.2.2.2.m1.1.1.2.cmml">S</mi><mn id="S2.T2.6.2.2.2.m1.1.1.3" xref="S2.T2.6.2.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T2.6.2.2.2.m1.1b"><apply id="S2.T2.6.2.2.2.m1.1.1.cmml" xref="S2.T2.6.2.2.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.6.2.2.2.m1.1.1.1.cmml" xref="S2.T2.6.2.2.2.m1.1.1">subscript</csymbol><ci id="S2.T2.6.2.2.2.m1.1.1.2.cmml" xref="S2.T2.6.2.2.2.m1.1.1.2">𝑆</ci><cn id="S2.T2.6.2.2.2.m1.1.1.3.cmml" type="integer" xref="S2.T2.6.2.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.6.2.2.2.m1.1c">S_{2}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.6.2.2.2.m1.1d">italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.7.3.3.3"><math alttext="S_{3}" class="ltx_Math" display="inline" id="S2.T2.7.3.3.3.m1.1"><semantics id="S2.T2.7.3.3.3.m1.1a"><msub id="S2.T2.7.3.3.3.m1.1.1" xref="S2.T2.7.3.3.3.m1.1.1.cmml"><mi id="S2.T2.7.3.3.3.m1.1.1.2" xref="S2.T2.7.3.3.3.m1.1.1.2.cmml">S</mi><mn id="S2.T2.7.3.3.3.m1.1.1.3" xref="S2.T2.7.3.3.3.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T2.7.3.3.3.m1.1b"><apply id="S2.T2.7.3.3.3.m1.1.1.cmml" xref="S2.T2.7.3.3.3.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.7.3.3.3.m1.1.1.1.cmml" xref="S2.T2.7.3.3.3.m1.1.1">subscript</csymbol><ci id="S2.T2.7.3.3.3.m1.1.1.2.cmml" xref="S2.T2.7.3.3.3.m1.1.1.2">𝑆</ci><cn id="S2.T2.7.3.3.3.m1.1.1.3.cmml" type="integer" xref="S2.T2.7.3.3.3.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.7.3.3.3.m1.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.7.3.3.3.m1.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.8.4.4.4"><math alttext="S_{4}" class="ltx_Math" display="inline" id="S2.T2.8.4.4.4.m1.1"><semantics id="S2.T2.8.4.4.4.m1.1a"><msub id="S2.T2.8.4.4.4.m1.1.1" xref="S2.T2.8.4.4.4.m1.1.1.cmml"><mi id="S2.T2.8.4.4.4.m1.1.1.2" xref="S2.T2.8.4.4.4.m1.1.1.2.cmml">S</mi><mn id="S2.T2.8.4.4.4.m1.1.1.3" xref="S2.T2.8.4.4.4.m1.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T2.8.4.4.4.m1.1b"><apply id="S2.T2.8.4.4.4.m1.1.1.cmml" xref="S2.T2.8.4.4.4.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.8.4.4.4.m1.1.1.1.cmml" xref="S2.T2.8.4.4.4.m1.1.1">subscript</csymbol><ci id="S2.T2.8.4.4.4.m1.1.1.2.cmml" xref="S2.T2.8.4.4.4.m1.1.1.2">𝑆</ci><cn id="S2.T2.8.4.4.4.m1.1.1.3.cmml" type="integer" xref="S2.T2.8.4.4.4.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.8.4.4.4.m1.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.8.4.4.4.m1.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.9.5.5.5"><math alttext="S_{5}" class="ltx_Math" display="inline" id="S2.T2.9.5.5.5.m1.1"><semantics id="S2.T2.9.5.5.5.m1.1a"><msub id="S2.T2.9.5.5.5.m1.1.1" xref="S2.T2.9.5.5.5.m1.1.1.cmml"><mi id="S2.T2.9.5.5.5.m1.1.1.2" xref="S2.T2.9.5.5.5.m1.1.1.2.cmml">S</mi><mn id="S2.T2.9.5.5.5.m1.1.1.3" xref="S2.T2.9.5.5.5.m1.1.1.3.cmml">5</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T2.9.5.5.5.m1.1b"><apply id="S2.T2.9.5.5.5.m1.1.1.cmml" xref="S2.T2.9.5.5.5.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.9.5.5.5.m1.1.1.1.cmml" xref="S2.T2.9.5.5.5.m1.1.1">subscript</csymbol><ci id="S2.T2.9.5.5.5.m1.1.1.2.cmml" xref="S2.T2.9.5.5.5.m1.1.1.2">𝑆</ci><cn id="S2.T2.9.5.5.5.m1.1.1.3.cmml" type="integer" xref="S2.T2.9.5.5.5.m1.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.9.5.5.5.m1.1c">S_{5}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.9.5.5.5.m1.1d">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.10.6.6.6"> <math alttext="P_{I}" class="ltx_Math" display="inline" id="S2.T2.10.6.6.6.m1.1"><semantics id="S2.T2.10.6.6.6.m1.1a"><msub id="S2.T2.10.6.6.6.m1.1.1" xref="S2.T2.10.6.6.6.m1.1.1.cmml"><mi id="S2.T2.10.6.6.6.m1.1.1.2" xref="S2.T2.10.6.6.6.m1.1.1.2.cmml">P</mi><mi id="S2.T2.10.6.6.6.m1.1.1.3" xref="S2.T2.10.6.6.6.m1.1.1.3.cmml">I</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T2.10.6.6.6.m1.1b"><apply id="S2.T2.10.6.6.6.m1.1.1.cmml" xref="S2.T2.10.6.6.6.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.10.6.6.6.m1.1.1.1.cmml" xref="S2.T2.10.6.6.6.m1.1.1">subscript</csymbol><ci id="S2.T2.10.6.6.6.m1.1.1.2.cmml" xref="S2.T2.10.6.6.6.m1.1.1.2">𝑃</ci><ci id="S2.T2.10.6.6.6.m1.1.1.3.cmml" xref="S2.T2.10.6.6.6.m1.1.1.3">𝐼</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.10.6.6.6.m1.1c">P_{I}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.10.6.6.6.m1.1d">italic_P start_POSTSUBSCRIPT italic_I end_POSTSUBSCRIPT</annotation></semantics></math> [h]</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.11.7.7.7"> <math alttext="P_{R}" class="ltx_Math" display="inline" id="S2.T2.11.7.7.7.m1.1"><semantics id="S2.T2.11.7.7.7.m1.1a"><msub id="S2.T2.11.7.7.7.m1.1.1" xref="S2.T2.11.7.7.7.m1.1.1.cmml"><mi id="S2.T2.11.7.7.7.m1.1.1.2" xref="S2.T2.11.7.7.7.m1.1.1.2.cmml">P</mi><mi id="S2.T2.11.7.7.7.m1.1.1.3" xref="S2.T2.11.7.7.7.m1.1.1.3.cmml">R</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T2.11.7.7.7.m1.1b"><apply id="S2.T2.11.7.7.7.m1.1.1.cmml" xref="S2.T2.11.7.7.7.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.11.7.7.7.m1.1.1.1.cmml" xref="S2.T2.11.7.7.7.m1.1.1">subscript</csymbol><ci id="S2.T2.11.7.7.7.m1.1.1.2.cmml" xref="S2.T2.11.7.7.7.m1.1.1.2">𝑃</ci><ci id="S2.T2.11.7.7.7.m1.1.1.3.cmml" xref="S2.T2.11.7.7.7.m1.1.1.3">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.11.7.7.7.m1.1c">P_{R}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.11.7.7.7.m1.1d">italic_P start_POSTSUBSCRIPT italic_R end_POSTSUBSCRIPT</annotation></semantics></math> [h]</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.12.8.8.8"> <math alttext="P_{C}" class="ltx_Math" display="inline" id="S2.T2.12.8.8.8.m1.1"><semantics id="S2.T2.12.8.8.8.m1.1a"><msub id="S2.T2.12.8.8.8.m1.1.1" xref="S2.T2.12.8.8.8.m1.1.1.cmml"><mi id="S2.T2.12.8.8.8.m1.1.1.2" xref="S2.T2.12.8.8.8.m1.1.1.2.cmml">P</mi><mi id="S2.T2.12.8.8.8.m1.1.1.3" xref="S2.T2.12.8.8.8.m1.1.1.3.cmml">C</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T2.12.8.8.8.m1.1b"><apply id="S2.T2.12.8.8.8.m1.1.1.cmml" xref="S2.T2.12.8.8.8.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.12.8.8.8.m1.1.1.1.cmml" xref="S2.T2.12.8.8.8.m1.1.1">subscript</csymbol><ci id="S2.T2.12.8.8.8.m1.1.1.2.cmml" xref="S2.T2.12.8.8.8.m1.1.1.2">𝑃</ci><ci id="S2.T2.12.8.8.8.m1.1.1.3.cmml" xref="S2.T2.12.8.8.8.m1.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.12.8.8.8.m1.1c">P_{C}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.12.8.8.8.m1.1d">italic_P start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT</annotation></semantics></math> [h]</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.13.9.9.9"> <math alttext="P_{L}" class="ltx_Math" display="inline" id="S2.T2.13.9.9.9.m1.1"><semantics id="S2.T2.13.9.9.9.m1.1a"><msub id="S2.T2.13.9.9.9.m1.1.1" xref="S2.T2.13.9.9.9.m1.1.1.cmml"><mi id="S2.T2.13.9.9.9.m1.1.1.2" xref="S2.T2.13.9.9.9.m1.1.1.2.cmml">P</mi><mi id="S2.T2.13.9.9.9.m1.1.1.3" xref="S2.T2.13.9.9.9.m1.1.1.3.cmml">L</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T2.13.9.9.9.m1.1b"><apply id="S2.T2.13.9.9.9.m1.1.1.cmml" xref="S2.T2.13.9.9.9.m1.1.1"><csymbol cd="ambiguous" id="S2.T2.13.9.9.9.m1.1.1.1.cmml" xref="S2.T2.13.9.9.9.m1.1.1">subscript</csymbol><ci id="S2.T2.13.9.9.9.m1.1.1.2.cmml" xref="S2.T2.13.9.9.9.m1.1.1.2">𝑃</ci><ci id="S2.T2.13.9.9.9.m1.1.1.3.cmml" xref="S2.T2.13.9.9.9.m1.1.1.3">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T2.13.9.9.9.m1.1c">P_{L}</annotation><annotation encoding="application/x-llamapun" id="S2.T2.13.9.9.9.m1.1d">italic_P start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT</annotation></semantics></math> [h]</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T2.13.9.9.11">Time To Resolve [h]</td> </tr> <tr class="ltx_tr" id="S2.T2.13.9.10"> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.1">w20mcckg1748</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.2">17:56</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.3">20:39</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.4">20:49</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.5">20:56</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.6">2024-04-18 00:01</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.7">2.72</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.8">0.17</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.9">0.12</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.10">171.08</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T2.13.9.10.11">3.00</td> </tr> </table> </div> </figure> <div class="ltx_para" id="S2.SS1.p4"> Incident visualization and model parameters: Focusing on the major ChatGPT outage on April 10, 2024, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.F2" title="In 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 2</a> visualizes the failure-recovery process as reported by ChatGPT, overlapping it with the number of user reports as reported by DownDetector. Around 2024-04-10 17:30, with the service believed to operate normally, some faults started to happen and the number of user reports increased to an abnormal level. This triggered an alert to the ChatGPT operational team, who started investigating at 17:56 (status <math alttext="S_{1}" class="ltx_Math" display="inline" id="S2.SS1.p4.1.m1.1"><semantics id="S2.SS1.p4.1.m1.1a"><msub id="S2.SS1.p4.1.m1.1.1" xref="S2.SS1.p4.1.m1.1.1.cmml"><mi id="S2.SS1.p4.1.m1.1.1.2" xref="S2.SS1.p4.1.m1.1.1.2.cmml">S</mi><mn id="S2.SS1.p4.1.m1.1.1.3" xref="S2.SS1.p4.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.1.m1.1b"><apply id="S2.SS1.p4.1.m1.1.1.cmml" xref="S2.SS1.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.1.m1.1.1.1.cmml" xref="S2.SS1.p4.1.m1.1.1">subscript</csymbol><ci id="S2.SS1.p4.1.m1.1.1.2.cmml" xref="S2.SS1.p4.1.m1.1.1.2">𝑆</ci><cn id="S2.SS1.p4.1.m1.1.1.3.cmml" type="integer" xref="S2.SS1.p4.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.1.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p4.1.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>). At 20:39, they identified the issue (<math alttext="S_{2}" class="ltx_Math" display="inline" id="S2.SS1.p4.2.m2.1"><semantics id="S2.SS1.p4.2.m2.1a"><msub id="S2.SS1.p4.2.m2.1.1" xref="S2.SS1.p4.2.m2.1.1.cmml"><mi id="S2.SS1.p4.2.m2.1.1.2" xref="S2.SS1.p4.2.m2.1.1.2.cmml">S</mi><mn id="S2.SS1.p4.2.m2.1.1.3" xref="S2.SS1.p4.2.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.2.m2.1b"><apply id="S2.SS1.p4.2.m2.1.1.cmml" xref="S2.SS1.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.2.m2.1.1.1.cmml" xref="S2.SS1.p4.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p4.2.m2.1.1.2.cmml" xref="S2.SS1.p4.2.m2.1.1.2">𝑆</ci><cn id="S2.SS1.p4.2.m2.1.1.3.cmml" type="integer" xref="S2.SS1.p4.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.2.m2.1c">S_{2}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p4.2.m2.1d">italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math>). They quickly implemented a fix, which they released and started monitoring at 20:49 (<math alttext="S_{3}" class="ltx_Math" display="inline" id="S2.SS1.p4.3.m3.1"><semantics id="S2.SS1.p4.3.m3.1a"><msub id="S2.SS1.p4.3.m3.1.1" xref="S2.SS1.p4.3.m3.1.1.cmml"><mi id="S2.SS1.p4.3.m3.1.1.2" xref="S2.SS1.p4.3.m3.1.1.2.cmml">S</mi><mn id="S2.SS1.p4.3.m3.1.1.3" xref="S2.SS1.p4.3.m3.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.3.m3.1b"><apply id="S2.SS1.p4.3.m3.1.1.cmml" xref="S2.SS1.p4.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.3.m3.1.1.1.cmml" xref="S2.SS1.p4.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.p4.3.m3.1.1.2.cmml" xref="S2.SS1.p4.3.m3.1.1.2">𝑆</ci><cn id="S2.SS1.p4.3.m3.1.1.3.cmml" type="integer" xref="S2.SS1.p4.3.m3.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.3.m3.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p4.3.m3.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math>). They confirmed that the issue had been resolved at 20:56 (<math alttext="S_{4}" class="ltx_Math" display="inline" id="S2.SS1.p4.4.m4.1"><semantics id="S2.SS1.p4.4.m4.1a"><msub id="S2.SS1.p4.4.m4.1.1" xref="S2.SS1.p4.4.m4.1.1.cmml"><mi id="S2.SS1.p4.4.m4.1.1.2" xref="S2.SS1.p4.4.m4.1.1.2.cmml">S</mi><mn id="S2.SS1.p4.4.m4.1.1.3" xref="S2.SS1.p4.4.m4.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.4.m4.1b"><apply id="S2.SS1.p4.4.m4.1.1.cmml" xref="S2.SS1.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.4.m4.1.1.1.cmml" xref="S2.SS1.p4.4.m4.1.1">subscript</csymbol><ci id="S2.SS1.p4.4.m4.1.1.2.cmml" xref="S2.SS1.p4.4.m4.1.1.2">𝑆</ci><cn id="S2.SS1.p4.4.m4.1.1.3.cmml" type="integer" xref="S2.SS1.p4.4.m4.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.4.m4.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p4.4.m4.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math>). During this period, the user reports increased to a peak, around which it fluctuated until the fix was released to increasingly more users, at which point a sharp drop of user reports, toward a normal level, can be seen in the figure. Finally, after a period of postmortem analysis, an incident summary was released by the ChatGPT team, to explain the cause of this incident to its users (<math alttext="S_{5}" class="ltx_Math" display="inline" id="S2.SS1.p4.5.m5.1"><semantics id="S2.SS1.p4.5.m5.1a"><msub id="S2.SS1.p4.5.m5.1.1" xref="S2.SS1.p4.5.m5.1.1.cmml"><mi id="S2.SS1.p4.5.m5.1.1.2" xref="S2.SS1.p4.5.m5.1.1.2.cmml">S</mi><mn id="S2.SS1.p4.5.m5.1.1.3" xref="S2.SS1.p4.5.m5.1.1.3.cmml">5</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.5.m5.1b"><apply id="S2.SS1.p4.5.m5.1.1.cmml" xref="S2.SS1.p4.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.5.m5.1.1.1.cmml" xref="S2.SS1.p4.5.m5.1.1">subscript</csymbol><ci id="S2.SS1.p4.5.m5.1.1.2.cmml" xref="S2.SS1.p4.5.m5.1.1.2">𝑆</ci><cn id="S2.SS1.p4.5.m5.1.1.3.cmml" type="integer" xref="S2.SS1.p4.5.m5.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.5.m5.1c">S_{5}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p4.5.m5.1d">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT</annotation></semantics></math>). </div> <div class="ltx_para" id="S2.SS1.p5"> This incident exemplifies a failure-recovery process with five key status-markers, <math alttext="S_{1}" class="ltx_Math" display="inline" id="S2.SS1.p5.1.m1.1"><semantics id="S2.SS1.p5.1.m1.1a"><msub id="S2.SS1.p5.1.m1.1.1" xref="S2.SS1.p5.1.m1.1.1.cmml"><mi id="S2.SS1.p5.1.m1.1.1.2" xref="S2.SS1.p5.1.m1.1.1.2.cmml">S</mi><mn id="S2.SS1.p5.1.m1.1.1.3" xref="S2.SS1.p5.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.1.m1.1b"><apply id="S2.SS1.p5.1.m1.1.1.cmml" xref="S2.SS1.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.1.m1.1.1.1.cmml" xref="S2.SS1.p5.1.m1.1.1">subscript</csymbol><ci id="S2.SS1.p5.1.m1.1.1.2.cmml" xref="S2.SS1.p5.1.m1.1.1.2">𝑆</ci><cn id="S2.SS1.p5.1.m1.1.1.3.cmml" type="integer" xref="S2.SS1.p5.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.1.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p5.1.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> through <math alttext="S_{5}" class="ltx_Math" display="inline" id="S2.SS1.p5.2.m2.1"><semantics id="S2.SS1.p5.2.m2.1a"><msub id="S2.SS1.p5.2.m2.1.1" xref="S2.SS1.p5.2.m2.1.1.cmml"><mi id="S2.SS1.p5.2.m2.1.1.2" xref="S2.SS1.p5.2.m2.1.1.2.cmml">S</mi><mn id="S2.SS1.p5.2.m2.1.1.3" xref="S2.SS1.p5.2.m2.1.1.3.cmml">5</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.2.m2.1b"><apply id="S2.SS1.p5.2.m2.1.1.cmml" xref="S2.SS1.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.2.m2.1.1.1.cmml" xref="S2.SS1.p5.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p5.2.m2.1.1.2.cmml" xref="S2.SS1.p5.2.m2.1.1.2">𝑆</ci><cn id="S2.SS1.p5.2.m2.1.1.3.cmml" type="integer" xref="S2.SS1.p5.2.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.2.m2.1c">S_{5}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p5.2.m2.1d">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT</annotation></semantics></math>. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.T1" title="In 2.1. Model and Real-World Example ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 1</a> summarizes the industry-wide model, including these status-markers, the periods they delimit, and the operational metrics. Among the periods, whereas <math alttext="P_{I}" class="ltx_Math" display="inline" id="S2.SS1.p5.3.m3.1"><semantics id="S2.SS1.p5.3.m3.1a"><msub id="S2.SS1.p5.3.m3.1.1" xref="S2.SS1.p5.3.m3.1.1.cmml"><mi id="S2.SS1.p5.3.m3.1.1.2" xref="S2.SS1.p5.3.m3.1.1.2.cmml">P</mi><mi id="S2.SS1.p5.3.m3.1.1.3" xref="S2.SS1.p5.3.m3.1.1.3.cmml">I</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.3.m3.1b"><apply id="S2.SS1.p5.3.m3.1.1.cmml" xref="S2.SS1.p5.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.3.m3.1.1.1.cmml" xref="S2.SS1.p5.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.p5.3.m3.1.1.2.cmml" xref="S2.SS1.p5.3.m3.1.1.2">𝑃</ci><ci id="S2.SS1.p5.3.m3.1.1.3.cmml" xref="S2.SS1.p5.3.m3.1.1.3">𝐼</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.3.m3.1c">P_{I}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p5.3.m3.1d">italic_P start_POSTSUBSCRIPT italic_I end_POSTSUBSCRIPT</annotation></semantics></math>, <math alttext="P_{R}" class="ltx_Math" display="inline" id="S2.SS1.p5.4.m4.1"><semantics id="S2.SS1.p5.4.m4.1a"><msub id="S2.SS1.p5.4.m4.1.1" xref="S2.SS1.p5.4.m4.1.1.cmml"><mi id="S2.SS1.p5.4.m4.1.1.2" xref="S2.SS1.p5.4.m4.1.1.2.cmml">P</mi><mi id="S2.SS1.p5.4.m4.1.1.3" xref="S2.SS1.p5.4.m4.1.1.3.cmml">R</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.4.m4.1b"><apply id="S2.SS1.p5.4.m4.1.1.cmml" xref="S2.SS1.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.4.m4.1.1.1.cmml" xref="S2.SS1.p5.4.m4.1.1">subscript</csymbol><ci id="S2.SS1.p5.4.m4.1.1.2.cmml" xref="S2.SS1.p5.4.m4.1.1.2">𝑃</ci><ci id="S2.SS1.p5.4.m4.1.1.3.cmml" xref="S2.SS1.p5.4.m4.1.1.3">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.4.m4.1c">P_{R}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p5.4.m4.1d">italic_P start_POSTSUBSCRIPT italic_R end_POSTSUBSCRIPT</annotation></semantics></math>, and <math alttext="P_{C}" class="ltx_Math" display="inline" id="S2.SS1.p5.5.m5.1"><semantics id="S2.SS1.p5.5.m5.1a"><msub id="S2.SS1.p5.5.m5.1.1" xref="S2.SS1.p5.5.m5.1.1.cmml"><mi id="S2.SS1.p5.5.m5.1.1.2" xref="S2.SS1.p5.5.m5.1.1.2.cmml">P</mi><mi id="S2.SS1.p5.5.m5.1.1.3" xref="S2.SS1.p5.5.m5.1.1.3.cmml">C</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.5.m5.1b"><apply id="S2.SS1.p5.5.m5.1.1.cmml" xref="S2.SS1.p5.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.5.m5.1.1.1.cmml" xref="S2.SS1.p5.5.m5.1.1">subscript</csymbol><ci id="S2.SS1.p5.5.m5.1.1.2.cmml" xref="S2.SS1.p5.5.m5.1.1.2">𝑃</ci><ci id="S2.SS1.p5.5.m5.1.1.3.cmml" xref="S2.SS1.p5.5.m5.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.5.m5.1c">P_{C}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p5.5.m5.1d">italic_P start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT</annotation></semantics></math> capture how the operational team resolved the outage and are thus synchronous with users experiencing the incident, <math alttext="P_{L}" class="ltx_Math" display="inline" id="S2.SS1.p5.6.m6.1"><semantics id="S2.SS1.p5.6.m6.1a"><msub id="S2.SS1.p5.6.m6.1.1" xref="S2.SS1.p5.6.m6.1.1.cmml"><mi id="S2.SS1.p5.6.m6.1.1.2" xref="S2.SS1.p5.6.m6.1.1.2.cmml">P</mi><mi id="S2.SS1.p5.6.m6.1.1.3" xref="S2.SS1.p5.6.m6.1.1.3.cmml">L</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.6.m6.1b"><apply id="S2.SS1.p5.6.m6.1.1.cmml" xref="S2.SS1.p5.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.6.m6.1.1.1.cmml" xref="S2.SS1.p5.6.m6.1.1">subscript</csymbol><ci id="S2.SS1.p5.6.m6.1.1.2.cmml" xref="S2.SS1.p5.6.m6.1.1.2">𝑃</ci><ci id="S2.SS1.p5.6.m6.1.1.3.cmml" xref="S2.SS1.p5.6.m6.1.1.3">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.6.m6.1c">P_{L}</annotation><annotation encoding="application/x-llamapun" id="S2.SS1.p5.6.m6.1d">italic_P start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT</annotation></semantics></math> captures the period, sometimes ending long after the incident has been resolved, during which the operational team analyses the incident and devises new measures to prevent it and related incidents from happening again. </div> <div class="ltx_para" id="S2.SS1.p6"> The model outputs include the industry-standard Time To Resolve and, with knowledge about prior and following incidents, Time Between Failures. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.T2" title="In 2.1. Model and Real-World Example ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 2</a> summarizes the status-markers, the periods spanning the failure-recovery process, and the Time To Resolve for this incident. It shows how the operational team took much longer than usual to find the cause of this incident, 2.72 h vs. the 0.65 h found in our analysis in <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4" title="4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 4</a> (see <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.T6" title="In 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 6</a>), but then it was able to resolve the failure and restore service much faster than normal. However, by then it was already late, as the media has picked up the incident. </div> </section> <section class="ltx_subsection" id="S2.SS2"> <h3 class="ltx_title ltx_title_subsection"> 2.2. LLM-Specific Terms and Metrics</h3> <div class="ltx_para" id="S2.SS2.p1"> Incident: An operational issue that may cause a service outage, e.g., ”getting an error of having reached a limit of GPT-4 usage” <cite class="ltx_cite ltx_citemacro_citep">(Archive, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib8" title="">2024</a>)</cite>. Once an incident happens, a textual report of its failure-recovery process is produced and is (expectedly) disclosed to the public. An incident can have different impact levels on single or multiple services, which include critical, major, minor, minimal, and maintenance, and are similarly defined by the LLM operators. </div> <div class="ltx_para" id="S2.SS2.p2"> Outage: Time when the service is unavailable. An outage can have multiple impact ranges: major outage, where most of the service’s users experience it, and partial outage, where a relatively small fraction of the users experience the outage. Operators such as OpenAI scale (discount) partial outages as being about 30% (<math alttext="0.3\times" class="ltx_math_unparsed" display="inline" id="S2.SS2.p2.1.m1.1"><semantics id="S2.SS2.p2.1.m1.1a"><mrow id="S2.SS2.p2.1.m1.1b"><mn id="S2.SS2.p2.1.m1.1.1">0.3</mn><mo id="S2.SS2.p2.1.m1.1.2" lspace="0.222em">×</mo></mrow><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.1c">0.3\times</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.p2.1.m1.1d">0.3 ×</annotation></semantics></math>) as bad as major outages <cite class="ltx_cite ltx_citemacro_citep">(Atlassian, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib9" title="">2024</a>)</cite>. </div> <div class="ltx_para" id="S2.SS2.p3"> Outage duration: For an operator, per day, let <math alttext="T_{M}" class="ltx_Math" display="inline" id="S2.SS2.p3.1.m1.1"><semantics id="S2.SS2.p3.1.m1.1a"><msub id="S2.SS2.p3.1.m1.1.1" xref="S2.SS2.p3.1.m1.1.1.cmml"><mi id="S2.SS2.p3.1.m1.1.1.2" xref="S2.SS2.p3.1.m1.1.1.2.cmml">T</mi><mi id="S2.SS2.p3.1.m1.1.1.3" xref="S2.SS2.p3.1.m1.1.1.3.cmml">M</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.1b"><apply id="S2.SS2.p3.1.m1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.p3.1.m1.1.1.2.cmml" xref="S2.SS2.p3.1.m1.1.1.2">𝑇</ci><ci id="S2.SS2.p3.1.m1.1.1.3.cmml" xref="S2.SS2.p3.1.m1.1.1.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.1c">T_{M}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.p3.1.m1.1d">italic_T start_POSTSUBSCRIPT italic_M end_POSTSUBSCRIPT</annotation></semantics></math> be the duration of major outage minutes and <math alttext="T_{P}" class="ltx_Math" display="inline" id="S2.SS2.p3.2.m2.1"><semantics id="S2.SS2.p3.2.m2.1a"><msub id="S2.SS2.p3.2.m2.1.1" xref="S2.SS2.p3.2.m2.1.1.cmml"><mi id="S2.SS2.p3.2.m2.1.1.2" xref="S2.SS2.p3.2.m2.1.1.2.cmml">T</mi><mi id="S2.SS2.p3.2.m2.1.1.3" xref="S2.SS2.p3.2.m2.1.1.3.cmml">P</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><apply id="S2.SS2.p3.2.m2.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.2.m2.1.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1">subscript</csymbol><ci id="S2.SS2.p3.2.m2.1.1.2.cmml" xref="S2.SS2.p3.2.m2.1.1.2">𝑇</ci><ci id="S2.SS2.p3.2.m2.1.1.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3">𝑃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">T_{P}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.p3.2.m2.1d">italic_T start_POSTSUBSCRIPT italic_P end_POSTSUBSCRIPT</annotation></semantics></math> be the partial outage minutes. The formula to calculate the daily total outage minutes (<math alttext="T" class="ltx_Math" display="inline" id="S2.SS2.p3.3.1.m1.1"><semantics id="S2.SS2.p3.3.1.m1.1a"><mi id="S2.SS2.p3.3.1.m1.1.1" xref="S2.SS2.p3.3.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.3.1.m1.1b"><ci id="S2.SS2.p3.3.1.m1.1.1.cmml" xref="S2.SS2.p3.3.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.3.1.m1.1c">T</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.p3.3.1.m1.1d">italic_T</annotation></semantics></math>) is: <table class="ltx_equation ltx_eqn_table" id="S2.E1"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1">(1)</td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="T=T_{M}+T_{P}" class="ltx_Math" display="block" id="S2.E1.m1.1"><semantics id="S2.E1.m1.1a"><mrow id="S2.E1.m1.1.1" xref="S2.E1.m1.1.1.cmml"><mi id="S2.E1.m1.1.1.2" xref="S2.E1.m1.1.1.2.cmml">T</mi><mo id="S2.E1.m1.1.1.1" xref="S2.E1.m1.1.1.1.cmml">=</mo><mrow id="S2.E1.m1.1.1.3" xref="S2.E1.m1.1.1.3.cmml"><msub id="S2.E1.m1.1.1.3.2" xref="S2.E1.m1.1.1.3.2.cmml"><mi id="S2.E1.m1.1.1.3.2.2" xref="S2.E1.m1.1.1.3.2.2.cmml">T</mi><mi id="S2.E1.m1.1.1.3.2.3" xref="S2.E1.m1.1.1.3.2.3.cmml">M</mi></msub><mo id="S2.E1.m1.1.1.3.1" xref="S2.E1.m1.1.1.3.1.cmml">+</mo><msub id="S2.E1.m1.1.1.3.3" xref="S2.E1.m1.1.1.3.3.cmml"><mi id="S2.E1.m1.1.1.3.3.2" xref="S2.E1.m1.1.1.3.3.2.cmml">T</mi><mi id="S2.E1.m1.1.1.3.3.3" xref="S2.E1.m1.1.1.3.3.3.cmml">P</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><apply id="S2.E1.m1.1.1.cmml" xref="S2.E1.m1.1.1"><eq id="S2.E1.m1.1.1.1.cmml" xref="S2.E1.m1.1.1.1"></eq><ci id="S2.E1.m1.1.1.2.cmml" xref="S2.E1.m1.1.1.2">𝑇</ci><apply id="S2.E1.m1.1.1.3.cmml" xref="S2.E1.m1.1.1.3"><plus id="S2.E1.m1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.3.1"></plus><apply id="S2.E1.m1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.3.2.1.cmml" xref="S2.E1.m1.1.1.3.2">subscript</csymbol><ci id="S2.E1.m1.1.1.3.2.2.cmml" xref="S2.E1.m1.1.1.3.2.2">𝑇</ci><ci id="S2.E1.m1.1.1.3.2.3.cmml" xref="S2.E1.m1.1.1.3.2.3">𝑀</ci></apply><apply id="S2.E1.m1.1.1.3.3.cmml" xref="S2.E1.m1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.3.3.1.cmml" xref="S2.E1.m1.1.1.3.3">subscript</csymbol><ci id="S2.E1.m1.1.1.3.3.2.cmml" xref="S2.E1.m1.1.1.3.3.2">𝑇</ci><ci id="S2.E1.m1.1.1.3.3.3.cmml" xref="S2.E1.m1.1.1.3.3.3">𝑃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">T=T_{M}+T_{P}</annotation><annotation encoding="application/x-llamapun" id="S2.E1.m1.1d">italic_T = italic_T start_POSTSUBSCRIPT italic_M end_POSTSUBSCRIPT + italic_T start_POSTSUBSCRIPT italic_P end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S2.SS2.p4"> Similarly, the daily scaled outage minutes (<math alttext="T_{S}" class="ltx_Math" display="inline" id="S2.SS2.p4.1.1.m1.1"><semantics id="S2.SS2.p4.1.1.m1.1a"><msub id="S2.SS2.p4.1.1.m1.1.1" xref="S2.SS2.p4.1.1.m1.1.1.cmml"><mi id="S2.SS2.p4.1.1.m1.1.1.2" xref="S2.SS2.p4.1.1.m1.1.1.2.cmml">T</mi><mi id="S2.SS2.p4.1.1.m1.1.1.3" xref="S2.SS2.p4.1.1.m1.1.1.3.cmml">S</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p4.1.1.m1.1b"><apply id="S2.SS2.p4.1.1.m1.1.1.cmml" xref="S2.SS2.p4.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p4.1.1.m1.1.1.1.cmml" xref="S2.SS2.p4.1.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.p4.1.1.m1.1.1.2.cmml" xref="S2.SS2.p4.1.1.m1.1.1.2">𝑇</ci><ci id="S2.SS2.p4.1.1.m1.1.1.3.cmml" xref="S2.SS2.p4.1.1.m1.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p4.1.1.m1.1c">T_{S}</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.p4.1.1.m1.1d">italic_T start_POSTSUBSCRIPT italic_S end_POSTSUBSCRIPT</annotation></semantics></math>) has the formula: <table class="ltx_equation ltx_eqn_table" id="S2.E2"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1">(2)</td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="T_{S}=T_{M}+(T_{P}\times 0.3)" class="ltx_Math" display="block" id="S2.E2.m1.1"><semantics id="S2.E2.m1.1a"><mrow id="S2.E2.m1.1.1" xref="S2.E2.m1.1.1.cmml"><msub id="S2.E2.m1.1.1.3" xref="S2.E2.m1.1.1.3.cmml"><mi id="S2.E2.m1.1.1.3.2" xref="S2.E2.m1.1.1.3.2.cmml">T</mi><mi id="S2.E2.m1.1.1.3.3" xref="S2.E2.m1.1.1.3.3.cmml">S</mi></msub><mo id="S2.E2.m1.1.1.2" xref="S2.E2.m1.1.1.2.cmml">=</mo><mrow id="S2.E2.m1.1.1.1" xref="S2.E2.m1.1.1.1.cmml"><msub id="S2.E2.m1.1.1.1.3" xref="S2.E2.m1.1.1.1.3.cmml"><mi id="S2.E2.m1.1.1.1.3.2" xref="S2.E2.m1.1.1.1.3.2.cmml">T</mi><mi id="S2.E2.m1.1.1.1.3.3" xref="S2.E2.m1.1.1.1.3.3.cmml">M</mi></msub><mo id="S2.E2.m1.1.1.1.2" xref="S2.E2.m1.1.1.1.2.cmml">+</mo><mrow id="S2.E2.m1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.cmml"><mo id="S2.E2.m1.1.1.1.1.1.2" stretchy="false" xref="S2.E2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E2.m1.1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.cmml"><msub id="S2.E2.m1.1.1.1.1.1.1.2" xref="S2.E2.m1.1.1.1.1.1.1.2.cmml"><mi id="S2.E2.m1.1.1.1.1.1.1.2.2" xref="S2.E2.m1.1.1.1.1.1.1.2.2.cmml">T</mi><mi id="S2.E2.m1.1.1.1.1.1.1.2.3" xref="S2.E2.m1.1.1.1.1.1.1.2.3.cmml">P</mi></msub><mo id="S2.E2.m1.1.1.1.1.1.1.1" lspace="0.222em" rspace="0.222em" xref="S2.E2.m1.1.1.1.1.1.1.1.cmml">×</mo><mn id="S2.E2.m1.1.1.1.1.1.1.3" xref="S2.E2.m1.1.1.1.1.1.1.3.cmml">0.3</mn></mrow><mo id="S2.E2.m1.1.1.1.1.1.3" stretchy="false" xref="S2.E2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.1b"><apply id="S2.E2.m1.1.1.cmml" xref="S2.E2.m1.1.1"><eq id="S2.E2.m1.1.1.2.cmml" xref="S2.E2.m1.1.1.2"></eq><apply id="S2.E2.m1.1.1.3.cmml" xref="S2.E2.m1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.3.1.cmml" xref="S2.E2.m1.1.1.3">subscript</csymbol><ci id="S2.E2.m1.1.1.3.2.cmml" xref="S2.E2.m1.1.1.3.2">𝑇</ci><ci id="S2.E2.m1.1.1.3.3.cmml" xref="S2.E2.m1.1.1.3.3">𝑆</ci></apply><apply id="S2.E2.m1.1.1.1.cmml" xref="S2.E2.m1.1.1.1"><plus id="S2.E2.m1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.2"></plus><apply id="S2.E2.m1.1.1.1.3.cmml" xref="S2.E2.m1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.3.1.cmml" xref="S2.E2.m1.1.1.1.3">subscript</csymbol><ci id="S2.E2.m1.1.1.1.3.2.cmml" xref="S2.E2.m1.1.1.1.3.2">𝑇</ci><ci id="S2.E2.m1.1.1.1.3.3.cmml" xref="S2.E2.m1.1.1.1.3.3">𝑀</ci></apply><apply id="S2.E2.m1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1"><times id="S2.E2.m1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1"></times><apply id="S2.E2.m1.1.1.1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E2.m1.1.1.1.1.1.1.2.2.cmml" xref="S2.E2.m1.1.1.1.1.1.1.2.2">𝑇</ci><ci id="S2.E2.m1.1.1.1.1.1.1.2.3.cmml" xref="S2.E2.m1.1.1.1.1.1.1.2.3">𝑃</ci></apply><cn id="S2.E2.m1.1.1.1.1.1.1.3.cmml" type="float" xref="S2.E2.m1.1.1.1.1.1.1.3">0.3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.1c">T_{S}=T_{M}+(T_{P}\times 0.3)</annotation><annotation encoding="application/x-llamapun" id="S2.E2.m1.1d">italic_T start_POSTSUBSCRIPT italic_S end_POSTSUBSCRIPT = italic_T start_POSTSUBSCRIPT italic_M end_POSTSUBSCRIPT + ( italic_T start_POSTSUBSCRIPT italic_P end_POSTSUBSCRIPT × 0.3 )</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <div class="ltx_para" id="S2.SS2.p5"> Organizations such as OpenAI use primarily the scaled outage minutes to assess and report outage impact <cite class="ltx_cite ltx_citemacro_citep">(Atlassian, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib9" title="">2024</a>)</cite>. </div> <div class="ltx_para" id="S2.SS2.p6"> Availability: Derived from the daily scaled outage minutes, we define the daily availability, <math alttext="A" class="ltx_Math" display="inline" id="S2.SS2.p6.1.m1.1"><semantics id="S2.SS2.p6.1.m1.1a"><mi id="S2.SS2.p6.1.m1.1.1" xref="S2.SS2.p6.1.m1.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.1.m1.1b"><ci id="S2.SS2.p6.1.m1.1.1.cmml" xref="S2.SS2.p6.1.m1.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.1.m1.1c">A</annotation><annotation encoding="application/x-llamapun" id="S2.SS2.p6.1.m1.1d">italic_A</annotation></semantics></math>, as the percentage of time a service or a group of services are available, given by the formula: <table class="ltx_equation ltx_eqn_table" id="S2.E3"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1">(3)</td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="A=(1-\frac{T_{S}}{24\times 60})\times 100\%" class="ltx_Math" display="block" id="S2.E3.m1.1"><semantics id="S2.E3.m1.1a"><mrow id="S2.E3.m1.1.1" xref="S2.E3.m1.1.1.cmml"><mi id="S2.E3.m1.1.1.3" xref="S2.E3.m1.1.1.3.cmml">A</mi><mo id="S2.E3.m1.1.1.2" xref="S2.E3.m1.1.1.2.cmml">=</mo><mrow id="S2.E3.m1.1.1.1" xref="S2.E3.m1.1.1.1.cmml"><mrow id="S2.E3.m1.1.1.1.1.1" xref="S2.E3.m1.1.1.1.1.1.1.cmml"><mo id="S2.E3.m1.1.1.1.1.1.2" stretchy="false" xref="S2.E3.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E3.m1.1.1.1.1.1.1" xref="S2.E3.m1.1.1.1.1.1.1.cmml"><mn id="S2.E3.m1.1.1.1.1.1.1.2" xref="S2.E3.m1.1.1.1.1.1.1.2.cmml">1</mn><mo id="S2.E3.m1.1.1.1.1.1.1.1" xref="S2.E3.m1.1.1.1.1.1.1.1.cmml">−</mo><mfrac id="S2.E3.m1.1.1.1.1.1.1.3" xref="S2.E3.m1.1.1.1.1.1.1.3.cmml"><msub id="S2.E3.m1.1.1.1.1.1.1.3.2" xref="S2.E3.m1.1.1.1.1.1.1.3.2.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.3.2.2" xref="S2.E3.m1.1.1.1.1.1.1.3.2.2.cmml">T</mi><mi id="S2.E3.m1.1.1.1.1.1.1.3.2.3" xref="S2.E3.m1.1.1.1.1.1.1.3.2.3.cmml">S</mi></msub><mrow id="S2.E3.m1.1.1.1.1.1.1.3.3" xref="S2.E3.m1.1.1.1.1.1.1.3.3.cmml"><mn id="S2.E3.m1.1.1.1.1.1.1.3.3.2" xref="S2.E3.m1.1.1.1.1.1.1.3.3.2.cmml">24</mn><mo id="S2.E3.m1.1.1.1.1.1.1.3.3.1" lspace="0.222em" rspace="0.222em" xref="S2.E3.m1.1.1.1.1.1.1.3.3.1.cmml">×</mo><mn id="S2.E3.m1.1.1.1.1.1.1.3.3.3" xref="S2.E3.m1.1.1.1.1.1.1.3.3.3.cmml">60</mn></mrow></mfrac></mrow><mo id="S2.E3.m1.1.1.1.1.1.3" rspace="0.055em" stretchy="false" xref="S2.E3.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo id="S2.E3.m1.1.1.1.2" rspace="0.222em" xref="S2.E3.m1.1.1.1.2.cmml">×</mo><mrow id="S2.E3.m1.1.1.1.3" xref="S2.E3.m1.1.1.1.3.cmml"><mn id="S2.E3.m1.1.1.1.3.2" xref="S2.E3.m1.1.1.1.3.2.cmml">100</mn><mo id="S2.E3.m1.1.1.1.3.1" xref="S2.E3.m1.1.1.1.3.1.cmml">%</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E3.m1.1b"><apply id="S2.E3.m1.1.1.cmml" xref="S2.E3.m1.1.1"><eq id="S2.E3.m1.1.1.2.cmml" xref="S2.E3.m1.1.1.2"></eq><ci id="S2.E3.m1.1.1.3.cmml" xref="S2.E3.m1.1.1.3">𝐴</ci><apply id="S2.E3.m1.1.1.1.cmml" xref="S2.E3.m1.1.1.1"><times id="S2.E3.m1.1.1.1.2.cmml" xref="S2.E3.m1.1.1.1.2"></times><apply id="S2.E3.m1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1"><minus id="S2.E3.m1.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.1"></minus><cn id="S2.E3.m1.1.1.1.1.1.1.2.cmml" type="integer" xref="S2.E3.m1.1.1.1.1.1.1.2">1</cn><apply id="S2.E3.m1.1.1.1.1.1.1.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3"><divide id="S2.E3.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3"></divide><apply id="S2.E3.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3.2.2">𝑇</ci><ci id="S2.E3.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3.2.3">𝑆</ci></apply><apply id="S2.E3.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3.3"><times id="S2.E3.m1.1.1.1.1.1.1.3.3.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3.3.1"></times><cn id="S2.E3.m1.1.1.1.1.1.1.3.3.2.cmml" type="integer" xref="S2.E3.m1.1.1.1.1.1.1.3.3.2">24</cn><cn id="S2.E3.m1.1.1.1.1.1.1.3.3.3.cmml" type="integer" xref="S2.E3.m1.1.1.1.1.1.1.3.3.3">60</cn></apply></apply></apply><apply id="S2.E3.m1.1.1.1.3.cmml" xref="S2.E3.m1.1.1.1.3"><csymbol cd="latexml" id="S2.E3.m1.1.1.1.3.1.cmml" xref="S2.E3.m1.1.1.1.3.1">percent</csymbol><cn id="S2.E3.m1.1.1.1.3.2.cmml" type="integer" xref="S2.E3.m1.1.1.1.3.2">100</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3.m1.1c">A=(1-\frac{T_{S}}{24\times 60})\times 100\%</annotation><annotation encoding="application/x-llamapun" id="S2.E3.m1.1d">italic_A = ( 1 - divide start_ARG italic_T start_POSTSUBSCRIPT italic_S end_POSTSUBSCRIPT end_ARG start_ARG 24 × 60 end_ARG ) × 100 %</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> </div> <figure class="ltx_table" id="S2.T3"> <figcaption class="ltx_caption ltx_centering">Table 3. Summary of LLM outages, per service. Legend: Incident Count <math alttext="=" class="ltx_Math" display="inline" id="S2.T3.2.1.m1.1"><semantics id="S2.T3.2.1.m1.1b"><mo id="S2.T3.2.1.m1.1.1" xref="S2.T3.2.1.m1.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S2.T3.2.1.m1.1c"><eq id="S2.T3.2.1.m1.1.1.cmml" xref="S2.T3.2.1.m1.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.2.1.m1.1d">=</annotation><annotation encoding="application/x-llamapun" id="S2.T3.2.1.m1.1e">=</annotation></semantics></math> the number of related incidents.</figcaption> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S2.T3.10" style="width:433.6pt;height:136.5pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_align_middle" id="S2.T3.10.8"> <tr class="ltx_tr" id="S2.T3.10.8.9"> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T3.10.8.9.1" rowspan="2">ID</td> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T3.10.8.9.2" rowspan="2">Service</td> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T3.10.8.9.3" rowspan="2">Provider</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T3.10.8.9.4" rowspan="2">Start</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T3.10.8.9.5" rowspan="2">End</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T3.10.8.9.6" rowspan="2">Months</td> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_tt" colspan="3" id="S2.T3.10.8.9.7">Outage Count</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="2" id="S2.T3.10.8.9.8">Outage Minutes</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T3.10.8.9.9">Incident</td> </tr> <tr class="ltx_tr" id="S2.T3.10.8.10"> <td class="ltx_td ltx_align_right ltx_border_l ltx_border_t" id="S2.T3.10.8.10.1">Total</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T3.10.8.10.2">Major</td> <td class="ltx_td ltx_align_right ltx_border_r ltx_border_t" id="S2.T3.10.8.10.3">Partial</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T3.10.8.10.4">Total</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_t" id="S2.T3.10.8.10.5">Scaled</td> <td class="ltx_td ltx_align_right" id="S2.T3.10.8.10.6">Count</td> </tr> <tr class="ltx_tr" id="S2.T3.3.1.1"> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T3.3.1.1.1"><math alttext="O_{1}" class="ltx_Math" display="inline" id="S2.T3.3.1.1.1.m1.1"><semantics id="S2.T3.3.1.1.1.m1.1a"><msub id="S2.T3.3.1.1.1.m1.1.1" xref="S2.T3.3.1.1.1.m1.1.1.cmml"><mi id="S2.T3.3.1.1.1.m1.1.1.2" xref="S2.T3.3.1.1.1.m1.1.1.2.cmml">O</mi><mn id="S2.T3.3.1.1.1.m1.1.1.3" xref="S2.T3.3.1.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T3.3.1.1.1.m1.1b"><apply id="S2.T3.3.1.1.1.m1.1.1.cmml" xref="S2.T3.3.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T3.3.1.1.1.m1.1.1.1.cmml" xref="S2.T3.3.1.1.1.m1.1.1">subscript</csymbol><ci id="S2.T3.3.1.1.1.m1.1.1.2.cmml" xref="S2.T3.3.1.1.1.m1.1.1.2">𝑂</ci><cn id="S2.T3.3.1.1.1.m1.1.1.3.cmml" type="integer" xref="S2.T3.3.1.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.3.1.1.1.m1.1c">O_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T3.3.1.1.1.m1.1d">italic_O start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T3.3.1.1.2">API</td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T3.3.1.1.3" rowspan="4">OpenAI</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S2.T3.3.1.1.4">2021-02-11</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S2.T3.3.1.1.5">2024-08-31</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S2.T3.3.1.1.6">43</td> <td class="ltx_td ltx_align_right ltx_border_l ltx_border_t" id="S2.T3.3.1.1.7">104</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T3.3.1.1.8">26</td> <td class="ltx_td ltx_align_right ltx_border_r ltx_border_t" id="S2.T3.3.1.1.9">78</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T3.3.1.1.10">7,891</td> <td class="ltx_td ltx_align_right ltx_border_r ltx_border_t" id="S2.T3.3.1.1.11">3,340</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T3.3.1.1.12">242</td> </tr> <tr class="ltx_tr" id="S2.T3.4.2.2"> <td class="ltx_td ltx_align_left" id="S2.T3.4.2.2.1"><math alttext="O_{2}" class="ltx_Math" display="inline" id="S2.T3.4.2.2.1.m1.1"><semantics id="S2.T3.4.2.2.1.m1.1a"><msub id="S2.T3.4.2.2.1.m1.1.1" xref="S2.T3.4.2.2.1.m1.1.1.cmml"><mi id="S2.T3.4.2.2.1.m1.1.1.2" xref="S2.T3.4.2.2.1.m1.1.1.2.cmml">O</mi><mn id="S2.T3.4.2.2.1.m1.1.1.3" xref="S2.T3.4.2.2.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T3.4.2.2.1.m1.1b"><apply id="S2.T3.4.2.2.1.m1.1.1.cmml" xref="S2.T3.4.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T3.4.2.2.1.m1.1.1.1.cmml" xref="S2.T3.4.2.2.1.m1.1.1">subscript</csymbol><ci id="S2.T3.4.2.2.1.m1.1.1.2.cmml" xref="S2.T3.4.2.2.1.m1.1.1.2">𝑂</ci><cn id="S2.T3.4.2.2.1.m1.1.1.3.cmml" type="integer" xref="S2.T3.4.2.2.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.4.2.2.1.m1.1c">O_{2}</annotation><annotation encoding="application/x-llamapun" id="S2.T3.4.2.2.1.m1.1d">italic_O start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T3.4.2.2.2">ChatGPT</td> <td class="ltx_td ltx_align_center" id="S2.T3.4.2.2.3">2023-02-14</td> <td class="ltx_td ltx_align_center" id="S2.T3.4.2.2.4">2024-08-31</td> <td class="ltx_td ltx_align_center" id="S2.T3.4.2.2.5">19</td> <td class="ltx_td ltx_align_right ltx_border_l" id="S2.T3.4.2.2.6">70</td> <td class="ltx_td ltx_align_right" id="S2.T3.4.2.2.7">28</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.4.2.2.8">42</td> <td class="ltx_td ltx_align_right" id="S2.T3.4.2.2.9">5,185</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.4.2.2.10">2,744</td> <td class="ltx_td ltx_align_right" id="S2.T3.4.2.2.11">157</td> </tr> <tr class="ltx_tr" id="S2.T3.5.3.3"> <td class="ltx_td ltx_align_left" id="S2.T3.5.3.3.1"><math alttext="O_{3}" class="ltx_Math" display="inline" id="S2.T3.5.3.3.1.m1.1"><semantics id="S2.T3.5.3.3.1.m1.1a"><msub id="S2.T3.5.3.3.1.m1.1.1" xref="S2.T3.5.3.3.1.m1.1.1.cmml"><mi id="S2.T3.5.3.3.1.m1.1.1.2" xref="S2.T3.5.3.3.1.m1.1.1.2.cmml">O</mi><mn id="S2.T3.5.3.3.1.m1.1.1.3" xref="S2.T3.5.3.3.1.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T3.5.3.3.1.m1.1b"><apply id="S2.T3.5.3.3.1.m1.1.1.cmml" xref="S2.T3.5.3.3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T3.5.3.3.1.m1.1.1.1.cmml" xref="S2.T3.5.3.3.1.m1.1.1">subscript</csymbol><ci id="S2.T3.5.3.3.1.m1.1.1.2.cmml" xref="S2.T3.5.3.3.1.m1.1.1.2">𝑂</ci><cn id="S2.T3.5.3.3.1.m1.1.1.3.cmml" type="integer" xref="S2.T3.5.3.3.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.5.3.3.1.m1.1c">O_{3}</annotation><annotation encoding="application/x-llamapun" id="S2.T3.5.3.3.1.m1.1d">italic_O start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T3.5.3.3.2">DALL·E</td> <td class="ltx_td ltx_align_center" id="S2.T3.5.3.3.3">2023-02-21</td> <td class="ltx_td ltx_align_center" id="S2.T3.5.3.3.4">2024-08-31</td> <td class="ltx_td ltx_align_center" id="S2.T3.5.3.3.5">19</td> <td class="ltx_td ltx_align_right ltx_border_l" id="S2.T3.5.3.3.6">27</td> <td class="ltx_td ltx_align_right" id="S2.T3.5.3.3.7">13</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.5.3.3.8">14</td> <td class="ltx_td ltx_align_right" id="S2.T3.5.3.3.9">2,821</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.5.3.3.10">1,748</td> <td class="ltx_td ltx_align_right" id="S2.T3.5.3.3.11">34</td> </tr> <tr class="ltx_tr" id="S2.T3.6.4.4"> <td class="ltx_td ltx_align_left" id="S2.T3.6.4.4.1"><math alttext="O_{4}" class="ltx_Math" display="inline" id="S2.T3.6.4.4.1.m1.1"><semantics id="S2.T3.6.4.4.1.m1.1a"><msub id="S2.T3.6.4.4.1.m1.1.1" xref="S2.T3.6.4.4.1.m1.1.1.cmml"><mi id="S2.T3.6.4.4.1.m1.1.1.2" xref="S2.T3.6.4.4.1.m1.1.1.2.cmml">O</mi><mn id="S2.T3.6.4.4.1.m1.1.1.3" xref="S2.T3.6.4.4.1.m1.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T3.6.4.4.1.m1.1b"><apply id="S2.T3.6.4.4.1.m1.1.1.cmml" xref="S2.T3.6.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T3.6.4.4.1.m1.1.1.1.cmml" xref="S2.T3.6.4.4.1.m1.1.1">subscript</csymbol><ci id="S2.T3.6.4.4.1.m1.1.1.2.cmml" xref="S2.T3.6.4.4.1.m1.1.1.2">𝑂</ci><cn id="S2.T3.6.4.4.1.m1.1.1.3.cmml" type="integer" xref="S2.T3.6.4.4.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.6.4.4.1.m1.1c">O_{4}</annotation><annotation encoding="application/x-llamapun" id="S2.T3.6.4.4.1.m1.1d">italic_O start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T3.6.4.4.2">Playground</td> <td class="ltx_td ltx_align_center" id="S2.T3.6.4.4.3">2021-03-31</td> <td class="ltx_td ltx_align_center" id="S2.T3.6.4.4.4">2024-08-31</td> <td class="ltx_td ltx_align_center" id="S2.T3.6.4.4.5">42</td> <td class="ltx_td ltx_align_right ltx_border_l" id="S2.T3.6.4.4.6">24</td> <td class="ltx_td ltx_align_right" id="S2.T3.6.4.4.7">12</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.6.4.4.8">12</td> <td class="ltx_td ltx_align_right" id="S2.T3.6.4.4.9">1,636</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.6.4.4.10">1,018</td> <td class="ltx_td ltx_align_right" id="S2.T3.6.4.4.11">36</td> </tr> <tr class="ltx_tr" id="S2.T3.7.5.5"> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T3.7.5.5.1"><math alttext="A_{1}" class="ltx_Math" display="inline" id="S2.T3.7.5.5.1.m1.1"><semantics id="S2.T3.7.5.5.1.m1.1a"><msub id="S2.T3.7.5.5.1.m1.1.1" xref="S2.T3.7.5.5.1.m1.1.1.cmml"><mi id="S2.T3.7.5.5.1.m1.1.1.2" xref="S2.T3.7.5.5.1.m1.1.1.2.cmml">A</mi><mn id="S2.T3.7.5.5.1.m1.1.1.3" xref="S2.T3.7.5.5.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T3.7.5.5.1.m1.1b"><apply id="S2.T3.7.5.5.1.m1.1.1.cmml" xref="S2.T3.7.5.5.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T3.7.5.5.1.m1.1.1.1.cmml" xref="S2.T3.7.5.5.1.m1.1.1">subscript</csymbol><ci id="S2.T3.7.5.5.1.m1.1.1.2.cmml" xref="S2.T3.7.5.5.1.m1.1.1.2">𝐴</ci><cn id="S2.T3.7.5.5.1.m1.1.1.3.cmml" type="integer" xref="S2.T3.7.5.5.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.7.5.5.1.m1.1c">A_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T3.7.5.5.1.m1.1d">italic_A start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T3.7.5.5.2">API</td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T3.7.5.5.3" rowspan="3">Anthropic</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S2.T3.7.5.5.4">2023-07-11</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S2.T3.7.5.5.5">2024-08-31</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S2.T3.7.5.5.6">14</td> <td class="ltx_td ltx_align_right ltx_border_l ltx_border_t" id="S2.T3.7.5.5.7">25</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T3.7.5.5.8">0</td> <td class="ltx_td ltx_align_right ltx_border_r ltx_border_t" id="S2.T3.7.5.5.9">25</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T3.7.5.5.10">1,675</td> <td class="ltx_td ltx_align_right ltx_border_r ltx_border_t" id="S2.T3.7.5.5.11">502</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T3.7.5.5.12">80</td> </tr> <tr class="ltx_tr" id="S2.T3.8.6.6"> <td class="ltx_td ltx_align_left" id="S2.T3.8.6.6.1"><math alttext="A_{2}" class="ltx_Math" display="inline" id="S2.T3.8.6.6.1.m1.1"><semantics id="S2.T3.8.6.6.1.m1.1a"><msub id="S2.T3.8.6.6.1.m1.1.1" xref="S2.T3.8.6.6.1.m1.1.1.cmml"><mi id="S2.T3.8.6.6.1.m1.1.1.2" xref="S2.T3.8.6.6.1.m1.1.1.2.cmml">A</mi><mn id="S2.T3.8.6.6.1.m1.1.1.3" xref="S2.T3.8.6.6.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T3.8.6.6.1.m1.1b"><apply id="S2.T3.8.6.6.1.m1.1.1.cmml" xref="S2.T3.8.6.6.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T3.8.6.6.1.m1.1.1.1.cmml" xref="S2.T3.8.6.6.1.m1.1.1">subscript</csymbol><ci id="S2.T3.8.6.6.1.m1.1.1.2.cmml" xref="S2.T3.8.6.6.1.m1.1.1.2">𝐴</ci><cn id="S2.T3.8.6.6.1.m1.1.1.3.cmml" type="integer" xref="S2.T3.8.6.6.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.8.6.6.1.m1.1c">A_{2}</annotation><annotation encoding="application/x-llamapun" id="S2.T3.8.6.6.1.m1.1d">italic_A start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T3.8.6.6.2">Claude</td> <td class="ltx_td ltx_align_center" id="S2.T3.8.6.6.3">2023-07-11</td> <td class="ltx_td ltx_align_center" id="S2.T3.8.6.6.4">2024-08-31</td> <td class="ltx_td ltx_align_center" id="S2.T3.8.6.6.5">14</td> <td class="ltx_td ltx_align_right ltx_border_l" id="S2.T3.8.6.6.6">30</td> <td class="ltx_td ltx_align_right" id="S2.T3.8.6.6.7">2</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.8.6.6.8">28</td> <td class="ltx_td ltx_align_right" id="S2.T3.8.6.6.9">3,017</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.8.6.6.10">983</td> <td class="ltx_td ltx_align_right" id="S2.T3.8.6.6.11">90</td> </tr> <tr class="ltx_tr" id="S2.T3.9.7.7"> <td class="ltx_td ltx_align_left" id="S2.T3.9.7.7.1"><math alttext="A_{3}" class="ltx_Math" display="inline" id="S2.T3.9.7.7.1.m1.1"><semantics id="S2.T3.9.7.7.1.m1.1a"><msub id="S2.T3.9.7.7.1.m1.1.1" xref="S2.T3.9.7.7.1.m1.1.1.cmml"><mi id="S2.T3.9.7.7.1.m1.1.1.2" xref="S2.T3.9.7.7.1.m1.1.1.2.cmml">A</mi><mn id="S2.T3.9.7.7.1.m1.1.1.3" xref="S2.T3.9.7.7.1.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T3.9.7.7.1.m1.1b"><apply id="S2.T3.9.7.7.1.m1.1.1.cmml" xref="S2.T3.9.7.7.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T3.9.7.7.1.m1.1.1.1.cmml" xref="S2.T3.9.7.7.1.m1.1.1">subscript</csymbol><ci id="S2.T3.9.7.7.1.m1.1.1.2.cmml" xref="S2.T3.9.7.7.1.m1.1.1.2">𝐴</ci><cn id="S2.T3.9.7.7.1.m1.1.1.3.cmml" type="integer" xref="S2.T3.9.7.7.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.9.7.7.1.m1.1c">A_{3}</annotation><annotation encoding="application/x-llamapun" id="S2.T3.9.7.7.1.m1.1d">italic_A start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T3.9.7.7.2">Console</td> <td class="ltx_td ltx_align_center" id="S2.T3.9.7.7.3">2023-07-11</td> <td class="ltx_td ltx_align_center" id="S2.T3.9.7.7.4">2024-08-31</td> <td class="ltx_td ltx_align_center" id="S2.T3.9.7.7.5">14</td> <td class="ltx_td ltx_align_right ltx_border_l" id="S2.T3.9.7.7.6">27</td> <td class="ltx_td ltx_align_right" id="S2.T3.9.7.7.7">1</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.9.7.7.8">26</td> <td class="ltx_td ltx_align_right" id="S2.T3.9.7.7.9">2,032</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T3.9.7.7.10">662</td> <td class="ltx_td ltx_align_right" id="S2.T3.9.7.7.11">72</td> </tr> <tr class="ltx_tr" id="S2.T3.10.8.8"> <td class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.1"><math alttext="C_{1}" class="ltx_Math" display="inline" id="S2.T3.10.8.8.1.m1.1"><semantics id="S2.T3.10.8.8.1.m1.1a"><msub id="S2.T3.10.8.8.1.m1.1.1" xref="S2.T3.10.8.8.1.m1.1.1.cmml"><mi id="S2.T3.10.8.8.1.m1.1.1.2" xref="S2.T3.10.8.8.1.m1.1.1.2.cmml">C</mi><mn id="S2.T3.10.8.8.1.m1.1.1.3" xref="S2.T3.10.8.8.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T3.10.8.8.1.m1.1b"><apply id="S2.T3.10.8.8.1.m1.1.1.cmml" xref="S2.T3.10.8.8.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T3.10.8.8.1.m1.1.1.1.cmml" xref="S2.T3.10.8.8.1.m1.1.1">subscript</csymbol><ci id="S2.T3.10.8.8.1.m1.1.1.2.cmml" xref="S2.T3.10.8.8.1.m1.1.1.2">𝐶</ci><cn id="S2.T3.10.8.8.1.m1.1.1.3.cmml" type="integer" xref="S2.T3.10.8.8.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T3.10.8.8.1.m1.1c">C_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T3.10.8.8.1.m1.1d">italic_C start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.2">Character.AI</td> <td class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.3">Character.AI</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.4">2023-10-19</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.5">2024-08-31</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.6">11</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_l ltx_border_t" id="S2.T3.10.8.8.7">32</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.8">17</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_r ltx_border_t" id="S2.T3.10.8.8.9">15</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.10">3,351</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_r ltx_border_t" id="S2.T3.10.8.8.11">1,878</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S2.T3.10.8.8.12">41</td> </tr> </table> </div> </figure> <figure class="ltx_table" id="S2.T4"> <figcaption class="ltx_caption ltx_centering">Table 4. Summary of LLM incident reports, per service. Legend: Maint. = Maintenance; Inv. = Investigating; PM = Postmortem.</figcaption> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S2.T4.3" style="width:433.6pt;height:57.5pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_align_middle" id="S2.T4.3.3"> <tr class="ltx_tr" id="S2.T4.3.3.4"> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T4.3.3.4.1" rowspan="2">ID</td> <td class="ltx_td ltx_align_left ltx_border_tt" id="S2.T4.3.3.4.2" rowspan="2">Provider</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T4.3.3.4.3" rowspan="2">First Date</td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S2.T4.3.3.4.4" rowspan="2">Last Date</td> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_tt" id="S2.T4.3.3.4.5"># of</td> <td class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="5" id="S2.T4.3.3.4.6"># of Impact Levels</td> <td class="ltx_td ltx_align_center ltx_border_tt" colspan="5" id="S2.T4.3.3.4.7"># of Failure-Recovery Status</td> </tr> <tr class="ltx_tr" id="S2.T4.3.3.5"> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r" id="S2.T4.3.3.5.1">Reports</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.2">Critical</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.3">Major</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.4">Minor</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.5">None</td> <td class="ltx_td ltx_align_right ltx_border_r ltx_border_t" id="S2.T4.3.3.5.6">Maint.</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.7">Inv.</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.8">Identified</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.9">Monitoring</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.10">Resolved</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.3.3.5.11">PM</td> </tr> <tr class="ltx_tr" id="S2.T4.1.1.1"> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T4.1.1.1.1"><math alttext="P_{1}" class="ltx_Math" display="inline" id="S2.T4.1.1.1.1.m1.1"><semantics id="S2.T4.1.1.1.1.m1.1a"><msub id="S2.T4.1.1.1.1.m1.1.1" xref="S2.T4.1.1.1.1.m1.1.1.cmml"><mi id="S2.T4.1.1.1.1.m1.1.1.2" xref="S2.T4.1.1.1.1.m1.1.1.2.cmml">P</mi><mn id="S2.T4.1.1.1.1.m1.1.1.3" xref="S2.T4.1.1.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T4.1.1.1.1.m1.1b"><apply id="S2.T4.1.1.1.1.m1.1.1.cmml" xref="S2.T4.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T4.1.1.1.1.m1.1.1.1.cmml" xref="S2.T4.1.1.1.1.m1.1.1">subscript</csymbol><ci id="S2.T4.1.1.1.1.m1.1.1.2.cmml" xref="S2.T4.1.1.1.1.m1.1.1.2">𝑃</ci><cn id="S2.T4.1.1.1.1.m1.1.1.3.cmml" type="integer" xref="S2.T4.1.1.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T4.1.1.1.1.m1.1c">P_{1}</annotation><annotation encoding="application/x-llamapun" id="S2.T4.1.1.1.1.m1.1d">italic_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S2.T4.1.1.1.2">OpenAI</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S2.T4.1.1.1.3">2021-02-09</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S2.T4.1.1.1.4">2024-08-28</td> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" id="S2.T4.1.1.1.5">365</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.6">46</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.7">125</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.8">141</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.9">52</td> <td class="ltx_td ltx_align_right ltx_border_r ltx_border_t" id="S2.T4.1.1.1.10">1</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.11">259</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.12">144</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.13">225</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.14">365</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S2.T4.1.1.1.15">29</td> </tr> <tr class="ltx_tr" id="S2.T4.2.2.2"> <td class="ltx_td ltx_align_left" id="S2.T4.2.2.2.1"><math alttext="P_{2}" class="ltx_Math" display="inline" id="S2.T4.2.2.2.1.m1.1"><semantics id="S2.T4.2.2.2.1.m1.1a"><msub id="S2.T4.2.2.2.1.m1.1.1" xref="S2.T4.2.2.2.1.m1.1.1.cmml"><mi id="S2.T4.2.2.2.1.m1.1.1.2" xref="S2.T4.2.2.2.1.m1.1.1.2.cmml">P</mi><mn id="S2.T4.2.2.2.1.m1.1.1.3" xref="S2.T4.2.2.2.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T4.2.2.2.1.m1.1b"><apply id="S2.T4.2.2.2.1.m1.1.1.cmml" xref="S2.T4.2.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T4.2.2.2.1.m1.1.1.1.cmml" xref="S2.T4.2.2.2.1.m1.1.1">subscript</csymbol><ci id="S2.T4.2.2.2.1.m1.1.1.2.cmml" xref="S2.T4.2.2.2.1.m1.1.1.2">𝑃</ci><cn id="S2.T4.2.2.2.1.m1.1.1.3.cmml" type="integer" xref="S2.T4.2.2.2.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T4.2.2.2.1.m1.1c">P_{2}</annotation><annotation encoding="application/x-llamapun" id="S2.T4.2.2.2.1.m1.1d">italic_P start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S2.T4.2.2.2.2">Anthropic</td> <td class="ltx_td ltx_align_center" id="S2.T4.2.2.2.3">2023-03-25</td> <td class="ltx_td ltx_align_center" id="S2.T4.2.2.2.4">2024-08-30</td> <td class="ltx_td ltx_align_center ltx_border_l ltx_border_r" id="S2.T4.2.2.2.5">141</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.6">5</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.7">43</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.8">48</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.9">44</td> <td class="ltx_td ltx_align_right ltx_border_r" id="S2.T4.2.2.2.10">1</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.11">96</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.12">45</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.13">51</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.14">141</td> <td class="ltx_td ltx_align_right" id="S2.T4.2.2.2.15">2</td> </tr> <tr class="ltx_tr" id="S2.T4.3.3.3"> <td class="ltx_td ltx_align_left ltx_border_bb" id="S2.T4.3.3.3.1"><math alttext="P_{3}" class="ltx_Math" display="inline" id="S2.T4.3.3.3.1.m1.1"><semantics id="S2.T4.3.3.3.1.m1.1a"><msub id="S2.T4.3.3.3.1.m1.1.1" xref="S2.T4.3.3.3.1.m1.1.1.cmml"><mi id="S2.T4.3.3.3.1.m1.1.1.2" xref="S2.T4.3.3.3.1.m1.1.1.2.cmml">P</mi><mn id="S2.T4.3.3.3.1.m1.1.1.3" xref="S2.T4.3.3.3.1.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T4.3.3.3.1.m1.1b"><apply id="S2.T4.3.3.3.1.m1.1.1.cmml" xref="S2.T4.3.3.3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T4.3.3.3.1.m1.1.1.1.cmml" xref="S2.T4.3.3.3.1.m1.1.1">subscript</csymbol><ci id="S2.T4.3.3.3.1.m1.1.1.2.cmml" xref="S2.T4.3.3.3.1.m1.1.1.2">𝑃</ci><cn id="S2.T4.3.3.3.1.m1.1.1.3.cmml" type="integer" xref="S2.T4.3.3.3.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T4.3.3.3.1.m1.1c">P_{3}</annotation><annotation encoding="application/x-llamapun" id="S2.T4.3.3.3.1.m1.1d">italic_P start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_bb" id="S2.T4.3.3.3.2">Character.AI</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S2.T4.3.3.3.3">2023-10-24</td> <td class="ltx_td ltx_align_center ltx_border_bb" id="S2.T4.3.3.3.4">2024-08-07</td> <td class="ltx_td ltx_align_center ltx_border_bb ltx_border_l ltx_border_r" id="S2.T4.3.3.3.5"> 36</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.6">19</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.7">11</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.8">4</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.9">2</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" id="S2.T4.3.3.3.10">0</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.11">26</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.12">16</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.13">15</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.14">36</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S2.T4.3.3.3.15">2</td> </tr> </table> </div> </figure> </section> </section> <section class="ltx_section" id="S3"> <h2 class="ltx_title ltx_title_section"> 3. Dataset Collection and Preparation</h2> <div class="ltx_para" id="S3.p1"> We collect for this research long-term datasets from 8 LLM services across 3 LLM service providers. We then process these datasets to prepare data useful to characterize LLM service outages and incidents. Tables <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.T3" title="Table 3 ‣ 2.2. LLM-Specific Terms and Metrics ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">3</a> and <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.T4" title="Table 4 ‣ 2.2. LLM-Specific Terms and Metrics ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">4</a> summarize the processed outage and incident datasets, respectively. </div> <section class="ltx_subsection" id="S3.SS1"> <h3 class="ltx_title ltx_title_subsection"> 3.1. Selection and Introduction of LLM Services</h3> <div class="ltx_para" id="S3.SS1.p1"> Selection process: Addressing the main challenge of lacking longitudinal failure data about LLM services, particularly under the same failure model, we carefully investigate the current LLM services, and select 8 LLM services from 3 service providers based on the following reasons: (1) Data availability: Selected services should have public status pages running for long durations, so our data collection can provide rich datasets for the community to further analyze. (2) Popularity: Selected services should be popular, with many users and applications with daily use, so the impact of outages is significant, and there is high likelihood users and media will also report on such outages if left unattended. This pressures operators to respond quickly, so the data we collect represents the best performance LLM operators can currently deliver. (3) Diversity: Selected services should cover most types of LLM services provided by different companies. This will ensure the generality of our results. </div> <div class="ltx_para" id="S3.SS1.p2"> Selected LLM services. (1) OpenAI API: The OpenAI API allows developers to access and use advanced LLM models provided by OpenAI through API keys without building or training from scratch. (2) ChatGPT: ChatGPT is a chatbot that interacts with users conversationally. ChatGPT can answer follow-up questions with prompts and provide a detailed response. (3) Labs (DALL·E): DALL·E is a text-to-image model that can create original, realistic images from a short text description. (4) Playground: Playground is a web-based interface for users to interact with and experiment with OpenAI’s language models. (5) Anthropic API: Similar to OpenAI API, Anthropic API allows developers to integrate language models such as Claude, into their applications and services. (6) Claude Similar to OpenAI’s ChatGPT, Claude is an AI chatbot and is trained to have natural, text-based conversations with users. (7) Console: Similar to the OpenAI’s playground, the Anthropic Console is a web-based interface that allows users to interact with Anthropic’s AI models directly. (8) Character.AI: is an innovative chatbot platform that leverages LLMs to facilitate a series of chatbots that emulate the personas of various figures, such as historical icons, fictional heroes, modern celebrities, etc. </div> </section> <section class="ltx_subsection" id="S3.SS2"> <h3 class="ltx_title ltx_title_subsection"> 3.2. Data Collection and Dataset Preparation</h3> <div class="ltx_para" id="S3.SS2.p1"> We collected all available outage and incident data reported publicly by of OpenAI, Anthropic, and Character.AI, up to 2024-08-31, on their public status pages <cite class="ltx_cite ltx_citemacro_citep">(Uptime, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib59" title="">2024c</a>, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib57" title="">a</a>, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib58" title="">b</a>)</cite> and incident pages <cite class="ltx_cite ltx_citemacro_citep">(Incident, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib30" title="">2024c</a>, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib28" title="">a</a>, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib29" title="">b</a>)</cite>. The starting dates differ: OpenAI has started reporting on February 11, Anthropic on July 11, and Character.AI on October 19, all dates in 2023. Our study misses none of the published reports. </div> <div class="ltx_para" id="S3.SS2.p2"> The industry has standardized presenting outage data in a calendar format, with separate information for each service. Each outage history page displays a 3-month calendar view. By hovering over the calendar, one can reveal detailed information about outages, including the occurrence and duration of partial and major outages and any related incidents. Incident reports provide detailed records of past issues, organized chronologically by month. Each incident report includes a title, a timeline of incident status updates with detailed descriptions, and the services affected. Not all outages have corresponding incident reports. Conversely, some incidents, e.g., with minimal impact, do not report a service outage. </div> <div class="ltx_para" id="S3.SS2.p3"> We developed an automated data-collection method, able to collect industry-standard outage and incident reports. Our tools leverage Python Selenium WebDriver <cite class="ltx_cite ltx_citemacro_citep">(Selenium, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib52" title="">2024</a>)</cite>, a robust tool allowing native browser automation by simulating real-user interactions. Our tools implement exception-handling mechanisms, addressing potential issues such as network problems, stale elements, and unexpected page layouts. They parse and extract information from the dynamic pages and store them as raw outage and incident datasets. After that, we performed a series of data transformations for the raw datasets, including filling in missing values, extracting data from text, processing JSON formats, splitting columns, and performing feature calculations to get the metrics used in this study. </div> <figure class="ltx_table" id="S3.T5"> <figcaption class="ltx_caption ltx_centering">Table 5. Status counts of incident reports (see <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.T4" title="In 2.2. LLM-Specific Terms and Metrics ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 4</a>).</figcaption> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S3.T5.5" style="width:130.1pt;height:189.3pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_align_middle" id="S3.T5.5.5"> <tr class="ltx_tr" id="S3.T5.5.5.5"> <td class="ltx_td ltx_align_center ltx_border_tt" id="S3.T5.1.1.1.1"><math alttext="S_{1}" class="ltx_Math" display="inline" id="S3.T5.1.1.1.1.m1.1"><semantics id="S3.T5.1.1.1.1.m1.1a"><msub id="S3.T5.1.1.1.1.m1.1.1" xref="S3.T5.1.1.1.1.m1.1.1.cmml"><mi id="S3.T5.1.1.1.1.m1.1.1.2" xref="S3.T5.1.1.1.1.m1.1.1.2.cmml">S</mi><mn id="S3.T5.1.1.1.1.m1.1.1.3" xref="S3.T5.1.1.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T5.1.1.1.1.m1.1b"><apply id="S3.T5.1.1.1.1.m1.1.1.cmml" xref="S3.T5.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T5.1.1.1.1.m1.1.1.1.cmml" xref="S3.T5.1.1.1.1.m1.1.1">subscript</csymbol><ci id="S3.T5.1.1.1.1.m1.1.1.2.cmml" xref="S3.T5.1.1.1.1.m1.1.1.2">𝑆</ci><cn id="S3.T5.1.1.1.1.m1.1.1.3.cmml" type="integer" xref="S3.T5.1.1.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.1.1.1.1.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S3.T5.1.1.1.1.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S3.T5.2.2.2.2"><math alttext="S_{2}" class="ltx_Math" display="inline" id="S3.T5.2.2.2.2.m1.1"><semantics id="S3.T5.2.2.2.2.m1.1a"><msub id="S3.T5.2.2.2.2.m1.1.1" xref="S3.T5.2.2.2.2.m1.1.1.cmml"><mi id="S3.T5.2.2.2.2.m1.1.1.2" xref="S3.T5.2.2.2.2.m1.1.1.2.cmml">S</mi><mn id="S3.T5.2.2.2.2.m1.1.1.3" xref="S3.T5.2.2.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T5.2.2.2.2.m1.1b"><apply id="S3.T5.2.2.2.2.m1.1.1.cmml" xref="S3.T5.2.2.2.2.m1.1.1"><csymbol cd="ambiguous" id="S3.T5.2.2.2.2.m1.1.1.1.cmml" xref="S3.T5.2.2.2.2.m1.1.1">subscript</csymbol><ci id="S3.T5.2.2.2.2.m1.1.1.2.cmml" xref="S3.T5.2.2.2.2.m1.1.1.2">𝑆</ci><cn id="S3.T5.2.2.2.2.m1.1.1.3.cmml" type="integer" xref="S3.T5.2.2.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.2.2.2.2.m1.1c">S_{2}</annotation><annotation encoding="application/x-llamapun" id="S3.T5.2.2.2.2.m1.1d">italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S3.T5.3.3.3.3"><math alttext="S_{3}" class="ltx_Math" display="inline" id="S3.T5.3.3.3.3.m1.1"><semantics id="S3.T5.3.3.3.3.m1.1a"><msub id="S3.T5.3.3.3.3.m1.1.1" xref="S3.T5.3.3.3.3.m1.1.1.cmml"><mi id="S3.T5.3.3.3.3.m1.1.1.2" xref="S3.T5.3.3.3.3.m1.1.1.2.cmml">S</mi><mn id="S3.T5.3.3.3.3.m1.1.1.3" xref="S3.T5.3.3.3.3.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T5.3.3.3.3.m1.1b"><apply id="S3.T5.3.3.3.3.m1.1.1.cmml" xref="S3.T5.3.3.3.3.m1.1.1"><csymbol cd="ambiguous" id="S3.T5.3.3.3.3.m1.1.1.1.cmml" xref="S3.T5.3.3.3.3.m1.1.1">subscript</csymbol><ci id="S3.T5.3.3.3.3.m1.1.1.2.cmml" xref="S3.T5.3.3.3.3.m1.1.1.2">𝑆</ci><cn id="S3.T5.3.3.3.3.m1.1.1.3.cmml" type="integer" xref="S3.T5.3.3.3.3.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.3.3.3.3.m1.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S3.T5.3.3.3.3.m1.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S3.T5.4.4.4.4"><math alttext="S_{4}" class="ltx_Math" display="inline" id="S3.T5.4.4.4.4.m1.1"><semantics id="S3.T5.4.4.4.4.m1.1a"><msub id="S3.T5.4.4.4.4.m1.1.1" xref="S3.T5.4.4.4.4.m1.1.1.cmml"><mi id="S3.T5.4.4.4.4.m1.1.1.2" xref="S3.T5.4.4.4.4.m1.1.1.2.cmml">S</mi><mn id="S3.T5.4.4.4.4.m1.1.1.3" xref="S3.T5.4.4.4.4.m1.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T5.4.4.4.4.m1.1b"><apply id="S3.T5.4.4.4.4.m1.1.1.cmml" xref="S3.T5.4.4.4.4.m1.1.1"><csymbol cd="ambiguous" id="S3.T5.4.4.4.4.m1.1.1.1.cmml" xref="S3.T5.4.4.4.4.m1.1.1">subscript</csymbol><ci id="S3.T5.4.4.4.4.m1.1.1.2.cmml" xref="S3.T5.4.4.4.4.m1.1.1.2">𝑆</ci><cn id="S3.T5.4.4.4.4.m1.1.1.3.cmml" type="integer" xref="S3.T5.4.4.4.4.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.4.4.4.4.m1.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S3.T5.4.4.4.4.m1.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_center ltx_border_tt" id="S3.T5.5.5.5.5"><math alttext="S_{5}" class="ltx_Math" display="inline" id="S3.T5.5.5.5.5.m1.1"><semantics id="S3.T5.5.5.5.5.m1.1a"><msub id="S3.T5.5.5.5.5.m1.1.1" xref="S3.T5.5.5.5.5.m1.1.1.cmml"><mi id="S3.T5.5.5.5.5.m1.1.1.2" xref="S3.T5.5.5.5.5.m1.1.1.2.cmml">S</mi><mn id="S3.T5.5.5.5.5.m1.1.1.3" xref="S3.T5.5.5.5.5.m1.1.1.3.cmml">5</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T5.5.5.5.5.m1.1b"><apply id="S3.T5.5.5.5.5.m1.1.1.cmml" xref="S3.T5.5.5.5.5.m1.1.1"><csymbol cd="ambiguous" id="S3.T5.5.5.5.5.m1.1.1.1.cmml" xref="S3.T5.5.5.5.5.m1.1.1">subscript</csymbol><ci id="S3.T5.5.5.5.5.m1.1.1.2.cmml" xref="S3.T5.5.5.5.5.m1.1.1.2">𝑆</ci><cn id="S3.T5.5.5.5.5.m1.1.1.3.cmml" type="integer" xref="S3.T5.5.5.5.5.m1.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.5.5.5.5.m1.1c">S_{5}</annotation><annotation encoding="application/x-llamapun" id="S3.T5.5.5.5.5.m1.1d">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S3.T5.5.5.5.6">Count</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S3.T5.5.5.5.7">Percent</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.6"> <td class="ltx_td ltx_align_center ltx_border_t" id="S3.T5.5.5.6.1">✓</td> <td class="ltx_td ltx_border_t" id="S3.T5.5.5.6.2"></td> <td class="ltx_td ltx_align_center ltx_border_t" id="S3.T5.5.5.6.3">✓</td> <td class="ltx_td ltx_align_center ltx_border_t" id="S3.T5.5.5.6.4">✓</td> <td class="ltx_td ltx_border_t" id="S3.T5.5.5.6.5"></td> <td class="ltx_td ltx_align_right ltx_border_t" id="S3.T5.5.5.6.6">131</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S3.T5.5.5.6.7">24.39%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.7"> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.7.1">✓</td> <td class="ltx_td" id="S3.T5.5.5.7.2"></td> <td class="ltx_td" id="S3.T5.5.5.7.3"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.7.4">✓</td> <td class="ltx_td" id="S3.T5.5.5.7.5"></td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.7.6">110</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.7.7">20.48%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.8"> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.8.1">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.8.2">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.8.3">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.8.4">✓</td> <td class="ltx_td" id="S3.T5.5.5.8.5"></td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.8.6">77</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.8.7">14.34%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.9"> <td class="ltx_td" id="S3.T5.5.5.9.1"></td> <td class="ltx_td" id="S3.T5.5.5.9.2"></td> <td class="ltx_td" id="S3.T5.5.5.9.3"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.9.4">✓</td> <td class="ltx_td" id="S3.T5.5.5.9.5"></td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.9.6">62</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.9.7">11.55%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.10"> <td class="ltx_td" id="S3.T5.5.5.10.1"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.10.2">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.10.3">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.10.4">✓</td> <td class="ltx_td" id="S3.T5.5.5.10.5"></td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.10.6">39</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.10.7">7.26%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.11"> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.11.1">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.11.2">✓</td> <td class="ltx_td" id="S3.T5.5.5.11.3"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.11.4">✓</td> <td class="ltx_td" id="S3.T5.5.5.11.5"></td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.11.6">35</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.11.7">6.52%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.12"> <td class="ltx_td" id="S3.T5.5.5.12.1"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.12.2">✓</td> <td class="ltx_td" id="S3.T5.5.5.12.3"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.12.4">✓</td> <td class="ltx_td" id="S3.T5.5.5.12.5"></td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.12.6">32</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.12.7">5.96%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.13"> <td class="ltx_td" id="S3.T5.5.5.13.1"></td> <td class="ltx_td" id="S3.T5.5.5.13.2"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.13.3">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.13.4">✓</td> <td class="ltx_td" id="S3.T5.5.5.13.5"></td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.13.6">18</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.13.7">3.35%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.14"> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.14.1">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.14.2">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.14.3">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.14.4">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.14.5">✓</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.14.6">12</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.14.7">2.23%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.15"> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.15.1">✓</td> <td class="ltx_td" id="S3.T5.5.5.15.2"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.15.3">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.15.4">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.15.5">✓</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.15.6">7</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.15.7">1.30%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.16"> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.16.1">✓</td> <td class="ltx_td" id="S3.T5.5.5.16.2"></td> <td class="ltx_td" id="S3.T5.5.5.16.3"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.16.4">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.16.5">✓</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.16.6">5</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.16.7">0.93%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.17"> <td class="ltx_td" id="S3.T5.5.5.17.1"></td> <td class="ltx_td" id="S3.T5.5.5.17.2"></td> <td class="ltx_td" id="S3.T5.5.5.17.3"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.17.4">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.17.5">✓</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.17.6">4</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.17.7">0.74%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.18"> <td class="ltx_td" id="S3.T5.5.5.18.1"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.18.2">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.18.3">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.18.4">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.18.5">✓</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.18.6">3</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.18.7">0.56%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.19"> <td class="ltx_td" id="S3.T5.5.5.19.1"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.19.2">✓</td> <td class="ltx_td" id="S3.T5.5.5.19.3"></td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.19.4">✓</td> <td class="ltx_td ltx_align_center" id="S3.T5.5.5.19.5">✓</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.19.6">2</td> <td class="ltx_td ltx_align_right" id="S3.T5.5.5.19.7">0.37%</td> </tr> <tr class="ltx_tr" id="S3.T5.5.5.20"> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" colspan="5" id="S3.T5.5.5.20.1">TOTAL</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S3.T5.5.5.20.2">537</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S3.T5.5.5.20.3">100.00%</td> </tr> </table> </div> </figure> <figure class="ltx_figure" id="S3.F3"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="519" id="S3.F3.g1" src="x3.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 3. Presence of different status combinations, by service [%]. Due to small counts, status combinations with <math alttext="S_{5}" class="ltx_Math" display="inline" id="S3.F3.2.1.m1.1"><semantics id="S3.F3.2.1.m1.1b"><msub id="S3.F3.2.1.m1.1.1" xref="S3.F3.2.1.m1.1.1.cmml"><mi id="S3.F3.2.1.m1.1.1.2" xref="S3.F3.2.1.m1.1.1.2.cmml">S</mi><mn id="S3.F3.2.1.m1.1.1.3" xref="S3.F3.2.1.m1.1.1.3.cmml">5</mn></msub><annotation-xml encoding="MathML-Content" id="S3.F3.2.1.m1.1c"><apply id="S3.F3.2.1.m1.1.1.cmml" xref="S3.F3.2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.F3.2.1.m1.1.1.1.cmml" xref="S3.F3.2.1.m1.1.1">subscript</csymbol><ci id="S3.F3.2.1.m1.1.1.2.cmml" xref="S3.F3.2.1.m1.1.1.2">𝑆</ci><cn id="S3.F3.2.1.m1.1.1.3.cmml" type="integer" xref="S3.F3.2.1.m1.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.2.1.m1.1d">S_{5}</annotation><annotation encoding="application/x-llamapun" id="S3.F3.2.1.m1.1e">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT</annotation></semantics></math> are merged into ‘All-with-S5’. (<a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.T3" title="In 2.2. LLM-Specific Terms and Metrics ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 3</a> indexes the services.)</figcaption> </figure> <div class="ltx_para" id="S3.SS2.p4"> Data cleanup related to the failure-recovery model in <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S2.SS1" title="2.1. Model and Real-World Example ‣ 2. Anatomy of an LLM-service Incident: Model and Example ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 2.1</a>: (1) The incident reports from, e.g., ChatGPT, include 6 statuses: investigating, identified, monitoring, update, resolved, and postmortem. The update status is not considered in the model, but in all the reports we have analyzed it seems to be used only as a keep-alive of the recovery process, to mark the operational team is still actively working on the incident, so we do not consider it in our analysis; (2) Out of the over 500 hundreds of incidents we analyzed in this work, only 5 cases do not follow the order of status markers <math alttext="S_{1}" class="ltx_Math" display="inline" id="S3.SS2.p4.1.m1.1"><semantics id="S3.SS2.p4.1.m1.1a"><msub id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml"><mi id="S3.SS2.p4.1.m1.1.1.2" xref="S3.SS2.p4.1.m1.1.1.2.cmml">S</mi><mn id="S3.SS2.p4.1.m1.1.1.3" xref="S3.SS2.p4.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.1b"><apply id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p4.1.m1.1.1.2.cmml" xref="S3.SS2.p4.1.m1.1.1.2">𝑆</ci><cn id="S3.SS2.p4.1.m1.1.1.3.cmml" type="integer" xref="S3.SS2.p4.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.p4.1.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math> through <math alttext="S_{5}" class="ltx_Math" display="inline" id="S3.SS2.p4.2.m2.1"><semantics id="S3.SS2.p4.2.m2.1a"><msub id="S3.SS2.p4.2.m2.1.1" xref="S3.SS2.p4.2.m2.1.1.cmml"><mi id="S3.SS2.p4.2.m2.1.1.2" xref="S3.SS2.p4.2.m2.1.1.2.cmml">S</mi><mn id="S3.SS2.p4.2.m2.1.1.3" xref="S3.SS2.p4.2.m2.1.1.3.cmml">5</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.2.m2.1b"><apply id="S3.SS2.p4.2.m2.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p4.2.m2.1.1.2.cmml" xref="S3.SS2.p4.2.m2.1.1.2">𝑆</ci><cn id="S3.SS2.p4.2.m2.1.1.3.cmml" type="integer" xref="S3.SS2.p4.2.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.1c">S_{5}</annotation><annotation encoding="application/x-llamapun" id="S3.SS2.p4.2.m2.1d">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT</annotation></semantics></math>. In 2 of these cases, the status-marker identified comes before investigating, and in 3 other cases, the status-marker monitoring comes before identified. None of these cases involves unusual durations or recovery times, so we safely exclude these 5 corner cases in our analysis. </div> </section> </section> <section class="ltx_section" id="S4"> <h2 class="ltx_title ltx_title_section"> 4. Failure-Recovery Analysis</h2> <div class="ltx_para" id="S4.p1"> We investigate the time spent on the key operational metrics (MTTR, MTBF) and compare the failure-recovery performance across the 8 LLM services and 3 service providers. We conduct several types of analysis to investigate the failure-recovery processes of LLM services: (1) Statues count and percent of different services; (2) Mean values for main model parameters; (3) Percent of different periods in the MTTR process; (4) Distribution of MTTR and MTTF duration by service; (5) Distribution of MTTR and MTTF duration by providers. For each analysis, we begin with key observations, followed by detailed descriptions and discussions. </div> <div class="ltx_para ltx_noindent" id="S4.p2"> Observation #1: Most incident reports lack information for all statuses. Although 100% of the incidents have been resolved, only 6.15% of reports disclose a postmortem. </div> <figure class="ltx_table" id="S4.T6"> <figcaption class="ltx_caption">Table 6. Mean value for model parameters by service. Legend: h<math alttext="=" class="ltx_Math" display="inline" id="S4.T6.3.1.m1.1"><semantics id="S4.T6.3.1.m1.1b"><mo id="S4.T6.3.1.m1.1.1" xref="S4.T6.3.1.m1.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S4.T6.3.1.m1.1c"><eq id="S4.T6.3.1.m1.1.1.cmml" xref="S4.T6.3.1.m1.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.3.1.m1.1d">=</annotation><annotation encoding="application/x-llamapun" id="S4.T6.3.1.m1.1e">=</annotation></semantics></math>hour(s), D<math alttext="=" class="ltx_Math" display="inline" id="S4.T6.4.2.m2.1"><semantics id="S4.T6.4.2.m2.1b"><mo id="S4.T6.4.2.m2.1.1" xref="S4.T6.4.2.m2.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S4.T6.4.2.m2.1c"><eq id="S4.T6.4.2.m2.1.1.cmml" xref="S4.T6.4.2.m2.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.4.2.m2.1d">=</annotation><annotation encoding="application/x-llamapun" id="S4.T6.4.2.m2.1e">=</annotation></semantics></math>day(s).</figcaption> <div class="ltx_inline-block ltx_transformed_outer" id="S4.T6.18" style="width:433.6pt;height:225.9pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_align_middle" id="S4.T6.18.14"> <tr class="ltx_tr" id="S4.T6.10.6.6"> <td class="ltx_td ltx_align_left ltx_border_tt" id="S4.T6.10.6.6.7">ID</td> <td class="ltx_td ltx_align_left ltx_border_tt" id="S4.T6.10.6.6.8">Service</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S4.T6.5.1.1.1"> <math alttext="P_{I}" class="ltx_Math" display="inline" id="S4.T6.5.1.1.1.m1.1"><semantics id="S4.T6.5.1.1.1.m1.1a"><msub id="S4.T6.5.1.1.1.m1.1.1" xref="S4.T6.5.1.1.1.m1.1.1.cmml"><mi id="S4.T6.5.1.1.1.m1.1.1.2" xref="S4.T6.5.1.1.1.m1.1.1.2.cmml">P</mi><mi id="S4.T6.5.1.1.1.m1.1.1.3" xref="S4.T6.5.1.1.1.m1.1.1.3.cmml">I</mi></msub><annotation-xml encoding="MathML-Content" id="S4.T6.5.1.1.1.m1.1b"><apply id="S4.T6.5.1.1.1.m1.1.1.cmml" xref="S4.T6.5.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.5.1.1.1.m1.1.1.1.cmml" xref="S4.T6.5.1.1.1.m1.1.1">subscript</csymbol><ci id="S4.T6.5.1.1.1.m1.1.1.2.cmml" xref="S4.T6.5.1.1.1.m1.1.1.2">𝑃</ci><ci id="S4.T6.5.1.1.1.m1.1.1.3.cmml" xref="S4.T6.5.1.1.1.m1.1.1.3">𝐼</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.5.1.1.1.m1.1c">P_{I}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.5.1.1.1.m1.1d">italic_P start_POSTSUBSCRIPT italic_I end_POSTSUBSCRIPT</annotation></semantics></math> [h] </td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S4.T6.6.2.2.2"> <math alttext="P_{R}" class="ltx_Math" display="inline" id="S4.T6.6.2.2.2.m1.1"><semantics id="S4.T6.6.2.2.2.m1.1a"><msub id="S4.T6.6.2.2.2.m1.1.1" xref="S4.T6.6.2.2.2.m1.1.1.cmml"><mi id="S4.T6.6.2.2.2.m1.1.1.2" xref="S4.T6.6.2.2.2.m1.1.1.2.cmml">P</mi><mi id="S4.T6.6.2.2.2.m1.1.1.3" xref="S4.T6.6.2.2.2.m1.1.1.3.cmml">R</mi></msub><annotation-xml encoding="MathML-Content" id="S4.T6.6.2.2.2.m1.1b"><apply id="S4.T6.6.2.2.2.m1.1.1.cmml" xref="S4.T6.6.2.2.2.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.6.2.2.2.m1.1.1.1.cmml" xref="S4.T6.6.2.2.2.m1.1.1">subscript</csymbol><ci id="S4.T6.6.2.2.2.m1.1.1.2.cmml" xref="S4.T6.6.2.2.2.m1.1.1.2">𝑃</ci><ci id="S4.T6.6.2.2.2.m1.1.1.3.cmml" xref="S4.T6.6.2.2.2.m1.1.1.3">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.6.2.2.2.m1.1c">P_{R}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.6.2.2.2.m1.1d">italic_P start_POSTSUBSCRIPT italic_R end_POSTSUBSCRIPT</annotation></semantics></math> [h] </td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S4.T6.7.3.3.3"> <math alttext="P_{C}" class="ltx_Math" display="inline" id="S4.T6.7.3.3.3.m1.1"><semantics id="S4.T6.7.3.3.3.m1.1a"><msub id="S4.T6.7.3.3.3.m1.1.1" xref="S4.T6.7.3.3.3.m1.1.1.cmml"><mi id="S4.T6.7.3.3.3.m1.1.1.2" xref="S4.T6.7.3.3.3.m1.1.1.2.cmml">P</mi><mi id="S4.T6.7.3.3.3.m1.1.1.3" xref="S4.T6.7.3.3.3.m1.1.1.3.cmml">C</mi></msub><annotation-xml encoding="MathML-Content" id="S4.T6.7.3.3.3.m1.1b"><apply id="S4.T6.7.3.3.3.m1.1.1.cmml" xref="S4.T6.7.3.3.3.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.7.3.3.3.m1.1.1.1.cmml" xref="S4.T6.7.3.3.3.m1.1.1">subscript</csymbol><ci id="S4.T6.7.3.3.3.m1.1.1.2.cmml" xref="S4.T6.7.3.3.3.m1.1.1.2">𝑃</ci><ci id="S4.T6.7.3.3.3.m1.1.1.3.cmml" xref="S4.T6.7.3.3.3.m1.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.7.3.3.3.m1.1c">P_{C}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.7.3.3.3.m1.1d">italic_P start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT</annotation></semantics></math> [h] </td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S4.T6.8.4.4.4"> <math alttext="P_{L}" class="ltx_Math" display="inline" id="S4.T6.8.4.4.4.m1.1"><semantics id="S4.T6.8.4.4.4.m1.1a"><msub id="S4.T6.8.4.4.4.m1.1.1" xref="S4.T6.8.4.4.4.m1.1.1.cmml"><mi id="S4.T6.8.4.4.4.m1.1.1.2" xref="S4.T6.8.4.4.4.m1.1.1.2.cmml">P</mi><mi id="S4.T6.8.4.4.4.m1.1.1.3" xref="S4.T6.8.4.4.4.m1.1.1.3.cmml">L</mi></msub><annotation-xml encoding="MathML-Content" id="S4.T6.8.4.4.4.m1.1b"><apply id="S4.T6.8.4.4.4.m1.1.1.cmml" xref="S4.T6.8.4.4.4.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.8.4.4.4.m1.1.1.1.cmml" xref="S4.T6.8.4.4.4.m1.1.1">subscript</csymbol><ci id="S4.T6.8.4.4.4.m1.1.1.2.cmml" xref="S4.T6.8.4.4.4.m1.1.1.2">𝑃</ci><ci id="S4.T6.8.4.4.4.m1.1.1.3.cmml" xref="S4.T6.8.4.4.4.m1.1.1.3">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.8.4.4.4.m1.1c">P_{L}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.8.4.4.4.m1.1d">italic_P start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT</annotation></semantics></math> [D] </td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S4.T6.9.5.5.5"> <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.T6.9.5.5.5.m1.1"><semantics id="S4.T6.9.5.5.5.m1.1a"><mrow id="S4.T6.9.5.5.5.m1.1.1" xref="S4.T6.9.5.5.5.m1.1.1.cmml"><mi id="S4.T6.9.5.5.5.m1.1.1.2" xref="S4.T6.9.5.5.5.m1.1.1.2.cmml">M</mi><mo id="S4.T6.9.5.5.5.m1.1.1.1" xref="S4.T6.9.5.5.5.m1.1.1.1.cmml">⁢</mo><mi id="S4.T6.9.5.5.5.m1.1.1.3" xref="S4.T6.9.5.5.5.m1.1.1.3.cmml">T</mi><mo id="S4.T6.9.5.5.5.m1.1.1.1a" xref="S4.T6.9.5.5.5.m1.1.1.1.cmml">⁢</mo><mi id="S4.T6.9.5.5.5.m1.1.1.4" xref="S4.T6.9.5.5.5.m1.1.1.4.cmml">T</mi><mo id="S4.T6.9.5.5.5.m1.1.1.1b" xref="S4.T6.9.5.5.5.m1.1.1.1.cmml">⁢</mo><mi id="S4.T6.9.5.5.5.m1.1.1.5" xref="S4.T6.9.5.5.5.m1.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T6.9.5.5.5.m1.1b"><apply id="S4.T6.9.5.5.5.m1.1.1.cmml" xref="S4.T6.9.5.5.5.m1.1.1"><times id="S4.T6.9.5.5.5.m1.1.1.1.cmml" xref="S4.T6.9.5.5.5.m1.1.1.1"></times><ci id="S4.T6.9.5.5.5.m1.1.1.2.cmml" xref="S4.T6.9.5.5.5.m1.1.1.2">𝑀</ci><ci id="S4.T6.9.5.5.5.m1.1.1.3.cmml" xref="S4.T6.9.5.5.5.m1.1.1.3">𝑇</ci><ci id="S4.T6.9.5.5.5.m1.1.1.4.cmml" xref="S4.T6.9.5.5.5.m1.1.1.4">𝑇</ci><ci id="S4.T6.9.5.5.5.m1.1.1.5.cmml" xref="S4.T6.9.5.5.5.m1.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.9.5.5.5.m1.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.T6.9.5.5.5.m1.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math> [h] </td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S4.T6.10.6.6.6"> <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.T6.10.6.6.6.m1.1"><semantics id="S4.T6.10.6.6.6.m1.1a"><mrow id="S4.T6.10.6.6.6.m1.1.1" xref="S4.T6.10.6.6.6.m1.1.1.cmml"><mi id="S4.T6.10.6.6.6.m1.1.1.2" xref="S4.T6.10.6.6.6.m1.1.1.2.cmml">M</mi><mo id="S4.T6.10.6.6.6.m1.1.1.1" xref="S4.T6.10.6.6.6.m1.1.1.1.cmml">⁢</mo><mi id="S4.T6.10.6.6.6.m1.1.1.3" xref="S4.T6.10.6.6.6.m1.1.1.3.cmml">T</mi><mo id="S4.T6.10.6.6.6.m1.1.1.1a" xref="S4.T6.10.6.6.6.m1.1.1.1.cmml">⁢</mo><mi id="S4.T6.10.6.6.6.m1.1.1.4" xref="S4.T6.10.6.6.6.m1.1.1.4.cmml">B</mi><mo id="S4.T6.10.6.6.6.m1.1.1.1b" xref="S4.T6.10.6.6.6.m1.1.1.1.cmml">⁢</mo><mi id="S4.T6.10.6.6.6.m1.1.1.5" xref="S4.T6.10.6.6.6.m1.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T6.10.6.6.6.m1.1b"><apply id="S4.T6.10.6.6.6.m1.1.1.cmml" xref="S4.T6.10.6.6.6.m1.1.1"><times id="S4.T6.10.6.6.6.m1.1.1.1.cmml" xref="S4.T6.10.6.6.6.m1.1.1.1"></times><ci id="S4.T6.10.6.6.6.m1.1.1.2.cmml" xref="S4.T6.10.6.6.6.m1.1.1.2">𝑀</ci><ci id="S4.T6.10.6.6.6.m1.1.1.3.cmml" xref="S4.T6.10.6.6.6.m1.1.1.3">𝑇</ci><ci id="S4.T6.10.6.6.6.m1.1.1.4.cmml" xref="S4.T6.10.6.6.6.m1.1.1.4">𝐵</ci><ci id="S4.T6.10.6.6.6.m1.1.1.5.cmml" xref="S4.T6.10.6.6.6.m1.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.10.6.6.6.m1.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.T6.10.6.6.6.m1.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> [D] </td> </tr> <tr class="ltx_tr" id="S4.T6.11.7.7"> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T6.11.7.7.1"><math alttext="O_{1}" class="ltx_Math" display="inline" id="S4.T6.11.7.7.1.m1.1"><semantics id="S4.T6.11.7.7.1.m1.1a"><msub id="S4.T6.11.7.7.1.m1.1.1" xref="S4.T6.11.7.7.1.m1.1.1.cmml"><mi id="S4.T6.11.7.7.1.m1.1.1.2" xref="S4.T6.11.7.7.1.m1.1.1.2.cmml">O</mi><mn id="S4.T6.11.7.7.1.m1.1.1.3" xref="S4.T6.11.7.7.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T6.11.7.7.1.m1.1b"><apply id="S4.T6.11.7.7.1.m1.1.1.cmml" xref="S4.T6.11.7.7.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.11.7.7.1.m1.1.1.1.cmml" xref="S4.T6.11.7.7.1.m1.1.1">subscript</csymbol><ci id="S4.T6.11.7.7.1.m1.1.1.2.cmml" xref="S4.T6.11.7.7.1.m1.1.1.2">𝑂</ci><cn id="S4.T6.11.7.7.1.m1.1.1.3.cmml" type="integer" xref="S4.T6.11.7.7.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.11.7.7.1.m1.1c">O_{1}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.11.7.7.1.m1.1d">italic_O start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T6.11.7.7.2">API-OpenAI</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.11.7.7.3">0.72</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.11.7.7.4">1.63</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.11.7.7.5">1.46</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.11.7.7.6">4.10</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.11.7.7.7">2.56</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.11.7.7.8">5.64</td> </tr> <tr class="ltx_tr" id="S4.T6.12.8.8"> <td class="ltx_td ltx_align_left" id="S4.T6.12.8.8.1"><math alttext="O_{2}" class="ltx_Math" display="inline" id="S4.T6.12.8.8.1.m1.1"><semantics id="S4.T6.12.8.8.1.m1.1a"><msub id="S4.T6.12.8.8.1.m1.1.1" xref="S4.T6.12.8.8.1.m1.1.1.cmml"><mi id="S4.T6.12.8.8.1.m1.1.1.2" xref="S4.T6.12.8.8.1.m1.1.1.2.cmml">O</mi><mn id="S4.T6.12.8.8.1.m1.1.1.3" xref="S4.T6.12.8.8.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T6.12.8.8.1.m1.1b"><apply id="S4.T6.12.8.8.1.m1.1.1.cmml" xref="S4.T6.12.8.8.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.12.8.8.1.m1.1.1.1.cmml" xref="S4.T6.12.8.8.1.m1.1.1">subscript</csymbol><ci id="S4.T6.12.8.8.1.m1.1.1.2.cmml" xref="S4.T6.12.8.8.1.m1.1.1.2">𝑂</ci><cn id="S4.T6.12.8.8.1.m1.1.1.3.cmml" type="integer" xref="S4.T6.12.8.8.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.12.8.8.1.m1.1c">O_{2}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.12.8.8.1.m1.1d">italic_O start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S4.T6.12.8.8.2">ChatGPT</td> <td class="ltx_td ltx_align_right" id="S4.T6.12.8.8.3">0.65</td> <td class="ltx_td ltx_align_right" id="S4.T6.12.8.8.4">1.64</td> <td class="ltx_td ltx_align_right" id="S4.T6.12.8.8.5">1.73</td> <td class="ltx_td ltx_align_right" id="S4.T6.12.8.8.6">4.79</td> <td class="ltx_td ltx_align_right" id="S4.T6.12.8.8.7">3.64</td> <td class="ltx_td ltx_align_right" id="S4.T6.12.8.8.8">4.01</td> </tr> <tr class="ltx_tr" id="S4.T6.13.9.9"> <td class="ltx_td ltx_align_left" id="S4.T6.13.9.9.1"><math alttext="O_{3}" class="ltx_Math" display="inline" id="S4.T6.13.9.9.1.m1.1"><semantics id="S4.T6.13.9.9.1.m1.1a"><msub id="S4.T6.13.9.9.1.m1.1.1" xref="S4.T6.13.9.9.1.m1.1.1.cmml"><mi id="S4.T6.13.9.9.1.m1.1.1.2" xref="S4.T6.13.9.9.1.m1.1.1.2.cmml">O</mi><mn id="S4.T6.13.9.9.1.m1.1.1.3" xref="S4.T6.13.9.9.1.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T6.13.9.9.1.m1.1b"><apply id="S4.T6.13.9.9.1.m1.1.1.cmml" xref="S4.T6.13.9.9.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.13.9.9.1.m1.1.1.1.cmml" xref="S4.T6.13.9.9.1.m1.1.1">subscript</csymbol><ci id="S4.T6.13.9.9.1.m1.1.1.2.cmml" xref="S4.T6.13.9.9.1.m1.1.1.2">𝑂</ci><cn id="S4.T6.13.9.9.1.m1.1.1.3.cmml" type="integer" xref="S4.T6.13.9.9.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.13.9.9.1.m1.1c">O_{3}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.13.9.9.1.m1.1d">italic_O start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S4.T6.13.9.9.2">DALL·E</td> <td class="ltx_td ltx_align_right" id="S4.T6.13.9.9.3">1.01</td> <td class="ltx_td ltx_align_right" id="S4.T6.13.9.9.4">0.96</td> <td class="ltx_td ltx_align_right" id="S4.T6.13.9.9.5">1.81</td> <td class="ltx_td ltx_align_right" id="S4.T6.13.9.9.6">1.86</td> <td class="ltx_td ltx_align_right" id="S4.T6.13.9.9.7">3.03</td> <td class="ltx_td ltx_align_right" id="S4.T6.13.9.9.8">18.24</td> </tr> <tr class="ltx_tr" id="S4.T6.14.10.10"> <td class="ltx_td ltx_align_left" id="S4.T6.14.10.10.1"><math alttext="O_{4}" class="ltx_Math" display="inline" id="S4.T6.14.10.10.1.m1.1"><semantics id="S4.T6.14.10.10.1.m1.1a"><msub id="S4.T6.14.10.10.1.m1.1.1" xref="S4.T6.14.10.10.1.m1.1.1.cmml"><mi id="S4.T6.14.10.10.1.m1.1.1.2" xref="S4.T6.14.10.10.1.m1.1.1.2.cmml">O</mi><mn id="S4.T6.14.10.10.1.m1.1.1.3" xref="S4.T6.14.10.10.1.m1.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T6.14.10.10.1.m1.1b"><apply id="S4.T6.14.10.10.1.m1.1.1.cmml" xref="S4.T6.14.10.10.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.14.10.10.1.m1.1.1.1.cmml" xref="S4.T6.14.10.10.1.m1.1.1">subscript</csymbol><ci id="S4.T6.14.10.10.1.m1.1.1.2.cmml" xref="S4.T6.14.10.10.1.m1.1.1.2">𝑂</ci><cn id="S4.T6.14.10.10.1.m1.1.1.3.cmml" type="integer" xref="S4.T6.14.10.10.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.14.10.10.1.m1.1c">O_{4}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.14.10.10.1.m1.1d">italic_O start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S4.T6.14.10.10.2">Playground</td> <td class="ltx_td ltx_align_right" id="S4.T6.14.10.10.3">0.37</td> <td class="ltx_td ltx_align_right" id="S4.T6.14.10.10.4">1.56</td> <td class="ltx_td ltx_align_right" id="S4.T6.14.10.10.5">2.22</td> <td class="ltx_td ltx_align_right" id="S4.T6.14.10.10.6">4.30</td> <td class="ltx_td ltx_align_right" id="S4.T6.14.10.10.7">2.95</td> <td class="ltx_td ltx_align_right" id="S4.T6.14.10.10.8">39.93</td> </tr> <tr class="ltx_tr" id="S4.T6.15.11.11"> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T6.15.11.11.1"><math alttext="A_{1}" class="ltx_Math" display="inline" id="S4.T6.15.11.11.1.m1.1"><semantics id="S4.T6.15.11.11.1.m1.1a"><msub id="S4.T6.15.11.11.1.m1.1.1" xref="S4.T6.15.11.11.1.m1.1.1.cmml"><mi id="S4.T6.15.11.11.1.m1.1.1.2" xref="S4.T6.15.11.11.1.m1.1.1.2.cmml">A</mi><mn id="S4.T6.15.11.11.1.m1.1.1.3" xref="S4.T6.15.11.11.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T6.15.11.11.1.m1.1b"><apply id="S4.T6.15.11.11.1.m1.1.1.cmml" xref="S4.T6.15.11.11.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.15.11.11.1.m1.1.1.1.cmml" xref="S4.T6.15.11.11.1.m1.1.1">subscript</csymbol><ci id="S4.T6.15.11.11.1.m1.1.1.2.cmml" xref="S4.T6.15.11.11.1.m1.1.1.2">𝐴</ci><cn id="S4.T6.15.11.11.1.m1.1.1.3.cmml" type="integer" xref="S4.T6.15.11.11.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.15.11.11.1.m1.1c">A_{1}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.15.11.11.1.m1.1d">italic_A start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T6.15.11.11.2">API-Anthropic</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.15.11.11.3">1.04</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.15.11.11.4">1.11</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.15.11.11.5">1.37</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.15.11.11.6">-</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.15.11.11.7">2.81</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.15.11.11.8">5.22</td> </tr> <tr class="ltx_tr" id="S4.T6.16.12.12"> <td class="ltx_td ltx_align_left" id="S4.T6.16.12.12.1"><math alttext="A_{2}" class="ltx_Math" display="inline" id="S4.T6.16.12.12.1.m1.1"><semantics id="S4.T6.16.12.12.1.m1.1a"><msub id="S4.T6.16.12.12.1.m1.1.1" xref="S4.T6.16.12.12.1.m1.1.1.cmml"><mi id="S4.T6.16.12.12.1.m1.1.1.2" xref="S4.T6.16.12.12.1.m1.1.1.2.cmml">A</mi><mn id="S4.T6.16.12.12.1.m1.1.1.3" xref="S4.T6.16.12.12.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T6.16.12.12.1.m1.1b"><apply id="S4.T6.16.12.12.1.m1.1.1.cmml" xref="S4.T6.16.12.12.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.16.12.12.1.m1.1.1.1.cmml" xref="S4.T6.16.12.12.1.m1.1.1">subscript</csymbol><ci id="S4.T6.16.12.12.1.m1.1.1.2.cmml" xref="S4.T6.16.12.12.1.m1.1.1.2">𝐴</ci><cn id="S4.T6.16.12.12.1.m1.1.1.3.cmml" type="integer" xref="S4.T6.16.12.12.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.16.12.12.1.m1.1c">A_{2}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.16.12.12.1.m1.1d">italic_A start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S4.T6.16.12.12.2">Claude</td> <td class="ltx_td ltx_align_right" id="S4.T6.16.12.12.3">1.35</td> <td class="ltx_td ltx_align_right" id="S4.T6.16.12.12.4">1.72</td> <td class="ltx_td ltx_align_right" id="S4.T6.16.12.12.5">2.05</td> <td class="ltx_td ltx_align_right" id="S4.T6.16.12.12.6">0.21</td> <td class="ltx_td ltx_align_right" id="S4.T6.16.12.12.7">3.16</td> <td class="ltx_td ltx_align_right" id="S4.T6.16.12.12.8">4.79</td> </tr> <tr class="ltx_tr" id="S4.T6.17.13.13"> <td class="ltx_td ltx_align_left" id="S4.T6.17.13.13.1"><math alttext="A_{3}" class="ltx_Math" display="inline" id="S4.T6.17.13.13.1.m1.1"><semantics id="S4.T6.17.13.13.1.m1.1a"><msub id="S4.T6.17.13.13.1.m1.1.1" xref="S4.T6.17.13.13.1.m1.1.1.cmml"><mi id="S4.T6.17.13.13.1.m1.1.1.2" xref="S4.T6.17.13.13.1.m1.1.1.2.cmml">A</mi><mn id="S4.T6.17.13.13.1.m1.1.1.3" xref="S4.T6.17.13.13.1.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T6.17.13.13.1.m1.1b"><apply id="S4.T6.17.13.13.1.m1.1.1.cmml" xref="S4.T6.17.13.13.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.17.13.13.1.m1.1.1.1.cmml" xref="S4.T6.17.13.13.1.m1.1.1">subscript</csymbol><ci id="S4.T6.17.13.13.1.m1.1.1.2.cmml" xref="S4.T6.17.13.13.1.m1.1.1.2">𝐴</ci><cn id="S4.T6.17.13.13.1.m1.1.1.3.cmml" type="integer" xref="S4.T6.17.13.13.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.17.13.13.1.m1.1c">A_{3}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.17.13.13.1.m1.1d">italic_A start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left" id="S4.T6.17.13.13.2">Console</td> <td class="ltx_td ltx_align_right" id="S4.T6.17.13.13.3">0.94</td> <td class="ltx_td ltx_align_right" id="S4.T6.17.13.13.4">0.34</td> <td class="ltx_td ltx_align_right" id="S4.T6.17.13.13.5">0.58</td> <td class="ltx_td ltx_align_right" id="S4.T6.17.13.13.6">-</td> <td class="ltx_td ltx_align_right" id="S4.T6.17.13.13.7">2.05</td> <td class="ltx_td ltx_align_right" id="S4.T6.17.13.13.8">5.73</td> </tr> <tr class="ltx_tr" id="S4.T6.18.14.14"> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T6.18.14.14.1"><math alttext="C_{1}" class="ltx_Math" display="inline" id="S4.T6.18.14.14.1.m1.1"><semantics id="S4.T6.18.14.14.1.m1.1a"><msub id="S4.T6.18.14.14.1.m1.1.1" xref="S4.T6.18.14.14.1.m1.1.1.cmml"><mi id="S4.T6.18.14.14.1.m1.1.1.2" xref="S4.T6.18.14.14.1.m1.1.1.2.cmml">C</mi><mn id="S4.T6.18.14.14.1.m1.1.1.3" xref="S4.T6.18.14.14.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T6.18.14.14.1.m1.1b"><apply id="S4.T6.18.14.14.1.m1.1.1.cmml" xref="S4.T6.18.14.14.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T6.18.14.14.1.m1.1.1.1.cmml" xref="S4.T6.18.14.14.1.m1.1.1">subscript</csymbol><ci id="S4.T6.18.14.14.1.m1.1.1.2.cmml" xref="S4.T6.18.14.14.1.m1.1.1.2">𝐶</ci><cn id="S4.T6.18.14.14.1.m1.1.1.3.cmml" type="integer" xref="S4.T6.18.14.14.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.18.14.14.1.m1.1c">C_{1}</annotation><annotation encoding="application/x-llamapun" id="S4.T6.18.14.14.1.m1.1d">italic_C start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T6.18.14.14.2">Character.AI</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.14.3">0.40</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.14.4">0.50</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.14.5">1.73</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.14.6">3.61</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.14.7">3.95</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.14.8">8.74</td> </tr> <tr class="ltx_tr" id="S4.T6.18.14.15"> <td class="ltx_td ltx_border_t" id="S4.T6.18.14.15.1"></td> <td class="ltx_td ltx_align_left ltx_border_t" id="S4.T6.18.14.15.2">Arith. Mean</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.15.3">0.84</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.15.4">1.40</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.15.5">1.58</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.15.6">4.01</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.15.7">2.94</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S4.T6.18.14.15.8">7.41</td> </tr> <tr class="ltx_tr" id="S4.T6.18.14.16"> <td class="ltx_td ltx_border_bb" id="S4.T6.18.14.16.1"></td> <td class="ltx_td ltx_align_left ltx_border_bb" id="S4.T6.18.14.16.2">Geom. Mean</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S4.T6.18.14.16.3">0.53</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S4.T6.18.14.16.4">1.15</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S4.T6.18.14.16.5">0.87</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S4.T6.18.14.16.6">3.45</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S4.T6.18.14.16.7">3.99</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S4.T6.18.14.16.8">3.26</td> </tr> </table> </div> </figure> <figure class="ltx_figure" id="S4.F4"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="363" id="S4.F4.g1" src="x4.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 4. Percent of time spent in the Investigating, Repairing, and Checking periods, from the overall duration for failure resolution [%].</figcaption> </figure> <div class="ltx_para" id="S4.p3"> Updated status information is important for users waiting for a service to recover so that they can plan their work, recovery, and communication with customers. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S3.T5" title="In 3.2. Data Collection and Dataset Preparation ‣ 3. Dataset Collection and Preparation ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 5</a> gives the numbers and percent of different combinations of statuses for all reports. In most cases, incident reports do not include every status. Cases with <math alttext="S_{5}(postmortem)" class="ltx_Math" display="inline" id="S4.p3.1.m1.1"><semantics id="S4.p3.1.m1.1a"><mrow id="S4.p3.1.m1.1.1" xref="S4.p3.1.m1.1.1.cmml"><msub id="S4.p3.1.m1.1.1.3" xref="S4.p3.1.m1.1.1.3.cmml"><mi id="S4.p3.1.m1.1.1.3.2" xref="S4.p3.1.m1.1.1.3.2.cmml">S</mi><mn id="S4.p3.1.m1.1.1.3.3" xref="S4.p3.1.m1.1.1.3.3.cmml">5</mn></msub><mo id="S4.p3.1.m1.1.1.2" xref="S4.p3.1.m1.1.1.2.cmml">⁢</mo><mrow id="S4.p3.1.m1.1.1.1.1" xref="S4.p3.1.m1.1.1.1.1.1.cmml"><mo id="S4.p3.1.m1.1.1.1.1.2" stretchy="false" xref="S4.p3.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.p3.1.m1.1.1.1.1.1" xref="S4.p3.1.m1.1.1.1.1.1.cmml"><mi id="S4.p3.1.m1.1.1.1.1.1.2" xref="S4.p3.1.m1.1.1.1.1.1.2.cmml">p</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.3" xref="S4.p3.1.m1.1.1.1.1.1.3.cmml">o</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1a" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.4" xref="S4.p3.1.m1.1.1.1.1.1.4.cmml">s</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1b" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.5" xref="S4.p3.1.m1.1.1.1.1.1.5.cmml">t</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1c" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.6" xref="S4.p3.1.m1.1.1.1.1.1.6.cmml">m</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1d" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.7" xref="S4.p3.1.m1.1.1.1.1.1.7.cmml">o</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1e" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.8" xref="S4.p3.1.m1.1.1.1.1.1.8.cmml">r</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1f" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.9" xref="S4.p3.1.m1.1.1.1.1.1.9.cmml">t</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1g" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.10" xref="S4.p3.1.m1.1.1.1.1.1.10.cmml">e</mi><mo id="S4.p3.1.m1.1.1.1.1.1.1h" xref="S4.p3.1.m1.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.1.m1.1.1.1.1.1.11" xref="S4.p3.1.m1.1.1.1.1.1.11.cmml">m</mi></mrow><mo id="S4.p3.1.m1.1.1.1.1.3" stretchy="false" xref="S4.p3.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p3.1.m1.1b"><apply id="S4.p3.1.m1.1.1.cmml" xref="S4.p3.1.m1.1.1"><times id="S4.p3.1.m1.1.1.2.cmml" xref="S4.p3.1.m1.1.1.2"></times><apply id="S4.p3.1.m1.1.1.3.cmml" xref="S4.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.p3.1.m1.1.1.3.1.cmml" xref="S4.p3.1.m1.1.1.3">subscript</csymbol><ci id="S4.p3.1.m1.1.1.3.2.cmml" xref="S4.p3.1.m1.1.1.3.2">𝑆</ci><cn id="S4.p3.1.m1.1.1.3.3.cmml" type="integer" xref="S4.p3.1.m1.1.1.3.3">5</cn></apply><apply id="S4.p3.1.m1.1.1.1.1.1.cmml" xref="S4.p3.1.m1.1.1.1.1"><times id="S4.p3.1.m1.1.1.1.1.1.1.cmml" xref="S4.p3.1.m1.1.1.1.1.1.1"></times><ci id="S4.p3.1.m1.1.1.1.1.1.2.cmml" xref="S4.p3.1.m1.1.1.1.1.1.2">𝑝</ci><ci id="S4.p3.1.m1.1.1.1.1.1.3.cmml" xref="S4.p3.1.m1.1.1.1.1.1.3">𝑜</ci><ci id="S4.p3.1.m1.1.1.1.1.1.4.cmml" xref="S4.p3.1.m1.1.1.1.1.1.4">𝑠</ci><ci id="S4.p3.1.m1.1.1.1.1.1.5.cmml" xref="S4.p3.1.m1.1.1.1.1.1.5">𝑡</ci><ci id="S4.p3.1.m1.1.1.1.1.1.6.cmml" xref="S4.p3.1.m1.1.1.1.1.1.6">𝑚</ci><ci id="S4.p3.1.m1.1.1.1.1.1.7.cmml" xref="S4.p3.1.m1.1.1.1.1.1.7">𝑜</ci><ci id="S4.p3.1.m1.1.1.1.1.1.8.cmml" xref="S4.p3.1.m1.1.1.1.1.1.8">𝑟</ci><ci id="S4.p3.1.m1.1.1.1.1.1.9.cmml" xref="S4.p3.1.m1.1.1.1.1.1.9">𝑡</ci><ci id="S4.p3.1.m1.1.1.1.1.1.10.cmml" xref="S4.p3.1.m1.1.1.1.1.1.10">𝑒</ci><ci id="S4.p3.1.m1.1.1.1.1.1.11.cmml" xref="S4.p3.1.m1.1.1.1.1.1.11">𝑚</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.1.m1.1c">S_{5}(postmortem)</annotation><annotation encoding="application/x-llamapun" id="S4.p3.1.m1.1d">italic_S start_POSTSUBSCRIPT 5 end_POSTSUBSCRIPT ( italic_p italic_o italic_s italic_t italic_m italic_o italic_r italic_t italic_e italic_m )</annotation></semantics></math> account for the fewest percent, with only 6.15% of incident reports having a postmortem. The most prevailing case combination is <math alttext="S_{1}" class="ltx_Math" display="inline" id="S4.p3.2.m2.1"><semantics id="S4.p3.2.m2.1a"><msub id="S4.p3.2.m2.1.1" xref="S4.p3.2.m2.1.1.cmml"><mi id="S4.p3.2.m2.1.1.2" xref="S4.p3.2.m2.1.1.2.cmml">S</mi><mn id="S4.p3.2.m2.1.1.3" xref="S4.p3.2.m2.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p3.2.m2.1b"><apply id="S4.p3.2.m2.1.1.cmml" xref="S4.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S4.p3.2.m2.1.1.1.cmml" xref="S4.p3.2.m2.1.1">subscript</csymbol><ci id="S4.p3.2.m2.1.1.2.cmml" xref="S4.p3.2.m2.1.1.2">𝑆</ci><cn id="S4.p3.2.m2.1.1.3.cmml" type="integer" xref="S4.p3.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.2.m2.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S4.p3.2.m2.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>-<math alttext="S_{3}" class="ltx_Math" display="inline" id="S4.p3.3.m3.1"><semantics id="S4.p3.3.m3.1a"><msub id="S4.p3.3.m3.1.1" xref="S4.p3.3.m3.1.1.cmml"><mi id="S4.p3.3.m3.1.1.2" xref="S4.p3.3.m3.1.1.2.cmml">S</mi><mn id="S4.p3.3.m3.1.1.3" xref="S4.p3.3.m3.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p3.3.m3.1b"><apply id="S4.p3.3.m3.1.1.cmml" xref="S4.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S4.p3.3.m3.1.1.1.cmml" xref="S4.p3.3.m3.1.1">subscript</csymbol><ci id="S4.p3.3.m3.1.1.2.cmml" xref="S4.p3.3.m3.1.1.2">𝑆</ci><cn id="S4.p3.3.m3.1.1.3.cmml" type="integer" xref="S4.p3.3.m3.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.3.m3.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S4.p3.3.m3.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math>-<math alttext="S_{4}" class="ltx_Math" display="inline" id="S4.p3.4.m4.1"><semantics id="S4.p3.4.m4.1a"><msub id="S4.p3.4.m4.1.1" xref="S4.p3.4.m4.1.1.cmml"><mi id="S4.p3.4.m4.1.1.2" xref="S4.p3.4.m4.1.1.2.cmml">S</mi><mn id="S4.p3.4.m4.1.1.3" xref="S4.p3.4.m4.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p3.4.m4.1b"><apply id="S4.p3.4.m4.1.1.cmml" xref="S4.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S4.p3.4.m4.1.1.1.cmml" xref="S4.p3.4.m4.1.1">subscript</csymbol><ci id="S4.p3.4.m4.1.1.2.cmml" xref="S4.p3.4.m4.1.1.2">𝑆</ci><cn id="S4.p3.4.m4.1.1.3.cmml" type="integer" xref="S4.p3.4.m4.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.4.m4.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S4.p3.4.m4.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math> (24.39%), in which the important status <math alttext="S_{2}(identified)" class="ltx_Math" display="inline" id="S4.p3.5.m5.1"><semantics id="S4.p3.5.m5.1a"><mrow id="S4.p3.5.m5.1.1" xref="S4.p3.5.m5.1.1.cmml"><msub id="S4.p3.5.m5.1.1.3" xref="S4.p3.5.m5.1.1.3.cmml"><mi id="S4.p3.5.m5.1.1.3.2" xref="S4.p3.5.m5.1.1.3.2.cmml">S</mi><mn id="S4.p3.5.m5.1.1.3.3" xref="S4.p3.5.m5.1.1.3.3.cmml">2</mn></msub><mo id="S4.p3.5.m5.1.1.2" xref="S4.p3.5.m5.1.1.2.cmml">⁢</mo><mrow id="S4.p3.5.m5.1.1.1.1" xref="S4.p3.5.m5.1.1.1.1.1.cmml"><mo id="S4.p3.5.m5.1.1.1.1.2" stretchy="false" xref="S4.p3.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="S4.p3.5.m5.1.1.1.1.1" xref="S4.p3.5.m5.1.1.1.1.1.cmml"><mi id="S4.p3.5.m5.1.1.1.1.1.2" xref="S4.p3.5.m5.1.1.1.1.1.2.cmml">i</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.3" xref="S4.p3.5.m5.1.1.1.1.1.3.cmml">d</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1a" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.4" xref="S4.p3.5.m5.1.1.1.1.1.4.cmml">e</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1b" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.5" xref="S4.p3.5.m5.1.1.1.1.1.5.cmml">n</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1c" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.6" xref="S4.p3.5.m5.1.1.1.1.1.6.cmml">t</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1d" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.7" xref="S4.p3.5.m5.1.1.1.1.1.7.cmml">i</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1e" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.8" xref="S4.p3.5.m5.1.1.1.1.1.8.cmml">f</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1f" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.9" xref="S4.p3.5.m5.1.1.1.1.1.9.cmml">i</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1g" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.10" xref="S4.p3.5.m5.1.1.1.1.1.10.cmml">e</mi><mo id="S4.p3.5.m5.1.1.1.1.1.1h" xref="S4.p3.5.m5.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.5.m5.1.1.1.1.1.11" xref="S4.p3.5.m5.1.1.1.1.1.11.cmml">d</mi></mrow><mo id="S4.p3.5.m5.1.1.1.1.3" stretchy="false" xref="S4.p3.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p3.5.m5.1b"><apply id="S4.p3.5.m5.1.1.cmml" xref="S4.p3.5.m5.1.1"><times id="S4.p3.5.m5.1.1.2.cmml" xref="S4.p3.5.m5.1.1.2"></times><apply id="S4.p3.5.m5.1.1.3.cmml" xref="S4.p3.5.m5.1.1.3"><csymbol cd="ambiguous" id="S4.p3.5.m5.1.1.3.1.cmml" xref="S4.p3.5.m5.1.1.3">subscript</csymbol><ci id="S4.p3.5.m5.1.1.3.2.cmml" xref="S4.p3.5.m5.1.1.3.2">𝑆</ci><cn id="S4.p3.5.m5.1.1.3.3.cmml" type="integer" xref="S4.p3.5.m5.1.1.3.3">2</cn></apply><apply id="S4.p3.5.m5.1.1.1.1.1.cmml" xref="S4.p3.5.m5.1.1.1.1"><times id="S4.p3.5.m5.1.1.1.1.1.1.cmml" xref="S4.p3.5.m5.1.1.1.1.1.1"></times><ci id="S4.p3.5.m5.1.1.1.1.1.2.cmml" xref="S4.p3.5.m5.1.1.1.1.1.2">𝑖</ci><ci id="S4.p3.5.m5.1.1.1.1.1.3.cmml" xref="S4.p3.5.m5.1.1.1.1.1.3">𝑑</ci><ci id="S4.p3.5.m5.1.1.1.1.1.4.cmml" xref="S4.p3.5.m5.1.1.1.1.1.4">𝑒</ci><ci id="S4.p3.5.m5.1.1.1.1.1.5.cmml" xref="S4.p3.5.m5.1.1.1.1.1.5">𝑛</ci><ci id="S4.p3.5.m5.1.1.1.1.1.6.cmml" xref="S4.p3.5.m5.1.1.1.1.1.6">𝑡</ci><ci id="S4.p3.5.m5.1.1.1.1.1.7.cmml" xref="S4.p3.5.m5.1.1.1.1.1.7">𝑖</ci><ci id="S4.p3.5.m5.1.1.1.1.1.8.cmml" xref="S4.p3.5.m5.1.1.1.1.1.8">𝑓</ci><ci id="S4.p3.5.m5.1.1.1.1.1.9.cmml" xref="S4.p3.5.m5.1.1.1.1.1.9">𝑖</ci><ci id="S4.p3.5.m5.1.1.1.1.1.10.cmml" xref="S4.p3.5.m5.1.1.1.1.1.10">𝑒</ci><ci id="S4.p3.5.m5.1.1.1.1.1.11.cmml" xref="S4.p3.5.m5.1.1.1.1.1.11">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.5.m5.1c">S_{2}(identified)</annotation><annotation encoding="application/x-llamapun" id="S4.p3.5.m5.1d">italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_i italic_d italic_e italic_n italic_t italic_i italic_f italic_i italic_e italic_d )</annotation></semantics></math> is missing. This means the operators do not communicate to the users that they have identified the issue. 20.48% of cases only provide information about <math alttext="S_{1}(investigating)" class="ltx_Math" display="inline" id="S4.p3.6.m6.1"><semantics id="S4.p3.6.m6.1a"><mrow id="S4.p3.6.m6.1.1" xref="S4.p3.6.m6.1.1.cmml"><msub id="S4.p3.6.m6.1.1.3" xref="S4.p3.6.m6.1.1.3.cmml"><mi id="S4.p3.6.m6.1.1.3.2" xref="S4.p3.6.m6.1.1.3.2.cmml">S</mi><mn id="S4.p3.6.m6.1.1.3.3" xref="S4.p3.6.m6.1.1.3.3.cmml">1</mn></msub><mo id="S4.p3.6.m6.1.1.2" xref="S4.p3.6.m6.1.1.2.cmml">⁢</mo><mrow id="S4.p3.6.m6.1.1.1.1" xref="S4.p3.6.m6.1.1.1.1.1.cmml"><mo id="S4.p3.6.m6.1.1.1.1.2" stretchy="false" xref="S4.p3.6.m6.1.1.1.1.1.cmml">(</mo><mrow id="S4.p3.6.m6.1.1.1.1.1" xref="S4.p3.6.m6.1.1.1.1.1.cmml"><mi id="S4.p3.6.m6.1.1.1.1.1.2" xref="S4.p3.6.m6.1.1.1.1.1.2.cmml">i</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.3" xref="S4.p3.6.m6.1.1.1.1.1.3.cmml">n</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1a" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.4" xref="S4.p3.6.m6.1.1.1.1.1.4.cmml">v</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1b" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.5" xref="S4.p3.6.m6.1.1.1.1.1.5.cmml">e</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1c" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.6" xref="S4.p3.6.m6.1.1.1.1.1.6.cmml">s</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1d" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.7" xref="S4.p3.6.m6.1.1.1.1.1.7.cmml">t</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1e" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.8" xref="S4.p3.6.m6.1.1.1.1.1.8.cmml">i</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1f" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.9" xref="S4.p3.6.m6.1.1.1.1.1.9.cmml">g</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1g" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.10" xref="S4.p3.6.m6.1.1.1.1.1.10.cmml">a</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1h" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.11" xref="S4.p3.6.m6.1.1.1.1.1.11.cmml">t</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1i" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.12" xref="S4.p3.6.m6.1.1.1.1.1.12.cmml">i</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1j" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.13" xref="S4.p3.6.m6.1.1.1.1.1.13.cmml">n</mi><mo id="S4.p3.6.m6.1.1.1.1.1.1k" xref="S4.p3.6.m6.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.6.m6.1.1.1.1.1.14" xref="S4.p3.6.m6.1.1.1.1.1.14.cmml">g</mi></mrow><mo id="S4.p3.6.m6.1.1.1.1.3" stretchy="false" xref="S4.p3.6.m6.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p3.6.m6.1b"><apply id="S4.p3.6.m6.1.1.cmml" xref="S4.p3.6.m6.1.1"><times id="S4.p3.6.m6.1.1.2.cmml" xref="S4.p3.6.m6.1.1.2"></times><apply id="S4.p3.6.m6.1.1.3.cmml" xref="S4.p3.6.m6.1.1.3"><csymbol cd="ambiguous" id="S4.p3.6.m6.1.1.3.1.cmml" xref="S4.p3.6.m6.1.1.3">subscript</csymbol><ci id="S4.p3.6.m6.1.1.3.2.cmml" xref="S4.p3.6.m6.1.1.3.2">𝑆</ci><cn id="S4.p3.6.m6.1.1.3.3.cmml" type="integer" xref="S4.p3.6.m6.1.1.3.3">1</cn></apply><apply id="S4.p3.6.m6.1.1.1.1.1.cmml" xref="S4.p3.6.m6.1.1.1.1"><times id="S4.p3.6.m6.1.1.1.1.1.1.cmml" xref="S4.p3.6.m6.1.1.1.1.1.1"></times><ci id="S4.p3.6.m6.1.1.1.1.1.2.cmml" xref="S4.p3.6.m6.1.1.1.1.1.2">𝑖</ci><ci id="S4.p3.6.m6.1.1.1.1.1.3.cmml" xref="S4.p3.6.m6.1.1.1.1.1.3">𝑛</ci><ci id="S4.p3.6.m6.1.1.1.1.1.4.cmml" xref="S4.p3.6.m6.1.1.1.1.1.4">𝑣</ci><ci id="S4.p3.6.m6.1.1.1.1.1.5.cmml" xref="S4.p3.6.m6.1.1.1.1.1.5">𝑒</ci><ci id="S4.p3.6.m6.1.1.1.1.1.6.cmml" xref="S4.p3.6.m6.1.1.1.1.1.6">𝑠</ci><ci id="S4.p3.6.m6.1.1.1.1.1.7.cmml" xref="S4.p3.6.m6.1.1.1.1.1.7">𝑡</ci><ci id="S4.p3.6.m6.1.1.1.1.1.8.cmml" xref="S4.p3.6.m6.1.1.1.1.1.8">𝑖</ci><ci id="S4.p3.6.m6.1.1.1.1.1.9.cmml" xref="S4.p3.6.m6.1.1.1.1.1.9">𝑔</ci><ci id="S4.p3.6.m6.1.1.1.1.1.10.cmml" xref="S4.p3.6.m6.1.1.1.1.1.10">𝑎</ci><ci id="S4.p3.6.m6.1.1.1.1.1.11.cmml" xref="S4.p3.6.m6.1.1.1.1.1.11">𝑡</ci><ci id="S4.p3.6.m6.1.1.1.1.1.12.cmml" xref="S4.p3.6.m6.1.1.1.1.1.12">𝑖</ci><ci id="S4.p3.6.m6.1.1.1.1.1.13.cmml" xref="S4.p3.6.m6.1.1.1.1.1.13">𝑛</ci><ci id="S4.p3.6.m6.1.1.1.1.1.14.cmml" xref="S4.p3.6.m6.1.1.1.1.1.14">𝑔</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.6.m6.1c">S_{1}(investigating)</annotation><annotation encoding="application/x-llamapun" id="S4.p3.6.m6.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_i italic_n italic_v italic_e italic_s italic_t italic_i italic_g italic_a italic_t italic_i italic_n italic_g )</annotation></semantics></math> and <math alttext="S_{4}(resolved)" class="ltx_Math" display="inline" id="S4.p3.7.m7.1"><semantics id="S4.p3.7.m7.1a"><mrow id="S4.p3.7.m7.1.1" xref="S4.p3.7.m7.1.1.cmml"><msub id="S4.p3.7.m7.1.1.3" xref="S4.p3.7.m7.1.1.3.cmml"><mi id="S4.p3.7.m7.1.1.3.2" xref="S4.p3.7.m7.1.1.3.2.cmml">S</mi><mn id="S4.p3.7.m7.1.1.3.3" xref="S4.p3.7.m7.1.1.3.3.cmml">4</mn></msub><mo id="S4.p3.7.m7.1.1.2" xref="S4.p3.7.m7.1.1.2.cmml">⁢</mo><mrow id="S4.p3.7.m7.1.1.1.1" xref="S4.p3.7.m7.1.1.1.1.1.cmml"><mo id="S4.p3.7.m7.1.1.1.1.2" stretchy="false" xref="S4.p3.7.m7.1.1.1.1.1.cmml">(</mo><mrow id="S4.p3.7.m7.1.1.1.1.1" xref="S4.p3.7.m7.1.1.1.1.1.cmml"><mi id="S4.p3.7.m7.1.1.1.1.1.2" xref="S4.p3.7.m7.1.1.1.1.1.2.cmml">r</mi><mo id="S4.p3.7.m7.1.1.1.1.1.1" xref="S4.p3.7.m7.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.7.m7.1.1.1.1.1.3" xref="S4.p3.7.m7.1.1.1.1.1.3.cmml">e</mi><mo id="S4.p3.7.m7.1.1.1.1.1.1a" xref="S4.p3.7.m7.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.7.m7.1.1.1.1.1.4" xref="S4.p3.7.m7.1.1.1.1.1.4.cmml">s</mi><mo id="S4.p3.7.m7.1.1.1.1.1.1b" xref="S4.p3.7.m7.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.7.m7.1.1.1.1.1.5" xref="S4.p3.7.m7.1.1.1.1.1.5.cmml">o</mi><mo id="S4.p3.7.m7.1.1.1.1.1.1c" xref="S4.p3.7.m7.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.7.m7.1.1.1.1.1.6" xref="S4.p3.7.m7.1.1.1.1.1.6.cmml">l</mi><mo id="S4.p3.7.m7.1.1.1.1.1.1d" xref="S4.p3.7.m7.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.7.m7.1.1.1.1.1.7" xref="S4.p3.7.m7.1.1.1.1.1.7.cmml">v</mi><mo id="S4.p3.7.m7.1.1.1.1.1.1e" xref="S4.p3.7.m7.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.7.m7.1.1.1.1.1.8" xref="S4.p3.7.m7.1.1.1.1.1.8.cmml">e</mi><mo id="S4.p3.7.m7.1.1.1.1.1.1f" xref="S4.p3.7.m7.1.1.1.1.1.1.cmml">⁢</mo><mi id="S4.p3.7.m7.1.1.1.1.1.9" xref="S4.p3.7.m7.1.1.1.1.1.9.cmml">d</mi></mrow><mo id="S4.p3.7.m7.1.1.1.1.3" stretchy="false" xref="S4.p3.7.m7.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p3.7.m7.1b"><apply id="S4.p3.7.m7.1.1.cmml" xref="S4.p3.7.m7.1.1"><times id="S4.p3.7.m7.1.1.2.cmml" xref="S4.p3.7.m7.1.1.2"></times><apply id="S4.p3.7.m7.1.1.3.cmml" xref="S4.p3.7.m7.1.1.3"><csymbol cd="ambiguous" id="S4.p3.7.m7.1.1.3.1.cmml" xref="S4.p3.7.m7.1.1.3">subscript</csymbol><ci id="S4.p3.7.m7.1.1.3.2.cmml" xref="S4.p3.7.m7.1.1.3.2">𝑆</ci><cn id="S4.p3.7.m7.1.1.3.3.cmml" type="integer" xref="S4.p3.7.m7.1.1.3.3">4</cn></apply><apply id="S4.p3.7.m7.1.1.1.1.1.cmml" xref="S4.p3.7.m7.1.1.1.1"><times id="S4.p3.7.m7.1.1.1.1.1.1.cmml" xref="S4.p3.7.m7.1.1.1.1.1.1"></times><ci id="S4.p3.7.m7.1.1.1.1.1.2.cmml" xref="S4.p3.7.m7.1.1.1.1.1.2">𝑟</ci><ci id="S4.p3.7.m7.1.1.1.1.1.3.cmml" xref="S4.p3.7.m7.1.1.1.1.1.3">𝑒</ci><ci id="S4.p3.7.m7.1.1.1.1.1.4.cmml" xref="S4.p3.7.m7.1.1.1.1.1.4">𝑠</ci><ci id="S4.p3.7.m7.1.1.1.1.1.5.cmml" xref="S4.p3.7.m7.1.1.1.1.1.5">𝑜</ci><ci id="S4.p3.7.m7.1.1.1.1.1.6.cmml" xref="S4.p3.7.m7.1.1.1.1.1.6">𝑙</ci><ci id="S4.p3.7.m7.1.1.1.1.1.7.cmml" xref="S4.p3.7.m7.1.1.1.1.1.7">𝑣</ci><ci id="S4.p3.7.m7.1.1.1.1.1.8.cmml" xref="S4.p3.7.m7.1.1.1.1.1.8">𝑒</ci><ci id="S4.p3.7.m7.1.1.1.1.1.9.cmml" xref="S4.p3.7.m7.1.1.1.1.1.9">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.7.m7.1c">S_{4}(resolved)</annotation><annotation encoding="application/x-llamapun" id="S4.p3.7.m7.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT ( italic_r italic_e italic_s italic_o italic_l italic_v italic_e italic_d )</annotation></semantics></math> statuses. 14.34% of reports update every status (<math alttext="S_{1}" class="ltx_Math" display="inline" id="S4.p3.8.m8.1"><semantics id="S4.p3.8.m8.1a"><msub id="S4.p3.8.m8.1.1" xref="S4.p3.8.m8.1.1.cmml"><mi id="S4.p3.8.m8.1.1.2" xref="S4.p3.8.m8.1.1.2.cmml">S</mi><mn id="S4.p3.8.m8.1.1.3" xref="S4.p3.8.m8.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p3.8.m8.1b"><apply id="S4.p3.8.m8.1.1.cmml" xref="S4.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S4.p3.8.m8.1.1.1.cmml" xref="S4.p3.8.m8.1.1">subscript</csymbol><ci id="S4.p3.8.m8.1.1.2.cmml" xref="S4.p3.8.m8.1.1.2">𝑆</ci><cn id="S4.p3.8.m8.1.1.3.cmml" type="integer" xref="S4.p3.8.m8.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.8.m8.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S4.p3.8.m8.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>-<math alttext="S_{2}" class="ltx_Math" display="inline" id="S4.p3.9.m9.1"><semantics id="S4.p3.9.m9.1a"><msub id="S4.p3.9.m9.1.1" xref="S4.p3.9.m9.1.1.cmml"><mi id="S4.p3.9.m9.1.1.2" xref="S4.p3.9.m9.1.1.2.cmml">S</mi><mn id="S4.p3.9.m9.1.1.3" xref="S4.p3.9.m9.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p3.9.m9.1b"><apply id="S4.p3.9.m9.1.1.cmml" xref="S4.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S4.p3.9.m9.1.1.1.cmml" xref="S4.p3.9.m9.1.1">subscript</csymbol><ci id="S4.p3.9.m9.1.1.2.cmml" xref="S4.p3.9.m9.1.1.2">𝑆</ci><cn id="S4.p3.9.m9.1.1.3.cmml" type="integer" xref="S4.p3.9.m9.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.9.m9.1c">S_{2}</annotation><annotation encoding="application/x-llamapun" id="S4.p3.9.m9.1d">italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT</annotation></semantics></math>-<math alttext="S_{3}" class="ltx_Math" display="inline" id="S4.p3.10.m10.1"><semantics id="S4.p3.10.m10.1a"><msub id="S4.p3.10.m10.1.1" xref="S4.p3.10.m10.1.1.cmml"><mi id="S4.p3.10.m10.1.1.2" xref="S4.p3.10.m10.1.1.2.cmml">S</mi><mn id="S4.p3.10.m10.1.1.3" xref="S4.p3.10.m10.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p3.10.m10.1b"><apply id="S4.p3.10.m10.1.1.cmml" xref="S4.p3.10.m10.1.1"><csymbol cd="ambiguous" id="S4.p3.10.m10.1.1.1.cmml" xref="S4.p3.10.m10.1.1">subscript</csymbol><ci id="S4.p3.10.m10.1.1.2.cmml" xref="S4.p3.10.m10.1.1.2">𝑆</ci><cn id="S4.p3.10.m10.1.1.3.cmml" type="integer" xref="S4.p3.10.m10.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.10.m10.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S4.p3.10.m10.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math>-<math alttext="S_{4}" class="ltx_Math" display="inline" id="S4.p3.11.m11.1"><semantics id="S4.p3.11.m11.1a"><msub id="S4.p3.11.m11.1.1" xref="S4.p3.11.m11.1.1.cmml"><mi id="S4.p3.11.m11.1.1.2" xref="S4.p3.11.m11.1.1.2.cmml">S</mi><mn id="S4.p3.11.m11.1.1.3" xref="S4.p3.11.m11.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p3.11.m11.1b"><apply id="S4.p3.11.m11.1.1.cmml" xref="S4.p3.11.m11.1.1"><csymbol cd="ambiguous" id="S4.p3.11.m11.1.1.1.cmml" xref="S4.p3.11.m11.1.1">subscript</csymbol><ci id="S4.p3.11.m11.1.1.2.cmml" xref="S4.p3.11.m11.1.1.2">𝑆</ci><cn id="S4.p3.11.m11.1.1.3.cmml" type="integer" xref="S4.p3.11.m11.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.11.m11.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S4.p3.11.m11.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math>) throughout the duration of incidents, while 11.55% only update once the incidents have been resolved (<math alttext="S_{4}" class="ltx_Math" display="inline" id="S4.p3.12.m12.1"><semantics id="S4.p3.12.m12.1a"><msub id="S4.p3.12.m12.1.1" xref="S4.p3.12.m12.1.1.cmml"><mi id="S4.p3.12.m12.1.1.2" xref="S4.p3.12.m12.1.1.2.cmml">S</mi><mn id="S4.p3.12.m12.1.1.3" xref="S4.p3.12.m12.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p3.12.m12.1b"><apply id="S4.p3.12.m12.1.1.cmml" xref="S4.p3.12.m12.1.1"><csymbol cd="ambiguous" id="S4.p3.12.m12.1.1.1.cmml" xref="S4.p3.12.m12.1.1">subscript</csymbol><ci id="S4.p3.12.m12.1.1.2.cmml" xref="S4.p3.12.m12.1.1.2">𝑆</ci><cn id="S4.p3.12.m12.1.1.3.cmml" type="integer" xref="S4.p3.12.m12.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.12.m12.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S4.p3.12.m12.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math>). </div> <div class="ltx_para" id="S4.p4"> These results indicate that operators fail to communicate the state of a failure to the user and update the user about potential fix times. Users must create their own failure models <cite class="ltx_cite ltx_citemacro_citep">(Garg et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib18" title="">2018</a>)</cite> and fault-tolerance systems <cite class="ltx_cite ltx_citemacro_citep">(Primorac et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib44" title="">2021</a>)</cite> and expect little input from the operator. </div> <figure class="ltx_figure ltx_minipage ltx_align_center ltx_align_middle" id="S4.F5" style="width:433.6pt;"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_1"> <figure class="ltx_figure ltx_figure_panel" id="S4.F5.sf1"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="498" id="S4.F5.sf1.g1" src="x5.png" width="830"/> <figcaption class="ltx_caption">(a) Mean Time To Resolve (Shorter is better).</figcaption> </figure> </div> <div class="ltx_flex_break"></div> <div class="ltx_flex_cell ltx_flex_size_1"> <figure class="ltx_figure ltx_figure_panel" id="S4.F5.sf2"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="498" id="S4.F5.sf2.g1" src="x6.png" width="830"/> <figcaption class="ltx_caption">(b) Mean Time Between Failures (Longer is better).</figcaption> </figure> </div> </div> <figcaption class="ltx_caption ltx_centering">Figure 5. Distribution of MTTR and MTBF by service, with median values indicated.</figcaption> </figure> <div class="ltx_para ltx_noindent" id="S4.p5"> Observation #2: ChatGPT includes postmortems in 12.91% of its reports. Anthropic’s reports contain the fewest postmortems, with none provided for its API and Console services. </div> <div class="ltx_para" id="S4.p6"> The status combinations also vary depending on different services, as <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S3.F3" title="In 3.2. Data Collection and Dataset Preparation ‣ 3. Dataset Collection and Preparation ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 3</a> shows. The primary status combination for OpenAI services is <math alttext="S_{1}" class="ltx_Math" display="inline" id="S4.p6.1.m1.1"><semantics id="S4.p6.1.m1.1a"><msub id="S4.p6.1.m1.1.1" xref="S4.p6.1.m1.1.1.cmml"><mi id="S4.p6.1.m1.1.1.2" xref="S4.p6.1.m1.1.1.2.cmml">S</mi><mn id="S4.p6.1.m1.1.1.3" xref="S4.p6.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p6.1.m1.1b"><apply id="S4.p6.1.m1.1.1.cmml" xref="S4.p6.1.m1.1.1"><csymbol cd="ambiguous" id="S4.p6.1.m1.1.1.1.cmml" xref="S4.p6.1.m1.1.1">subscript</csymbol><ci id="S4.p6.1.m1.1.1.2.cmml" xref="S4.p6.1.m1.1.1.2">𝑆</ci><cn id="S4.p6.1.m1.1.1.3.cmml" type="integer" xref="S4.p6.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.1.m1.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S4.p6.1.m1.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>-<math alttext="S_{3}" class="ltx_Math" display="inline" id="S4.p6.2.m2.1"><semantics id="S4.p6.2.m2.1a"><msub id="S4.p6.2.m2.1.1" xref="S4.p6.2.m2.1.1.cmml"><mi id="S4.p6.2.m2.1.1.2" xref="S4.p6.2.m2.1.1.2.cmml">S</mi><mn id="S4.p6.2.m2.1.1.3" xref="S4.p6.2.m2.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p6.2.m2.1b"><apply id="S4.p6.2.m2.1.1.cmml" xref="S4.p6.2.m2.1.1"><csymbol cd="ambiguous" id="S4.p6.2.m2.1.1.1.cmml" xref="S4.p6.2.m2.1.1">subscript</csymbol><ci id="S4.p6.2.m2.1.1.2.cmml" xref="S4.p6.2.m2.1.1.2">𝑆</ci><cn id="S4.p6.2.m2.1.1.3.cmml" type="integer" xref="S4.p6.2.m2.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.2.m2.1c">S_{3}</annotation><annotation encoding="application/x-llamapun" id="S4.p6.2.m2.1d">italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT</annotation></semantics></math>-<math alttext="S_{4}" class="ltx_Math" display="inline" id="S4.p6.3.m3.1"><semantics id="S4.p6.3.m3.1a"><msub id="S4.p6.3.m3.1.1" xref="S4.p6.3.m3.1.1.cmml"><mi id="S4.p6.3.m3.1.1.2" xref="S4.p6.3.m3.1.1.2.cmml">S</mi><mn id="S4.p6.3.m3.1.1.3" xref="S4.p6.3.m3.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p6.3.m3.1b"><apply id="S4.p6.3.m3.1.1.cmml" xref="S4.p6.3.m3.1.1"><csymbol cd="ambiguous" id="S4.p6.3.m3.1.1.1.cmml" xref="S4.p6.3.m3.1.1">subscript</csymbol><ci id="S4.p6.3.m3.1.1.2.cmml" xref="S4.p6.3.m3.1.1.2">𝑆</ci><cn id="S4.p6.3.m3.1.1.3.cmml" type="integer" xref="S4.p6.3.m3.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.3.m3.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S4.p6.3.m3.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math>, representing 29.96% for API, 41.13% for ChatGPT, 38.71% for DALL·E, and 31.25% for Playground. In contrast, Anthropic and Character.AI primarily use the <math alttext="S_{1}" class="ltx_Math" display="inline" id="S4.p6.4.m4.1"><semantics id="S4.p6.4.m4.1a"><msub id="S4.p6.4.m4.1.1" xref="S4.p6.4.m4.1.1.cmml"><mi id="S4.p6.4.m4.1.1.2" xref="S4.p6.4.m4.1.1.2.cmml">S</mi><mn id="S4.p6.4.m4.1.1.3" xref="S4.p6.4.m4.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p6.4.m4.1b"><apply id="S4.p6.4.m4.1.1.cmml" xref="S4.p6.4.m4.1.1"><csymbol cd="ambiguous" id="S4.p6.4.m4.1.1.1.cmml" xref="S4.p6.4.m4.1.1">subscript</csymbol><ci id="S4.p6.4.m4.1.1.2.cmml" xref="S4.p6.4.m4.1.1.2">𝑆</ci><cn id="S4.p6.4.m4.1.1.3.cmml" type="integer" xref="S4.p6.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.4.m4.1c">S_{1}</annotation><annotation encoding="application/x-llamapun" id="S4.p6.4.m4.1d">italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT</annotation></semantics></math>-<math alttext="S_{4}" class="ltx_Math" display="inline" id="S4.p6.5.m5.1"><semantics id="S4.p6.5.m5.1a"><msub id="S4.p6.5.m5.1.1" xref="S4.p6.5.m5.1.1.cmml"><mi id="S4.p6.5.m5.1.1.2" xref="S4.p6.5.m5.1.1.2.cmml">S</mi><mn id="S4.p6.5.m5.1.1.3" xref="S4.p6.5.m5.1.1.3.cmml">4</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p6.5.m5.1b"><apply id="S4.p6.5.m5.1.1.cmml" xref="S4.p6.5.m5.1.1"><csymbol cd="ambiguous" id="S4.p6.5.m5.1.1.1.cmml" xref="S4.p6.5.m5.1.1">subscript</csymbol><ci id="S4.p6.5.m5.1.1.2.cmml" xref="S4.p6.5.m5.1.1.2">𝑆</ci><cn id="S4.p6.5.m5.1.1.3.cmml" type="integer" xref="S4.p6.5.m5.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.5.m5.1c">S_{4}</annotation><annotation encoding="application/x-llamapun" id="S4.p6.5.m5.1d">italic_S start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT</annotation></semantics></math> combination, accounting for over 30% of each service. OpenAI publishes more status information in incident reports compared to Anthropic and Character.AI. </div> <div class="ltx_para ltx_noindent" id="S4.p7"> Observation #3: Claude spent the longest time on investigating (1.35 hours), repairing (1.72 hours), and checking (2.05 hours). </div> <div class="ltx_para" id="S4.p8"> The time a service takes to resolve incidents affects the fault-tolerance strategies a user can use. For example, users can maintain a local cache to tolerate very short failures. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.T6" title="In 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 6</a> shows the mean value of different model parameters. For MTTR and MTBF, because some records don’t have <math alttext="investigating" class="ltx_Math" display="inline" id="S4.p8.1.m1.1"><semantics id="S4.p8.1.m1.1a"><mrow id="S4.p8.1.m1.1.1" xref="S4.p8.1.m1.1.1.cmml"><mi id="S4.p8.1.m1.1.1.2" xref="S4.p8.1.m1.1.1.2.cmml">i</mi><mo id="S4.p8.1.m1.1.1.1" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.3" xref="S4.p8.1.m1.1.1.3.cmml">n</mi><mo id="S4.p8.1.m1.1.1.1a" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.4" xref="S4.p8.1.m1.1.1.4.cmml">v</mi><mo id="S4.p8.1.m1.1.1.1b" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.5" xref="S4.p8.1.m1.1.1.5.cmml">e</mi><mo id="S4.p8.1.m1.1.1.1c" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.6" xref="S4.p8.1.m1.1.1.6.cmml">s</mi><mo id="S4.p8.1.m1.1.1.1d" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.7" xref="S4.p8.1.m1.1.1.7.cmml">t</mi><mo id="S4.p8.1.m1.1.1.1e" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.8" xref="S4.p8.1.m1.1.1.8.cmml">i</mi><mo id="S4.p8.1.m1.1.1.1f" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.9" xref="S4.p8.1.m1.1.1.9.cmml">g</mi><mo id="S4.p8.1.m1.1.1.1g" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.10" xref="S4.p8.1.m1.1.1.10.cmml">a</mi><mo id="S4.p8.1.m1.1.1.1h" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.11" xref="S4.p8.1.m1.1.1.11.cmml">t</mi><mo id="S4.p8.1.m1.1.1.1i" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.12" xref="S4.p8.1.m1.1.1.12.cmml">i</mi><mo id="S4.p8.1.m1.1.1.1j" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.13" xref="S4.p8.1.m1.1.1.13.cmml">n</mi><mo id="S4.p8.1.m1.1.1.1k" xref="S4.p8.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p8.1.m1.1.1.14" xref="S4.p8.1.m1.1.1.14.cmml">g</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p8.1.m1.1b"><apply id="S4.p8.1.m1.1.1.cmml" xref="S4.p8.1.m1.1.1"><times id="S4.p8.1.m1.1.1.1.cmml" xref="S4.p8.1.m1.1.1.1"></times><ci id="S4.p8.1.m1.1.1.2.cmml" xref="S4.p8.1.m1.1.1.2">𝑖</ci><ci id="S4.p8.1.m1.1.1.3.cmml" xref="S4.p8.1.m1.1.1.3">𝑛</ci><ci id="S4.p8.1.m1.1.1.4.cmml" xref="S4.p8.1.m1.1.1.4">𝑣</ci><ci id="S4.p8.1.m1.1.1.5.cmml" xref="S4.p8.1.m1.1.1.5">𝑒</ci><ci id="S4.p8.1.m1.1.1.6.cmml" xref="S4.p8.1.m1.1.1.6">𝑠</ci><ci id="S4.p8.1.m1.1.1.7.cmml" xref="S4.p8.1.m1.1.1.7">𝑡</ci><ci id="S4.p8.1.m1.1.1.8.cmml" xref="S4.p8.1.m1.1.1.8">𝑖</ci><ci id="S4.p8.1.m1.1.1.9.cmml" xref="S4.p8.1.m1.1.1.9">𝑔</ci><ci id="S4.p8.1.m1.1.1.10.cmml" xref="S4.p8.1.m1.1.1.10">𝑎</ci><ci id="S4.p8.1.m1.1.1.11.cmml" xref="S4.p8.1.m1.1.1.11">𝑡</ci><ci id="S4.p8.1.m1.1.1.12.cmml" xref="S4.p8.1.m1.1.1.12">𝑖</ci><ci id="S4.p8.1.m1.1.1.13.cmml" xref="S4.p8.1.m1.1.1.13">𝑛</ci><ci id="S4.p8.1.m1.1.1.14.cmml" xref="S4.p8.1.m1.1.1.14">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p8.1.m1.1c">investigating</annotation><annotation encoding="application/x-llamapun" id="S4.p8.1.m1.1d">italic_i italic_n italic_v italic_e italic_s italic_t italic_i italic_g italic_a italic_t italic_i italic_n italic_g</annotation></semantics></math> timestamps, we use the minimum ones before the resolved status here (This also explains why Claude has the longest <math alttext="P_{I}" class="ltx_Math" display="inline" id="S4.p8.2.m2.1"><semantics id="S4.p8.2.m2.1a"><msub id="S4.p8.2.m2.1.1" xref="S4.p8.2.m2.1.1.cmml"><mi id="S4.p8.2.m2.1.1.2" xref="S4.p8.2.m2.1.1.2.cmml">P</mi><mi id="S4.p8.2.m2.1.1.3" xref="S4.p8.2.m2.1.1.3.cmml">I</mi></msub><annotation-xml encoding="MathML-Content" id="S4.p8.2.m2.1b"><apply id="S4.p8.2.m2.1.1.cmml" xref="S4.p8.2.m2.1.1"><csymbol cd="ambiguous" id="S4.p8.2.m2.1.1.1.cmml" xref="S4.p8.2.m2.1.1">subscript</csymbol><ci id="S4.p8.2.m2.1.1.2.cmml" xref="S4.p8.2.m2.1.1.2">𝑃</ci><ci id="S4.p8.2.m2.1.1.3.cmml" xref="S4.p8.2.m2.1.1.3">𝐼</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p8.2.m2.1c">P_{I}</annotation><annotation encoding="application/x-llamapun" id="S4.p8.2.m2.1d">italic_P start_POSTSUBSCRIPT italic_I end_POSTSUBSCRIPT</annotation></semantics></math>, <math alttext="P_{R}" class="ltx_Math" display="inline" id="S4.p8.3.m3.1"><semantics id="S4.p8.3.m3.1a"><msub id="S4.p8.3.m3.1.1" xref="S4.p8.3.m3.1.1.cmml"><mi id="S4.p8.3.m3.1.1.2" xref="S4.p8.3.m3.1.1.2.cmml">P</mi><mi id="S4.p8.3.m3.1.1.3" xref="S4.p8.3.m3.1.1.3.cmml">R</mi></msub><annotation-xml encoding="MathML-Content" id="S4.p8.3.m3.1b"><apply id="S4.p8.3.m3.1.1.cmml" xref="S4.p8.3.m3.1.1"><csymbol cd="ambiguous" id="S4.p8.3.m3.1.1.1.cmml" xref="S4.p8.3.m3.1.1">subscript</csymbol><ci id="S4.p8.3.m3.1.1.2.cmml" xref="S4.p8.3.m3.1.1.2">𝑃</ci><ci id="S4.p8.3.m3.1.1.3.cmml" xref="S4.p8.3.m3.1.1.3">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p8.3.m3.1c">P_{R}</annotation><annotation encoding="application/x-llamapun" id="S4.p8.3.m3.1d">italic_P start_POSTSUBSCRIPT italic_R end_POSTSUBSCRIPT</annotation></semantics></math>, and <math alttext="P_{C}" class="ltx_Math" display="inline" id="S4.p8.4.m4.1"><semantics id="S4.p8.4.m4.1a"><msub id="S4.p8.4.m4.1.1" xref="S4.p8.4.m4.1.1.cmml"><mi id="S4.p8.4.m4.1.1.2" xref="S4.p8.4.m4.1.1.2.cmml">P</mi><mi id="S4.p8.4.m4.1.1.3" xref="S4.p8.4.m4.1.1.3.cmml">C</mi></msub><annotation-xml encoding="MathML-Content" id="S4.p8.4.m4.1b"><apply id="S4.p8.4.m4.1.1.cmml" xref="S4.p8.4.m4.1.1"><csymbol cd="ambiguous" id="S4.p8.4.m4.1.1.1.cmml" xref="S4.p8.4.m4.1.1">subscript</csymbol><ci id="S4.p8.4.m4.1.1.2.cmml" xref="S4.p8.4.m4.1.1.2">𝑃</ci><ci id="S4.p8.4.m4.1.1.3.cmml" xref="S4.p8.4.m4.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p8.4.m4.1c">P_{C}</annotation><annotation encoding="application/x-llamapun" id="S4.p8.4.m4.1d">italic_P start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT</annotation></semantics></math> respectively, but it does not have the longest <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.p8.5.m5.1"><semantics id="S4.p8.5.m5.1a"><mrow id="S4.p8.5.m5.1.1" xref="S4.p8.5.m5.1.1.cmml"><mi id="S4.p8.5.m5.1.1.2" xref="S4.p8.5.m5.1.1.2.cmml">M</mi><mo id="S4.p8.5.m5.1.1.1" xref="S4.p8.5.m5.1.1.1.cmml">⁢</mo><mi id="S4.p8.5.m5.1.1.3" xref="S4.p8.5.m5.1.1.3.cmml">T</mi><mo id="S4.p8.5.m5.1.1.1a" xref="S4.p8.5.m5.1.1.1.cmml">⁢</mo><mi id="S4.p8.5.m5.1.1.4" xref="S4.p8.5.m5.1.1.4.cmml">T</mi><mo id="S4.p8.5.m5.1.1.1b" xref="S4.p8.5.m5.1.1.1.cmml">⁢</mo><mi id="S4.p8.5.m5.1.1.5" xref="S4.p8.5.m5.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p8.5.m5.1b"><apply id="S4.p8.5.m5.1.1.cmml" xref="S4.p8.5.m5.1.1"><times id="S4.p8.5.m5.1.1.1.cmml" xref="S4.p8.5.m5.1.1.1"></times><ci id="S4.p8.5.m5.1.1.2.cmml" xref="S4.p8.5.m5.1.1.2">𝑀</ci><ci id="S4.p8.5.m5.1.1.3.cmml" xref="S4.p8.5.m5.1.1.3">𝑇</ci><ci id="S4.p8.5.m5.1.1.4.cmml" xref="S4.p8.5.m5.1.1.4">𝑇</ci><ci id="S4.p8.5.m5.1.1.5.cmml" xref="S4.p8.5.m5.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p8.5.m5.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.p8.5.m5.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math>). The learning period (<math alttext="P_{L}" class="ltx_Math" display="inline" id="S4.p8.6.m6.1"><semantics id="S4.p8.6.m6.1a"><msub id="S4.p8.6.m6.1.1" xref="S4.p8.6.m6.1.1.cmml"><mi id="S4.p8.6.m6.1.1.2" xref="S4.p8.6.m6.1.1.2.cmml">P</mi><mi id="S4.p8.6.m6.1.1.3" xref="S4.p8.6.m6.1.1.3.cmml">L</mi></msub><annotation-xml encoding="MathML-Content" id="S4.p8.6.m6.1b"><apply id="S4.p8.6.m6.1.1.cmml" xref="S4.p8.6.m6.1.1"><csymbol cd="ambiguous" id="S4.p8.6.m6.1.1.1.cmml" xref="S4.p8.6.m6.1.1">subscript</csymbol><ci id="S4.p8.6.m6.1.1.2.cmml" xref="S4.p8.6.m6.1.1.2">𝑃</ci><ci id="S4.p8.6.m6.1.1.3.cmml" xref="S4.p8.6.m6.1.1.3">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p8.6.m6.1c">P_{L}</annotation><annotation encoding="application/x-llamapun" id="S4.p8.6.m6.1d">italic_P start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT</annotation></semantics></math>) takes the longest time (2.94 days) in all services. </div> <div class="ltx_para ltx_noindent" id="S4.p9"> Observation #4: Significant differences are observed across the 8 services in the percentage of periods within failure resolutions. For Character.AI, 82.71% of the time is spent on monitoring and checking if the fix is stable and effective. Anthropic services spent more percent of the time for investigating and resolving issues than OpenAI services. </div> <div class="ltx_para" id="S4.p10"> <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.F4" title="In 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 4</a> shows the percent of the 3 periods (Investigating, Repairing, Checking) in <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.p10.1.m1.1"><semantics id="S4.p10.1.m1.1a"><mrow id="S4.p10.1.m1.1.1" xref="S4.p10.1.m1.1.1.cmml"><mi id="S4.p10.1.m1.1.1.2" xref="S4.p10.1.m1.1.1.2.cmml">M</mi><mo id="S4.p10.1.m1.1.1.1" xref="S4.p10.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p10.1.m1.1.1.3" xref="S4.p10.1.m1.1.1.3.cmml">T</mi><mo id="S4.p10.1.m1.1.1.1a" xref="S4.p10.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p10.1.m1.1.1.4" xref="S4.p10.1.m1.1.1.4.cmml">T</mi><mo id="S4.p10.1.m1.1.1.1b" xref="S4.p10.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p10.1.m1.1.1.5" xref="S4.p10.1.m1.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p10.1.m1.1b"><apply id="S4.p10.1.m1.1.1.cmml" xref="S4.p10.1.m1.1.1"><times id="S4.p10.1.m1.1.1.1.cmml" xref="S4.p10.1.m1.1.1.1"></times><ci id="S4.p10.1.m1.1.1.2.cmml" xref="S4.p10.1.m1.1.1.2">𝑀</ci><ci id="S4.p10.1.m1.1.1.3.cmml" xref="S4.p10.1.m1.1.1.3">𝑇</ci><ci id="S4.p10.1.m1.1.1.4.cmml" xref="S4.p10.1.m1.1.1.4">𝑇</ci><ci id="S4.p10.1.m1.1.1.5.cmml" xref="S4.p10.1.m1.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p10.1.m1.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.p10.1.m1.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math>. The majority of the resolution time is used for checking, ranging from the highest 82.71% for Character.AI to the lowest 43.75% for Console. Most services spent more percent of the time on repairing than investigating, except for API and Console from Anthropic. Anthropic API spent 26.70% of the time investigating issues, while Console spent 43.23% in the same period. The large fraction of the time used for checking indicates that deploying a fix to production takes a long time. Operators should employ faster testing and continuous deployment techniques to deploy fixes faster <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib69" title="">2018</a>)</cite>. However, this is challenging as LLMs are a new technology, and there isn’t much work on improving testing and deployment time. </div> <div class="ltx_para ltx_noindent" id="S4.p11"> Observation #5: OpenAI’s API and ChatGPT recover slower from failures than Anthropic’s API and Claude, with 1.6x and 1.4 longer <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.p11.1.1.1.1.1.m1.1"><semantics id="S4.p11.1.1.1.1.1.m1.1a"><mrow id="S4.p11.1.1.1.1.1.m1.1.1" xref="S4.p11.1.1.1.1.1.m1.1.1.cmml"><mi id="S4.p11.1.1.1.1.1.m1.1.1.2" xref="S4.p11.1.1.1.1.1.m1.1.1.2.cmml">M</mi><mo id="S4.p11.1.1.1.1.1.m1.1.1.1" xref="S4.p11.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p11.1.1.1.1.1.m1.1.1.3" xref="S4.p11.1.1.1.1.1.m1.1.1.3.cmml">T</mi><mo id="S4.p11.1.1.1.1.1.m1.1.1.1a" xref="S4.p11.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p11.1.1.1.1.1.m1.1.1.4" xref="S4.p11.1.1.1.1.1.m1.1.1.4.cmml">T</mi><mo id="S4.p11.1.1.1.1.1.m1.1.1.1b" xref="S4.p11.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p11.1.1.1.1.1.m1.1.1.5" xref="S4.p11.1.1.1.1.1.m1.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p11.1.1.1.1.1.m1.1b"><apply id="S4.p11.1.1.1.1.1.m1.1.1.cmml" xref="S4.p11.1.1.1.1.1.m1.1.1"><times id="S4.p11.1.1.1.1.1.m1.1.1.1.cmml" xref="S4.p11.1.1.1.1.1.m1.1.1.1"></times><ci id="S4.p11.1.1.1.1.1.m1.1.1.2.cmml" xref="S4.p11.1.1.1.1.1.m1.1.1.2">𝑀</ci><ci id="S4.p11.1.1.1.1.1.m1.1.1.3.cmml" xref="S4.p11.1.1.1.1.1.m1.1.1.3">𝑇</ci><ci id="S4.p11.1.1.1.1.1.m1.1.1.4.cmml" xref="S4.p11.1.1.1.1.1.m1.1.1.4">𝑇</ci><ci id="S4.p11.1.1.1.1.1.m1.1.1.5.cmml" xref="S4.p11.1.1.1.1.1.m1.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p11.1.1.1.1.1.m1.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.p11.1.1.1.1.1.m1.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math>, respectively. </div> <div class="ltx_para" id="S4.p12"> <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.F5.sf1" title="In Figure 5 ‣ 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 5(a)</a> depicts <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.p12.1.m1.1"><semantics id="S4.p12.1.m1.1a"><mrow id="S4.p12.1.m1.1.1" xref="S4.p12.1.m1.1.1.cmml"><mi id="S4.p12.1.m1.1.1.2" xref="S4.p12.1.m1.1.1.2.cmml">M</mi><mo id="S4.p12.1.m1.1.1.1" xref="S4.p12.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p12.1.m1.1.1.3" xref="S4.p12.1.m1.1.1.3.cmml">T</mi><mo id="S4.p12.1.m1.1.1.1a" xref="S4.p12.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p12.1.m1.1.1.4" xref="S4.p12.1.m1.1.1.4.cmml">T</mi><mo id="S4.p12.1.m1.1.1.1b" xref="S4.p12.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p12.1.m1.1.1.5" xref="S4.p12.1.m1.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p12.1.m1.1b"><apply id="S4.p12.1.m1.1.1.cmml" xref="S4.p12.1.m1.1.1"><times id="S4.p12.1.m1.1.1.1.cmml" xref="S4.p12.1.m1.1.1.1"></times><ci id="S4.p12.1.m1.1.1.2.cmml" xref="S4.p12.1.m1.1.1.2">𝑀</ci><ci id="S4.p12.1.m1.1.1.3.cmml" xref="S4.p12.1.m1.1.1.3">𝑇</ci><ci id="S4.p12.1.m1.1.1.4.cmml" xref="S4.p12.1.m1.1.1.4">𝑇</ci><ci id="S4.p12.1.m1.1.1.5.cmml" xref="S4.p12.1.m1.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p12.1.m1.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.p12.1.m1.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math> of the 8 LLM service incidents, showing how quickly failures are resolved. Most failures are resolved between 0.5 and 3 hours, with the median values around 1 hour. APIs and chatbots are the most popular LLM services. The median <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.p12.2.m2.1"><semantics id="S4.p12.2.m2.1a"><mrow id="S4.p12.2.m2.1.1" xref="S4.p12.2.m2.1.1.cmml"><mi id="S4.p12.2.m2.1.1.2" xref="S4.p12.2.m2.1.1.2.cmml">M</mi><mo id="S4.p12.2.m2.1.1.1" xref="S4.p12.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p12.2.m2.1.1.3" xref="S4.p12.2.m2.1.1.3.cmml">T</mi><mo id="S4.p12.2.m2.1.1.1a" xref="S4.p12.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p12.2.m2.1.1.4" xref="S4.p12.2.m2.1.1.4.cmml">T</mi><mo id="S4.p12.2.m2.1.1.1b" xref="S4.p12.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p12.2.m2.1.1.5" xref="S4.p12.2.m2.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p12.2.m2.1b"><apply id="S4.p12.2.m2.1.1.cmml" xref="S4.p12.2.m2.1.1"><times id="S4.p12.2.m2.1.1.1.cmml" xref="S4.p12.2.m2.1.1.1"></times><ci id="S4.p12.2.m2.1.1.2.cmml" xref="S4.p12.2.m2.1.1.2">𝑀</ci><ci id="S4.p12.2.m2.1.1.3.cmml" xref="S4.p12.2.m2.1.1.3">𝑇</ci><ci id="S4.p12.2.m2.1.1.4.cmml" xref="S4.p12.2.m2.1.1.4">𝑇</ci><ci id="S4.p12.2.m2.1.1.5.cmml" xref="S4.p12.2.m2.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p12.2.m2.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.p12.2.m2.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math> of OpenAI API (1.23 hours) is 1.6x longer than Anthropic API (0.77 hours). Similarly, the median <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.p12.3.m3.1"><semantics id="S4.p12.3.m3.1a"><mrow id="S4.p12.3.m3.1.1" xref="S4.p12.3.m3.1.1.cmml"><mi id="S4.p12.3.m3.1.1.2" xref="S4.p12.3.m3.1.1.2.cmml">M</mi><mo id="S4.p12.3.m3.1.1.1" xref="S4.p12.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p12.3.m3.1.1.3" xref="S4.p12.3.m3.1.1.3.cmml">T</mi><mo id="S4.p12.3.m3.1.1.1a" xref="S4.p12.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p12.3.m3.1.1.4" xref="S4.p12.3.m3.1.1.4.cmml">T</mi><mo id="S4.p12.3.m3.1.1.1b" xref="S4.p12.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p12.3.m3.1.1.5" xref="S4.p12.3.m3.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p12.3.m3.1b"><apply id="S4.p12.3.m3.1.1.cmml" xref="S4.p12.3.m3.1.1"><times id="S4.p12.3.m3.1.1.1.cmml" xref="S4.p12.3.m3.1.1.1"></times><ci id="S4.p12.3.m3.1.1.2.cmml" xref="S4.p12.3.m3.1.1.2">𝑀</ci><ci id="S4.p12.3.m3.1.1.3.cmml" xref="S4.p12.3.m3.1.1.3">𝑇</ci><ci id="S4.p12.3.m3.1.1.4.cmml" xref="S4.p12.3.m3.1.1.4">𝑇</ci><ci id="S4.p12.3.m3.1.1.5.cmml" xref="S4.p12.3.m3.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p12.3.m3.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.p12.3.m3.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math> of ChatGPT (1.32 hours) is 1.4x longer than Claude (0.93 hours). </div> <figure class="ltx_figure ltx_minipage ltx_align_center ltx_align_middle" id="S4.F6" style="width:433.6pt;"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_2"> <figure class="ltx_figure ltx_figure_panel" id="S4.F6.sf1"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="581" id="S4.F6.sf1.g1" src="x7.png" width="830"/> <figcaption class="ltx_caption">(a) MTTR.</figcaption> </figure> </div> <div class="ltx_flex_cell ltx_flex_size_2"> <figure class="ltx_figure ltx_figure_panel" id="S4.F6.sf2"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="581" id="S4.F6.sf2.g1" src="x8.png" width="830"/> <figcaption class="ltx_caption">(b) MTBF.</figcaption> </figure> </div> </div> <figcaption class="ltx_caption ltx_centering">Figure 6. MTTR and MTBF by provider, ECDF plot. The closer the line is to the upper left, the shorter the time it takes.</figcaption> </figure> <div class="ltx_para ltx_noindent" id="S4.p13"> Observation #6: Playground is the most reliable service (16.80 median <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p13.1.1.1.1.1.m1.1"><semantics id="S4.p13.1.1.1.1.1.m1.1a"><mrow id="S4.p13.1.1.1.1.1.m1.1.1" xref="S4.p13.1.1.1.1.1.m1.1.1.cmml"><mi id="S4.p13.1.1.1.1.1.m1.1.1.2" xref="S4.p13.1.1.1.1.1.m1.1.1.2.cmml">M</mi><mo id="S4.p13.1.1.1.1.1.m1.1.1.1" xref="S4.p13.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p13.1.1.1.1.1.m1.1.1.3" xref="S4.p13.1.1.1.1.1.m1.1.1.3.cmml">T</mi><mo id="S4.p13.1.1.1.1.1.m1.1.1.1a" xref="S4.p13.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p13.1.1.1.1.1.m1.1.1.4" xref="S4.p13.1.1.1.1.1.m1.1.1.4.cmml">B</mi><mo id="S4.p13.1.1.1.1.1.m1.1.1.1b" xref="S4.p13.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p13.1.1.1.1.1.m1.1.1.5" xref="S4.p13.1.1.1.1.1.m1.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p13.1.1.1.1.1.m1.1b"><apply id="S4.p13.1.1.1.1.1.m1.1.1.cmml" xref="S4.p13.1.1.1.1.1.m1.1.1"><times id="S4.p13.1.1.1.1.1.m1.1.1.1.cmml" xref="S4.p13.1.1.1.1.1.m1.1.1.1"></times><ci id="S4.p13.1.1.1.1.1.m1.1.1.2.cmml" xref="S4.p13.1.1.1.1.1.m1.1.1.2">𝑀</ci><ci id="S4.p13.1.1.1.1.1.m1.1.1.3.cmml" xref="S4.p13.1.1.1.1.1.m1.1.1.3">𝑇</ci><ci id="S4.p13.1.1.1.1.1.m1.1.1.4.cmml" xref="S4.p13.1.1.1.1.1.m1.1.1.4">𝐵</ci><ci id="S4.p13.1.1.1.1.1.m1.1.1.5.cmml" xref="S4.p13.1.1.1.1.1.m1.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p13.1.1.1.1.1.m1.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p13.1.1.1.1.1.m1.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math>), followed by DALL·E (4.53 median <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p13.2.2.2.2.2.m2.1"><semantics id="S4.p13.2.2.2.2.2.m2.1a"><mrow id="S4.p13.2.2.2.2.2.m2.1.1" xref="S4.p13.2.2.2.2.2.m2.1.1.cmml"><mi id="S4.p13.2.2.2.2.2.m2.1.1.2" xref="S4.p13.2.2.2.2.2.m2.1.1.2.cmml">M</mi><mo id="S4.p13.2.2.2.2.2.m2.1.1.1" xref="S4.p13.2.2.2.2.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p13.2.2.2.2.2.m2.1.1.3" xref="S4.p13.2.2.2.2.2.m2.1.1.3.cmml">T</mi><mo id="S4.p13.2.2.2.2.2.m2.1.1.1a" xref="S4.p13.2.2.2.2.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p13.2.2.2.2.2.m2.1.1.4" xref="S4.p13.2.2.2.2.2.m2.1.1.4.cmml">B</mi><mo id="S4.p13.2.2.2.2.2.m2.1.1.1b" xref="S4.p13.2.2.2.2.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p13.2.2.2.2.2.m2.1.1.5" xref="S4.p13.2.2.2.2.2.m2.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p13.2.2.2.2.2.m2.1b"><apply id="S4.p13.2.2.2.2.2.m2.1.1.cmml" xref="S4.p13.2.2.2.2.2.m2.1.1"><times id="S4.p13.2.2.2.2.2.m2.1.1.1.cmml" xref="S4.p13.2.2.2.2.2.m2.1.1.1"></times><ci id="S4.p13.2.2.2.2.2.m2.1.1.2.cmml" xref="S4.p13.2.2.2.2.2.m2.1.1.2">𝑀</ci><ci id="S4.p13.2.2.2.2.2.m2.1.1.3.cmml" xref="S4.p13.2.2.2.2.2.m2.1.1.3">𝑇</ci><ci id="S4.p13.2.2.2.2.2.m2.1.1.4.cmml" xref="S4.p13.2.2.2.2.2.m2.1.1.4">𝐵</ci><ci id="S4.p13.2.2.2.2.2.m2.1.1.5.cmml" xref="S4.p13.2.2.2.2.2.m2.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p13.2.2.2.2.2.m2.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p13.2.2.2.2.2.m2.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math>) and Character.AI (3.94 median <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p13.3.3.3.3.3.m3.1"><semantics id="S4.p13.3.3.3.3.3.m3.1a"><mrow id="S4.p13.3.3.3.3.3.m3.1.1" xref="S4.p13.3.3.3.3.3.m3.1.1.cmml"><mi id="S4.p13.3.3.3.3.3.m3.1.1.2" xref="S4.p13.3.3.3.3.3.m3.1.1.2.cmml">M</mi><mo id="S4.p13.3.3.3.3.3.m3.1.1.1" xref="S4.p13.3.3.3.3.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p13.3.3.3.3.3.m3.1.1.3" xref="S4.p13.3.3.3.3.3.m3.1.1.3.cmml">T</mi><mo id="S4.p13.3.3.3.3.3.m3.1.1.1a" xref="S4.p13.3.3.3.3.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p13.3.3.3.3.3.m3.1.1.4" xref="S4.p13.3.3.3.3.3.m3.1.1.4.cmml">B</mi><mo id="S4.p13.3.3.3.3.3.m3.1.1.1b" xref="S4.p13.3.3.3.3.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p13.3.3.3.3.3.m3.1.1.5" xref="S4.p13.3.3.3.3.3.m3.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p13.3.3.3.3.3.m3.1b"><apply id="S4.p13.3.3.3.3.3.m3.1.1.cmml" xref="S4.p13.3.3.3.3.3.m3.1.1"><times id="S4.p13.3.3.3.3.3.m3.1.1.1.cmml" xref="S4.p13.3.3.3.3.3.m3.1.1.1"></times><ci id="S4.p13.3.3.3.3.3.m3.1.1.2.cmml" xref="S4.p13.3.3.3.3.3.m3.1.1.2">𝑀</ci><ci id="S4.p13.3.3.3.3.3.m3.1.1.3.cmml" xref="S4.p13.3.3.3.3.3.m3.1.1.3">𝑇</ci><ci id="S4.p13.3.3.3.3.3.m3.1.1.4.cmml" xref="S4.p13.3.3.3.3.3.m3.1.1.4">𝐵</ci><ci id="S4.p13.3.3.3.3.3.m3.1.1.5.cmml" xref="S4.p13.3.3.3.3.3.m3.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p13.3.3.3.3.3.m3.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p13.3.3.3.3.3.m3.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math>). OpenAI’s ChatGPT is more reliable than Anthropic’s Claude, though its API is less reliable in comparison. </div> <div class="ltx_para" id="S4.p14"> Awareness of how frequently a service fails is important for users to assess the reliability they can offer when they depend on the service. It’s also important to assess which fault-tolerance mechanisms they should use as each has a different overhead. For example, active replication, frequent checkpointing, or infrequent checkpointing. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.F5.sf2" title="In Figure 5 ‣ 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 5(b)</a> depicts <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p14.1.m1.1"><semantics id="S4.p14.1.m1.1a"><mrow id="S4.p14.1.m1.1.1" xref="S4.p14.1.m1.1.1.cmml"><mi id="S4.p14.1.m1.1.1.2" xref="S4.p14.1.m1.1.1.2.cmml">M</mi><mo id="S4.p14.1.m1.1.1.1" xref="S4.p14.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p14.1.m1.1.1.3" xref="S4.p14.1.m1.1.1.3.cmml">T</mi><mo id="S4.p14.1.m1.1.1.1a" xref="S4.p14.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p14.1.m1.1.1.4" xref="S4.p14.1.m1.1.1.4.cmml">B</mi><mo id="S4.p14.1.m1.1.1.1b" xref="S4.p14.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p14.1.m1.1.1.5" xref="S4.p14.1.m1.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p14.1.m1.1b"><apply id="S4.p14.1.m1.1.1.cmml" xref="S4.p14.1.m1.1.1"><times id="S4.p14.1.m1.1.1.1.cmml" xref="S4.p14.1.m1.1.1.1"></times><ci id="S4.p14.1.m1.1.1.2.cmml" xref="S4.p14.1.m1.1.1.2">𝑀</ci><ci id="S4.p14.1.m1.1.1.3.cmml" xref="S4.p14.1.m1.1.1.3">𝑇</ci><ci id="S4.p14.1.m1.1.1.4.cmml" xref="S4.p14.1.m1.1.1.4">𝐵</ci><ci id="S4.p14.1.m1.1.1.5.cmml" xref="S4.p14.1.m1.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p14.1.m1.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p14.1.m1.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> of the 8 LLM incidents, showing how frequently failures occur. The <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p14.2.m2.1"><semantics id="S4.p14.2.m2.1a"><mrow id="S4.p14.2.m2.1.1" xref="S4.p14.2.m2.1.1.cmml"><mi id="S4.p14.2.m2.1.1.2" xref="S4.p14.2.m2.1.1.2.cmml">M</mi><mo id="S4.p14.2.m2.1.1.1" xref="S4.p14.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p14.2.m2.1.1.3" xref="S4.p14.2.m2.1.1.3.cmml">T</mi><mo id="S4.p14.2.m2.1.1.1a" xref="S4.p14.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p14.2.m2.1.1.4" xref="S4.p14.2.m2.1.1.4.cmml">B</mi><mo id="S4.p14.2.m2.1.1.1b" xref="S4.p14.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p14.2.m2.1.1.5" xref="S4.p14.2.m2.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p14.2.m2.1b"><apply id="S4.p14.2.m2.1.1.cmml" xref="S4.p14.2.m2.1.1"><times id="S4.p14.2.m2.1.1.1.cmml" xref="S4.p14.2.m2.1.1.1"></times><ci id="S4.p14.2.m2.1.1.2.cmml" xref="S4.p14.2.m2.1.1.2">𝑀</ci><ci id="S4.p14.2.m2.1.1.3.cmml" xref="S4.p14.2.m2.1.1.3">𝑇</ci><ci id="S4.p14.2.m2.1.1.4.cmml" xref="S4.p14.2.m2.1.1.4">𝐵</ci><ci id="S4.p14.2.m2.1.1.5.cmml" xref="S4.p14.2.m2.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p14.2.m2.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p14.2.m2.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> of failures varies significantly across services. The most reliable service is Playground, with a median <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p14.3.m3.1"><semantics id="S4.p14.3.m3.1a"><mrow id="S4.p14.3.m3.1.1" xref="S4.p14.3.m3.1.1.cmml"><mi id="S4.p14.3.m3.1.1.2" xref="S4.p14.3.m3.1.1.2.cmml">M</mi><mo id="S4.p14.3.m3.1.1.1" xref="S4.p14.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p14.3.m3.1.1.3" xref="S4.p14.3.m3.1.1.3.cmml">T</mi><mo id="S4.p14.3.m3.1.1.1a" xref="S4.p14.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p14.3.m3.1.1.4" xref="S4.p14.3.m3.1.1.4.cmml">B</mi><mo id="S4.p14.3.m3.1.1.1b" xref="S4.p14.3.m3.1.1.1.cmml">⁢</mo><mi id="S4.p14.3.m3.1.1.5" xref="S4.p14.3.m3.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p14.3.m3.1b"><apply id="S4.p14.3.m3.1.1.cmml" xref="S4.p14.3.m3.1.1"><times id="S4.p14.3.m3.1.1.1.cmml" xref="S4.p14.3.m3.1.1.1"></times><ci id="S4.p14.3.m3.1.1.2.cmml" xref="S4.p14.3.m3.1.1.2">𝑀</ci><ci id="S4.p14.3.m3.1.1.3.cmml" xref="S4.p14.3.m3.1.1.3">𝑇</ci><ci id="S4.p14.3.m3.1.1.4.cmml" xref="S4.p14.3.m3.1.1.4">𝐵</ci><ci id="S4.p14.3.m3.1.1.5.cmml" xref="S4.p14.3.m3.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p14.3.m3.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p14.3.m3.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> of 16.80 days, which is nearly 9.66 times higher than the lowest median <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p14.4.m4.1"><semantics id="S4.p14.4.m4.1a"><mrow id="S4.p14.4.m4.1.1" xref="S4.p14.4.m4.1.1.cmml"><mi id="S4.p14.4.m4.1.1.2" xref="S4.p14.4.m4.1.1.2.cmml">M</mi><mo id="S4.p14.4.m4.1.1.1" xref="S4.p14.4.m4.1.1.1.cmml">⁢</mo><mi id="S4.p14.4.m4.1.1.3" xref="S4.p14.4.m4.1.1.3.cmml">T</mi><mo id="S4.p14.4.m4.1.1.1a" xref="S4.p14.4.m4.1.1.1.cmml">⁢</mo><mi id="S4.p14.4.m4.1.1.4" xref="S4.p14.4.m4.1.1.4.cmml">B</mi><mo id="S4.p14.4.m4.1.1.1b" xref="S4.p14.4.m4.1.1.1.cmml">⁢</mo><mi id="S4.p14.4.m4.1.1.5" xref="S4.p14.4.m4.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p14.4.m4.1b"><apply id="S4.p14.4.m4.1.1.cmml" xref="S4.p14.4.m4.1.1"><times id="S4.p14.4.m4.1.1.1.cmml" xref="S4.p14.4.m4.1.1.1"></times><ci id="S4.p14.4.m4.1.1.2.cmml" xref="S4.p14.4.m4.1.1.2">𝑀</ci><ci id="S4.p14.4.m4.1.1.3.cmml" xref="S4.p14.4.m4.1.1.3">𝑇</ci><ci id="S4.p14.4.m4.1.1.4.cmml" xref="S4.p14.4.m4.1.1.4">𝐵</ci><ci id="S4.p14.4.m4.1.1.5.cmml" xref="S4.p14.4.m4.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p14.4.m4.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p14.4.m4.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> of 1.74 days from Claude. The median <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p14.5.m5.1"><semantics id="S4.p14.5.m5.1a"><mrow id="S4.p14.5.m5.1.1" xref="S4.p14.5.m5.1.1.cmml"><mi id="S4.p14.5.m5.1.1.2" xref="S4.p14.5.m5.1.1.2.cmml">M</mi><mo id="S4.p14.5.m5.1.1.1" xref="S4.p14.5.m5.1.1.1.cmml">⁢</mo><mi id="S4.p14.5.m5.1.1.3" xref="S4.p14.5.m5.1.1.3.cmml">T</mi><mo id="S4.p14.5.m5.1.1.1a" xref="S4.p14.5.m5.1.1.1.cmml">⁢</mo><mi id="S4.p14.5.m5.1.1.4" xref="S4.p14.5.m5.1.1.4.cmml">B</mi><mo id="S4.p14.5.m5.1.1.1b" xref="S4.p14.5.m5.1.1.1.cmml">⁢</mo><mi id="S4.p14.5.m5.1.1.5" xref="S4.p14.5.m5.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p14.5.m5.1b"><apply id="S4.p14.5.m5.1.1.cmml" xref="S4.p14.5.m5.1.1"><times id="S4.p14.5.m5.1.1.1.cmml" xref="S4.p14.5.m5.1.1.1"></times><ci id="S4.p14.5.m5.1.1.2.cmml" xref="S4.p14.5.m5.1.1.2">𝑀</ci><ci id="S4.p14.5.m5.1.1.3.cmml" xref="S4.p14.5.m5.1.1.3">𝑇</ci><ci id="S4.p14.5.m5.1.1.4.cmml" xref="S4.p14.5.m5.1.1.4">𝐵</ci><ci id="S4.p14.5.m5.1.1.5.cmml" xref="S4.p14.5.m5.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p14.5.m5.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p14.5.m5.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> for OpenAI’s API is 1.99 days, which is lower than Anthropic’s API at 2.09 days; however, ChatGPT at 2.04 days is higher than Claude’s 1.74 days. DALL·E and Character.AI are relatively reliable services, with median <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p14.6.m6.1"><semantics id="S4.p14.6.m6.1a"><mrow id="S4.p14.6.m6.1.1" xref="S4.p14.6.m6.1.1.cmml"><mi id="S4.p14.6.m6.1.1.2" xref="S4.p14.6.m6.1.1.2.cmml">M</mi><mo id="S4.p14.6.m6.1.1.1" xref="S4.p14.6.m6.1.1.1.cmml">⁢</mo><mi id="S4.p14.6.m6.1.1.3" xref="S4.p14.6.m6.1.1.3.cmml">T</mi><mo id="S4.p14.6.m6.1.1.1a" xref="S4.p14.6.m6.1.1.1.cmml">⁢</mo><mi id="S4.p14.6.m6.1.1.4" xref="S4.p14.6.m6.1.1.4.cmml">B</mi><mo id="S4.p14.6.m6.1.1.1b" xref="S4.p14.6.m6.1.1.1.cmml">⁢</mo><mi id="S4.p14.6.m6.1.1.5" xref="S4.p14.6.m6.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p14.6.m6.1b"><apply id="S4.p14.6.m6.1.1.cmml" xref="S4.p14.6.m6.1.1"><times id="S4.p14.6.m6.1.1.1.cmml" xref="S4.p14.6.m6.1.1.1"></times><ci id="S4.p14.6.m6.1.1.2.cmml" xref="S4.p14.6.m6.1.1.2">𝑀</ci><ci id="S4.p14.6.m6.1.1.3.cmml" xref="S4.p14.6.m6.1.1.3">𝑇</ci><ci id="S4.p14.6.m6.1.1.4.cmml" xref="S4.p14.6.m6.1.1.4">𝐵</ci><ci id="S4.p14.6.m6.1.1.5.cmml" xref="S4.p14.6.m6.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p14.6.m6.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p14.6.m6.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> values of 4.63 days and 3.94 days, respectively. </div> <div class="ltx_para" id="S4.p15"> The MTBF of LLM services (4-40 days) is much higher than the MTBF of single-node failure (0.25-1 day) and system-wide failure (6.6 days) in other large-scale systems <cite class="ltx_cite ltx_citemacro_citep">(Martino et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib40" title="">2014</a>; Gupta et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib21" title="">2017</a>)</cite>. This indicates that LLM operators use effective fault-tolerance mechanisms. It also indicates that users can use low-overhead fault-tolerance techniques like infrequent checkpointing to provide a reliable service that depends on LLMs. </div> <div class="ltx_para ltx_noindent" id="S4.p16"> Observation #7: Over 90% of the incidents end within 10 hours for all measured providers. Specifically, Anthropic’s services resolved failures more quickly but also experienced the highest frequency of incidents, based on its <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.p16.1.1.1.1.1.m1.1"><semantics id="S4.p16.1.1.1.1.1.m1.1a"><mrow id="S4.p16.1.1.1.1.1.m1.1.1" xref="S4.p16.1.1.1.1.1.m1.1.1.cmml"><mi id="S4.p16.1.1.1.1.1.m1.1.1.2" xref="S4.p16.1.1.1.1.1.m1.1.1.2.cmml">M</mi><mo id="S4.p16.1.1.1.1.1.m1.1.1.1" xref="S4.p16.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p16.1.1.1.1.1.m1.1.1.3" xref="S4.p16.1.1.1.1.1.m1.1.1.3.cmml">T</mi><mo id="S4.p16.1.1.1.1.1.m1.1.1.1a" xref="S4.p16.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p16.1.1.1.1.1.m1.1.1.4" xref="S4.p16.1.1.1.1.1.m1.1.1.4.cmml">T</mi><mo id="S4.p16.1.1.1.1.1.m1.1.1.1b" xref="S4.p16.1.1.1.1.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p16.1.1.1.1.1.m1.1.1.5" xref="S4.p16.1.1.1.1.1.m1.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p16.1.1.1.1.1.m1.1b"><apply id="S4.p16.1.1.1.1.1.m1.1.1.cmml" xref="S4.p16.1.1.1.1.1.m1.1.1"><times id="S4.p16.1.1.1.1.1.m1.1.1.1.cmml" xref="S4.p16.1.1.1.1.1.m1.1.1.1"></times><ci id="S4.p16.1.1.1.1.1.m1.1.1.2.cmml" xref="S4.p16.1.1.1.1.1.m1.1.1.2">𝑀</ci><ci id="S4.p16.1.1.1.1.1.m1.1.1.3.cmml" xref="S4.p16.1.1.1.1.1.m1.1.1.3">𝑇</ci><ci id="S4.p16.1.1.1.1.1.m1.1.1.4.cmml" xref="S4.p16.1.1.1.1.1.m1.1.1.4">𝑇</ci><ci id="S4.p16.1.1.1.1.1.m1.1.1.5.cmml" xref="S4.p16.1.1.1.1.1.m1.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p16.1.1.1.1.1.m1.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.p16.1.1.1.1.1.m1.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math> (2.70 hours) and <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p16.2.2.2.2.2.m2.1"><semantics id="S4.p16.2.2.2.2.2.m2.1a"><mrow id="S4.p16.2.2.2.2.2.m2.1.1" xref="S4.p16.2.2.2.2.2.m2.1.1.cmml"><mi id="S4.p16.2.2.2.2.2.m2.1.1.2" xref="S4.p16.2.2.2.2.2.m2.1.1.2.cmml">M</mi><mo id="S4.p16.2.2.2.2.2.m2.1.1.1" xref="S4.p16.2.2.2.2.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p16.2.2.2.2.2.m2.1.1.3" xref="S4.p16.2.2.2.2.2.m2.1.1.3.cmml">T</mi><mo id="S4.p16.2.2.2.2.2.m2.1.1.1a" xref="S4.p16.2.2.2.2.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p16.2.2.2.2.2.m2.1.1.4" xref="S4.p16.2.2.2.2.2.m2.1.1.4.cmml">B</mi><mo id="S4.p16.2.2.2.2.2.m2.1.1.1b" xref="S4.p16.2.2.2.2.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p16.2.2.2.2.2.m2.1.1.5" xref="S4.p16.2.2.2.2.2.m2.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p16.2.2.2.2.2.m2.1b"><apply id="S4.p16.2.2.2.2.2.m2.1.1.cmml" xref="S4.p16.2.2.2.2.2.m2.1.1"><times id="S4.p16.2.2.2.2.2.m2.1.1.1.cmml" xref="S4.p16.2.2.2.2.2.m2.1.1.1"></times><ci id="S4.p16.2.2.2.2.2.m2.1.1.2.cmml" xref="S4.p16.2.2.2.2.2.m2.1.1.2">𝑀</ci><ci id="S4.p16.2.2.2.2.2.m2.1.1.3.cmml" xref="S4.p16.2.2.2.2.2.m2.1.1.3">𝑇</ci><ci id="S4.p16.2.2.2.2.2.m2.1.1.4.cmml" xref="S4.p16.2.2.2.2.2.m2.1.1.4">𝐵</ci><ci id="S4.p16.2.2.2.2.2.m2.1.1.5.cmml" xref="S4.p16.2.2.2.2.2.m2.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p16.2.2.2.2.2.m2.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p16.2.2.2.2.2.m2.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> (5.22 days) on average. </div> <figure class="ltx_figure" id="S4.F7"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_1"> <figure class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" id="S4.F7.sf1" style="width:433.6pt;"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="125" id="S4.F7.sf1.g1" src="x9.png" width="681"/> <figcaption class="ltx_caption ltx_centering">(a) The number of incidents by day of week.</figcaption> </figure> </div> <div class="ltx_flex_break"></div> <div class="ltx_flex_cell ltx_flex_size_1"> <figure class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" id="S4.F7.sf2" style="width:433.6pt;"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="125" id="S4.F7.sf2.g1" src="x10.png" width="681"/> <figcaption class="ltx_caption ltx_centering">(b) The number of incidents by hour of day.</figcaption> </figure> </div> </div> <figcaption class="ltx_caption ltx_centering">Figure 7. Temporal distributions for incidents, PDT time.</figcaption> </figure> <div class="ltx_para" id="S4.p17"> To understand how the MTTR and MTBF values are distributed and compare the distributions for different LLM providers, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.F6" title="In 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 6</a> displays the Empirical Cumulative Distribution Function (ECDF) plot of <math alttext="MTTR" class="ltx_Math" display="inline" id="S4.p17.1.m1.1"><semantics id="S4.p17.1.m1.1a"><mrow id="S4.p17.1.m1.1.1" xref="S4.p17.1.m1.1.1.cmml"><mi id="S4.p17.1.m1.1.1.2" xref="S4.p17.1.m1.1.1.2.cmml">M</mi><mo id="S4.p17.1.m1.1.1.1" xref="S4.p17.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p17.1.m1.1.1.3" xref="S4.p17.1.m1.1.1.3.cmml">T</mi><mo id="S4.p17.1.m1.1.1.1a" xref="S4.p17.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p17.1.m1.1.1.4" xref="S4.p17.1.m1.1.1.4.cmml">T</mi><mo id="S4.p17.1.m1.1.1.1b" xref="S4.p17.1.m1.1.1.1.cmml">⁢</mo><mi id="S4.p17.1.m1.1.1.5" xref="S4.p17.1.m1.1.1.5.cmml">R</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p17.1.m1.1b"><apply id="S4.p17.1.m1.1.1.cmml" xref="S4.p17.1.m1.1.1"><times id="S4.p17.1.m1.1.1.1.cmml" xref="S4.p17.1.m1.1.1.1"></times><ci id="S4.p17.1.m1.1.1.2.cmml" xref="S4.p17.1.m1.1.1.2">𝑀</ci><ci id="S4.p17.1.m1.1.1.3.cmml" xref="S4.p17.1.m1.1.1.3">𝑇</ci><ci id="S4.p17.1.m1.1.1.4.cmml" xref="S4.p17.1.m1.1.1.4">𝑇</ci><ci id="S4.p17.1.m1.1.1.5.cmml" xref="S4.p17.1.m1.1.1.5">𝑅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p17.1.m1.1c">MTTR</annotation><annotation encoding="application/x-llamapun" id="S4.p17.1.m1.1d">italic_M italic_T italic_T italic_R</annotation></semantics></math> in hours and <math alttext="MTBF" class="ltx_Math" display="inline" id="S4.p17.2.m2.1"><semantics id="S4.p17.2.m2.1a"><mrow id="S4.p17.2.m2.1.1" xref="S4.p17.2.m2.1.1.cmml"><mi id="S4.p17.2.m2.1.1.2" xref="S4.p17.2.m2.1.1.2.cmml">M</mi><mo id="S4.p17.2.m2.1.1.1" xref="S4.p17.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p17.2.m2.1.1.3" xref="S4.p17.2.m2.1.1.3.cmml">T</mi><mo id="S4.p17.2.m2.1.1.1a" xref="S4.p17.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p17.2.m2.1.1.4" xref="S4.p17.2.m2.1.1.4.cmml">B</mi><mo id="S4.p17.2.m2.1.1.1b" xref="S4.p17.2.m2.1.1.1.cmml">⁢</mo><mi id="S4.p17.2.m2.1.1.5" xref="S4.p17.2.m2.1.1.5.cmml">F</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p17.2.m2.1b"><apply id="S4.p17.2.m2.1.1.cmml" xref="S4.p17.2.m2.1.1"><times id="S4.p17.2.m2.1.1.1.cmml" xref="S4.p17.2.m2.1.1.1"></times><ci id="S4.p17.2.m2.1.1.2.cmml" xref="S4.p17.2.m2.1.1.2">𝑀</ci><ci id="S4.p17.2.m2.1.1.3.cmml" xref="S4.p17.2.m2.1.1.3">𝑇</ci><ci id="S4.p17.2.m2.1.1.4.cmml" xref="S4.p17.2.m2.1.1.4">𝐵</ci><ci id="S4.p17.2.m2.1.1.5.cmml" xref="S4.p17.2.m2.1.1.5">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p17.2.m2.1c">MTBF</annotation><annotation encoding="application/x-llamapun" id="S4.p17.2.m2.1d">italic_M italic_T italic_B italic_F</annotation></semantics></math> in days, grouped by provider. It also marks vertically different time points for better observation and comparison. </div> <div class="ltx_para" id="S4.p18"> A small percentage of incidents can be resolved within 10 minutes, such as 8.55% for Anthropic. Anthropic also solved the highest percent of incidents (37.18%) in 0.5 hours, significantly more than OpenAI (19.72%) and Character.AI (22.86%). Most failures are addressed within 3 hours, with 74.25% for OpenAI, 82.91% for Anthropic, and 68.57% for Character.AI. After 10 hours, 92.34% of OpenAI, 90.60% of Anthropic, and 91.43% of Character.AI’s failures are solved. However, a small proposition of failures for all providers lasted over 1 day, with 6.03%, 7.69%, and 5.71%, respectively. Overall, Anthropic resolved failures more quickly, despite a higher percentage of extreme cases lasting over 1 day. </div> <div class="ltx_para" id="S4.p19"> Although Anthropic resolves failures the fastest, it also encounters them most frequently, with every 5.22 days on average. In contrast, OpenAI and Character.AI are more reliable, with failure occurring every 8.48 and 8.74 days, respectively. A notable percentage of incidents occur within a day: 35.47% for Anthropic, 28.77% for OpenAI, and 20.00% for Character.AI. Within 1 week interval, nearly three-quarters of failures occur for OpenAI (75.64%) and for Anthorpic (78.63%), with a slightly lower rate for Character.AI (60.00%). Over 90% of incidents for all providers happen within a month of each other. Our findings indicate that users of LLMs should expect failures regularly (at least once a month). Therefore, failure should not be an exceptional event but should be baked into the users’ normal operating procedure. </div> </section> <section class="ltx_section" id="S5"> <h2 class="ltx_title ltx_title_section"> 5. Failure Patterns Over Time</h2> <div class="ltx_para" id="S5.p1"> This section conducts time series analyses to examine the failure patterns over time, including: (1) Weekly and daily incident distributions, (2) Auto-correlations in different time intervals; and (3) Daily service available time. </div> <figure class="ltx_figure ltx_minipage ltx_align_center ltx_align_middle" id="S5.F8" style="width:433.6pt;"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_1"> <figure class="ltx_figure ltx_figure_panel" id="S5.F8.sf1"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="249" id="S5.F8.sf1.g1" src="x11.png" width="747"/> <figcaption class="ltx_caption">(a) OpenAI.</figcaption> </figure> </div> <div class="ltx_flex_break"></div> <div class="ltx_flex_cell ltx_flex_size_1"> <figure class="ltx_figure ltx_figure_panel" id="S5.F8.sf2"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="226" id="S5.F8.sf2.g1" src="x12.png" width="747"/> <figcaption class="ltx_caption">(b) Anthropic.</figcaption> </figure> </div> </div> <figcaption class="ltx_caption ltx_centering">Figure 8. Auto-correlations with the numbers of incidents aggregated at different time granularities.</figcaption> </figure> <section class="ltx_subsection" id="S5.SS1"> <h3 class="ltx_title ltx_title_subsection"> 5.1. Temporal Distributions</h3> <div class="ltx_para ltx_noindent" id="S5.SS1.p1"> Observation #8: OpenAI and Anthropic exhibit more failures on weekdays, while Character.AI has fewer on Tuesdays and Wednesdays. All services show peak failures from 8:00 to 16:00. </div> <div class="ltx_para" id="S5.SS1.p2"> To investigate the temporal distributions of LLM incidents, we aggregate service incidents by day of week in <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.F7.sf1" title="In Figure 7 ‣ 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 7(a)</a>, and hour of day in <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.F7.sf2" title="In Figure 7 ‣ 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 7(b)</a>. Incident times are given in local time (PDT) as they were originally reported in PDT. OpenAI and Anthropic’s services display a clear weekday pattern in incidents, with significantly more failures on weekdays than on weekends. In contrast, Character.AI follows a different pattern, with fewer failures occurring on Tuesdays and Wednesdays. This may be due to the differing purposes of using LLM services: Character.AI is primarily used for leisure <cite class="ltx_cite ltx_citemacro_citep">(WIRED, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib65" title="">2024</a>)</cite>, while API and conversational services are more often used for work-related tasks, such as writing and coding <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib62" title="">2024b</a>)</cite>. All services exhibit a diurnal pattern, with incident peaks occurring during typical work hours, such as 8:00 to 16:00, and lower at night hours. Similar periodic failure patterns are also found in machine learning jobs <cite class="ltx_cite ltx_citemacro_citep">(Chu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib15" title="">2023</a>, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib14" title="">2024</a>; Versluis et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib61" title="">2023</a>)</cite>, deep learning jobs <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib35" title="">2022</a>)</cite>, and general user request in BurstGPT workloads <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib63" title="">2024a</a>)</cite>. </div> <figure class="ltx_figure" id="S5.F9"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="217" id="S5.F9.g1" src="x13.png" width="830"/> <figcaption class="ltx_caption ltx_centering">Figure 9. Service daily availability by scaled outage minutes [%]. Some services started reporting later (see <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S3.SS2" title="3.2. Data Collection and Dataset Preparation ‣ 3. Dataset Collection and Preparation ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 3.2</a>).</figcaption> </figure> <figure class="ltx_table" id="S5.T7"> <figcaption class="ltx_caption ltx_centering">Table 7. Service availability by scaled outage minutes (from all periods).</figcaption> <div class="ltx_inline-block ltx_align_center ltx_transformed_outer" id="S5.T7.4" style="width:363.8pt;height:100.1pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_align_middle" id="S5.T7.4.1"> <tr class="ltx_tr" id="S5.T7.4.1.1"> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.1">Service</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.2">Min</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.3">Max</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.4">Mean</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.5">Median</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.6">=100%</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.7">¿=99.999%</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.8">¿=99.99%</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.9">¿=99.9%</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.10">¿=99%</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.11">¿=90%</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S5.T7.4.1.1.12">¡90%</td> </tr> <tr class="ltx_tr" id="S5.T7.4.1.2"> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.1">API-OpenAI</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.2">80.57%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.3">100.0%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.4">99.82%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.5">100.0%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.6">92.68%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.7">92.68%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.8">92.68%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.9">92.84%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.10">95.30%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.11">99.77%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.2.12">0.23%</td> </tr> <tr class="ltx_tr" id="S5.T7.4.1.3"> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.1">ChatGPT</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.2">77.15%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.3">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.4">99.66%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.5">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.6">88.85%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.7">88.85%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.8">88.85%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.9">89.20%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.10">93.10%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.11">99.12%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.3.12">0.88%</td> </tr> <tr class="ltx_tr" id="S5.T7.4.1.4"> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.1">DALL·E</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.2">74.17%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.3">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.4">99.78%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.5">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.6">95.88%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.7">95.88%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.8">95.88%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.9">95.88%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.10">96.77%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.11">99.28%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.4.12">0.72%</td> </tr> <tr class="ltx_tr" id="S5.T7.4.1.5"> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.1">Playground</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.2">82.57%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.3">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.4">99.94%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.5">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.6">98.08%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.7">98.08%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.8">98.08%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.9">98.16%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.10">98.88%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.11">99.84%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.5.12">0.16%</td> </tr> <tr class="ltx_tr" id="S5.T7.4.1.6"> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.1">API-Anthropic</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.2">85.44%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.3">100.0%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.4">99.92%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.5">100.0%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.6">94.02%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.7">94.02%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.8">94.02%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.9">94.26%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.10">97.61%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.11">99.76%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S5.T7.4.1.6.12">0.24%</td> </tr> <tr class="ltx_tr" id="S5.T7.4.1.7"> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.1">Claude</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.2">82.02%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.3">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.4">99.84%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.5">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.6">93.06%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.7">93.06%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.8">93.06%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.9">93.06%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.10">97.13%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.11">99.52%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.7.12">0.48%</td> </tr> <tr class="ltx_tr" id="S5.T7.4.1.8"> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.1">Console</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.2">82.56%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.3">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.4">99.89%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.5">100.0%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.6">93.54%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.7">93.54%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.8">93.54%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.9">93.54%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.10">97.61%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.11">99.76%</td> <td class="ltx_td ltx_align_right" id="S5.T7.4.1.8.12">0.24%</td> </tr> <tr class="ltx_tr" id="S5.T7.4.1.9"> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.1">Character.AI</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.2">85.33%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.3">100.0%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.4">99.59%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.5">100.0%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.6">90.88%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.7">90.88%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.8">90.88%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.9">91.19%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.10">94.03%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.11">98.11%</td> <td class="ltx_td ltx_align_right ltx_border_bb ltx_border_t" id="S5.T7.4.1.9.12">1.89%</td> </tr> </table> </div> </figure> </section> <section class="ltx_subsection" id="S5.SS2"> <h3 class="ltx_title ltx_title_subsection"> 5.2. Auto-correlations</h3> <div class="ltx_para ltx_noindent" id="S5.SS2.p1"> Observation #9: LLM service failures have strong monthly auto-correlations, with OpenAI incidents showing longer-lasting correlations than Anthropic. Both services display distinct weekly periodicity. </div> <div class="ltx_para" id="S5.SS2.p2"> We investigate if a failure is immediately followed by another failure and how often it happens. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S5.F8" title="In 5. Failure Patterns Over Time ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 8</a> depicts the auto-correlation for the number of incidents at month, week, and day granularities. We use the autocorrelation function (ACF) to measure the degree of correlation based on the temporal incidents data. Confidence intervals are drawn as the blue area. By default, this is set to a 95% confidence interval, suggesting that correlation values outside of this area are significant, which are real patterns rather than random noise. Lags represent the time intervals at which a time series is compared to itself, and autocorrelation measures how similar a time series is to itself at different lags. </div> <div class="ltx_para" id="S5.SS2.p3"> For OpenAI, the auto-correlation plots display significant positive correlations up to lag 3 on a monthly scale and up to lag 12 on a weekly scale, indicating that both monthly and weekly incidents are strongly related to their previous values. Anthropic shows similar correlations with shorter lags, with up to lag 1 for monthly data and lag 7 for weekly data, likely affected by Anthropic’s shorter operational history. The consistent but gradual decay in auto-correlations at every 7-day interval for both OpenAI and Anthropic suggests strong weekly periodic behavior, supporting our previous findings in <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S4.F7" title="In 4. Failure-Recovery Analysis ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 7</a>. Compared to the auto-correlations observed in ML failures from the previous study <cite class="ltx_cite ltx_citemacro_citep">(Chu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib15" title="">2023</a>)</cite>, the auto-correlation in LLM service failures shows stronger periodic trends. The periodic characteristics can be utilized to predict future incidents, similar to workload failure predictions <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib37" title="">2023</a>)</cite>. </div> </section> <section class="ltx_subsection" id="S5.SS3"> <h3 class="ltx_title ltx_title_subsection"> 5.3. Service Availability Over Time</h3> <div class="ltx_para ltx_noindent" id="S5.SS3.p1"> Observation #10: ChatGPT is the least consistently available service, with only 88.85% of days fully available, followed by Character.AI at 90.88%. Availability of Anthropic’s services declined after April 2024, possibly due to product release and the sharp increase in user demands. </div> <div class="ltx_para" id="S5.SS3.p2"> We provide a high-level view of what level of service reliability a user can expect in this section. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S5.F9" title="In 5.1. Temporal Distributions ‣ 5. Failure Patterns Over Time ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 9</a> shows the service daily availability by scaled outage minutes, from February 2023 to August 2024. We categorized availability into five levels based on their value ranges. Days without outages, which mean full service availability, are colored green, while days with longer outage durations are represented by colors closer to red. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S5.T7" title="In 5.1. Temporal Distributions ‣ 5. Failure Patterns Over Time ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Table 7</a> gives the specific statistics of service availability. DALL·E and Playground have the highest availability, with 95.88% and 98.08% of days fully accessible, respectively. In contrast, ChatGPT is the least available service, with only 88.85% of days fully accessible. Availability of Anthropic’s services declined after April 2024, possibly due to product release and the sharp increase in user demands <cite class="ltx_cite ltx_citemacro_citep">(Anthropic, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib7" title="">2024b</a>, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib6" title="">a</a>)</cite>. Character.AI also shows noticeable instability, with only 90.88% of days fully available and over 1.89% of days with availability falling below 90%. </div> </section> </section> <section class="ltx_section" id="S6"> <h2 class="ltx_title ltx_title_section"> 6. Co-occurrence of Failures</h2> <div class="ltx_para" id="S6.p1"> This section examines the co-occurrence of failures across services. When an outage occurs in one service, do other services also experience outages? Is there any co-occurrence within and across different providers, given that services may share the same cloud infrastructure? For instance, both Anthropic and Character.AI rely on GCP <cite class="ltx_cite ltx_citemacro_citep">(gcp, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib4" title="">2025b</a>, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib3" title="">a</a>)</cite>. How about the impacted range of incidents for different services and providers? To address these questions, we analyze (1) the co-occurrence of outages, and (2) the impact range of incidents. </div> <section class="ltx_subsection" id="S6.SS1"> <h3 class="ltx_title ltx_title_subsection"> 6.1. Co-occurrence of Outages</h3> <figure class="ltx_figure ltx_minipage ltx_align_center ltx_align_middle" id="S6.F10" style="width:433.6pt;"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_1"> <figure class="ltx_figure ltx_figure_panel" id="S6.F10.sf1"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="651" id="S6.F10.sf1.g1" src="x14.png" width="830"/> <figcaption class="ltx_caption">(a) Co-occurrence outages in days count.</figcaption> </figure> </div> <div class="ltx_flex_break"></div> <div class="ltx_flex_cell ltx_flex_size_1"> <figure class="ltx_figure ltx_figure_panel" id="S6.F10.sf2"><img alt="Refer to caption" class="ltx_graphics ltx_img_landscape" height="649" id="S6.F10.sf2.g1" src="x15.png" width="830"/> <figcaption class="ltx_caption">(b) Conditional probabilities of co-occurrence outages. Notes: y-axis <math alttext="=" class="ltx_Math" display="inline" id="S6.F10.sf2.4.1.m1.1"><semantics id="S6.F10.sf2.4.1.m1.1b"><mo id="S6.F10.sf2.4.1.m1.1.1" xref="S6.F10.sf2.4.1.m1.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S6.F10.sf2.4.1.m1.1c"><eq id="S6.F10.sf2.4.1.m1.1.1.cmml" xref="S6.F10.sf2.4.1.m1.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S6.F10.sf2.4.1.m1.1d">=</annotation><annotation encoding="application/x-llamapun" id="S6.F10.sf2.4.1.m1.1e">=</annotation></semantics></math> service A, x-axis <math alttext="=" class="ltx_Math" display="inline" id="S6.F10.sf2.5.2.m2.1"><semantics id="S6.F10.sf2.5.2.m2.1b"><mo id="S6.F10.sf2.5.2.m2.1.1" xref="S6.F10.sf2.5.2.m2.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S6.F10.sf2.5.2.m2.1c"><eq id="S6.F10.sf2.5.2.m2.1.1.cmml" xref="S6.F10.sf2.5.2.m2.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S6.F10.sf2.5.2.m2.1d">=</annotation><annotation encoding="application/x-llamapun" id="S6.F10.sf2.5.2.m2.1e">=</annotation></semantics></math> service B, cells <math alttext="=" class="ltx_Math" display="inline" id="S6.F10.sf2.6.3.m3.1"><semantics id="S6.F10.sf2.6.3.m3.1b"><mo id="S6.F10.sf2.6.3.m3.1.1" xref="S6.F10.sf2.6.3.m3.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S6.F10.sf2.6.3.m3.1c"><eq id="S6.F10.sf2.6.3.m3.1.1.cmml" xref="S6.F10.sf2.6.3.m3.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S6.F10.sf2.6.3.m3.1d">=</annotation><annotation encoding="application/x-llamapun" id="S6.F10.sf2.6.3.m3.1e">=</annotation></semantics></math> P(A—B).</figcaption> </figure> </div> </div> <figcaption class="ltx_caption ltx_centering">Figure 10. Co-occurrence of outages between service pairs.</figcaption> </figure> <div class="ltx_para ltx_noindent" id="S6.SS1.p1"> Observation #11: Co-occurrence is particularly high among services from the same provider, suggesting a strong interdependence between those services. For Anthropic’s services, the likelihood of any two services experiencing outages on the same day is over 80%, indicating a severe lack of isolation across different services. </div> <div class="ltx_para" id="S6.SS1.p2"> The <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S6.F10.sf1" title="In Figure 10 ‣ 6.1. Co-occurrence of Outages ‣ 6. Co-occurrence of Failures ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 10(a)</a> shows the number of co-occurring outages across different services on the same day. The counts of outages may be affected by the maximum number of outages. For example, the number of co-occurrences among Anthropic services is lower than for OpenAI services, however, the probability of co-occurrence among Anthropic services is higher. To avoid this impact of the number of outages, we also give the conditional probabilities of co-occurring outages in <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S6.F10.sf2" title="In Figure 10 ‣ 6.1. Co-occurrence of Outages ‣ 6. Co-occurrence of Failures ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 10(b)</a>. The conditional probability indicates the likelihood that if service B experiences an outage, service A will also experience an outage. For instance, the 49.21% in row 1, column 2 means that if ChatGPT is down, there is a 49.21% chance that OpenAI’s API will also experience an outage on the same day. The probability that service A is also outage while service B is outage can be formulated in: <table class="ltx_equation ltx_eqn_table" id="S6.E4"> <tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline"> <td class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left" rowspan="1">(4)</td> <td class="ltx_eqn_cell ltx_eqn_center_padleft"></td> <td class="ltx_eqn_cell ltx_align_center"><math alttext="P(A|B)=\frac{P(A\cap B)}{P(A)\times P(B)}=\frac{O_{AB}}{O_{B}}" class="ltx_Math" display="block" id="S6.E4.m1.4"><semantics id="S6.E4.m1.4a"><mrow id="S6.E4.m1.4.4" xref="S6.E4.m1.4.4.cmml"><mrow id="S6.E4.m1.4.4.1" xref="S6.E4.m1.4.4.1.cmml"><mi id="S6.E4.m1.4.4.1.3" xref="S6.E4.m1.4.4.1.3.cmml">P</mi><mo id="S6.E4.m1.4.4.1.2" xref="S6.E4.m1.4.4.1.2.cmml">⁢</mo><mrow id="S6.E4.m1.4.4.1.1.1" xref="S6.E4.m1.4.4.1.1.1.1.cmml"><mo id="S6.E4.m1.4.4.1.1.1.2" stretchy="false" xref="S6.E4.m1.4.4.1.1.1.1.cmml">(</mo><mrow id="S6.E4.m1.4.4.1.1.1.1" xref="S6.E4.m1.4.4.1.1.1.1.cmml"><mi id="S6.E4.m1.4.4.1.1.1.1.2" xref="S6.E4.m1.4.4.1.1.1.1.2.cmml">A</mi><mo fence="false" id="S6.E4.m1.4.4.1.1.1.1.1" xref="S6.E4.m1.4.4.1.1.1.1.1.cmml">|</mo><mi id="S6.E4.m1.4.4.1.1.1.1.3" xref="S6.E4.m1.4.4.1.1.1.1.3.cmml">B</mi></mrow><mo id="S6.E4.m1.4.4.1.1.1.3" stretchy="false" xref="S6.E4.m1.4.4.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S6.E4.m1.4.4.3" xref="S6.E4.m1.4.4.3.cmml">=</mo><mfrac id="S6.E4.m1.3.3" xref="S6.E4.m1.3.3.cmml"><mrow id="S6.E4.m1.1.1.1" xref="S6.E4.m1.1.1.1.cmml"><mi id="S6.E4.m1.1.1.1.3" xref="S6.E4.m1.1.1.1.3.cmml">P</mi><mo id="S6.E4.m1.1.1.1.2" xref="S6.E4.m1.1.1.1.2.cmml">⁢</mo><mrow id="S6.E4.m1.1.1.1.1.1" xref="S6.E4.m1.1.1.1.1.1.1.cmml"><mo id="S6.E4.m1.1.1.1.1.1.2" stretchy="false" xref="S6.E4.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S6.E4.m1.1.1.1.1.1.1" xref="S6.E4.m1.1.1.1.1.1.1.cmml"><mi id="S6.E4.m1.1.1.1.1.1.1.2" xref="S6.E4.m1.1.1.1.1.1.1.2.cmml">A</mi><mo id="S6.E4.m1.1.1.1.1.1.1.1" xref="S6.E4.m1.1.1.1.1.1.1.1.cmml">∩</mo><mi id="S6.E4.m1.1.1.1.1.1.1.3" xref="S6.E4.m1.1.1.1.1.1.1.3.cmml">B</mi></mrow><mo id="S6.E4.m1.1.1.1.1.1.3" stretchy="false" xref="S6.E4.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="S6.E4.m1.3.3.3" xref="S6.E4.m1.3.3.3.cmml"><mrow id="S6.E4.m1.3.3.3.4" xref="S6.E4.m1.3.3.3.4.cmml"><mrow id="S6.E4.m1.3.3.3.4.2" xref="S6.E4.m1.3.3.3.4.2.cmml"><mi id="S6.E4.m1.3.3.3.4.2.2" xref="S6.E4.m1.3.3.3.4.2.2.cmml">P</mi><mo id="S6.E4.m1.3.3.3.4.2.1" xref="S6.E4.m1.3.3.3.4.2.1.cmml">⁢</mo><mrow id="S6.E4.m1.3.3.3.4.2.3.2" xref="S6.E4.m1.3.3.3.4.2.cmml"><mo id="S6.E4.m1.3.3.3.4.2.3.2.1" stretchy="false" xref="S6.E4.m1.3.3.3.4.2.cmml">(</mo><mi id="S6.E4.m1.2.2.2.1" xref="S6.E4.m1.2.2.2.1.cmml">A</mi><mo id="S6.E4.m1.3.3.3.4.2.3.2.2" rspace="0.055em" stretchy="false" xref="S6.E4.m1.3.3.3.4.2.cmml">)</mo></mrow></mrow><mo id="S6.E4.m1.3.3.3.4.1" rspace="0.222em" xref="S6.E4.m1.3.3.3.4.1.cmml">×</mo><mi id="S6.E4.m1.3.3.3.4.3" xref="S6.E4.m1.3.3.3.4.3.cmml">P</mi></mrow><mo id="S6.E4.m1.3.3.3.3" xref="S6.E4.m1.3.3.3.3.cmml">⁢</mo><mrow id="S6.E4.m1.3.3.3.5.2" xref="S6.E4.m1.3.3.3.cmml"><mo id="S6.E4.m1.3.3.3.5.2.1" stretchy="false" xref="S6.E4.m1.3.3.3.cmml">(</mo><mi id="S6.E4.m1.3.3.3.2" xref="S6.E4.m1.3.3.3.2.cmml">B</mi><mo id="S6.E4.m1.3.3.3.5.2.2" stretchy="false" xref="S6.E4.m1.3.3.3.cmml">)</mo></mrow></mrow></mfrac><mo id="S6.E4.m1.4.4.4" xref="S6.E4.m1.4.4.4.cmml">=</mo><mfrac id="S6.E4.m1.4.4.5" xref="S6.E4.m1.4.4.5.cmml"><msub id="S6.E4.m1.4.4.5.2" xref="S6.E4.m1.4.4.5.2.cmml"><mi id="S6.E4.m1.4.4.5.2.2" xref="S6.E4.m1.4.4.5.2.2.cmml">O</mi><mrow id="S6.E4.m1.4.4.5.2.3" xref="S6.E4.m1.4.4.5.2.3.cmml"><mi id="S6.E4.m1.4.4.5.2.3.2" xref="S6.E4.m1.4.4.5.2.3.2.cmml">A</mi><mo id="S6.E4.m1.4.4.5.2.3.1" xref="S6.E4.m1.4.4.5.2.3.1.cmml">⁢</mo><mi id="S6.E4.m1.4.4.5.2.3.3" xref="S6.E4.m1.4.4.5.2.3.3.cmml">B</mi></mrow></msub><msub id="S6.E4.m1.4.4.5.3" xref="S6.E4.m1.4.4.5.3.cmml"><mi id="S6.E4.m1.4.4.5.3.2" xref="S6.E4.m1.4.4.5.3.2.cmml">O</mi><mi id="S6.E4.m1.4.4.5.3.3" xref="S6.E4.m1.4.4.5.3.3.cmml">B</mi></msub></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S6.E4.m1.4b"><apply id="S6.E4.m1.4.4.cmml" xref="S6.E4.m1.4.4"><and id="S6.E4.m1.4.4a.cmml" xref="S6.E4.m1.4.4"></and><apply id="S6.E4.m1.4.4b.cmml" xref="S6.E4.m1.4.4"><eq id="S6.E4.m1.4.4.3.cmml" xref="S6.E4.m1.4.4.3"></eq><apply id="S6.E4.m1.4.4.1.cmml" xref="S6.E4.m1.4.4.1"><times id="S6.E4.m1.4.4.1.2.cmml" xref="S6.E4.m1.4.4.1.2"></times><ci id="S6.E4.m1.4.4.1.3.cmml" xref="S6.E4.m1.4.4.1.3">𝑃</ci><apply id="S6.E4.m1.4.4.1.1.1.1.cmml" xref="S6.E4.m1.4.4.1.1.1"><csymbol cd="latexml" id="S6.E4.m1.4.4.1.1.1.1.1.cmml" xref="S6.E4.m1.4.4.1.1.1.1.1">conditional</csymbol><ci id="S6.E4.m1.4.4.1.1.1.1.2.cmml" xref="S6.E4.m1.4.4.1.1.1.1.2">𝐴</ci><ci id="S6.E4.m1.4.4.1.1.1.1.3.cmml" xref="S6.E4.m1.4.4.1.1.1.1.3">𝐵</ci></apply></apply><apply id="S6.E4.m1.3.3.cmml" xref="S6.E4.m1.3.3"><divide id="S6.E4.m1.3.3.4.cmml" xref="S6.E4.m1.3.3"></divide><apply id="S6.E4.m1.1.1.1.cmml" xref="S6.E4.m1.1.1.1"><times id="S6.E4.m1.1.1.1.2.cmml" xref="S6.E4.m1.1.1.1.2"></times><ci id="S6.E4.m1.1.1.1.3.cmml" xref="S6.E4.m1.1.1.1.3">𝑃</ci><apply id="S6.E4.m1.1.1.1.1.1.1.cmml" xref="S6.E4.m1.1.1.1.1.1"><intersect id="S6.E4.m1.1.1.1.1.1.1.1.cmml" xref="S6.E4.m1.1.1.1.1.1.1.1"></intersect><ci id="S6.E4.m1.1.1.1.1.1.1.2.cmml" xref="S6.E4.m1.1.1.1.1.1.1.2">𝐴</ci><ci id="S6.E4.m1.1.1.1.1.1.1.3.cmml" xref="S6.E4.m1.1.1.1.1.1.1.3">𝐵</ci></apply></apply><apply id="S6.E4.m1.3.3.3.cmml" xref="S6.E4.m1.3.3.3"><times id="S6.E4.m1.3.3.3.3.cmml" xref="S6.E4.m1.3.3.3.3"></times><apply id="S6.E4.m1.3.3.3.4.cmml" xref="S6.E4.m1.3.3.3.4"><times id="S6.E4.m1.3.3.3.4.1.cmml" xref="S6.E4.m1.3.3.3.4.1"></times><apply id="S6.E4.m1.3.3.3.4.2.cmml" xref="S6.E4.m1.3.3.3.4.2"><times id="S6.E4.m1.3.3.3.4.2.1.cmml" xref="S6.E4.m1.3.3.3.4.2.1"></times><ci id="S6.E4.m1.3.3.3.4.2.2.cmml" xref="S6.E4.m1.3.3.3.4.2.2">𝑃</ci><ci id="S6.E4.m1.2.2.2.1.cmml" xref="S6.E4.m1.2.2.2.1">𝐴</ci></apply><ci id="S6.E4.m1.3.3.3.4.3.cmml" xref="S6.E4.m1.3.3.3.4.3">𝑃</ci></apply><ci id="S6.E4.m1.3.3.3.2.cmml" xref="S6.E4.m1.3.3.3.2">𝐵</ci></apply></apply></apply><apply id="S6.E4.m1.4.4c.cmml" xref="S6.E4.m1.4.4"><eq id="S6.E4.m1.4.4.4.cmml" xref="S6.E4.m1.4.4.4"></eq><share href="https://arxiv.org/html/2501.12469v2#S6.E4.m1.3.3.cmml" id="S6.E4.m1.4.4d.cmml" xref="S6.E4.m1.4.4"></share><apply id="S6.E4.m1.4.4.5.cmml" xref="S6.E4.m1.4.4.5"><divide id="S6.E4.m1.4.4.5.1.cmml" xref="S6.E4.m1.4.4.5"></divide><apply id="S6.E4.m1.4.4.5.2.cmml" xref="S6.E4.m1.4.4.5.2"><csymbol cd="ambiguous" id="S6.E4.m1.4.4.5.2.1.cmml" xref="S6.E4.m1.4.4.5.2">subscript</csymbol><ci id="S6.E4.m1.4.4.5.2.2.cmml" xref="S6.E4.m1.4.4.5.2.2">𝑂</ci><apply id="S6.E4.m1.4.4.5.2.3.cmml" xref="S6.E4.m1.4.4.5.2.3"><times id="S6.E4.m1.4.4.5.2.3.1.cmml" xref="S6.E4.m1.4.4.5.2.3.1"></times><ci id="S6.E4.m1.4.4.5.2.3.2.cmml" xref="S6.E4.m1.4.4.5.2.3.2">𝐴</ci><ci id="S6.E4.m1.4.4.5.2.3.3.cmml" xref="S6.E4.m1.4.4.5.2.3.3">𝐵</ci></apply></apply><apply id="S6.E4.m1.4.4.5.3.cmml" xref="S6.E4.m1.4.4.5.3"><csymbol cd="ambiguous" id="S6.E4.m1.4.4.5.3.1.cmml" xref="S6.E4.m1.4.4.5.3">subscript</csymbol><ci id="S6.E4.m1.4.4.5.3.2.cmml" xref="S6.E4.m1.4.4.5.3.2">𝑂</ci><ci id="S6.E4.m1.4.4.5.3.3.cmml" xref="S6.E4.m1.4.4.5.3.3">𝐵</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.E4.m1.4c">P(A|B)=\frac{P(A\cap B)}{P(A)\times P(B)}=\frac{O_{AB}}{O_{B}}</annotation><annotation encoding="application/x-llamapun" id="S6.E4.m1.4d">italic_P ( italic_A | italic_B ) = divide start_ARG italic_P ( italic_A ∩ italic_B ) end_ARG start_ARG italic_P ( italic_A ) × italic_P ( italic_B ) end_ARG = divide start_ARG italic_O start_POSTSUBSCRIPT italic_A italic_B end_POSTSUBSCRIPT end_ARG start_ARG italic_O start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT end_ARG</annotation></semantics></math></td> <td class="ltx_eqn_cell ltx_eqn_center_padright"></td> </tr></tbody> </table> <math alttext="O_{AB}" class="ltx_Math" display="inline" id="S6.SS1.p2.1.m1.1"><semantics id="S6.SS1.p2.1.m1.1a"><msub id="S6.SS1.p2.1.m1.1.1" xref="S6.SS1.p2.1.m1.1.1.cmml"><mi id="S6.SS1.p2.1.m1.1.1.2" xref="S6.SS1.p2.1.m1.1.1.2.cmml">O</mi><mrow id="S6.SS1.p2.1.m1.1.1.3" xref="S6.SS1.p2.1.m1.1.1.3.cmml"><mi id="S6.SS1.p2.1.m1.1.1.3.2" xref="S6.SS1.p2.1.m1.1.1.3.2.cmml">A</mi><mo id="S6.SS1.p2.1.m1.1.1.3.1" xref="S6.SS1.p2.1.m1.1.1.3.1.cmml">⁢</mo><mi id="S6.SS1.p2.1.m1.1.1.3.3" xref="S6.SS1.p2.1.m1.1.1.3.3.cmml">B</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.SS1.p2.1.m1.1b"><apply id="S6.SS1.p2.1.m1.1.1.cmml" xref="S6.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S6.SS1.p2.1.m1.1.1.1.cmml" xref="S6.SS1.p2.1.m1.1.1">subscript</csymbol><ci id="S6.SS1.p2.1.m1.1.1.2.cmml" xref="S6.SS1.p2.1.m1.1.1.2">𝑂</ci><apply id="S6.SS1.p2.1.m1.1.1.3.cmml" xref="S6.SS1.p2.1.m1.1.1.3"><times id="S6.SS1.p2.1.m1.1.1.3.1.cmml" xref="S6.SS1.p2.1.m1.1.1.3.1"></times><ci id="S6.SS1.p2.1.m1.1.1.3.2.cmml" xref="S6.SS1.p2.1.m1.1.1.3.2">𝐴</ci><ci id="S6.SS1.p2.1.m1.1.1.3.3.cmml" xref="S6.SS1.p2.1.m1.1.1.3.3">𝐵</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p2.1.m1.1c">O_{AB}</annotation><annotation encoding="application/x-llamapun" id="S6.SS1.p2.1.m1.1d">italic_O start_POSTSUBSCRIPT italic_A italic_B end_POSTSUBSCRIPT</annotation></semantics></math> represents the number of days when both services A and B experience outages simultaneously, while <math alttext="O_{B}" class="ltx_Math" display="inline" id="S6.SS1.p2.2.m2.1"><semantics id="S6.SS1.p2.2.m2.1a"><msub id="S6.SS1.p2.2.m2.1.1" xref="S6.SS1.p2.2.m2.1.1.cmml"><mi id="S6.SS1.p2.2.m2.1.1.2" xref="S6.SS1.p2.2.m2.1.1.2.cmml">O</mi><mi id="S6.SS1.p2.2.m2.1.1.3" xref="S6.SS1.p2.2.m2.1.1.3.cmml">B</mi></msub><annotation-xml encoding="MathML-Content" id="S6.SS1.p2.2.m2.1b"><apply id="S6.SS1.p2.2.m2.1.1.cmml" xref="S6.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S6.SS1.p2.2.m2.1.1.1.cmml" xref="S6.SS1.p2.2.m2.1.1">subscript</csymbol><ci id="S6.SS1.p2.2.m2.1.1.2.cmml" xref="S6.SS1.p2.2.m2.1.1.2">𝑂</ci><ci id="S6.SS1.p2.2.m2.1.1.3.cmml" xref="S6.SS1.p2.2.m2.1.1.3">𝐵</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p2.2.m2.1c">O_{B}</annotation><annotation encoding="application/x-llamapun" id="S6.SS1.p2.2.m2.1d">italic_O start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT</annotation></semantics></math> indicates the number of days that service B has an outage. </div> <div class="ltx_para" id="S6.SS1.p3"> The heatmaps show that co-occurrence is notably high among services from the same provider. For OpenAI services, the API is more likely to have an outage with DALL·E (78.26%) and Playground (87.50%) than ChatGPT (49.21%). For Anthropic’s services, the likelihood of any two services experiencing outages on the same day is extremely high over 80%, this may be caused by a lack of isolation across different services. There is no correlation observed between services from different providers. The lack of correlation suggests that user can use one service as the other’s backup to increase their reliability. The difference in co-occurrence between OpenAI and Anthropic suggests that outages could be reduced through better service isolation. Based on publicly available information, this may be caused by the different cloud infrastructures used by LLMs: OpenAI relies on Azure <cite class="ltx_cite ltx_citemacro_citep">(azu, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib2" title="">2025</a>)</cite>, while Anthropic uses GCP <cite class="ltx_cite ltx_citemacro_citep">(gcp, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib4" title="">2025b</a>)</cite>. </div> </section> <section class="ltx_subsection" id="S6.SS2"> <h3 class="ltx_title ltx_title_subsection"> 6.2. Impact Range of Incidents</h3> <figure class="ltx_figure ltx_minipage ltx_align_center ltx_align_middle" id="S6.F11" style="width:433.6pt;"> <div class="ltx_flex_figure"> <div class="ltx_flex_cell ltx_flex_size_2"> <figure class="ltx_figure ltx_figure_panel" id="S6.F11.1.1"><img alt="Refer to caption" class="ltx_graphics ltx_img_square" height="672" id="S6.F11.1.1.g1" src="x16.png" width="831"/> </figure> </div> <div class="ltx_flex_cell ltx_flex_size_2"> <figure class="ltx_figure ltx_figure_panel" id="S6.F11.2.2"><img alt="Refer to caption" class="ltx_graphics ltx_img_square" height="728" id="S6.F11.2.2.g1" src="x17.png" width="829"/> </figure> </div> </div> <figcaption class="ltx_caption ltx_centering">Figure 11. Impact Services of OpenAI and Anthropic incidents, respectively.</figcaption> </figure> <figure class="ltx_table" id="S6.T8"> <figcaption class="ltx_caption">Table 8. Percentage of the number of impacted services.</figcaption> <div class="ltx_inline-block ltx_transformed_outer" id="S6.T8.4" style="width:294.9pt;height:70.2pt;vertical-align:-0.0pt;"> <table class="ltx_tabular ltx_align_middle" id="S6.T8.4.1"> <tr class="ltx_tr" id="S6.T8.4.1.1"> <td class="ltx_td ltx_border_tt" id="S6.T8.4.1.1.1"></td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S6.T8.4.1.1.2">1</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S6.T8.4.1.1.3">2</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S6.T8.4.1.1.4">3</td> <td class="ltx_td ltx_align_right ltx_border_tt" id="S6.T8.4.1.1.5">4</td> </tr> <tr class="ltx_tr" id="S6.T8.4.1.2"> <td class="ltx_td ltx_align_left ltx_border_t" id="S6.T8.4.1.2.1">OpenAI</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S6.T8.4.1.2.2">57.63%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S6.T8.4.1.2.3">31.44%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S6.T8.4.1.2.4">2.73%</td> <td class="ltx_td ltx_align_right ltx_border_t" id="S6.T8.4.1.2.5">8.20%</td> </tr> <tr class="ltx_tr" id="S6.T8.4.1.3"> <td class="ltx_td ltx_align_left ltx_border_bb" id="S6.T8.4.1.3.1">Anthropic</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S6.T8.4.1.3.2">12.82%</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S6.T8.4.1.3.3">15.38%</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S6.T8.4.1.3.4">71.79%</td> <td class="ltx_td ltx_align_right ltx_border_bb" id="S6.T8.4.1.3.5">-</td> </tr> </table> </div> </figure> <div class="ltx_para ltx_noindent" id="S6.SS2.p1"> Observation #12: 71.79% of Anthropic incidents affect all its services, compared to only 8.20% for OpenAI. </div> <div class="ltx_para" id="S6.SS2.p2"> The incident reports indicate that one incident can impact several services, which is the impact range of incidents. <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S6.F11" title="In 6.2. Impact Range of Incidents ‣ 6. Co-occurrence of Failures ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Figure 11</a> gives the impacted services combinations of OpenAI and Anthropic incidents based on their reports, respectively. For Anthropic, the majority of incidents (71.79%) impact its 3 services jointly. However, for OpenAI, only 8.20% of services are impacted together, and over half (57.63%) of incidents affect a single service. </div> </section> </section> <section class="ltx_section" id="S7"> <h2 class="ltx_title ltx_title_section"> 7. Limitations and Validity</h2> <div class="ltx_para" id="S7.p1"> The generality of our work is limited to the services we analyze. We analyze LLM-related services from three popular operators, including the currently most popular (OpenAI). However, other, very different services could exist, e.g., allowing users to self-host LLMs (e.g., Anyscale), and LLM services from big cloud providers and/or used primarily internally (e.g., Google Gemini). </div> <div class="ltx_para" id="S7.p2"> The accuracy of our failure dataset is limited to what the LLM operators themselves report. That makes our data subject to the operators’ bias. Prior work <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib23" title="">2020</a>; Talluri et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib55" title="">2021</a>)</cite> suggests the operator’s reports already capture the most user-visible failures as those generate widespread social media coverage, making it difficult for the operators to hide failure; to confirm this for LLM services, we need to collect data from other sources such as the user devices <cite class="ltx_cite ltx_citemacro_citep">(Burnett et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib12" title="">2020</a>)</cite> or user failure reports <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib23" title="">2020</a>; Talluri et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib55" title="">2021</a>)</cite>. </div> <div class="ltx_para" id="S7.p3"> The depth of our analysis regarding the root cause of failures is limited. We glean limited information from the operators’ failure reports regarding the hardware and software infrastructure. To confirm our findings, we would ideally use detailed infrastructure and application-level data. However, this requires active help from the service operators, e.g., releasing their system traces as Google did with its cluster workloads <cite class="ltx_cite ltx_citemacro_citep">(Google, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib20" title="">2019</a>)</cite>. </div> <div class="ltx_para" id="S7.p4"> The scope of our work is limited to LLM services. We ignore other deep learning services such as Image Generation (e.g., Stable Diffusion, Midjourney), Translation (e.g., DeepL), etc. However, we believe the operational characteristics of LLM services are valuable in and of themselves. LLMs have gained broad general public adoption and mindshare, as described in <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#S1" title="1. Introduction ‣ An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models">Section 1</a>. LLM services now also support multi-modal use cases such as image generation and image-based question answering, making them some of the most general deep learning tools currently available. </div> </section> <section class="ltx_section" id="S8"> <h2 class="ltx_title ltx_title_section"> 8. Related Work</h2> <div class="ltx_para" id="S8.p1"> Overall, this work complements the existing body of work on failure characterization, modeling, and more generally failure-recovery, with a focus on the emerging area of LLM services. Ours is the first comprehensive, longitudinal data collection and empirical characterization of public LLM services. </div> <div class="ltx_para ltx_noindent" id="S8.p2"> Operational failure characterization of workstations <cite class="ltx_cite ltx_citemacro_citep">(Javadi et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib31" title="">2013</a>)</cite>, HPC sites <cite class="ltx_cite ltx_citemacro_citep">(Gupta et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib21" title="">2017</a>)</cite>, clouds <cite class="ltx_cite ltx_citemacro_citep">(Garraghan et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib19" title="">2014</a>)</cite>, big data jobs <cite class="ltx_cite ltx_citemacro_citep">(Rosà et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib50" title="">2015</a>)</cite>, networks <cite class="ltx_cite ltx_citemacro_citep">(Potharaju and Jain, <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib43" title="">2013</a>)</cite>, storage devices <cite class="ltx_cite ltx_citemacro_citep">(Alter et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib5" title="">2019</a>)</cite>, CPUs <cite class="ltx_cite ltx_citemacro_citep">(Hochschild et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib22" title="">2021</a>)</cite>, and GPUs <cite class="ltx_cite ltx_citemacro_citep">(Tiwari et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib56" title="">2015</a>)</cite> has led to improved application designs and fault-tolerance mechanisms. Leading from these, we have better failure detection <cite class="ltx_cite ltx_citemacro_citep">(Burnett et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib12" title="">2020</a>; Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib23" title="">2020</a>)</cite>, checkpointing <cite class="ltx_cite ltx_citemacro_citep">(Garg et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib18" title="">2018</a>)</cite>, retry <cite class="ltx_cite ltx_citemacro_citep">(Primorac et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib44" title="">2021</a>)</cite>, and replication mechanisms <cite class="ltx_cite ltx_citemacro_citep">(Shen et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib53" title="">2015</a>)</cite>. However, these do not cover deep learning and particularly LLM services. </div> <div class="ltx_para" id="S8.p3"> There is existing work on operational characteristics of GPUs for deep learning <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib36" title="">2017</a>)</cite>, ML jobs on HPC clusters <cite class="ltx_cite ltx_citemacro_citep">(Chu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib14" title="">2024</a>)</cite>, and deep-learning clusters <cite class="ltx_cite ltx_citemacro_citep">(Jeon et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib32" title="">2019</a>)</cite>. However, no work has described the operational failure characteristics of user-facing deep learning services. Our study addresses this gap, focusing on LLMs. </div> <div class="ltx_para ltx_noindent" id="S8.p4"> Deep learning workloads have been characterized including their GPU utilization <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib24" title="">2021</a>; Li et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib35" title="">2022</a>)</cite>, network characteristics <cite class="ltx_cite ltx_citemacro_citep">(Awan et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib11" title="">2020</a>)</cite>, and storage characteristics <cite class="ltx_cite ltx_citemacro_citep">(Chien et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib13" title="">2018</a>)</cite>. User-facing machine learning workloads have also been characterized <cite class="ltx_cite ltx_citemacro_citep">(Weng et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib64" title="">2022</a>)</cite>. The studies complement our work as they explore different hardware/software stack layers. We complement the studies by enhancing the community’s understanding of LLM failures at the user-facing application layer. </div> <div class="ltx_para ltx_noindent" id="S8.p5"> LLM workloads have been characterized at the preliminary-level for training <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib25" title="">2024</a>)</cite>, fine-tuning <cite class="ltx_cite ltx_citemacro_citep">(Xia et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib66" title="">2024</a>; Wang et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib63" title="">2024a</a>)</cite>, and inference <cite class="ltx_cite ltx_citemacro_citep">(Łazuka et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib34" title="">2024</a>)</cite>. Failures have been assessed briefly; e.g., found to occur frequently (<math alttext="\sim" class="ltx_Math" display="inline" id="S8.p5.1.m1.1"><semantics id="S8.p5.1.m1.1a"><mo id="S8.p5.1.m1.1.1" xref="S8.p5.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S8.p5.1.m1.1b"><csymbol cd="latexml" id="S8.p5.1.m1.1.1.cmml" xref="S8.p5.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S8.p5.1.m1.1c">\sim</annotation><annotation encoding="application/x-llamapun" id="S8.p5.1.m1.1d">∼</annotation></semantics></math>9 hour MTBF) in LLM training <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a class="ltx_ref" href="https://arxiv.org/html/2501.12469v2#bib.bib25" title="">2024</a>)</cite>, compared to around 4 days MTBF for the user-facing services in this work. Fine-tuning and inference workloads have not been characterized, especially concerning failures. Ours is the first study to focus on failures occurring in public LLM services, with unique contributions in longitudinal analysis and in collecting comprehensive data from multiple services. </div> </section> <section class="ltx_section" id="S9"> <h2 class="ltx_title ltx_title_section"> 9. Conclusion</h2> <div class="ltx_para" id="S9.p1"> Understanding the characteristics of failures in the operation of public LLM services has become a stringent problem, driven by the rapid increase in the popularity of such services, market competitiveness, and increasingly self-reported presence of such failures by LLM service providers. Addressing this problem, in this work we have conducted a comprehensive empirical characterization of long-term outages and incidents in public LLM services. </div> <div class="ltx_para" id="S9.p2"> Our main findings includes: (1) Different LLM services take varying amounts of time at different stages of the failure-recovery process. For example, Anthropic services spent more time for investigating and resolving issues than OpenAI services. (2) OpenAI and Anthropic’s services failures exhibit periodic patterns that are more frequent on weekdays than on weekends. However, Character.AI has fewer failures on Tuesdays and Wednesdays. (3) Co-occurrence is significantly higher among services within the same provider, with no clear co-occurrence observed between services from different providers. </div> <div class="ltx_para" id="S9.p3"> Overall, we emphasized over 10 observations, which scientists, engineers, and users could include directly in their knowledge base, and from which improvements to LLM systems could occur in time. For the future work, we aim to lead a community effort where LLM service availability datasets, collected long-term and processed to provide similar information, can be shared. Future analysis could include promising emerging LLM services in different countries such as Google Gemini, Mistral AI, and DeepSeek. </div> <div class="ltx_acknowledgements"> <h6 class="ltx_title ltx_title_acknowledgements">Acknowledgements.</h6> This work was supported by the EU Horizon Graph Massivizer and the EU MSCA Cloudstars projects. This research was partly supported by a National Growth Fund through the Dutch 6G flagship project ”Future Network Services”. We thank the China Scholarship Council (CSC) for supporting Xiaoyu Chu. We thank Daniel Hofstätter, Krijn Doekemeijer, and Radu Nicolae for their critical review of the manuscript. </div> </section> <section class="ltx_bibliography" id="bib"> <h2 class="ltx_title ltx_title_bibliography">References</h2> <ul class="ltx_biblist"> <li class="ltx_bibitem" id="bib.bib1"> (1) </li> <li class="ltx_bibitem" id="bib.bib2"> azu (2025) 2025. Azure OpenAI Service. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://azure.microsoft.com/en-us/products/ai-services/openai-service" title="">https://azure.microsoft.com/en-us/products/ai-services/openai-service</a>. </li> <li class="ltx_bibitem" id="bib.bib3"> gcp (2025a) 2025a. Scaling Character.AI. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://cloud.google.com/blog/products/databases/why-characterai-chose-spanner-and-alloydb-for-postgresql" title="">https://cloud.google.com/blog/products/databases/why-characterai-chose-spanner-and-alloydb-for-postgresql</a>. </li> <li class="ltx_bibitem" id="bib.bib4"> gcp (2025b) 2025b. Use Anthropic’s Claude models. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude" title="">https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude</a>. </li> <li class="ltx_bibitem" id="bib.bib5"> Alter et al. (2019) Jacob Alter, Ji Xue, Alma Dimnaku, and Evgenia Smirni. 2019. SSD failures in the field: symptoms, causes, and prediction models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019, Denver, Colorado, USA, November 17-19, 2019, Michela Taufer, Pavan Balaji, and Antonio J. Peña (Eds.). ACM, 75:1–75:14. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3295500.3356172" title="">doi:10.1145/3295500.3356172</a> </li> <li class="ltx_bibitem" id="bib.bib6"> Anthropic (2024a) Anthropic. 2024a. Release notes of Anthropic API. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://docs.anthropic.com/en/release-notes/api" title="">https://docs.anthropic.com/en/release-notes/api</a>. </li> <li class="ltx_bibitem" id="bib.bib7"> Anthropic (2024b) Anthropic. 2024b. Release notes of Claude. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://docs.anthropic.com/en/release-notes/claude-apps" title="">https://docs.anthropic.com/en/release-notes/claude-apps</a>. </li> <li class="ltx_bibitem" id="bib.bib8"> Archive (2024) Internet Archive. 2024. Archived page of OpenAI status at 2024/04/10. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://web.archive.org/web/20240410235249/https://downdetector.com/status/openai/" title="">https://web.archive.org/web/20240410235249/https://downdetector.com/status/openai/</a>, Accessed: 2024-09-30. </li> <li class="ltx_bibitem" id="bib.bib9"> Atlassian (2024) Atlassian. 2024. Atlassian Support. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://support.atlassian.com/statuspage/docs/display-historical-uptime-of-components/" title="">https://support.atlassian.com/statuspage/docs/display-historical-uptime-of-components/</a>. </li> <li class="ltx_bibitem" id="bib.bib10"> Avizienis et al. (2004) Algirdas Avizienis, Jean-Claude Laprie, Brian Randell, and Carl E. Landwehr. 2004. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Trans. Dependable Secur. Comput. 1, 1 (2004), 11–33. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/TDSC.2004.2" title="">doi:10.1109/TDSC.2004.2</a> </li> <li class="ltx_bibitem" id="bib.bib11"> Awan et al. (2020) Ammar Ahmad Awan, Arpan Jain, Ching-Hsiang Chu, Hari Subramoni, and Dhabaleswar K. Panda. 2020. Communication Profiling and Characterization of Deep-Learning Workloads on Clusters With High-Performance Interconnects. IEEE Micro 40, 1 (2020), 35–43. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/MM.2019.2949986" title="">doi:10.1109/MM.2019.2949986</a> </li> <li class="ltx_bibitem" id="bib.bib12"> Burnett et al. (2020) Sam Burnett, Lily Chen, Douglas A. Creager, Misha Efimov, Ilya Grigorik, Ben Jones, Harsha V. Madhyastha, Pavlos Papageorge, Brian Rogan, Charles Stahl, and Julia Tuttle. 2020. Network Error Logging: Client-side measurement of end-to-end web service reliability. In 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020, Santa Clara, CA, USA, February 25-27, 2020, Ranjita Bhagwan and George Porter (Eds.). USENIX Association, 985–998. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.usenix.org/conference/nsdi20/presentation/burnett" title="">https://www.usenix.org/conference/nsdi20/presentation/burnett</a> </li> <li class="ltx_bibitem" id="bib.bib13"> Chien et al. (2018) Steven Wei Der Chien, Stefano Markidis, Chaitanya Prasad Sishtla, Luís Santos, Pawel Andrzej Herman, Sai Narasimhamurthy, and Erwin Laure. 2018. Characterizing Deep-Learning I/O Workloads in TensorFlow. In 3rd IEEE/ACM International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems, PDSW-DISCS@SC 2018, Dallas, TX, USA, November 12, 2018. IEEE, 54–63. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/PDSW-DISCS.2018.00011" title="">doi:10.1109/PDSW-DISCS.2018.00011</a> </li> <li class="ltx_bibitem" id="bib.bib14"> Chu et al. (2024) Xiaoyu Chu, Daniel Hofstätter, Shashikant Ilager, Sacheendra Talluri, Duncan Kampert, Damian Podareanu, Dmitry Duplyakin, Ivona Brandic, and Alexandru Iosup. 2024. Generic and ML Workloads in an HPC Datacenter: Node Energy, Job Failures, and Node-Job Analysis. arXiv:2409.08949 [cs.DC] <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://arxiv.org/abs/2409.08949" title="">https://arxiv.org/abs/2409.08949</a> </li> <li class="ltx_bibitem" id="bib.bib15"> Chu et al. (2023) Xiaoyu Chu, Sacheendra Talluri, Laurens Versluis, and Alexandru Iosup. 2023. How Do ML Jobs Fail in Datacenters? Analysis of a Long-Term Dataset from an HPC Cluster. In Proceedings of the International Conference on Performance Engineering, Coimbra, Portugal, April, 2023. </li> <li class="ltx_bibitem" id="bib.bib16"> DownDetector (2024) DownDetector. 2024. OpenAI user reports. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://downdetector.com/status/openai/" title="">https://downdetector.com/status/openai/</a>. </li> <li class="ltx_bibitem" id="bib.bib17"> Gamell et al. (2014) Marc Gamell, Daniel S. Katz, Hemanth Kolla, Jacqueline Chen, Scott Klasky, and Manish Parashar. 2014. Exploring Automatic, Online Failure Recovery for Scientific Applications at Extreme Scales. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2014, New Orleans, LA, USA, November 16-21, 2014, Trish Damkroger and Jack J. Dongarra (Eds.). IEEE Computer Society, 895–906. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/SC.2014.78" title="">doi:10.1109/SC.2014.78</a> </li> <li class="ltx_bibitem" id="bib.bib18"> Garg et al. (2018) Rohan Garg, Tirthak Patel, Gene Cooperman, and Devesh Tiwari. 2018. Shiraz: Exploiting System Reliability and Application Resilience Characteristics to Improve Large Scale System Throughput. In 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018, Luxembourg City, Luxembourg, June 25-28, 2018. IEEE Computer Society, 83–94. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/DSN.2018.00021" title="">doi:10.1109/DSN.2018.00021</a> </li> <li class="ltx_bibitem" id="bib.bib19"> Garraghan et al. (2014) Peter Garraghan, Paul Townend, and Jie Xu. 2014. An Empirical Failure-Analysis of a Large-Scale Cloud Computing Environment. In 15th International IEEE Symposium on High-Assurance Systems Engineering, HASE 2014, Miami Beach, FL, USA, January 9-11, 2014. IEEE Computer Society, 113–120. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/HASE.2014.24" title="">doi:10.1109/HASE.2014.24</a> </li> <li class="ltx_bibitem" id="bib.bib20"> Google (2019) Google. 2019. Google Cluster Traces 2019. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://github.com/google/cluster-data?tab=readme-ov-file" title="">https://github.com/google/cluster-data?tab=readme-ov-file</a>. </li> <li class="ltx_bibitem" id="bib.bib21"> Gupta et al. (2017) Saurabh Gupta, Tirthak Patel, Christian Engelmann, and Devesh Tiwari. 2017. Failures in large scale systems: long-term measurement, analysis, and implications. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, Denver, CO, USA, November 12 - 17, 2017, Bernd Mohr and Padma Raghavan (Eds.). ACM, 44. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3126908.3126937" title="">doi:10.1145/3126908.3126937</a> </li> <li class="ltx_bibitem" id="bib.bib22"> Hochschild et al. (2021) Peter H. Hochschild, Paul Turner, Jeffrey C. Mogul, Rama Govindaraju, Parthasarathy Ranganathan, David E. Culler, and Amin Vahdat. 2021. Cores that don’t count. In HotOS ’21: Workshop on Hot Topics in Operating Systems, Ann Arbor, Michigan, USA, June, 1-3, 2021, Sebastian Angel, Baris Kasikci, and Eddie Kohler (Eds.). ACM, 9–16. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3458336.3465297" title="">doi:10.1145/3458336.3465297</a> </li> <li class="ltx_bibitem" id="bib.bib23"> Hu et al. (2020) Jiyao Hu, Zhenyu Zhou, Xiaowei Yang, Jacob Malone, and Jonathan W. Williams. 2020. CableMon: Improving the Reliability of Cable Broadband Networks via Proactive Network Maintenance. In 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020, Santa Clara, CA, USA, February 25-27, 2020, Ranjita Bhagwan and George Porter (Eds.). USENIX Association, 619–632. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.usenix.org/conference/nsdi20/presentation/hu-jiyao" title="">https://www.usenix.org/conference/nsdi20/presentation/hu-jiyao</a> </li> <li class="ltx_bibitem" id="bib.bib24"> Hu et al. (2021) Qinghao Hu, Peng Sun, Shengen Yan, Yonggang Wen, and Tianwei Zhang. 2021. Characterization and prediction of deep learning workloads in large-scale GPU datacenters. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021, Bronis R. de Supinski, Mary W. Hall, and Todd Gamblin (Eds.). ACM, 104. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3458817.3476223" title="">doi:10.1145/3458817.3476223</a> </li> <li class="ltx_bibitem" id="bib.bib25"> Hu et al. (2024) Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, Yonggang Wen, and Tianwei Zhang. 2024. Characterization of Large Language Model Development in the Datacenter. In 21st USENIX Symposium on Networked Systems Design and Implementation, NSDI 2024, Santa Clara, CA, April 15-17, 2024, Laurent Vanbever and Irene Zhang (Eds.). USENIX Association, 709–729. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.usenix.org/conference/nsdi24/presentation/hu" title="">https://www.usenix.org/conference/nsdi24/presentation/hu</a> </li> <li class="ltx_bibitem" id="bib.bib26"> Hub (2022) Singularity Hub. 2022. OpenAI Says DALL-E Is Generating Over 2 Million Images a Day. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://singularityhub.com/2022/10/03/openai-says-dall-e-is-generating-over-2-million-images-a-day-and-thats-just-table-stakes/" title="">https://singularityhub.com/2022/10/03/openai-says-dall-e-is-generating-over-2-million-images-a-day-and-thats-just-table-stakes/</a>. </li> <li class="ltx_bibitem" id="bib.bib27"> IDC (2024) IDC. 2024. A Deep Dive Into Global AI and Generative AI Spending. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://blogs.idc.com/2024/08/16/a-deep-dive-into-idcs-global-ai-and-generative-ai-spending/" title="">https://blogs.idc.com/2024/08/16/a-deep-dive-into-idcs-global-ai-and-generative-ai-spending/</a>. </li> <li class="ltx_bibitem" id="bib.bib28"> Incident (2024a) Anthropic Incident. 2024a. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://status.anthropic.com/history" title="">https://status.anthropic.com/history</a>. </li> <li class="ltx_bibitem" id="bib.bib29"> Incident (2024b) Character.AI Incident. 2024b. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://status.character.ai/history" title="">https://status.character.ai/history</a>. </li> <li class="ltx_bibitem" id="bib.bib30"> Incident (2024c) OpenAI Incident. 2024c. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://status.openai.com/history" title="">https://status.openai.com/history</a>. </li> <li class="ltx_bibitem" id="bib.bib31"> Javadi et al. (2013) Bahman Javadi, Derrick Kondo, Alexandru Iosup, and Dick H. J. Epema. 2013. The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems. J. Parallel Distributed Comput. 73, 8 (2013), 1208–1223. <a class="ltx_ref ltx_href" href="https://doi.org/10.1016/J.JPDC.2013.04.002" title="">doi:10.1016/J.JPDC.2013.04.002</a> </li> <li class="ltx_bibitem" id="bib.bib32"> Jeon et al. (2019) Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. In Proceedings of the 2019 USENIX Annual Technical Conference, USENIX ATC 2019, Renton, WA, USA, July 10-12, 2019, Dahlia Malkhi and Dan Tsafrir (Eds.). USENIX Association, 947–960. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.usenix.org/conference/atc19/presentation/jeon" title="">https://www.usenix.org/conference/atc19/presentation/jeon</a> </li> <li class="ltx_bibitem" id="bib.bib33"> Koh et al. (2023) Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. 2023. Generating Images with Multimodal Language Models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). <a class="ltx_ref ltx_url ltx_font_typewriter" href="http://papers.nips.cc/paper_files/paper/2023/hash/43a69d143273bd8215578bde887bb552-Abstract-Conference.html" title="">http://papers.nips.cc/paper_files/paper/2023/hash/43a69d143273bd8215578bde887bb552-Abstract-Conference.html</a> </li> <li class="ltx_bibitem" id="bib.bib34"> Łazuka et al. (2024) Małgorzata Łazuka, Andreea Anghel, and Thomas Parnell. 2024. LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services. arXiv preprint arXiv:2410.02425 (2024). </li> <li class="ltx_bibitem" id="bib.bib35"> Li et al. (2022) Baolin Li, Rohin Arora, Siddharth Samsi, Tirthak Patel, William Arcand, David Bestor, Chansup Byun, Rohan Basu Roy, Bill Bergeron, John T. Holodnak, Michael Houle, Matthew Hubbell, Michael Jones, Jeremy Kepner, Anna Klein, Peter Michaleas, Joseph McDonald, Lauren Milechin, Julie Mullen, Andrew Prout, Benjamin Price, Albert Reuther, Antonio Rosa, Matthew L. Weiss, Charles Yee, Daniel Edelman, Allan Vanterpool, Anson Cheng, Vijay Gadepally, and Devesh Tiwari. 2022. AI-Enabling Workloads on Large-Scale GPU-Accelerated System: Characterization, Opportunities, and Implications. In IEEE International Symposium on High-Performance Computer Architecture, HPCA 2022, Seoul, South Korea, April 2-6, 2022. IEEE, 1224–1237. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/HPCA53966.2022.00093" title="">doi:10.1109/HPCA53966.2022.00093</a> </li> <li class="ltx_bibitem" id="bib.bib36"> Li et al. (2017) Guanpeng Li, Siva Kumar Sastry Hari, Michael B. Sullivan, Timothy Tsai, Karthik Pattabiraman, Joel S. Emer, and Stephen W. Keckler. 2017. Understanding error propagation in deep learning neural network (DNN) accelerators and applications. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, Denver, CO, USA, November 12 - 17, 2017, Bernd Mohr and Padma Raghavan (Eds.). ACM, 8. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3126908.3126964" title="">doi:10.1145/3126908.3126964</a> </li> <li class="ltx_bibitem" id="bib.bib37"> Li et al. (2023) Jie Li, Rui Wang, Ghazanfar Ali, Tommy Dang, Alan Sill, and Yong Chen. 2023. Workload Failure Prediction for Data Centers. In 16th IEEE International Conference on Cloud Computing, CLOUD 2023, Chicago, IL, USA, July 2-8, 2023. IEEE, 479–485. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/CLOUD60044.2023.00064" title="">doi:10.1109/CLOUD60044.2023.00064</a> </li> <li class="ltx_bibitem" id="bib.bib38"> Liu et al. (2023) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). <a class="ltx_ref ltx_url ltx_font_typewriter" href="http://papers.nips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html" title="">http://papers.nips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html</a> </li> <li class="ltx_bibitem" id="bib.bib39"> Lu et al. (2023) Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. 2023. LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). <a class="ltx_ref ltx_url ltx_font_typewriter" href="http://papers.nips.cc/paper_files/paper/2023/hash/47f30d67bce3e9824928267e9355420f-Abstract-Conference.html" title="">http://papers.nips.cc/paper_files/paper/2023/hash/47f30d67bce3e9824928267e9355420f-Abstract-Conference.html</a> </li> <li class="ltx_bibitem" id="bib.bib40"> Martino et al. (2014) Catello Di Martino, Zbigniew T. Kalbarczyk, Ravishankar K. Iyer, Fabio Baccanico, Joseph Fullop, and William Kramer. 2014. Lessons Learned from the Analysis of System Failures at Petascale: The Case of Blue Waters. In 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2014, Atlanta, GA, USA, June 23-26, 2014. IEEE Computer Society, 610–621. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/DSN.2014.62" title="">doi:10.1109/DSN.2014.62</a> </li> <li class="ltx_bibitem" id="bib.bib41"> OpenAI (2024a) OpenAI. 2024a. Elevated errors in ChatGPT. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://status.openai.com/incidents/w20mcckg1748" title="">https://status.openai.com/incidents/w20mcckg1748</a>, Accessed: 2024-09-30. </li> <li class="ltx_bibitem" id="bib.bib42"> OpenAI (2024b) OpenAI. 2024b. Start using ChatGPT instantly (Apr 1, 2024). <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://help.openai.com/en/articles/6825453-chatgpt-release-notes" title="">https://help.openai.com/en/articles/6825453-chatgpt-release-notes</a>. </li> <li class="ltx_bibitem" id="bib.bib43"> Potharaju and Jain (2013) Rahul Potharaju and Navendu Jain. 2013. When the network crumbles: an empirical study of cloud network failures and their impact on services. In ACM Symposium on Cloud Computing, SOCC ’13, Santa Clara, CA, USA, October 1-3, 2013, Guy M. Lohman (Ed.). ACM, 15:1–15:17. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/2523616.2523638" title="">doi:10.1145/2523616.2523638</a> </li> <li class="ltx_bibitem" id="bib.bib44"> Primorac et al. (2021) Mia Primorac, Katerina J. Argyraki, and Edouard Bugnion. 2021. When to Hedge in Interactive Services. In 18th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2021, April 12-14, 2021, James Mickens and Renata Teixeira (Eds.). USENIX Association, 373–387. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.usenix.org/conference/nsdi21/presentation/primorac" title="">https://www.usenix.org/conference/nsdi21/presentation/primorac</a> </li> <li class="ltx_bibitem" id="bib.bib45"> Pro (2024) Similarweb Pro. 2024. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://pro.similarweb.com/" title="">https://pro.similarweb.com/</a>. </li> <li class="ltx_bibitem" id="bib.bib46"> Reiss et al. (2012) Charles Reiss, Alexey Tumanov, Gregory R. Ganger, Randy H. Katz, and Michael A. Kozuch. 2012. Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In ACM Symposium on Cloud Computing, SOCC ’12, San Jose, CA, USA, October 14-17, 2012, Michael J. Carey and Steven Hand (Eds.). ACM, 7. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/2391229.2391236" title="">doi:10.1145/2391229.2391236</a> </li> <li class="ltx_bibitem" id="bib.bib47"> Reuters (2023) Reuters. 2023. ChatGPT sets record for fastest-growing user base - analyst note. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/" title="">https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/</a> </li> <li class="ltx_bibitem" id="bib.bib48"> Reuters (2024a) Reuters. 2024a. Google-backed Anthropic releases Claude chatbot across Europe. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.reuters.com/technology/google-backed-anthropic-releases-claude-chatbot-across-europe-2024-05-13/" title="">https://www.reuters.com/technology/google-backed-anthropic-releases-claude-chatbot-across-europe-2024-05-13/</a>. </li> <li class="ltx_bibitem" id="bib.bib49"> Reuters (2024b) Reuters. 2024b. OpenAI says ChatGPT’s weekly users have grown to 200 million. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.reuters.com/technology/artificial-intelligence/openai-says-chatgpts-weekly-users-have-grown-200-million-2024-08-29/" title="">https://www.reuters.com/technology/artificial-intelligence/openai-says-chatgpts-weekly-users-have-grown-200-million-2024-08-29/</a>. </li> <li class="ltx_bibitem" id="bib.bib50"> Rosà et al. (2015) Andrea Rosà, Lydia Y. Chen, and Walter Binder. 2015. Predicting and Mitigating Jobs Failures in Big Data Clusters. In 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2015, Shenzhen, China, May 4-7, 2015. IEEE Computer Society, 221–230. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/CCGRID.2015.139" title="">doi:10.1109/CCGRID.2015.139</a> </li> <li class="ltx_bibitem" id="bib.bib51"> Roumeliotis and Tselikas (2023) Konstantinos I. Roumeliotis and Nikolaos D. Tselikas. 2023. ChatGPT and Open-AI Models: A Preliminary Review. Future Internet 15, 6 (2023). <a class="ltx_ref ltx_href" href="https://doi.org/10.3390/fi15060192" title="">doi:10.3390/fi15060192</a> </li> <li class="ltx_bibitem" id="bib.bib52"> Selenium (2024) Selenium. 2024. WebDriver Documentation. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.selenium.dev/documentation/webdriver/" title="">https://www.selenium.dev/documentation/webdriver/</a>. </li> <li class="ltx_bibitem" id="bib.bib53"> Shen et al. (2015) Siqi Shen, Alexandru Iosup, Assaf Israel, Walfredo Cirne, Danny Raz, and Dick H. J. Epema. 2015. An Availability-on-Demand Mechanism for Datacenters. In 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2015, Shenzhen, China, May 4-7, 2015. IEEE Computer Society, 495–504. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/CCGRID.2015.58" title="">doi:10.1109/CCGRID.2015.58</a> </li> <li class="ltx_bibitem" id="bib.bib54"> Talluri et al. (2019) Sacheendra Talluri, Alicja Luszczak, Cristina L. Abad, and Alexandru Iosup. 2019. Characterization of a Big Data Storage Workload in the Cloud. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering, ICPE 2019, Mumbai, India, April 7-11, 2019, Varsha Apte, Antinisca Di Marco, Marin Litoiu, and José Merseguer (Eds.). ACM, 33–44. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3297663.3310302" title="">doi:10.1145/3297663.3310302</a> </li> <li class="ltx_bibitem" id="bib.bib55"> Talluri et al. (2021) Sacheendra Talluri, Leon Overweel, Laurens Versluis, Animesh Trivedi, and Alexandru Iosup. 2021. Empirical Characterization of User Reports about Cloud Failures. In IEEE International Conference on Autonomic Computing and Self-Organizing Systems, ACSOS 2021, Washington, DC, USA, September 27 - Oct. 1, 2021, Esam El-Araby, Vana Kalogeraki, Danilo Pianini, Frédéric Lassabe, Barry Porter, Sona Ghahremani, Ingrid Nunes, Mohamed Bakhouya, and Sven Tomforde (Eds.). IEEE, 158–163. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/ACSOS52086.2021.00039" title="">doi:10.1109/ACSOS52086.2021.00039</a> </li> <li class="ltx_bibitem" id="bib.bib56"> Tiwari et al. (2015) Devesh Tiwari, Saurabh Gupta, James H. Rogers, Don Maxwell, Paolo Rech, Sudharshan S. Vazhkudai, Daniel Oliveira, Dave Londo, Nathan DeBardeleben, Philippe Olivier Alexandre Navaux, Luigi Carro, and Arthur S. Bland. 2015. Understanding GPU errors on large-scale HPC systems and the implications for system design and operation. In 21st IEEE International Symposium on High Performance Computer Architecture, HPCA 2015, Burlingame, CA, USA, February 7-11, 2015. IEEE Computer Society, 331–342. <a class="ltx_ref ltx_href" href="https://doi.org/10.1109/HPCA.2015.7056044" title="">doi:10.1109/HPCA.2015.7056044</a> </li> <li class="ltx_bibitem" id="bib.bib57"> Uptime (2024a) Anthropic Uptime. 2024a. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://status.anthropic.com/uptime" title="">https://status.anthropic.com/uptime</a>. </li> <li class="ltx_bibitem" id="bib.bib58"> Uptime (2024b) Character.AI Uptime. 2024b. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://status.character.ai/uptime" title="">https://status.character.ai/uptime</a>. </li> <li class="ltx_bibitem" id="bib.bib59"> Uptime (2024c) OpenAI Uptime. 2024c. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://status.openai.com/uptime" title="">https://status.openai.com/uptime</a>. </li> <li class="ltx_bibitem" id="bib.bib60"> Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, Extended Abstracts, Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, and David A. Shamma (Eds.). ACM, 332:1–332:7. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3491101.3519665" title="">doi:10.1145/3491101.3519665</a> </li> <li class="ltx_bibitem" id="bib.bib61"> Versluis et al. (2023) Laurens Versluis, Mehmet Çetin, Caspar Greeven, Kristian Laursen, Damian Podareanu, Valeriu Codreanu, Alexandru Uta, and Alexandru Iosup. 2023. Less is not more: We need rich datasets to explore. Future Gener. Comput. Syst. 142 (2023), 117–130. <a class="ltx_ref ltx_href" href="https://doi.org/10.1016/J.FUTURE.2022.12.022" title="">doi:10.1016/J.FUTURE.2022.12.022</a> </li> <li class="ltx_bibitem" id="bib.bib62"> Wang et al. (2024b) Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. 2024b. Understanding User Experience in Large Language Model Interactions. CoRR abs/2401.08329 (2024). <a class="ltx_ref ltx_href" href="https://doi.org/10.48550/ARXIV.2401.08329" title="">doi:10.48550/ARXIV.2401.08329</a> arXiv:2401.08329 </li> <li class="ltx_bibitem" id="bib.bib63"> Wang et al. (2024a) Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2024a. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems. arXiv:2401.17644 </li> <li class="ltx_bibitem" id="bib.bib64"> Weng et al. (2022) Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2022, Renton, WA, USA, April 4-6, 2022, Amar Phanishayee and Vyas Sekar (Eds.). USENIX Association, 945–960. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.usenix.org/conference/nsdi22/presentation/weng" title="">https://www.usenix.org/conference/nsdi22/presentation/weng</a> </li> <li class="ltx_bibitem" id="bib.bib65"> WIRED (2024) WIRED. 2024. The obsession with Character AI is becoming more common. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://wired.me/technology/character-ai-obsession/" title="">https://wired.me/technology/character-ai-obsession/</a>. </li> <li class="ltx_bibitem" id="bib.bib66"> Xia et al. (2024) Yuchen Xia, Jiho Kim, Yuhan Chen, Haojie Ye, Souvik Kundu, Cong Hao, and Nishil Talati. 2024. Understanding the Performance and Estimating the Cost of LLM Fine-Tuning. CoRR abs/2408.04693 (2024). <a class="ltx_ref ltx_href" href="https://doi.org/10.48550/ARXIV.2408.04693" title="">doi:10.48550/ARXIV.2408.04693</a> arXiv:2408.04693 </li> <li class="ltx_bibitem" id="bib.bib67"> Yang et al. (2024) Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Ben Hu. 2024. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond. ACM Trans. Knowl. Discov. Data 18, 6 (2024), 160:1–160:32. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3649506" title="">doi:10.1145/3649506</a> </li> <li class="ltx_bibitem" id="bib.bib68"> York (2024) FOX5 New York. 2024. ChatGPT recovers following outage affecting thousands of users. <a class="ltx_ref ltx_url ltx_font_typewriter" href="https://www.fox5ny.com/news/chatgpt-down-users-openai-internet" title="">https://www.fox5ny.com/news/chatgpt-down-users-openai-internet</a>, Accessed: 2024-09-30. </li> <li class="ltx_bibitem" id="bib.bib69"> Zhang et al. (2018) Yang Zhang, Bogdan Vasilescu, Huaimin Wang, and Vladimir Filkov. 2018. One size does not fit all: an empirical study of containerized continuous deployment workflows. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018, Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.). ACM, 295–306. <a class="ltx_ref ltx_href" href="https://doi.org/10.1145/3236024.3236033" title="">doi:10.1145/3236024.3236033</a> </li> </ul> </section> <div class="ltx_pagination ltx_role_newpage"></div> </article> </div> <footer class="ltx_page_footer"> <div class="ltx_page_logo">Generated on Sat Mar 15 16:14:35 2025 by <a class="ltx_LaTeXML_logo" href="http://dlmf.nist.gov/LaTeXML/">LaTeXML<img alt="Mascot Sammy" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg=="/></a> </div></footer> </div> </body> </html>