PagerDuty Incident Response Documentation

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8" /> <title>PagerDuty Incident Response Documentation</title>  <meta name="dcterms.license" content="http://www.apache.org/licenses/LICENSE-2.0" /> <meta name="author" content="PagerDuty, Inc." /> <meta name="dcterms.rightsHolder" content="PagerDuty, Inc." /> <meta name="dcterms.rights" content="Copyright © PagerDuty, Inc." />  <meta name="description" content="A collection of information about the PagerDuty incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work." /> <meta name="robots" content="index, follow, archive" /> <meta name="generator" content="mkdocs-1.6.1, mkdocs-theme-pagerduty" />  <link rel="canonical" href="https://response.pagerduty.com/">  <link rel="shortcut icon" href="/assets/images/favicon.png" type="image/png" /> <link rel="icon" href="/assets/images/favicon.png" type="image/png" /> <link rel="apple-touch-icon" href="/assets/images/favicon.png" />  <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0" /> <meta name="theme-color" content="#005a24" />  <meta name="apple-mobile-web-app-title" content="PagerDuty Incident Response Documentation" /> <meta name="apple-mobile-web-app-capable" content="yes" /> <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />  <meta property="og:url" content="https://response.pagerduty.com/" /> <meta property="og:title" content="PagerDuty Incident Response Documentation" /> <meta property="og:site_name" content="PagerDuty Incident Response Documentation" /> <meta property="og:description" content="A collection of information about the PagerDuty incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work." /> <meta property="og:image" content="https://response.pagerduty.com/assets/img/covers/incident_response_docs.png" /> <meta property="og:locale" content="en_US" /> <meta property="og:type" content="website" />  <meta name="twitter:card" content="summary_large_image" /> <meta name="twitter:title" content="PagerDuty Incident Response Documentation" /> <meta name="twitter:description" content="A collection of information about the PagerDuty incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work." /> <meta name="twitter:image" content="https://response.pagerduty.com/assets/img/covers/incident_response_docs.png" />  <link rel="stylesheet" href="assets/stylesheets/fonts.css" /> <link rel="stylesheet" href="assets/stylesheets/reset.css" /> <link rel="stylesheet" href="assets/stylesheets/pagerduty.css" /> <link rel="stylesheet" href="assets/stylesheets/code.css" /> <link rel="stylesheet" href="assets/css/custom.css" /> </head> <body class=""> <header> <div class="content-wrapper"> <div id="logo"> <a href="." title="Go to Incident Response homepage."> <img src="./assets/images/logo.png" title="PagerDuty" /> </a> <span class="subtitle"> Incident Response </span> </div>  <input id="mobile-search-link" type="checkbox" /> <label id="mobile-search-link-selector" for="mobile-search-link" role="button"></label> <div id="search"> <input name="q" id="mkdocs-search-query" type="search" placeholder="Search..." /> <div id="mkdocs-search-results"><p>No results found</p></div> </div>  <script>var base_url = '/';</script> </div> </header> <nav id="breadcrumbs"> <div class="content-wrapper"> <p><a href=".">Incident Response</a></p> <h1>Home</h1> </div> </nav> <section id="hero"> <img alt="Incident Response at PagerDuty" src="assets/img/headers/iStock-1097331490-3992x2242-e4f3f2d.png" /> </section> <main>  <input id="mobile-nav-link" type="checkbox" /> <label id="mobile-nav-link-selector" for="mobile-nav-link" role="button"></label> <nav> <div id="nav-title">Incident Response</div> <ul> <li class="active"> <a title="Home" href=".">Home</a> <ul id="toc"> <li class="anchor"> <a title="Being On-Call" href="#being-on-call">Being On-Call</a> </li> <li class="anchor"> <a title="Before an Incident" href="#before-an-incident">Before an Incident</a> </li> <li class="anchor"> <a title="During an Incident" href="#during-an-incident">During an Incident</a> </li> <li class="anchor"> <a title="After an Incident" href="#after-an-incident">After an Incident</a> </li> <li class="anchor"> <a title="Crisis Response" href="#crisis-response">Crisis Response</a> </li> <li class="anchor"> <a title="Training" href="#training">Training</a> </li> <li class="anchor"> <a title="Additional Resources" href="#additional-resources">Additional Resources</a> </li> </ul> </li> <li> <a title="Getting Started" href="getting_started/">Getting Started</a> </li> <li> <span class="subsection">On-Call</span> <ul> <li> <a title="Being On-Call" href="oncall/being_oncall/">Being On-Call</a> </li> <li> <a title="Who's On-Call?" href="oncall/whos_oncall/">Who's On-Call?</a> </li> <li> <a title="Alerting Principles" href="oncall/alerting_principles/">Alerting Principles</a> </li> </ul> </li> <li> <span class="subsection">Before an Incident</span> <ul> <li> <a title="What is an Incident?" href="before/what_is_an_incident/">What is an Incident?</a> </li> <li> <a title="Severity Levels" href="before/severity_levels/">Severity Levels</a> </li> <li> <a title="Different Roles" href="before/different_roles/">Different Roles</a> </li> <li> <a title="Call Etiquette" href="before/call_etiquette/">Call Etiquette</a> </li> <li> <a title="Complex Incidents" href="before/complex_incidents/">Complex Incidents</a> </li> </ul> </li> <li> <span class="subsection">During an Incident</span> <ul> <li> <a title="During an Incident" href="during/during_an_incident/">During an Incident</a> </li> <li> <a title="External Communication Guidelines" href="during/external_communication_guidelines/">External Communication Guidelines</a> </li> <li> <a title="Security Incident" href="during/security_incident_response/">Security Incident</a> </li> </ul> </li> <li> <span class="subsection">After an Incident</span> <ul> <li> <a title="After an Incident" href="after/after_an_incident/">After an Incident</a> </li> <li> <a title="Postmortem Process" href="after/post_mortem_process/">Postmortem Process</a> </li> <li> <a title="Postmortem Template" href="after/post_mortem_template/">Postmortem Template</a> </li> <li> <a title="Effective Postmortems" href="after/effective_post_mortems/">Effective Postmortems</a> </li> </ul> </li> <li> <span class="subsection">Crisis Response</span> <ul> <li> <a title="Crisis Response Introduction" href="crisis/crisis_intro/">Crisis Response Introduction</a> </li> <li> <a title="Terminology" href="crisis/terms/">Terminology</a> </li> <li> <a title="Crisis Leadership" href="crisis/leadership/">Crisis Leadership</a> </li> <li> <a title="Crisis Response Operations" href="crisis/operations/">Crisis Response Operations</a> </li> <li> <a title="Pre-crisis Phase" href="crisis/prework/">Pre-crisis Phase</a> </li> <li> <a title="PagerDuty for CRMOps" href="crisis/pagerduty/">PagerDuty for CRMOps</a> </li> </ul> </li> <li> <span class="subsection">Training</span> <ul> <li> <a title="Training Overview" href="training/overview/">Training Overview</a> </li> <li> <a title="Incident Commander" href="training/incident_commander/">Incident Commander</a> </li> <li> <a title="Deputy" href="training/deputy/">Deputy</a> </li> <li> <a title="Scribe" href="training/scribe/">Scribe</a> </li> <li> <a title="Subject Matter Expert" href="training/subject_matter_expert/">Subject Matter Expert</a> </li> <li> <a title="Customer Liaison" href="training/customer_liaison/">Customer Liaison</a> </li> <li> <a title="Internal Liaison" href="training/internal_liaison/">Internal Liaison</a> </li> <li> <a title="Glossary" href="training/glossary/">Glossary</a> </li> </ul> </li> <li> <span class="subsection">Training Courses</span> <ul> <li> <a title="Incident Response" href="training/courses/incident_response/">Incident Response</a> </li> </ul> </li> <li> <span class="subsection">Additional Resources</span> <ul> <li> <a title="Reading" href="resources/reading/">Reading</a> </li> <li> <a title="ChatOps" href="resources/chatops/">ChatOps</a> </li> <li> <a title="Anti-Patterns" href="resources/anti_patterns/">Anti-Patterns</a> </li> </ul> </li> <li> <a title="About" href="about/">About</a> </li> </ul> </nav> <article> <h1>Home</h1> <section id="downloads"> </section> <p>This documentation covers parts of the PagerDuty Incident Response process. It is a cut-down version of our internal documentation used at PagerDuty for any major incidents and to prepare new employees for on-call responsibilities. It provides information not only on preparing for an incident, but also what to do during and after the incident. It is intended to be used by on-call practitioners and those involved in an operational incident response process (or those wishing to enact a formal incident response process). See the <a href="about/">about page</a> for more information on what this documentation is and why it exists.</p> <div class="admonition tip"> <p class="admonition-title">Don't know where to start?</p> <p>If you're new to incident response and don't yet have a formal process in your organization, we recommend looking at our <a href="getting_started/">Getting Started</a> page for a quick list of things you can do to begin. Additionally, you can check out our <a href="training/courses/incident_response/">Training Course</a> page for a more detailed overview of our process.</p> </div> <h2 id="being-on-call">Being On-Call<a class="headerlink" href="#being-on-call" title="Permanent link">#</a></h2> <p>If you've never been on-call before, you might be wondering what it's all about. These pages describe what the expectations of being on-call are, along with some resources to help you.</p> <ul> <li><a href="oncall/being_oncall/">Being On-Call</a> - <em>A guide to being on-call, both what your responsibilities are, and what they are not.</em></li> <li><a href="oncall/alerting_principles/">Alerting Principles</a> - <em>The principles we use to determine what things page an engineer, and what time of day they page.</em></li> </ul> <h2 id="before-an-incident">Before an Incident<a class="headerlink" href="#before-an-incident" title="Permanent link">#</a></h2> <p>Reading material for things you probably want to know before an incident occurs. You likely don't want to be reading these during an actual incident.</p> <ul> <li><a href="before/what_is_an_incident/">What is an Incident?</a> - <em>Before we can talk about incident response, we need to define what an incident actually is.</em></li> <li><a href="before/severity_levels/">Severity Levels</a> - <em>Information on our severity level classification. What constitutes a SEV-3 vs SEV-1? What response do they get?</em></li> <li><a href="before/different_roles/">Different Roles for Incidents</a> - <em>Information on the roles during an incident(i.e. Incident Commander (IC), Scribe, etc.)</em></li> <li><a href="before/call_etiquette/">Incident Call Etiquette</a> - <em>Our etiquette guidelines for incident calls (before you find yourself in one).</em></li> <li><a href="before/complex_incidents/">Complex Incidents</a> - <em>Our guide for handling larger, more complex incidents.</em></li> </ul> <h2 id="during-an-incident">During an Incident<a class="headerlink" href="#during-an-incident" title="Permanent link">#</a></h2> <p>Information and processes during a major incident.</p> <ul> <li><a href="during/during_an_incident/">During an Incident</a> - <em>Information on what to do during an incident and how to constructively contribute.</em></li> <li><a href="during/security_incident_response/">Security Incident Response</a> - <em>Security incidents are handled differently than normal operational incidents.</em></li> </ul> <h2 id="after-an-incident">After an Incident<a class="headerlink" href="#after-an-incident" title="Permanent link">#</a></h2> <p>Our followup-processes, how we make sure we don't repeat mistakes, and are always improving.</p> <ul> <li><a href="after/after_an_incident/">After an Incident</a> - <em>Information on what to do after an incident is resolved.</em></li> <li><a href="after/post_mortem_process/">Postmortem Process</a> - <em>Information on our postmortem process, what's involved, and how to write or run a postmortem.</em></li> <li><a href="after/post_mortem_template/">Postmortem Template</a> - <em>The template we use for writing our postmortems for major incidents.</em></li> <li><a href="after/effective_post_mortems/">Effective Postmortems</a> - <em>A guide for writing effective postmortems.</em></li> </ul> <h2 id="crisis-response">Crisis Response<a class="headerlink" href="#crisis-response" title="Permanent link">#</a></h2> <p>Incident response is about more than dealing with technical incidents. A crisis can happen at any time. Are you ready for it? The way you handle your worst day will leave lasting impressions about your brand and its perceived value in the eyes of your current and potential customers.</p> <ul> <li><a href="crisis/crisis_intro/">Introduction</a> - <em>An introduction to crisis response and who this document is intended for.</em></li> <li><a href="crisis/terms/">Terminology</a> - <em>A list of key terms and concepts used in this guide.</em></li> <li><a href="crisis/leadership/">Crisis Leadership</a> - <em>Incorporating basic principles and your values in your response.</em></li> <li><a href="crisis/operations/">Crisis Response Operations</a> - <em>Activating your crisis response plans.</em></li> <li><a href="crisis/prework/">Pre-crisis Phase</a> - <em>Capitalizing on preparedness activities to keep your teams ready and engaged.</em></li> <li><a href="crisis/pagerduty/">PagerDuty for CRMOps</a> - <em>How PagerDuty leverages PagerDuty for crisis response management operations.</em></li> </ul> <h2 id="training">Training<a class="headerlink" href="#training" title="Permanent link">#</a></h2> <p>So you want to learn about incident response? You've come to the right place.</p> <ul> <li><a href="training/overview/">Training Overview</a> - <em>An overview of our training guides and additional training material from third parties.</em></li> <li><a href="training/glossary/">Glossary of Incident Response Terms</a> - <em>A collection of terms that you may hear being used along with their definitions.</em></li> </ul> <h3 id="training-guides">Training Guides<a class="headerlink" href="#training-guides" title="Permanent link">#</a></h3> <ul> <li><a href="training/incident_commander/">Incident Commander Training</a> - <em>A guide to becoming an Incident Commander.</em></li> <li><a href="training/deputy/">Deputy Training</a> - <em>How to be a Deputy and back up the Incident Commander.</em></li> <li><a href="training/scribe/">Scribe Training</a> - <em>A guide to Scribing.</em></li> <li><a href="training/subject_matter_expert/">Subject Matter Expert Training</a> - <em>A guide on responsibilities and behavior for all participants in a major incident.</em></li> <li><a href="training/customer_liaison/">Customer Liaison Training</a> - <em>A guide on how to act as our public representative during an incident.</em></li> <li><a href="training/internal_liaison/">Internal Liaison Training</a> - <em>A guide on how to liaise with internal teams during an incident.</em></li> </ul> <h3 id="training-courses">Training Courses<a class="headerlink" href="#training-courses" title="Permanent link">#</a></h3> <ul> <li><a href="training/courses/incident_response/">Incident Response Training Course</a> - <em>An introductory course on incident response and the role of the Incident Commander (IC).</em></li> </ul> <h2 id="additional-resources">Additional Resources<a class="headerlink" href="#additional-resources" title="Permanent link">#</a></h2> <p>Useful material and resources from external parties that are relevant to incident response.</p> <ul> <li><a href="resources/reading/">Reading</a> - <em>Recommended reading material relevant to incident response.</em></li> <li><a href="resources/chatops/">ChatOps</a> - <em>Description of the chat bot commands we've referenced in this documentation.</em></li> <li><a href="resources/anti_patterns/">Anti-Patterns</a> - <em>List of things we've tried and then rejected - learn from our mistakes.</em></li> </ul> </article> </main> <nav id="pagination"> <div class="content-wrapper"> <div class="previous"> </div> <div class="next"> <a href="getting_started/" title="Getting Started"> <span class="label">Next</span> <div class="page"> <div class="title"> Getting Started </div> <div class="button" role="button" aria-label="Next"> <i class="icon icon-next"></i> </div> </div> </a> </div> </div> </nav> <footer> <div class="content-wrapper"> <ul id="ops-guides"> <h2>Ops Guides</h2> <p>Explore our other guides!</p> <li id="og-incident-response" class="ops-guide"> <a href="https://response.pagerduty.com">Incident Response</a> <p>A detailed outline of response processes for technical incidents practices by PagerDuty and our leading customers.</p> </li> <li id="og-security-training" class="ops-guide"> <a href="https://sudo.pagerduty.com">Security Training</a> <p>Open source security training used at PagerDuty - adaptable for your own technical and non-technical teams.</p> </li> <li id="og-postmortems" class="ops-guide"> <a href="https://postmortems.pagerduty.com/">Postmortems</a> <p>Comprehensive guide on how to conduct effective postmortems. Learn how to build a culture of blamelessness.</p> </li> <li id="og-operational-reviews" class="ops-guide"> <a href="https://reviews.pagerduty.com/">Operational Reviews</a> <p>Gauge incident impact using data-driven regularly scheduled reviews to better manage the hidden cost of real-time ops.</p> </li> <li id="og-business-response" class="ops-guide"> <a href="https://business-response.pagerduty.com/">Business Response</a> <p>Learn how to align the business needs with technical needs when severe technical incidents occur.</p> </li> <li id="og-stakeholder-comms" class="ops-guide"> <a href="https://stakeholders.pagerduty.com/">Stakeholder Comms</a> <p>Build an effective communiction strategy for your internal stakeholders during major incidents.</p> </li> <li id="og-retrospectives" class="ops-guide"> <a href="https://retrospectives.pagerduty.com/">Retrospectives</a> <p>Iteratively learn from working processes and behaviors while cultivating a culture of continuous improvement.</p> </li> <li id="og-ownership" class="ops-guide"> <a href="https://ownership.pagerduty.com">Ownership</a> <p>Ensure the reliability of systems & services through a deeper understanding of how code functions in production.</p> </li> <li id="og-devsecops" class="ops-guide"> <a href="https://devsecops.pagerduty.com/">DevSecOps</a> <p>New to DevSecOps, or wondering what it is and how to implement it? This guide will help you get started.</p> </li> <li id="og-goingoncall" class="ops-guide"> <a href="https://goingoncall.pagerduty.com/">Going On-Call</a> <p>The complete resource to going on call for teams and managers.</p> </li> <li id="og-autoremediation" class="ops-guide"> <a href="https://autoremediation.pagerduty.com/">Auto Remediation</a> <p>This guide will help you to leverage automation in your Incident Response process.</p> </li> <li id="og-customerservice" class="ops-guide"> <a href="https://customerserviceops.pagerduty.com/">Customer Service</a> <p>Excellent Customer Service means excellent customer experience, even during incidents.</p> </li> </ul> <p id="outro"> Made with <span class="material-icons love" aria-label="Love">favorite</span> at PagerDuty. </p> <p id="github-repo-link"> <a target="_blank" title="View the source for this documentation on GitHub." href="https://github.com/pagerduty/incident-response-docs">View Source on GitHub</a> </p>  <ul id="social-links"> <li class="twitter"> <a title="View the @PagerDuty Twitter feed." href="https://twitter.com/PagerDuty">Twitter</a> </li> <li class="github"> <a title="View PagerDuty on GitHub" href="https://github.com/PagerDuty">GitHub</a> </li> <li class="pagerduty"> <a title="View the PagerDuty website." href="https://www.pagerduty.com/">PagerDuty</a> </li> </ul> </div> </footer> <script src="search/main.js"></script> <script> /* Google Analytics */ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-8759953-1', 'auto'); ga('send', 'pageview'); /* Track outbound links */ var buttons = document.querySelectorAll('a'); Array.prototype.map.call(buttons, function(item) { if (item.host != document.location.host) { item.addEventListener('click', function() { var action = item.getAttribute('data-action') || 'follow'; ga('send', 'event', 'outbound', action, item.href); }); } }); /* Register handler to log search on blur */ var query = document.querySelector('#mkdocs-search-query'); query.addEventListener('blur', function() { if (this.value) { var path = document.location.pathname; ga('send', 'pageview', path + '?q=' + this.value); } }); </script> </body> </html>

CINXE.COM

PagerDuty Incident Response Documentation