When One Line Took Thousands of Websites Offline

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">     <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://ogp.me/ns/fb#"> <head> <title>When One Line Took Thousands of Websites Offline - CERN Document Server</title> <link href='https://framework.web.cern.ch/framework/2.0/fonts/PTSansWeb/PTSansWeb.css' rel='stylesheet' type='text/css' /> <link rel="stylesheet" href="https://cds.cern.ch/img/invenio.css?v=20141127" type="text/css" /> <link rel="stylesheet" href="https://cds.cern.ch/img/cern_theme/css/cern_theme.css?v=20141127" type="text/css" /> <link rel="stylesheet"href="/css/font-awesome.min.css"> <meta http-equiv="X-UA-Compatible" content="IE=Edge"/> <link rel="stylesheet" href="https://cds.cern.ch/img/cern_toolbar/css/toolbar.css" type="text/css" />    <link rel="canonical" href="https://cds.cern.ch/record/2875365" /> <link rel="alternate" hreflang="el" href="https://cds.cern.ch/record/2875365?ln=el" /> <link rel="alternate" hreflang="fr" href="https://cds.cern.ch/record/2875365?ln=fr" /> <link rel="alternate" hreflang="bg" href="https://cds.cern.ch/record/2875365?ln=bg" /> <link rel="alternate" hreflang="zh-TW" href="https://cds.cern.ch/record/2875365?ln=zh_TW" /> <link rel="alternate" hreflang="pt" href="https://cds.cern.ch/record/2875365?ln=pt" /> <link rel="alternate" hreflang="no" href="https://cds.cern.ch/record/2875365?ln=no" /> <link rel="alternate" hreflang="hr" href="https://cds.cern.ch/record/2875365?ln=hr" /> <link rel="alternate" hreflang="ca" href="https://cds.cern.ch/record/2875365?ln=ca" /> <link rel="alternate" hreflang="de" href="https://cds.cern.ch/record/2875365?ln=de" /> <link rel="alternate" hreflang="it" href="https://cds.cern.ch/record/2875365?ln=it" /> <link rel="alternate" hreflang="zh-CN" href="https://cds.cern.ch/record/2875365?ln=zh_CN" /> <link rel="alternate" hreflang="sv" href="https://cds.cern.ch/record/2875365?ln=sv" /> <link rel="alternate" hreflang="sk" href="https://cds.cern.ch/record/2875365?ln=sk" /> <link rel="alternate" hreflang="en" href="https://cds.cern.ch/record/2875365?ln=en" /> <link rel="alternate" hreflang="pl" href="https://cds.cern.ch/record/2875365?ln=pl" /> <link rel="alternate" hreflang="ru" href="https://cds.cern.ch/record/2875365?ln=ru" /> <link rel="alternate" hreflang="ka" href="https://cds.cern.ch/record/2875365?ln=ka" /> <link rel="alternate" hreflang="ja" href="https://cds.cern.ch/record/2875365?ln=ja" /> <link rel="alternate" hreflang="es" href="https://cds.cern.ch/record/2875365?ln=es" /> <link rel="alternate" type="application/rss+xml" title="CERN Document Server RSS" href="/rss?ln=en" /> <link rel="search" type="application/opensearchdescription+xml" href="https://cds.cern.ch/opensearchdescription" title="CERN Document Server" /> <link rel="unapi-server" type="application/xml" title="unAPI" href="https://cds.cern.ch/unapi" /> <link rel="apple-touch-icon" href="/apple-touch-icon.png"/> <link rel="apple-touch-icon-precomposed" href="/apple-touch-icon-precomposed.png"/> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta http-equiv="Content-Language" content="en" /> <meta name="description" content="This talk describes an incident where an innocuous change in a configuration management system caused a highly-visible unavailability of thousands of websites, which was followed by an intense recovery procedure. The talk covers the part of the infrastructure that prevented more widespread damage, the lessons learned (in terms of infrastructure design and operational procedures) as well as improvements significant improvements that have been implemented since then. All of this happened on Kubernetes infrastructure, therefore the talk will dive into the topics of Kubernetes operators, automation, manual intervention, configuration management and backups. Henschel, Jack; Borges Aurindo Barros, Francisco" /> <meta name="keywords" content="Information Technology" /> <script type="text/javascript" src="https://cds.cern.ch/js/jquery.min.js"></script>  <link rel="stylesheet" href="https://cds.cern.ch/img/webnews.css" type="text/css" />  <script type="text/javascript" src="https://cds.cern.ch/js/webnews.js?v=20131009"></script> <meta property="fb:app_id" content="137353533001720"/> <script type="text/x-mathjax-config"> MathJax.Hub.Config({ tex2jax: {inlineMath: [['$','$']], processEscapes: true}, showProcessingMessages: false, messageStyle: "none" }); </script> <script src="/MathJax/MathJax.js?config=TeX-AMS_CHTML" type="text/javascript"> </script>  <meta content="When One Line Took Thousands of Websites Offline" name="citation_title" /> <meta content="Henschel, Jack" name="citation_author" /> <meta content="Borges Aurindo Barros, Francisco" name="citation_author" /> <meta content="2023/10/11" name="citation_publication_date" /> <meta name="citation_online_date" content="2023/10/11"> <meta content="IT-TALK-2012-008" name="citation_technical_report_number" /> <meta name="citation_pdf_url" content="https://cds.cern.ch/record/2875365/files/presentation_print.pdf" />  <meta content="SREcon EMEA 2023" property="og:title" /> <meta content="When One Line Took Thousands of Websites Offline" property="og:title" /> <meta content="website" property="og:type" /> <meta content="website" property="og:type" /> <meta content="https://cds.cern.ch/record/2875365" property="og:url" /> <meta name="twitter:image" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta property="og:image" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta property="og:image:secure_url" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta property="og:image" content="https://cds.cern.ch/record/2875365/files/presentation_print.gif?subformat=icon" /> <meta property="og:image:secure_url" content="https://cds.cern.ch/record/2875365/files/presentation_print.gif?subformat=icon" /> <meta property="og:image" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta property="og:image:secure_url" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta content="CERN Document Server" property="og:site_name" /> <meta content="This talk describes an incident where an innocuous change in a configuration management system caused a highly-visible unavailability of thousands of websites, which was followed by an intense recovery procedure. The talk covers the part of the infrastructure that prevented more widespread damage, the lessons learned (in terms of infrastructure design and operational procedures) as well as improvements significant improvements that have been implemented since then. All of this happened on Kubernetes infrastructure, therefore the talk will dive into the topics of Kubernetes operators, automation, manual intervention, configuration management and backups." property="og:description" />  <meta content="summary" name="twitter:card" /> <meta content="summary" name="twitter:card" /> <style></style> </head> <body class="IT32Slides search" lang="en">  <div id="cern-toolbar"> <h1><a href="http://cern.ch" title="CERN">CERN <span>Accelerating science</span></a></h1> <ul> <li class="cern-accountlinks"><a class="cern-account" href="https://cds.cern.ch/youraccount/login?ln=en&referer=https%3A//cds.cern.ch/record/2875365" title="Sign in to your CERN account">Sign in</a></li> <li><a class="cern-directory" href="http://cern.ch/directory" title="Search CERN resources and browse the directory">Directory</a></li> </ul> </div>   <div role="banner" class="clearfix" id="header"> <div class="header-inner inner"> <hgroup class="clearfix"> <h2 id="site-name"> <a rel="home" title="Home" href="/"><span>CERN Document Server</span></a> </h2> <h3 id="site-slogan">Access articles, reports and multimedia content in HEP</h3> </hgroup> <div role="navigation" id="main-navigation" class="cdsmenu"> <h2 class="element-invisible">Main menu</h2><ul class="links inline clearfix"> <li class="menu-386 first active-trail"><a class="active-trail" href="https://cds.cern.ch/?ln=en">Search</a></li> <li class="menu-444 "><a class="" title="" href="https://cds.cern.ch/submit?ln=en">Submit</a></li> <li class="menu-426 "><a class="" href="https://cds.cern.ch/help/?ln=en">Help</a></li> <li class="leaf hassubcdsmenu"> <a hreflang="en" class="header" href="https://cds.cern.ch/youraccount/display?ln=en">Personalize</a> <ul class="subsubcdsmenu"><li><a href="https://cds.cern.ch/youralerts/list?ln=en">Your alerts</a></li><li><a href="https://cds.cern.ch/yourbaskets/display?ln=en">Your baskets</a></li><li><a href="https://cds.cern.ch/yourcomments?ln=en">Your comments</a></li><li><a href="https://cds.cern.ch/youralerts/display?ln=en">Your searches</a></li></ul></li> </ul> </div> </div> </div>  <table class="navtrailbox"> <tr> <td class="navtrailboxbody"> <a href="/?ln=en" class="navtrail">Home</a> > <a href="/collection/Articles%20%26%20Preprints?ln=en" class="navtrail">Articles & Preprints</a> > <a href="/collection/CERN%20Internal%20Notes?ln=en" class="navtrail">CERN Notes</a> > <a href="/collection/IT%20Slides?ln=en" class="navtrail">IT Slides</a> > When One Line Took Thousands of Websites Offline </td> </tr> </table> </div> <div class="pagebody"><div class="pagebodystripemiddle"> <div class="detailedrecordbox"> <div class="detailedrecordtabs"> <div> <ul class="detailedrecordtabs"><li class="on first"><a href="/record/2875365/?ln=en">Information </a></li><li class=""><a href="/record/2875365/comments?ln=en">Discussion (0) </a></li><li class=""><a href="/record/2875365/files?ln=en">Files </a></li></ul> <div id="tabsSpacer" style="clear:both;height:0px"> </div></div> </div> <div class="detailedrecordboxcontent"> <div class="top-left-folded"></div> <div class="top-right-folded"></div> <div class="inside">  <abbr class="unapi-id" title="2875365"></abbr> <style type="text/css">  </style> <table class="formatRecordTableFullWidth"> <tr> <td class="formatRecordHeader" style="background-image: url('https://cds.cern.ch/img/journals.jpg');" colspan="2">  </td> </tr> <tr><td class="formatRecordLabel"> Report number </td><td style="padding-left:5px;">IT-TALK-2012-008</td></tr> <tr><td class="formatRecordLabel"> Title </td><td style="padding-left:5px;"><b>When One Line Took Thousands of Websites Offline</b></td></tr> <tr><td class="formatRecordLabel"> Category </td><td style="padding-left:5px;">Talk</td></tr> <tr><td class="formatRecordLabel"> <span style="white-space:nowrap;">Author(s)</span> </td><td style="padding-left:5px;"> <a href="https://cds.cern.ch/search?f=author&p=Henschel%2C%20Jack&ln=en">Henschel, Jack</a> (CERN) ; <a href="https://cds.cern.ch/search?f=author&p=Borges%20Aurindo%20Barros%2C%20Francisco&ln=en">Borges Aurindo Barros, Francisco</a> (CERN) </td></tr> <tr><td class="formatRecordLabel"> Group </td><td style="padding-left:5px;">PW</td></tr> <tr><td class="formatRecordLabel"> Imprint </td><td style="padding-left:5px;">2023-10-11</td></tr> <tr><td class="formatRecordLabel">Presented at</td><td style="padding-left:5px;"><a href="https://www.usenix.org/conference/srecon23emea">SREcon EMEA 2023</a>, Dublin, Ireland, 10-12 Oct 2023</td></tr> <tr><td class="formatRecordLabel"> Keywords </td><td style="padding-left:5px;"><a href="https://cds.cern.ch/search?f=keyword&p=Information%20Technology&ln=en">Information Technology</a></td></tr> <tr><td class="formatRecordLabel"> Abstract </td><td style="padding-left:5px;">This talk describes an incident where an innocuous change in a configuration management system caused a highly-visible unavailability of thousands of websites, which was followed by an intense recovery procedure. The talk covers the part of the infrastructure that prevented more widespread damage, the lessons learned (in terms of infrastructure design and operational procedures) as well as improvements significant improvements that have been implemented since then. All of this happened on Kubernetes infrastructure, therefore the talk will dive into the topics of Kubernetes operators, automation, manual intervention, configuration management and backups.</td></tr> </table> <br/>Email contact: <a href="mailto:jack.henschel@cern.ch">jack.henschel@cern.ch</a> <br/><div><div style="clear: both;"> </div></div> <script type="text/javascript"> // Initially hide: $(".longCaption").hide(); // Allow to toggle visibility: $(".toggleLongCaption").toggle(function(){ $(this).siblings(".longCaption").show('fast'); var thisElem = $(this); thisElem.text(thisElem.text() === "more" ? "less" : "more"); },function(){ $(this).siblings(".longCaption").hide('fast'); var thisElem = $(this); thisElem.text(thisElem.text() === "more" ? "less" : "more"); }); </script> <br/> <br/><br/><div align="right"><div style="padding-bottom:2px;padding-top:30px;"><span class="moreinfo" style="margin-right:10px;"> <a href="" class="moreinfo">Back to search</a> </span></div></div> <div class="bottom-left-folded"><div class="recordlastmodifiedbox" style="position:relative;margin-left:1px"> Record created 2023-10-11, last modified 2023-10-11</div></div> <div class="bottom-right-folded" style="text-align:right;padding-bottom:2px;"> <span class="moreinfo" style="margin-right:10px;"><a href="/search?ln=en&p=recid%3A2875365&rm=wrd" class="moreinfo">Similar records</a></span></div> </div> </div> </div> <br/> <br /> <div class="detailedrecordminipanel"> <div class="top-left"></div><div class="top-right"></div> <div class="inside"> <div id="detailedrecordminipanelfile" style="width:33%;float:left;text-align:center;margin-top:0"> <div><small class="detailedRecordActions">Fulltext:</small> <br /><a href="/record/2875365/files/presentation_print.pdf"><img style="border:none" src="/img/file-icon-text-34x48.gif" alt="Download fulltext" /><br />PDF</a><br /></div> </div> <div id="detailedrecordminipanelreview" style="width:30%;float:left;text-align:center"> </div> <div id="detailedrecordminipanelactions" style="width:36%;float:right;text-align:right;"> <ul class="detailedrecordactions"> <li><a href="/yourbaskets/add?ln=en&recid=2875365">Add to personal basket</a></li> <li>Export as <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/hx?ln=en">BibTeX</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/hm?ln=en">MARC</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xm?ln=en">MARCXML</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xd?ln=en">DC</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xe?ln=en">EndNote</a>,  <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xn?ln=en">NLM</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xw?ln=en">RefWorks</a> </li> </ul> <div style='padding-left: 13px;'>  <div id="bookmark"></div> <div id="bookmark_sciencewise"></div> <style type="text/css"> #bookmark_sciencewise, #bookmark {float: left;} #bookmark_sciencewise li {padding: 2px; width: 25px;} #bookmark_sciencewise ul, #bookmark ul {list-style-image: none;} </style> <script type="text/javascript" src="/js/jquery.bookmark.min.js"></script> <style type="text/css">@import "/css/jquery.bookmark.css";</style> <script type="text/javascript">// <![CDATA[ $.bookmark.addSite('sciencewise', 'ScienceWise.info', 'https://cds.cern.ch/img/sciencewise.png', 'en', 'bookmark', 'http://sciencewise.info/bookmarks/cds:2875365/add'); $('#bookmark_sciencewise').bookmark({sites: ['sciencewise']}); $('#bookmark').bookmark({ sites: ['facebook', 'twitter', 'linkedin', 'google_plusone'], icons: '/img/bookmarks.png', url: 'https://cds.cern.ch/record/2875365', addEmail: true, title: "When One Line Took Thousands of Websites Offline", description: "This talk describes an incident where an innocuous change in a configuration management system caused a highly-visible unavailability of thousands of websites, which was followed by an intense recovery procedure. The talk covers the part of the infrastructure that prevented more widespread damage, the lessons learned (in terms of infrastructure design and operational procedures) as well as improvements significant improvements that have been implemented since then. All of this happened on Kubernetes infrastructure, therefore the talk will dive into the topics of Kubernetes operators, automation, manual intervention, configuration management and backups." }); // ]]> </script>  </div> </div> <div style="clear:both;margin-bottom: 0;"></div> </div> <div class="bottom-left"></div><div class="bottom-right"></div> </div> </div></div> <footer id="footer" class="pagefooter clearfix">  <div class="pagefooterstripeleft"> CERN Document Server :: <a class="footer" href="https://cds.cern.ch/?ln=en">Search</a> :: <a class="footer" href="https://cds.cern.ch/submit?ln=en">Submit</a> :: <a class="footer" href="https://cds.cern.ch/youraccount/display?ln=en">Personalize</a> :: <a class="footer" href="https://cds.cern.ch/help/?ln=en">Help</a> :: <a class="footer" href="https://cern.service-now.com/service-portal?id=privacy_policy&se=CDS-Service" target="_blank">Privacy Notice</a> <br /> Powered by <a class="footer" href="http://invenio-software.org/">Invenio</a> <br /> Maintained by <a class="footer" href="https://cern.service-now.com/service-portal?id=service_element&name=CDS-Service">CDS Service</a> - Need help? Contact <a href="https://cern.service-now.com/service-portal?id=service_element&name=CDS-Service">CDS Support</a>. <br /> </div> <div class="pagefooterstriperight"> <div class="cern-logo"> <a id="logo" href="http://cern.ch" title="CERN" rel="CERN" ><img src="https://cds.cern.ch/img/cern_theme/img/cern-logo-large.png" alt="CERN" /></a> </div> <div class="cern-languagebox"> This site is also available in the following languages:<br /><a href="/record/2875365?ln=bg" class="langinfo">Български</a>  <a href="/record/2875365?ln=ca" class="langinfo">Català</a>  <a href="/record/2875365?ln=de" class="langinfo">Deutsch</a>  <a href="/record/2875365?ln=el" class="langinfo">Ελληνικά</a>  <span class="langinfo">English</span>  <a href="/record/2875365?ln=es" class="langinfo">Español</a>  <a href="/record/2875365?ln=fr" class="langinfo">Français</a>  <a href="/record/2875365?ln=hr" class="langinfo">Hrvatski</a>  <a href="/record/2875365?ln=it" class="langinfo">Italiano</a>  <a href="/record/2875365?ln=ja" class="langinfo">日本語</a>  <a href="/record/2875365?ln=ka" class="langinfo">ქართული</a>  <a href="/record/2875365?ln=no" class="langinfo">Norsk/Bokmål</a>  <a href="/record/2875365?ln=pl" class="langinfo">Polski</a>  <a href="/record/2875365?ln=pt" class="langinfo">Português</a>  <a href="/record/2875365?ln=ru" class="langinfo">Русский</a>  <a href="/record/2875365?ln=sk" class="langinfo">Slovensky</a>  <a href="/record/2875365?ln=sv" class="langinfo">Svenska</a>  <a href="/record/2875365?ln=zh_CN" class="langinfo">中文(简)</a>  <a href="/record/2875365?ln=zh_TW" class="langinfo">中文(繁)</a> </div> </div>  </footer> <script type="text/javascript"> var SyndeticsBookCovers = (function() { var SMALL_SIZE = "sc.gif", MEDIUM_SIZE = "mc.gif", RAW_URL = "https://secure.syndetics.com/index.aspx?isbn=THEISBN/THESIZE&client=cernlibrary"; replaceCover = function(imgElement, isbns, hdFormat) { var img = new Image(), size = hdFormat ? MEDIUM_SIZE : SMALL_SIZE; var _isbns = isbns.sort(function(a, b) { // sort from shortest to longest ISBN (more modern) return a.length > b.length ? 1 : -1; }); function next() { var isbn = _isbns.pop(); if (isbn) { var url = RAW_URL.replace("THEISBN", isbn).replace("THESIZE", size); img.src = url; } } function done() { imgElement.src = img.src; } img.onload = function() { if (this.width > 1) { done(); } else { next(); } }; next(); }; return { replaceCover: replaceCover }; })(); $(document).ready(function() { // get book covers $("img.book-cover").each(function() { var $this = $(this), strIsbns = $this.data("isbns") || "", isbnsArray = String(strIsbns).split(","), hdFormat = $this.hasClass("hd"); SyndeticsBookCovers.replaceCover(this, isbnsArray, hdFormat); }); // WebNews tooltips $.ajax({ url: "/news/tooltips", success: function(data) { create_tooltips(data); }, dataType: "json", cache: false }); }); </script>  <script src="//cds.cern.ch/js/feedback.js"></script>   <script> var _paq = window._paq = window._paq || []; /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { var u="https://webanalytics.web.cern.ch/"; _paq.push(['setTrackerUrl', u+'matomo.php']); _paq.push(['setSiteId', '756']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); })(); </script>  </body> </html>

CINXE.COM

When One Line Took Thousands of Websites Offline - CERN Document Server