When One Line Took Thousands of Websites Offline

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">     <html xmlns="http://www.w3.org/1999/xhtml" lang="ja" xml:lang="ja" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://ogp.me/ns/fb#"> <head> <title>When One Line Took Thousands of Websites Offline - CERN Document Server</title> <link href='https://framework.web.cern.ch/framework/2.0/fonts/PTSansWeb/PTSansWeb.css' rel='stylesheet' type='text/css' /> <link rel="stylesheet" href="https://cds.cern.ch/img/invenio.css?v=20141127" type="text/css" /> <link rel="stylesheet" href="https://cds.cern.ch/img/cern_theme/css/cern_theme.css?v=20141127" type="text/css" /> <link rel="stylesheet"href="/css/font-awesome.min.css"> <meta http-equiv="X-UA-Compatible" content="IE=Edge"/> <link rel="stylesheet" href="https://cds.cern.ch/img/cern_toolbar/css/toolbar.css" type="text/css" />    <link rel="canonical" href="https://cds.cern.ch/record/2875365" /> <link rel="alternate" hreflang="el" href="https://cds.cern.ch/record/2875365?ln=el" /> <link rel="alternate" hreflang="fr" href="https://cds.cern.ch/record/2875365?ln=fr" /> <link rel="alternate" hreflang="bg" href="https://cds.cern.ch/record/2875365?ln=bg" /> <link rel="alternate" hreflang="zh-TW" href="https://cds.cern.ch/record/2875365?ln=zh_TW" /> <link rel="alternate" hreflang="pt" href="https://cds.cern.ch/record/2875365?ln=pt" /> <link rel="alternate" hreflang="no" href="https://cds.cern.ch/record/2875365?ln=no" /> <link rel="alternate" hreflang="hr" href="https://cds.cern.ch/record/2875365?ln=hr" /> <link rel="alternate" hreflang="ca" href="https://cds.cern.ch/record/2875365?ln=ca" /> <link rel="alternate" hreflang="de" href="https://cds.cern.ch/record/2875365?ln=de" /> <link rel="alternate" hreflang="it" href="https://cds.cern.ch/record/2875365?ln=it" /> <link rel="alternate" hreflang="zh-CN" href="https://cds.cern.ch/record/2875365?ln=zh_CN" /> <link rel="alternate" hreflang="sv" href="https://cds.cern.ch/record/2875365?ln=sv" /> <link rel="alternate" hreflang="sk" href="https://cds.cern.ch/record/2875365?ln=sk" /> <link rel="alternate" hreflang="en" href="https://cds.cern.ch/record/2875365?ln=en" /> <link rel="alternate" hreflang="pl" href="https://cds.cern.ch/record/2875365?ln=pl" /> <link rel="alternate" hreflang="ru" href="https://cds.cern.ch/record/2875365?ln=ru" /> <link rel="alternate" hreflang="ka" href="https://cds.cern.ch/record/2875365?ln=ka" /> <link rel="alternate" hreflang="ja" href="https://cds.cern.ch/record/2875365?ln=ja" /> <link rel="alternate" hreflang="es" href="https://cds.cern.ch/record/2875365?ln=es" /> <link rel="alternate" type="application/rss+xml" title="CERN Document Server RSS" href="/rss?ln=ja" /> <link rel="search" type="application/opensearchdescription+xml" href="https://cds.cern.ch/opensearchdescription" title="CERN Document Server" /> <link rel="unapi-server" type="application/xml" title="unAPI" href="https://cds.cern.ch/unapi" /> <link rel="apple-touch-icon" href="/apple-touch-icon.png"/> <link rel="apple-touch-icon-precomposed" href="/apple-touch-icon-precomposed.png"/> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta http-equiv="Content-Language" content="ja" /> <meta name="description" content="This talk describes an incident where an innocuous change in a configuration management system caused a highly-visible unavailability of thousands of websites, which was followed by an intense recovery procedure. The talk covers the part of the infrastructure that prevented more widespread damage, the lessons learned (in terms of infrastructure design and operational procedures) as well as improvements significant improvements that have been implemented since then. All of this happened on Kubernetes infrastructure, therefore the talk will dive into the topics of Kubernetes operators, automation, manual intervention, configuration management and backups. Henschel, Jack; Borges Aurindo Barros, Francisco" /> <meta name="keywords" content="Information Technology" /> <script type="text/javascript" src="https://cds.cern.ch/js/jquery.min.js"></script>  <link rel="stylesheet" href="https://cds.cern.ch/img/webnews.css" type="text/css" />  <script type="text/javascript" src="https://cds.cern.ch/js/webnews.js?v=20131009"></script> <meta property="fb:app_id" content="137353533001720"/> <script type="text/x-mathjax-config"> MathJax.Hub.Config({ tex2jax: {inlineMath: [['$','$']], processEscapes: true}, showProcessingMessages: false, messageStyle: "none" }); </script> <script src="/MathJax/MathJax.js?config=TeX-AMS_CHTML" type="text/javascript"> </script>  <meta content="When One Line Took Thousands of Websites Offline" name="citation_title" /> <meta content="Henschel, Jack" name="citation_author" /> <meta content="Borges Aurindo Barros, Francisco" name="citation_author" /> <meta content="2023/10/11" name="citation_publication_date" /> <meta name="citation_online_date" content="2023/10/11"> <meta content="IT-TALK-2012-008" name="citation_technical_report_number" /> <meta name="citation_pdf_url" content="https://cds.cern.ch/record/2875365/files/presentation_print.pdf" />  <meta content="SREcon EMEA 2023" property="og:title" /> <meta content="When One Line Took Thousands of Websites Offline" property="og:title" /> <meta content="website" property="og:type" /> <meta content="website" property="og:type" /> <meta content="https://cds.cern.ch/record/2875365" property="og:url" /> <meta name="twitter:image" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta property="og:image" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta property="og:image:secure_url" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta property="og:image" content="https://cds.cern.ch/record/2875365/files/presentation_print.gif?subformat=icon" /> <meta property="og:image:secure_url" content="https://cds.cern.ch/record/2875365/files/presentation_print.gif?subformat=icon" /> <meta property="og:image" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta property="og:image:secure_url" content="https://cds.cern.ch/record/2875365/files/presentation_print.jpg?subformat=icon-180" /> <meta content="CERN Document Server" property="og:site_name" /> <meta content="This talk describes an incident where an innocuous change in a configuration management system caused a highly-visible unavailability of thousands of websites, which was followed by an intense recovery procedure. The talk covers the part of the infrastructure that prevented more widespread damage, the lessons learned (in terms of infrastructure design and operational procedures) as well as improvements significant improvements that have been implemented since then. All of this happened on Kubernetes infrastructure, therefore the talk will dive into the topics of Kubernetes operators, automation, manual intervention, configuration management and backups." property="og:description" />  <meta content="summary" name="twitter:card" /> <meta content="summary" name="twitter:card" /> <style></style> </head> <body class="IT32Slides search" lang="ja">  <div id="cern-toolbar"> <h1><a href="http://cern.ch" title="CERN">CERN <span>Accelerating science</span></a></h1> <ul> <li class="cern-accountlinks"><a class="cern-account" href="https://cds.cern.ch/youraccount/login?ln=ja&referer=https%3A//cds.cern.ch/record/2875365%3Fln%3Dja" title="Sign in to your CERN account">Sign in</a></li> <li><a class="cern-directory" href="http://cern.ch/directory" title="Search CERN resources and browse the directory">Directory</a></li> </ul> </div>   <div role="banner" class="clearfix" id="header"> <div class="header-inner inner"> <hgroup class="clearfix"> <h2 id="site-name"> <a rel="home" title="Home" href="/"><span>CERN Document Server</span></a> </h2> <h3 id="site-slogan">Access articles, reports and multimedia content in HEP</h3> </hgroup> <div role="navigation" id="main-navigation" class="cdsmenu"> <h2 class="element-invisible">Main menu</h2><ul class="links inline clearfix"> <li class="menu-386 first active-trail"><a class="active-trail" href="https://cds.cern.ch/?ln=ja">検索</a></li> <li class="menu-444 "><a class="" title="" href="https://cds.cern.ch/submit?ln=ja">アップロード</a></li> <li class="menu-426 "><a class="" href="https://cds.cern.ch/help/?ln=ja">ヘルプ</a></li> <li class="leaf hassubcdsmenu"> <a hreflang="en" class="header" href="https://cds.cern.ch/youraccount/display?ln=ja">あなたのページ</a> <ul class="subsubcdsmenu"><li><a href="https://cds.cern.ch/youralerts/list?ln=ja">Your alerts</a></li><li><a href="https://cds.cern.ch/yourbaskets/display?ln=ja">Your baskets</a></li><li><a href="https://cds.cern.ch/yourcomments?ln=ja">Your comments</a></li><li><a href="https://cds.cern.ch/youralerts/display?ln=ja">Your searches</a></li></ul></li> </ul> </div> </div> </div>  <table class="navtrailbox"> <tr> <td class="navtrailboxbody"> <a href="/?ln=ja" class="navtrail">ホーム</a> > <a href="/collection/Articles%20%26%20Preprints?ln=ja" class="navtrail">Articles & Preprints</a> > <a href="/collection/CERN%20Internal%20Notes?ln=ja" class="navtrail">CERN Notes</a> > <a href="/collection/IT%20Slides?ln=ja" class="navtrail">IT Slides</a> > When One Line Took Thousands of Websites Offline </td> </tr> </table> </div> <div class="pagebody"><div class="pagebodystripemiddle"> <div class="detailedrecordbox"> <div class="detailedrecordtabs"> <div> <ul class="detailedrecordtabs"><li class="on first"><a href="/record/2875365/?ln=ja">Information </a></li><li class=""><a href="/record/2875365/comments?ln=ja">Discussion (0) </a></li><li class=""><a href="/record/2875365/files?ln=ja">Files </a></li></ul> <div id="tabsSpacer" style="clear:both;height:0px"> </div></div> </div> <div class="detailedrecordboxcontent"> <div class="top-left-folded"></div> <div class="top-right-folded"></div> <div class="inside">  <abbr class="unapi-id" title="2875365"></abbr> <style type="text/css">  </style> <table class="formatRecordTableFullWidth"> <tr> <td class="formatRecordHeader" style="background-image: url('https://cds.cern.ch/img/journals.jpg');" colspan="2">  </td> </tr> <tr><td class="formatRecordLabel"> Report number </td><td style="padding-left:5px;">IT-TALK-2012-008</td></tr> <tr><td class="formatRecordLabel"> Title </td><td style="padding-left:5px;"><b>When One Line Took Thousands of Websites Offline</b></td></tr> <tr><td class="formatRecordLabel"> Category </td><td style="padding-left:5px;">Talk</td></tr> <tr><td class="formatRecordLabel"> <span style="white-space:nowrap;">Author(s)</span> </td><td style="padding-left:5px;"> <a href="https://cds.cern.ch/search?f=author&p=Henschel%2C%20Jack&ln=ja">Henschel, Jack</a> (CERN) ; <a href="https://cds.cern.ch/search?f=author&p=Borges%20Aurindo%20Barros%2C%20Francisco&ln=ja">Borges Aurindo Barros, Francisco</a> (CERN) </td></tr> <tr><td class="formatRecordLabel"> Group </td><td style="padding-left:5px;">PW</td></tr> <tr><td class="formatRecordLabel"> Imprint </td><td style="padding-left:5px;">2023-10-11</td></tr> <tr><td class="formatRecordLabel">Presented at</td><td style="padding-left:5px;"><a href="https://www.usenix.org/conference/srecon23emea">SREcon EMEA 2023</a>, Dublin, Ireland, 10-12 Oct 2023</td></tr> <tr><td class="formatRecordLabel"> Keywords </td><td style="padding-left:5px;"><a href="https://cds.cern.ch/search?f=keyword&p=Information%20Technology&ln=ja">Information Technology</a></td></tr> <tr><td class="formatRecordLabel"> Abstract </td><td style="padding-left:5px;">This talk describes an incident where an innocuous change in a configuration management system caused a highly-visible unavailability of thousands of websites, which was followed by an intense recovery procedure. The talk covers the part of the infrastructure that prevented more widespread damage, the lessons learned (in terms of infrastructure design and operational procedures) as well as improvements significant improvements that have been implemented since then. All of this happened on Kubernetes infrastructure, therefore the talk will dive into the topics of Kubernetes operators, automation, manual intervention, configuration management and backups.</td></tr> </table> <br/>Email contact: <a href="mailto:jack.henschel@cern.ch">jack.henschel@cern.ch</a> <br/><div><div style="clear: both;"> </div></div> <script type="text/javascript"> // Initially hide: $(".longCaption").hide(); // Allow to toggle visibility: $(".toggleLongCaption").toggle(function(){ $(this).siblings(".longCaption").show('fast'); var thisElem = $(this); thisElem.text(thisElem.text() === "さらに" ? "less" : "さらに"); },function(){ $(this).siblings(".longCaption").hide('fast'); var thisElem = $(this); thisElem.text(thisElem.text() === "さらに" ? "less" : "さらに"); }); </script> <br/> <br/><br/><div align="right"><div style="padding-bottom:2px;padding-top:30px;"><span class="moreinfo" style="margin-right:10px;"> <a href="" class="moreinfo">Back to search</a> </span></div></div> <div class="bottom-left-folded"><div class="recordlastmodifiedbox" style="position:relative;margin-left:1px"> レコード　生成： 2023-10-11, 最終変更： 2023-10-11</div></div> <div class="bottom-right-folded" style="text-align:right;padding-bottom:2px;"> <span class="moreinfo" style="margin-right:10px;"><a href="/search?ln=ja&p=recid%3A2875365&rm=wrd" class="moreinfo">ほとんど同じレコード</a></span></div> </div> </div> </div> <br/> <br /> <div class="detailedrecordminipanel"> <div class="top-left"></div><div class="top-right"></div> <div class="inside"> <div id="detailedrecordminipanelfile" style="width:33%;float:left;text-align:center;margin-top:0"> <div><small class="detailedRecordActions">フルテキスト:</small> <br /><a href="/record/2875365/files/presentation_print.pdf"><img style="border:none" src="/img/file-icon-text-34x48.gif" alt="Download fulltext" /><br />PDF</a><br /></div> </div> <div id="detailedrecordminipanelreview" style="width:30%;float:left;text-align:center"> </div> <div id="detailedrecordminipanelactions" style="width:36%;float:right;text-align:right;"> <ul class="detailedrecordactions"> <li><a href="/yourbaskets/add?ln=ja&recid=2875365">Add to personal basket</a></li> <li>Export as <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/hx?ln=ja">BibTeX</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/hm?ln=ja">MARC</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xm?ln=ja">MARCXML</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xd?ln=ja">DC</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xe?ln=ja">EndNote</a>,  <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xn?ln=ja">NLM</a>, <a style="text-decoration:underline;font-weight:normal" href="/record/2875365/export/xw?ln=ja">RefWorks</a> </li> </ul> <div style='padding-left: 13px;'>  <div id="bookmark"></div> <div id="bookmark_sciencewise"></div> <style type="text/css"> #bookmark_sciencewise, #bookmark {float: left;} #bookmark_sciencewise li {padding: 2px; width: 25px;} #bookmark_sciencewise ul, #bookmark ul {list-style-image: none;} </style> <script type="text/javascript" src="/js/jquery.bookmark.min.js"></script> <style type="text/css">@import "/css/jquery.bookmark.css";</style> <script type="text/javascript">// <![CDATA[ $.bookmark.addSite('sciencewise', 'ScienceWise.info', 'https://cds.cern.ch/img/sciencewise.png', 'en', 'bookmark', 'http://sciencewise.info/bookmarks/cds:2875365/add'); $('#bookmark_sciencewise').bookmark({sites: ['sciencewise']}); $('#bookmark').bookmark({ sites: ['facebook', 'twitter', 'linkedin', 'google_plusone'], icons: '/img/bookmarks.png', url: 'https://cds.cern.ch/record/2875365', addEmail: true, title: "When One Line Took Thousands of Websites Offline", description: "This talk describes an incident where an innocuous change in a configuration management system caused a highly-visible unavailability of thousands of websites, which was followed by an intense recovery procedure. The talk covers the part of the infrastructure that prevented more widespread damage, the lessons learned (in terms of infrastructure design and operational procedures) as well as improvements significant improvements that have been implemented since then. All of this happened on Kubernetes infrastructure, therefore the talk will dive into the topics of Kubernetes operators, automation, manual intervention, configuration management and backups." }); // ]]> </script>  </div> </div> <div style="clear:both;margin-bottom: 0;"></div> </div> <div class="bottom-left"></div><div class="bottom-right"></div> </div> </div></div> <footer id="footer" class="pagefooter clearfix">  <div class="pagefooterstripeleft"> CERN Document Server :: <a class="footer" href="https://cds.cern.ch/?ln=ja">検索</a> :: <a class="footer" href="https://cds.cern.ch/submit?ln=ja">アップロード</a> :: <a class="footer" href="https://cds.cern.ch/youraccount/display?ln=ja">あなたのページ</a> :: <a class="footer" href="https://cds.cern.ch/help/?ln=ja">ヘルプ</a> :: <a class="footer" href="https://cern.service-now.com/service-portal?id=privacy_policy&se=CDS-Service" target="_blank">Privacy Notice</a> <br /> Powered by <a class="footer" href="http://invenio-software.org/">Invenio</a> <br /> 管理者 <a class="footer" href="https://cern.service-now.com/service-portal?id=service_element&name=CDS-Service">CDS Service</a> - Need help? Contact <a href="https://cern.service-now.com/service-portal?id=service_element&name=CDS-Service">CDS Support</a>. <br /> </div> <div class="pagefooterstriperight"> <div class="cern-logo"> <a id="logo" href="http://cern.ch" title="CERN" rel="CERN" ><img src="https://cds.cern.ch/img/cern_theme/img/cern-logo-large.png" alt="CERN" /></a> </div> <div class="cern-languagebox"> この場所は次の言語でまた利用できる:<br /><a href="/record/2875365?ln=bg" class="langinfo">Български</a>  <a href="/record/2875365?ln=ca" class="langinfo">Català</a>  <a href="/record/2875365?ln=de" class="langinfo">Deutsch</a>  <a href="/record/2875365?ln=el" class="langinfo">Ελληνικά</a>  <a href="/record/2875365?ln=en" class="langinfo">English</a>  <a href="/record/2875365?ln=es" class="langinfo">Español</a>  <a href="/record/2875365?ln=fr" class="langinfo">Français</a>  <a href="/record/2875365?ln=hr" class="langinfo">Hrvatski</a>  <a href="/record/2875365?ln=it" class="langinfo">Italiano</a>  <span class="langinfo">日本語</span>  <a href="/record/2875365?ln=ka" class="langinfo">ქართული</a>  <a href="/record/2875365?ln=no" class="langinfo">Norsk/Bokmål</a>  <a href="/record/2875365?ln=pl" class="langinfo">Polski</a>  <a href="/record/2875365?ln=pt" class="langinfo">Português</a>  <a href="/record/2875365?ln=ru" class="langinfo">Русский</a>  <a href="/record/2875365?ln=sk" class="langinfo">Slovensky</a>  <a href="/record/2875365?ln=sv" class="langinfo">Svenska</a>  <a href="/record/2875365?ln=zh_CN" class="langinfo">中文(简)</a>  <a href="/record/2875365?ln=zh_TW" class="langinfo">中文(繁)</a> </div> </div>  </footer> <script type="text/javascript"> var SyndeticsBookCovers = (function() { var SMALL_SIZE = "sc.gif", MEDIUM_SIZE = "mc.gif", RAW_URL = "https://secure.syndetics.com/index.aspx?isbn=THEISBN/THESIZE&client=cernlibrary"; replaceCover = function(imgElement, isbns, hdFormat) { var img = new Image(), size = hdFormat ? MEDIUM_SIZE : SMALL_SIZE; var _isbns = isbns.sort(function(a, b) { // sort from shortest to longest ISBN (more modern) return a.length > b.length ? 1 : -1; }); function next() { var isbn = _isbns.pop(); if (isbn) { var url = RAW_URL.replace("THEISBN", isbn).replace("THESIZE", size); img.src = url; } } function done() { imgElement.src = img.src; } img.onload = function() { if (this.width > 1) { done(); } else { next(); } }; next(); }; return { replaceCover: replaceCover }; })(); $(document).ready(function() { // get book covers $("img.book-cover").each(function() { var $this = $(this), strIsbns = $this.data("isbns") || "", isbnsArray = String(strIsbns).split(","), hdFormat = $this.hasClass("hd"); SyndeticsBookCovers.replaceCover(this, isbnsArray, hdFormat); }); // WebNews tooltips $.ajax({ url: "/news/tooltips", success: function(data) { create_tooltips(data); }, dataType: "json", cache: false }); }); </script>  <script src="//cds.cern.ch/js/feedback.js"></script>   <script> var _paq = window._paq = window._paq || []; /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { var u="https://webanalytics.web.cern.ch/"; _paq.push(['setTrackerUrl', u+'matomo.php']); _paq.push(['setSiteId', '756']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); })(); </script>  </body> </html>

CINXE.COM

When One Line Took Thousands of Websites Offline - CERN Document Server