CINXE.COM

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head><script type="text/javascript" src="/_static/js/bundle-playback.js?v=HxkREWBo" charset="utf-8"></script> <script type="text/javascript" src="/_static/js/wombat.js?v=txqj7nKC" charset="utf-8"></script> <script>window.RufflePlayer=window.RufflePlayer||{};window.RufflePlayer.config={"autoplay":"on","unmuteOverlay":"hidden"};</script> <script type="text/javascript" src="/_static/js/ruffle/ruffle.js"></script> <script type="text/javascript"> __wm.init("http://web.archive.org/web"); __wm.wombat("http://www.sifry.com/alerts/archives/000241.html","20090712211427","http://web.archive.org/","web","/_static/", "1247433267"); </script> <link rel="stylesheet" type="text/css" href="/_static/css/banner-styles.css?v=S1zqJCYt" /> <link rel="stylesheet" type="text/css" href="/_static/css/iconochive.css?v=3PDvdIFv" />  <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <title>Sifry's Alerts: Putting out colo fires (literally!), aka “A weekend of downtime”, grrr.</title> <link rel="stylesheet" href="http://web.archive.org/web/20090712211427cs_/http://www.sifry.com/alerts/styles-site.css" type="text/css"/> <link rel="alternate" type="application/rss+xml" title="RSS" href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/index.rdf"/> <link rel="start" href="http://www.sifry.com/alerts/" title="Home"/> <link rel="prev" href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000240.html" title="Sputnik releases Control Center 3.0, and hosted service: SputnikNet"/> <link rel="next" href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000242.html" title="And we're back"/> <script type="text/javascript" language="javascript">  </script>  </head> <body> <div id="banner"> <h1><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/" accesskey="1">Sifry's Alerts</a></h1> <span class="description">David Sifry's musings</span> </div> <div id="container"> <div class="blog"> <div id="menu"> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000240.html">« Sputnik releases Control Center 3.0, and hosted service: SputnikNet</a> | <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/">Main</a> | <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000242.html">And we're back »</a> </div> </div> <div class="blog"> <h2 class="date">September 26, 2004</h2> <div class="blogbody"> <h3 class="title">Putting out colo fires (literally!), aka “A weekend of downtime”, grrr.</h3> <p> Ugh. What a horrible weekend. I and the team have been spending the entire weekend dealing with massive data corruption caused - of all things - by an electrical fire on the main electrical line coming into our colo here in San Francisco. </p><p> The colo fire has led to a cascade of failures that has caused the Technorati service to be down for most of the weekend. It's also giving me a lot more respect for people who build and maintain 100% uptime of services, the trials and tribulations they go through, and also the cost of being operationally excellent. </p><p> <strong>What Happened</strong> </p><p> At about 9:30PM PST Friday, there was an electrical fire on the power main inside our colocation center, where our entire server infrastructure is housed. This caused our battery backup power supplies to kick in, but the independent power generator at the colo never kicked in - possibly because the problem was a fire inside the building rather than a general power blackout of the neighborhood. Well, the fire was only problem #1. We weren't expecting or planning for an outage of that kind. It caused a cascade of other problems that made the rest of the weekend a huge PITA. Problem #2 was that we didn't have a good enough emergency plan in place that would shut our systems down cleanly when power ran out like that. Unfortunately, that meant that when the batteries died, our server farm went down quite ungracefully - causing problem #3, which was data corruption due to the unclean shutdown. </p><p> The rest of the weekend has been spent recovering from these failures - we've had to do consistency checks and then rebuilds of the data sets that got corrupted, and we're doing that for over a hundred machines. Bad bad bad. At least we've been performing regular daily backups, and we're able to use that as starting points on our road to recovery. The current ETA to get services back up and running is by Monday morning, which will mean a weekend of unplanned downtime. </p><p> <strong>What we're doing about it</strong> </p><p> Clearly, this is unacceptable, but the damage is already done, so the big question on my mind is how to learn from this outage and make sure that it <strong>never</strong> happens again. One of the important things learned is that there's a reason why some colocation centers are called “Tier 1” (and priced that way) and others are not. Tier 1 means that everything is overprovisioned, and there's plenty of infrastructure backup already built into the place - electrical, network, fire suppression, environmental, security, etc. We have been planning a move to a new colocation center, but this most recent incident just underscores the need to move asap. Second, it illustrates a hole we had in our emergency plan - we had built our emergency plan based on a threat level of a short outage followed by a quick electrical recovery. We had planned for a shutdown of critical systems if battery power fell below a certain threshold (upsd for you techies out there), but we hadn't gotten it implemented given that we were planning the move to the new colo. That of course, led to the data corruption that is keeping the team up all weekend. </p><p> Once we get past this crisis and get the service back up and running again, I guarantee that we'll be doing a post-mortem analysis to see where the failure points were, and how we can avoid them in the future. I have learned a lot from this experience about the value (and implicit cost) of planning and building systems around unreliable components, and doing everything you can to eliminate risk - and also about planning for quick recovery when the unthinkable happens. I have a lot of respect for the folks at Google, Yahoo, eBay, and the like for their ability to build and maintain a solid world-class infrastructure, have it scale, and also innovate with new applications as well. </p><p> <strong>Planning for Murphy's Law <br/></strong> <br/>Count me as one of the humbled. To our users and customers: Ouch this hurts, and we're working on making sure that outages like this never happen again. To the folks down in ops and engineering at companies around the globe keeping these systems running and useful, no matter what Murphy throws at you, you've got my appreciation. </p><p> I'll post as we have more updates on service status as the day progresses. </p><p> <strong>UPDATE: </strong>As of 10:00 PST Monday, the infrastructure is back online, and we're live again, but we're still making sure that everything is back 100%. Searches should work, and we're monitoring our response time. </p> <a name="more"></a> <span class="posted">Posted by dsifry at September 26, 2004 02:19 PM | <script src="http://web.archive.org/web/20090712211427js_/http://embed.technorati.com/linkcount" type="text/javascript"></script><a href="http://web.archive.org/web/20090712211427/http://technorati.com/search/http%3A%2F%2Fwww.sifry.com%2Falerts%2Farchives%2F000241.html?sub=nscosmos" rel="linkcount">View blog reactions</a> <br/></span> </div> </div> </div> <div id="links"> <div class="sidetitle"> Search </div> <div class="side"> <form method="get" action="http://web.archive.org/web/20090712211427/http://technorati.com/cosmos/search.html"> <fieldset style="border:0"> <legend style="display:inline"><a href="http://web.archive.org/web/20090712211427/http://www.technorati.com/">Technorati</a> search</legend> <input type="hidden" name="sub" value="searchlet"/> <input type="text" name="url" style="width:12em" maxlength="255" value=""/> <input type="submit" value="Search"/><br/><label style="display:block;padding-top:3px" for="ts_yb" title="Sifry's Alerts"> <input type="radio" checked="checked" name="from" id="ts_yb" value="sifry.com/alerts"/> this blog</label> <label for="ts_www"> <input type="radio" name="from" id="ts_www" value=""/> all blogs</label> </fieldset> </form> </div> <iframe src="http://web.archive.org/web/20090712211427if_/http://widgets.technorati.com/faves/dsifry?t=posts" marginwidth="0" marginheight="0" width="200" height="400" frameborder="0" scrolling="auto" style="padding:0;border:1px solid #000"></iframe>  <div class="sidetoptitle"> Who is David Sifry? </div> <div class="side"> <a href="http://web.archive.org/web/20090712211427/http://www.technorati.com/profile/dsifry/7517/c5204f63935568e33ee55f0ee9ccdde2"><img src="http://web.archive.org/web/20090712211427im_/http://www.technorati.com/progimages/photo.jpg?uid=1" height="60" width="40" border="0" align="left" hspace="5" vspace="5" alt="Picture of me"/></a>I'm founder and CEO of <a href="http://web.archive.org/web/20090712211427/http://www.technorati.com/">Technorati</a>. I was the cofounder and CTO of <a href="http://web.archive.org/web/20090712211427/http://www.sputnik.com/">Sputnik</a> and <a href="http://web.archive.org/web/20090712211427/http://www.linuxcare.com/">Linuxcare</a>, founding board member of <a href="http://web.archive.org/web/20090712211427/http://www.li.org/">Linux International</a>, and board member of <a href="http://web.archive.org/web/20090712211427/http://www.aotc.info/">AOTC</a>. I've been around the block a few times. Some might call me a serial entrepreneur. A more personal but less-updated <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/">site</a> has more information about me - pictures, speeches, software, etc. You can contact me at <a href="http://web.archive.org/web/20090712211427/mailto:david-blog@sifry.com">david-blog@sifry.com</a>. </div> <a name="snippets"></a> <div class="sidetitle"> Snippets </div> <div class="side"> <ul> [file:///home/sifry/rss/linkblog.xml: page not found] </ul> </div> <div class="sidetitle"> Recent Entries </div> <div class="side"> <ul><li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000475.html">Posting from all my devices</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000474.html">Fantastic video - helpdesk support for earlier technologies</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000473.html">You have new Picture Mail!</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000472.html">You have new Picture Mail!</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000461.html">Boston Photowalk Tuesday Feb 20?</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000460.html">Watching the WTFs roll in</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000459.html">Technorati Joins Startups in a Viral Ad Campaign for the Big Game</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000458.html">WTF? It's a new feature!</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000457.html">Report from Davos: Steve Case Interview</a></li> <li><a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000456.html">Reminds me of an Escher drawing</a></li> </ul> </div> <div align="center" class="calendar"> <table border="0" cellspacing="4" cellpadding="0" summary="Monthly calendar with links to each day's posts"> <caption class="calendarhead">February 2007</caption> <tr> <th abbr="Sunday" align="center"><span class="calendar">Sun</span></th> <th abbr="Monday" align="center"><span class="calendar">Mon</span></th> <th abbr="Tuesday" align="center"><span class="calendar">Tue</span></th> <th abbr="Wednesday" align="center"><span class="calendar">Wed</span></th> <th abbr="Thursday" align="center"><span class="calendar">Thu</span></th> <th abbr="Friday" align="center"><span class="calendar">Fri</span></th> <th abbr="Saturday" align="center"><span class="calendar">Sat</span></th> </tr> <tr> <td align="center"><span class="calendar">  </span></td> <td align="center"><span class="calendar">  </span></td> <td align="center"><span class="calendar">  </span></td> <td align="center"><span class="calendar">  </span></td> <td align="center"><span class="calendar"> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000460.html">1</a></span></td> <td align="center"><span class="calendar"> 2</span></td> <td align="center"><span class="calendar"> 3</span></td></tr><tr> <td align="center"><span class="calendar"> 4</span></td> <td align="center"><span class="calendar"> 5</span></td> <td align="center"><span class="calendar"> 6</span></td> <td align="center"><span class="calendar"> 7</span></td> <td align="center"><span class="calendar"> 8</span></td> <td align="center"><span class="calendar"> 9</span></td> <td align="center"><span class="calendar"> 10</span></td></tr><tr> <td align="center"><span class="calendar"> 11</span></td> <td align="center"><span class="calendar"> 12</span></td> <td align="center"><span class="calendar"> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000461.html">13</a></span></td> <td align="center"><span class="calendar"> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/000475.html">14</a></span></td> <td align="center"><span class="calendar"> 15</span></td> <td align="center"><span class="calendar"> 16</span></td> <td align="center"><span class="calendar"> 17</span></td></tr><tr> <td align="center"><span class="calendar"> 18</span></td> <td align="center"><span class="calendar"> 19</span></td> <td align="center"><span class="calendar"> 20</span></td> <td align="center"><span class="calendar"> 21</span></td> <td align="center"><span class="calendar"> 22</span></td> <td align="center"><span class="calendar"> 23</span></td> <td align="center"><span class="calendar"> 24</span></td></tr><tr> <td align="center"><span class="calendar"> 25</span></td> <td align="center"><span class="calendar"> 26</span></td> <td align="center"><span class="calendar"> 27</span></td> <td align="center"><span class="calendar"> 28</span></td> <td align="center"><span class="calendar">  </span></td> <td align="center"><span class="calendar">  </span></td> <td align="center"><span class="calendar">  </span></td></tr> </table> </div> <div class="sidetitle"> By Month </div> <div class="side"> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2007_02.html">February 2007</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2007_01.html">January 2007</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_12.html">December 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_11.html">November 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_10.html">October 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_09.html">September 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_08.html">August 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_07.html">July 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_05.html">May 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_04.html">April 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_02.html">February 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2006_01.html">January 2006</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_12.html">December 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_11.html">November 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_10.html">October 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_09.html">September 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_08.html">August 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_07.html">July 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_06.html">June 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_05.html">May 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_04.html">April 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_03.html">March 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_02.html">February 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2005_01.html">January 2005</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_12.html">December 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_11.html">November 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_10.html">October 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_09.html">September 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_08.html">August 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_07.html">July 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_06.html">June 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_05.html">May 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_04.html">April 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_03.html">March 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_02.html">February 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2004_01.html">January 2004</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_12.html">December 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_11.html">November 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_10.html">October 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_09.html">September 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_08.html">August 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_07.html">July 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_06.html">June 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_05.html">May 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_04.html">April 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_03.html">March 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_02.html">February 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2003_01.html">January 2003</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_12.html">December 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_11.html">November 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_10.html">October 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_09.html">September 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_08.html">August 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_07.html">July 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_06.html">June 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_05.html">May 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_04.html">April 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_02.html">February 2002</a><br/> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/archives/2002_01.html">January 2002</a><br/> </div> <a name="linkcosmos"></a> <div class="sidetitle"> Link Cosmos </div> <div class="side"> <ul> </ul> <p>Last 25 inbound links. <a href="http://web.archive.org/web/20090712211427/http://technorati.com/">Powered by Technorati. </a> </div> <div class="side"> <script type="text/javascript" src="http://web.archive.org/web/20090712211427js_/http://technorati.com/embed/0KBKJWTED.js"> </script> </div> <div class="syndicate"> <a href="http://web.archive.org/web/20090712211427/http://www.sifry.com/alerts/index.rdf">Syndicate this site (XML)</a> </div> <div class="syndicate"> <a href="http://web.archive.org/web/20090712211427/http://creativecommons.org/licenses/by/1.0/"><img alt="Creative Commons License" border="0" src="http://web.archive.org/web/20090712211427im_/http://creativecommons.org/images/public/somerights"/></a><br/> This weblog is licensed under a <a href="http://web.archive.org/web/20090712211427/http://creativecommons.org/licenses/by/1.0/">Creative Commons License</a>. </div> <div class="powered"> Powered by<br/><a href="http://web.archive.org/web/20090712211427/http://www.movabletype.org/">Movable Type 3.17</a><br/> </div> </div> </body> </html>