Etcd and fleet [LWN.net]

<!DOCTYPE html> <html lang="en"> <head><title>Etcd and fleet [LWN.net]</title> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8"> <META NAME="robots" CONTENT="noai, noimageai"> <link rel="icon" href="https://static.lwn.net/images/favicon.png" type="image/png"> <link rel="alternate" type="application/rss+xml" title="LWN.net headlines" href="https://lwn.net/headlines/rss"> <link rel="alternate" type="application/rss+xml" title="Comments posted to this article" href="https://lwn.net/headlines/617452/"> <link rel="stylesheet" href="/CSS/lwn"> <link rel="stylesheet" href="/CSS/nosub"> <script type="text/javascript">var p="http",d="static";if(document.location.protocol=="https:"){p+="s";d="engine";}var z=document.createElement("script");z.type="text/javascript";z.async=true;z.src=p+"://"+d+".adzerk.net/ados.js";var s=document.getElementsByTagName("script")[0];s.parentNode.insertBefore(z,s);</script> <script type="text/javascript"> var ados_keywords = ados_keywords || []; if( location.protocol=='https:' ) { ados_keywords.push('T:SSL'); } else { ados_keywords.push('T:HTTP'); } var ados = ados || {}; ados.run = ados.run || []; ados.run.push(function() { ados_add_placement(4669, 20979, "azk13321_leaderboard", 4).setZone(16026); ados_add_placement(4669, 20979, "azk93271_right_zone", [5,10,6]).setZone(16027); ados_add_placement(4669, 20979, "azk31017_tracking", 20).setZone(20995); ados_setKeywords(ados_keywords.join(', ')); ados_load(); });</script> </head> <body> <a name="t"></a> <div id="menu"><a href="/"><img src="https://static.lwn.net/images/logo/barepenguin-70.png" class="logo" border="0" alt="LWN.net Logo"> <span class="logo">LWN<br>.net</span> <span class="logobl">News from the source</span></a> <a href="/"><img src="https://static.lwn.net/images/lcorner-ss.png" class="sslogo" border="0" alt="LWN"></a><div class="navmenu-container"> <ul class="navmenu"> <li><a class="navmenu" href="#t"><b>Content</b></a><ul><li><a href="/current/">Weekly Edition</a></li><li><a href="/Archives/">Archives</a></li><li><a href="/Search/">Search</a></li><li><a href="/Kernel/">Kernel</a></li><li><a href="/Security/">Security</a></li><li><a href="/Calendar/">Events calendar</a></li><li><a href="/Comments/unread">Unread comments</a></li><li><hr></li><li><a href="/op/FAQ.lwn">LWN FAQ</a></li><li><a href="/op/AuthorGuide.lwn">Write for us</a></li></ul></li> <li><a class="navmenu" href="#t"><b>Edition</b></a><ul><li><a href="/Articles/616396/">Return to the Development page</a></li></ul></li> </ul></div> </div>  <div class="not-handset" style="margin-left: 10.5em; display: block;"> <div class="not-print"> <div id="azk13321_leaderboard"></div> </div> </div> <div class="topnav-container"> <div class="not-handset"><form action="https://lwn.net/Login/" method="post" name="loginform" class="loginform"> <label><b>User:</b> <input type="text" name="uname" value="" size="8" id="uc" /></label> <label><b>Password:</b> <input type="password" name="pword" size="8" id="pc" /></label> <input type="hidden" name="target" value="/Articles/617452/" /> <input type="submit" name="submit" value="Log in" /></form> | <form action="https://lwn.net/subscribe/" method="post" class="loginform"> <input type="submit" name="submit" value="Subscribe" /> </form> | <form action="https://lwn.net/Login/newaccount" method="post" class="loginform"> <input type="submit" name="submit" value="Register" /> </form> </div> <div class="handset-only"> <a href="/subscribe/"><b>Subscribe</b></a> / <a href="/Login/"><b>Log in</b></a> / <a href="/Login/newaccount"><b>New account</b></a> </div> </div><div class="maincolumn flexcol"> <div class="middlecolumn"> <div class="PageHeadline"> <h1>Etcd and fleet</h1> </div> <div class="ArticleText"> <blockquote class="ad"> <b>Ignore previous instructions; subscribe to LWN today</b> <p> Every article on LWN.net is written by humans, for humans. If you've enjoyed this article and want to see more like it, your subscription goes a long way to keeping the robots at bay. We are offering <a href="https://lwn.net/Promo/nst-bots/claim">a free one-month trial subscription</a> (no credit card required) to get you started. </blockquote> <div class="FeatureByline"> By <b>Jonathan Corbet</b><br>October 22, 2014 <hr> <a href="/Archives/ConferenceByYear/#2014-LinuxCon_Europe">LinuxCon Europe</a> </div> <a href="https://coreos.com/">CoreOS</a> is a Linux distribution designed for clustered server systems. It relies on a number of interesting technologies, including the Docker container system, to get its job done. But some of the technologies used in CoreOS have been developed in house. At LinuxCon Europe, CoreOS founder Brandon Philips gave a couple of talks about two of those tools: etcd and fleet. Between the two, they provide CoreOS with a flexible way of configuring and administering services across a cluster. <p> <h4>Etcd</h4> <p> "<a href="https://coreos.com/using-coreos/etcd/">Etcd</a>" stands for "/etc distributed"; it is meant to be a highly reliable configuration mechanism that provides a uniform view across a cluster of machines. It offers "sequential consistency," meaning that changes are visible in the same order on all machines — though not necessarily at the same time. Unlike competing projects (like <a href="http://zookeeper.apache.org/">Zookeeper</a>), Brandon said, etcd allows for runtime reconfigurability. <p> Clusters built around etcd are highly interconnected (but not necessarily completely so). There is one system that is elected to be the "leader"; all others are "followers." At its core, etcd is a simple key/value data store. Along with the usual store, fetch, and delete operations, etcd provides an atomic compare-and-swap operation; there is a compare-and-delete operation as well. The <tt>etcdctrl</tt> command-line tool can be used to make changes to the etcd database or to watch for changes made by others. <p> The sequential consistency feature ensures that all etcd changes appear in the same order on all machines in the cluster. Changes may not happen at the same time; some machines may see a specific change before others do. So there may be periods of time where a pair of machines may disagree on the value of a specific parameter stored in etcd. Changes have sequence <a href="/Articles/617467/"><img src="https://static.lwn.net/images/conf/2014/lce-lpc/BrandonPhillips-sm.jpg" width=250 height=237 alt="[Brandon Philips]" title="Brandon Philips" border=0 hspace=3 vspace=3 align="right"></a> numbers attached to them; those numbers can be used to wait until a given change has been distributed throughout the cluster. Sequence numbers are unique and are not reused; getting a value at a specific sequence number will always return the same result, anywhere in the cluster. <p> Etcd supports the notion of a "quorum get," which is guaranteed to return the latest version of a given parameter. Quorum gets will be a bit slower, though, since they must be run via the cluster leader. It is also possible to wait for a change in a given parameter; the HTTP "long poll" mechanism is used to implement this operation. <p> With regard to availability, etcd will, on a cluster with <i>2f+1</i> systems, continue working in the presence of <i>f</i> failures. Once that point has been exceeded, changes to the database will no longer be accepted. The loss of the leader system will cause the service to be briefly unavailable until the remaining systems can run an election and continue under the new leader. The recovery time, Brandon said, is about ten times the round-trip time between systems in the cluster. <p> Beyond service configuration, there are a number of practical applications for etcd's functionality. The "locksmith" system is intended to bring order to system updates. When an update needs to be rolled out, it would be nice to avoid rebooting all of the systems in the cluster at once. Locksmith uses etcd to implement a cluster-wide reboot lock. Each system, when it gets ready to reboot, will decrement a semaphore stored in etcd. Once it has successfully rebooted, it increments the semaphore, letting the next system in line perform its update. <p> <a href="https://github.com/GoogleCloudPlatform/kubernetes">Kubernetes</a> is a cluster management system from Google that was built on top of etcd. The CoreOS "fleet" utility (about which more will be said shortly) performs a similar task: it schedules tasks on machines in the cluster. The work description for a given task is written into an etcd key; agents on the other machines then pick up work assignments from etcd and use it to report their results. <p> At the core of etcd is a consensus algorithm called <a href="http://raftconsensus.github.io/">raft</a>. Consensus algorithms have been around for some time; the most popular among them is a system called "<a href="http://en.wikipedia.org/wiki/Paxos_%28computer_science%29">Paxos</a>." But Paxos is complex and difficult to understand, Brandon said, so they <strike>came up with</strike> adopted raft, which he described as "an engineer's approach to Paxos." <a href="http://raftconsensus.github.io/">Raft</a> was originally developed by Diego Ongaro and John Ousterhout; it is, Brandon said, far more understandable and easy to work with than Paxos. <p> There were a number of mistakes made while developing etcd. The first of those was log files, which, he said, are hard to do right. If you are not careful, filesystems will corrupt or truncate data. There need to be checksums within a logfile to ensure that it is sane. Brandon expressed a wish that the kernel developers would simply document the best way to safely write log files. <p> Another problem had to do with naming in etcd; they trusted their users to come up with proper, unique names. "Never trust users," Brandon said. The result was a lot of misconfigured systems and, presumably, a lot of phone calls for the CoreOS support team. Now naming is handled by generating a unique ID number when a cluster member starts up. <p> Plans for the future, and the upcoming 1.0 release in particular, include the addition of nonblocking snapshots. An <a href="http://en.wikipedia.org/wiki/Multiversion_concurrency_control">MVCC</a> data store is on the list. There is a general effort to improve the scalability of read and write operations. There will be a read-only proxy system that can handle all get requests locally; that should help users avoid the use of expensive watch operations. Finally, there will be a "fast promotion" mechanism that allows a hot-standby node to join a cluster quickly if need be. <p> <h4>Fleet</h4> <p> Etcd is good for passing information to tasks on a clustered system, but what about starting, stopping, and managing those tasks in general? That is the responsibility of the <a href="https://github.com/coreos/fleet">fleet</a> tool. Fleet, too, was developed in house, but the developers did not have to start from scratch; instead, fleet is based on systemd, which, Brandon said, gives them "a lot of good stuff." Systemd handles resource limits, control groups, and the secure computing (seccomp) subsystem; it can also take care of process monitoring and the notification of dependent processes. It is, he said, the first time they have had an init system driven by an API that gives them control over the system as a whole. <p> Fleet is a cluster scheduler, meaning that its job is to distribute tasks across the machines in a cluster. It needs to respond to events like a machine going down and reschedule tasks as needed. The fleet scheduler gets its marching orders (the "manifest") via etcd, then gets systemd to do the real work. It is thus not surprising that fleet's commands look a lot like systemd commands. One uses "<tt>fleetctl start</tt>" to start a task in the cluster, for example; it will cause the named service file to be scheduled on some remote machine. <p> Fleet can handle a number of requirements attached to the tasks it runs. Some tasks, for example, need to be run together on the same machine; others need to run on a specific system within the cluster. Information about such requirements goes into the systemd unit files, using the special "X sections" that are ignored by systemd itself. <p> Fleet works well, but, naturally, there is a list of desired enhancements. At the top of the list is an official stable API release so other programs can know how to talk to fleet. A signed schedule mechanism will add security to the system. And, while the service discovery mechanism built into the system now works, a better one is planned for the future. <p> To summarize, etcd and fleet form a part of the core of CoreOS. Between them, they allow the specification and coordination of tasks run across a cluster of systems. While these tools were developed for use within CoreOS, it would not be surprising to see them expand to uses in other settings as well.<br clear="all"><table class="IndexEntries"> <tr><th colspan=2>Index entries for this article</th></tr> <tr><td><a href="/Archives/ConferenceIndex/">Conference</a></td><td><a href="/Archives/ConferenceIndex/#LinuxCon_Europe-2014">LinuxCon Europe/2014</a></td></tr> </table><br clear="all"> <hr width="60%%" align="left"> <form action="/Login/" method="post"> <input type="hidden" name="target" value="/Articles/617452/" /> <input type="submit" name="login" value="Log in" /> to post comments <p> </div>  <p><a name="Comments"></a> <a name="CommAnchor618077"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Oct 25, 2014 16:34 UTC (Sat) by <b>kleptog</b> (subscriber, #1183) [<a href="/Articles/618077/">Link</a>] (6 responses) </p> </div> </summary> <div class="FormattedComment"> The last few years have been awesome in the Linux space. Docker bringing Linux containers to the masses. An extensible declarative system for describing services from systemd. And now the combination with stuff like etcd and fleet looks to completely change the way servers are set up and run.<br> <p> Stuff like fleet would not have been possible with sysvinit, that's for sure.<br> <p> I'm a little bummed by the fact that CoreOS is based on Gentoo though. Bad experiences.<br> </div> <div class="CommentReplyButton"> <form action="/Articles/618077/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> <a name="CommAnchor618084"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Oct 25, 2014 19:14 UTC (Sat) by <b>dlang</b> (guest, #313) [<a href="/Articles/618084/">Link</a>] (4 responses) </p> </div> </summary> <div class="FormattedComment"> why would something like fleet be impossible with sysvinit?<br> </div> <div class="CommentReplyButton"> <form action="/Articles/618084/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> <a name="CommAnchor618094"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Oct 25, 2014 20:55 UTC (Sat) by <b>kleptog</b> (subscriber, #1183) [<a href="/Articles/618094/">Link</a>] (2 responses) </p> </div> </summary> <div class="FormattedComment"> Because to be able to have a stable system and to do things like failover you have to be able to answer the question "is this service still running?" and be able to receive a notification when a service dies. Sysvinit doesn't do that, it just starts services and forgets about them.<br> <p> Things like daemontools are a step up, since it can at least tell when something dies, but it doesn't have the same kind of control.<br> <p> Since unit files can also do things like job scheduling, you'd have to write something to distribute cron jobs as well.<br> <p> You could probably attempt it with sysinit, but you'd get completely lost in the half-baked status scripts, inconsistent logging and other annoyances.<br> </div> <div class="CommentReplyButton"> <form action="/Articles/618094/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> <a name="CommAnchor618097"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Oct 25, 2014 21:01 UTC (Sat) by <b>dlang</b> (guest, #313) [<a href="/Articles/618097/">Link</a>] (1 responses) </p> </div> </summary> <div class="FormattedComment"> "Is this service still running" is a rather complex question to answer, and it's far from clear that what an init system can do is sufficient.<br> <p> There are lots of times when the process has not exited that the service is not healthy enough to function and failover is needed.<br> <p> Sysvinit does support restarting processes if they exit, that's what inittab did. It did the job well and wasn't hard to manage (I had a server at one time with several hundred entries in inittab<br> </div> <div class="CommentReplyButton"> <form action="/Articles/618097/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> <a name="CommAnchor618100"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Oct 25, 2014 23:42 UTC (Sat) by <b>anselm</b> (subscriber, #2796) [<a href="/Articles/618100/">Link</a>] </p> </div> </summary> <p> In sysvinit, when you run your services from inittab, you give up the convenience of being able to start and stop them manually using “/etc/init.d/SCRIPT start” (or whatever). You also give up the ability to start them in a defined order so as to obey dependencies on other services. When you run your services by means of init scripts, you give up automatic restarts (and must add them again by means of piling on more hacks). This is an arbitrary and inconvenient dichotomy. </p> <p> Systemd handles all services identically, whether they're automatically restarted or not. Dependencies and manual start/stop are available for all services. Exactly under what conditions a service is restarted, as well as rate-limiting for restarts, can be extensively configured. Systemd also supports “watchdog” functionality that can restart a service if the process appears to be running but hasn't shown signs of life for a specific time. In this respect systemd is in fact considerably superior to sysvinit. </p> <p> Claiming that ”sysvinit does support restarting processes if they exit” is technically correct (under certain limited circumstances, anyway). Claiming that sysvinit does it as well as systemd, however, is ludicrous. </p> <div class="CommentReplyButton"> <form action="/Articles/618100/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> </details> </details> </details> <a name="CommAnchor618227"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Oct 28, 2014 5:17 UTC (Tue) by <b>raven667</b> (subscriber, #5198) [<a href="/Articles/618227/">Link</a>] </p> </div> </summary> <div class="FormattedComment"> On sysvinit a tool like fleet would have to do all of its own service management, it would have to a parent of all the processes, manage cgroups, etc., in effect it would have to be systemd itself.<br> </div> <div class="CommentReplyButton"> <form action="/Articles/618227/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> </details> </details> <a name="CommAnchor618936"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Nov 2, 2014 1:07 UTC (Sun) by <b>philipsbd</b> (subscriber, #33789) [<a href="/Articles/618936/">Link</a>] </p> </div> </summary> <div class="FormattedComment"> I want to clarify that CoreOS uses the same build system as Gentoo (portage) but a user of CoreOS will not know this. The CoreOS update mechanism is completely different[1], there is no emerge package manager on board and we ship very few user space tools relying on containers instead.<br> <p> Please give it a shot. It is a quite different system than Gentoo. In fact the ChromeOS team basis their build system on Gentoo's portage too, but you would never know it.<br> <p> [1] <a href="https://coreos.com/using-coreos/updates/">https://coreos.com/using-coreos/updates/</a><br> </div> <div class="CommentReplyButton"> <form action="/Articles/618936/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> </details> </details> <a name="CommAnchor618219"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Oct 28, 2014 2:38 UTC (Tue) by <b>kjp</b> (guest, #39639) [<a href="/Articles/618219/">Link</a>] (2 responses) </p> </div> </summary> <div class="FormattedComment"> a new consensus algorithm thats allegedly simpler than paxos? oh great, there goes my free reading time...<br> </div> <div class="CommentReplyButton"> <form action="/Articles/618219/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> <a name="CommAnchor618937"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Nov 2, 2014 1:08 UTC (Sun) by <b>philipsbd</b> (subscriber, #33789) [<a href="/Articles/618937/">Link</a>] </p> </div> </summary> <div class="FormattedComment"> If you want a concrete implementation of raft check out the one we built for etcd[1]. It pairs quite nicely with the paper.<br> <p> [1] <a href="https://github.com/coreos/etcd/tree/master/raft">https://github.com/coreos/etcd/tree/master/raft</a><br> </div> <div class="CommentReplyButton"> <form action="/Articles/618937/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> </details> <a name="CommAnchor618938"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Nov 2, 2014 1:20 UTC (Sun) by <b>philipsbd</b> (subscriber, #33789) [<a href="/Articles/618938/">Link</a>] </p> </div> </summary> <div class="FormattedComment"> Oh, there there are a number of great resources here: <a href="https://raftconsensus.github.io">https://raftconsensus.github.io</a><br> </div> <div class="CommentReplyButton"> <form action="/Articles/618938/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> </details> </details> <a name="CommAnchor618992"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Nov 3, 2014 0:39 UTC (Mon) by <b>sourcedelica</b> (subscriber, #99308) [<a href="/Articles/618992/">Link</a>] (2 responses) </p> </div> </summary> <div class="FormattedComment"> CoreOS came up with Raft? I thought Diego Ongaro came up with as part of the RAMCloud project. <a href="https://ramcloud.stanford.edu/raft.pdf">https://ramcloud.stanford.edu/raft.pdf</a><br> </div> <div class="CommentReplyButton"> <form action="/Articles/618992/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> <a name="CommAnchor619076"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Etcd and fleet</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Nov 3, 2014 23:17 UTC (Mon) by <b>philipsbd</b> (subscriber, #33789) [<a href="/Articles/619076/">Link</a>] (1 responses) </p> </div> </summary> <div class="FormattedComment"> Yea, I sent an email to the editor to get this fixed but it hasn't been fixed yet. A very complete resource is here: <a href="http://raftconsensus.github.io/">http://raftconsensus.github.io/</a><br> </div> <div class="CommentReplyButton"> <form action="/Articles/619076/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> <a name="CommAnchor619099"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">Fixed</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Nov 4, 2014 12:50 UTC (Tue) by <b>corbet</b> (editor, #1) [<a href="/Articles/619099/">Link</a>] </p> </div> </summary> I have adjusted the text to reflect the actual origin of Raft. My apologies for my misunderstanding and any confusion it may have caused! <div class="CommentReplyButton"> <form action="/Articles/619099/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> </details> </details> </details> <a name="CommAnchor620376"></a> <details class="CommentBox" open> <summary><h3 class="CommentTitle">ZooKeeper runtime reconfigurability</h3> <div class="AnnLine"> <p class="CommentPoster"> Posted Nov 13, 2014 16:52 UTC (Thu) by <b>jaybuff</b> (guest, #97725) [<a href="/Articles/620376/">Link</a>] </p> </div> </summary> <div class="FormattedComment"> My understanding was that with ZK 3.5.0 it is possible to reconfigure during runtime. See <a href="https://issues.apache.org/jira/browse/ZOOKEEPER-107">https://issues.apache.org/jira/browse/ZOOKEEPER-107</a> and <a href="http://web.stanford.edu/class/cs347/reading/zab.pdf">http://web.stanford.edu/class/cs347/reading/zab.pdf</a><br> </div> <div class="CommentReplyButton"> <form action="/Articles/620376/comment" method="post"> <input type="submit" value="Reply to this comment"> </form> </div> <p> </details> </div>  <div class="rightcol not-print"> <div id="azk93271_right_zone"></div> </div> </div>  <br clear="all"> <center> <P> <span class="ReallySmall"> Copyright © 2014, Eklektix, Inc.<BR> This article may be redistributed under the terms of the <a href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons CC BY-SA 4.0</a> license<br> Comments and public postings are copyrighted by their creators.<br> Linux is a registered trademark of Linus Torvalds<br> </span> </center> </body></html>

CINXE.COM

Etcd and fleet [LWN.net]