Improving Reliability of Software RAID : Blueprints : Ubuntu

<!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr"> <head> <base href="https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-software-raid/+index" /> <meta charset="UTF-8" /> <title>Improving Reliability of Software RAID : Blueprints : Ubuntu</title> <link rel="apple-touch-icon" sizes="180x180" href="/@@/apple-touch-icon.png?v=2022" /> <link rel="icon" type="image/png" sizes="32x32" href="/@@/favicon-32x32.png?v=2022" /> <link rel="icon" type="image/png" sizes="16x16" href="/@@/favicon-16x16.png?v=2022" /> <link rel="manifest" href="/@@/site.webmanifest?v=2022" /> <link rel="mask-icon" href="/@@/safari-pinned-tab.svg?v=2022" color="#e9531f" /> <link rel="shortcut icon" href="/@@/favicon.ico?v=2022" /> <meta name="msapplication-TileColor" content="#da532c" /> <meta name="msapplication-config" content="/@@/browserconfig.xml?v=2022" /> <meta name="theme-color" content="#ffffff" /> <link type="text/css" rel="stylesheet" media="screen, print" href="/+icing/revaa29ae0fff49e4e804b39147c9f259d2fb023199/combo.css" /> <meta name="description" content="mdadm currently has a gaggle of open bugs, and every cycle the RAID ISO tests produce new and interesting bugs. It seems like we're doing something a bit wrong with Software RAID. There are some valid solutions here: https://wiki.ubuntu.com/ReliableRaid which should be discussed and either refuted or implemented." /> <meta property="og:description" content="mdadm currently has a gaggle of open bugs, and every cycle the RAID ISO tests produce new and interesting bugs. It seems like we're doing something a bit wrong with Software RAID. There are some valid solutions here: https://wiki.ubuntu.com/ReliableRaid which should be discussed and either refuted or implemented." /> <meta property="og:title" content="Improving Reliability of Software RAID : Blueprints : Ubuntu" /> <meta property="og:type" content="website" /> <meta property="og:image" content="/@@/launchpad-og-image.png" /> <meta property="og:url" content="https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-software-raid/+index" /> <meta property="og:site_name" content="Launchpad" /> <script type="text/javascript"> var LP = { cache: {}, links: {} }; </script> <script type="text/javascript">var cookie_scope = '; Path=/; Secure; Domain=.launchpad.net';</script> <script type="text/javascript" src="/+combo/revaa29ae0fff49e4e804b39147c9f259d2fb023199/?yui/yui/yui-min.js&lp/meta.js&yui/loader/loader-min.js"></script> <script type="text/javascript"> var raw = null; if (LP.devmode) { raw = 'raw'; } YUI.GlobalConfig = { combine: true, comboBase: '/+combo/revaa29ae0fff49e4e804b39147c9f259d2fb023199/?', root: 'yui/', filter: raw, debug: false, fetchCSS: false, maxURLLength: 2000, groups: { lp: { combine: true, base: '/+combo/revaa29ae0fff49e4e804b39147c9f259d2fb023199/?lp/', comboBase: '/+combo/revaa29ae0fff49e4e804b39147c9f259d2fb023199/?', root: 'lp/', // comes from including lp/meta.js modules: LP_MODULES, fetchCSS: false } } }</script> <script type="text/javascript"> // we need this to create a single YUI instance all events and code // talks across. All instances of YUI().use should be based off of // LPJS instead. var LPJS = new YUI(); </script> <script id="base-layout-load-scripts" type="text/javascript"> //<![CDATA[ LPJS.use('base', 'node', 'console', 'event', 'oop', 'lp', 'lp.app.foldables','lp.app.sorttable', 'lp.app.inlinehelp', 'lp.app.links', 'lp.bugs.bugtask_index', 'lp.bugs.subscribers', 'lp.app.ellipsis', 'lp.code.branchmergeproposal.diff', 'lp.views.global', function(Y) { Y.on("domready", function () { var global_view = new Y.lp.views.Global(); global_view.render(); Y.lp.app.sorttable.SortTable.init(); Y.lp.app.inlinehelp.init_help(); Y.lp.activate_collapsibles(); Y.lp.app.foldables.activate(); Y.lp.app.links.check_valid_lp_links(); }); Y.on('lp:context:web_link:changed', function(e) { window.location = e.new_value; }); }); //]]> </script> <script id="base-helper-functions" type="text/javascript"> //<![CDATA[ // This code is pulled from lp.js that needs to be available on every // request. Pulling here to get it outside the scope of the YUI block. function setFocusByName(name) { // Focus the first element matching the given name which can be focused. var nodes = document.getElementsByName(name); var i, node; for (i = 0; i < nodes.length; i++) { node = nodes[i]; if (node.focus) { try { // Trying to focus a hidden element throws an error in IE8. if (node.offsetHeight !== 0) { node.focus(); } } catch (e) { LPJS.use('console', function(Y) { Y.log('In setFocusByName(<' + node.tagName + ' type=' + node.type + '>): ' + e); }); } break; } } } function selectWidget(widget_name, event) { if (event && (event.keyCode === 9 || event.keyCode === 13)) { // Avoid firing if user is tabbing through or simply pressing // enter to submit the form. return; } document.getElementById(widget_name).checked = true; } //]]> </script> <style type="text/css"> #implementation div.yui3-ichoicesource { display: inline; } </style> </head> <body id="document" itemscope="" itemtype="http://schema.org/WebPage" class="tab-specifications main_side public yui3-skin-sam"> <div class="yui-d0"> <div id="locationbar" class="login-logout"> <div id="logincontrol"><a href="https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-software-raid/+login">Log in / Register</a></div> </div> <div id="watermark" class="watermark-apps-portlet"> <div> <a href="https://launchpad.net/ubuntu"><img alt="" width="64" height="64" src="https://launchpadlibrarian.net/606381979/CoF%2064px.png" /></a> </div> <div class="wide"> <h2 id="watermark-heading"><a href="https://launchpad.net/ubuntu">Ubuntu</a></h2> </div>  <ul class="facetmenu"> <li class="overview"><a href="https://launchpad.net/ubuntu">Overview</a></li> <li class="branches"><a href="https://code.launchpad.net/ubuntu">Code</a></li> <li class="bugs"><a href="https://bugs.launchpad.net/ubuntu">Bugs</a></li> <li class="specifications active"><a href="https://blueprints.launchpad.net/ubuntu">Blueprints</a></li> <li class="translations"><a href="https://translations.launchpad.net/ubuntu">Translations</a></li> <li class="answers"><a href="https://answers.launchpad.net/ubuntu">Answers</a></li> </ul> </div> <div class="yui-t4"> <div id="maincontent" class="yui-main"> <div class="yui-b" dir="ltr"> <div class="context-publication"> <h1 id="edit-title"> <span class="yui3-editable_text-text ellipsis" style="max-width: 95%;"> Improving Reliability of Software RAID </span> </h1> <div id="registration" class="registering"> Registered by <a href="https://launchpad.net/~clint-fewbar" class="sprite person">Clint Byrum</a> <time title="2011-04-28 16:01:19 UTC" datetime="2011-04-28T16:01:19.801948+00:00">on 2011-04-28</time> </div> </div> <div id="request-notifications"> </div> <div> <div id="maincontentsub"> <div class="top-portlet"> <div> <div class="lazr-multiline-edit" id="edit-summary"> <div class="clearfix"> </div> <div class="yui3-editable_text-text"><p>mdadm currently has a gaggle of open bugs, and every cycle the RAID ISO tests produce new and interesting bugs. It seems like we're doing something a bit wrong with Software RAID. There are some valid solutions here: <a rel="nofollow" href="https://wiki.ubuntu.com/ReliableRaid">https:/<wbr />/wiki.ubuntu.<wbr />com/ReliableRai<wbr />d</a> which should be discussed and either refuted or implemented.</p></div> </div> </div> <ul class="horizontal"> <li id="spec-url"> <a class="sprite external-link" rel="nofollow" href="https://wiki.ubuntu.com/Oneiric/SoftwareRaid">Read the full specification</a> </li> </ul> </div> <div class="yui-g"> <div class="portlet"> <h2>Blueprint information</h2> <div class="yui-u first"> <div class="two-column-list"> <dl> <dt>Status:</dt> <dd id="lifecycle-status">Started</dd> </dl> <dl> <dt>Approver:</dt> <dd> <span id="edit-approver"> <span class="yui3-activator-data-box"> <a href="https://launchpad.net/~vorlon" class="sprite person-inactive">Steve Langasek</a> </span> </span> </dd> </dl> <dl> <dt>Priority:</dt> <dd><span id="edit-priority"> <span class="value specpriorityHIGH">High</span> </span> </dd> </dl> <dl> <dt>Drafter:</dt> <dd> <span id="edit-drafter"> <span class="yui3-activator-data-box"> <a href="https://launchpad.net/~xnox" class="sprite person">Dimitri John Ledkov</a> </span> </span> </dd> </dl> <dl> <dt>Direction:</dt> <dd><span id="edit-direction_approved"> <span class="value">Approved</span> </span> </dd> </dl> <dl> <dt>Assignee:</dt> <dd> <span id="edit-assignee"> <span class="yui3-activator-data-box"> <a href="https://launchpad.net/~xnox" class="sprite person">Dimitri John Ledkov</a> </span> </span> </dd> </dl> <dl> <dt>Definition:</dt> <dd><span id="edit-definition_status"> <span class="value specstatusAPPROVED">Approved</span> </span> </dd> </dl> <dl id="series-goal"> <dt>Series goal:</dt> <dd> Accepted for <a href="/ubuntu/raring">raring</a> </dd> </dl> <dl> <dt>Implementation:</dt> <dd id="implementation"> <img src="/@@/info" alt="Informational" title="Informational blueprint, no implementation necessary" id="informational-icon" class="hidden" /> <span id="edit-implementation_status"> <span class="value specdeliverySTARTED">Started</span> </span> </dd> </dl> <dl id="milestone-target"> <dt>Milestone target:</dt> <dd> <a href="https://launchpad.net/ubuntu/+milestone/ubuntu-13.04"> <img alt="milestone icon" src="/@@/milestone" /> ubuntu-13.04</a> </dd> </dl> <dl id="started-by" class="visible"> <dt>Started by</dt> <dd> <a href="https://launchpad.net/~vorlon" class="sprite person-inactive">Steve Langasek</a> <time title="2012-05-30 16:28:52 UTC" datetime="2012-05-30T16:28:52.069172+00:00">on 2012-05-30</time> </dd> </dl> <dl id="completed-by" class="hidden"> <dt>Completed by</dt> <dd> </dd> </dl> </div> </div> <div class="yui-u"> <div id="linked_branches"> <h3>Related branches</h3> </div> <div id="bug_links"> <h3>Related bugs</h3> <table> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/136252" class="sprite bug-medium" title="Medium - Confirmed">Bug #136252: [->UUIDudev] mdadm.conf w/o ARRAY lines but udev/mdadm not assembling arrays. (boot & hotplug fails)</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/158918" class="sprite bug-high" title="High - Triaged">Bug #158918: [->UUIDudev] installing mdadm (or outdated mdadm.conf) breaks bootup</a> </td> <td> <span class="statusTRIAGED">Triaged</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/244808" class="sprite bug-undecided" title="Undecided - Confirmed">Bug #244808: {upstream} --incremental --scan --run does not start anything</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/251164" class="sprite bug-medium" title="Medium - Triaged">Bug #251164: boot impossible due to missing initramfs failure hook / event driven initramfs</a> </td> <td> <span class="statusTRIAGED">Triaged</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/251646" class="sprite bug-wishlist" title="Wishlist - Confirmed">Bug #251646: {upstream} missing command to start *single* arrays in "auto-read" mode (like --incremental)</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/252345" class="sprite bug-undecided" title="Undecided - Confirmed">Bug #252345: raid setups fail due to mdadm.conf with explicit ARRAY statements and HOMEHOST !=any</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/259145" class="sprite bug-undecided" title="Undecided - Confirmed">Bug #259145: degraded NON-root raids never --run on boot</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/320638" class="sprite bug-critical" title="Critical - Won't Fix">Bug #320638: hot-add/remove in mixed (IDE/SATA/USB/SD-card/...) RAIDs with device mapper on top => data corruption (bio too big device md0 (248 > 240))</a> </td> <td> <span class="statusWONTFIX">Won't Fix</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/488317" class="sprite bug-undecided" title="Undecided - Confirmed">Bug #488317: [->Initramfs] no degraded boot (arbitrary lvm, luks, raid, auth, ... combinations)</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/serverguide/+bug/496478" class="sprite bug-high" title="High - Expired">Bug #496478: server guide: warn about using ubuntu with raid or provide reliable raid config</a> </td> <td> <span class="statusEXPIRED">Expired</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/497186" class="sprite bug-undecided" title="Undecided - Confirmed">Bug #497186: initramfs' init-premount degrades *all* arrays (not just those required to boot)</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/535417" class="sprite bug-undecided" title="Undecided - Confirmed">Bug #535417: mdadm monitor feature broken, email notification not set up, nor using beep/wall/notify-send</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+bug/804427" class="sprite bug-undecided" title="Undecided - Invalid">Bug #804427: RAID fails after suspend</a> </td> <td> <span class="statusINVALID">Invalid</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/925280" class="sprite bug-medium" title="Medium - Confirmed">Bug #925280: Software RAID fails to rebuild after testing degraded cold boot</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/942106" class="sprite bug-medium" title="Medium - Fix Released">Bug #942106: software raid doesn't assemble before mount on boot</a> </td> <td> <span class="statusFIXRELEASED">Fix Released</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/957494" class="sprite bug-medium" title="Medium - Fix Released">Bug #957494: Missing added utility 'mdmon'</a> </td> <td> <span class="statusFIXRELEASED">Fix Released</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/968074" class="sprite bug-undecided" title="Undecided - Fix Released">Bug #968074: Partitionable raid ignored by 65-mdadm-blkid.rules</a> </td> <td> <span class="statusFIXRELEASED">Fix Released</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/969384" class="sprite bug-critical" title="Critical - Fix Released">Bug #969384: mdadm --detail --scan segfaults during update-initramfs</a> </td> <td> <span class="statusFIXRELEASED">Fix Released</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1002357" class="sprite bug-undecided" title="Undecided - Fix Released">Bug #1002357: sort out udev rules madness (3 editions installed into 4 files)</a> </td> <td> <span class="statusFIXRELEASED">Fix Released</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1009973" class="sprite bug-wishlist" title="Wishlist - Fix Released">Bug #1009973: SRU upstream bugfix micro point release 3.2.5</a> </td> <td> <span class="statusFIXRELEASED">Fix Released</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1020914" class="sprite bug-medium" title="Medium - New">Bug #1020914: The disk drive for /boot is not yet ready or not present</a> </td> <td> <span class="statusNEW">New</span> </td> </tr> <tr> <td> <a href="https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1088532" class="sprite bug-undecided" title="Undecided - Confirmed">Bug #1088532: pluging in a missing raid member does not (re)add it to array</a> </td> <td> <span class="statusCONFIRMED">Confirmed</span> </td> </tr> </table> <ul class="horizontal"> </ul> </div> <div> <h3>Sprints</h3> <ul> <li> <span><img src="/@@/yes" alt="Approved" title="Approved for the meeting agenda" /></span> <a href="/sprints/uds-o">uds-o</a> </li> <li> <span><img src="/@@/yes" alt="Approved" title="Approved for the meeting agenda" /></span> <a href="/sprints/uds-p">uds-p</a> </li> <li> <span><img src="/@@/yes" alt="Approved" title="Approved for the meeting agenda" /></span> <a href="/sprints/uds-q">uds-q</a> </li> </ul> <ul class="horizontal"> </ul> </div> </div> </div> </div> <div class="portlet"> <div class="wide"><div> <div class="lazr-multiline-edit" id="edit-whiteboard"> <div class="clearfix"> <h3>Whiteboard</h3> </div> <div class="yui3-editable_text-text"><p>Past Points:<br /> [kees] Collect historical work done on improving raid: TODO<br /> [kees] Write detailed specification of mdadm initramfs requirements: TODO<br /> [kees] Write detailed specification of mdadm post-initramfs requirements: TODO<br /> Test RAID over LVM and LVM over RAID: TODO</p> <p>Notes from etherpad:</p> <p>there are a lot of bugs</p> <p>- put together a tree of failure conditions<br /> - map the intention of how to deal with it<br /> - check existing code against intention, fix deltas<br /> - suggest/recommend smart monitoring in servers<br /> - Look into automated testing of ALL supported RAID modes<br /> - Test case for LVM over RAID<br /> - Investigate and test booting without mdadm.conf<br /> - Investigate not autostarting certain arrays<br /> - Interface with upstream for feedback (invite to UDS-P)<br /> - Fully document (maybe in conjunction with upstream?) and review existing documentation around software raid debugging and general maintenance.<br /> Multiple locations for various disk/array information can be used to diagnose what is where, and who's done what, for example:<br /> ** ll /dev/disk/by-id/<br /> ** mdadm --detail /dev/md127<br /> ** cat /proc/mdstat<br /> ** messages from kernel include references to "ata*.*" with no easy way to trace it back to a "real" /dev/sd* device<br /> ** lshw<br /> ** how to identify out of the drives, which is which? often "dd if=/dev/sd* of=/dev/null" (where * is the failed drive) and see which drive is solidly active.<br /> - The upstream documentation <a rel="nofollow" href="https://raid.wiki.kernel.org/index.php/Linux_Raid">https:/<wbr />/raid.wiki.<wbr />kernel.<wbr />org/index.<wbr />php/Linux_<wbr />Raid</a> is very basic when it comes to diagnostics or failure conditions, also quite outdated in many situations.</p> <p>Intentions<br /> - Preserve Data Integrity<br /> - Detect Known Failure Modes as early as possible<br /> - Allow system to run and reboot fine even if partial hardware failures occur(ed).<br /> - Provide Options for how to handle failure modes:<br /> - The BOOT_DEGRADED=false option allows admins to safeguard against mdadm bugs and to do recovery manually, but should default to true.</p> <p>Failure Modes<br /> - Degraded array at boot<br /> - Failed drive at runtime<br /> - Removed drive at runtime (where metadata is intact)<br /> - Adding out-of-sync drive<br /> - Adding failed drive<br /> - Drives producing corrupt reads without failure<br /> - Old RAID configuration resurrection<br /> - LVM starting up on mirror halves<br /> - Hardware is fail_ing_ (SMART details)<br /> &nbsp;&nbsp;- can we link the SMART data to some kind of user reporting?</p> <p>Validate the behavior of drivers/<wbr />arrays/<wbr />drives<br /> - does the driver notice a yanked drive?<br /> - does the driver notice a failed drive?<br /> - how does the driver react to a new drive getting inserted?<br /> ** New drive being inserted with alternate raid config<br /> ** Some controllers are hot swap, some not, how to identify?</p> <p>Debugging failure cases (user side?)<br /> - logic to align dm/ata information as expressed in dmesg etc with the /dev/sdx devices that mdadm knows about</p> <p>Links:<br /> <a rel="nofollow" href="https://wiki.ubuntu.com/HotplugRaid">https:/<wbr />/wiki.ubuntu.<wbr />com/HotplugRaid</a></p> <p>drussell 2011-05-17: Added more content to the etherpad session post UDS...<br /> slangasek 2011-10-31: this has been reproposed for a session at UDS-P, but I don't think any of the facts have changed. Why is another session needed for this?<br /> cbyrum 2011-10-31: Agreed Steve, this just needs to get done, nothing has changed.<br /> drussell - 2011-10-31: Absolutely agreed... so how do we focus on getting this done?<br /> dmitrij.ledkov 2012-05-18: Adding foundations-<wbr />q-degraded-<wbr />hw-notification as a dependency for sending degraded raid notifications to the user & integrating SMART notifications.</p> <p>foundations-<wbr />q-event-<wbr />based-initramfs dependency is not fully determined yet. It is pending on investigation of current RAID deficiencies. Only then event-based-<wbr />initramfs might become a hard dependency.</p></div> </div> </div> </div> </div> <div class="portlet"> <a href="/+help-blueprints/workitems-help.html" target="help" class="sprite maybe action-icon">(?)</a> <div class="wide"><div> <div class="lazr-multiline-edit" id="edit-workitems_text"> <div class="clearfix"> <h3>Work Items</h3> </div> <div class="yui3-editable_text-text"><p>Work items:<br /> create (Ubuntu|Upstream) RAID Architecture Specification: INPROGRESS<br /> update existing (i.e. out-of-date) RAID documentation: TODO<br /> identify and document all failure conditions: TODO<br /> investigate if foundations-<wbr />q-event-<wbr />based-initramfs is required for completing this spec (UPDATE it is due to cryptsetup): DONE<br /> test existing handling of failure modes for RAID: TODO<br /> establish automated RAID testing and RAID failure condition testing: BLOCKED<br /> review blueprint linked bugs for likelihood of fixing this cycle: DONE<br /> get input from kees on the whole topic of Reliable Raid: DONE<br /> backport critical reliability fixes to 12.04 LTS: DONE</p></div> </div> </div> </div> </div> <div class="portlet"> <h2>Dependency tree</h2> <img src="deptree.png" usemap="#deptree" /> <map id="deptree" name="deptree"> <area shape="poly" id="node1" href="https://blueprints.launchpad.net/ubuntu/+spec/boot-degraded-raid" title="Boot Support for Degraded RAID" alt="" coords="202,103,197,95,183,88,161,82,134,78,103,77,73,78,45,82,23,88,9,95,5,103,9,111,23,118,45,124,73,127,103,129,134,127,161,124,183,118,197,111"/> <area shape="poly" id="node2" title="Improving Reliability of Software RAID" alt="" coords="512,175,505,167,486,160,457,154,420,150,379,149,337,150,300,154,271,160,252,167,245,175,252,183,271,190,300,196,337,200,379,201,420,200,457,196,486,190,505,183"/> <area shape="poly" id="node3" href="https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-degraded-hw-notification" title="Degraded Hardware Notifications aka 'things you rather not see'" alt="" coords="375,30,366,22,340,15,299,10,247,6,190,5,133,6,81,10,40,15,14,22,5,30,14,38,40,46,81,51,133,55,190,56,247,55,299,51,340,46,366,38"/> <area shape="poly" id="node4" href="https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-event-based-initramfs" title="Event-based initramfs" alt="" coords="551,103,542,95,518,88,480,82,432,78,379,77,325,78,277,82,239,88,215,95,206,103,215,111,239,118,277,124,325,127,379,129,432,127,480,124,518,118,542,111"/> <area shape="poly" id="node5" href="https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-upstart-stateful-re-exec" title="Upstart stateful re-exec" alt="" coords="735,30,726,22,701,15,661,10,611,6,556,5,500,6,450,10,411,15,385,22,376,30,385,38,411,46,450,51,500,55,556,56,611,55,661,51,701,46,726,38"/> <area shape="poly" id="node6" href="https://blueprints.launchpad.net/ubuntu/+spec/udev-lvm-mdadm-evms-gutsy" title="Udev and synthetic block devices - rework for gutsy" alt="" coords="843,103,836,95,815,88,783,82,742,78,697,77,652,78,612,82,580,88,559,95,552,103,559,111,580,118,612,124,652,127,697,129,742,127,783,124,815,118,836,111"/> <area shape="poly" id="node7" href="https://blueprints.launchpad.net/ubuntu/+spec/udev-mdadm" title="Make udev and mdadm play nicely together" alt="" coords="878,30,875,22,865,15,849,10,829,6,807,5,785,6,765,10,749,15,739,22,736,30,739,38,749,46,765,51,785,55,807,56,829,55,849,51,865,46,875,38"/> </map> <p> * Blueprints in grey have been implemented. </p> <ul class="horizontal"> </ul> </div> <div class="related"> <h2>Nearby</h2> <ul> <li> <a href="/ubuntu/raring"> Other Raring blueprints </a> </li> </ul> </div> </div> <script type="text/javascript"> LPJS.use('lp.anim', 'lp.client', 'lp.deprecated.ui', 'lp.app.information_type', 'node', 'widget', function(Y) { Y.on('domready', function(){ var privacy_link = Y.one('#privacy-link'); Y.lp.app.information_type.setup_choice( privacy_link, new Y.lp.client.Launchpad(), LP.cache.context); }); Y.on('lp:context:implementation_status:changed', function(e) { var icon = Y.one('#informational-icon'); if (e.new_value == 'Informational') { icon.removeClass('hidden'); } else { icon.addClass('hidden'); } }); Y.on('lp:context:lifecycle_status:changed', function(e) { Y.lp.deprecated.ui.update_field('#lifecycle-status', e.new_value); }); Y.on('lp:context:is_started:changed', function(e) { var started = Y.one('#started-by'); if (e.new_value) { started.removeClass('hidden'); started.one('dd').setContent(e.entry.getHTML('starter')); Y.lp.anim.green_flash({node:started}).run(); } else { started.addClass('hidden'); } }); Y.on('lp:context:is_complete:changed', function(e) { var completed = Y.one('#completed-by'); if (e.new_value) { completed.removeClass('hidden'); completed.one('dd').setContent(e.entry.getHTML('completer')); Y.lp.anim.green_flash({node:completed}).run(); } else { completed.addClass('hidden'); } }); Y.on('lp:context:title:changed', function(e) { // change the window title and breadcrumb. Y.lp.deprecated.ui.update_field('ol.breadcrumbs li:last-child', e.new_value); var title = window.document.title; title = e.new_value + title.substring(e.old_value.length); window.document.title = title; }); // Watch for the whiteboard for edit mode so we can show/hide a // message to the user to make sure not to put work items in there. var whiteboard_node = Y.one('#edit-whiteboard'); var whiteboard = Y.Widget.getByNode(whiteboard_node); var notice_node = Y.Node.create('<p/>'); notice_node.set('id', 'wimessage'); notice_node.addClass('informational message'); notice_node.setContent('Please note that work items go in the separate Work Items input field below.'); whiteboard.editor.on('visibleChange', function (ev) { var par = whiteboard_node.get('parentNode'); // If we're visible, show the message if (ev.newVal) { par.insertBefore(notice_node, whiteboard_node); } else { // Otherwise we need to remove the node par.removeChild(notice_node) } }); }); </script> </div> </div> </div> <div id="side-portlets" class="yui-b side"> <div id="privacy" class="portlet public"> <div id="privacy-text"> <span id="information-type-summary" class="sprite public">This blueprint contains <strong id="information-type">Public</strong> information</span>  <div id="information-type-description" style="padding-top: 5px">Everyone can see this information. </div> </div> </div> <div class="portlet vertical" id="portlet-subscribers"> <div class="section"> <div class="subscribed-false"><a class="menu-link-subscription sprite modify edit" href="https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-software-raid/+subscribe">Edit subscription</a></div> <div><a class="menu-link-addsubscriber sprite add" href="https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-software-raid/+addsubscriber">Subscribe someone else</a></div> </div> <div id="subscribers"> <h2>Subscribers</h2> <div id="subscribers-links"> <div class="subscriber-246094 subscriber" id="subscription-subscriber-246094"> <img src="/@@/subscriber-inessential" /> <a href="/~apw">Andy Whitcroft</a> </div> <div class="subscriber-4990181 subscriber" id="subscription-subscriber-4990181"> <img src="/@@/subscriber-inessential" /> <a href="/~blood-stone">blood-stone</a> </div> <div class="subscriber-972620 subscriber" id="subscription-subscriber-972620"> <img src="/@@/subscriber-inessential" /> <a href="/~brian-murray">Brian Murray</a> </div> <div class="subscriber-513235 subscriber" id="subscription-subscriber-513235"> <img src="/@@/subscriber-inessential" /> <a href="/~ceg">ceg</a> </div> <div class="subscriber-3406492 subscriber" id="subscription-subscriber-3406492"> <img src="/@@/subscriber-inessential" /> <a href="/~townsend">Christopher Townsend</a> </div> <div class="subscriber-18141 subscriber" id="subscription-subscriber-18141"> <img src="/@@/subscriber-inessential" /> <a href="/~zulcss">Chuck Short</a> </div> <div class="subscriber-2895653 subscriber" id="subscription-subscriber-2895653"> <img src="/@@/subscriber-essential" /> <a href="/~clint-fewbar">Clint Byrum</a> </div> <div class="subscriber-91 subscriber" id="subscription-subscriber-91"> <img src="/@@/subscriber-inessential" /> <a href="/~cjwatson">Colin Watson</a> </div> <div class="subscriber-1132275 subscriber" id="subscription-subscriber-1132275"> <img src="/@@/subscriber-inessential" /> <a href="/~craig.magina">Craig Magina</a> </div> <div class="subscriber-3508017 subscriber" id="subscription-subscriber-3508017"> <img src="/@@/subscriber-essential" /> <a href="/~drussell">Dave Russell</a> </div> <div class="subscriber-1228861 subscriber" id="subscription-subscriber-1228861"> <img src="/@@/subscriber-essential" /> <a href="/~xnox">Dimitri John Ledkov</a> </div> <div class="subscriber-923336 subscriber" id="subscription-subscriber-923336"> <img src="/@@/subscriber-inessential" /> <a href="/~jamesodhunt">James Hunt</a> </div> <div class="subscriber-3315043 subscriber" id="subscription-subscriber-3315043"> <img src="/@@/subscriber-inessential" /> <a href="/~james-page">James Page</a> </div> <div class="subscriber-433401 subscriber" id="subscription-subscriber-433401"> <img src="/@@/subscriber-inessential" /> <a href="/~jdstrand">Jamie Strandboge</a> </div> <div class="subscriber-1034989 subscriber" id="subscription-subscriber-1034989"> <img src="/@@/subscriber-inessential" /> <a href="/~fevrier">Jean-Philippe Gu茅rard</a> </div> <div class="subscriber-3426129 subscriber" id="subscription-subscriber-3426129"> <img src="/@@/subscriber-inessential" /> <a href="/~jeff-underhill">Jeff Underhill</a> </div> <div class="subscriber-2990863 subscriber" id="subscription-subscriber-2990863"> <img src="/@@/subscriber-inessential" /> <a href="/~john-center">John Center</a> </div> <div class="subscriber-5005278 subscriber" id="subscription-subscriber-5005278"> <img src="/@@/subscriber-essential" /> <a href="/~fortins5">john</a> </div> <div class="subscriber-15840 subscriber" id="subscription-subscriber-15840"> <img src="/@@/subscriber-inessential" /> <a href="/~jools">Jools Wills</a> </div> <div class="subscriber-669 subscriber" id="subscription-subscriber-669"> <img src="/@@/subscriber-inessential" /> <a href="/~jorge">Jorge Castro</a> </div> <div class="subscriber-323839 subscriber" id="subscription-subscriber-323839"> <img src="/@@/subscriber-inessential" /> <a href="/~kees">Kees Cook</a> </div> <div class="subscriber-3358048 subscriber" id="subscription-subscriber-3358048"> <img src="/@@/subscriber-essential" /> <a href="/~mahmoh">mahmoh</a> </div> <div class="subscriber-3478996 subscriber" id="subscription-subscriber-3478996"> <img src="/@@/subscriber-inessential" /> <a href="/~lynxman">Marc Cluet</a> </div> <div class="subscriber-908270 subscriber" id="subscription-subscriber-908270"> <img src="/@@/subscriber-inessential" /> <a href="/~mikedep333">Michael DePaulo</a> </div> <div class="subscriber-611547 subscriber" id="subscription-subscriber-611547"> <img src="/@@/subscriber-inessential" /> <a href="/~ubuntu-michael-prokop">Michael Prokop</a> </div> <div class="subscriber-1122129 subscriber" id="subscription-subscriber-1122129"> <img src="/@@/subscriber-inessential" /> <a href="/~petermatulis">Peter Matulis</a> </div> <div class="subscriber-3239612 subscriber" id="subscription-subscriber-3239612"> <img src="/@@/subscriber-essential" /> <a href="/~peter-petrakis">Peter Petrakis</a> </div> <div class="subscriber-17269 subscriber" id="subscription-subscriber-17269"> <img src="/@@/subscriber-inessential" /> <a href="/~chihchun">Rex Tsai</a> </div> <div class="subscriber-1733822 subscriber" id="subscription-subscriber-1733822"> <img src="/@@/subscriber-inessential" /> <a href="/~smb">Stefan Bader</a> </div> <div class="subscriber-284829 subscriber" id="subscription-subscriber-284829"> <img src="/@@/subscriber-inessential" /> <a href="/~stefanor">Stefano Rivera</a> </div> <div class="subscriber-1273614 subscriber" id="subscription-subscriber-1273614"> <img src="/@@/subscriber-inessential" /> <a href="/~sbeattie">Steve Beattie</a> </div> <div class="subscriber-6739 subscriber" id="subscription-subscriber-6739"> <img src="/@@/subscriber-essential" /> <a href="/~vorlon">Steve Langasek</a> </div> <div class="subscriber-258358 subscriber" id="subscription-subscriber-258358"> <img src="/@@/subscriber-inessential" /> <a href="/~stgraber">St茅phane Graber</a> </div> <div class="subscriber-3047517 subscriber" id="subscription-subscriber-3047517"> <img src="/@@/subscriber-essential" /> <a href="/~csurbhi">Surbhi Palande</a> </div> <div class="subscriber-1889244 subscriber" id="subscription-subscriber-1889244"> <img src="/@@/subscriber-inessential" /> <a href="/~webwurst">Tobias Bradtke</a> </div> </div> </div> </div> </div> </div> <div id="footer" class="footer"> <div class="lp-arcana"> <div class="lp-branding"> <a href="https://launchpad.net/"><img src="/@@/launchpad-footer-logo.svg" alt="Launchpad" width="65" height="18" /></a>  •  <a href="https://launchpad.net/+tour">Take the tour</a>  •  <a href="https://help.launchpad.net/">Read the guide</a>   <form id="globalsearch" method="get" accept-charset="UTF-8" action="https://launchpad.net/+search"> <input type="search" id="search-text" name="field.text" /> <input type="image" src="/@@/search" style="vertical-align:5%" alt="Search Launchpad" /> </form> </div> </div> <div class="colophon"> © 2004 <a href="http://canonical.com/">Canonical Ltd.</a>  •  <a href="https://launchpad.net/legal">Terms of use</a>  •  <a href="https://www.ubuntu.com/legal/dataprivacy">Data privacy</a>  •  <a href="/feedback">Contact Launchpad Support</a>  •  <a href="http://blog.launchpad.net/">Blog</a>  •  <a href="https://canonical.com/careers">Careers</a>  •  <a href="https://ubuntu.social/@launchpadstatus">System status</a> <span id="lp-version">  •  aa29ae0 (<a href="https://dev.launchpad.net/">Get the code!</a>) </span> </div> </div> </div> <script id="json-cache-script">LP.cache = {"related_features": {}, "information_type_data": {"PUBLIC": {"value": "PUBLIC", "description": "Everyone can see this information.\n", "name": "Public", "order": 0, "is_private": false, "description_css_class": "choice-description"}}, "context": {"self_link": "https://blueprints.launchpad.net/api/devel/ubuntu/+spec/foundations-r-software-raid", "web_link": "https://blueprints.launchpad.net/ubuntu/+spec/foundations-r-software-raid", "resource_type_link": "https://blueprints.launchpad.net/api/devel/#specification", "bugs_collection_link": "https://blueprints.launchpad.net/api/devel/ubuntu/+spec/foundations-r-software-raid/bugs", "information_type": "Public", "name": "foundations-r-software-raid", "title": "Improving Reliability of Software RAID", "specification_url": "https://wiki.ubuntu.com/Oneiric/SoftwareRaid", "summary": "mdadm currently has a gaggle of open bugs, and every cycle the RAID ISO tests produce new and interesting bugs. It seems like we're doing something a bit wrong with Software RAID. There are some valid solutions here: https://wiki.ubuntu.com/ReliableRaid which should be discussed and either refuted or implemented.", "definition_status": "Approved", "assignee_link": "https://blueprints.launchpad.net/api/devel/~xnox", "drafter_link": "https://blueprints.launchpad.net/api/devel/~xnox", "approver_link": "https://blueprints.launchpad.net/api/devel/~vorlon", "priority": "High", "date_created": "2011-04-28T16:01:19.801948+00:00", "owner_link": "https://blueprints.launchpad.net/api/devel/~clint-fewbar", "target_link": "https://blueprints.launchpad.net/api/devel/ubuntu", "milestone_link": "https://blueprints.launchpad.net/api/devel/ubuntu/+milestone/ubuntu-13.04", "whiteboard": "Past Points:\n[kees] Collect historical work done on improving raid: TODO\n[kees] Write detailed specification of mdadm initramfs requirements: TODO\n[kees] Write detailed specification of mdadm post-initramfs requirements: TODO\nTest RAID over LVM and LVM over RAID: TODO\n\nNotes from etherpad:\n\nthere are a lot of bugs\n\n- put together a tree of failure conditions\n- map the intention of how to deal with it\n- check existing code against intention, fix deltas\n- suggest/recommend smart monitoring in servers\n- Look into automated testing of ALL supported RAID modes\n- Test case for LVM over RAID\n- Investigate and test booting without mdadm.conf\n- Investigate not autostarting certain arrays\n- Interface with upstream for feedback (invite to UDS-P)\n- Fully document (maybe in conjunction with upstream?) and review existing documentation around software raid debugging and general maintenance.\nMultiple locations for various disk/array information can be used to diagnose what is where, and who's done what, for example:\n** ll /dev/disk/by-id/\n** mdadm --detail /dev/md127\n** cat /proc/mdstat\n** messages from kernel include references to \"ata*.*\" with no easy way to trace it back to a \"real\" /dev/sd* device\n** lshw\n** how to identify out of the drives, which is which? often \"dd if=/dev/sd* of=/dev/null\" (where * is the failed drive) and see which drive is solidly active.\n- The upstream documentation https://raid.wiki.kernel.org/index.php/Linux_Raid is very basic when it comes to diagnostics or failure conditions, also quite outdated in many situations.\n\nIntentions\n- Preserve Data Integrity\n- Detect Known Failure Modes as early as possible\n- Allow system to run and reboot fine even if partial hardware failures occur(ed).\n- Provide Options for how to handle failure modes:\n- The BOOT_DEGRADED=false option allows admins to safeguard against mdadm bugs and to do recovery manually, but should default to true.\n\nFailure Modes\n- Degraded array at boot\n- Failed drive at runtime\n- Removed drive at runtime (where metadata is intact)\n- Adding out-of-sync drive\n- Adding failed drive\n- Drives producing corrupt reads without failure\n- Old RAID configuration resurrection\n- LVM starting up on mirror halves\n- Hardware is fail_ing_ (SMART details)\n\u0026nbsp;\u0026nbsp;- can we link the SMART data to some kind of user reporting?\n\nValidate the behavior of drivers/arrays/drives\n- does the driver notice a yanked drive?\n- does the driver notice a failed drive?\n- how does the driver react to a new drive getting inserted?\n** New drive being inserted with alternate raid config\n** Some controllers are hot swap, some not, how to identify?\n\nDebugging failure cases (user side?)\n- logic to align dm/ata information as expressed in dmesg etc with the /dev/sdx devices that mdadm knows about\n\nLinks:\nhttps://wiki.ubuntu.com/HotplugRaid\n\ndrussell 2011-05-17: Added more content to the etherpad session post UDS...\nslangasek 2011-10-31: this has been reproposed for a session at UDS-P, but I don't think any of the facts have changed. Why is another session needed for this?\ncbyrum 2011-10-31: Agreed Steve, this just needs to get done, nothing has changed.\ndrussell - 2011-10-31: Absolutely agreed... so how do we focus on getting this done?\ndmitrij.ledkov 2012-05-18: Adding foundations-q-degraded-hw-notification as a dependency for sending degraded raid notifications to the user \u0026 integrating SMART notifications.\n\nfoundations-q-event-based-initramfs dependency is not fully determined yet. It is pending on investigation of current RAID deficiencies. Only then event-based-initramfs might become a hard dependency.", "workitems_text": "Work items:\ncreate (Ubuntu|Upstream) RAID Architecture Specification: INPROGRESS\nupdate existing (i.e. out-of-date) RAID documentation: TODO\nidentify and document all failure conditions: TODO\ninvestigate if foundations-q-event-based-initramfs is required for completing this spec (UPDATE it is due to cryptsetup): DONE\ntest existing handling of failure modes for RAID: TODO\nestablish automated RAID testing and RAID failure condition testing: BLOCKED\nreview blueprint linked bugs for likelihood of fixing this cycle: DONE\nget input from kees on the whole topic of Reliable Raid: DONE\nbackport critical reliability fixes to 12.04 LTS: DONE", "direction_approved": true, "implementation_status": "Started", "starter_link": "https://blueprints.launchpad.net/api/devel/~vorlon", "date_started": "2012-05-30T16:28:52.069172+00:00", "completer_link": null, "date_completed": null, "dependencies_collection_link": "https://blueprints.launchpad.net/api/devel/ubuntu/+spec/foundations-r-software-raid/dependencies", "linked_branches_collection_link": "https://blueprints.launchpad.net/api/devel/ubuntu/+spec/foundations-r-software-raid/linked_branches", "is_complete": false, "is_started": true, "lifecycle_status": "Started", "has_accepted_goal": true, "http_etag": "\"3167d2ad117a0c9ed75078971cdb1f7b837428a8-ba864fcfd96797edbf9c24f00f7965d59e1d82da\""}};</script> </body>  </html>

CINXE.COM

Improving Reliability of Software RAID : Blueprints : Ubuntu