{"id":225,"date":"2011-06-29T13:01:26","date_gmt":"2011-06-29T20:01:26","guid":{"rendered":"https:\/\/analogman.org\/?p=225"},"modified":"2011-06-29T13:01:26","modified_gmt":"2011-06-29T20:01:26","slug":"recovering-a-broken-mirrored-drive-ubuntu-9-04","status":"publish","type":"post","link":"https:\/\/www.analogman.org\/?p=225","title":{"rendered":"Recovering a broken mirrored drive – ubuntu 9.04"},"content":{"rendered":"

The other day I got an email from mdadm, a process running on some of our servers that keeps an eye on the raid array.<\/p>\n

—–<\/p>\n

This is an automatically generated mail message from mdadm\u00a0running on woo<\/em><\/address>\n
A DegradedArray event<\/strong> had been detected on md device \/dev\/md0.<\/em><\/address>\n
Faithfully yours, etc.<\/em><\/address>\n
P.S. The \/proc\/mdstat file currently contains the following:<\/em><\/address>\n
Personalities <\/strong>: [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]<\/em><\/address>\n
md1 :<\/strong> active raid1 sda2[0]<\/em><\/address>\n
303805120 blocks [2\/1] [U_]<\/em><\/address>\n
md0 :<\/strong> active raid1 sda1[0]<\/em><\/address>\n
7815488 blocks [2\/1] [U_]<\/em><\/address>\n
unused devices:<\/strong> <none><\/em><\/address>\n

—–<\/p>\n

This was not a happy event – looks like one of the two drives in the array was no longer working.<\/p>\n

    \n
  • This is a system that’s a couple of years old running software RAID 1 (mirrored) 320gb SATA drives.<\/li>\n
  • OS is Ubuntu 9.04 running web services.<\/li>\n
  • The failed drive is no longer readable by the system.<\/li>\n
  • There are only two partitions on the drive : System and Swap.<\/li>\n<\/ul>\n

    Easiest thing to do here is to replace the drive (first making a new backup).<\/p>\n

    I just ran a quick check on the raid status to confirm the email I had received.<\/p>\n

    cat \/proc\/mdstat (maybe you will need to sudo this command)<\/em><\/strong><\/p>\n

    This is my output<\/p>\n

     <\/p>\n

    —–<\/p>\n

    Sun Jun 26:02:27 PM:~$ cat \/proc\/mdstat<\/strong><\/em><\/address>\n
    Personalities : <\/strong>[linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]<\/em><\/address>\n
    md1 :<\/strong> active raid1 sda2[0]<\/em><\/address>\n
    303805120 blocks [2\/1] [U_]<\/em><\/address>\n
    md0 : <\/strong>active raid1 sda1[0]<\/em><\/address>\n
    7815488 blocks [2\/1] [U_]<\/em><\/address>\n
    unused devices: <none><\/em><\/address>\n
    Sun Jun 26:02:28 PM:~$<\/em><\/strong><\/address>\n
    —–<\/address>\n

    Now I had to pull the bad drive and replace it.<\/p>\n

    1- Sometimes you can find out which is the bad drive by looking in dmesg for the read failure on the device.<\/p>\n

    dmesg | grep ata (or whatever is appropriate for you)<\/em><\/strong><\/p>\n

    2- Shutdown and unplug the suspect drive – reboot to confirm you have the correct device unplugged.<\/p>\n

    3- Plug in the new drive (best if it is unpartitioned\/unformatted)<\/p>\n

    reboot and watch the boot up to see if the drive shows up –\u00a0if you don’t see it go by on the screen (i always get attracted to something else and forget to watch carefully).<\/p>\n

    Once the box is booted up – grep the output of dmesg to find the new device.<\/p>\n

    4- You can also check (and get important info for the next steps) by running<\/p>\n

    sudo fdisk -l<\/strong><\/em><\/p>\n

    ——–<\/p>\n

    Sun Jun 26:02:30 PM:~$<\/strong> sudo fdisk -l<\/em><\/address>\n
    [sudo] password for ken:<\/em><\/address>\n
    Disk \/dev\/sda:<\/strong> 320.0 GB, 320072933376 bytes<\/em><\/address>\n
    255 heads, 63 sectors\/track, 38913 cylinders<\/em><\/address>\n
    Units = cylinders of 16065 * 512 = 8225280 bytes<\/em><\/address>\n
    Disk identifier: 0x59b728b7<\/em><\/address>\n
    Device \u00a0 \u00a0 \u00a0 Boot\u00a0 \u00a0 \u00a0 Start \u00a0 \u00a0 \u00a0 \u00a0 End\u00a0 \u00a0 \u00a0 Blocks \u00a0 \u00a0 \u00a0Id\u00a0 System<\/em><\/strong><\/address>\n
    \/dev\/sda1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 973 \u00a0 \u00a0 \u00a0 7815591 \u00a0 \u00a0 \u00a0 \u00a0fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sda2 \u00a0 * \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 974 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a038795 \u00a0 303805215 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    Disk \/dev\/sdb<\/strong>: 500.1 GB, 500107862016 bytes<\/em><\/address>\n
    255 heads, 63 sectors\/track, 60801 cylinders<\/em><\/address>\n
    Units = cylinders of 16065 * 512 = 8225280 bytes<\/em><\/address>\n
    Disk identifier: 0x00000000<\/em><\/address>\n
    Disk \/dev\/sdb<\/strong> doesn’t contain a valid partition table<\/em><\/address>\n
    Disk \/dev\/md0:<\/strong> 8003 MB, 8003059712 bytes<\/em><\/address>\n
    2 heads, 4 sectors\/track, 1953872 cylinders<\/em><\/address>\n
    Units = cylinders of 8 * 512 = 4096 bytes<\/em><\/address>\n
    Disk identifier: 0x00000000<\/em><\/address>\n
    Disk \/dev\/md0<\/strong> doesn’t contain a valid partition table<\/em><\/address>\n
    Disk \/dev\/md1: <\/strong>311.0 GB, 311096442880 bytes<\/em><\/address>\n
    2 heads, 4 sectors\/track, 75951280 cylinders<\/em><\/address>\n
    Units = cylinders of 8 * 512 = 4096 bytes<\/em><\/address>\n
    Disk identifier: 0x00000000<\/em><\/address>\n
    Disk \/dev\/md1 <\/strong>doesn’t contain a valid partition table<\/em><\/address>\n
    Sun Jun 26:02:35 PM:~$<\/em><\/strong><\/address>\n
    —–<\/address>\n

    Note that my working drive is sda with a couple of partitions.<\/p>\n

    The device sdb doesn’t have a valid partition table. Your milage (and drive designations will vary).<\/p>\n

    5- Now to get the raid back on track we need to copy the existing partition table from the functioning raid drive to the newly installed drive.<\/p>\n

    (Dangerous stuff here – I have never tried it but would almost bet money that getting the drives backwards would not be ‘good’)<\/p>\n

    So here is my output for sudo sfdisk -l<\/p>\n

    —–<\/p>\n

    Sun Jun 26:02:49 PM:~$<\/strong> sudo sfdisk -l<\/em><\/address>\n
    Disk \/dev\/sda: <\/strong>38913 cylinders, 255 heads, 63 sectors\/track<\/em><\/address>\n
    Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0<\/em><\/address>\n
    Device Boot Start \u00a0 \u00a0 End \u00a0 #cyls\u00a0 \u00a0 #blocks \u00a0 Id\u00a0 System<\/em><\/address>\n
    \/dev\/sda1\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0+\u00a0 \u00a0 972 \u00a0 \u00a0 973- \u00a0 7815591 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sda2 \u00a0 *\u00a0 \u00a0 973 \u00a0 38794 \u00a0 37822\u00a0 303805215 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sda3\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 \u00a0 \u00a0 – \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 0\u00a0 Empty<\/em><\/address>\n
    \/dev\/sda4\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 \u00a0 \u00a0 – \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 0\u00a0 Empty<\/em><\/address>\n
    Disk \/dev\/sdb:<\/strong> 60801 cylinders, 255 heads, 63 sectors\/track<\/em><\/address>\n
    sfdisk:<\/strong> ERROR: sector 0 does not have an msdos signature<\/em><\/address>\n
    \/dev\/sdb<\/strong>: unrecognized partition table type<\/em><\/address>\n
    No partitions found<\/em><\/address>\n
    Disk \/dev\/md0:<\/strong> 1953872 cylinders, 2 heads, 4 sectors\/track<\/em><\/address>\n
    sfdisk:<\/strong> ERROR: sector 0 does not have an msdos signature<\/em><\/address>\n
    \/dev\/md0:<\/strong> unrecognized partition table type<\/em><\/address>\n
    No partitions found<\/em><\/address>\n
    Disk \/dev\/md1:<\/strong> 75951280 cylinders, 2 heads, 4 sectors\/track<\/em><\/address>\n
    sfdisk: <\/strong>ERROR: sector 0 does not have an msdos signature<\/em><\/address>\n
    \/dev\/md1:<\/strong> unrecognized partition table type<\/em><\/address>\n
    No partitions found<\/em><\/address>\n
    Sun Jun 26:02:49 PM:~$<\/strong><\/em><\/address>\n
    —–<\/address>\n

    6- Check out the man page for sfdisk and read through some of the stuff there.<\/p>\n

    We are going to use the -d option which should give us the partition information<\/p>\n

    about one device and pipe that through to the other device – hopefully using the partition information gleaned from the good drive to recreate the same partitions on the new drive\u2026(fingers crossed here)<\/p>\n

     <\/p>\n

    sudo sfdisk -d \/dev\/sda | sudo sfdisk \/dev\/sdb<\/strong><\/em><\/p>\n

     <\/p>\n

    So again we are just piping the output of the first sfdisk command into the input of the second.<\/p>\n

    If you want see the output of the first part of the command before you commit to destroying whatever is on the target of the second sfdisk command you can enter just that portion and see what you get.<\/p>\n

    sudo sfdisk -d \/dev\/sda<\/strong><\/em> (again use the appropriate drive designation here for your system – not mine)<\/p>\n

    You should get some output that sort of makes sense to you\u2026<\/p>\n

     <\/p>\n

    —–<\/p>\n

    Sun Jun 26:02:49 PM:~$<\/strong> sudo sfdisk -d \/dev\/sda<\/em><\/address>\n
    # partition table of \/dev\/sda<\/em><\/address>\n
    unit: sectors<\/em><\/address>\n
    \/dev\/sda1 : start= \u00a0 \u00a0 \u00a0 63, size= 15631182, Id=fd<\/em><\/address>\n
    \/dev\/sda2 : start= 15631245, size=607610430, Id=fd, bootable<\/em><\/address>\n
    \/dev\/sda3 : start=\u00a0 \u00a0 \u00a0 \u00a0 0, size=\u00a0 \u00a0 \u00a0 \u00a0 0, Id= 0<\/em><\/address>\n
    \/dev\/sda4 : start=\u00a0 \u00a0 \u00a0 \u00a0 0, size=\u00a0 \u00a0 \u00a0 \u00a0 0, Id= 0<\/em><\/address>\n
    Sun Jun 26:02:57 PM:~$<\/em><\/strong><\/address>\n
    —–<\/address>\n

    If you point this command at the newly installed drive you should get an error (unless it has an existing partition table that sfdisk recognizes).<\/p>\n

    Here is mine again<\/p>\n

     <\/p>\n

    —–<\/p>\n

    Sun Jun 26:02:57 PM:~$<\/strong> sudo sfdisk -d \/dev\/sdb<\/em><\/address>\n
    sfdisk: ERROR: sector 0 does not have an msdos signature<\/em><\/address>\n
    \/dev\/sdb: unrecognized partition table type<\/em><\/address>\n
    No partitions found<\/em><\/address>\n
    Sun Jun 26:02:59 PM:~$<\/strong><\/em><\/address>\n

    —–<\/p>\n

     <\/p>\n

    All of this double checking makes me feel a little better about continuing\u2026<\/p>\n

     <\/p>\n

    —–<\/p>\n

    Sun Jun 26:02:59 PM:~$ <\/strong>sudo sfdisk -d \/dev\/sda | sudo sfdisk \/dev\/sdb<\/em><\/address>\n
    Checking that no-one is using this disk right now …<\/em><\/address>\n
    OK<\/em><\/address>\n
    Disk \/dev\/sdb: 60801 cylinders, 255 heads, 63 sectors\/track<\/em><\/address>\n
    Old situation:<\/em><\/address>\n
    Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0<\/em><\/address>\n
    Device Boot Start \u00a0 \u00a0 End \u00a0 #cyls\u00a0 \u00a0 #blocks \u00a0 Id\u00a0 System<\/em><\/address>\n
    \/dev\/sdb1\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0+\u00a0 \u00a0 972 \u00a0 \u00a0 973- \u00a0 7815591 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sdb2 \u00a0 *\u00a0 \u00a0 973 \u00a0 38794 \u00a0 37822\u00a0 303805215 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sdb3\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 \u00a0 \u00a0 – \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 0\u00a0 Empty<\/em><\/address>\n
    \/dev\/sdb4\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 \u00a0 \u00a0 – \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 0\u00a0 Empty<\/em><\/address>\n
    New situation:<\/em><\/address>\n
    Units = sectors of 512 bytes, counting from 0<\/em><\/address>\n
    Device Boot\u00a0 \u00a0 Start \u00a0 \u00a0 \u00a0 End \u00a0 #sectors\u00a0 Id\u00a0 System<\/em><\/address>\n
    \/dev\/sdb1\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 63\u00a0 15631244 \u00a0 15631182\u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sdb2 \u00a0 *\u00a0 15631245 623241674\u00a0 607610430\u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sdb3 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 \u00a0 \u00a0 \u00a0 –\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 0\u00a0 Empty<\/em><\/address>\n
    \/dev\/sdb4 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 \u00a0 \u00a0 \u00a0 –\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 0\u00a0 Empty<\/em><\/address>\n
    Successfully wrote the new partition table<\/em><\/address>\n
    Re-reading the partition table …<\/em><\/address>\n
    If you created or changed a DOS partition, \/dev\/foo7, say, then use dd(1)<\/em><\/address>\n
    to zero the first 512 bytes:\u00a0 dd if=\/dev\/zero of=\/dev\/foo7 bs=512 count=1<\/em><\/address>\n
    (See fdisk(8).)<\/em><\/address>\n
    Sun Jun 26:03:01 PM:~$<\/strong><\/em><\/address>\n
    —–<\/address>\n

    Whew\u2026I always get a\u00a0 little butterfly thing no matter how many drives i break\u2026<\/p>\n

    (P.S. right at the moment I am listening to a shredder from the early 90s Gary Hoey. No Joe Satriani but still fun sometimes)<\/p>\n

     <\/p>\n

    7- So now I want to take another quick look at all of the partitions with<\/p>\n

    sudo fdisk -l<\/em><\/strong><\/p>\n

    —–<\/p>\n

    Sun Jun 26:03:02 PM:~$<\/strong> sudo fdisk -l<\/em><\/address>\n
    Disk \/dev\/sda: 320.0 GB, 320072933376 bytes<\/em><\/address>\n
    255 heads, 63 sectors\/track, 38913 cylinders<\/em><\/address>\n
    Units = cylinders of 16065 * 512 = 8225280 bytes<\/em><\/address>\n
    Disk identifier: 0x59b728b7<\/em><\/address>\n
    Device Boot\u00a0 \u00a0 \u00a0 Start \u00a0 \u00a0 \u00a0 \u00a0 End\u00a0 \u00a0 \u00a0 Blocks \u00a0 Id\u00a0 System<\/em><\/address>\n
    \/dev\/sda1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 973 \u00a0 \u00a0 7815591 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sda2 \u00a0 * \u00a0 \u00a0 \u00a0 \u00a0 974 \u00a0 \u00a0 \u00a0 38795 \u00a0 303805215 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    Disk \/dev\/sdb: 500.1 GB, 500107862016 bytes<\/em><\/address>\n
    255 heads, 63 sectors\/track, 60801 cylinders<\/em><\/address>\n
    Units = cylinders of 16065 * 512 = 8225280 bytes<\/em><\/address>\n
    Disk identifier: 0x00000000<\/em><\/address>\n
    Device Boot\u00a0 \u00a0 \u00a0 Start \u00a0 \u00a0 \u00a0 \u00a0 End\u00a0 \u00a0 \u00a0 Blocks \u00a0 Id\u00a0 System<\/em><\/address>\n
    \/dev\/sdb1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 973 \u00a0 \u00a0 7815591 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    \/dev\/sdb2 \u00a0 * \u00a0 \u00a0 \u00a0 \u00a0 974 \u00a0 \u00a0 \u00a0 38795 \u00a0 303805215 \u00a0 fd\u00a0 Linux raid autodetect<\/em><\/address>\n
    Disk \/dev\/md0: 8003 MB, 8003059712 bytes<\/em><\/address>\n
    2 heads, 4 sectors\/track, 1953872 cylinders<\/em><\/address>\n
    Units = cylinders of 8 * 512 = 4096 bytes<\/em><\/address>\n
    Disk identifier: 0x00000000<\/em><\/address>\n
    Disk \/dev\/md0 doesn’t contain a valid partition table<\/em><\/address>\n
    Disk \/dev\/md1: 311.0 GB, 311096442880 bytes<\/em><\/address>\n
    2 heads, 4 sectors\/track, 75951280 cylinders<\/em><\/address>\n
    Units = cylinders of 8 * 512 = 4096 bytes<\/em><\/address>\n
    Disk identifier: 0x00000000<\/em><\/address>\n
    Disk \/dev\/md1 doesn’t contain a valid partition table<\/em><\/address>\n

    —–<\/p>\n

    8- Nice – there is the second drive with appropriate partitions but still not a happy raid camper.<\/p>\n

    Again and again – use the correct nomenclature for your particular system configuration<\/p>\n

     <\/p>\n

    In the case of our example system we will use these commands<\/p>\n

    This is for the swap partition<\/p>\n

    sudo mdadam –add \/dev\/md0 \/dev\/sdb1<\/strong><\/em><\/p>\n

    —–<\/p>\n

    Sun Jun 26:03:13 PM:~$<\/strong><\/em> sudo mdadm –add \/dev\/md0 \/dev\/sdb1<\/address>\n
    mdadm: added \/dev\/sdb1<\/address>\n
    —–<\/address>\n

    and this is for the system partition<\/p>\n

    sudo madam –add \/dev\/md1 \/dev\/sdb2<\/strong><\/em><\/p>\n

    —–<\/strong><\/p>\n

    Sun Jun 26:03:13 PM:~$<\/strong> sudo mdadm –add \/dev\/md1 \/dev\/sdb2<\/em><\/address>\n
    mdadm: added \/dev\/sdb2<\/em><\/address>\n
    —–<\/span><\/address>\n
    so now that we did that -lets see what is going on \u00a0by looking at mdstat again.<\/address>\n

    —–<\/p>\n

    Sun Jun 26:03:16 PM:~$<\/strong> cat \/proc\/mdstat<\/em><\/address>\n
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]<\/em><\/address>\n
    md1 : active raid1 sdb2[2] sda2[0]<\/em><\/address>\n
    303805120 blocks [2\/1] [U_]<\/em><\/address>\n
    [>………………..]\u00a0 recovery =\u00a0 0.9% (2881792\/303805120) finish=80.2min speed=62493K\/sec<\/em><\/address>\n
    md0 : active raid1 sdb1[1] sda1[0]<\/em><\/address>\n
    7815488 blocks [2\/2] [UU]<\/em><\/address>\n
    unused devices: <none><\/em><\/address>\n
    Sun Jun 26:03:16 PM:~$<\/em><\/strong><\/address>\n

    —–<\/p>\n

    Awesome stuff. Look, the computer machine is working to bring the newly added device up to snuff. Love it.<\/p>\n

     <\/p>\n

    You can also get additional information using<\/p>\n

    sudo mdadm –detail \/dev\/md1<\/strong><\/em><\/p>\n

    sudo mdadm –detail \/dev\/md0<\/strong><\/em><\/p>\n

    —–<\/p>\n

    Sun Jun 26:04:02 PM:~$ sudo mdadm –detail \/dev\/md0<\/em><\/address>\n
    \/dev\/md0:<\/em><\/address>\n
    Version : 00.90<\/em><\/address>\n
    Creation Time : Sat Sep 19 19:59:31 2009<\/em><\/address>\n
    Raid Level : raid1<\/em><\/address>\n
    Array Size : 7815488 (7.45 GiB 8.00 GB)<\/em><\/address>\n
    Used Dev Size : 7815488 (7.45 GiB 8.00 GB)<\/em><\/address>\n
    Raid Devices : 2<\/em><\/address>\n
    Total Devices : 2<\/em><\/address>\n
    Preferred Minor : 0<\/em><\/address>\n
    Persistence : Superblock is persistent<\/em><\/address>\n
    Update Time : Sun Jun 26 15:15:35 2011<\/em><\/address>\n
    State : clean<\/em><\/address>\n
    Active Devices : 2<\/em><\/address>\n
    Working Devices : 2<\/em><\/address>\n
    Failed Devices : 0<\/em><\/address>\n
    Spare Devices : 0<\/em><\/address>\n
    UUID : c6fe5bf1:47145c2e:8f53a666:581a3da1<\/em><\/address>\n
    Events : 0.604<\/em><\/address>\n
    Number \u00a0 Major \u00a0 Minor \u00a0 RaidDevice State<\/em><\/address>\n
    0 \u00a0 \u00a0 \u00a0 8\u00a0 \u00a0 \u00a0 \u00a0 1\u00a0 \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 \u00a0 active sync \u00a0 \/dev\/sda1<\/em><\/address>\n
    1 \u00a0 \u00a0 \u00a0 8 \u00a0 \u00a0 \u00a0 17\u00a0 \u00a0 \u00a0 \u00a0 1\u00a0 \u00a0 \u00a0 active sync \u00a0 \/dev\/sdb1<\/em><\/address>\n
    Sun Jun 26:04:04 PM:~$<\/em><\/address>\n
    -----<\/em><\/pre>\n
    Sun Jun 26:03:30 PM:~$ sudo mdadm –detail \/dev\/md1<\/em><\/address>\n
    \/dev\/md1:<\/em><\/address>\n
    Version : 00.90<\/em><\/address>\n
    Creation Time : Sat Sep 19 19:59:48 2009<\/em><\/address>\n
    Raid Level : raid1<\/em><\/address>\n
    Array Size : 303805120 (289.73 GiB 311.10 GB)<\/em><\/address>\n
    Used Dev Size : 303805120 (289.73 GiB 311.10 GB)<\/em><\/address>\n
    Raid Devices : 2<\/em><\/address>\n
    Total Devices : 2<\/em><\/address>\n
    Preferred Minor : 1<\/em><\/address>\n
    Persistence : Superblock is persistent<\/em><\/address>\n
    Update Time : Sun Jun 26 16:02:40 2011<\/em><\/address>\n
    State : active, degraded, recovering<\/em><\/address>\n
    Active Devices : 1<\/em><\/address>\n
    Working Devices : 2<\/em><\/address>\n
    Failed Devices : 0<\/em><\/address>\n
    Spare Devices : 1<\/em><\/address>\n
    Rebuild Status : 55% complete<\/em><\/address>\n
    UUID : a004ba5a:4a61bca9:f20d5c50:35d36b51<\/em><\/address>\n
    Events : 0.4423477<\/em><\/address>\n
    Number \u00a0 Major \u00a0 Minor \u00a0 RaidDevice State<\/em><\/address>\n
    0 \u00a0 \u00a0 \u00a0 8\u00a0 \u00a0 \u00a0 \u00a0 2\u00a0 \u00a0 \u00a0 \u00a0 0\u00a0 \u00a0 \u00a0 active sync \u00a0 \/dev\/sda2<\/em><\/address>\n
    2 \u00a0 \u00a0 \u00a0 8 \u00a0 \u00a0 \u00a0 18\u00a0 \u00a0 \u00a0 \u00a0 1\u00a0 \u00a0 \u00a0 spare rebuilding \u00a0 \/dev\/sdb2<\/em><\/address>\n
    —–<\/em><\/address>\n
    9- Now go get a cup of coffee, tea, water\u2026what ever you enjoy. this recovery will take a bit of time to complete. I am going to have some left over pasta from dinner last night.<\/p>\n
    10- Finally we want to install GRUB on to the new drive<\/p>\n
    sudo grub-install \/dev\/md1<\/em><\/strong><\/address>\n
     <\/address>\n
    Good luck, ken.<\/em><\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n","protected":false},"excerpt":{"rendered":"
    The other day I got an email from mdadm, a process running on some of our servers that keeps an eye on the raid array. —– This is an automatically generated mail message from mdadm\u00a0running on woo A DegradedArray event had been detected on md device \/dev\/md0. Faithfully yours, etc. P.S. The \/proc\/mdstat file currently […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[27,8,10],"tags":[30,29,28],"class_list":["post-225","post","type-post","status-publish","format-standard","hentry","category-linux","category-mods","category-training","tag-broken","tag-linux-2","tag-raid"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.analogman.org\/index.php?rest_route=\/wp\/v2\/posts\/225","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.analogman.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.analogman.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.analogman.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.analogman.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=225"}],"version-history":[{"count":7,"href":"https:\/\/www.analogman.org\/index.php?rest_route=\/wp\/v2\/posts\/225\/revisions"}],"predecessor-version":[{"id":232,"href":"https:\/\/www.analogman.org\/index.php?rest_route=\/wp\/v2\/posts\/225\/revisions\/232"}],"wp:attachment":[{"href":"https:\/\/www.analogman.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=225"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.analogman.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=225"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.analogman.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=225"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}