Resurrecting a lost hard disk...

My main desktop system had been powered off for a few weeks and when I powered it on last weekend it wouldn't boot up completely.  It was complaining about errors, specifically errors on the hard drive that contains all my documents.  The 1.5TB hard drive had been my home directory repository for the past 2 years and never gave any signs of trouble.

I grabbed my copy of SpinRite and set it loose, but it too had troubles and would get a "divide by zero" error less than 30 seconds after starting.  Not good.  I got a bit further by removing it and trying to access it via a USB-to-SATA adapter I had.  This time SpinRite could work on the drive (albeit VERY slowly), but I still couldn't mount it or see the data.

After playing with it for most of last weekend I resigned to the fact that it was dead.  Thankfully, when I purchased it via NewEgg.com in November 2010, I spent the extra money and got the Western Digital Caviar Black 1.5TB SATA drive - it came with a 5 year warranty.  It was well within the WD warranty period, and when I got to their on-line support page, all I had to do was enter the serial number and choose cross-ship, and the new drive was waiting for me two days later.

While waiting for the HDD, I did some research and came across the "ddrescue" utility - a Linux utility that tries to read raw sectors from one place and dump them to a new file or other drive.  The tool will attempt to re-read sectors a few times, but it will log the bad sectors in a file, then continue with the rest of the disk.  I decided to give it a try, and I'm glad I did.

When the replacement drive came, I put it into the case and attached the failing drive to the USB adapter.  After running SpinRite on the new drive for a couple days (even level two took 20+ hours - I couldn't really start on recovery until Friday night), I decided to try the ddrescue tool.  I ended up booting from an Ubuntu "LiveCD" and running a very simple command:

ddrescue -v /dev/sdc /dev/sda disk_rescue.001.wri

That setup ddrescue to be verbose ("-v"), try coping everything from /dev/sdc (the failing drive), to /dev/sda (the new drive), and save all your notes in a file named "disk_rescue.001.wri".

It was slow going at first.  It was maxing out at about 500KByte/second - that was going to take a long time.  As I let it sit for about 15 minutes, I noticed that it started leveling out at a better 6MByte/second.  Still not great but better.

Since ddrescue will auto-restart if it has a log file to reference, I killed it and re-ran it with the "-c 1024" option.  Using "-c" told the program to take a bigger "bite" each time it copied from the disk, and that seemed to bring the speed up to 32MByte/second.  Playing with different values didn't help the speed much, so I let it run over night and it was still running Saturday morning.  It ran all day Saturday, but finally finished sometime early Sunday morning.

I could tell tha some stuff was being copied - I used "dd if=/dev/sda count=25 bs=1024k | strings | more" and could see some text - but mounting the new filesystem reported a lot of errors about the superblock.

I did some additional searching and found this blog entry:

How to repair a broken ext4 superblock in Ubuntu 

In short I ran "mke2fs -n /dev/sda1" and found a number of superblocks listed so I chose one more in the middle.  I then ran "e2fsck -b {block_number} /dev/sda1" and let it run.  The first time I ran it, there were a lot of errors that I had to answer "y" to, and a large but smaller set the second time.  I then re-ran it with the "-y" option to tell it to fix everything and it finally came up clean enough for testing.

I was finally able to mount it and look around.  To my initial suprise, the home directory wasn't there, only a "lost+found" directory.  Checking there, were a few new files, and under there was my "dan" home directory.  I moved it to the correct location on the disk and rebooted my system.

WOW!  My system came up and I was able to login with my "dan" account.  Much more suprising, all of the videos and bigger files I have checked so far have been fine.

I think I dodged a bullet this time - I really need to look into a true RAID NAS solution for my home directory.

P.S. I know that a RAID system does NOT take the place of a true backup.  We do have an off-site backup solution that I use to keep important files in: JungleDisk with Amazon Simple Storage Service.  It's not the cheapest solution, but it encrypts with my private key BEFORE sending it out ot the Amazon servers, and I can use the program on as many systems as I need.  Highly recommended.

Mounting disks using UUID rather than device names.

The other day I reformatted my external USB hard drive to use as a backup device for my linux systems.  I was trying to come up with a single command to do the backup, but each system that I plugged the device into gave it a different device name.  One would call it "/dev/sdc", while another called it "/dev/sdb".  I wanted a consistent way to have it mounted at the same location each time so I could keep one backup script for all the systems.  Thankfully I'm far from the first person to have encountered this program, and the Linux community has come up with a solution.

That solution is based on the UUID - Universally Unique IDentifier - a string of 32 HEX characters that are unique in most cases.  For the most part, these will be unique - see the Wikipedia entry for a full stastical analysis.

In general, each filesystem has a UUID stamped into it when the filesystem is created.  These UUIDs look something like this when we view them:

d48416cc-f4d0-4455-8f8f-8ba0f69fbc11

The UUID assigned to each partition is visible by using the"blkid" command (found in the util-linux package).

I plugged in my USB drive with the EXT4 filesystem, then ran blkid and got output like this:

root@titan:~# blkid
/dev/sda1: UUID="e48416fc-f4d0-4455-8f8f-8ba0f69fbc11" TYPE="ext4"
/dev/sdb1: UUID="00a5db59-7cfe-4d6a-9237-92afaa64ca19" TYPE="ext3"
/dev/sdb5: UUID="40edf22c-a364-3068-84af-d2714cf98350" TYPE="ext4"
/dev/sdc1: UUID="f45eb6da-ca86-4837-afed-0fc4023b43f6" TYPE="ext4"

In my case, I wanted my external USB drive (/dev/sdc1 in the output above) to always be mounted in /mnt/backup.  If the system always adds this device as "/dev/sdc1", then this line in the /etc/fstab will mount it to "/mnt/backup" each time:

/dev/sdc1 /mnt/backup ext4 defaults 0 2

Or, to mount it from the command line:

mount /dev/sdc1 /mnt/backup

But, I want to automate my backup script including having it check and mount the drive at the proper location.  Since each system uses a different /dev/sdXX name, I'll have to use UUIDs.  The /etc/fstab entry is fairly simple:

UUID=f45eb6da-ca86-4837-afed-0fc4023b43f6 /mnt/backup ext4 defaults 0 2

And mounting from the command line is also similar:

mount /dev/disk/by-uuid/f45eb6da-ca86-4837-afed-0fc4023b43f6 /mnt/backup

Original notes: http://www.cyberciti.biz/faq/linux-finding-using-uuids-to-update-fstab/

Geezer-Geek and the Kids

On a mailing list I am on, the discussion turned to the new "Raspberry Pi" sub-$40 computer on a card.

It was mentioned that part of the reason it was produced was to give school kids something that was powerfull enough for them to play with and inexpensive enough to use in a classroom setting.  The old "back in my day" discussion came up - "We had to type in hundreds of lines of code from a magazine to get a stick-figure person to walk" and other such nostalga.

I tended to agree with that being that I cut my teeth on a Timex/Sinclair 1000 and remember having to learn skills debugging the magazine programs that have served me today (patience and careful observation being two key lessons).

At the same time I came across this article: Is raspberry pi a mid-life crisis?

The jist of this article is that we're old codgers, and our children are probably going to blaze just as new trails as we did.  Mostly because of the advanced tools they have now that were built on what we learned.

Hmm...not what I wanted to hear, but it made me thing about my experiences with tech as a kid and how it relates to my daughters.

 I've struggled trying to find a geek interest that I can share with my daughters (11 and 6), but so far their only interest in computers is to use them to view "Cute Kitten" YouTube videos and play some on-line games for school.

But, what I'm missing is that I was very much into the "hands-on technical" tinkering on anything (building carts, modifying my bike, building with legos, playing with my dads electronics tools, etc).  My 11 year old is showing an aptitude for science and literature - I need to change *myself* to find something in those fields that I can apply my technical interests in and show her how to use computers/technology to do the next great thing in those fields.

Now, I just need to find a way to make the Raspberry Pi help my daughter produce great literature while displaying videos of cute kittens...