Recovering From Fracked or I Am Awesome

I get home late last night and the power is out. After a couple hours, the power finally comes back on. I turn on my machines, but something has gone wrong.

The webserver is down and the hard drive partition that houses everything is fracked.

Definition of "fracked" you ask?

It won't mount, and e2fsck won't fix it because the journal is on a superblock that "reads short" so I feel pretty much screwed. . .

But then I remember - I'm - freaking - awesome.

So I hook up the DVD drive and boot sysresccd. Try to e2fsck from there. Still no good. I'm still awesome, so I keep looking for a solution. Looks like

# tune2fs -O ^has_journal /dev/sda5

followed by e2fsck could help or could frag it. Don't want to do that without a backup of the drive as it is. . .

I decide that I'll pull the data off the bad drive into a file using dd_rescue. I hook up the external hard drive and start that process.

# dd_rescue /dev/sda5 sda5.img

Then I go to sleep. It's a long process.

When I wake up I run

# tune2fs -O ^has_journal sda5.img

and it does it's thing. Then I can run

# testdisk sda5.img

which lets me look through the disk image for alternate superblocks. I pick one and run

# e2fsck -b 32768 -y sda5.img

This checks the filesystem on the drive and answers "y" to any questions. Trying to autorepair the filesystem. It's another long process. I go do some other stuff.

When that's done, I try to mount the image as a loopback device

# mkdir /mnt/tempo
# mount -t ext2 -o loop sda5.img /mnt/tempo
# cd /mnt/tempo
# ls

HOLY CRAP! There's data there. Recovered data! I check through it and it's all looking good. I dance around. Then I start the next part of this journey.

# dd sda5.img /dev/sda5

That's another very-long-process. It puts the fixed data back on the partition.

After the fixed data is back on the partition I finally run

# e2fsck -c /dev/sda5

Which checks the partition for bad blocks and puts all the other stuff back in order.

Then I hit ctrl-alt-del and pray that there's a happy restart!

The system clicks and whirs and restarts. The little Ubuntu logo flashes by and. . . it starts running fsck automagically one more time. . . hmmm. . . how will this end?

Starting

Starting
Starting

WHOOOOOOOOOOOO!!!!!!!!!!!

As they say in Brazil, GOOOOOOOOOOOAAAAAAAAAAAAAAAALLLLLLLLLLLLLLLL!

I have recovered my system. The webserver is running. All is right in the world. . .

Oh and:

I - Am - Awesome.