Data Recovery with KNOPPIX, RAID, and LVM

Say What?

Recently I had a power failure which, in lieu of my wonderful UPS / NUT setup, resulted in a hardware failure with one of my servers. Continuing my streak of horrific luck, my UPS gave out 3 seconds before the power came back on. Furthermore, I didn't tweak the remaining battery percentage before NUT sends the shutdown command to all my boxes so one of them (the most crucial one) didn't have enough time to power down. As a result, I now have my most important server no longer able to POST. Sweet.

It was inevitable I guess. The days of running a mini datacenter with business class internet are over - just ended a little premature. So how do I get the data off my server which no longer powers up?

What I Had

Before the death of the box, I was running a Xen host with 7 guests. These VMs were running off a software RAID 5 comprised of SATA drives with LVM on top of that. Each partition within the VM had its own logical volume. So, in order to recover data from this box, I needed:

A server which would actually POST (thankfully I had one lying around)
A server with at least 3 SATA ports
A KNOPPIX CD
Some free time

The Recovery

After I downloaded the latest KNOPPIX CD, it was time to boot up and start recovering some data. Unfortunately this wasn't a simple 'boot into KNOPPIX, mount the partition, call it a day' kind of ordeal. 2 things made this relatively complex: RAID and LVM. So ...

Note
Although above I speak of a RAID 5 of 3 SATA drives being the culprit, I first tested this out with the server that didn't die which contained a RAID 1 of 2 SATA drives. Thus these instructions were based on the 2 drive setup, not the one which I'll be performing the actual recovery with.

Boot into KNOPPIX.
Ensure all the devices are there. In my case, I ran something like
```
root@Microknoppix:~# ls -al /dev/sd*
```

Before we get to assembling the array, let's load some modules if they're not already loaded:

root@Microknoppix:~# lsmod | grep -i md
root@Microknoppix:~# modprobe md
root@Microknoppix:~# modprobe dm_mod

Now it's time to recreate the RAID. First, it might be helpful to use mdadm to read the RAID information.
```
root@Microknoppix:~# mdadm --examine --scan /dev/sd*
```
This will give you an idea of how the RAID was setup when it was originally running. Take note to the RAID device (i.e. /dev/md*).
If it doesn't already exist, create the device
```
root@Microknoppix:~# mknod /dev/md1 b 9 1
```

Now assemble the RAID array

root@Microknoppix:~# mdadm --assemble /dev/md1 /dev/sdb1 /dev/sdc1

Make sure the RAID looks legit
```
root@Microknoppix:~# cat /proc/mdstat
```
Now you should be able to see your logical volumes. Run some basic LVM commands to make sure things are in working order
```
root@Microknoppix:~# pvs
root@Microknoppix:~# vgs
root@Microknoppix:~# lvs
root@Microknoppix:~# lvscan
```
You might find that your logical volumes are inactive. To activate each logical volume, run
```
root@Microknoppix:~# lvchange -ay /dev/mapper/<volume_group>-<logical_volume>
```
Obviously being sure to replace volume_group and logical_volume with the appropriate names. If you are unsure, running lvscan will tell you any inactive volumes you may have. Simply plug in the full path of the volume from this output to the lvchange command and you should be in business.

Finally, mount the logical volume as you normally would.

root@Microknoppix:~# mkdir /mnt/tmp
root@Microknoppix:~# mount /dev/mapper/vg_xen_vms-lv_vm_splunk /mnt/tmp

Ta dah!

Data Recovery with KNOPPIX, RAID, and LVM

Say What?

What I Had

The Recovery

Labels

Comments (1)

Anonymous says:

Anonymous replies:

Help Tips

Examples

Description