|
h3. Say What?
|
... Recently I had a power failure which, in lieu of my wonderful UPS / NUT setup, resulted in a hardware failure with one of my servers. Continuing my streak of horrific luck, my UPS gave out 3 seconds before the power came back on. Furthermore, I didn't tweak the remaining battery percentage before NUT sends the shutdown command to all my boxes so one of them (the most crucial one) didn't have enough time to power down. As a result, I now have my most important server no longer able to POST. Sweet.
It was inevitable I guess. The days of running a mini datacenter with business class internet are over - just ended a little premature. So how do I get the data off my server which no longer powers up?
h3. What I Had
Before the death of the box, I was running a Xen host with 7 guests. These VMs were running off a software RAID 5 comprised of SATA drives with LVM on top of that. Each partition within the VM had its own logical volume. So, in order to recover data from this box, I needed:
# A server which would actually POST (thankfully I had one lying around)
# A server with at least 3 SATA ports
# A [KNOPPIX|http://knopper.net/knoppix-mirrors/index-en.html] CD
# Some free time
h3. The Recovery
|
After I downloaded the latest KNOPPIX CD, it was time to boot up and start recovering some data. Unfortunately this wasn't a simple 'boot into KNOPPIX, mount the partition, call it a day' kind of ordeal. 2 things made this relatively complex: RAID and LVM. So ...
|
{info:title=Note}
Although above I speak of a RAID 5 of 3 SATA drives being the culprit, I first tested this out with the server which didn't die which contained a RAID 1 of 2 SATA drives. Thus these instructions were based on the 2 drive setup, not the one which I'll be performing the actual recovery with.
{info}
|
# Boot into KNOPPIX.
# Ensure all the devices are there. In my case, I ran something like
|
... {code}
root@Microknoppix:~# ls -al /dev/sd*
{code}
# Before we get to assembling the array, let's load some modules if they're not already loaded:
{code}
root@Microknoppix:~# lsmod | grep -i md
root@Microknoppix:~# modprobe md
root@Microknoppix:~# modprobe dm_mod
{code}
# Now it's time to recreate the RAID. First, it might be helpful to use {{mdadm}} to read the RAID information.
{code}
root@Microknoppix:~# mdadm --examine --scan /dev/sd*
{code}
# This will give you an idea of how the RAID was setup when it was originally running. Take note to the RAID device (i.e. {{/dev/md*}}).
# If it doesn't already exist, create the device
{code}
root@Microknoppix:~# mknod /dev/md1 b 9 1
{code}
# Now assemble the RAID array
{code}
root@Microknoppix:~# mdadm --assemble /dev/md1 /dev/sdb1 /dev/sdc1
{code}
# Make sure the RAID looks legit
{code}
root@Microknoppix:~# cat /proc/mdstat
{code}
# Now you should be able to see your logical volumes. Run some basic LVM commands to make sure things are in working order
{code}
root@Microknoppix:~# pvs
root@Microknoppix:~# vgs
root@Microknoppix:~# lvs
root@Microknoppix:~# lvscan
{code}
# You might find that your logical volumes are inactive. To activate each logical volume, run
{code}
root@Microknoppix:~# lvchange -ay /dev/mapper/<volume_group>-<logical_volume>
{code}
Obviously being sure to replace {{volume_group}} and {{logical_volume}} with the appropriate names. If you are unsure, running {{lvscan}} will tell you any inactive volumes you may have. Simply plug in the full path of the volume from this output to the {{lvchange}} command and you should be in business.
# Finally, mount the logical volume as you normally would.
{code}
root@Microknoppix:~# mkdir /mnt/tmp
root@Microknoppix:~# mount /dev/mapper/vg_xen_vms-lv_vm_splunk /mnt/tmp
{code}
Ta dah!
|
