lost my FC5 LVM volumes

[Joel Rees]

For the record, I was able to recover the volume group. I failed to  
copy the text of the shell session to a file, so this is from memory,  
but I'm pretty sure it's accurate enough to provide clues to anyone  
in a similar situation.

What I did was sacrifice the two base volumes hda2 and hda4 below.  
hda4 was converted to a large ext3 partition to hold what I needed  
out of hda2, and to hold the results of using photorec to recover  
what it could find of my files which had been stored physically in  
hde5. hda2 was then sacrificed and converted to an ext3 partition, to  
hold a single-partition minimal install of FC5.

While I was using photorec to recover my files, all 70,000+ of them  
scattered across 170 or so recup_xxx directories, I also used  
Christophe Grenier's testdisk (which photorec is distributed with) to  
see if it would find the lvm information in hda3 and hde5. It did,  
and I told it to write the results of scanning hda3 to hda3, or at  
least I think I did. I don't know if that step helped or not. Neither  
of those tools are what I would call verbose.

By refraining from using LVM in the minimal install in hda2, I think  
I was able to avoid further conflicting LVM headers, and was able to  
boot the install of FC6 in hde2 once or twice more to mount /dev/hda1  
and /dev/hda2, and get the name of the kernel and boot files in hda2  
and copy them into /boot/grub/grub.conf in hda1 so I could boot the  
FC5 in hda2. (I had just chained to grub on hde1 from grub in hda1,  
so that simplified things a bit.)

So I booted the new install of FC5 from hda2, re-read the man entries  
on lvm and its subcommands, pvdisplay was able to see both hda3 and  
hde5, so I took hope.

I mounted /dev/hda1, cd-ed into the old fc5's /etc/lvm and got the  
uuids of both physical volumes from backup/fc5 in there. I checked  
dates, and checked the uuids in the entries in the archive directory  
to make sure they matched, and that I wasn't going to end up trying  
to restore the wrong headers.

One point of possible confusion is that the uuids of the logical  
volumes which are recorded there won't help if the headers for the  
physical volumes have been written over. The original problem (before  
I screwed things up worse in my initial attempts at recovery) was  
that attempting to use logical volumes in installing ubuntu on hde2  
had overwritten the headers of hde5, and then lvm couldn't find all  
of the fc5 logical volume group and refused to mount any of it.

Then I did pvcreate --restorefile backup/fc5 --uuid <uuid-of-physical- 
hde5> /dev/hde5 . It first complained that it couldn't find the  
volume, and then it said volume created.

So then I did pvcreate --restorefile backup/fc5 --uuid <uuid-of-of- 
physical-hda3> /dev/hda3 with no complaints.

So I next did the vgcrestorefile --file backup/fc5 (I think that was  
the parameters, wish I had saved the session text) and that gave me a  
success message. So I checked the file system with the graphical lvm  
widget, and it saw all the lost logical volumes, including the one in  
hde5.

In order to boot from hda1 again, I had to get rid of the entries I  
had in /etc/fstab referencing the old fc4 leftovers in hda2 that were  
no longer there. But now it boots fine. I'll see if booting fc6  
works, and whether it screws things up, but only after backing up my  
important stuff from the old fc5 install.

Things I learned:

(1) Read the lvm howto at http://tldp.org/HOWTO/LVM-HOWTO/ before  
trying anything fancy like including physical volumes from more than  
one drive in a volume group.

(2) Don't multiboot other distros that use LVM if do mix physical  
drives in a volume group. (I guess this is kind of the same issue as,  
when multi-booting MSWxx and Linux or BSD distros, let MSWindwoes or  
Partition Magic, you only want to do the base partitions once and  
then leave them alone.)

(3) I didn't have much luck with using the LVM tools from one distro  
on another (Ubuntu live DVD and Knoppix CD). Probably not a good idea.

(4) I might want to consider not having /usr on an LVM volumes, even  
though Fedora Core installs all optional packages there instead of / 
usr/local. Part of the problem was that, with /usr in an LVM managed  
volume, I couldn't access the tools I needed to fix the LVM volumes.  
Alternatively, I might want to keep a live CD with the LVM tools on  
it. In fact, maybe I should make a feature request to put the LVM  
tools and man pages on the FC restore CDs. (The tools are incomplete,  
and the man pages not present on the current restore CDs.)

(0) Backup! Even my kids' experiments with the digital camera have  
more value than the cost of a few DVD-Rs and/or CD-Rs or CD-R/Ws. (I  
wonder if I'll really believe it this time. I thought I had learned  
this lesson when resizing the partitions for the sixth time trying to  
find an optimal balance for MSW2k and openbsd botched and left me  
backing up development work from the MSW2k partition with MSDOS  
through a floppy, but I guess the lesson didn't come home with me  
from work.)

On 2006/12/16, at 14:55, Joel Rees wrote:

> I was going to give Ubuntu a try and started an install. Since I  
> wanted to leave my FC5 intact and dual-boot, I was using the Ubuntu  
> LVM tools to set up LVM and walked on the FC5 LVM headers or  
> whatever they are called.
>
> My partition setup:
> hda1:	boot FC5 base partition
> hda2: 	LVM volume, leftovers of FC4 that I've been too lazy to take  
> a backup of, still intact
> hda3:	the lost LVM volumes of FC5, /usr et. al., except for one  
> volume on hde5
> hda4:	unused (freeBSD install currently sitting idle)
>
> hde1:	base partition, failed installing Ubuntu boot here, installed  
> FC6 instead for LVM tools
> hde2:	FC6 /usr et. al., in LVM (which I am now sure was a mistake)
> hde3 or hde4, I don't remember which:	DOS extended
> hde5:	LVM PV which was used as shared volume on FC5
> hde6:	swap
> hde7:	FAT32 share
>
> I think that where things went south was when I added another  
> volume group on hde using the Ubuntu LVM tools. I saw on
>
>     http://tldp.org/HOWTO/LVM-HOWTO/
>
> a few warning words to the effect that one shouldn't add physical  
> partitions (hde5) from a second drive to a volume group on one  
> drive (in hda3), and then set up a separate volume group (the  
> Ubuntu volumes) on the second drive. After partitioning, at the  
> step where Ubuntu would start actually installing, it complained  
> that LVM could no longer find the PV that had been in hde5 in the  
> FC5 volume group.
>
> I found this in a blog by one jalal, apparently of gnomedia.com  
> (now apparently only in google cache):
>
>> Here is what I did: First find out the old UID’s of the  
>> partitions, this is in the /etc/lvm/backup/system file. They are  
>> quite long… make sure you get the UID for the physical volumes.
>> $pvcreate -u sdSD-2343-SD939-adIda2 /dev/hda6
>> $pvcreate -u dk33kd-929293nd-adfja298a /dev/hdd1
>> $vgcreate -v system /dev/hda7 /dev/hdd1
>> $vgcfgrestore -f /etc/lvm/backup/system system
>>
>> and lo!, all data present and correct!
>>
>> In fact, I just rebooted the system and was back where I had  
>> started with the additional benefit of an extra 60Gb of disk  
>> space, because now I had the extra partition properly included.
>>
>> [Note: in the lines using pvcreate... above I could have used:
>> $ pvcreate –restorefile /etc/lvm/backup/system
>> to automatically find the ID’s but I hadn’t realized that at  
>> the time. Without the UID’s then the vgcfgrestore will not find  
>> the physical volumes that it needs to recreate the volume group.]
>
> I tried following that but forget the vgcfgrestore step, and  
> apparently have lost what headers remained. vgscan no longer sees  
> the FC5 volume group at all. I do still have FC5's /etc/lvm in the  
> boot partition (which will try to boot and fail, of course). (Kind  
> of wish the lvm commands had been under /sbin instead of /usr/sbin .)
>
> Any pointers where I can go to find information that might help me  
> recover the data?
>
> I'm thinking maybe I should install another FC5 in hda4, just  
> enough to use the command-line pvcreate tools, and all in one  
> partition, so I can scrub the FC6 install and get that other LVM  
> volume off of hde .