Slightly off-topic question about Xen+LVM

[Aaron Metzger]

First let me say that I love the way virtualization has been integrated 
into Fedora!  Thanks to all who have done this work.

I have a simple question which is a little off-topic but I know that the 
experience is contained in this group of folks.

Using the final release of Fedora 8 plus all released patches, I created 
an LVM volume group and an Ext3 logical volume upon which I am storing 
all of my Xen virtual machine images as normal files.

I tried to use the system-config-lvm tool to create a snapshot of the 
logical volume and assigned it to a mount point that I created under 
"/mnt/vmbackups" and checked the box to have it mounted at boot time.

When I rebooted the machine, the boot failed when It tried to access the 
snapshot logical volume.  I'm not going to include the specific errors 
here because my question is much more general.

Do snapshots of logical volumes work under Fedora 8?  Google revealed 
some historical discussions about snapshots not working in the 
transition from LVM1 to LVM2.  Is that the case?  Is anyone successful 
creating logical volume snapshots under Fedora 8?

Alternatively, is there any other reliable way to make a self-consistent 
copy of a running Xen virtual machine image for the case where the Xen 
virtual machine is held in a regular file system file?  Is LVM my only 
option?

Thanks in advance for any and all help.

--
Aaron

[Sam Folk-Williams]

Hi,

Aaron Metzger wrote:
> First let me say that I love the way virtualization has been integrated 
> into Fedora!  Thanks to all who have done this work.
>
> I have a simple question which is a little off-topic but I know that the 
> experience is contained in this group of folks.
>
> Using the final release of Fedora 8 plus all released patches, I created 
> an LVM volume group and an Ext3 logical volume upon which I am storing 
> all of my Xen virtual machine images as normal files.
>
> I tried to use the system-config-lvm tool to create a snapshot of the 
> logical volume and assigned it to a mount point that I created under 
> "/mnt/vmbackups" and checked the box to have it mounted at boot time.
>
> When I rebooted the machine, the boot failed when It tried to access the 
> snapshot logical volume.  I'm not going to include the specific errors 
> here because my question is much more general.
>
> Do snapshots of logical volumes work under Fedora 8?  Google revealed 
> some historical discussions about snapshots not working in the 
> transition from LVM1 to LVM2.  Is that the case?  Is anyone successful 
> creating logical volume snapshots under Fedora 8?

They definitely work. Thought there might be a better approach. For 
example, the simplest way is to make a separate LV for each quest. Then 
you make a snapshot of that LV to "clone" that guest, or to save it as a 
backup.
> Alternatively, is there any other reliable way to make a self-consistent 
> copy of a running Xen virtual machine image for the case where the Xen 
> virtual machine is held in a regular file system file?  Is LVM my only 
> option?

For a single file you can just copy the file... with, say, cp or dd.

-Sam
> Thanks in advance for any and all help.
>
> -- 
> Aaron
>
> -- 
> Fedora-xen mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-xen

[Aaron Metzger]

Sam Folk-Williams wrote:
> > Alternatively, is there any other reliable way to make a 
> > self-consistent copy of a running Xen virtual machine image for the 
> > case where the Xen virtual machine is held in a regular file system 
> > file?  Is LVM my only option?
>
> For a single file you can just copy the file... with, say, cp or dd.

Thanks for the reply, but I thought that this is not a valid thing to do 
for a RUNNING guest.  I think that the file will be changing too rapidly 
(guest machine swap etc) to get a self-consistent copy using just "cp" 
or "dd".  Right?

--
Aaron M.

[Markus Armbruster]

Aaron Metzger  writes:
> Sam Folk-Williams wrote:
> > > Alternatively, is there any other reliable way to make a
> > > self-consistent copy of a running Xen virtual machine image for the
> > > case where the Xen virtual machine is held in a regular file system
> > > file?  Is LVM my only option?
> >
> > For a single file you can just copy the file... with, say, cp or dd.
>
> Thanks for the reply, but I thought that this is not a valid thing to
> do for a RUNNING guest.  I think that the file will be changing too
> rapidly (guest machine swap etc) to get a self-consistent copy using
> just "cp" or "dd".  Right?

Right.

[Stephen C. Tweedie]

Hi,

On Tue, 2007-11-20 at 13:55 -0500, Aaron Metzger wrote:
> I tried to use the system-config-lvm tool to create a snapshot of the 
> logical volume and assigned it to a mount point that I created under 
> "/mnt/vmbackups" and checked the box to have it mounted at boot time.

OK, that should work... *IF* you are careful about some of the gotchas
concerning snapshots.

Namely... if you're mounting them, be aware that the original filesystem
and its snapshot will both have the same UUID and filesystem label.  So
if you try to mount-by-label (eg. "LABEL=foo" in fstab), you can't
guarantee which one will be picked up.

Now, filesystems set up by anaconda on LVM should be specified
in /etc/fstab by their LVM path, not by their label.  But if you have
set up the fstab entry yourself by label, then you might well confuse
the system.
> When I rebooted the machine, the boot failed when It tried to access the 
> snapshot logical volume.  I'm not going to include the specific errors 
> here because my question is much more general.

Well, in general it should work.. we'd need to see the error to know if
there's a special-case you've fallen into.

Cheers,
 Stephen

[Aaron Metzger]

Stephen C. Tweedie wrote:
> Hi,
> Well, in general it should work.. we'd need to see the error to know if
> there's a special-case you've fallen into.

A follow-up to describe what I have found regarding Fedora 8 and my 
attempts to use LVM snapshots to hold Xen guest images.  Maybe it will 
help others.

In general, I was unable to create any LVM snapshot with an out of the 
box configuration of a 64 bit Fedora 8 until I manually loaded the 
"dm-snapshot" kernel module.

modprobe dm-snapshot

Then, I was able to create a snapshot volume and all seemed fine until I 
rebooted the machine.  The boot sequence failed with a message about 
"unable to expand snapshot volume" which I assume was caused again by 
the lack of the "dm-snapshot" module.  I used the rescue disk to remove 
the snapshot volume and was able to boot my machine once again.

I found a number of postings on the debian mailing lists from 2005 
regarding the same issue and peoples attempts to get the "dm-snapshot" 
module to load during boot.

I played around with modules.conf.d configuration to try to get the 
module to load on boot but didn't have any success.

Do I need to make a custom boot/initrd script or am I on the wrong 
track?  Ideas are welcome.

In the mean time, I have resigned myself to just not ever rebooting the 
machine while any snapshot volumes exist.

--
Aaron

[Dale Bewley]

> Do I need to make a custom boot/initrd script or am I on the wrong 
> track?  Ideas are welcome.

Does your initrd contain dm-snapshot.ko?

# cat initrd-2.6.21-2950.fc8xen.img | gunzip | cpio -vt | grep snapshot
-rw-------   1 root     root        34608 Nov 27 08:43 lib/dm-snapshot.ko

--
Dale Bewley - Unix Administrator - Shields Library - UC Davis
GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD  1753 064D 2583 B098 A0F3

[Aaron Metzger]

Dale Bewley wrote:
> > Do I need to make a custom boot/initrd script or am I on the wrong 
> > track?  Ideas are welcome.
>
> Does your initrd contain dm-snapshot.ko?
>
> # cat initrd-2.6.21-2950.fc8xen.img | gunzip | cpio -vt | grep snapshot
> -rw-------   1 root     root        34608 Nov 27 08:43 lib/dm-snapshot.ko

Dale:

No, it does not.

Is the solution as simple as adding it?
If so, what is the procedure to correctly create/build a custom initrd?

Thanks,
Aaron

[Richard W.M. Jones]

Aaron Metzger wrote:
> Dale Bewley wrote:
> > > Do I need to make a custom boot/initrd script or am I on the wrong 
> > > track?  Ideas are welcome.
> >
> > Does your initrd contain dm-snapshot.ko?
> >
> > # cat initrd-2.6.21-2950.fc8xen.img | gunzip | cpio -vt | grep snapshot
> > -rw-------   1 root     root        34608 Nov 27 08:43 lib/dm-snapshot.ko
>
> Dale:
>
> No, it does not.
>
> Is the solution as simple as adding it?
> If so, what is the procedure to correctly create/build a custom initrd?

"initrd" is built for you when you install the kernel package.  The step 
  by step process goes something like this:

(1) When the kernel RPM is installed, an RPM script runs (see 'rpm -q 
--scripts kernel'):

/sbin/new-kernel-pkg --package kernel --mkinitrd --depmod --install 
$version

(2) new-kernel-pkg is a shell script which amongst other things runs 
mkinitrd:

/sbin/mkinitrd --allow-missing -f $initrdfile $version

(3) mkinitrd is also a shell script which probes the current machine and 
decides which drivers the machine needs to boot and mount the root 
filesystem, and combines those drivers and a big shell script together 
into an initrd image.  (Which is a kind of ramdisk available before the 
kernel has finished booting).

The whole thing is a bit crap to be honest ...

Anyway, if you read the mkinitrd script you'll see that if a dm module 
other than 'linear' is used in some dmtable entry (see 'dmsetup table' 
command) then mkinitrd will include the snapshot module, in case it's 
being used.  Actually, on my system mkinitrd appears to be broken so 
that it never probes for anything in the dmtable at all, so maybe 
there's a real bug going on here.

You can run new-kernel-pkg at any time to rebuild initrd, and the 
details above should tell you what's going on.  Do file a bugzilla if 
there is a real problem in mkinitrd script.

Rich.

[Dale Bewley]

----- "Aaron Metzger"  wrote:
> Dale Bewley wrote:
> > > Do I need to make a custom boot/initrd script or am I on the wrong
> > > track?  Ideas are welcome.
> >
> > Does your initrd contain dm-snapshot.ko?
> >
> > # cat initrd-2.6.21-2950.fc8xen.img | gunzip | cpio -vt | grep
>
> snapshot
> > -rw-------   1 root     root        34608 Nov 27 08:43
>
> lib/dm-snapshot.ko
> >
>
> Dale:
>
> No, it does not.
>
> Is the solution as simple as adding it?
> If so, what is the procedure to correctly create/build a custom
> initrd?

If mkinitrd doesn't include what you want (in my case it doesn't know to include raid1 and raid456 modules) you can include it by hand. Example:

VER=`uname -r`
INITRD=/boot/initrd-${VER}.img
if [ -f $INITRD ]; then
    mv $INITRD $INITRD.bak
    mkinitrd --with=raid1 --with=raid456 $INITRD $VER
else
    echo "Can not find $INITRD"
    echo "Are you really fixing kernel ${VER}?"
fi

--
Dale Bewley - Unix Administrator - Shields Library - UC Davis
GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD  1753 064D 2583 B098 A0F3

[Aaron Metzger]

Dale Bewley wrote:
> If mkinitrd doesn't include what you want (in my case it doesn't know to include raid1 and raid456 modules) you can include it by hand. Example:
>
> VER=`uname -r`
> INITRD=/boot/initrd-${VER}.img
> if [ -f $INITRD ]; then
>    mv $INITRD $INITRD.bak
>    mkinitrd --with=raid1 --with=raid456 $INITRD $VER
> else
>    echo "Can not find $INITRD"
>    echo "Are you really fixing kernel ${VER}?"
> fi

Dale:

Your instructions worked perfectly to allow me to add "dm-snapshot" to 
my initrd.

Thank you very much for your insight.  You made my day.

--
Aaron