P2V and grub bootloader

[Dennis Jacobfeuerborn]

Hi,
I'm currently experimenting with converting a physical system to a virtual 
one. Using losetup, kpartx, etc. i was able to transfer the system to a 
volume group in an image file.
My problem is that I don't know how to install grub on that image file. I 
can boot the VM with a rescue CD and then install it from there but I'd 
like to do this without having to boot the VM first. On the net I found 
some tutorials that mention mount-binding /dev to /new-root/dev, chrooting 
to /new-root and then doing a "grub-install /dev/sda" but wouldn't that 
overwrite the boot sector of my physical disk rather than the virtual one?
I tried doing a "grub-install /dev/loop0" (with loop0 beeing the device 
node for the image file) but there I only get "/dev/loop0 does not have any 
corresponding BIOS drive".
How can I install grub from outside the VM?

Regards,
   Dennis

[Richard W.M. Jones]

On Thu, Apr 30, 2009 at 11:28:51PM +0200, Dennis J. wrote:
>  Hi,
>  I'm currently experimenting with converting a physical system to a 
>  virtual one. Using losetup, kpartx, etc. i was able to transfer the 
>  system to a volume group in an image file. 
Why not just copy across the whole block device?  This is essentially
what virt-p2v does (http://et.redhat.com/~rjones/virt-p2v/), although
virt-p2v is doing nothing more than automating what you could do by
hand.

Anyway, if you copy the whole block device (eg /dev/sda) then
presumably grub will be copied across along with it.

[...]
>  How can I install grub from outside the VM? 
I haven't tried it, but it's possibly something you can do from
libguestfs, or if not, it's something that we could add to libguestfs.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Richard W.M. Jones]

On Thu, Apr 30, 2009 at 10:39:37PM +0100, Richard W.M. Jones wrote:
>  I haven't tried it, but it's possibly something you can do
> from
>  libguestfs, or if not, it's something that we could add to libguestfs. 
OK, I just added grub-install to libguestfs:

http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=b55bf8158f0b7f6...

(Make sure you have a back up of your guest image before experimenting
with this ...)

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Dennis Jacobfeuerborn]

On 05/01/2009 12:11 AM, Richard W.M. Jones wrote:
>  On Thu, Apr 30, 2009 at 10:39:37PM +0100, Richard W.M. Jones wrote:
> > I haven't tried it, but it's possibly something you can do from
> > libguestfs, or if not, it's something that we could add to libguestfs.
>
>  OK, I just added grub-install to libguestfs:
>
>
> http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=b55bf8158f0b7f6...
>
>  (Make sure you have a back up of your guest image before experimenting
>  with this ...) 
Looking at that patch wouldn't specifying /dev/sda as device actually 
overwrite the physical boot sector on the host system rather than the one 
in the image?
Or does libguestfs manipulate the device table so that access to /dev/sda 
get automatically gets re-routed to access the image instead?

Regards,
   Dennis

[Richard W.M. Jones]

On Fri, May 01, 2009 at 12:25:44AM +0200, Dennis J. wrote:
>  On 05/01/2009 12:11 AM, Richard W.M. Jones wrote:
> > On Thu, Apr 30, 2009 at 10:39:37PM +0100, Richard W.M. Jones wrote:
> >> I haven't tried it, but it's possibly something you can do from
> >> libguestfs, or if not, it's something that we could add to libguestfs.
> >
> > OK, I just added grub-install to libguestfs:
> >
> >
> http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=b55bf8158f0b7f6...
> >
> > (Make sure you have a back up of your guest image before experimenting
> > with this ...)
>
>  Looking at that patch wouldn't specifying /dev/sda as device actually  
>  overwrite the physical boot sector on the host system rather than the one 
>  in the image?
>  Or does libguestfs manipulate the device table so that access to /dev/sda 
>  get automatically gets re-routed to access the image instead? 
Neither.  libguestfs boots a mini virtual machine, so /dev/sda really
is the right device all along.

Example usage:

$ guestfish 
Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

> <fs> alloc /tmp/test 100M
> <fs> run 
qemu: loading initrd (0x157fd37 bytes) at 0x000000000f470000
> <fs> sfdisk /dev/sda 0 0 0 ,
> <fs> list-partitions  
/dev/sda1
> <fs> mkfs ext2 /dev/sda1
> <fs> mount /dev/sda1 /
> <fs> ll / 
total 13
drwxr-xr-x  3 root root  1024 Apr 30 18:28 .
drwxr-xr-x 19 root root     0 Apr 30 18:27 ..
drwx------  2 root root 12288 Apr 30 18:28 lost+found

> <fs> grub-install / /dev/sda
> <fs> ll /boot/grub/ 
total 236
drwxr-xr-x 2 root root   1024 Apr 30 18:28 .
drwxr-xr-x 3 root root   1024 Apr 30 18:28 ..
-rw-r--r-- 1 root root     30 Apr 30 18:28 device.map
-rw-r--r-- 1 root root  11736 Apr 30 18:28 e2fs_stage1_5
-rw-r--r-- 1 root root  11512 Apr 30 18:28 fat_stage1_5
-rw-r--r-- 1 root root  10744 Apr 30 18:28 ffs_stage1_5
-rw-r--r-- 1 root root  10736 Apr 30 18:28 iso9660_stage1_5
-rw-r--r-- 1 root root  12328 Apr 30 18:28 jfs_stage1_5
-rw-r--r-- 1 root root  10968 Apr 30 18:28 minix_stage1_5
-rw-r--r-- 1 root root  13360 Apr 30 18:28 reiserfs_stage1_5
-rw-r--r-- 1 root root    512 Apr 30 18:28 stage1
-rw-r--r-- 1 root root 111020 Apr 30 18:28 stage2
-rw-r--r-- 1 root root  11008 Apr 30 18:28 ufs2_stage1_5
-rw-r--r-- 1 root root  10344 Apr 30 18:28 vstafs_stage1_5
-rw-r--r-- 1 root root  12984 Apr 30 18:28 xfs_stage1_5

> <fs> exit 
$ ll /tmp/test
-rw-rw-r--. 1 rjones rjones 104857600 2009-04-30 23:28 /tmp/test
$ file /tmp/test
/tmp/test: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, 1st sector
stage2 0x25003, GRUB version 0.94; partition 1: ID=0x83, starthead 0, startsector 1,
192779 sectors, extended partition table (last)\011, code offset 0x48

Note that this does not require root privileges.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Dennis Jacobfeuerborn]

On 05/01/2009 12:30 AM, Richard W.M. Jones wrote:
>  On Fri, May 01, 2009 at 12:25:44AM +0200, Dennis J. wrote:
> > On 05/01/2009 12:11 AM, Richard W.M. Jones wrote:
> >> On Thu, Apr 30, 2009 at 10:39:37PM +0100, Richard W.M. Jones wrote:
> >>> I haven't tried it, but it's possibly something you can do from
> >>> libguestfs, or if not, it's something that we could add to libguestfs.
> >>
> >> OK, I just added grub-install to libguestfs:
> >>
> >>
> http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=b55bf8158f0b7f6...
> >>
> >> (Make sure you have a back up of your guest image before experimenting
> >> with this ...)
> >
> > Looking at that patch wouldn't specifying /dev/sda as device actually
> > overwrite the physical boot sector on the host system rather than the one
> > in the image?
> > Or does libguestfs manipulate the device table so that access to /dev/sda
> > get automatically gets re-routed to access the image instead?
>
>  Neither.  libguestfs boots a mini virtual machine, so /dev/sda really
>  is the right device all along. 
Ah, I didn't know that libguestfs actually uses its own vm.
I just tried compiling libguestfs from git on f10 but it doesn't seem to 
like that at all:

...
mv initramfs.fedora-10.i686.img initramfs.fedora-10.i686.img.bak
mv: cannot stat `initramfs.fedora-10.i686.img': No such file or directory
make[2]: [initramfs/fakeroot.log] Error 1 (ignored)
mv vmlinuz.fedora-10.i686 vmlinuz.fedora-10.i686.bak
mv: cannot stat `vmlinuz.fedora-10.i686': No such file or directory
make[2]: [initramfs/fakeroot.log] Error 1 (ignored)
if ! bash ./make-initramfs.sh; then rm -f initramfs/fakeroot.log; exit 1; fi
run_yum: line 1: 19874 Segmentation fault      yum -y -c 
"$tmpdir"/febootstrap.repo --disablerepo=* --enablerepo=febootstrap 
--noplugins --nogpgcheck --installroot="$target" install "$@"
chroot: cannot run command `chmod': No such file or directory
make[2]: *** [initramfs/fakeroot.log] Error 1
make[2]: Leaving directory `/home/dennis/Downloads/source/libguestfs-1.0.16'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/dennis/Downloads/source/libguestfs-1.0.16'
make: *** [all] Error 2

Regards,
   Dennis

[Richard W.M. Jones]

On Fri, May 01, 2009 at 02:42:27AM +0200, Dennis J. wrote:
>  Ah, I didn't know that libguestfs actually uses its own vm.
>  I just tried compiling libguestfs from git on f10 but it doesn't seem to  
>  like that at all:
>
>  ...
>  mv initramfs.fedora-10.i686.img initramfs.fedora-10.i686.img.bak
>  mv: cannot stat `initramfs.fedora-10.i686.img': No such file or directory
>  make[2]: [initramfs/fakeroot.log] Error 1 (ignored)
>  mv vmlinuz.fedora-10.i686 vmlinuz.fedora-10.i686.bak
>  mv: cannot stat `vmlinuz.fedora-10.i686': No such file or directory
>  make[2]: [initramfs/fakeroot.log] Error 1 (ignored)
>  if ! bash ./make-initramfs.sh; then rm -f initramfs/fakeroot.log; exit 1; fi
>  run_yum: line 1: 19874 Segmentation fault      yum -y -c  
>  "$tmpdir"/febootstrap.repo --disablerepo=* --enablerepo=febootstrap  
>  --noplugins --nogpgcheck --installroot="$target" install "$@"
>  chroot: cannot run command `chmod': No such file or directory
>  make[2]: *** [initramfs/fakeroot.log] Error 1
>  make[2]: Leaving directory `/home/dennis/Downloads/source/libguestfs-1.0.16'
>  make[1]: *** [all-recursive] Error 1
>  make[1]: Leaving directory `/home/dennis/Downloads/source/libguestfs-1.0.16'
>  make: *** [all] Error 2 
While it's possible to get libguestfs working on F10, it's not
our minimum target, which is F11.

In particular on F10 you will need to get the F11/F12 versions of the
following packages:

  febootstrap >= 1.7
  fakechroot >= 2.9-22
  fakeroot >= 1.12.2
  qemu >= 0.10-7

The following command should work _in theory_, but last time I
tried, it didn't because of the F11 freeze:

  # yum --enablerepo=rawhide install febootstrap fakechroot fakeroot qemu

so instead you need to get the packages from
http://koji.fedoraproject.org (type the package name in the box at the
top right).

You could also try:

  rpmbuild -ta libguestfs-1.0.17.tar.gz

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Emre Erenoglu]

On Fri, May 1, 2009 at 10:34 AM, Richard W.M. Jones <rjones(a)redhat.com&gt;wrote:

>  While it's possible to get libguestfs working on F10, it's not
>  our minimum target, which is F11.
>
>  In particular on F10 you will need to get the F11/F12 versions of the
>  following packages:
>
>   febootstrap >= 1.7
>   fakechroot >= 2.9-22
>   fakeroot >= 1.12.2
>   qemu >= 0.10-7
>  
I would like to use libguestfs in my distro, but is the dependency to
debootstrap mandatory? (as my distro is not Fedora?)
-- 
Emre

[Richard W.M. Jones]

On Fri, May 01, 2009 at 01:40:13PM +0200, Emre Erenoglu wrote:
>  On Fri, May 1, 2009 at 10:34 AM, Richard W.M. Jones
> <rjones(a)redhat.com&gt;wrote:
>  > While it's possible to get libguestfs working on F10, it's not
>  > our minimum target, which is F11.
>  >
>  > In particular on F10 you will need to get the F11/F12 versions of the
>  > following packages:
>  >
>  >  febootstrap >= 1.7
>  >  fakechroot >= 2.9-22
>  >  fakeroot >= 1.12.2
>  >  qemu >= 0.10-7
>  
>  I would like to use libguestfs in my distro, but is the dependency to
>  debootstrap mandatory? (as my distro is not Fedora?) 
Not sure if you misspelled 'febootstrap' there or if you really
mean debootstrap.

Anyhow it does need something like febootstrap/debootstrap,
ie. something which can build appliances as non-root.

So depending on your level of skill and free-software purity you have
several choices:

(1) Port febootstrap plus its deps to your distro.  Approximately
those deps would be: yum, python-rpm, fakeroot, fakechroot >= 2.9.

or:

(2) Port libguestfs to use debootstrap - if you're on Debian or Ubuntu
or another Debian derivative this would make most sense.  This is not
as much effort as it sounds, since febootstrap is loosely based on
debootstrap anyway.  Also because we have debootstrap in Fedora-land,
I can collaborate and test your patches.

or:

(3) Copy the binary blobs (the appliance) from /usr/lib64/guestfs in
the Fedora RPMs:

$ ll /usr/lib64/guestfs/
total 23968
-rw-r--r--. 1 root root 21869259 2009-04-30 20:35 initramfs.fedora-10.x86_64.img
-rw-r--r--. 1 root root  2637056 2009-04-30 20:35 vmlinuz.fedora-10.x86_64

Choice (3) limits your freedom to modify the code and may have GPL
implications, so check the licenses carefully.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

Libguestfs on Debian (was: Re: P2V and grub bootloader)

[Richard W.M. Jones]

On Fri, May 01, 2009 at 01:18:28PM +0100, Richard W.M. Jones wrote:
>  (1) Port febootstrap plus its deps to your distro.  Approximately
>  those deps would be: yum, python-rpm, fakeroot, fakechroot >= 2.9. 
So this is the approach I took, and libguestfs + febootstrap now
compiles and passes most of the tests on Debian.

I'll try to post some more detailed instructions later, or even get
the package into Debian, but for now let me just say that you do need
to use the * latest versions of everything from git *, and keep your
wits about you.

Also you can following postings on my blog: http://rwmj.wordpress.com/
and look at libguestfs bugs if you some specific issue:
https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=...

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Dennis Jacobfeuerborn]

On 05/01/2009 10:34 AM, Richard W.M. Jones wrote:
>  On Fri, May 01, 2009 at 02:42:27AM +0200, Dennis J. wrote:
> > Ah, I didn't know that libguestfs actually uses its own vm.
> > I just tried compiling libguestfs from git on f10 but it doesn't seem to
> > like that at all:
> >
> > ...
> > mv initramfs.fedora-10.i686.img initramfs.fedora-10.i686.img.bak
> > mv: cannot stat `initramfs.fedora-10.i686.img': No such file or directory
> > make[2]: [initramfs/fakeroot.log] Error 1 (ignored)
> > mv vmlinuz.fedora-10.i686 vmlinuz.fedora-10.i686.bak
> > mv: cannot stat `vmlinuz.fedora-10.i686': No such file or directory
> > make[2]: [initramfs/fakeroot.log] Error 1 (ignored)
> > if ! bash ./make-initramfs.sh; then rm -f initramfs/fakeroot.log; exit 1; fi
> > run_yum: line 1: 19874 Segmentation fault      yum -y -c
> > "$tmpdir"/febootstrap.repo --disablerepo=* --enablerepo=febootstrap
> > --noplugins --nogpgcheck --installroot="$target" install "$@"
> > chroot: cannot run command `chmod': No such file or directory
> > make[2]: *** [initramfs/fakeroot.log] Error 1
> > make[2]: Leaving directory `/home/dennis/Downloads/source/libguestfs-1.0.16'
> > make[1]: *** [all-recursive] Error 1
> > make[1]: Leaving directory `/home/dennis/Downloads/source/libguestfs-1.0.16'
> > make: *** [all] Error 2
>
>  While it's possible to get libguestfs working on F10, it's not
>  our minimum target, which is F11.
>
>  In particular on F10 you will need to get the F11/F12 versions of the
>  following packages:
>
>     febootstrap>= 1.7
>     fakechroot>= 2.9-22
>     fakeroot>= 1.12.2
>     qemu>= 0.10-7 
I've rebuilt all of those and installed them:

febootstrap-1.7-1.fc10.1.noarch
fakechroot-2.9-22.fc10.i386
fakeroot-1.12.2-21.fc10.i386
qemu-0.10-15.fc10.i386

Also I'm using the brand-new libguestfs-1.0.18-1.fc11.src.rpm to try to 
rebuild it for f10 but I still get fatal errors:

Lots of these:
/usr/bin/pod2html: inspector/virt-inspector.pl: unknown pod directive 
'encoding' in paragraph 5.  ignoring.
/usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve 
L<guestfish(1)> in paragraph 44.
/usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve 
L<guestfish(1)> in paragraph 69.
/usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve 
L<guestfs(3)> in paragraph 139.
...

Then I see this:
malloc: 4103338240: Cannot allocate memory
DB_ENV->set_lk_detect: unknown deadlock detection mode specified
DB_ENV->txn_stat_print interface requires an environment configured for the 
transaction subsystem
run_yum: line 1:  5326 Segmentation fault      yum -y -c 
"$tmpdir"/febootstrap.repo --disablerepo=* --enablerepo=febootstrap 
--noplugins --nogpgcheck --installroot="$target" install "$@"

and finally this:
chroot: cannot run command `chmod': No such file or directory

Regards,
   Dennis

[Richard W.M. Jones]

On Fri, May 01, 2009 at 02:34:22PM +0200, Dennis J. wrote:
>  I've rebuilt all of those and installed them:
>
>  febootstrap-1.7-1.fc10.1.noarch
>  fakechroot-2.9-22.fc10.i386
>  fakeroot-1.12.2-21.fc10.i386
>  qemu-0.10-15.fc10.i386
>
>  Also I'm using the brand-new libguestfs-1.0.18-1.fc11.src.rpm to try to  
>  rebuild it for f10 but I still get fatal errors:
>
>  Lots of these:
>  /usr/bin/pod2html: inspector/virt-inspector.pl: unknown pod directive  
>  'encoding' in paragraph 5.  ignoring.
>  /usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve  
>  L<guestfish(1)> in paragraph 44.
>  /usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve  
>  L<guestfish(1)> in paragraph 69.
>  /usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve  
>  L<guestfs(3)> in paragraph 139. 
Ignore these, they're just documentation warnings.

>  Then I see this:
>  malloc: 4103338240: Cannot allocate memory
>  DB_ENV->set_lk_detect: unknown deadlock detection mode specified
>  DB_ENV->txn_stat_print interface requires an environment configured for 
>  the transaction subsystem 
These seem to be from the Python BerkleyDB library.  Trying to
allocate 4 GB of memory, nice one Python.

I don't have a Fedora 10 i386 box to test against, but my suggestion
is to upgrade to the latest Yum (especially) and Python (if possible)
from F11/F12.

This is what I'm using on F11 i386:

$ rpm -q python yum
python-2.6-7.fc11.i586
yum-3.2.22-4.fc11.noarch

[...]
>  and finally this:
>  chroot: cannot run command `chmod': No such file or directory 
This is a consequence of the earlier failure of yum to do anything
useful.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Ján ONDREJ (SAL)]

On Fri, May 01, 2009 at 01:55:48PM +0100, Richard W.M. Jones wrote:
>  On Fri, May 01, 2009 at 02:34:22PM +0200, Dennis J. wrote:
>  > I've rebuilt all of those and installed them:
>  >
>  > febootstrap-1.7-1.fc10.1.noarch 
This one still has CDPATH problems:
  https://admin.fedoraproject.org/updates/febootstrap-1.7-1.fc10.1
Can you please update this?

>  > fakechroot-2.9-22.fc10.i386
>  > fakeroot-1.12.2-21.fc10.i386
>  > qemu-0.10-15.fc10.i386 
I have qemu-kvm-new copyed into ~/bin from qemu-system-x86-0.10-15 package,
because after update to qemu-0.10 on my Fedora 10, my virtual machines
corrupts images. Then:
  ./autogen.sh --with-java-home=no --with-qemu=qemu-kvm-new
And now it's working for me. You don't need whole qemu>0.10, just qemu-kvm
or qemu binary + another qemu installation.

>  > Then I see this:
>  > malloc: 4103338240: Cannot allocate memory
>  > DB_ENV->set_lk_detect: unknown deadlock detection mode specified
>  > DB_ENV->txn_stat_print interface requires an environment configured for 
>  > the transaction subsystem
>  
>  These seem to be from the Python BerkleyDB library.  Trying to
>  allocate 4 GB of memory, nice one Python. 
I have no similar problems on Fedora 10 i386.

>  I don't have a Fedora 10 i386 box to test against, but my
> suggestion
>  is to upgrade to the latest Yum (especially) and Python (if possible)
>  from F11/F12. 
May be you can install one as virtual machine. :)

		SAL

[Richard W.M. Jones]

On Fri, May 01, 2009 at 03:17:53PM +0200, Ján ONDREJ (SAL) wrote:
>  This one still has CDPATH problems:
>    https://admin.fedoraproject.org/updates/febootstrap-1.7-1.fc10.1
>  Can you please update this? 
I pushed a new version which should fix any problems.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 75 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

[Richard W.M. Jones]

I just uploaded some Fedora 11 / i586 binary RPMs to the site:

http://et.redhat.com/~rjones/libguestfs/files/

These _may_ work on Fedora 10, I haven't tried.  They will certainly
require qemu >= 0.10 from Fedora 11 though.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Dennis Jacobfeuerborn]

On 05/01/2009 03:49 PM, Richard W.M. Jones wrote:
>  I just uploaded some Fedora 11 / i586 binary RPMs to the site:
>
>  http://et.redhat.com/~rjones/libguestfs/files/
>
>  These _may_ work on Fedora 10, I haven't tried.  They will certainly
>  require qemu>= 0.10 from Fedora 11 though. 
This seems to work nicely for me on f10 so far, thanks!

The grub-install command still doesn't work though:

[root@nexus ~]# guestfish -a /var/lib/libvirt/images/test.img -m 
/dev/vg_mobile/lv_root
qemu: loading initrd (0x15b0c16 bytes) at 0x000000000b53f000

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
       'quit' to quit the shell

> <fs> cat /etc/redhat-release 
Fedora release 10.92
(Rawhide)

> <fs> grub-install / /dev/sda 
libguestfs: error:
grub-install: /dev/mapper/vg_mobile-lv_root does not 
have any corresponding BIOS drive.

Regards,
   Dennis

[Richard W.M. Jones]

On Fri, May 01, 2009 at 04:16:23PM +0200, Dennis J. wrote:
>  [root@nexus ~]# guestfish -a /var/lib/libvirt/images/test.img -m  
>  /dev/vg_mobile/lv_root 
[...]
>  ><fs> grub-install / /dev/sda
>  libguestfs: error: grub-install: /dev/mapper/vg_mobile-lv_root does not  
>  have any corresponding BIOS drive. 
Grub just doesn't work on LVs.  You have to install it on a 'real'
partition.  This is why Fedora uses a separate /boot partition.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Richard W.M. Jones]

Here's an example where I've used a separate /boot partition and root
is on an LV:

-------------------------------------------------- /tmp/test.sh 
#!/bin/sh -

guestfish <<EOF
alloc /tmp/test.img 500MB
run
sfdisk /dev/sda 0 0 0 ",10 ,"

echo Size of /dev/sda1:
blockdev-getsize64 /dev/sda1
echo Size of /dev/sda2:
blockdev-getsize64 /dev/sda2

mkfs ext2 /dev/sda1
pvcreate /dev/sda2
vgcreate VG /dev/sda2
lvcreate LV VG 400M
mkfs ext2 /dev/VG/LV

mount /dev/VG/LV /
mkdir /boot
mount /dev/sda1 /boot

grub-install / /dev/sda

echo Grub is installed on /dev/sda
ll /boot
ll /boot/grub
EOF

ls -l /tmp/test.img
file /tmp/test.img
--------------------------------------------------

$ time /tmp/test.sh 
qemu: loading initrd (0x15b0097 bytes) at 0x000000000b53f000
Size of /dev/sda1:
82252288
Size of /dev/sda2:
435939840
Grub is installed on /dev/sda
total 15
drwxr-xr-x 4 root root  1024 May  1 10:58 .
drwxr-xr-x 4 root root  1024 May  1 10:58 ..
drwxr-xr-x 2 root root  1024 May  1 10:58 grub
drwx------ 2 root root 12288 May  1 10:58 lost+found

total 236
drwxr-xr-x 2 root root   1024 May  1 10:58 .
drwxr-xr-x 4 root root   1024 May  1 10:58 ..
-rw-r--r-- 1 root root     30 May  1 10:58 device.map
-rw-r--r-- 1 root root  11736 May  1 10:58 e2fs_stage1_5
-rw-r--r-- 1 root root  11512 May  1 10:58 fat_stage1_5
-rw-r--r-- 1 root root  10744 May  1 10:58 ffs_stage1_5
-rw-r--r-- 1 root root  10736 May  1 10:58 iso9660_stage1_5
-rw-r--r-- 1 root root  12328 May  1 10:58 jfs_stage1_5
-rw-r--r-- 1 root root  10968 May  1 10:58 minix_stage1_5
-rw-r--r-- 1 root root  13360 May  1 10:58 reiserfs_stage1_5
-rw-r--r-- 1 root root    512 May  1 10:58 stage1
-rw-r--r-- 1 root root 111004 May  1 10:58 stage2
-rw-r--r-- 1 root root  11008 May  1 10:58 ufs2_stage1_5
-rw-r--r-- 1 root root  10344 May  1 10:58 vstafs_stage1_5
-rw-r--r-- 1 root root  12984 May  1 10:58 xfs_stage1_5

-rw-rw-r--. 1 rjones rjones 524288000 2009-05-01 15:58 /tmp/test.img
/tmp/test.img: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, 1st sector
stage2 0x4c03, GRUB version 0.94; partition 1: ID=0x83, starthead 0, startsector 1, 160649
sectors; partition 2: ID=0x83, starthead 0, startsector 160650, 851445 sectors, code
offset 0x48

real	       0m48.690s
user	       0m0.057s
sys	       0m3.109s


-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Dennis Jacobfeuerborn]

On 05/01/2009 05:36 PM, Richard W.M. Jones wrote:
>  Here's an example where I've used a separate /boot partition
> and root
>  is on an LV:
>
>  -------------------------------------------------- /tmp/test.sh
>  #!/bin/sh -
>
>  guestfish<<EOF
>  alloc /tmp/test.img 500MB
>  run
>  sfdisk /dev/sda 0 0 0 ",10 ,"
>
>  echo Size of /dev/sda1:
>  blockdev-getsize64 /dev/sda1
>  echo Size of /dev/sda2:
>  blockdev-getsize64 /dev/sda2
>
>  mkfs ext2 /dev/sda1
>  pvcreate /dev/sda2
>  vgcreate VG /dev/sda2
>  lvcreate LV VG 400M
>  mkfs ext2 /dev/VG/LV
>
>  mount /dev/VG/LV /
>  mkdir /boot
>  mount /dev/sda1 /boot
>
>  grub-install / /dev/sda
>  
I'm actually using the same setup but I forgot to mount the /boot 
partition. After doing that grub-install worked fine although I'm not sure 
why I got the BIOS error message simply because /boot wasn't mounted. Isn't 
the MBR written to the first sector of the disk and as a result independent 
of any particular partition? Why would the mounting of a partition have an 
influence on that? (Of course without the mount grub's boot files get 
written to the wrong place but that doesn't explain grub complaining about 
the device layout)

Regards,
   Dennis

[Richard W.M. Jones]

On Fri, May 01, 2009 at 08:12:23PM +0200, Dennis J. wrote:
>  I'm actually using the same setup but I forgot to mount the /boot
>  
>  partition. After doing that grub-install worked fine although I'm not 
>  sure why I got the BIOS error message simply because /boot wasn't 
>  mounted. Isn't the MBR written to the first sector of the disk and as a 
>  result independent of any particular partition? Why would the mounting of 
>  a partition have an influence on that? (Of course without the mount 
>  grub's boot files get written to the wrong place but that doesn't explain 
>  grub complaining about the device layout) 
Well this is a question for a grub mailing list, but the basic reason
is because the boot sector needs to know on which partition the first
stage is located.  The boot sector is very limited and has access to
just BIOS disks and a simple MBR-based partition decoder.  Thus the
first stage must be on an MBR partition (not an LV) on a BIOS disk --
/boot or /dev/sda1 in this example.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Dennis Jacobfeuerborn]

On 05/01/2009 02:55 PM, Richard W.M. Jones wrote:
>  On Fri, May 01, 2009 at 02:34:22PM +0200, Dennis J. wrote:
> > I've rebuilt all of those and installed them:
> >
> > febootstrap-1.7-1.fc10.1.noarch
> > fakechroot-2.9-22.fc10.i386
> > fakeroot-1.12.2-21.fc10.i386
> > qemu-0.10-15.fc10.i386
> >
> > Also I'm using the brand-new libguestfs-1.0.18-1.fc11.src.rpm to try to
> > rebuild it for f10 but I still get fatal errors:
> >
> > Lots of these:
> > /usr/bin/pod2html: inspector/virt-inspector.pl: unknown pod directive
> > 'encoding' in paragraph 5.  ignoring.
> > /usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve
> > L<guestfish(1)>  in paragraph 44.
> > /usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve
> > L<guestfish(1)>  in paragraph 69.
> > /usr/bin/pod2html: inspector/virt-inspector.pl: cannot resolve
> > L<guestfs(3)>  in paragraph 139.
>
>  Ignore these, they're just documentation warnings.
>
> > Then I see this:
> > malloc: 4103338240: Cannot allocate memory
> > DB_ENV->set_lk_detect: unknown deadlock detection mode specified
> > DB_ENV->txn_stat_print interface requires an environment configured for
> > the transaction subsystem
>
>  These seem to be from the Python BerkleyDB library.  Trying to
>  allocate 4 GB of memory, nice one Python.
>
>  I don't have a Fedora 10 i386 box to test against, but my suggestion
>  is to upgrade to the latest Yum (especially) and Python (if possible)
>  from F11/F12.
>
>  This is what I'm using on F11 i386:
>
>  $ rpm -q python yum
>  python-2.6-7.fc11.i586
>  yum-3.2.22-4.fc11.noarch 
I updated yum but still get the segfault but the malloc error didn't reappear.

Updating Python really isn't an option as that basically forces a 
distribution upgrade anyway due to its massive dependency tail.

In the line:
run_yum: line 1: 13054 Segmentation fault      yum -y -c 
"$tmpdir"/febootstrap.repo --disablerepo=* --enablerepo=febootstrap 
--noplugins --nogpgcheck --installroot="$target" install "$@"

what are the contents of "$tmpdir/febootstrap.repo"? I tried finding it in 
the build directory after the failure but (also run_yum) but I couldn't 
find them anywhere. I want to execute that command manually in order to 
track down the cause of the segfault.

Regards,
   Dennis

[Richard W.M. Jones]

On Fri, May 01, 2009 at 03:57:39PM +0200, Dennis J. wrote:
>  In the line:
>  run_yum: line 1: 13054 Segmentation fault      yum -y -c  
>  "$tmpdir"/febootstrap.repo --disablerepo=* --enablerepo=febootstrap  
>  --noplugins --nogpgcheck --installroot="$target" install "$@"
>
>  what are the contents of "$tmpdir/febootstrap.repo"? I tried finding it 
>  in the build directory after the failure but (also run_yum) but I 
>  couldn't find them anywhere. I want to execute that command manually in 
>  order to track down the cause of the segfault. 
Have a look at febootstrap (it's just a big shell script).

febootstrap.repo will contain:

  [febootstrap]
  name=febootstrap $repo $arch
  failovermethod=priority
  enabled=1
  gpgcheck=0
  $proxy

where $repo = fedora-10, $arch = i386 and $proxy will be empty in this
case.

'run_yum' is a function in the febootstrap shell script.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 75 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

[Dennis Jacobfeuerborn]

On 04/30/2009 11:39 PM, Richard W.M. Jones wrote:
>  On Thu, Apr 30, 2009 at 11:28:51PM +0200, Dennis J. wrote:
> > Hi,
> > I'm currently experimenting with converting a physical system to a
> > virtual one. Using losetup, kpartx, etc. i was able to transfer the
> > system to a volume group in an image file.
>
>  Why not just copy across the whole block device?  This is essentially
>  what virt-p2v does (http://et.redhat.com/~rjones/virt-p2v/), although
>  virt-p2v is doing nothing more than automating what you could do by
>  hand.
>
>  Anyway, if you copy the whole block device (eg /dev/sda) then
>  presumably grub will be copied across along with it. 
virt-p2v's achilles heel for me is that it can only copy on the block 
device level. /dev/sda is 50gb in size, the root logical volume only 10gb 
and copying the files over by piping a tar into ssh allowed me to shrink it 
to 5gb in the process. also I plan to move machines that use regular 
partitioning into VMs that use LVM volumes. Also copying the 3gb of files 
is obviously faster then copying 50gb of mostly empty space.

> > How can I install grub from outside the VM?
>
>  I haven't tried it, but it's possibly something you can do from
>  libguestfs, or if not, it's something that we could add to libguestfs. 
That would be great. What I'm a little confused about is the different 
perspectives of the device-space you get when simply chroot-ing into the 
VMs root filesystem and actually booting the VM. In the VM booting case 
/dev/sda is /dev/sda so a "grub-install /dev/sda" does the trick but in the 
chrooting case /dev/sda is still the physical /dev/sda (if I'm not making a 
mistake in my thinking here). That's why I thought I could "grub-install 
/dev/loop0" to treat the image file as the virtual /dev/sda.
Unfortunately that seems to confuse grub. I think I'll have to dive into 
the grub-install source to see why exactly it is complaining and how to get 
around that.

Regards,
   Dennis

[Richard W.M. Jones]

On Fri, May 01, 2009 at 12:16:29AM +0200, Dennis J. wrote:
>  On 04/30/2009 11:39 PM, Richard W.M. Jones wrote:
> > Why not just copy across the whole block device?  This is essentially
> > what virt-p2v does (http://et.redhat.com/~rjones/virt-p2v/), although
> > virt-p2v is doing nothing more than automating what you could do by
> > hand. 
[...]
>  virt-p2v's achilles heel for me is that it can only copy on the
> block  
>  device level. /dev/sda is 50gb in size, the root logical volume only 10gb 
>  and copying the files over by piping a tar into ssh allowed me to shrink 
>  it to 5gb in the process. also I plan to move machines that use regular  
>  partitioning into VMs that use LVM volumes. Also copying the 3gb of files 
>  is obviously faster then copying 50gb of mostly empty space. 
Right, and that's one of the reasons for writing libguestfs.  When
libguestfs is more mature I'll be able to rewrite virt-p2v to make it
do this.

> >> How can I install grub from outside the VM?
> >
> > I haven't tried it, but it's possibly something you can do from
> > libguestfs, or if not, it's something that we could add to libguestfs.
>
>  That would be great. What I'm a little confused about is the different  
>  perspectives of the device-space you get when simply chroot-ing into the  
>  VMs root filesystem and actually booting the VM. In the VM booting case  
>  /dev/sda is /dev/sda so a "grub-install /dev/sda" does the trick but in 
>  the chrooting case /dev/sda is still the physical /dev/sda (if I'm not 
>  making a mistake in my thinking here). That's why I thought I could 
>  "grub-install /dev/loop0" to treat the image file as the virtual 
>  /dev/sda.
>  Unfortunately that seems to confuse grub. I think I'll have to dive into  
>  the grub-install source to see why exactly it is complaining and how to 
>  get around that. 
It's highly doubtful that grub-install would work on a loop device.
Grub needs to query the kernel for the real CHS geometry of the disk,
and I don't think /dev/loop0 will give that information, only the real
/dev/sda device.

In any case, libguestfs boots a mini-VM, so this is a non-issue.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw