info on save function in virt-manager

[Gianluca Cecchi]

Is the save function in virt-manager for a vm equivalent to these steps:

1) pause vm (for example in virt-manager gui)
2) save disk image to file
3) run vm again
?
Is any mem state preserved in this save operation?

Can we think it as crash-consistent image ot the VM (aka equivalent of
rebooting a server after a hard shutdown)?
What exactly happens when you do a restore of a saved VM?

What could they be differences between a save operation and this one below:

- imagine VM disk is an LV
- pause VM
- do an lvm snapshot of the LVs that compose the VM
- exit VM from pause state (unchek inside virt-manager gui)
- backup the snapshot LVs

I think this latest is crash-consistent, correct?

Thanks for your comments.

Gianluca

[Chris Lalancette]

Gianluca Cecchi wrote:
>  Is the save function in virt-manager for a vm equivalent to these
> steps:
>  
>  1) pause vm (for example in virt-manager gui)
>  2) save disk image to file
>  3) run vm again
>  ?
>  Is any mem state preserved in this save operation? 
It's equivalent to steps 1) and 2), followed by a kill of the paused VM.  And
yes, the memory state is preserved.  On a restore, the memory state is restored,
and the VM is set running again.

The idea behind it is that during host shutdown (or other times), you want to
preserve the state of your guests and then resume them from where they were.

Note very carefully that this is *not* the same as a full snapshot; if you save
a guest, and then restore it multiple times from the same save file, you will
certainly corrupt your disk image.  That is, a save does not take an "snapshot"
of the underlying storage at the same time.  That functionality is planned, but
really needs a new API (and cooperation from the underlying qemu bits) to work
properly.

-- 
Chris Lalancette

[Gianluca Cecchi]

On Mon, Sep 21, 2009 at 8:28 AM, Chris Lalancette <clalance(a)redhat.com&gt;wrote:

>  Gianluca Cecchi wrote:
>  > Is the save function in virt-manager for a vm equivalent to these steps:
>  >
>  > 1) pause vm (for example in virt-manager gui)
>  > 2) save disk image to file
>  > 3) run vm again
>  > ?
>  > Is any mem state preserved in this save operation?
>
>  It's equivalent to steps 1) and 2), followed by a kill of the paused VM.
>   And
>  yes, the memory state is preserved.  On a restore, the memory state is
>  restored,
>  and the VM is set running again.
>
>
>  
And is it supposed to work at this time on F11 + fedora-virt-preview repo?
Because I get the window saying "operation in progress" but it seems
blocked.

In fact also a "virsh list" command at this point remains pending.....

The process seems to have only being written about 1400 bytes of the saved
file:
-rw------- 1 root root     1400 2009-09-21 17:46 save_vm_prova2

An fuser command on that file gives nothing.
An strace on virt-manager process gives:
[root@virtfedbis tmp]# strace -p 648
Process 648 attached - interrupt to quit
futex(0x2ff705c, FUTEX_WAIT_PRIVATE, 1, NULL

The Vm is no more reachable and was doing almost nothing when the "save"
operation was started.
It seems nothing relevant in messages or virt-manager.log or vm log
prova2.log....

If it is supposed to work I'm going to open a bug...

Thanks,
Gianluca

[Chris Lalancette]

Gianluca Cecchi wrote:
>  And is it supposed to work at this time on F11 + fedora-virt-preview
> repo?
>  Because I get the window saying "operation in progress" but it seems
>  blocked. 
Yeah, I think there's a bug currently in the code.  I believe markmc has
recently ported that patch back, though, so upgrading to a newer version (when
he pushes it) should fix the bug.

-- 
Chris Lalancette

[Gianluca Cecchi]

On Tue, Sep 22, 2009 at 8:28 AM, Chris Lalancette <clalance(a)redhat.com&gt;wrote:

>  Gianluca Cecchi wrote:
>  > And is it supposed to work at this time on F11 + fedora-virt-preview
>  repo?
>  > Because I get the window saying "operation in progress" but it seems
>  > blocked.
>
>  Yeah, I think there's a bug currently in the code.  I believe markmc has
>  recently ported that patch back, though, so upgrading to a newer version
>  (when
>  he pushes it) should fix the bug.
>
>  --
>  Chris Lalancette
>  
OK, I'll wait for the fix and then I test.
In the mean time, even when fixed, this implies a shutdown of the VM....
Can the other kind of approach considered crash-consistent?

The one involving:
- VM disks are LVs of the host
- pause VM
- do an lvm snapshot of the LVs that compose the VM
- exit VM from pause state (unchek inside virt-manager gui)
- backup the snapshot LVs

Thanks,
Gianluca

[Chris Lalancette]

Gianluca Cecchi wrote:
>  OK, I'll wait for the fix and then I test.
>  In the mean time, even when fixed, this implies a shutdown of the VM....
>  Can the other kind of approach considered crash-consistent?
>  
>  The one involving:
>  - VM disks are LVs of the host
>  - pause VM
>  - do an lvm snapshot of the LVs that compose the VM
>  - exit VM from pause state (unchek inside virt-manager gui)
>  - backup the snapshot LVs 
Yes, there was a proposal a while ago for an new API to do this, but it hasn't
been implemented yet.  It's also a bit tricky; while we can do the snapshot like
thing on qcow2 and LVM disks, you can't do it on raw files or partitions, for
instance.  And even then, LVM has limitations with the number of snapshots you
can make.

So while it can be done, it will probably be a bit of time before we have
something implemented.

-- 
Chris Lalancette

[Mark McLoughlin]

On Tue, 2009-09-22 at 10:05 +0200, Gianluca Cecchi wrote:
>  > Yeah, I think there's a bug currently in the code.  I
> believe markmc
>  has
>  > recently ported that patch back, though, so upgrading to a newer
>  version
>  > (when
>  > he pushes it) should fix the bug.
>  >
>  > --
>  > Chris Lalancette
>  >
>  
>  OK, I'll wait for the fix and then I test. 
I think the fix Chris means is in libvirt-0.7.1-5

Cheers,
Mark.

[Gianluca Cecchi]

On Tue, Sep 22, 2009 at 3:00 PM, Mark McLoughlin <markmc(a)redhat.com&gt; wrote:

>  On Tue, 2009-09-22 at 10:05 +0200, Gianluca Cecchi wrote:
>  > > Yeah, I think there's a bug currently in the code.  I believe markmc
>  > has
>  > > recently ported that patch back, though, so upgrading to a newer
>  > version
>  > > (when
>  > > he pushes it) should fix the bug.
>  > >
>  > > --
>  > > Chris Lalancette
>  > >
>  >
>  > OK, I'll wait for the fix and then I test.
>
>  I think the fix Chris means is in libvirt-0.7.1-5
>
>  Cheers,
>  Mark.
>
>  I confirm that with that version of libvirt I'm able to save my VM (or at 
least
the command completes, not tried yet to restore '-)
Inside the data file saved I can see that there is also the xml config for
the guest embedded ....
As the VM is killed at the end of the save, can I then
- copy the disk(s) of the VM to another host
- copy the save file to another host
- restore the save on that secondary host based on the fact that it retains
the same config?

At this time, trying to restore on another host that has not the same
storage poool config gives this:
Error restoring domain '/mnt/rhel54_x86_64': internal error unable to start
guest: qemu: could not open disk image /dev/vg_qemu01/rhel53_64

and in details window:
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/manager.py", line 458, in
restore_saved_callback
    newconn.restore(file_to_load)
  File "/usr/share/virt-manager/virtManager/connection.py", line 642, in
restore
    self.vmm.restore(frm)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1420, in
restore
    if ret == -1: raise libvirtError ('virDomainRestore() failed',
conn=self)
libvirtError: internal error unable to start guest: qemu: could not open
disk image /dev/vg_qemu01/rhel53_64

Supposing I recreate a pool with the same pointers and the hard disk of the
VM, are there other limitations/restrictions to restore on another host?

[Gianluca Cecchi]

On Thu, Sep 24, 2009 at 5:42 PM, gianluca.cecchi
<gianluca.cecchi(a)gmail.com&gt;wrote:

>  On Tue, Sep 22, 2009 at 3:00 PM, Mark McLoughlin
> <markmc(a)redhat.com&gt;wrote:
>
> > On Tue, 2009-09-22 at 10:05 +0200, Gianluca Cecchi wrote:
> > > > Yeah, I think there's a bug currently in the code.  I believe markmc
> > > has
> > > > recently ported that patch back, though, so upgrading to a newer
> > > version
> > > > (when
> > > > he pushes it) should fix the bug.
> > > >
> > > > --
> > > > Chris Lalancette
> > > >
> > >
> > > OK, I'll wait for the fix and then I test.
> >
> > I think the fix Chris means is in libvirt-0.7.1-5
> >
> > Cheers,
> > Mark.
> >
> > I confirm that with that version of libvirt I'm able to save my VM (or at
>  least the command completes, not tried yet to restore '-)
>  Inside the data file saved I can see that there is also the xml config for
>  the guest embedded ....
>  As the VM is killed at the end of the save, can I then
>  - copy the disk(s) of the VM to another host
>  - copy the save file to another host
>  - restore the save on that secondary host based on the fact that it retains
>  the same config?
>
>  At this time, trying to restore on another host that has not the same
>  storage poool config gives this:
>  Error restoring domain '/mnt/rhel54_x86_64': internal error unable to start
>  guest: qemu: could not open disk image /dev/vg_qemu01/rhel53_64
>
>  and in details window:
>  Traceback (most recent call last):
>    File "/usr/share/virt-manager/virtManager/manager.py", line 458, in
>  restore_saved_callback
>      newconn.restore(file_to_load)
>    File "/usr/share/virt-manager/virtManager/connection.py", line 642, in
>  restore
>      self.vmm.restore(frm)
>    File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1420, in
>  restore
>      if ret == -1: raise libvirtError ('virDomainRestore() failed',
>  conn=self)
>  libvirtError: internal error unable to start guest: qemu: could not open
>  disk image /dev/vg_qemu01/rhel53_64
>
>  Supposing I recreate a pool with the same pointers and the hard disk of the
>  VM, are there other limitations/restrictions to restore on another host?
>
>  
BTW: the restore on the same host (with the VM still stopped as at the end
of the save), doesn't work either. I get:
Error restoring domain '/mnt/rhel54_240909_1750.save': Unable to read QEMU
help output: Interrupted system call

And inside the details window:
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/manager.py", line 458, in
restore_saved_callback
    newconn.restore(file_to_load)
  File "/usr/share/virt-manager/virtManager/connection.py", line 642, in
restore
    self.vmm.restore(frm)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1420, in
restore
    if ret == -1: raise libvirtError ('virDomainRestore() failed',
conn=self)
libvirtError: Unable to read QEMU help output: Interrupted system call

[root@virtfedbis qemu]# ll /mnt/rhel54_240909_1750.save
-rw------- 1 root root 334221504 2009-09-24 17:51
/mnt/rhel54_240909_1750.save

The VM has an hd of 8Gb of size and 4Gb of ram.

If I want to debug setting LIBVIRTD_DEBUG=1 and then run the restore command
via command line, what would it be, so that I can bugzilla this (if not
already there)?

Inside /var/log/messages I have this:
Sep 24 17:51:51 virtfedbis kernel: libvirtd[25820]: segfault at 0 ip
00000039762a4822 sp 00007f20a7546880 error 4 in libc-2.10.1.so
[3976200000+164000]
Sep 24 17:51:51 virtfedbis libvirtd: 17:51:51.379: error :
qemudExtractVersionInfo:1032 : Unable to read QEMU help output: Interrupted
system call

In virt-manager.log I have the same messages as in the virt-manager window.
My packages releases are:
kernel 2.6.30.5-43.fc11.x86_64
[root@virtfedbis qemu]# rpm -qa libvirt* qemu*
qemu-system-x86-0.10.92-5.fc11.x86_64
qemu-img-0.10.92-5.fc11.x86_64
qemu-kvm-0.10.92-5.fc11.x86_64
libvirt-0.7.1-5.fc11.x86_64
libvirt-python-0.7.1-5.fc11.x86_64
libvirt-client-0.7.1-5.fc11.x86_64
qemu-common-0.10.92-5.fc11.x86_64

Thanks,
Gianluca

[Chris Lalancette]

gianluca.cecchi wrote:
>  On Thu, Sep 24, 2009 at 5:42 PM, gianluca.cecchi
>  <gianluca.cecchi(a)gmail.com <mailto:[e-mail]>> wrote:
>  
>      On Tue, Sep 22, 2009 at 3:00 PM, Mark McLoughlin <markmc(a)redhat.com
>      <mailto:[e-mail]>> wrote:
>  
>          On Tue, 2009-09-22 at 10:05 +0200, Gianluca Cecchi wrote:
>          > > Yeah, I think there's a bug currently in the code.  I
>          believe markmc
>          > has
>          > > recently ported that patch back, though, so upgrading to a newer
>          > version
>          > > (when
>          > > he pushes it) should fix the bug.
>          > >
>          > > --
>          > > Chris Lalancette
>          > >
>          >
>          > OK, I'll wait for the fix and then I test.
>  
>          I think the fix Chris means is in libvirt-0.7.1-5
>  
>          Cheers,
>          Mark.
>  
>      I confirm that with that version of libvirt I'm able to save my VM
>      (or at least the command completes, not tried yet to restore '-)
>      Inside the data file saved I can see that there is also the xml
>      config for the guest embedded ....
>      As the VM is killed at the end of the save, can I then
>      - copy the disk(s) of the VM to another host
>      - copy the save file to another host
>      - restore the save on that secondary host based on the fact that it
>      retains the same config?
>  
>      At this time, trying to restore on another host that has not the
>      same storage poool config gives this:
>      Error restoring domain '/mnt/rhel54_x86_64': internal error unable
>      to start guest: qemu: could not open disk image /dev/vg_qemu01/rhel53_64
>  
>      and in details window:
>      Traceback (most recent call last):
>        File "/usr/share/virt-manager/virtManager/manager.py", line 458,
>      in restore_saved_callback
>          newconn.restore(file_to_load)
>        File "/usr/share/virt-manager/virtManager/connection.py", line
>      642, in restore
>          self.vmm.restore(frm)
>        File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1420,
>      in restore
>          if ret == -1: raise libvirtError ('virDomainRestore() failed',
>      conn=self)
>      libvirtError: internal error unable to start guest: qemu: could not
>      open disk image /dev/vg_qemu01/rhel53_64
>  
>      Supposing I recreate a pool with the same pointers and the hard disk
>      of the VM, are there other limitations/restrictions to restore on
>      another host?
>  
>  
>  BTW: the restore on the same host (with the VM still stopped as at the
>  end of the save), doesn't work either. I get:
>  Error restoring domain '/mnt/rhel54_240909_1750.save': Unable to read
>  QEMU help output: Interrupted system call
>  
>  And inside the details window:
>  Traceback (most recent call last):
>    File "/usr/share/virt-manager/virtManager/manager.py", line 458, in
>  restore_saved_callback
>      newconn.restore(file_to_load)
>    File "/usr/share/virt-manager/virtManager/connection.py", line 642, in
>  restore
>      self.vmm.restore(frm)
>    File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1420, in
>  restore
>      if ret == -1: raise libvirtError ('virDomainRestore() failed',
>  conn=self)
>  libvirtError: Unable to read QEMU help output: Interrupted system call
>  
>  [root@virtfedbis qemu]# ll /mnt/rhel54_240909_1750.save
>  -rw------- 1 root root 334221504 2009-09-24 17:51
>  /mnt/rhel54_240909_1750.save
>  
>  The VM has an hd of 8Gb of size and 4Gb of ram.
>  
>  If I want to debug setting LIBVIRTD_DEBUG=1 and then run the restore
>  command via command line, what would it be, so that I can bugzilla this
>  (if not already there)?
>  
>  Inside /var/log/messages I have this:
>  Sep 24 17:51:51 virtfedbis kernel: libvirtd[25820]: segfault at 0 ip
>  00000039762a4822 sp 00007f20a7546880 error 4 in libc-2.10.1.so
>  <http://libc-2.10.1.so>[3976200000+164000]
>  Sep 24 17:51:51 virtfedbis libvirtd: 17:51:51.379: error :
>  qemudExtractVersionInfo:1032 : Unable to read QEMU help output:
>  Interrupted system call 
OK, yeah, it looks like libvirtd segfaulted, which is obviously not good.
You'll want to stop libvirtd, then run (as root):

# ulimit -c unlimited
# LIBVIRT_DEBUG=1 /usr/sbin/libvirtd --verbose

Assuming you get a core out of it, the best information to get is the output of
"thread apply all bt" with gdb.

-- 
Chris Lalancette

[Paul Jenner]

On Fri, 2009-09-25 at 08:52 +0200, Chris Lalancette wrote: 
>  gianluca.cecchi wrote:
>  > BTW: the restore on the same host (with the VM still stopped as at the
>  > end of the save), doesn't work either. I get:
>  > Error restoring domain '/mnt/rhel54_240909_1750.save': Unable to read
>  > QEMU help output: Interrupted system call
>  > 
>  > And inside the details window:
>  > Traceback (most recent call last):
>  >   File "/usr/share/virt-manager/virtManager/manager.py", line 458, in
>  > restore_saved_callback
>  >     newconn.restore(file_to_load)
>  >   File "/usr/share/virt-manager/virtManager/connection.py", line 642, in
>  > restore
>  >     self.vmm.restore(frm)
>  >   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1420, in
>  > restore
>  >     if ret == -1: raise libvirtError ('virDomainRestore() failed',
>  > conn=self)
>  > libvirtError: Unable to read QEMU help output: Interrupted system call
>  > 
>  > [root@virtfedbis qemu]# ll /mnt/rhel54_240909_1750.save
>  > -rw------- 1 root root 334221504 2009-09-24 17:51
>  > /mnt/rhel54_240909_1750.save
>  > 
>  > The VM has an hd of 8Gb of size and 4Gb of ram.
>  > 
>  > If I want to debug setting LIBVIRTD_DEBUG=1 and then run the restore
>  > command via command line, what would it be, so that I can bugzilla this
>  > (if not already there)?
>  > 
>  > Inside /var/log/messages I have this:
>  > Sep 24 17:51:51 virtfedbis kernel: libvirtd[25820]: segfault at 0 ip
>  > 00000039762a4822 sp 00007f20a7546880 error 4 in libc-2.10.1.so
>  > <http://libc-2.10.1.so>[3976200000+164000]
>  > Sep 24 17:51:51 virtfedbis libvirtd: 17:51:51.379: error :
>  > qemudExtractVersionInfo:1032 : Unable to read QEMU help output:
>  > Interrupted system call
>  
>  OK, yeah, it looks like libvirtd segfaulted, which is obviously not good.
>  You'll want to stop libvirtd, then run (as root):
>  
>  # ulimit -c unlimited
>  # LIBVIRT_DEBUG=1 /usr/sbin/libvirtd --verbose
>  
>  Assuming you get a core out of it, the best information to get is the output of
>  "thread apply all bt" with gdb. 
This looks similar to Charles Duffy's recent post to libvir-list here:

https://www.redhat.com/archives/libvir-list/2009-September/msg00663.html

Paul

-- 
Paul Jenner <psj(a)familyjenner.co.uk&gt;

[Chris Lalancette]

gianluca.cecchi wrote:
>  I confirm that with that version of libvirt I'm able to save my
> VM (or
>  at least the command completes, not tried yet to restore '-)
>  Inside the data file saved I can see that there is also the xml config
>  for the guest embedded ....
>  As the VM is killed at the end of the save, can I then
>  - copy the disk(s) of the VM to another host
>  - copy the save file to another host
>  - restore the save on that secondary host based on the fact that it
>  retains the same config? 
The answer is "maybe".  If the two machines are exactly the same in every way,
including the paths to storage, the network configuration, the CPUs, then yes,
this will probably work.  Note that this is more-or-less live migration, without
the "live" part.  However, as you found out, if the paths are different, or the
CPUs expose different flags, then chances are this won't work.

Note that there are currently proposals in libvirt to expose a
"least-common-denominator" for CPU, so that you can migrate VMs between two
machines of differing generations and models.  However, there is no code yet.

-- 
Chris Lalancette