virsh output discrepancy with vol-list

[Shergill, Gurinder]

Hello,

I am seeing following discrepancy with virsh output for volume list, which goes away when
I restart the libvirtd daemon.

Here virsh shows 4 volumes. Also, notice that first 2 volumes are duplicates.

$ virsh vol-list --pool default
Name                 Path                                    
-----------------------------------------
crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
fish.vol.0.img       /var/lib/libvirt/images/fish.vol.0.img  
fish.vol.1.img       /var/lib/libvirt/images/fish.vol.1.img  

However, none of the above volumes currently exist (as I had just deleted them using,
again, using virsh).

$ ll /var/lib/libvirt/images/
total 0

Now, I restart libvirt. And, after that virsh output is fine.

$ virsh vol-list --pool default
Name                 Path                                    
-----------------------------------------

Any idea what is going on here ? Is this a known issue ?

I am using Fedora 17.

Sunny

[Kashyap Chamarthy]

On 04/11/2013 08:36 AM, Shergill, Gurinder wrote:
>  Hello,
>  
>  I am seeing following discrepancy with virsh output for volume list, which goes away when
> I restart the libvirtd daemon.
>  
>  Here virsh shows 4 volumes. Also, notice that first 2 volumes are duplicates.
>  
>  $ virsh vol-list --pool default
>  Name                 Path                                    
>  -----------------------------------------
>  crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
>  crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
>  fish.vol.0.img       /var/lib/libvirt/images/fish.vol.0.img  
>  fish.vol.1.img       /var/lib/libvirt/images/fish.vol.1.img  
>  
>  However, none of the above volumes currently exist (as I had just deleted them using,
> again, using virsh).
>  
>  $ ll /var/lib/libvirt/images/
>  total 0
>  
>  Now, I restart libvirt. And, after that virsh output is fine.
>  
>  $ virsh vol-list --pool default
>  Name                 Path                                    
>  -----------------------------------------
>  
>  Any idea what is going on here ? Is this a known issue ? 
Not sure at the first glance.
>  
>  I am using Fedora 17. 
Version of libvirt ?

I'm using this version & I can't reproduce it -- libvirt-1.0.4-1.fc18.x86_64

Do you find the same issue with using --
http://fedorapeople.org/groups/virt/virt-preview/

For further debugging:

Add the below in in libvirtd.log
====
log_level = 1
log_outputs = 1:file:/var/tmp/libvirtd.log
====

$ service libvirtd restart

Monitor the log for interesting info.

/kashyap

>  
>  Sunny
>  
>  _______________________________________________
>  virt mailing list
>  virt(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/virt
>

[Shergill, Gurinder]

>  -----Original Message-----
>  From: Kashyap Chamarthy [mailto:[e-mail]]
>  Sent: Thursday, April 11, 2013 4:09 AM
>  To: Shergill, Gurinder
>  Cc: virt(a)lists.fedoraproject.org
>  Subject: Re: [fedora-virt] virsh output discrepancy with vol-list
>  
>  On 04/11/2013 08:36 AM, Shergill, Gurinder wrote:
>  > Hello,
>  >
>  > I am seeing following discrepancy with virsh output for volume list,
>  which goes away when I restart the libvirtd daemon.
>  >
>  > Here virsh shows 4 volumes. Also, notice that first 2 volumes are
>  duplicates.
>  >
>  > $ virsh vol-list --pool default
>  > Name                 Path
>  > -----------------------------------------
>  > crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img
>  > crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img
>  > fish.vol.0.img       /var/lib/libvirt/images/fish.vol.0.img
>  > fish.vol.1.img       /var/lib/libvirt/images/fish.vol.1.img
>  >
>  > However, none of the above volumes currently exist (as I had just
>  deleted them using, again, using virsh).
>  >
>  > $ ll /var/lib/libvirt/images/
>  > total 0
>  >
>  > Now, I restart libvirt. And, after that virsh output is fine.
>  >
>  > $ virsh vol-list --pool default
>  > Name                 Path
>  > -----------------------------------------
>  >
>  > Any idea what is going on here ? Is this a known issue ?
>  
>  Not sure at the first glance.
>  >
>  > I am using Fedora 17.
>  
>  Version of libvirt ?
>  
$ rpm -qa | grep libvirt
libvirt-daemon-0.9.11.9-1.fc17.x86_64
libvirt-python-0.9.11.9-1.fc17.x86_64
libvirt-daemon-config-network-0.9.11.9-1.fc17.x86_64
libvirt-client-0.9.11.9-1.fc17.x86_64
libvirt-daemon-qemu-0.9.11.9-1.fc17.x86_64
libvirt-0.9.11.9-1.fc17.x86_64
libvirt-daemon-config-nwfilter-0.9.11.9-1.fc17.x86_64

>  
>  I'm using this version & I can't reproduce it -- libvirt-1.0.4-
>  1.fc18.x86_64
>  
The issue doesn't happen all the time or right away. I am working on benchmarking
various KVM storage I/O options, for this I routinely create/destroy guests, and after a
while I'll notice this discrepancy.

>  
>  Do you find the same issue with using --
>  http://fedorapeople.org/groups/virt/virt-preview/
>  
I haven't tried that yet.

>  
>  For further debugging:
>  
>  Add the below in in libvirtd.log
>  ====
>  log_level = 1
>  log_outputs = 1:file:/var/tmp/libvirtd.log ====
>  
>  $ service libvirtd restart
>  
>  Monitor the log for interesting info.
>  
Okay. I'll try that.

Sunny

[Kashyap Chamarthy]

[Please wrap lines in your email client. Long lines are a bit hard to read.]

> >> I am using Fedora 17.
> >
> > Version of libvirt ?
>  
>  $ rpm -qa | grep libvirt
>  libvirt-daemon-0.9.11.9-1.fc17.x86_64
>  libvirt-python-0.9.11.9-1.fc17.x86_64
>  libvirt-daemon-config-network-0.9.11.9-1.fc17.x86_64
>  libvirt-client-0.9.11.9-1.fc17.x86_64
>  libvirt-daemon-qemu-0.9.11.9-1.fc17.x86_64
>  libvirt-0.9.11.9-1.fc17.x86_64
>  libvirt-daemon-config-nwfilter-0.9.11.9-1.fc17.x86_64 
That's a little old. Can you please try w/ libvirt-daemon-kvm-1.0.4-1.fc18.x86_64

If some odd error hits me, I try, in that order:
	- pull up virt-preview to test latests pkgs
		- Info on enabling it - http://fedoraproject.org/wiki/Virtualization_Preview_Repository

	- if the above fails, build qemu & libvirt from upstream git, if you can still
reproduce,
file a bz with clear details/reproducer.

>  
> >
> > I'm using this version & I can't reproduce it -- libvirt-1.0.4-
> > 1.fc18.x86_64
>  
>  The issue doesn't happen all the time or right away. I am working on benchmarking
> various KVM storage I/O options, for this I routinely create/destroy guests, and after a
> while I'll notice this discrepancy. 
If I were you, doing bench-marking, I'd try with newest available packages of kernel,
qemu, libvirt.

Hope that helps.

--
/kashyap

[Shergill, Gurinder]

>  -----Original Message-----
>  From: Kashyap Chamarthy [mailto:[e-mail]]
>  Sent: Thursday, April 11, 2013 8:51 AM
>  To: Shergill, Gurinder
>  Cc: virt(a)lists.fedoraproject.org
>  Subject: Re: [fedora-virt] virsh output discrepancy with vol-list
>  
>  [Please wrap lines in your email client. Long lines are a bit hard to
>  read.]
>  
>  >>> I am using Fedora 17.
>  >>
>  >> Version of libvirt ?
>  >
>  > $ rpm -qa | grep libvirt
>  > libvirt-daemon-0.9.11.9-1.fc17.x86_64
>  > libvirt-python-0.9.11.9-1.fc17.x86_64
>  > libvirt-daemon-config-network-0.9.11.9-1.fc17.x86_64
>  > libvirt-client-0.9.11.9-1.fc17.x86_64
>  > libvirt-daemon-qemu-0.9.11.9-1.fc17.x86_64
>  > libvirt-0.9.11.9-1.fc17.x86_64
>  > libvirt-daemon-config-nwfilter-0.9.11.9-1.fc17.x86_64
>  
>  That's a little old. Can you please try w/ libvirt-daemon-kvm-1.0.4-
>  1.fc18.x86_64
>  
Okay. Will move to F18 and try.

>  
>  If some odd error hits me, I try, in that order:
>  	- pull up virt-preview to test latests pkgs
>  		- Info on enabling it -
>  http://fedoraproject.org/wiki/Virtualization_Preview_Repository
>  
>  	- if the above fails, build qemu & libvirt from upstream git, if you
>  can still reproduce, file a bz with clear details/reproducer.
>  
>  >
>  >>
>  >> I'm using this version & I can't reproduce it -- libvirt-1.0.4-
>  >> 1.fc18.x86_64
>  >
>  > The issue doesn't happen all the time or right away. I am working on
>  benchmarking various KVM storage I/O options, for this I routinely
>  create/destroy guests, and after a while I'll notice this discrepancy.
>  
>  If I were you, doing bench-marking, I'd try with newest available packages
>  of kernel, qemu, libvirt.
>  
I agree, however, I am trying to find the right balance between stability and newest
available packages. That said, point taken -- I'll move to F18, and try out the
preview repos. 

>  
>  Hope that helps.
>  
Yeap. Thanks,

Sunny

[Shergill, Gurinder]

>  -----Original Message-----
>  From: Kashyap Chamarthy [mailto:[e-mail]]
>  Sent: Thursday, April 11, 2013 4:09 AM
>  To: Shergill, Gurinder
>  Cc: virt(a)lists.fedoraproject.org
>  Subject: Re: [fedora-virt] virsh output discrepancy with vol-list
>  
>  On 04/11/2013 08:36 AM, Shergill, Gurinder wrote:
>  > Hello,
>  >
>  > I am seeing following discrepancy with virsh output for volume list,
>  which goes away when I restart the libvirtd daemon.
>  >
>  > Here virsh shows 4 volumes. Also, notice that first 2 volumes are
>  duplicates.
>  >
>  > $ virsh vol-list --pool default
>  > Name                 Path
>  > -----------------------------------------
>  > crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img
>  > crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img
>  > fish.vol.0.img       /var/lib/libvirt/images/fish.vol.0.img
>  > fish.vol.1.img       /var/lib/libvirt/images/fish.vol.1.img
>  >
>  > However, none of the above volumes currently exist (as I had just
>  deleted them using, again, using virsh).
>  >
>  > $ ll /var/lib/libvirt/images/
>  > total 0
>  >
>  > Now, I restart libvirt. And, after that virsh output is fine.
>  >
>  > $ virsh vol-list --pool default
>  > Name                 Path
>  > -----------------------------------------
>  >
>  > Any idea what is going on here ? Is this a known issue ?
>  
>  Not sure at the first glance.
>  >
>  > I am using Fedora 17.
>  
>  Version of libvirt ?
>  
>  I'm using this version & I can't reproduce it -- libvirt-1.0.4-
>  1.fc18.x86_64
>  
>  Do you find the same issue with using --
>  http://fedorapeople.org/groups/virt/virt-preview/
>  
>  For further debugging:
>  
>  Add the below in in libvirtd.log
>  ====
>  log_level = 1
>  log_outputs = 1:file:/var/tmp/libvirtd.log ====
>  
>  $ service libvirtd restart
>  
I enabled the log output and set the level to 1, just like you suggested. Interestingly,
the issue happened almost right away and, this time, I only had to create a guest from
scratch.

$ virsh vol-list --pool default
Name                 Path                                    
-----------------------------------------
crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
crab.vol.1.img       /var/lib/libvirt/images/crab.vol.1.img  
crab.vol.1.img       /var/lib/libvirt/images/crab.vol.1.img  
crab.vol.2.img       /var/lib/libvirt/images/crab.vol.2.img  
crab.vol.2.img       /var/lib/libvirt/images/crab.vol.2.img  
crab.vol.3.img       /var/lib/libvirt/images/crab.vol.3.img  
crab.vol.3.img       /var/lib/libvirt/images/crab.vol.3.img  

$ ll /var/lib/libvirt/images/
total 1248240
-rw-------. 1 root root 1271201792 Apr 10 21:53 crab.vol.0.img
-rw-------. 1 root root    2490368 Apr 10 21:47 crab.vol.1.img
-rw-------. 1 root root    2490368 Apr 10 21:47 crab.vol.2.img
-rw-------. 1 root root    2490368 Apr 10 21:47 crab.vol.3.img

Restarted libvirt ... and virsh gives correct information.

$ virsh vol-list --pool default
Name                 Path                                    
-----------------------------------------
crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
crab.vol.1.img       /var/lib/libvirt/images/crab.vol.1.img  
crab.vol.2.img       /var/lib/libvirt/images/crab.vol.2.img  
crab.vol.3.img       /var/lib/libvirt/images/crab.vol.3.img  

>  
>  Monitor the log for interesting info.
>
>  
The log file is HUGE - about 400k lines ! ... any suggestions on what would classify as
interesting info for this issue ?

Sunny

[Cole Robinson]

On 04/10/2013 11:06 PM, Shergill, Gurinder wrote:
>  Hello,
>  
>  I am seeing following discrepancy with virsh output for volume list, which goes away when
> I restart the libvirtd daemon.
>  
>  Here virsh shows 4 volumes. Also, notice that first 2 volumes are duplicates.
>  
>  $ virsh vol-list --pool default
>  Name                 Path                                    
>  -----------------------------------------
>  crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
>  crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img  
>  fish.vol.0.img       /var/lib/libvirt/images/fish.vol.0.img  
>  fish.vol.1.img       /var/lib/libvirt/images/fish.vol.1.img  
>  
>  However, none of the above volumes currently exist (as I had just deleted them using,
> again, using virsh).
>  
>  $ ll /var/lib/libvirt/images/
>  total 0
>  
>  Now, I restart libvirt. And, after that virsh output is fine.
>  
>  $ virsh vol-list --pool default
>  Name                 Path                                    
>  -----------------------------------------
>  
>  Any idea what is going on here ? Is this a known issue ?
>   
When you delete files behind libvirt's back, it doesn't notice. You need to do

  virsh vol-refresh default

This was a deliberate decision by libvirt but for directory pools it's caused
so much grief it should be fixed. There's a bug about a possible solution:

https://bugzilla.redhat.com/show_bug.cgi?id=821508

- Cole

[Shergill, Gurinder]

>  -----Original Message-----
>  From: Cole Robinson [mailto:[e-mail]]
>  Sent: Saturday, April 13, 2013 3:47 PM
>  To: Shergill, Gurinder
>  Cc: virt(a)lists.fedoraproject.org
>  Subject: Re: [fedora-virt] virsh output discrepancy with vol-list
>  
>  On 04/10/2013 11:06 PM, Shergill, Gurinder wrote:
>  > Hello,
>  >
>  > I am seeing following discrepancy with virsh output for volume list,
>  which goes away when I restart the libvirtd daemon.
>  >
>  > Here virsh shows 4 volumes. Also, notice that first 2 volumes are
>  duplicates.
>  >
>  > $ virsh vol-list --pool default
>  > Name                 Path
>  > -----------------------------------------
>  > crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img
>  > crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img
>  > fish.vol.0.img       /var/lib/libvirt/images/fish.vol.0.img
>  > fish.vol.1.img       /var/lib/libvirt/images/fish.vol.1.img
>  >
>  > However, none of the above volumes currently exist (as I had just
>  deleted them using, again, using virsh).
>  >
>  > $ ll /var/lib/libvirt/images/
>  > total 0
>  >
>  > Now, I restart libvirt. And, after that virsh output is fine.
>  >
>  > $ virsh vol-list --pool default
>  > Name                 Path
>  > -----------------------------------------
>  >
>  > Any idea what is going on here ? Is this a known issue ?
>  >
>  
>  When you delete files behind libvirt's back, it doesn't notice. 
Although, in my case, I used "virsh vol-delete" and did not go behind libvirt.
And, the last instance I reported, issue happened without anything being deleted.

http://lists.fedoraproject.org/pipermail/virt/2013-April/003637.html

>  You need to do
>  
>    virsh vol-refresh default 
Good to know.

>  
>  This was a deliberate decision by libvirt but for directory pools it's
>  caused so much grief it should be fixed. There's a bug about a possible
>  solution:
>  
>  https://bugzilla.redhat.com/show_bug.cgi?id=821508
>  
>  - Cole 
I am moving to F18, as per Kashyap's suggestion, will report back how it goes.

Sunny

[Cole Robinson]

On 04/13/2013 10:25 PM, Shergill, Gurinder wrote:
> > -----Original Message-----
> > From: Cole Robinson [mailto:[e-mail]]
> > Sent: Saturday, April 13, 2013 3:47 PM
> > To: Shergill, Gurinder
> > Cc: virt(a)lists.fedoraproject.org
> > Subject: Re: [fedora-virt] virsh output discrepancy with vol-list
> >
> > On 04/10/2013 11:06 PM, Shergill, Gurinder wrote:
> >> Hello,
> >>
> >> I am seeing following discrepancy with virsh output for volume list,
> > which goes away when I restart the libvirtd daemon.
> >>
> >> Here virsh shows 4 volumes. Also, notice that first 2 volumes are
> > duplicates.
> >>
> >> $ virsh vol-list --pool default
> >> Name                 Path
> >> -----------------------------------------
> >> crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img
> >> crab.vol.0.img       /var/lib/libvirt/images/crab.vol.0.img
> >> fish.vol.0.img       /var/lib/libvirt/images/fish.vol.0.img
> >> fish.vol.1.img       /var/lib/libvirt/images/fish.vol.1.img
> >>
> >> However, none of the above volumes currently exist (as I had just
> > deleted them using, again, using virsh).
> >>
> >> $ ll /var/lib/libvirt/images/
> >> total 0
> >>
> >> Now, I restart libvirt. And, after that virsh output is fine.
> >>
> >> $ virsh vol-list --pool default
> >> Name                 Path
> >> -----------------------------------------
> >>
> >> Any idea what is going on here ? Is this a known issue ?
> >>
> >
> > When you delete files behind libvirt's back, it doesn't notice.
>  
>  Although, in my case, I used "virsh vol-delete" and did not go behind libvirt.
> And, the last instance I reported, issue happened without anything being deleted.
>  
>  http://lists.fedoraproject.org/pipermail/virt/2013-April/003637.html 
Hmm, sorry, must have misread.

>  
> > You need to do
> >
> >   virsh vol-refresh default
>  
>  Good to know.
>   
Man I really butchered this mail :) The correct command is actually

  virsh pool-refresh default

- Cole