[fedora-virt] Problems with Rawhide VM on CentOS 6.2

[Richard W.M. Jones]

On Sun, Apr 01, 2012 at 06:52:27PM +0200, Tim Niemueller wrote:
> Hi all.
> 
> To test our robot software against the newest and shiniest compilers and
> libs we have a rawhide VM. Today I noticed a very annoying problem
> (after not using the VM for a while), but I'm not sure if it's on our
> side or if I should file a bug.
> 
> The VM is frequently unable to access its root partition (using virtio
> for this VM). SSH connection end with "broken pipe", if locally on the
> machine trying to execute "dmesg" it only says "cannot find /bin/dmesg".
> In the time when the error appears until I loose the connection was once
> able to do "cat /proc/kmsg" showing errors on /dev/vda. In the libvirt
> log on the host (CentOS 6.2 using libvirtd and virt-manager) I see the
> following entries popping up in these situations:
> 
> block I/O error in device 'drive-virtio-disk0': Permission denied (13)
> 
> SELinux' audit.log is quiet. And the VM runs, and sometimes I can work
> on it for 30mins straight, sometimes not even for 30secs. Any idea what
> might be causing this intermittent problem?
> 
> During investigation I noticed that the very same message appears every
> now and then in other VM's logs as well. They are running CentOS 6.2,
> and one is FreeBSD 9.0. But in those I do not see the broken connection
> or "cannot find file" problems.
> 
> The VM's disks are logical volumes in the same volume group. The machine
> is a Dual Xeon Quad-Core with a speedy RAID array hosting the VG.
> 
> Does someone have a similar problem, or can give ideas how to fix or
> investigate, and if to file a bug, which component to target? I'm not
> even sure if it's a CentOS or a rawhide thing, though I assume the latter...

Is the host RAID array using 4K sectors?

Look at the files in /sys/block/<device>/queue and compare with:
http://libguestfs.org/virt-alignment-scan.1.html#linux_host_block_and_i_o_size

Secondly, attach a virtual serial port to the guest and boot it using
the kernel command line parameter: console=ttyS0,115200.  Then take a
look at the boot messages.

Thirdly, look at /var/log/libvirt/qemu/*.log

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org