I have some very weird behavior with my new ubuntu 9.10 virtual machines running as KVMs under fedora 11.
I run testbeds on lots of different linuxii in ssh sessions to virtual machines, and the ssh session just stops randomly when talking to ubuntu 9.10 (no problem with 9.04, or 8.10, no problem with various fedora versions or openSUSE versions).
I can still ssh into the KVM with a new ssh connection, but the original connection is frozen up, it never even times out an disconnects, it acts like it is in the middle of talking and is just waiting for the next thing to happen.
Does this strike a familiar note to anyone? I've tried both virtio and emulated NIC modes, and see the same results.
On Thu, 2009-11-05 at 12:03 -0500, Tom Horsley wrote:
I have some very weird behavior with my new ubuntu 9.10 virtual machines running as KVMs under fedora 11.
I run testbeds on lots of different linuxii in ssh sessions to virtual machines, and the ssh session just stops randomly when talking to ubuntu 9.10 (no problem with 9.04, or 8.10, no problem with various fedora versions or openSUSE versions).
I can still ssh into the KVM with a new ssh connection, but the original connection is frozen up, it never even times out an disconnects, it acts like it is in the middle of talking and is just waiting for the next thing to happen.
Does this strike a familiar note to anyone? I've tried both virtio and emulated NIC modes, and see the same results.
I don't recall hearing about this bug before
I'd be curious as to whether you can reproduce in with the Fedora 12 bits from virt-preview. A tcpdump might help too
Also might be worth asking Ubuntu developers, it could just be a guest bug
Cheers, Mark.
On Mon, 09 Nov 2009 14:15:17 +0000 Mark McLoughlin wrote:
On Thu, 2009-11-05 at 12:03 -0500, Tom Horsley wrote:
... I can still ssh into the KVM with a new ssh connection, but the original connection is frozen up, it never even times out an disconnects, it acts like it is in the middle of talking and is just waiting for the next thing to happen.
Does this strike a familiar note to anyone? I've tried both virtio and emulated NIC modes, and see the same results.
I don't recall hearing about this bug before
Well, ubuntu 9.10 was just released, and it is the only system I've had this problem with.
I'd be curious as to whether you can reproduce in with the Fedora 12 bits from virt-preview. A tcpdump might help too
Yea, I'm curious about those too, but I'm not really in the position to switch the host software around a lot (though I may indeed go to fedora 12 in the not too distant future). Not sure if I have enough disk space to record a tcpdump :-).
Also might be worth asking Ubuntu developers, it could just be a guest bug
Yea, I've asked over on ubuntuforums, but no response there as yet.
One thing I did was install ubuntu 9.10 on some real hardware, and that system has run the testbeds several times with no problems, so it does seem to be something unique to the KVM installs.
On Mon, 9 Nov 2009 12:05:50 -0500 Tom Horsley wrote:
On Mon, 09 Nov 2009 14:15:17 +0000 Mark McLoughlin wrote:
On Thu, 2009-11-05 at 12:03 -0500, Tom Horsley wrote:
... I can still ssh into the KVM with a new ssh connection, but the original connection is frozen up, it never even times out an disconnects, it acts like it is in the middle of talking and is just waiting for the next thing to happen.
Woa! Just happened to notice new symptom. I had an ssh session to one of these ubuntu KVM machines open at the same time I had watched it boot in the console window I opened from virt-manager. Not only did the ssh session freeze, but the console window froze at the same time. I could still ssh in with a new ssh session, but I'm not sure what would take out an ssh session and the console at the same time while leaving the KVM machine up.
On Mon, 2009-11-09 at 15:17 -0500, Tom Horsley wrote:
On Mon, 9 Nov 2009 12:05:50 -0500 Tom Horsley wrote:
On Mon, 09 Nov 2009 14:15:17 +0000 Mark McLoughlin wrote:
On Thu, 2009-11-05 at 12:03 -0500, Tom Horsley wrote:
... I can still ssh into the KVM with a new ssh connection, but the original connection is frozen up, it never even times out an disconnects, it acts like it is in the middle of talking and is just waiting for the next thing to happen.
Woa! Just happened to notice new symptom. I had an ssh session to one of these ubuntu KVM machines open at the same time I had watched it boot in the console window I opened from virt-manager. Not only did the ssh session freeze, but the console window froze at the same time. I could still ssh in with a new ssh session, but I'm not sure what would take out an ssh session and the console at the same time while leaving the KVM machine up.
Are you running virt-manager on the same machine as the guests? i.e. is virt-manager connected to qemu:///system ?
If it's all local, the VNC connection is simply a localhost TCP socket between virt-manager and qemu, whereas the ssh is a TCP socket over virbr0/vnet0 ethernet to the guest sshd. I can't imagine how both would lock up at the same time while qemu is still happily processing packets to form a new ssh connection.
Cheers, Mark.
On Tue, 10 Nov 2009 08:03:47 +0000 Mark McLoughlin wrote:
Woa! Just happened to notice new symptom. I had an ssh session to one of these ubuntu KVM machines open at the same time I had watched it boot in the console window I opened from virt-manager. Not only did the ssh session freeze, but the console window froze at the same time. I could still ssh in with a new ssh session, but I'm not sure what would take out an ssh session and the console at the same time while leaving the KVM machine up.
Are you running virt-manager on the same machine as the guests? i.e. is virt-manager connected to qemu:///system ?
If it's all local, the VNC connection is simply a localhost TCP socket between virt-manager and qemu, whereas the ssh is a TCP socket over virbr0/vnet0 ethernet to the guest sshd. I can't imagine how both would lock up at the same time while qemu is still happily processing packets to form a new ssh connection.
Virt-manager was running local on the host machine, but displaying remotely to my desktop over a forwarded X connection. The ssh session was from my desktop directly to the virtual machine (all the VMs are setup using a bridge network). It is indeed weird. Other versions of linux running in VMs configured pretty much the same way on the same host don't have these problems. The same ubuntu 9.10 on real hardware doesn't have the problem. Worse yet, I installed some ubuntu 9.10 virtual machines at home on a very similar (but much less loaded) KVM setup as I have at work, and they have never shown the problem. I guess there are just some things man was not meant to know :-). I'll see what happens when we install fedora 12 on the host after the official release.
On Thu, Nov 05, 2009 at 12:03:01PM -0500, Tom Horsley wrote:
I have some very weird behavior with my new ubuntu 9.10 virtual machines running as KVMs under fedora 11.
I run testbeds on lots of different linuxii in ssh sessions to virtual machines, and the ssh session just stops randomly when talking to ubuntu 9.10 (no problem with 9.04, or 8.10, no problem with various fedora versions or openSUSE versions).
Do the guests have libvirtd running inside them? libvirtd tends to create a default network (192.168.122.*) which conflicts with the host virbr0 network. This usually breaks networking completely -- I've not seen is cause hangs on an existing working network, unless ssh is hanging while something is starting up libvirtd.
https://bugzilla.redhat.com/show_bug.cgi?id=509458
If not that, then these problems are often caused by specific classes of packets getting through (eg. packets that are too large, have broken checksums etc). So as Mark said, a tcpdump is vital to solve the issue.
Rich.
On Tue, Nov 10, 2009 at 05:31:10PM +0000, Richard W.M. Jones wrote:
of packets getting through (eg. packets that are too large, have
... not getting through ...
Rich.
On Tue, 10 Nov 2009 17:31:10 +0000 Richard W.M. Jones wrote:
So as Mark said, a tcpdump is vital to solve the issue.
I still have no ideas why the virt-viewer console stopped working at the same time, but going through logs on the guest machines showed $#@! NetworkManager deciding it needed to do a brand new DHCPDISCOVER for the network connection which was already up, resulting in the IP address changing out from under my original connection. After much cursing and poking of sticks at NetworkManager I think the connection is now staying up. Why my "real hardware" ubuntu 9.10 machine doesn't suffer from the same problem, I don't know.
-
Mark McLoughlin -
Richard W.M. Jones -
Tom Horsley