11:03 a.m.
I have some very weird behavior with my new ubuntu 9.10
virtual machines running as KVMs under fedora 11.
I run testbeds on lots of different linuxii in ssh sessions
to virtual machines, and the ssh session just stops randomly
when talking to ubuntu 9.10 (no problem with 9.04, or 8.10,
no problem with various fedora versions or openSUSE versions).
I can still ssh into the KVM with a new ssh connection, but
the original connection is frozen up, it never even times
out an disconnects, it acts like it is in the middle of
talking and is just waiting for the next thing to happen.
Does this strike a familiar note to anyone? I've tried
both virtio and emulated NIC modes, and see the same results.
8:15 a.m.
On Thu, 2009-11-05 at 12:03 -0500, Tom Horsley wrote:
I have some very weird behavior with my new ubuntu 9.10
virtual machines running as KVMs under fedora 11.
I run testbeds on lots of different linuxii in ssh sessions
to virtual machines, and the ssh session just stops randomly
when talking to ubuntu 9.10 (no problem with 9.04, or 8.10,
no problem with various fedora versions or openSUSE versions).
I can still ssh into the KVM with a new ssh connection, but
the original connection is frozen up, it never even times
out an disconnects, it acts like it is in the middle of
talking and is just waiting for the next thing to happen.
Does this strike a familiar note to anyone? I've tried
both virtio and emulated NIC modes, and see the same results.
I don't recall hearing about this bug before
I'd be curious as to whether you can reproduce in with the Fedora 12
bits from virt-preview. A tcpdump might help too
Also might be worth asking Ubuntu developers, it could just be a guest
bug
Cheers,
Mark.
11:05 a.m.
On Mon, 09 Nov 2009 14:15:17 +0000
Mark McLoughlin wrote:
On Thu, 2009-11-05 at 12:03 -0500, Tom Horsley wrote:
> ...
> I can still ssh into the KVM with a new ssh connection, but
> the original connection is frozen up, it never even times
> out an disconnects, it acts like it is in the middle of
> talking and is just waiting for the next thing to happen.
>
> Does this strike a familiar note to anyone? I've tried
> both virtio and emulated NIC modes, and see the same results.
I don't recall hearing about this bug before
Well, ubuntu 9.10 was just released, and it is the only system
I've had this problem with.
I'd be curious as to whether you can reproduce in with the Fedora
12
bits from virt-preview. A tcpdump might help too
Yea, I'm curious about those too, but I'm not really in the position
to switch the host software around a lot (though I may indeed go to
fedora 12 in the not too distant future). Not sure if I have enough
disk space to record a tcpdump :-).
Also might be worth asking Ubuntu developers, it could just be a
guest
bug
Yea, I've asked over on ubuntuforums, but no response there as yet.
One thing I did was install ubuntu 9.10 on some real hardware, and
that system has run the testbeds several times with no problems,
so it does seem to be something unique to the KVM installs.
2:17 p.m.
On Mon, 9 Nov 2009 12:05:50 -0500
Tom Horsley wrote:
On Mon, 09 Nov 2009 14:15:17 +0000
Mark McLoughlin wrote:
> On Thu, 2009-11-05 at 12:03 -0500, Tom Horsley wrote:
> > ...
> > I can still ssh into the KVM with a new ssh connection, but
> > the original connection is frozen up, it never even times
> > out an disconnects, it acts like it is in the middle of
> > talking and is just waiting for the next thing to happen.
Woa! Just happened to notice new symptom. I had an ssh session
to one of these ubuntu KVM machines open at the same time I had
watched it boot in the console window I opened from virt-manager.
Not only did the ssh session freeze, but the console window froze
at the same time. I could still ssh in with a new ssh session, but
I'm not sure what would take out an ssh session and the console
at the same time while leaving the KVM machine up.
2:03 a.m.
On Mon, 2009-11-09 at 15:17 -0500, Tom Horsley wrote:
On Mon, 9 Nov 2009 12:05:50 -0500
Tom Horsley wrote:
> On Mon, 09 Nov 2009 14:15:17 +0000
> Mark McLoughlin wrote:
>
> > On Thu, 2009-11-05 at 12:03 -0500, Tom Horsley wrote:
> > > ...
> > > I can still ssh into the KVM with a new ssh connection, but
> > > the original connection is frozen up, it never even times
> > > out an disconnects, it acts like it is in the middle of
> > > talking and is just waiting for the next thing to happen.
Woa! Just happened to notice new symptom. I had an ssh session
to one of these ubuntu KVM machines open at the same time I had
watched it boot in the console window I opened from virt-manager.
Not only did the ssh session freeze, but the console window froze
at the same time. I could still ssh in with a new ssh session, but
I'm not sure what would take out an ssh session and the console
at the same time while leaving the KVM machine up.
Are you running virt-manager on the same machine as the guests? i.e. is
virt-manager connected to qemu:///system ?
If it's all local, the VNC connection is simply a localhost TCP socket
between virt-manager and qemu, whereas the ssh is a TCP socket over
virbr0/vnet0 ethernet to the guest sshd. I can't imagine how both would
lock up at the same time while qemu is still happily processing packets
to form a new ssh connection.
Cheers,
Mark.
6:40 a.m.
On Tue, 10 Nov 2009 08:03:47 +0000
Mark McLoughlin wrote:
> Woa! Just happened to notice new symptom. I had an ssh session
> to one of these ubuntu KVM machines open at the same time I had
> watched it boot in the console window I opened from virt-manager.
> Not only did the ssh session freeze, but the console window froze
> at the same time. I could still ssh in with a new ssh session, but
> I'm not sure what would take out an ssh session and the console
> at the same time while leaving the KVM machine up.
Are you running virt-manager on the same machine as the guests? i.e. is
virt-manager connected to qemu:///system ?
If it's all local, the VNC connection is simply a localhost TCP socket
between virt-manager and qemu, whereas the ssh is a TCP socket over
virbr0/vnet0 ethernet to the guest sshd. I can't imagine how both would
lock up at the same time while qemu is still happily processing packets
to form a new ssh connection.
Virt-manager was running local on the host machine, but displaying
remotely to my desktop over a forwarded X connection. The ssh session
was from my desktop directly to the virtual machine (all the VMs
are setup using a bridge network). It is indeed weird. Other versions
of linux running in VMs configured pretty much the same way on the same
host don't have these problems. The same ubuntu 9.10 on real hardware
doesn't have the problem. Worse yet, I installed some ubuntu 9.10 virtual
machines at home on a very similar (but much less loaded) KVM setup
as I have at work, and they have never shown the problem. I guess
there are just some things man was not meant to know :-). I'll see
what happens when we install fedora 12 on the host after the official
release.
11:31 a.m.
On Thu, Nov 05, 2009 at 12:03:01PM -0500, Tom Horsley wrote:
I have some very weird behavior with my new ubuntu 9.10
virtual machines running as KVMs under fedora 11.
I run testbeds on lots of different linuxii in ssh sessions
to virtual machines, and the ssh session just stops randomly
when talking to ubuntu 9.10 (no problem with 9.04, or 8.10,
no problem with various fedora versions or openSUSE versions).
Do the guests have libvirtd running inside them? libvirtd tends to
create a default network (192.168.122.*) which conflicts with the host
virbr0 network. This usually breaks networking completely -- I've not
seen is cause hangs on an existing working network, unless ssh is
hanging while something is starting up libvirtd.
https://bugzilla.redhat.com/show_bug.cgi?id=509458
If not that, then these problems are often caused by specific classes
of packets getting through (eg. packets that are too large, have
broken checksums etc). So as Mark said, a tcpdump is vital to solve
the issue.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
11:35 a.m.
On Tue, Nov 10, 2009 at 05:31:10PM +0000, Richard W.M. Jones wrote:
of packets getting through (eg. packets that are too large, have
... not getting through ...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
12:25 p.m.
On Tue, 10 Nov 2009 17:31:10 +0000
Richard W.M. Jones wrote:
So as Mark said, a tcpdump is vital to solve
the issue.
I still have no ideas why the virt-viewer console stopped working
at the same time, but going through logs on the guest machines
showed $#@! NetworkManager deciding it needed to do a brand
new DHCPDISCOVER for the network connection which was already up,
resulting in the IP address changing out from under my original
connection. After much cursing and poking of sticks at
NetworkManager I think the connection is now staying up. Why
my "real hardware" ubuntu 9.10 machine doesn't suffer from the
same problem, I don't know.
5290
days inactive
5295
days old
8 comments
3 participants
participants (3)
-
Mark McLoughlin -
Richard W.M. Jones -
Tom Horsley