10:38 a.m.
Cleanup : cpp-4.8.0-0.7.fc19.x86_64 215/262
Cleanup : gdb-7.5.50.20130118-2.fc19.x86_64 216/262
Cleanup : 1:findutils-4.5.10-7.fc19.x86_64 217/262
Cleanup : spice-server-0.12.2-2.fc19.x86_64 218/262
Cleanup : cracklib-2.8.22-2.fc19.x86_64 219/262
Cleanup : libvirt-daemon-driver-interface-1.0.1-6.fc19.x86_64 220/262
Cleanup : libvirt-daemon-driver-nodedev-1.0.1-6.fc19.x86_64 221/262
Cleanup : libvirt-daemon-driver-nwfilter-1.0.1-6.fc19.x86_64 222/262
Cleanup : libvirt-daemon-driver-secret-1.0.1-6.fc19.x86_64 223/262
Cleanup : libvirt-daemon-1.0.1-6.fc19.x86_64 224/262
Cleanup : libvirt-client-1.0.1-6.fc19.x86_64 225/262
Cleanup : cyrus-sasl-2.1.25-2.fc19.x86_64 226/262
Cleanup : openldap-2.4.33-3.fc19.x86_64 227/262
Cleanup : nss-tools-3.14.1-3.fc19.x86_64 228/262
Cleanup : nss-sysinit-3.14.1-3.fc19.x86_64 229/262
Cleanup : nss-3.14.1-3.fc19.x86_64 230/262
(and here it hangs, for at least 20 minutes)
There's nothing obviously going on inside the virtual machine. Load
average is 0.00. The VM is still perfectly responsive, plenty of
free memory and so on.
It appears as if it's yum itself which is stuck:
4223 ? Ss 0:00 \_ sshd: rjones [priv]
4226 ? S 0:09 \_ sshd: rjones@pts/1
4227 pts/1 Ss 0:00 \_ -bash
13396 pts/1 S+ 0:00 \_ sudo yum update
13397 pts/1 Sl+ 1:01 \_ /usr/bin/python /bin/yum update
Yum is blocked on a futex:
$ sudo strace -p 13397
Process 13397 attached
futex(0x7f908292fc0c, FUTEX_WAIT, 1, NULL
Is there a way I can see what (if anything) is holding on to this
mutex?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
10:41 a.m.
On Mon, Feb 04, 2013 at 04:38:08PM +0000, Richard W.M. Jones wrote:
Yum is blocked on a futex:
I should add:
yum-3.4.3-57.fc19.noarch
Rawhide 64 bit, almost(!) up to date.
It looks as if rpm-libs was also updated moments earlier to
4.11.0-0.beta1.3.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
10:43 a.m.
On Mon, Feb 04, 2013 at 04:41:25PM +0000, Richard W.M. Jones wrote:
It looks as if rpm-libs was also updated moments earlier to
4.11.0-0.beta1.3.
Correction: RPM was just updated from 4.11.0-0.beta1.3 to 4.11.0.1-1.
It was beta1.2 which was the dodgy version, but I didn't have
that installed at any point.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
11:21 a.m.
I've not run into this on my rawhide laptop or rawhide vm today...
Perhaps some kind of race condition there?
kevin
11:01 a.m.
On Mon, Feb 04, 2013 at 04:38:08PM +0000, Richard W.M. Jones wrote:
Cleanup : cpp-4.8.0-0.7.fc19.x86_64 215/262
Cleanup : gdb-7.5.50.20130118-2.fc19.x86_64 216/262
Cleanup : 1:findutils-4.5.10-7.fc19.x86_64 217/262
Cleanup : spice-server-0.12.2-2.fc19.x86_64 218/262
Cleanup : cracklib-2.8.22-2.fc19.x86_64 219/262
Cleanup : libvirt-daemon-driver-interface-1.0.1-6.fc19.x86_64 220/262
Cleanup : libvirt-daemon-driver-nodedev-1.0.1-6.fc19.x86_64 221/262
Cleanup : libvirt-daemon-driver-nwfilter-1.0.1-6.fc19.x86_64 222/262
Cleanup : libvirt-daemon-driver-secret-1.0.1-6.fc19.x86_64 223/262
Cleanup : libvirt-daemon-1.0.1-6.fc19.x86_64 224/262
Cleanup : libvirt-client-1.0.1-6.fc19.x86_64 225/262
Cleanup : cyrus-sasl-2.1.25-2.fc19.x86_64 226/262
Cleanup : openldap-2.4.33-3.fc19.x86_64 227/262
Cleanup : nss-tools-3.14.1-3.fc19.x86_64 228/262
Cleanup : nss-sysinit-3.14.1-3.fc19.x86_64 229/262
Cleanup : nss-3.14.1-3.fc19.x86_64 230/262
(and here it hangs, for at least 20 minutes)
So how odd is this? Suddenly it leaps back into life, after maybe
30-40 minutes.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
11:17 a.m.
On 02/04/2013 07:01 PM, Richard W.M. Jones wrote:
On Mon, Feb 04, 2013 at 04:38:08PM +0000, Richard W.M. Jones wrote:
>
> Cleanup : cpp-4.8.0-0.7.fc19.x86_64 215/262
> Cleanup : gdb-7.5.50.20130118-2.fc19.x86_64 216/262
> Cleanup : 1:findutils-4.5.10-7.fc19.x86_64 217/262
> Cleanup : spice-server-0.12.2-2.fc19.x86_64 218/262
> Cleanup : cracklib-2.8.22-2.fc19.x86_64 219/262
> Cleanup : libvirt-daemon-driver-interface-1.0.1-6.fc19.x86_64 220/262
> Cleanup : libvirt-daemon-driver-nodedev-1.0.1-6.fc19.x86_64 221/262
> Cleanup : libvirt-daemon-driver-nwfilter-1.0.1-6.fc19.x86_64 222/262
> Cleanup : libvirt-daemon-driver-secret-1.0.1-6.fc19.x86_64 223/262
> Cleanup : libvirt-daemon-1.0.1-6.fc19.x86_64 224/262
> Cleanup : libvirt-client-1.0.1-6.fc19.x86_64 225/262
> Cleanup : cyrus-sasl-2.1.25-2.fc19.x86_64 226/262
> Cleanup : openldap-2.4.33-3.fc19.x86_64 227/262
> Cleanup : nss-tools-3.14.1-3.fc19.x86_64 228/262
> Cleanup : nss-sysinit-3.14.1-3.fc19.x86_64 229/262
> Cleanup : nss-3.14.1-3.fc19.x86_64 230/262
> (and here it hangs, for at least 20 minutes)
So how odd is this? Suddenly it leaps back into life, after maybe
30-40 minutes.
Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=860500
- Panu -
4:32 p.m.
On Mon, Feb 04, 2013 at 07:17:35PM +0200, Panu Matilainen wrote:
On 02/04/2013 07:01 PM, Richard W.M. Jones wrote:
>On Mon, Feb 04, 2013 at 04:38:08PM +0000, Richard W.M. Jones wrote:
>>
>> Cleanup : cpp-4.8.0-0.7.fc19.x86_64 215/262
>> Cleanup : gdb-7.5.50.20130118-2.fc19.x86_64 216/262
>> Cleanup : 1:findutils-4.5.10-7.fc19.x86_64 217/262
>> Cleanup : spice-server-0.12.2-2.fc19.x86_64 218/262
>> Cleanup : cracklib-2.8.22-2.fc19.x86_64 219/262
>> Cleanup : libvirt-daemon-driver-interface-1.0.1-6.fc19.x86_64 220/262
>> Cleanup : libvirt-daemon-driver-nodedev-1.0.1-6.fc19.x86_64 221/262
>> Cleanup : libvirt-daemon-driver-nwfilter-1.0.1-6.fc19.x86_64 222/262
>> Cleanup : libvirt-daemon-driver-secret-1.0.1-6.fc19.x86_64 223/262
>> Cleanup : libvirt-daemon-1.0.1-6.fc19.x86_64 224/262
>> Cleanup : libvirt-client-1.0.1-6.fc19.x86_64 225/262
>> Cleanup : cyrus-sasl-2.1.25-2.fc19.x86_64 226/262
>> Cleanup : openldap-2.4.33-3.fc19.x86_64 227/262
>> Cleanup : nss-tools-3.14.1-3.fc19.x86_64 228/262
>> Cleanup : nss-sysinit-3.14.1-3.fc19.x86_64 229/262
>> Cleanup : nss-3.14.1-3.fc19.x86_64 230/262
>>(and here it hangs, for at least 20 minutes)
>
>So how odd is this? Suddenly it leaps back into life, after maybe
>30-40 minutes.
Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=860500
Yes, this looks similar.
It's possible that I ran a non-root yum command in another terminal.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
1:24 a.m.
On 02/05/2013 12:32 AM, Richard W.M. Jones wrote:
On Mon, Feb 04, 2013 at 07:17:35PM +0200, Panu Matilainen wrote:
> On 02/04/2013 07:01 PM, Richard W.M. Jones wrote:
>> On Mon, Feb 04, 2013 at 04:38:08PM +0000, Richard W.M. Jones wrote:
>>>
>>> Cleanup : cpp-4.8.0-0.7.fc19.x86_64
215/262
>>> Cleanup : gdb-7.5.50.20130118-2.fc19.x86_64
216/262
>>> Cleanup : 1:findutils-4.5.10-7.fc19.x86_64
217/262
>>> Cleanup : spice-server-0.12.2-2.fc19.x86_64
218/262
>>> Cleanup : cracklib-2.8.22-2.fc19.x86_64
219/262
>>> Cleanup : libvirt-daemon-driver-interface-1.0.1-6.fc19.x86_64
220/262
>>> Cleanup : libvirt-daemon-driver-nodedev-1.0.1-6.fc19.x86_64
221/262
>>> Cleanup : libvirt-daemon-driver-nwfilter-1.0.1-6.fc19.x86_64
222/262
>>> Cleanup : libvirt-daemon-driver-secret-1.0.1-6.fc19.x86_64
223/262
>>> Cleanup : libvirt-daemon-1.0.1-6.fc19.x86_64
224/262
>>> Cleanup : libvirt-client-1.0.1-6.fc19.x86_64
225/262
>>> Cleanup : cyrus-sasl-2.1.25-2.fc19.x86_64
226/262
>>> Cleanup : openldap-2.4.33-3.fc19.x86_64
227/262
>>> Cleanup : nss-tools-3.14.1-3.fc19.x86_64
228/262
>>> Cleanup : nss-sysinit-3.14.1-3.fc19.x86_64
229/262
>>> Cleanup : nss-3.14.1-3.fc19.x86_64
230/262
>>> (and here it hangs, for at least 20 minutes)
>>
>> So how odd is this? Suddenly it leaps back into life, after maybe
>> 30-40 minutes.
>
> Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=860500
Yes, this looks similar.
It's possible that I ran a non-root yum command in another terminal.
A non-root yum/rpm/similar command wouldn't do. Only processes running
as root can participate in the shared environment (those __db.* files)
locking, others use a "private environment" which pretty much equals to
no locking at all.
So whatever it is that causes the jam is running as root, and equally
only a root-process can unjam it. Could even be the same thing that
caused the jam re-running, it's quite clearly something that runs
automatically in the background and does so more or less periodically,
occasionally exiting or crashing without freeing the rpmdb iterator it
holds. Whether its time-based or triggered by some other "external"
event I dunno. And when it causes a jam its either still running while
yum is started, or has started after yum.
Rpm uses Berkeley DB's "Concurrent Data Store" model for its database.
This is a simple model which supposedly provides a deadlock-free
operation without caller having to bother with explicit locking, but
unfortunately this only works when all callers are well-behaved. Not
entirely unlike multitasking in Windows 3.x... All it takes a single
buggy application forgetting to release its rpmdb iterators (or crashing
while holding them) to block a concurrent writer "forever". Stale locks
from no longer active processes are automatically cleaned but only on
rpmdb open, so a potentially long-running application like yum can get
stuck if the bad apple comes along after yum started.
Come to think of it, it should be possible to have rpm check for stale
locks when opening write-cursors. That would help some of the cases
(where the bad caller already exited/died) at least, but it'd still be
"vulnerable" to long-running process hanging on to iterators.
- Panu -
6:10 p.m.
On Mon, 2013-02-04 at 19:17 +0200, Panu Matilainen wrote:
On 02/04/2013 07:01 PM, Richard W.M. Jones wrote:
> On Mon, Feb 04, 2013 at 04:38:08PM +0000, Richard W.M. Jones wrote:
>>
>> Cleanup : cpp-4.8.0-0.7.fc19.x86_64
215/262
>> Cleanup : gdb-7.5.50.20130118-2.fc19.x86_64
216/262
>> Cleanup : 1:findutils-4.5.10-7.fc19.x86_64
217/262
>> Cleanup : spice-server-0.12.2-2.fc19.x86_64
218/262
>> Cleanup : cracklib-2.8.22-2.fc19.x86_64
219/262
>> Cleanup : libvirt-daemon-driver-interface-1.0.1-6.fc19.x86_64
220/262
>> Cleanup : libvirt-daemon-driver-nodedev-1.0.1-6.fc19.x86_64
221/262
>> Cleanup : libvirt-daemon-driver-nwfilter-1.0.1-6.fc19.x86_64
222/262
>> Cleanup : libvirt-daemon-driver-secret-1.0.1-6.fc19.x86_64
223/262
>> Cleanup : libvirt-daemon-1.0.1-6.fc19.x86_64
224/262
>> Cleanup : libvirt-client-1.0.1-6.fc19.x86_64
225/262
>> Cleanup : cyrus-sasl-2.1.25-2.fc19.x86_64
226/262
>> Cleanup : openldap-2.4.33-3.fc19.x86_64
227/262
>> Cleanup : nss-tools-3.14.1-3.fc19.x86_64
228/262
>> Cleanup : nss-sysinit-3.14.1-3.fc19.x86_64
229/262
>> Cleanup : nss-3.14.1-3.fc19.x86_64
230/262
>> (and here it hangs, for at least 20 minutes)
>
> So how odd is this? Suddenly it leaps back into life, after maybe
> 30-40 minutes.
Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=860500
Sure does. Funnily enough, when I first encountered it no-one else was,
and now other people are seeing it, I never seem to any more :/ So I
haven't been able to contribute any more to the report.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
4104
days inactive
4105
days old
8 comments
4 participants
participants (4)
-
Adam Williamson -
Kevin Fenzi -
Panu Matilainen -
Richard W.M. Jones