I posted this issue on the users mailing list but didn't get a response ( https://lists.fedoraproject.org/pipermail/users/2015-March/459083.html ), so I thought I'd try here.
I did a "yum remove" on F21 and it hung after the 2nd of 8 .rpms and systemd has been using ~40% of the CPU since then. Here's the stacktrace from yum:
#0 0xb76debac in __kernel_vsyscall () #1 0xb7510d03 in __waitpid_nocancel () from /lib/libpthread.so.0 #2 0xb6fa4842 in rpmScriptRun () from /lib/librpm.so.3 #3 0xb6f83c53 in runScript () from /lib/librpm.so.3 #4 0xb6f8434f in runInstScript () from /lib/librpm.so.3 #5 0xb6f8531b in rpmpsmRun () from /lib/librpm.so.3 #6 0xb6f9a3cb in rpmteProcess () from /lib/librpm.so.3 #7 0xb6fa1714 in rpmtsRun () from /lib/librpm.so.3 #8 0xb6fd0f0a in rpmts_Run () from /usr/lib/python2.7/site-packages/rpm/_rpm.so #9 0xb758a609 in PyCFunction_Call () from /lib/libpython2.7.so.1.0 #10 0xb754a645 in PyObject_Call () from /lib/libpython2.7.so.1.0 #11 0xb75e6f33 in PyEval_CallObjectWithKeywords () from /lib/libpython2.7.so.1.0 #12 0xb75616c6 in methoddescr_call () from /lib/libpython2.7.so.1.0 #13 0xb754a645 in PyObject_Call () from /lib/libpython2.7.so.1.0 #14 0xb75eb187 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #15 0xb75ecfe1 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #16 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #17 0xb75ecf11 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #18 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #19 0xb75ecf11 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #20 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #21 0xb75ecf11 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #22 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #23 0xb75ecf11 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #24 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #25 0xb75ee344 in PyEval_EvalCode () from /lib/libpython2.7.so.1.0 #26 0xb76078db in run_mod () from /lib/libpython2.7.so.1.0 #27 0xb7608d70 in PyRun_FileExFlags () from /lib/libpython2.7.so.1.0 #28 0xb760a163 in PyRun_SimpleFileExFlags () from /lib/libpython2.7.so.1.0 #29 0xb760a6c8 in PyRun_AnyFileExFlags () from /lib/libpython2.7.so.1.0 #30 0xb761c911 in Py_Main () from /lib/libpython2.7.so.1.0 #31 0x08048578 in main ()
It looks like yum is waiting on some process. Is there a way I can tell what the process is and why it hasn't returned yet? Any other ideas on how I can figure out what is going wrong?
Thanks, Dave
On 03/10/2015 03:13 AM, Dave Johansen wrote:
#0 0xb76debac in __kernel_vsyscall () #1 0xb7510d03 in __waitpid_nocancel () from /lib/libpthread.so.0 #2 0xb6fa4842 in rpmScriptRun () from /lib/librpm.so.3 #3 0xb6f83c53 in runScript () from /lib/librpm.so.3 #4 0xb6f8434f in runInstScript () from /lib/librpm.so.3 #5 0xb6f8531b in rpmpsmRun () from /lib/librpm.so.3 #6 0xb6f9a3cb in rpmteProcess () from /lib/librpm.so.3 #7 0xb6fa1714 in rpmtsRun () from /lib/librpm.so.3 [...] It looks like yum is waiting on some process.
It's waiting for a package scriptlet to finish.
Is there a way I can tell what the process is and why it hasn't returned yet? Any other ideas on how I can figure out what is going wrong?
You need to look for the child processes of yum and see what they're doing.
Michal
On Tue, Mar 10, 2015 at 5:10 AM, Michal Schmidt mschmidt@redhat.com wrote:
On 03/10/2015 03:13 AM, Dave Johansen wrote:
#0 0xb76debac in __kernel_vsyscall () #1 0xb7510d03 in __waitpid_nocancel () from /lib/libpthread.so.0 #2 0xb6fa4842 in rpmScriptRun () from /lib/librpm.so.3 #3 0xb6f83c53 in runScript () from /lib/librpm.so.3 #4 0xb6f8434f in runInstScript () from /lib/librpm.so.3 #5 0xb6f8531b in rpmpsmRun () from /lib/librpm.so.3 #6 0xb6f9a3cb in rpmteProcess () from /lib/librpm.so.3 #7 0xb6fa1714 in rpmtsRun () from /lib/librpm.so.3 [...] It looks like yum is waiting on some process.
It's waiting for a package scriptlet to finish.
Here's the contents of the script that it's waiting on: if [ $1 -eq 0 ] ; then # Package removal, not upgrade systemctl --no-reload disable minetest@default.service > /dev/null 2>&1 || : systemctl stop minetest@default.service > /dev/null 2>&1 || : fi
On Tue, Mar 10, 2015 at 7:53 PM, Dave Johansen davejohansen@gmail.com wrote:
On Tue, Mar 10, 2015 at 5:10 AM, Michal Schmidt mschmidt@redhat.com wrote:
On 03/10/2015 03:13 AM, Dave Johansen wrote:
#0 0xb76debac in __kernel_vsyscall () #1 0xb7510d03 in __waitpid_nocancel () from /lib/libpthread.so.0 #2 0xb6fa4842 in rpmScriptRun () from /lib/librpm.so.3 #3 0xb6f83c53 in runScript () from /lib/librpm.so.3 #4 0xb6f8434f in runInstScript () from /lib/librpm.so.3 #5 0xb6f8531b in rpmpsmRun () from /lib/librpm.so.3 #6 0xb6f9a3cb in rpmteProcess () from /lib/librpm.so.3 #7 0xb6fa1714 in rpmtsRun () from /lib/librpm.so.3 [...] It looks like yum is waiting on some process.
It's waiting for a package scriptlet to finish.
Here's the contents of the script that it's waiting on: if [ $1 -eq 0 ] ; then # Package removal, not upgrade systemctl --no-reload disable minetest@default.service > /dev/null 2>&1 || : systemctl stop minetest@default.service > /dev/null 2>&1 || : fi
Sorry for the multiple emails. It's stuck on the call to stop the service. When I run status, I prints the following: . minetest@default.service - Minetest multiplayer server w/ default.conf server config Loaded: loaded (/usr/lib/system/system/minetest@.service; disabled) Active: inactive (dead)
Any recommendations on what I should do next to diagnose the source of the problem?
Thanks, Dave
On Tue, Mar 10, 2015 at 08:15:38PM -0700, Dave Johansen wrote:
On Tue, Mar 10, 2015 at 7:53 PM, Dave Johansen davejohansen@gmail.com wrote:
On Tue, Mar 10, 2015 at 5:10 AM, Michal Schmidt mschmidt@redhat.com wrote:
On 03/10/2015 03:13 AM, Dave Johansen wrote:
#0 0xb76debac in __kernel_vsyscall () #1 0xb7510d03 in __waitpid_nocancel () from /lib/libpthread.so.0 #2 0xb6fa4842 in rpmScriptRun () from /lib/librpm.so.3 #3 0xb6f83c53 in runScript () from /lib/librpm.so.3 #4 0xb6f8434f in runInstScript () from /lib/librpm.so.3 #5 0xb6f8531b in rpmpsmRun () from /lib/librpm.so.3 #6 0xb6f9a3cb in rpmteProcess () from /lib/librpm.so.3 #7 0xb6fa1714 in rpmtsRun () from /lib/librpm.so.3 [...] It looks like yum is waiting on some process.
It's waiting for a package scriptlet to finish.
Here's the contents of the script that it's waiting on: if [ $1 -eq 0 ] ; then # Package removal, not upgrade systemctl --no-reload disable minetest@default.service > /dev/null 2>&1 || : systemctl stop minetest@default.service > /dev/null 2>&1 || : fi
Sorry for the multiple emails. It's stuck on the call to stop the service. When I run status, I prints the following: . minetest@default.service - Minetest multiplayer server w/ default.conf server config Loaded: loaded (/usr/lib/system/system/minetest@.service; disabled) Active: inactive (dead)
Any recommendations on what I should do next to diagnose the source of the problem?
How long does it stay in this state? systemctl times out connections after a timeout (25s ?), so even if it fails to communicate with PID1 for any reason, rpm should still continue at some point.
pstree -ap <pid-of-rpm> should show a list of processes. If it's systemctl that is hanging, can you post the backtrace?
Zbyszek
On Wed, Mar 11, 2015 at 6:40 AM, Zbigniew Jędrzejewski-Szmek < zbyszek@in.waw.pl> wrote:
On Tue, Mar 10, 2015 at 08:15:38PM -0700, Dave Johansen wrote:
On Tue, Mar 10, 2015 at 7:53 PM, Dave Johansen davejohansen@gmail.com wrote:
On Tue, Mar 10, 2015 at 5:10 AM, Michal Schmidt mschmidt@redhat.com wrote:
On 03/10/2015 03:13 AM, Dave Johansen wrote:
#0 0xb76debac in __kernel_vsyscall () #1 0xb7510d03 in __waitpid_nocancel () from /lib/libpthread.so.0 #2 0xb6fa4842 in rpmScriptRun () from /lib/librpm.so.3 #3 0xb6f83c53 in runScript () from /lib/librpm.so.3 #4 0xb6f8434f in runInstScript () from /lib/librpm.so.3 #5 0xb6f8531b in rpmpsmRun () from /lib/librpm.so.3 #6 0xb6f9a3cb in rpmteProcess () from /lib/librpm.so.3 #7 0xb6fa1714 in rpmtsRun () from /lib/librpm.so.3 [...] It looks like yum is waiting on some process.
It's waiting for a package scriptlet to finish.
Here's the contents of the script that it's waiting on: if [ $1 -eq 0 ] ; then # Package removal, not upgrade systemctl --no-reload disable minetest@default.service > /dev/null
2>&1
|| : systemctl stop minetest@default.service > /dev/null 2>&1 || : fi
Sorry for the multiple emails. It's stuck on the call to stop the
service.
When I run status, I prints the following: . minetest@default.service - Minetest multiplayer server w/ default.conf server config Loaded: loaded (/usr/lib/system/system/minetest@.service; disabled) Active: inactive (dead)
Any recommendations on what I should do next to diagnose the source of
the
problem?
How long does it stay in this state? systemctl times out connections after a timeout (25s ?), so even if it fails to communicate with PID1 for any reason, rpm should still continue at some point.
pstree -ap <pid-of-rpm> should show a list of processes. If it's systemctl that is hanging, can you post the backtrace?
It had been stuck for ~3 days and last night I got sick of my computer being sluggish, so I started trying to see if I could get it to break free. I should have gotten a stacktrace for systemctl first, but didn't think of that and here's what happened:
I ran "systemctl start minetest@default.service" Then after a few seconds, the system went into what I believe was "suspend mode" (screen went blank and I had to hit the power button to get things to come back). I then hit Ctrl-C after about 30 seconds because it appeared that the service wasn't actually starting (it technically was in the process of being removed after all).
After that, the CPU usage of systemd dropped back to being close to 0 and the status output the following: ● minetest@default.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead)
Mar 10 20:16:19 JohansenDev systemd[1]: Failed to start minetest@default.service.
On Wed, Mar 11, 2015 at 08:23:43PM -0700, Dave Johansen wrote:
On Wed, Mar 11, 2015 at 6:40 AM, Zbigniew Jędrzejewski-Szmek < zbyszek@in.waw.pl> wrote:
On Tue, Mar 10, 2015 at 08:15:38PM -0700, Dave Johansen wrote:
On Tue, Mar 10, 2015 at 7:53 PM, Dave Johansen davejohansen@gmail.com wrote:
On Tue, Mar 10, 2015 at 5:10 AM, Michal Schmidt mschmidt@redhat.com wrote:
On 03/10/2015 03:13 AM, Dave Johansen wrote:
#0 0xb76debac in __kernel_vsyscall () #1 0xb7510d03 in __waitpid_nocancel () from /lib/libpthread.so.0 #2 0xb6fa4842 in rpmScriptRun () from /lib/librpm.so.3 #3 0xb6f83c53 in runScript () from /lib/librpm.so.3 #4 0xb6f8434f in runInstScript () from /lib/librpm.so.3 #5 0xb6f8531b in rpmpsmRun () from /lib/librpm.so.3 #6 0xb6f9a3cb in rpmteProcess () from /lib/librpm.so.3 #7 0xb6fa1714 in rpmtsRun () from /lib/librpm.so.3 [...] It looks like yum is waiting on some process.
It's waiting for a package scriptlet to finish.
Here's the contents of the script that it's waiting on: if [ $1 -eq 0 ] ; then # Package removal, not upgrade systemctl --no-reload disable minetest@default.service > /dev/null
2>&1
|| : systemctl stop minetest@default.service > /dev/null 2>&1 || : fi
Sorry for the multiple emails. It's stuck on the call to stop the
service.
When I run status, I prints the following: . minetest@default.service - Minetest multiplayer server w/ default.conf server config Loaded: loaded (/usr/lib/system/system/minetest@.service; disabled) Active: inactive (dead)
Any recommendations on what I should do next to diagnose the source of
the
problem?
How long does it stay in this state? systemctl times out connections after a timeout (25s ?), so even if it fails to communicate with PID1 for any reason, rpm should still continue at some point.
pstree -ap <pid-of-rpm> should show a list of processes. If it's systemctl that is hanging, can you post the backtrace?
It had been stuck for ~3 days and last night I got sick of my computer being sluggish, so I started trying to see if I could get it to break free. I should have gotten a stacktrace for systemctl first, but didn't think of that and here's what happened:
I ran "systemctl start minetest@default.service" Then after a few seconds, the system went into what I believe was "suspend mode" (screen went blank and I had to hit the power button to get things to come back). I then hit Ctrl-C after about 30 seconds because it appeared that the service wasn't actually starting (it technically was in the process of being removed after all).
After that, the CPU usage of systemd dropped back to being close to 0 and the status output the following: ● minetest@default.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead)
Mar 10 20:16:19 JohansenDev systemd[1]: Failed to start minetest@default.service.
This could be somehow related to a restart of unit after it's file has been removed. We had some bugs in that area in the past.
Sorry, can't say what the problem was.
Zbyszek
Dave Johansen davejohansen@gmail.com writes:
I posted this issue on the users mailing list but didn't get a response ( https://lists.fedoraproject.org/pipermail/users/2015-March/459083.html ), so I thought I'd try here.
I did a "yum remove" on F21 and it hung after the 2nd of 8 .rpms and systemd has been using ~40% of the CPU since then. Here's the stacktrace from yum:
#0 0xb76debac in __kernel_vsyscall () #1 0xb7510d03 in __waitpid_nocancel () from /lib/libpthread.so.0 #2 0xb6fa4842 in rpmScriptRun () from /lib/librpm.so.3 #3 0xb6f83c53 in runScript () from /lib/librpm.so.3 #4 0xb6f8434f in runInstScript () from /lib/librpm.so.3 #5 0xb6f8531b in rpmpsmRun () from /lib/librpm.so.3 #6 0xb6f9a3cb in rpmteProcess () from /lib/librpm.so.3 #7 0xb6fa1714 in rpmtsRun () from /lib/librpm.so.3 #8 0xb6fd0f0a in rpmts_Run () from /usr/lib/python2.7/site-packages/rpm/_rpm.so #9 0xb758a609 in PyCFunction_Call () from /lib/libpython2.7.so.1.0 #10 0xb754a645 in PyObject_Call () from /lib/libpython2.7.so.1.0 #11 0xb75e6f33 in PyEval_CallObjectWithKeywords () from /lib/libpython2.7.so.1.0 #12 0xb75616c6 in methoddescr_call () from /lib/libpython2.7.so.1.0 #13 0xb754a645 in PyObject_Call () from /lib/libpython2.7.so.1.0 #14 0xb75eb187 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #15 0xb75ecfe1 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #16 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #17 0xb75ecf11 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #18 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #19 0xb75ecf11 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #20 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #21 0xb75ecf11 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #22 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #23 0xb75ecf11 in PyEval_EvalFrameEx () from /lib/libpython2.7.so.1.0 #24 0xb75ee1ea in PyEval_EvalCodeEx () from /lib/libpython2.7.so.1.0 #25 0xb75ee344 in PyEval_EvalCode () from /lib/libpython2.7.so.1.0 #26 0xb76078db in run_mod () from /lib/libpython2.7.so.1.0 #27 0xb7608d70 in PyRun_FileExFlags () from /lib/libpython2.7.so.1.0 #28 0xb760a163 in PyRun_SimpleFileExFlags () from /lib/libpython2.7.so.1.0 #29 0xb760a6c8 in PyRun_AnyFileExFlags () from /lib/libpython2.7.so.1.0 #30 0xb761c911 in Py_Main () from /lib/libpython2.7.so.1.0 #31 0x08048578 in main ()
It looks like yum is waiting on some process. Is there a way I can tell what the process is and why it hasn't returned yet? Any other ideas on how I can figure out what is going wrong?
Thanks, Dave
This is somewhat reminiscent of [1]. Any chance that the process that actually uses the cpu is dbus related?
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1186018
Cheers,
-
Dave Johansen -
Jan Synacek -
Michal Schmidt -
Zbigniew Jędrzejewski-Szmek