9:31 a.m.
Hi,
Since I upgraded my FC6 box from kernel-2.6.20-1.2962.fc6 to
kernel-2.6.22.1-32.fc6, and now 2.6.22.2-42.fc6, I am getting pauses
from the whole OS. They last ~1 second and occur every few minutes.
Every application becomes unresponsive for the duration. The 1 second
scheduler latency this causes is long enough for my music player to be
effected and the audio track to be interrupted. This makes the fault
very easy to hear. At the same time kernel thread events/0 seems to
use all the CPU time. Here is the first few lines of top's output
when a pause happens:
top - 21:39:41 up 1:17, 2 users, load average: 1.05, 1.20, 1.24
Tasks: 138 total, 4 running, 134 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.3%us, 29.3%sy, 68.4%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2074880k total, 1355244k used, 719636k free, 62144k buffers
Swap: 1004052k total, 0k used, 1004052k free, 882684k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5525 mike 39 19 202m 81m 4016 R 68.4 4.0 13:07.46 hadcm3transum_5
6 root 15 -5 0 0 0 S 28.5 0.0 0:53.40 events/0
3619 mike 20 0 45620 8264 5856 S 0.9 0.4 0:18.58 xmms
3318 root 20 0 328m 46m 8132 S 0.6 2.3 1:39.60 Xorg
3524 mike 20 0 76728 23m 9640 S 0.6 1.2 1:13.15 bittorrent
3557 mike 20 0 211m 115m 30m S 0.6 5.7 3:54.24 firefox-bin
3489 mike 20 0 57240 23m 15m S 0.3 1.1 0:04.02 gnome-terminal
5583 root 20 0 2204 1100 832 R 0.3 0.1 0:01.14 top
28.5% CPU time used by events/0 of a single 3 second top refresh is
0.85 seconds of CPU time. Rebooting back to kernel 2.6.20 completely
fixes it.
Has any one else seen this issue?
Does anyone know what kernel thread events/0 does?
Could this be related to CFS newly introduced into Fedora's kernel 2.6.22?
Can anybody suggest how to fix this issue?
Thanks,
Mike
12:30 p.m.
Mike Fleetwood wrote:
Hi,
Since I upgraded my FC6 box from kernel-2.6.20-1.2962.fc6 to
kernel-2.6.22.1-32.fc6, and now 2.6.22.2-42.fc6, I am getting pauses
from the whole OS. They last ~1 second and occur every few minutes.
Every application becomes unresponsive for the duration. The 1 second
scheduler latency this causes is long enough for my music player to be
effected and the audio track to be interrupted. This makes the fault
very easy to hear. At the same time kernel thread events/0 seems to
use all the CPU time. Here is the first few lines of top's output
when a pause happens:
top - 21:39:41 up 1:17, 2 users, load average: 1.05, 1.20, 1.24
Tasks: 138 total, 4 running, 134 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.3%us, 29.3%sy, 68.4%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2074880k total, 1355244k used, 719636k free, 62144k buffers
Swap: 1004052k total, 0k used, 1004052k free, 882684k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5525 mike 39 19 202m 81m 4016 R 68.4 4.0 13:07.46 hadcm3transum_5
6 root 15 -5 0 0 0 S 28.5 0.0 0:53.40 events/0
3619 mike 20 0 45620 8264 5856 S 0.9 0.4 0:18.58 xmms
3318 root 20 0 328m 46m 8132 S 0.6 2.3 1:39.60 Xorg
3524 mike 20 0 76728 23m 9640 S 0.6 1.2 1:13.15 bittorrent
3557 mike 20 0 211m 115m 30m S 0.6 5.7 3:54.24 firefox-bin
3489 mike 20 0 57240 23m 15m S 0.3 1.1 0:04.02 gnome-terminal
5583 root 20 0 2204 1100 832 R 0.3 0.1 0:01.14 top
28.5% CPU time used by events/0 of a single 3 second top refresh is
0.85 seconds of CPU time. Rebooting back to kernel 2.6.20 completely
fixes it.
Has any one else seen this issue?
Does anyone know what kernel thread events/0 does?
Could this be related to CFS newly introduced into Fedora's kernel 2.6.22?
Can anybody suggest how to fix this issue?
Thanks,
Mike
Running kernel.org 2.6.22.5 here on 5 boxes. Don't see this problem on
any of them. events/0 shows no cpu usage (don't know what it is). I
don't believe that CFS is in the stock 2.6.22 kernel. Looks like D
Jones and company might have screwed the pooch.
Regards,
John
2:19 p.m.
New subject: 2.6.22.4-65.fc7 and bcm43xx_mac8021 making wireless love not war
After a yum update that included the 2.6.22.4-65.fc7 kernel update my
Broadcom wireless just started working. Didn't have to do anything. I
noticed that the wireless button was glowing blue so clicked on Network
Manager and selected my Access Point and BINGO I was connected. Bless
all you kernel programmers. Thank you.
Steve Eberhart
HP Pavilion zd7000, now wireless FC7
12:54 a.m.
On 8/27/07, Mike Fleetwood <mike.fleetwood(a)googlemail.com> wrote:
Hi,
Since I upgraded my FC6 box from kernel-2.6.20-1.2962.fc6 to
kernel-2.6.22.1-32.fc6, and now 2.6.22.2-42.fc6, I am getting pauses
from the whole OS. They last ~1 second and occur every few minutes.
Every application becomes unresponsive for the duration. The 1 second
scheduler latency this causes is long enough for my music player to be
effected and the audio track to be interrupted. This makes the fault
very easy to hear. At the same time kernel thread events/0 seems to
use all the CPU time. Here is the first few lines of top's output
when a pause happens:
top - 21:39:41 up 1:17, 2 users, load average: 1.05, 1.20, 1.24
Tasks: 138 total, 4 running, 134 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.3%us, 29.3%sy, 68.4%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2074880k total, 1355244k used, 719636k free, 62144k buffers
Swap: 1004052k total, 0k used, 1004052k free, 882684k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5525 mike 39 19 202m 81m 4016 R 68.4 4.0 13:07.46 hadcm3transum_5
6 root 15 -5 0 0 0 S 28.5 0.0 0:53.40 events/0
3619 mike 20 0 45620 8264 5856 S 0.9 0.4 0:18.58 xmms
3318 root 20 0 328m 46m 8132 S 0.6 2.3 1:39.60 Xorg
3524 mike 20 0 76728 23m 9640 S 0.6 1.2 1:13.15 bittorrent
3557 mike 20 0 211m 115m 30m S 0.6 5.7 3:54.24 firefox-bin
3489 mike 20 0 57240 23m 15m S 0.3 1.1 0:04.02 gnome-terminal
5583 root 20 0 2204 1100 832 R 0.3 0.1 0:01.14 top
28.5% CPU time used by events/0 of a single 3 second top refresh is
0.85 seconds of CPU time. Rebooting back to kernel 2.6.20 completely
fixes it.
Has any one else seen this issue?
Does anyone know what kernel thread events/0 does?
It's a per processor work queue. This is a data structure used to
handle deferred work. As you have only one processor, you get an
events/0 (events/1, etc.. in SMP).
I've never seen that thread taking so much cpu time... maybe a kernel
developer can throw some light here
Could this be related to CFS newly introduced into Fedora's
kernel 2.6.22?
Can anybody suggest how to fix this issue?
Thanks,
Mike
--
fedora-list mailing list
fedora-list(a)redhat.com
To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list
2:52 p.m.
On 27 Aug 2007, Mike Fleetwood wrote:
Hi,
Since I upgraded my FC6 box from kernel-2.6.20-1.2962.fc6 to
kernel-2.6.22.1-32.fc6, and now 2.6.22.2-42.fc6, I am getting pauses
from the whole OS. They last ~1 second and occur every few minutes.
Every application becomes unresponsive for the duration. The 1 second
scheduler latency this causes is long enough for my music player to be
effected and the audio track to be interrupted. This makes the fault
very easy to hear. At the same time kernel thread events/0 seems to
use all the CPU time. Here is the first few lines of top's output
when a pause happens:
top - 21:39:41 up 1:17, 2 users, load average: 1.05, 1.20, 1.24
Tasks: 138 total, 4 running, 134 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.3%us, 29.3%sy, 68.4%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2074880k total, 1355244k used, 719636k free, 62144k buffers
Swap: 1004052k total, 0k used, 1004052k free, 882684k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5525 mike 39 19 202m 81m 4016 R 68.4 4.0 13:07.46 hadcm3transum_5
6 root 15 -5 0 0 0 S 28.5 0.0 0:53.40 events/0
3619 mike 20 0 45620 8264 5856 S 0.9 0.4 0:18.58 xmms
3318 root 20 0 328m 46m 8132 S 0.6 2.3 1:39.60 Xorg
3524 mike 20 0 76728 23m 9640 S 0.6 1.2 1:13.15 bittorrent
3557 mike 20 0 211m 115m 30m S 0.6 5.7 3:54.24 firefox-bin
3489 mike 20 0 57240 23m 15m S 0.3 1.1 0:04.02 gnome-terminal
5583 root 20 0 2204 1100 832 R 0.3 0.1 0:01.14 top
28.5% CPU time used by events/0 of a single 3 second top refresh is
0.85 seconds of CPU time. Rebooting back to kernel 2.6.20 completely
fixes it.
Has any one else seen this issue?
Does anyone know what kernel thread events/0 does?
Could this be related to CFS newly introduced into Fedora's kernel 2.6.22?
Can anybody suggest how to fix this issue?
Thanks for the replies so far.
1) Seems no one else has seen this issue.
2) Kernel has one events thread per CPU, numbered 0 upwards. They are
used to get kernel work do a little later, for example a device drive
interrupt handler might use one. Ref:
http://docs.blackfin.uclinux.org/doku.php?id=kernel_events
3) I have also just compiled Linus' kernel 2.6.22.2 and get the same
OS wide 1 second pauses. Therefore it's not a Fedora specific patch,
such as CFS, causing the fault.
4) Now off to try a binary search of Linus' kernel releases between
2.6.20 and 2.6.22.2 to see when it got introduced.
Still could be a kernel software fault, but as it seems no one else
has this issue perhaps it is buggy hardware the kernel no longer
handles as well. (I don't think that I run any unusual software or
uncommon hardware). Could be a while before I finish binary searching
kernel releases.
Mike
3:30 p.m.
Somebody in the thread at some point said:
On 27 Aug 2007, Mike Fleetwood wrote:
> Hi,
>
> Since I upgraded my FC6 box from kernel-2.6.20-1.2962.fc6 to
> kernel-2.6.22.1-32.fc6, and now 2.6.22.2-42.fc6, I am getting pauses
> from the whole OS. They last ~1 second and occur every few minutes.
How about running
slabtop
for a while and watch if anything is radically changing during the pause.
Also, is it a laptop? I saw this kind of behaviour a long time ago when
it was bitbanging to the battery sensor to find out the charge level.
-Andy
3:13 p.m.
On 29 Aug 2007, Andy Green wrote:
How about running
slabtop
for a while and watch if anything is radically changing during the pause.
Also, is it a laptop? I saw this kind of behaviour a long time ago when
it was bitbanging to the battery sensor to find out the charge level.
-Andy
I will give slabtop a go, but slabtop requires /proc/slabinfo which
requires kernel to be compiled with option CONFIG_SLAB which the
Fedora kernel and Linus' I compiled using the same configuration file
don't have. I will add it to future compiles of the kernel.
No this is a desktop. AMD Athlon 64 3200+ CPU and ASUS A8V Delux
motherboard. Still using 32bit Fedora and kernel though.
Thanks,
Mike
3:28 p.m.
Somebody in the thread at some point said:
I will give slabtop a go, but slabtop requires /proc/slabinfo which
requires kernel to be compiled with option CONFIG_SLAB which the
Fedora kernel and Linus' I compiled using the same configuration file
don't have. I will add it to future compiles of the kernel.
Hum seems Fedora uses the SLUB alternative. The kernel I am running
right now uses SLAB so.
-Andy
3:12 a.m.
On 27 Aug 2007, Mike Fleetwood wrote:
On 27 Aug 2007, Mike Fleetwood wrote:
> Hi,
>
> Since I upgraded my FC6 box from kernel-2.6.20-1.2962.fc6 to
> kernel-2.6.22.1-32.fc6, and now 2.6.22.2-42.fc6, I am getting pauses
> from the whole OS. They last ~1 second and occur every few minutes.
> Every application becomes unresponsive for the duration. The 1 second
> scheduler latency this causes is long enough for my music player to be
> effected and the audio track to be interrupted. This makes the fault
> very easy to hear. At the same time kernel thread events/0 seems to
> use all the CPU time. Here is the first few lines of top's output
> when a pause happens:
> top - 21:39:41 up 1:17, 2 users, load average: 1.05, 1.20, 1.24
> Tasks: 138 total, 4 running, 134 sleeping, 0 stopped, 0 zombie
> Cpu(s): 2.3%us, 29.3%sy, 68.4%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2074880k total, 1355244k used, 719636k free, 62144k buffers
> Swap: 1004052k total, 0k used, 1004052k free, 882684k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 5525 mike 39 19 202m 81m 4016 R 68.4 4.0 13:07.46 hadcm3transum_5
> 6 root 15 -5 0 0 0 S 28.5 0.0 0:53.40 events/0
> 3619 mike 20 0 45620 8264 5856 S 0.9 0.4 0:18.58 xmms
> 3318 root 20 0 328m 46m 8132 S 0.6 2.3 1:39.60 Xorg
> 3524 mike 20 0 76728 23m 9640 S 0.6 1.2 1:13.15 bittorrent
> 3557 mike 20 0 211m 115m 30m S 0.6 5.7 3:54.24 firefox-bin
> 3489 mike 20 0 57240 23m 15m S 0.3 1.1 0:04.02 gnome-terminal
> 5583 root 20 0 2204 1100 832 R 0.3 0.1 0:01.14 top
> 28.5% CPU time used by events/0 of a single 3 second top refresh is
> 0.85 seconds of CPU time. Rebooting back to kernel 2.6.20 completely
> fixes it.
>
> Has any one else seen this issue?
> Does anyone know what kernel thread events/0 does?
> Could this be related to CFS newly introduced into Fedora's kernel 2.6.22?
> Can anybody suggest how to fix this issue?
Thanks for the replies so far.
1) Seems no one else has seen this issue.
2) Kernel has one events thread per CPU, numbered 0 upwards. They are
used to get kernel work do a little later, for example a device drive
interrupt handler might use one. Ref:
http://docs.blackfin.uclinux.org/doku.php?id=kernel_events
3) I have also just compiled Linus' kernel 2.6.22.2 and get the same
OS wide 1 second pauses. Therefore it's not a Fedora specific patch,
such as CFS, causing the fault.
4) Now off to try a binary search of Linus' kernel releases between
2.6.20 and 2.6.22.2 to see when it got introduced.
Still could be a kernel software fault, but as it seems no one else
has this issue perhaps it is buggy hardware the kernel no longer
handles as well. (I don't think that I run any unusual software or
uncommon hardware). Could be a while before I finish binary searching
kernel releases.
After lots of kernel compiling and testing I finally tracked down
the
causes ...
1) Between Fedora kernels 2.6.20-1.2962.fc6 and 2.6.22.1-32.fc6 this
configuration change was the one which triggered the ~1 second OS pauses
to appear:
diff -y /boot/config-2.6.20-1.2962.fc6 /boot/config-2.6.22.1-22.fc6
...
CONFIG_RTC=y | # CONFIG_RTC is not set
CONFIG_GEN_RTC=y
CONFIG_GEN_RTC_X=y
...
2) I also run chrony (http://chrony.sunsite.dk/) rather than ntp to time
synchronise my machine. (It monitors and adjusts the hardware RTC as
well as the kernel software clock, hence accesses /dev/rtc). Running
chrony is probably very rare, hence why no one else has seen this issue.
I still want to understand what the above kernel configuration changes
actually do and which RTC related OS calls chrony is making to cause the
events thread to hog the CPU solidly for ~1 second.
Mike
6067
days inactive
6080
days old
8 comments
5 participants
participants (5)
-
Andy Green -
Fernando Apesteguía -
John Wendel -
Mike Fleetwood -
Steven Eberhart