4:57 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
One of the worst problems wrt energy savings we have today are all the
wakeups processes request. This is not just an issue for laptops, it
relevant everywhere.
To see the size of the problem run the attached systemtap script. On my
laptop I see the following output (47 secs runtime):
uid | poll select epoll itimer futex nanosle signal| process
29799 |15941 7971 0 0 0 0 0| npviewer.bin
29841 | 253 0 0 0 1531 0 0| thunderbird-bin
3017 | 447 0 0 0 0 0 0| pulseaudio
2467 | 76 0 0 0 0 0 0| hald
2471 | 8 0 0 0 0 0 0| hald-runner
2620 | 58 0 0 0 0 0 0| NetworkManager
13174 | 214 0 0 0 0 0 0| stapio
3044 | 48 0 0 0 0 0 0| gnome-panel
9115 | 16 0 0 0 0 0 0| nm-vpnc-service
3112 | 32 0 0 0 0 0 0| gpk-update-icon
3108 | 24 0 0 0 0 0 0| nm-applet
3052 | 274 0 0 1 0 0 0| gnome-terminal
3115 | 29 0 0 0 0 0 0| gnome-power-man
3055 | 16 0 0 0 0 0 0| bluetooth-apple
3093 | 16 0 0 0 0 0 0| krb5-auth-dialo
2633 | 16 0 0 0 0 0 0| gdm-binary
2724 | 16 0 0 0 0 0 0| gdm-simple-slav
2630 | 16 0 0 0 0 0 0| nm-system-setti
2470 | 16 0 0 0 0 0 0| console-kit-dae
2385 | 16 0 0 0 0 0 0| avahi-daemon
2397 | 16 0 0 0 0 0 0| libvirtd
2725 | 63 214 0 4 0 0 0| Xorg
2150 | 42 0 0 0 0 0 0| setroubleshootd
3010 | 47 0 0 0 0 0 0| gnome-settings-
3040 | 111 0 0 0 0 0 0| metacity
3526 | 37 0 0 0 0 0 0| notification-da
3085 | 49 0 0 0 0 0 0| wnck-applet
3043 | 50 0 0 0 0 0 0| gnome-screensav
3042 | 26 0 0 0 0 0 0| nautilus
3050 | 8 0 0 0 0 0 0| evince
9120 | 0 5 0 0 0 0 0| vpnc
3342 | 11 0 0 0 0 0 0| clock-applet
3329 | 0 6 0 0 0 0 0| pam_timestamp_c
2337 | 0 7 0 0 0 0 0| sendmail
2132 | 0 0 0 0 33 0 0| automount
2958 | 0 3 0 0 0 0 0| ssh-agent
2118 | 1 0 0 0 0 0 0| dbus-daemon
5732 | 1 0 0 0 0 0 0| gnome-vfs-daemo
29394 | 68 131 0 0 65 0 0| firefox
2105 | 1 0 0 0 0 0 0| audispd
1979 | 0 1 0 0 0 0 0| rsyslogd
As you can see, not all programs are equally bad and proprietary ones
(here: flash) are the worst.
Still, since the script records wakeups due to timeouts almost all
mentions on this list are bad. Programs should be woken based on
events. They shouldn't poll data (which is what usually happens after a
timeout).
I would hope the package maintainers can find some time and look at the
issues. Maybe at least document them in a BZ. I might try to do the
latter myself but given the large number of packages involved I'll most
likely be able to cover just a few packages. IMO it should be a release
criteria that a program does use polling.
- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iEYEARECAAYFAkh7y9oACgkQ2ijCOnn/RHTW6wCfWzYOn3AvjLvJ5mwWjSZwr8tX
Z84An2YYvqd9f2fkPGenE5uhDDdsan3y
=VWLu
-----END PGP SIGNATURE-----
# Copyright (C) 2008 Red Hat, Inc.
# Written by Ulrich Drepper <drepper(a)redhat.com>
global process
global poll_timeouts
global epoll_timeouts
global select_timeouts
global itimer_timeouts
global nanosleep_timeouts
global futex_timeouts
global signal_timeouts
probe kernel.function("do_sys_poll").return {
if ($return == 0) {
p = pid()
if (!(p in process))
process[p] = execname()
poll_timeouts[p]++
}
}
probe kernel.function("do_select").return {
if ($return == 0) {
p = pid()
if (!(p in process))
process[p] = execname()
select_timeouts[p]++
}
}
probe kernel.function("sys_epoll_wait").return {
if ($return == 0) {
p = pid()
if (!(p in process))
process[p] = execname()
epoll_timeouts[p]++
}
}
probe kernel.function("do_futex").return {
if ($return == -110) {
p = pid()
if (!(p in process))
process[p] = execname()
futex_timeouts[p]++
}
}
probe kernel.function("hrtimer_nanosleep").return,
kernel.function("sys_clock_nanosleep").return {
if ($return == 0) {
p = pid()
if (!(p in process))
process[p] = execname()
nanosleep_timeouts[p]++
}
}
probe kernel.function("it_real_fn") {
p = pid()
itimer_timeouts[p]++
if (!(p in process))
process[p] = execname()
}
probe kernel.function("sys_rt_sigtimedwait").return {
if ($return == -11) {
p = pid()
if (!(p in process))
process[p] = execname()
signal_timeouts[p]++
}
}
probe kernel.function("do_exit") {
p = pid()
if (process[p] == execname()) {
delete process[p]
poll_timeouts[p] = 0
epoll_timeouts[p] = 0
select_timeouts[p] = 0
itimer_timeouts[p] = 0
futex_timeouts[p] = 0
nanosleep_timeouts[p] = 0
signal_timeouts[p] = 0
}
}
probe timer.ms(1000) {
printf("\033[2J\033[1;1H") /* clear screen */
printf (" uid | poll select epoll itimer futex nanosle signal|
process\n");
foreach (p in process) {
if (poll_timeouts[p] != 0 || select_timeouts[p] != 0
|| epoll_timeouts[p] != 0 || itimer_timeouts[p] != 0
|| futex_timeouts[p] != 0 || nanosleep_timeouts[p] != 0
|| signal_timeouts[p] != 0)
printf ("%5d |%7d %7d %7d %7d %7d %7d %7d| %-.38s\n", p,
poll_timeouts[p], select_timeouts[p],
epoll_timeouts[p], itimer_timeouts[p],
futex_timeouts[p], nanosleep_timeouts[p],
signal_timeouts[p], process[p])
}
}
12:57 a.m.
On Mon, 14 Jul 2008 14:57:56 -0700
Ulrich Drepper <drepper(a)redhat.com> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
One of the worst problems wrt energy savings we have today are all the
wakeups processes request. This is not just an issue for laptops, it
relevant everywhere.
To see the size of the problem run the attached systemtap script. On
my laptop I see the following output (47 secs runtime):
just a question.. is there anything this script does that powertop
doesn't yet do?
I would hope the package maintainers can find some time and look at
the issues. Maybe at least document them in a BZ.
there is a metabug for these things in bugzilla with the "wakeup"
alias...
--
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
6:06 a.m.
On Mon, Jul 14, 2008 at 02:57:56PM -0700, Ulrich Drepper wrote:
I would hope the package maintainers can find some time and look at
the
issues. Maybe at least document them in a BZ. I might try to do the
latter myself but given the large number of packages involved I'll most
likely be able to cover just a few packages. IMO it should be a release
criteria that a program does use polling.
There are certain situations where polling is inevitable (such as
querying hardware for battery status or signal strength, pulling mail,
that kind of thing). Applications that do this should do it at
relatively low frequency, and where possible (ie, almost always) use
g_timeout_add_seconds or equivalent functionality. It's pretty
inevitable that we'll have one wakeup a second, and ensuring that all
applications that need to be woken during that second are woken at the
same time has a significant benefit on power consumption.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
6:30 a.m.
Matthew Garrett wrote:
On Mon, Jul 14, 2008 at 02:57:56PM -0700, Ulrich Drepper wrote:
> I would hope the package maintainers can find some time and look at the
> issues. Maybe at least document them in a BZ. I might try to do the
> latter myself but given the large number of packages involved I'll most
> likely be able to cover just a few packages. IMO it should be a release
> criteria that a program does use polling.
There are certain situations where polling is inevitable (such as
querying hardware for battery status or signal strength, pulling mail,
that kind of thing). Applications that do this should do it at
relatively low frequency, and where possible (ie, almost always) use
g_timeout_add_seconds or equivalent functionality. It's pretty
inevitable that we'll have one wakeup a second, and ensuring that all
applications that need to be woken during that second are woken at the
same time has a significant benefit on power consumption.
g_timeout_add_seconds does not "really" sync globally.
The kernel could schedule a common (across all apps) wakeup time a little bit
better than g_timeout_add_seconds(), if the application could specify a time
"range" of how long it would like to approx. sleep. Also g_timeout_add_seconds()
does not work with select/poll.
10:33 a.m.
On Tue, 2008-07-15 at 15:28 +0100, Matthew Garrett wrote:
On Tue, Jul 15, 2008 at 01:30:50PM +0200, Harald Hoyer wrote:
> g_timeout_add_seconds does not "really" sync globally.
Yeah, it's non-ideal. Anything better is going to need at least glibc
(and ideally kernel) support.
If it needs to be done per sleep event. On the other hand, if you can
set a per-thread flag or tunable which says 'you can always delay my
wake by up to N milliseconds', then you don't need special magic in
poll/select/nanosleep/etc for that. And it would make it easier to
modify userspace to use it too. In fact, you could even use it with
binary-only unmodified userspace, by executing a program with the flag
already set.
--
dwmw2
9:39 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Matthew Garrett wrote:
There are certain situations where polling is inevitable (such as
querying hardware for battery status or signal strength, pulling mail,
that kind of thing).
For querying hardware there should be better ways. Have a centralized
place where the information is poll and the broadcast changes. I though
hal etc is supposed to do this.
As for mail pulling: sure. But the frequency should be measured in
minutes and not in micro-seconds.
Applications that do this should do it at
relatively low frequency, and where possible (ie, almost always) use
g_timeout_add_seconds or equivalent functionality.
I assume this refers to the same thing Arjan tried to do: group wakeups
of multiple waiters together. He can probably recall the details more
easily than I can.
- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkh8tq0ACgkQ2ijCOnn/RHRF/QCfeGAbjuejLQl5MDhNQafy7edA
coYAn0a3cULC+vdaYVRAZOEHMA2rnQ/d
=rW6D
-----END PGP SIGNATURE-----
9:50 a.m.
On Tue, 15 Jul 2008 07:39:41 -0700
Ulrich Drepper <drepper(a)redhat.com> wrote:
> Applications that do this should do it at
> relatively low frequency, and where possible (ie, almost always)
> use g_timeout_add_seconds or equivalent functionality.
I assume this refers to the same thing Arjan tried to do: group
wakeups of multiple waiters together. He can probably recall the
details more easily than I can.
g_timeout_add_seconds() is what I did ;-)
it's not ideal (it doesn't allow for a range, and it only works for
glib based apps).. but it's the best we have for userspace so far.
we really ought to do a range based timeout (or a precision specifier)
--
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
5:35 a.m.
On Tue, 2008-07-15 at 07:39 -0700, Ulrich Drepper wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Matthew Garrett wrote:
> There are certain situations where polling is inevitable (such as
> querying hardware for battery status or signal strength, pulling mail,
> that kind of thing).
[...]
As for mail pulling: sure. But the frequency should be measured in
minutes and not in micro-seconds.
IMAP IDLE (the IMAP server telling connected clients when there is new
mail) should be able to fix that: http://en.wikipedia.org/wiki/IMAP_IDLE
We just have to ensure that our clients and servers support this
correctly (and that it is enabled by default).
Nils
--
Nils Philippsen / Red Hat / nphilipp(a)redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
8:04 a.m.
On Mon, 2008-07-14 at 14:57 -0700, Ulrich Drepper wrote:
3112 | 32 0 0 0 0 0 0|
gpk-update-icon
All wakeups fixed in git master -- should be essentially polling free
apart from when actually being used.
3115 | 29 0 0 0 0 0 0|
gnome-power-man
Harder to fix, as we still have to poll the xserver to see if dpms has
been enabled every now and then. There's a fix in the works for this, as
X will send out notification of the changed state.
Richard.
8:39 a.m.
On Monday 14 July 2008 17:57:56 Ulrich Drepper wrote:
Still, since the script records wakeups due to timeouts almost all
mentions on this list are bad. Programs should be woken based on
events. They shouldn't poll data (which is what usually happens after a
timeout).
One of the worst that I've found that's not on the list is mysqld. It has 2
threads that just never stop.
-Steve
10:28 a.m.
On Tuesday 15 July 2008 11:15:56 Bill Crawford wrote:
> One of the worst that I've found that's not on the list
is mysqld. It has
> 2 threads that just never stop.
I see about 1 wakeup per second from mysqld (it's currently idle,
though). What are you seeing there?
ps -ef | grep sql
root 2227 1 0 07:41 ?
00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql
--socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log
--pid-file=/var/run/mysqld/mysqld.pid
mysql 2286 2227 0 07:41 ?
00:00:12 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql
--pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking
--socket=/var/lib/mysql/mysql.sock
ls /proc/2286/task/
2286 2296 2297 2298 2299 2301 2304 2308 2309 2312 2460
then strace them.
strace -p 2301
Process 2301 attached - interrupt to quit
select(0, NULL, NULL, NULL, {0, 78000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}^C <unfinished ...>
...this is after about 5 seconds. This thread just never gets quiet and is
constantly churning. My system is idle too with nothing being added or
deleted or read from the database.
strace -p 2304
Process 2304 attached - interrupt to quit
select(0, NULL, NULL, NULL, {0, 101000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {2, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {2, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {2, 0}^C <unfinished ...>
Those are the 2 noisy threads.
-Steve
5:42 a.m.
2008/7/15 Steve Grubb <sgrubb(a)redhat.com>:
...
...this is after about 5 seconds. This thread just never gets quiet
and is
constantly churning. My system is idle too with nothing being added or
deleted or read from the database.
...
Those are the 2 noisy threads.
Ah, that does fit with what I see here, then. I took your "that just
never stop" to mean you were seeing significant CPU usage.
It's not good, certainly, but I guess it's designed on the assumption
that you're using it, ... so doesn't make quite the extraordinary
effort to avoid these wakeups that it otherwise might. I presume that
the timeout could be increased greatly when there's no activity to
monitor; but I have no knowledge at all of the mysql code, so I won't
claim it would be easy :o)
10:50 a.m.
On Mon, Jul 14, 2008 at 02:57:56PM -0700, Ulrich Drepper wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
One of the worst problems wrt energy savings we have today are all the
wakeups processes request. This is not just an issue for laptops, it
relevant everywhere.
To see the size of the problem run the attached systemtap script. On my
laptop I see the following output (47 secs runtime):
uid | poll select epoll itimer futex nanosle signal| process
2397 | 16 0 0 0 0 0 0| libvirtd
This isn't libvirtd's fault - we don't use any timeouts, and only
break out of poll() if there's real work todo. In this case it
appears we're being woken up once every 5 seconds, bt a message
from the DBus system bus saying
NameOwnerChanged (nil) 1.237
and then almost immediately
NameOwnerChanged 1.237 (nil)
Which basically means that every 6 seconds some app is connecting to
DBus and then almost immediately disconnecting again. What's most annoying
is that I've not actually requested that DBus send me this signal.
Anyway, a systemtap script probing 'unix_stream_connect' identifies the
cuplrits..
# cat conn.stap
probe kernel.function("unix_stream_connect").return {
if ($return == 0) {
p = pid()
n = execname()
printf ("%d %d %s\n", gettimeofday_s(), p, n)
}
}
[root@t60wlan ~]# stap conn.stap
1216136855 8215 hal-is-caller-p
1216136855 8217 hal-ipw-killswi
1216136861 8219 hal-is-caller-p
1216136861 8221 hal-ipw-killswi
1216136867 8223 hal-is-caller-p
1216136867 8225 hal-ipw-killswi
Aka
/usr/libexec/hal-ipw-killswitch-linux
/usr/bin/hal-is-caller-privileged
which both seem to be spawned once every 5-6 seconds :-(
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
12:22 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Daniel P. Berrange wrote:
> uid | poll select epoll itimer futex nanosle signal| process
> 2397 | 16 0 0 0 0 0 0| libvirtd
This isn't libvirtd's fault - we don't use any timeouts, and only
break out of poll() if there's real work todo.
The script counts returns from poll() where no file descriptor is ready.
This is no count of all poll() calls.
Note, that all calls in a process regardless of whether it's in the
binary or a DSO is counted.
Of course there could be a problem with systemtap but I doubt it
actually, the script is simple.
- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkh83MEACgkQ2ijCOnn/RHRUoACeIHb4a6oMyXlUm0pG95luohak
jlgAoMmz2cCKzdoPLBE3V+svauzXAfqC
=SZEe
-----END PGP SIGNATURE-----
2:13 p.m.
On Tue, Jul 15, 2008 at 10:22:09AM -0700, Ulrich Drepper wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Daniel P. Berrange wrote:
>> uid | poll select epoll itimer futex nanosle signal| process
>> 2397 | 16 0 0 0 0 0 0| libvirtd
>
> This isn't libvirtd's fault - we don't use any timeouts, and only
> break out of poll() if there's real work todo.
The script counts returns from poll() where no file descriptor is ready.
This is no count of all poll() calls.
Sorry I wasn't being clear. The call to poll() with a 0 second timeout
is an artifact of the way the DBus/Avahi/event loop integration works.
It gets a wakeup on a file descriptor, processes some messages and then
does one iteration through the main loop with a 0 second timeout to
flush out the queues, before finally going back to sleep for events on
its file descriptors. As such the poll() with timeout is harmless - the
process was already awake at this point. Its the wakeup from irrelevant
DBus signals that's the real problem that needs to be addressed - once
that's solved the poll() will timeout won't be done.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
10:52 a.m.
On Mon, 2008-07-14 at 14:57 -0700, Ulrich Drepper wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
One of the worst problems wrt energy savings we have today are all the
wakeups processes request. This is not just an issue for laptops, it
relevant everywhere.
To see the size of the problem run the attached systemtap script. On my
laptop I see the following output (47 secs runtime):
uid | poll select epoll itimer futex nanosle signal| process
29799 |15941 7971 0 0 0 0 0| npviewer.bin
29841 | 253 0 0 0 1531 0 0| thunderbird-bin
3017 | 447 0 0 0 0 0 0| pulseaudio
2467 | 76 0 0 0 0 0 0| hald
2471 | 8 0 0 0 0 0 0| hald-runner
2620 | 58 0 0 0 0 0 0| NetworkManager
While there are probably stupidities in NM, there are also known
externally-driven wakeup causes including:
1) until 2.6.27 lands, there's no way to get the rfkill switch
state-change events, so NM polls them every 6 seconds _iff_ they exist.
Unfortunately, they often exist in HAL even though there are no physical
switches on the laptop because the nice rfkill patches haven't landed
yet (again, scheduled for 2.6.27)
2) Intel-based wireless cards driven by ipw drivers send out a steady
stream of background scan events, which causes any process listening for
WEXT events to wake up. Somewhat fixed for ipw2200 in 2.6.24 and later
by 0b5316769774d1dc2fdd702e095f9e6992af269a, but still wakes up all WEXT
listeners every 4 seconds
3) Have to poll the driver every so often (every 6 seconds right now
maybe?) to get signal strength information; I'd love it if drivers would
just send this out over netlink or WEXT or whatever every few seconds.
Not sure if people want that kernel-side though since there's not always
a listener and the frequency of events should be somewhat selectable.
13174 | 214 0 0 0 0 0 0| stapio
3044 | 48 0 0 0 0 0 0| gnome-panel
9115 | 16 0 0 0 0 0 0| nm-vpnc-service
Interesting... I have no idea why it would be a significant wakeup
source. It just listens on the D-Bus system bus, and uses the glib
child-watch functionality to monitor vpnc child process state.
Dan
3112 | 32 0 0 0 0 0 0|
gpk-update-icon
3108 | 24 0 0 0 0 0 0| nm-applet
3052 | 274 0 0 1 0 0 0| gnome-terminal
3115 | 29 0 0 0 0 0 0| gnome-power-man
3055 | 16 0 0 0 0 0 0| bluetooth-apple
3093 | 16 0 0 0 0 0 0| krb5-auth-dialo
2633 | 16 0 0 0 0 0 0| gdm-binary
2724 | 16 0 0 0 0 0 0| gdm-simple-slav
2630 | 16 0 0 0 0 0 0| nm-system-setti
2470 | 16 0 0 0 0 0 0| console-kit-dae
2385 | 16 0 0 0 0 0 0| avahi-daemon
2397 | 16 0 0 0 0 0 0| libvirtd
2725 | 63 214 0 4 0 0 0| Xorg
2150 | 42 0 0 0 0 0 0| setroubleshootd
3010 | 47 0 0 0 0 0 0| gnome-settings-
3040 | 111 0 0 0 0 0 0| metacity
3526 | 37 0 0 0 0 0 0| notification-da
3085 | 49 0 0 0 0 0 0| wnck-applet
3043 | 50 0 0 0 0 0 0| gnome-screensav
3042 | 26 0 0 0 0 0 0| nautilus
3050 | 8 0 0 0 0 0 0| evince
9120 | 0 5 0 0 0 0 0| vpnc
3342 | 11 0 0 0 0 0 0| clock-applet
3329 | 0 6 0 0 0 0 0| pam_timestamp_c
2337 | 0 7 0 0 0 0 0| sendmail
2132 | 0 0 0 0 33 0 0| automount
2958 | 0 3 0 0 0 0 0| ssh-agent
2118 | 1 0 0 0 0 0 0| dbus-daemon
5732 | 1 0 0 0 0 0 0| gnome-vfs-daemo
29394 | 68 131 0 0 65 0 0| firefox
2105 | 1 0 0 0 0 0 0| audispd
1979 | 0 1 0 0 0 0 0| rsyslogd
As you can see, not all programs are equally bad and proprietary ones
(here: flash) are the worst.
Still, since the script records wakeups due to timeouts almost all
mentions on this list are bad. Programs should be woken based on
events. They shouldn't poll data (which is what usually happens after a
timeout).
I would hope the package maintainers can find some time and look at the
issues. Maybe at least document them in a BZ. I might try to do the
latter myself but given the large number of packages involved I'll most
likely be able to cover just a few packages. IMO it should be a release
criteria that a program does use polling.
- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iEYEARECAAYFAkh7y9oACgkQ2ijCOnn/RHTW6wCfWzYOn3AvjLvJ5mwWjSZwr8tX
Z84An2YYvqd9f2fkPGenE5uhDDdsan3y
=VWLu
-----END PGP SIGNATURE-----
plain text document attachment (timeout.stap)
# Copyright (C) 2008 Red Hat, Inc.
# Written by Ulrich Drepper <drepper(a)redhat.com>
global process
global poll_timeouts
global epoll_timeouts
global select_timeouts
global itimer_timeouts
global nanosleep_timeouts
global futex_timeouts
global signal_timeouts
probe kernel.function("do_sys_poll").return {
if ($return == 0) {
p = pid()
if (!(p in process))
process[p] = execname()
poll_timeouts[p]++
}
}
probe kernel.function("do_select").return {
if ($return == 0) {
p = pid()
if (!(p in process))
process[p] = execname()
select_timeouts[p]++
}
}
probe kernel.function("sys_epoll_wait").return {
if ($return == 0) {
p = pid()
if (!(p in process))
process[p] = execname()
epoll_timeouts[p]++
}
}
probe kernel.function("do_futex").return {
if ($return == -110) {
p = pid()
if (!(p in process))
process[p] = execname()
futex_timeouts[p]++
}
}
probe kernel.function("hrtimer_nanosleep").return,
kernel.function("sys_clock_nanosleep").return {
if ($return == 0) {
p = pid()
if (!(p in process))
process[p] = execname()
nanosleep_timeouts[p]++
}
}
probe kernel.function("it_real_fn") {
p = pid()
itimer_timeouts[p]++
if (!(p in process))
process[p] = execname()
}
probe kernel.function("sys_rt_sigtimedwait").return {
if ($return == -11) {
p = pid()
if (!(p in process))
process[p] = execname()
signal_timeouts[p]++
}
}
probe kernel.function("do_exit") {
p = pid()
if (process[p] == execname()) {
delete process[p]
poll_timeouts[p] = 0
epoll_timeouts[p] = 0
select_timeouts[p] = 0
itimer_timeouts[p] = 0
futex_timeouts[p] = 0
nanosleep_timeouts[p] = 0
signal_timeouts[p] = 0
}
}
probe timer.ms(1000) {
printf("\033[2J\033[1;1H") /* clear screen */
printf (" uid | poll select epoll itimer futex nanosle signal|
process\n");
foreach (p in process) {
if (poll_timeouts[p] != 0 || select_timeouts[p] != 0
|| epoll_timeouts[p] != 0 || itimer_timeouts[p] != 0
|| futex_timeouts[p] != 0 || nanosleep_timeouts[p] != 0
|| signal_timeouts[p] != 0)
printf ("%5d |%7d %7d %7d %7d %7d %7d %7d| %-.38s\n", p,
poll_timeouts[p], select_timeouts[p],
epoll_timeouts[p], itimer_timeouts[p],
futex_timeouts[p], nanosleep_timeouts[p],
signal_timeouts[p], process[p])
}
}
--
fedora-devel-list mailing list
fedora-devel-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
10:57 a.m.
On Tue, Jul 15, 2008 at 11:52:35AM -0400, Dan Williams wrote:
On Mon, 2008-07-14 at 14:57 -0700, Ulrich Drepper wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> One of the worst problems wrt energy savings we have today are all the
> wakeups processes request. This is not just an issue for laptops, it
> relevant everywhere.
>
> To see the size of the problem run the attached systemtap script. On my
> laptop I see the following output (47 secs runtime):
>
> uid | poll select epoll itimer futex nanosle signal| process
> 29799 |15941 7971 0 0 0 0 0| npviewer.bin
> 29841 | 253 0 0 0 1531 0 0| thunderbird-bin
> 3017 | 447 0 0 0 0 0 0| pulseaudio
> 2467 | 76 0 0 0 0 0 0| hald
> 2471 | 8 0 0 0 0 0 0| hald-runner
> 2620 | 58 0 0 0 0 0 0| NetworkManager
While there are probably stupidities in NM, there are also known
externally-driven wakeup causes including:
1) until 2.6.27 lands, there's no way to get the rfkill switch
state-change events, so NM polls them every 6 seconds _iff_ they exist.
Unfortunately, they often exist in HAL even though there are no physical
switches on the laptop because the nice rfkill patches haven't landed
yet (again, scheduled for 2.6.27)
Ahhh, so that's probably what's causing /usr/libexec/hal-ipw-killswitch-linux
to be run every 6 seconds, which in turns causes any app connected to DBus
system bus to be send a signal every 6 seconds and thus causes all the
hits against libvirtd - and a fair number of other apps in that list too
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
11:07 a.m.
On Tue, 2008-07-15 at 16:57 +0100, Daniel P. Berrange wrote:
On Tue, Jul 15, 2008 at 11:52:35AM -0400, Dan Williams wrote:
> On Mon, 2008-07-14 at 14:57 -0700, Ulrich Drepper wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > One of the worst problems wrt energy savings we have today are all the
> > wakeups processes request. This is not just an issue for laptops, it
> > relevant everywhere.
> >
> > To see the size of the problem run the attached systemtap script. On my
> > laptop I see the following output (47 secs runtime):
> >
> > uid | poll select epoll itimer futex nanosle signal| process
> > 29799 |15941 7971 0 0 0 0 0| npviewer.bin
> > 29841 | 253 0 0 0 1531 0 0| thunderbird-bin
> > 3017 | 447 0 0 0 0 0 0| pulseaudio
> > 2467 | 76 0 0 0 0 0 0| hald
> > 2471 | 8 0 0 0 0 0 0| hald-runner
> > 2620 | 58 0 0 0 0 0 0| NetworkManager
>
> While there are probably stupidities in NM, there are also known
> externally-driven wakeup causes including:
>
> 1) until 2.6.27 lands, there's no way to get the rfkill switch
> state-change events, so NM polls them every 6 seconds _iff_ they exist.
> Unfortunately, they often exist in HAL even though there are no physical
> switches on the laptop because the nice rfkill patches haven't landed
> yet (again, scheduled for 2.6.27)
Ahhh, so that's probably what's causing /usr/libexec/hal-ipw-killswitch-linux
to be run every 6 seconds, which in turns causes any app connected to DBus
system bus to be send a signal every 6 seconds and thus causes all the
hits against libvirtd - and a fair number of other apps in that list too
That's probably the cause, yes.
Is D-Bus signal filtering done client-side or bus-side? I forget. You
do have to explicitly subscribe to most signals (see dbus_bus_add_match)
with libdbus otherwise you won't get them, so maybe a library that
libvirt depends on is asking for the subscription or not properly
limiting its subscription.
Dan
11 a.m.
On Mon, 14.07.08 14:57, Ulrich Drepper (drepper(a)redhat.com) wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
One of the worst problems wrt energy savings we have today are all the
wakeups processes request. This is not just an issue for laptops, it
relevant everywhere.
To see the size of the problem run the attached systemtap script. On my
laptop I see the following output (47 secs runtime):
uid | poll select epoll itimer futex nanosle signal| process
29799 |15941 7971 0 0 0 0 0| npviewer.bin
29841 | 253 0 0 0 1531 0 0| thunderbird-bin
3017 | 447 0 0 0 0 0 0| pulseaudio
I would assume that this is just a followup by Flash/npviewer. Flash
opens an audio stream and never closes it again during its entire
runtime. While the audio stream is open. PA also needs to keep the
audio device open. As long as the audio device is open we we will
schedule audio via timers. The PA version from rawhide should
not generate any wakeups when completely idle (i.e. no streams
allocated or all streams that are allocated are properly paused)
But of course, I might be mistaken, I didn't run this test, and
pointing fingers to other people's software (flash) might just by an
automatic reflex of me. Last time I checked PA didn't show up in
powertop when idle, to say the least. I'll check again.
2467 | 76 0 0 0 0 0 0| hald
2471 | 8 0 0 0 0 0 0| hald-runner
2620 | 58 0 0 0 0 0 0| NetworkManager
13174 | 214 0 0 0 0 0 0| stapio
3044 | 48 0 0 0 0 0 0| gnome-panel
9115 | 16 0 0 0 0 0 0| nm-vpnc-service
3112 | 32 0 0 0 0 0 0| gpk-update-icon
3108 | 24 0 0 0 0 0 0| nm-applet
3052 | 274 0 0 1 0 0 0| gnome-terminal
3115 | 29 0 0 0 0 0 0| gnome-power-man
3055 | 16 0 0 0 0 0 0| bluetooth-apple
3093 | 16 0 0 0 0 0 0| krb5-auth-dialo
2633 | 16 0 0 0 0 0 0| gdm-binary
2724 | 16 0 0 0 0 0 0| gdm-simple-slav
2630 | 16 0 0 0 0 0 0| nm-system-setti
2470 | 16 0 0 0 0 0 0| console-kit-dae
2385 | 16 0 0 0 0 0 0| avahi-daemon
For avahi-daemon the wakeups are highly dependant on how much traffic
actually goes on on the network and how long Avahi has already been
running. I think the room for improvements here is rather minimal, I
would assume, due to the way the protocol works: in mDNS traffic is
scheduled based on time. Basically, every packet we send is implicitly
delayed to increase the chance that we can merge multiple messages
into a single packet.
Lennart
--
Lennart Poettering Red Hat, Inc.
lennart [at] poettering [dot] net ICQ# 11060553
http://0pointer.net/lennart/ GnuPG 0x1A015CC4
5771
days inactive
5773
days old
20 comments
12 participants
participants (12)
-
Arjan van de Ven -
Bill Crawford -
Dan Williams -
Daniel P. Berrange -
David Woodhouse -
Harald Hoyer -
Lennart Poettering -
Matthew Garrett -
Nils Philippsen -
Richard Hughes -
Steve Grubb -
Ulrich Drepper