boot-time / readahead investigations

[Eric Sandeen]

I found some problems with our (F8) readahead as it is.  I did some
testing w/ blktrace & seekwatcher during boot to see what the io
patterns look like.

First, the default/shipped readahead lists seem pretty incomplete, I saw
lots of fresh IO happening after the readahead is done.  How are the
default lists maintained?

I tried generating my own lists with init=/sbin/readahead-collector, but:

Bugzilla Bug 326891: readahead-collector doesn't collect binaries

I fixed that up, and generated nice long custom lists, but:

Bugzilla Bug 327041: readahead --sort generates file with "(null)"
instead of filename

so after the cron.daily goes off, those nice custom lists (or the
default lists) get nuked in the sort procedure.  (I think this bug only
hits on 32-bit platforms)

Ok, fixed that up, and did a bit of other tweaking:

1) moved readahead_early to S01 per davej's suggestion, and made the
readahead call *not* background itself.  This seems to help.

2) also played with "readahead_later" at S02, reading ahead in the
background.  This saved a little time, but it requires jumping through
hoops to make the lists properly.

3) tried adding a superearly readahead list for all the IO rc.sysinit
does, it helped a little but not a whole lot.

So, with the above bugfixes and 1) above, using (fixed) readahead *does*
speed up the boot process.  Testing w/o selinux so far,

with stock readahead (0:59):
http://sandeen.net/fedora_boot/bootchart-stockreadahead-noselinux.png
http://sandeen.net/fedora_boot/seekwatcher-stockreadahead-noselinux.png

without readahead (0:49):
http://sandeen.net/fedora_boot/bootchart-noreadahead-noselinux.png
http://sandeen.net/fedora_boot/seekwatcher-noreadahead-noselinux.png

with tweaked readahead (0:44):
(only tweak 1) above, with updated/sorted lists)
http://sandeen.net/fedora_boot/bootchart-fixedreadahead-noselinux.png
http://sandeen.net/fedora_boot/seekwatcher-fixedreadahead-noselinux.png

On the seekwatcher graphs, nice linear accesses should be faster, and if
all files are unfragmented and read-ahead in block order, we shouldn't
be doing a lot of seeking around.

I was able to get down to about 0:41 a few times with enough fiddling...

Also, some way to keep the readahead lists uptodate would be a help, I
think.

Selinux still does have a lot of overhead, haven't started looking
too closely at that case yet (for one, readahead probably doesn't pull
in the selinux xattrs)

Also, Chris Mason told me that the SuSE glibc has a patch which calls
fadvise(POSIX_FADV_WILLNEED) on any shared library that is opened; this
causes the kernel to launch its own readahead on it.  I have the patch
and will do some testing, sounds like an interesting approach.

I'll keep looking at this...

-Eric