Woohoo! 31% boot speedup (*so far*), and 4.5MB free...

[Douglas McClendon]

Disclaimer: take these claims with the proverbial grain of salt, as I 
can on occasion be a bit over-optimistic or over-committal...

Second Disclaimer: read the first disclaimer again.

BUT THAT SAID...

The Bad News: my "SuperDeviceMapperCaching" technique does not (yet[1]) 
appear to actually significantly improve LiveCD boot speed.

The Good News: Some of the things that I did which were prerequisites of 
SDMC, appear to have the potential to speed up LiveCD boot by at least 
31%, and even shrink the LiveCD by 4.5MB.

The Better News: those prerequisite things are actually far less 
invasive (non-invasive really) than SDMC.

Review: The SDMC idea was to use a virtualized boot of a LiveCD to 
determine which files were accessed during boot, then remaster the root 
filesystem so that those files would be grouped together, so that they 
could be read into a portion of the rootfs that would temporarily live 
in ram, via a devicemapper linear combination of the 'cached/preloaded' 
portion, and the normal rest of the rootfs on CDROM.

The Reality: For expediency, I opted to gather the list of files via a 
simple find atime command after a virtual boot settled.  I then sorted 
the files (and perhaps more importantly, directories and symlinks) by 
size, and then populated a filesystem, starting with the smallest 
boot-accessed file, and working my way up to the largest boot accessed 
file, and then just doing the rest.

The realization: At some point I realized that this was actually 
gathering the files at inner part of the cdrom, which is read half as 
fast as the outter part.

The sick future: I will get those files pushed to the outter edge...  no 
matter what sick devicemapper magic it takes ;)

The already optimistic numbers, that I really hope I'm not 
misinterpreting or overhyping-

I timed a fedora-8 livecd boot on my laptop.

I got about 2:30s to gdm, 3:30s to gnome desktop fully rendered, and 
3:50s to firefox launched and fully rendered.

I then (slight lie, did this last), timed a build of my very fedora-like 
livecd, built with my VirOS tools, but for all practical purposes, in no 
way radically different than the fedora livecd.

What I got was 2:06s to the first post rhgb cursor (I'm using gdm 
autologin, so this is either a gdm mouse arrow, or the gnome mouse 
arrow).  Then 3:19s to gnome fully rendered, and 3:23s to firefox fully 
rendered.  (firefox was autostarted and fighting with gnome a bit).

Then, I tried my SDMC livecd, but without SDMC, just the file 
reordering- and got 1:48s to first gdm mouse arrow, 2:29s to gnome fully 
rendered, and 2:34s to firefox fully rendered.

I think I was able to get my SDMC mechanism to shave 2-5 seconds off of 
that, but not really a terribly significant difference.

Another interesting fact, is that the reordered filesystem was 4.5MB 
smaller than unordered.  This is no doubt because when I sort those 
files by size, a lot of very similar files end up spatially closer 
together, and are thus better compressed by squashfs.

Thus, what I tenatively see, beyond the 4.5M of free space, is a 31% 
speedup (203s/154s)

So, going forward, I plan to

[1]A) do some sick stuff to get those early files on the outter rim of 
the cdrom, which I think may get further significant gains by itself, 
and perhaps even push my SDMC method into the significantly profitable 
area (though I don't much care at this point).

B) perform all of this semi-manually on the official F8-livecd, and see 
how much I can really improve 'the real deal'.

C) do boot file access data gathering on a pair of dissimilar physical 
hosts, and use the file diffs to determine classes of files (hw drivers 
mostly) which should get added to the list.  Probably /lib/modules and a 
few other obvious directories should just get thrown into the list.

D) I don't really care enough at this point to do a full systemtap or 
similar boot profile.  atime seems sufficient for now (though I wonder, 
do I miss a lot of files that get touched when root is still mounted 
readonly?)

E) do all of this for ubuntu, where I think my SDMC techniques, combined 
with their native squashfs usage, will yield as much, or perhaps even 
more benefit.

-dmc/jdog

Disclaimer: I really hope this all doesn't turn out to be fruitless... 
but I'm doing a pretty good job convincing myself that it all makes 
sense...  (I'm just disappointed that I failed to really eliminate 95% 
of all seeks during boot...  I still think that might be possible... but 
I need to go write a little C program to visualize the population of a 
sparse file... or instrument some filesystems....   Nah... I'm pretty 
sure I've got lower hanging fruit within my grasp...)

P.S.- When I say I 'did' all of this, I of course mean completely 
scripted from a single VirOS commandline that just takes nearly half a 
day to run :)

[Douglas McClendon]

Douglas McClendon wrote:
...
>  Thus, what I tenatively see, beyond the 4.5M of free space, is a 31%
>
>  speedup (203s/154s)
>  
>  So, going forward, I plan to
>  
>  [1]A) do some sick stuff to get those early files on the outter rim of 
>  the cdrom, which I think may get further significant gains by itself, 
>  and perhaps even push my SDMC method into the significantly profitable 
>  area (though I don't much care at this point). 
And here I was thinking of doing my own kludge, but yet AGAIN, Device 
Mapper Is Just So Cool - (dm-reverse)

http://sources.redhat.com/cgi-bin/cvsweb.cgi/device-mapper/contrib/?cvsro...

-dmc

[Douglas McClendon]

Just a status report on this, since 31% is such an outlandish claim...

I'm currently investigating why it is that my find atime command on my 
livecd typically shows about 20k/80k files accessed during a normal 
boot, versus 3k/80k on the f8 livecd.  If anyone would like to venture 
speculation or facts as to which of those numbers is more realistic... 
Looking at the diffs, I know that 3k is missing some stuff, like, 
/etc/xdg/menus (the directory entry) not showing up, even though files 
under the directory show up.  And I did extract the initramfs from the 
f8 livecd, change it's /etc/fstab, initramfs /init, and 
/sysroot/etc/fstab to all have atime,diratime instead of noatime and/or 
relatime.  And I echod 0 to /proc/sys/fs/default_relatime as soon as 
proc was mounted.

( find / -xdev -atime -7 -exec ls -ld '{}' ';' > boot.afiles.list.txt
)

Anyway, I still think I can get significant improvement, but there are a 
lot of experiments and trials to run.

In general, I'm also going to try ordering the filesystem with all 
directory entries together, and symlinks together, to see how that helps 
compression.  Likewise all text files, and all these things alpha and 
size sorted.  I'm particularly interested if anyone has speculation as 
to why clustering the directory entries and symlinks might be some bad 
cornercase for ext3 filesystem performance.

And unfortunately I can't get the dm-reverse kernel module to compile 
easily (don't think anyone's actually used it in years).  But I may go 
back to my own kludge, somewhat similar to the perl extent reversing 
example in the dm docs.

I'll let you all know how it goes...  Maybe nowhere, maybe it'll speed 
up boot 50% and make space for another 10MB of compressed stuff...

-dmc


Douglas McClendon wrote:
>  Douglas McClendon wrote:
>  ...
> > Thus, what I tenatively see, beyond the 4.5M of free space, is a 31% 
> > speedup (203s/154s)
> >
> > So, going forward, I plan to
> >
> > [1]A) do some sick stuff to get those early files on the outter rim of 
> > the cdrom, which I think may get further significant gains by itself, 
> > and perhaps even push my SDMC method into the significantly profitable 
> > area (though I don't much care at this point).
>  
>  And here I was thinking of doing my own kludge, but yet AGAIN, Device 
>  Mapper Is Just So Cool - (dm-reverse)
>  
>  http://sources.redhat.com/cgi-bin/cvsweb.cgi/device-mapper/contrib/?cvsro... 
>  
>  
>  -dmc
>  
>  -- 
>  Fedora-livecd-list mailing list
>  Fedora-livecd-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-livecd-list