Boot speedup with readahead

[Harald Hoyer]

Hello fellow Fedora developers,

recently readahead was modified to adapt to system file changes and to start very early in
the boot process 
via upstart.

I would like to encourage you to test readahead-1.4.5-3.fc10 from rawhide (even possible
on F9), which I built 
some minutes ago. It may take a day to reach your local mirror.

# yum --enablerepo=rawhide install readahead
or
# yum --enablerepo=rawhide update readahead

With the next reboot readahead-collector runs and collects the information which files are
used during the 
boot process. The next reboot then, readahead read ahead those files and the boot process
(from init start to 
gdm login screen) should be approx. 10% faster.

Hope it speeds up your boot process a little bit. Please report any changes in your boot
time (can be measured 
with a stop clock or bootchart).

You can modify /etc/sysconfig/readahead to turn readahead on/off.

[Eric Sandeen]

Harald Hoyer wrote:
>  Hello fellow Fedora developers,
>  
>  recently readahead was modified to adapt to system file changes and to start very early
> in the boot process 
>  via upstart. 
Glad to see that!  Sorry I helped put the hammer down on older readahead
in the F8 era... but it was pretty broken back then ... ;)

>  I would like to encourage you to test readahead-1.4.5-3.fc10 from
> rawhide (even possible on F9), which I built 
>  some minutes ago. It may take a day to reach your local mirror.
>  
>  # yum --enablerepo=rawhide install readahead
>  or
>  # yum --enablerepo=rawhide update readahead
>  
>  With the next reboot readahead-collector runs and collects the information which files
> are used during the 
>  boot process. The next reboot then, readahead read ahead those files and the boot process
> (from init start to 
>  gdm login screen) should be approx. 10% faster. 
So when / how often does readahead-collector run now?

Thanks,
-Eric

>  Hope it speeds up your boot process a little bit. Please report any
> changes in your boot time (can be measured 
>  with a stop clock or bootchart).
>
>
>  You can modify /etc/sysconfig/readahead to turn readahead on/off.
>

[Harald Hoyer]

Eric Sandeen schrieb:
>  Harald Hoyer wrote:
> > Hello fellow Fedora developers,
> >
> > recently readahead was modified to adapt to system file changes and to start very
> early in the boot process 
> > via upstart.
>  
>  Glad to see that!  Sorry I helped put the hammer down on older readahead
>  in the F8 era... but it was pretty broken back then ... ;)
>  
> > I would like to encourage you to test readahead-1.4.5-3.fc10 from rawhide (even
> possible on F9), which I built 
> > some minutes ago. It may take a day to reach your local mirror.
> >
> > # yum --enablerepo=rawhide install readahead
> > or
> > # yum --enablerepo=rawhide update readahead
> >
> > With the next reboot readahead-collector runs and collects the information which
> files are used during the 
> > boot process. The next reboot then, readahead read ahead those files and the boot
> process (from init start to 
> > gdm login screen) should be approx. 10% faster.
>  
>  So when / how often does readahead-collector run now?
>  
>  Thanks,
>  -Eric 
Every month.

/etc/cron.monthly/readahead-monthly.cron

[Eric Sandeen]

Harald Hoyer wrote:
>  Eric Sandeen schrieb: 
> > So when / how often does readahead-collector run now?
> >
> > Thanks,
> > -Eric
>  
>  Every month.
>  
>  /etc/cron.monthly/readahead-monthly.cron 
It'd be interesting to do a daily yum update / reboot and see how the
boot times look, graphed for a couple months.

Will things degrade until the next collection run?  I wonder if some
inotify magic might be interesting; if more than X% of the
previously-collected files have changed, then kick the collector on again?

-Eric

[Harald Hoyer]

Eric Sandeen schrieb:
>  Harald Hoyer wrote:
> > Eric Sandeen schrieb:
>  
> >> So when / how often does readahead-collector run now?
> >>
> >> Thanks,
> >> -Eric
> > Every month.
> >
> > /etc/cron.monthly/readahead-monthly.cron
>  
>  It'd be interesting to do a daily yum update / reboot and see how the
>  boot times look, graphed for a couple months.
>  
>  Will things degrade until the next collection run?  I wonder if some
>  inotify magic might be interesting; if more than X% of the
>  previously-collected files have changed, then kick the collector on again?
>  
>  -Eric
>   
hehe, yes, but how would you implement that? :) I don't think this is doable :)

[Panu Matilainen]

On Mon, 8 Sep 2008, Harald Hoyer wrote:

>  Eric Sandeen schrieb:
> > Harald Hoyer wrote:
> >> Eric Sandeen schrieb:
> > 
> >>> So when / how often does readahead-collector run now?
> >>> 
> >>> Thanks,
> >>> -Eric
> >> Every month.
> >> 
> >> /etc/cron.monthly/readahead-monthly.cron
> > 
> > It'd be interesting to do a daily yum update / reboot and see how the
> > boot times look, graphed for a couple months.
> > 
> > Will things degrade until the next collection run?  I wonder if some
> > inotify magic might be interesting; if more than X% of the
> > previously-collected files have changed, then kick the collector on again?
> > 
> > -Eric
> > 
>
>  hehe, yes, but how would you implement that? :) I don't think this is doable 
>  :) 
A yum-readahead plugin could look at the readahead list after an upgrade 
and calculate if refreshing is needed.

/me ducks and hides

 	- Panu -

[Seth Vidal]

On Mon, 2008-09-08 at 18:07 +0300, Panu Matilainen wrote:
>  On Mon, 8 Sep 2008, Harald Hoyer wrote:
>  
>  > Eric Sandeen schrieb:
>  >> Harald Hoyer wrote:
>  >>> Eric Sandeen schrieb:
>  >> 
>  >>>> So when / how often does readahead-collector run now?
>  >>>> 
>  >>>> Thanks,
>  >>>> -Eric
>  >>> Every month.
>  >>> 
>  >>> /etc/cron.monthly/readahead-monthly.cron
>  >> 
>  >> It'd be interesting to do a daily yum update / reboot and see how the
>  >> boot times look, graphed for a couple months.
>  >> 
>  >> Will things degrade until the next collection run?  I wonder if some
>  >> inotify magic might be interesting; if more than X% of the
>  >> previously-collected files have changed, then kick the collector on again?
>  >> 
>  >> -Eric
>  >> 
>  >
>  > hehe, yes, but how would you implement that? :) I don't think this is doable 
>  > :)
>  
>  A yum-readahead plugin could look at the readahead list after an upgrade 
>  and calculate if refreshing is needed.
>  
>  /me ducks and hides 
it'd be like 10 lines long.

-sv

[Jeremy Katz]

On Mon, 2008-09-08 at 11:12 -0400, Seth Vidal wrote:
>  On Mon, 2008-09-08 at 18:07 +0300, Panu Matilainen wrote:
>  > On Mon, 8 Sep 2008, Harald Hoyer wrote:
>  > > Eric Sandeen schrieb:
>  > >> Harald Hoyer wrote:
>  > >>> Eric Sandeen schrieb:
>  > >>>> So when / how often does readahead-collector run now?
>  > >>>> 
>  > >>>> Thanks,
>  > >>>> -Eric
>  > >>> Every month.
>  > >>> 
>  > >>> /etc/cron.monthly/readahead-monthly.cron
>  > >> 
>  > >> It'd be interesting to do a daily yum update / reboot and see how the
>  > >> boot times look, graphed for a couple months.
>  > >> 
>  > >> Will things degrade until the next collection run?  I wonder if some
>  > >> inotify magic might be interesting; if more than X% of the
>  > >> previously-collected files have changed, then kick the collector on again?
>  > >> 
>  > >> -Eric
>  > >> 
>  > >
>  > > hehe, yes, but how would you implement that? :) I don't think this is
> doable 
>  > > :)
>  > 
>  > A yum-readahead plugin could look at the readahead list after an upgrade 
>  > and calculate if refreshing is needed.
>  > 
>  > /me ducks and hides
>  
>  it'd be like 10 lines long. 
And doing it this way would be more effective than cron as there are
plenty of people (especially with laptops/desktops which aren't on all
the time) for whom cron really isn't appropriate for anything that we
actually care to have done

Jeremy

[Seth Vidal]

On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:

>  And doing it this way would be more effective than cron as there are
>  plenty of people (especially with laptops/desktops which aren't on all
>  the time) for whom cron really isn't appropriate for anything that we
>  actually care to have done
>   
What package does the readahead cron-monthly job live in?

-sv

[Eric Sandeen]

Seth Vidal wrote:
>  On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:
>  
> > And doing it this way would be more effective than cron as there are
> > plenty of people (especially with laptops/desktops which aren't on all
> > the time) for whom cron really isn't appropriate for anything that we
> > actually care to have done
> >
>  
>  What package does the readahead cron-monthly job live in?
>  
>  -sv
>  
>   
it's in readahead:

[root@box]# rpm -ql readahead | grep cron
/etc/cron.daily/readahead.cron
/etc/cron.monthly/readahead-monthly.cron

But we should probably wait a week to see what magic Arjan has before we
go off re-architecting any of this :)

-Eric

[Seth Vidal]

On Mon, 2008-09-08 at 11:08 -0500, Eric Sandeen wrote:
>  Seth Vidal wrote:
>  > On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:
>  > 
>  >> And doing it this way would be more effective than cron as there are
>  >> plenty of people (especially with laptops/desktops which aren't on all
>  >> the time) for whom cron really isn't appropriate for anything that we
>  >> actually care to have done
>  >>
>  > 
>  > What package does the readahead cron-monthly job live in?
>  > 
>  > -sv
>  > 
>  > 
>  
>  it's in readahead:
>  
>  [root@box]# rpm -ql readahead | grep cron
>  /etc/cron.daily/readahead.cron
>  /etc/cron.monthly/readahead-monthly.cron
>  
>  But we should probably wait a week to see what magic Arjan has before we
>  go off re-architecting any of this :) 
okay
:)

-sv

[Callum Lerwick]

On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:
>  And doing it this way would be more effective than cron as there are
>  plenty of people (especially with laptops/desktops which aren't on all
>  the time) for whom cron really isn't appropriate for anything that we
>  actually care to have done 
We still need to trigger pre-linking after yum transactions rather than
by a cron job...

[Jesse Keating]

On Mon, 2008-09-08 at 15:28 -0500, Callum Lerwick wrote:
>  
>  We still need to trigger pre-linking after yum transactions rather than
>  by a cron job... 
So that we can have an "Optimizing harddrive" phase during updates like
OSX does?  that'd be awesome...

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Eric Sandeen]

Jesse Keating wrote:
>  On Mon, 2008-09-08 at 15:28 -0500, Callum Lerwick wrote:
> > We still need to trigger pre-linking after yum transactions rather than
> > by a cron job...
>  
>  So that we can have an "Optimizing harddrive" phase during updates like
>  OSX does?  that'd be awesome... 
I guess you can wait now, or wait later.

But yeah I agree with osx it's:

boots up fast, awesome!
background download of updates, awesome!
updates install quickly (usually) - awesome!
updates force reboot - not awesome!
reboot takes AGES that first time - not aweseome!

-Eric

[Jesse Keating]

On Mon, 2008-09-08 at 16:11 -0500, Eric Sandeen wrote:
>  reboot takes AGES that first time - not aweseome! 
At least they moved it to the reboot.  It used to be tacked on to the
tail end of each update run, so you'd download fast, install fast, and
then sit there for unknown amount of time waiting for "optimizations"
that are likely not going to make up for the lost time.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Ola Thoresen]

Jesse Keating wrote:
>  On Mon, 2008-09-08 at 15:28 -0500, Callum Lerwick wrote:
> > We still need to trigger pre-linking after yum transactions rather than
> > by a cron job...
>  
>  So that we can have an "Optimizing harddrive" phase during updates like
>  OSX does?  that'd be awesome...
>   

Prelink is fine, but please don't "optimize" my brand new SSD-drive!
=;-)


Rgds.

Ola Thhoresen

-- 
         _,--',   _._.--._____
  .--.--';_'-.', ";_      _.,-'   Ola Thoresen
 .'--'.  _.'    {`'-;_ .-.>.'
       '-:_      )  / `' '=.      It is easier to fix Unix
         ) >     {_/,     /~)     than to live with Windows
         |/               `^ .'

[Ville Skyttä]

On Monday 08 September 2008, Callum Lerwick wrote:
>  On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:
>  > And doing it this way would be more effective than cron as there are
>  > plenty of people (especially with laptops/desktops which aren't on all
>  > the time) for whom cron really isn't appropriate for anything that we
>  > actually care to have done
>
>  We still need to trigger pre-linking after yum transactions rather than
>  by a cron job... 
Please keep users of other package managers than yum in mind, too.  Can things 
like these be hooked to rpm level transactions rather than yum 
("%triggerpostrans -- *" or something)?

[Jeremy Katz]

On Tue, 2008-09-09 at 00:24 +0300, Ville Skyttä wrote:
>  On Monday 08 September 2008, Callum Lerwick wrote:
>  > On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:
>  > > And doing it this way would be more effective than cron as there are
>  > > plenty of people (especially with laptops/desktops which aren't on all
>  > > the time) for whom cron really isn't appropriate for anything that we
>  > > actually care to have done
>  >
>  > We still need to trigger pre-linking after yum transactions rather than
>  > by a cron job...
>  
>  Please keep users of other package managers than yum in mind, too.  Can things 
>  like these be hooked to rpm level transactions rather than yum 
>  ("%triggerpostrans -- *" or something)? 
Not really -- for one thing, there isn't functionality for transaction
level triggers.  And even if there were, using them would introduce
other new and different problems.  It really boils down to the fact that
the Fedora package manager is yum.  If users of other package managers
want to get their package manager to do similar things, we won't stop
them -- but trying to make everything work for such a small minority is
going to be increasingly difficult.  Much like if you wanted to use dpkg
+ alien to install packages rather than using rpm

Jeremy

[Ville Skyttä]

On Tuesday 09 September 2008, Jeremy Katz wrote:

>  It really boils down to the fact that
>  the Fedora package manager is yum.  If users of other package managers
>  want to get their package manager to do similar things, we won't stop
>  them -- but trying to make everything work for such a small minority is
>  going to be increasingly difficult. 
It's not really about _making_ everything work everywhere I'm worried about.  
It's about people seemingly being very eager to _move_ existing, working 
functionality to yum or its plugins when someone finds a case that could be 
optimized by doing that, and dropping the existing generic 
functionality.  "We won't stop them to make their favourite package manager 
to do similar things" does not really cover it at all well, it's closer to 
actively making things harder for them.

Keeping other package manager (including plain rpm) users in mind does not 
mean that time should be invested to make everything work for them as well as 
the optimized case does for yum users.  It's very much enough to just not 
drop the existing functionality and do the yum-optimized cases so that they 
play nice together with that.  With the right mindset, I don't think that's 
going to be at all difficult to do in the vast majority of cases.

Just a couple of examples from this thread: readahead and prelink 
updates/re-runs.  Both are things that should Just Work out of the box even 
without any package manager (even rpm!) installed.   

If I misunderstood and the optimized cases are meant to be done so that 
they're just optimized for yum users, and the existing generic functionality 
is preserved for others, then no worries.

Oh, and by the way, after trying to use yum for several distro versions, 
always eventually failing and switching to some alternative for various 
reasons until now, I haven't even tried other package managers than yum 
during the time I've used F-9.  I still find myself disagreeing with some 
things and run into some bugs every now and then, but at this point I think 
overall it's the best depsolver I've used on Fedora.

[Seth Vidal]

On Mon, 2008-09-08 at 15:28 -0500, Callum Lerwick wrote:
>  On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:
>  > And doing it this way would be more effective than cron as there are
>  > plenty of people (especially with laptops/desktops which aren't on all
>  > the time) for whom cron really isn't appropriate for anything that we
>  > actually care to have done
>  
>  We still need to trigger pre-linking after yum transactions rather than
>  by a cron job... 

So we really need a generic yum-post-transaction trigger script.

something that says:

pkgname: action: command

ie:

glibc: *: /run/something/for/a/glibc/


How do we determine which pkgs changing need a new prelink run?

I can put this together probably today if that's what's needed.

-sv

[Colin Walters]

On Tue, Sep 9, 2008 at 9:51 AM, Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:
>  So we really need a generic yum-post-transaction trigger script.
>
>  something that says:
>
>  pkgname: action: command
>
>  ie:
>
>  glibc: *: /run/something/for/a/glibc/
>
>
>  How do we determine which pkgs changing need a new prelink run? 
This would be great and we've really needed it for a long time.
Ideally it would be structured to recognize packages based on their
content; for example, if a package installs an icon in
/usr/share/icons, we know to invoke gtk-update-icon-cache.

[Arjan van de Ven]

On Tue, 9 Sep 2008 10:01:09 -0400
"Colin Walters" <walters(a)verbum.org&gt; wrote:
>  This would be great and we've really needed it for a long time.
>  Ideally it would be structured to recognize packages based on their
>  content; for example, if a package installs an icon in
>  /usr/share/icons, we know to invoke gtk-update-icon-cache. 
that makes it a question if this should be in yum or in rpm....

>   

-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Seth Vidal]

On Tue, 2008-09-09 at 07:23 -0700, Arjan van de Ven wrote:
>  On Tue, 9 Sep 2008 10:01:09 -0400
>  "Colin Walters" <walters(a)verbum.org&gt; wrote:
>  > This would be great and we've really needed it for a long time.
>  > Ideally it would be structured to recognize packages based on their
>  > content; for example, if a package installs an icon in
>  > /usr/share/icons, we know to invoke gtk-update-icon-cache.
>  
>  that makes it a question if this should be in yum or in rpm....
>   
I'm fairly confident it'll be easier to do in yum.

-sv

[Arjan van de Ven]

On Tue, 09 Sep 2008 10:29:46 -0400
Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:

>  On Tue, 2008-09-09 at 07:23 -0700, Arjan van de Ven wrote:
>  > On Tue, 9 Sep 2008 10:01:09 -0400
>  > "Colin Walters" <walters(a)verbum.org&gt; wrote:
>  > > This would be great and we've really needed it for a long time.
>  > > Ideally it would be structured to recognize packages based on
>  > > their content; for example, if a package installs an icon in
>  > > /usr/share/icons, we know to invoke gtk-update-icon-cache.
>  > 
>  > that makes it a question if this should be in yum or in rpm....
>  > 
>  
>  I'm fairly confident it'll be easier to do in yum.
>   
I absolutely do not doubt that ;)

the question is more complex than 'what is easiest' if "correctness"
type things move into these checks rather than just optimization things.


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Seth Vidal]

On Tue, 2008-09-09 at 07:40 -0700, Arjan van de Ven wrote:
>  On Tue, 09 Sep 2008 10:29:46 -0400
>  Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:
>  
>  > On Tue, 2008-09-09 at 07:23 -0700, Arjan van de Ven wrote:
>  > > On Tue, 9 Sep 2008 10:01:09 -0400
>  > > "Colin Walters" <walters(a)verbum.org&gt; wrote:
>  > > > This would be great and we've really needed it for a long time.
>  > > > Ideally it would be structured to recognize packages based on
>  > > > their content; for example, if a package installs an icon in
>  > > > /usr/share/icons, we know to invoke gtk-update-icon-cache.
>  > > 
>  > > that makes it a question if this should be in yum or in rpm....
>  > > 
>  > 
>  > I'm fairly confident it'll be easier to do in yum.
>  > 
>  
>  I absolutely do not doubt that ;)
>  
>  the question is more complex than 'what is easiest' if "correctness"
>  type things move into these checks rather than just optimization things. 
I'll work on a simple plugin to handle this and we can see if it'll work
semi-correctly. If we decide it needs to be at the rpm-level then so be
it. :)

-sv

[Doug Ledford]

On Tue, 2008-09-09 at 10:29 -0400, Seth Vidal wrote:
>  On Tue, 2008-09-09 at 07:23 -0700, Arjan van de Ven wrote:
>  > On Tue, 9 Sep 2008 10:01:09 -0400
>  > "Colin Walters" <walters(a)verbum.org&gt; wrote:
>  > > This would be great and we've really needed it for a long time.
>  > > Ideally it would be structured to recognize packages based on their
>  > > content; for example, if a package installs an icon in
>  > > /usr/share/icons, we know to invoke gtk-update-icon-cache.
>  > 
>  > that makes it a question if this should be in yum or in rpm....
>  > 
>  
>  I'm fairly confident it'll be easier to do in yum. 
It doesn't matter where it's easy to do, what matters is it's done in
the right place.  Core system infrastructure should never be a case of
done the easy way, always the right way.

-- 
Doug Ledford <dledford(a)redhat.com&gt;
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

[Jesse Keating]

On Wed, 2008-09-10 at 11:35 -0400, Doug Ledford wrote:
>  > I'm fairly confident it'll be easier to do in yum.
>  
>  It doesn't matter where it's easy to do, what matters is it's done in
>  the right place.  Core system infrastructure should never be a case of
>  done the easy way, always the right way. 
Ever heard of prototyping? 

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Seth Vidal]

On Tue, 2008-09-09 at 10:01 -0400, Colin Walters wrote:
>  On Tue, Sep 9, 2008 at 9:51 AM, Seth Vidal
> <skvidal(a)fedoraproject.org&gt; wrote:
>  >
>  > So we really need a generic yum-post-transaction trigger script.
>  >
>  > something that says:
>  >
>  > pkgname: action: command
>  >
>  > ie:
>  >
>  > glibc: *: /run/something/for/a/glibc/
>  >
>  >
>  > How do we determine which pkgs changing need a new prelink run?
>  
>  This would be great and we've really needed it for a long time.
>  Ideally it would be structured to recognize packages based on their
>  content; for example, if a package installs an icon in
>  /usr/share/icons, we know to invoke gtk-update-icon-cache. 

Does it need to be run if the pkg removed an icon?

-sv

[Colin Walters]

On Tue, Sep 9, 2008 at 10:36 AM, Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:
>  Does it need to be run if the pkg removed an icon? 
Yeah, that would be more correct; but if we don't it just means that a
program will succeed in finding an icon it "shouldn't have", and I
can't offhand think of a reasonable case where that would be a
problem.

[Arjan van de Ven]

On Tue, 09 Sep 2008 09:51:37 -0400
Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:

>  On Mon, 2008-09-08 at 15:28 -0500, Callum Lerwick wrote:
>  > On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:
>  > > And doing it this way would be more effective than cron as there
>  > > are plenty of people (especially with laptops/desktops which
>  > > aren't on all the time) for whom cron really isn't appropriate
>  > > for anything that we actually care to have done
>  > 
>  > We still need to trigger pre-linking after yum transactions rather
>  > than by a cron job...
>  
>  
>  So we really need a generic yum-post-transaction trigger script.
>  
>  something that says:
>  
>  pkgname: action: command
>  
>  ie:
>  
>  glibc: *: /run/something/for/a/glibc/
>  
>  
>  How do we determine which pkgs changing need a new prelink run? 
anything that puts something in /usr/bin or /usr/lib (and their
variations)
so would be nice if it would glob on package name OR package content.



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Behdad Esfahbod]

Seth Vidal wrote:
>  On Mon, 2008-09-08 at 15:28 -0500, Callum Lerwick wrote:
> > On Mon, 2008-09-08 at 11:27 -0400, Jeremy Katz wrote:
> >> And doing it this way would be more effective than cron as there are
> >> plenty of people (especially with laptops/desktops which aren't on all
> >> the time) for whom cron really isn't appropriate for anything that we
> >> actually care to have done
> > We still need to trigger pre-linking after yum transactions rather than
> > by a cron job...
>  
>  
>  So we really need a generic yum-post-transaction trigger script.
>  
>  something that says:
>  
>  pkgname: action: command
>  
>  ie:
>  
>  glibc: *: /run/something/for/a/glibc/
>  
>  
>  How do we determine which pkgs changing need a new prelink run? 
Shouldn't this be done on RPM level?  Isn't this pretty much an extension of
RPM triggers?  Or it's so hard to do that that we now want to move stuff to yum?

Similar issues that can be automatically handled also:

  - Run gtk-update-icon-cache if icons have been added/removed.

  - Run fc-cache if fonts have been added/removed.

  - Run ldconfig when share objects are installed (this one is done already,
right?)


It's really the same problem.  Many of the RPM snippets we keep copying from
package to package can be automated.  I don't see how this one is different.


behdad


>  I can put this together probably today if that's what's
> needed.
>  
>  -sv

[Seth Vidal]

On Tue, 2008-09-09 at 13:51 -0400, Behdad Esfahbod wrote:
>  Shouldn't this be done on RPM level?  Isn't this pretty much
> an extension of
>  RPM triggers?  Or it's so hard to do that that we now want to move stuff to yum?

a proof of concept plugin for yum is almost finished. If we want to move
it to the rpm layer, fine. The rpm devs are busy and getting this added
to rpm will take some time. I don't really see the problem in working on
it here.


-sv

[Seth Vidal]

On Tue, 2008-09-09 at 13:57 -0400, Seth Vidal wrote:
>  On Tue, 2008-09-09 at 13:51 -0400, Behdad Esfahbod wrote:
>  > Shouldn't this be done on RPM level?  Isn't this pretty much an extension
> of
>  > RPM triggers?  Or it's so hard to do that that we now want to move stuff to
> yum?
>  
>  
>  a proof of concept plugin for yum is almost finished. If we want to move
>  it to the rpm layer, fine. The rpm devs are busy and getting this added
>  to rpm will take some time. I don't really see the problem in working on
>  it here.
>   
Here's a proof of concept plugin:

http://skvidal.fedorapeople.org/misc/post-transaction-actions/


put the .py file in /usr/lib/yum-plugins
put the .conf file in /etc/yum/pluginconf.d/
make an /etc/yum/post-actions/ and put the .action file in there - add
more .action files as you'd like.

run yum transactions and see what happens...

-sv

[Colin Walters]

On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:
>  Here's a proof of concept plugin:
>
>  http://skvidal.fedorapeople.org/misc/post-transaction-actions/ 
Ok cool - with one small fix to the plugin, this simple action:

# Update the GTK+ icon cache when an icon is installed.
# See: http://www.gtk.org/api/2.6/gtk/gtk-update-icon-cache.html
/usr/share/icons/hicolor/*:any:gtk-update-icon-cache --quiet --force
/usr/share/icons/hicolor

seems to work.  Now we can apply hundreds (thousand?) of patches of
the following form, and probably in many cases entirely eliminate the
single %post and %postun we've been copying and pasting around.  It
will be awesome.

Index: hotssh.spec
===================================================================
RCS file: /cvs/pkgs/rpms/hotssh/devel/hotssh.spec,v
retrieving revision 1.1
diff -u -r1.1 hotssh.spec
--- hotssh.spec	4 Aug 2008 19:01:32 -0000	1.1
+++ hotssh.spec	10 Sep 2008 18:14:01 -0000
@@ -4,7 +4,7 @@
 Summary: Secure Shell Client
 Name: hotssh
 Version: 0.2.5
-Release: 1%{?dist}
+Release: 2%{?dist}
 Source0: http://ftp.gnome.org/pub/GNOME/sources/hotssh/0.2/hotssh-%{version}.tar.bz2
 License: GPLv2+
 Group: User Interface/Desktops
@@ -55,15 +55,10 @@
 %dir %{python_sitelib}/hotssh/
 %{python_sitelib}/hotssh/*

-%post
-touch --no-create %{_datadir}/icons/hicolor || :
-%{_bindir}/gtk-update-icon-cache --quiet %{_datadir}/icons/hicolor || :
-
-%postun
-touch --no-create %{_datadir}/icons/hicolor || :
-%{_bindir}/gtk-update-icon-cache --quiet %{_datadir}/icons/hicolor || :
-
 %changelog
+* Wed Sep 10 2008 Colin Walters <walters(a)redhat.com&gt; - 0.2.5-2
+- Delete icon cache bits, now in yum post transaction
+
 * Fri Aug 01 2008 Colin Walters <walters(a)redhat.com&gt; - 0.2.5-1
 - New upstream

[Seth Vidal]

On Wed, 2008-09-10 at 14:14 -0400, Colin Walters wrote:
>  On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal
> <skvidal(a)fedoraproject.org&gt; wrote:
>  >
>  > Here's a proof of concept plugin:
>  >
>  > http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  
>  Ok cool - with one small fix to the plugin, this simple action:
>   

Thank you, I've applied this to the one I uploaded above.

So this seems to work - now the question is - should we push for this to
go into rpm instead?


-sv

[Doug Ledford]

On Wed, 2008-09-10 at 14:16 -0400, Seth Vidal wrote:
>  On Wed, 2008-09-10 at 14:14 -0400, Colin Walters wrote:
>  > On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:
>  > >
>  > > Here's a proof of concept plugin:
>  > >
>  > > http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  > 
>  > Ok cool - with one small fix to the plugin, this simple action:
>  > 
>  
>  
>  Thank you, I've applied this to the one I uploaded above.
>  
>  So this seems to work - now the question is - should we push for this to
>  go into rpm instead? 
Absolutely.  If it's going to exist at all, it needs to be in rpm.
Otherwise this and the corresponding rpm changes fail the litmus test of
"will my system be the same if I download the rpm(2) and run the
transaction manually using rpm as it is if I use yum to install the
rpm".  Any time you fail that litmus test, you've put the code in the
wrong place (at least IMO).

-- 
Doug Ledford <dledford(a)redhat.com&gt;
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

[Seth Vidal]

On Wed, 2008-09-10 at 14:24 -0400, Doug Ledford wrote:
>  On Wed, 2008-09-10 at 14:16 -0400, Seth Vidal wrote:
>  > On Wed, 2008-09-10 at 14:14 -0400, Colin Walters wrote:
>  > > On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt;
> wrote:
>  > > >
>  > > > Here's a proof of concept plugin:
>  > > >
>  > > > http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  > > 
>  > > Ok cool - with one small fix to the plugin, this simple action:
>  > > 
>  > 
>  > 
>  > Thank you, I've applied this to the one I uploaded above.
>  > 
>  > So this seems to work - now the question is - should we push for this to
>  > go into rpm instead?
>  
>  Absolutely.  If it's going to exist at all, it needs to be in rpm.
>  Otherwise this and the corresponding rpm changes fail the litmus test of
>  "will my system be the same if I download the rpm(2) and run the
>  transaction manually using rpm as it is if I use yum to install the
>  rpm".  Any time you fail that litmus test, you've put the code in the
>  wrong place (at least IMO). 
unless we bite the bullet and stop falling back on being able to do
everything from the rpm interface.

And just move forward with doing things from the yum-cli and/or yum
modules.

Using the rpm interface for rescue/recovery functionality only.

-sv

[Michael Stone]

> And just move forward with doing things from the yum-cli and/or yum
> modules.
>
> Using the rpm interface for rescue/recovery functionality only. 
Please remember that people use packaging tools for more than simple
system maintenance; for example, I use an image-builder which is now
based on smart (even though it was originally based on yum) because
smart is more convenient for some of my programmatic use cases [1].

Basically, diversity in this area is a Good Thing, in my humble opinion.
Please don't impair it without a really good reason when alternatives
exist (such as pushing this functionality into RPM).

Thanks,

Michael

[1]: In particular, because of its programmable mirror system, priority
system, support for alternate package formats, built-in ability to
download packages, builtin ability to save URLs for the packages it is
going to download, strictness options, etc.)

[Jef Spaleta]

On Wed, Sep 10, 2008 at 12:16 PM, Michael Stone <michael(a)laptop.org&gt; wrote:
>  [1]: In particular, because of its programmable mirror system,
> priority
>  system, support for alternate package formats, built-in ability to
>  download packages, builtin ability to save URLs for the packages it is
>  going to download, strictness options, etc.) 
Hey maybe we should push some of those features down the the rpm layer too.

-jef

[Seth Vidal]

On Wed, 2008-09-10 at 16:16 -0400, Michael Stone wrote:

>  Please remember that people use packaging tools for more than simple
>  system maintenance; for example, I use an image-builder which is now
>  based on smart (even though it was originally based on yum) because
>  smart is more convenient for some of my programmatic use cases [1]. 
I think I'd like to look at those use cases in more detail. If there's
anything that can be done in smart's interface that isn't doable in via
the yum python modules I'd be curious to know about it (excluding sudoku
b/c I just don't care)

>  [1]: In particular,
>  because of its programmable mirror system,  
- how much more programmable do you
want than being able to modify the
mirrorlist directly per repo?

>  priority system,  
available in a yum plugin - though
questionably useful to begin with.

>  support for alternate package formats, 
not sure how this helps
fedora.

>  built-in ability to download packages, 
yum can clearly download
pkgs - not sure why it being built in rather
than a plugin matters though.

Also you can download directly if you're using the yum modules.


>  builtin ability to save URLs for the packages it is going to
> download, 
see above.

>  strictness options, etc.) 
strictness of what?


explain a bit more about the features you're looking for and I'll see
what I can do to help.

-sv

[Michael Stone]

On Wed, Sep 10, 2008 at 04:46:27PM -0400, Seth Vidal wrote:
> On Wed, 2008-09-10 at 16:16 -0400, Michael Stone wrote:
>
> > Please remember that people use packaging tools for more than simple
> > system maintenance; for example, I use an image-builder which is now
> > based on smart (even though it was originally based on yum) because
> > smart is more convenient for some of my programmatic use cases [1].
>
> I think I'd like to look at those use cases in more detail.  
I'm happy to explain in more detail, though I'll do so in my own order.

First, some introduction: my software [1] consists of a family of python
programs (called "compilations", since each is a primitive compiler), designed
to be run in a mock chroot, which convert collections of packages and hacks
into publishables (for example a list of installed packages, a XO JFFS2 disk
image, and a tarball of the filesystem contents). 

   [1]: http://wiki.laptop.org/go/Puritan

Its primary goal is to ease the task of creating releaseable disk images for
OLPC XOs in a reproducible and verifiable fashion by storing all effective
inputs to the build process in git commits. Its secondary goal is to overcome
some of the infelicities of pilgrim with respect to error detection, cleanup,
and debuggability. It's third goal is to be useful to expert Python programmers
who maintain Linux builds for fixed hardware regardless of their preferred
distro and package-format. It therefore exists in contrast to general-purpose
tools like debian-installer, anaconda, revisor, and livecd-tools, which are
distro-specific build tools with an interest in shielding their users from the
tool's implementation's underlying error-detection, analysis, and correction
mechanisms and workflows, often by means of a domain-specific language.

Smart is convenient to these goals in that it has a thoroughly documented shell
interface which permits the following useful operations:

Smart permits me to control my selection of package repositories package more
succinctly and less distro-specifically than yum; e.g.

   smart channel -y --remove-all
   smart channel -y --add bootstrap type='rpm-md'
baseurl='http://dev.laptop.org/~mstone/puritan-repo/'
   smart channel -y --add bootstrap-f9 type='rpm-md'
baseurl='http://dev.laptop.org/~mstone/puritan-f9-repo/'
   smart channel -y --enable bootstrap
   smart channel -y --enable bootstrap-f9
   smart update
   smart install -y olpc-crcimg python-msutils

I was able to write these basic commands without reference to the smart source
code using only its man page; could I have done the same with yum?

   (In release compilations, the rpm repos being sourced would be included as
    git submodules of the compilation commit thereby achieving full history good
    reproducibility.)

Next, after adding two new repos (for olpc-specific packages and for general
F-9 packages), I run

   for pkg in firefox seamonkey mozplugger kdebase kernel; do
     smart priority --set $pkg olpc-joyride 100
   done

in order to bias several packages toward the OLPC repo. I could do something
similar in yum with the '--exclude' option or by modifying the yum repo
configuration, but it would be much more annoying either way in that I'd need
more complicated code to pipe the results of my configuration phase into the
installation phase.

Next, mirror configuration:

   smart mirror --add
http://koji.fedoraproject.org/static-repos/dist-olpc3-build-current/i386/
http://koji.fedoraproject.org/packages/

This command essentially lets me perform url-rewriting as needed. This ability
is not _necessary_ for my purpose, but it sure is convenient from time to time.

> > priority system, 
> available in a yum plugin - though questionably useful to begin with. 
Yum plugins seem, at first blush and in general rather than in specific, highly
inconvenient for my purpose because of issues like the:

  * lack of uniform quality control 
  * absence of centralized documentation 
  * potential lack of unified error handling conventions,
  * perceived difficulty of installation and potential for version mismatch
    between plugin and host,
  * lack of a API stability guarantees

(Plugins offer a different and compelling social benefit; namely, pluggability;
however, that's not the social benefit that _I_ happen to need at the moment.)

> > support for alternate package formats,
> not sure how this helps fedora. 
First, must it help Fedora in order to be valuable to me or to be important to
my particular use case? Live and let live, friend. 

Second, please acknowledge that I am, for the time being dedicated first to the
XO hardware and second to Fedora. This being what it is, and there lots of
other people out there who would like to get their pet distribution running on
the XO hardware, it's important to me to be able to collaborate with others
using the same tools that I (hope to soon) use during the rest of my day. Isn't
that going to benefit Fedora insofar as it helps make Fedora packages work
better on all 400k XOs we've shipped it on and insofar as it stimulates more
people to make useful contributions to upstream projects?

> > built-in ability to download packages,
> yum can clearly download pkgs - not sure why it being built in rather
> than a plugin matters though. 
For the reasons I discussed above -- less complexity for me, the tool-writer.

> > strictness options, etc.)
>
> strictness of what? 
Error-checking. Pilgrim was notorious for producing broken builds because
pilgrim failed or was unable to detect situations in which yum was unable to
install all the requested packages. Perhaps this is a documentation problem --
I note that the yum man page contains no information about what error codes yum
will or can be made to return (if any?).

In particular, 

   yum -d 10 -e 10 install grhlkjoei

returns 0. This needlessly complicates my life: I _DO NOT_ want to have to
parse the yum output in order to learn that a problem occurred.

> explain a bit more about the features you're looking for and
> I'll see
> what I can do to help. 
In conclusion, yum does lots of helpful things but it doesn't do the exact
useful things that I need. Yum can certainly be made better by polishing some
of these infelicities and I hope that my exposition offers you some useful
guidance on conventions and social properties that make my life, as a generic
Unix programmer working in Python, much easier were yum's goals a better match
for my own.

Regarding the possibility of using the yum python modules directly: 

  * what documentation exists?
  * what guarantees of API stability exist?
  * what sample code exists and is it sane or contorted?

Also, consider the simplicity of my solution: I accomplished everything I
needed to (repo configuration, priorities, mirrors, downloading, installation,
cross-distro tool, and full error-detection) in something like 40-60 lines of
my code which I was able to write from a single man-page and the online help
for my tool of choice. I'd be interested in knowing whether you can offer me
something comparable. 

Kind regards,

Michael

P.S. - I hope I've successfully communicated that I really am interested in my
continued ability to make use of all the hard work that goes into maintaining
Fedora's packages even though I find myself presently unable to justify doing
so with yum?

[Seth Vidal]

On Wed, 2008-09-10 at 20:48 -0400, Michael Stone wrote:
>  I'm happy to explain in more detail, though I'll do so in my
> own order.
>  
>  First, some introduction: my software [1] consists of a family of python
>  programs (called "compilations", since each is a primitive compiler), designed
>  to be run in a mock chroot, which convert collections of packages and hacks
>  into publishables (for example a list of installed packages, a XO JFFS2 disk
>  image, and a tarball of the filesystem contents). 
>  
>     [1]: http://wiki.laptop.org/go/Puritan
>  
>  Its primary goal is to ease the task of creating releaseable disk images for
>  OLPC XOs in a reproducible and verifiable fashion by storing all effective
>  inputs to the build process in git commits. Its secondary goal is to overcome
>  some of the infelicities of pilgrim with respect to error detection, cleanup,
>  and debuggability. It's third goal is to be useful to expert Python programmers
>  who maintain Linux builds for fixed hardware regardless of their preferred
>  distro and package-format. It therefore exists in contrast to general-purpose
>  tools like debian-installer, anaconda, revisor, and livecd-tools, which are
>  distro-specific build tools with an interest in shielding their users from the
>  tool's implementation's underlying error-detection, analysis, and correction
>  mechanisms and workflows, often by means of a domain-specific language.
>  
>  Smart is convenient to these goals in that it has a thoroughly documented shell
>  interface which permits the following useful operations:
>  
>  Smart permits me to control my selection of package repositories package more
>  succinctly and less distro-specifically than yum; e.g.
>  
>     smart channel -y --remove-all
>     smart channel -y --add bootstrap type='rpm-md'
> baseurl='http://dev.laptop.org/~mstone/puritan-repo/'
>     smart channel -y --add bootstrap-f9 type='rpm-md'
> baseurl='http://dev.laptop.org/~mstone/puritan-f9-repo/'
>     smart channel -y --enable bootstrap
>     smart channel -y --enable bootstrap-f9
>     smart update
>     smart install -y olpc-crcimg python-msutils
>  
>  I was able to write these basic commands without reference to the smart source
>  code using only its man page; could I have done the same with yum?
>  
>     (In release compilations, the rpm repos being sourced would be included as
>      git submodules of the compilation commit thereby achieving full history good
>      reproducibility.)
>  
>  Next, after adding two new repos (for olpc-specific packages and for general
>  F-9 packages), I run
>  
>     for pkg in firefox seamonkey mozplugger kdebase kernel; do
>       smart priority --set $pkg olpc-joyride 100
>     done
>   

I think you can do all of the above in the yum shell

yum shell
> repo enable foo
> repo disable bar
> config assumeyes 1
> update
> install pkg1 pkg2 pkg3
> run 
>  in order to bias several packages toward the OLPC repo. I could do
> something
>  similar in yum with the '--exclude' option or by modifying the yum repo
>  configuration, but it would be much more annoying either way in that I'd need
>  more complicated code to pipe the results of my configuration phase into the
>  installation phase. 
You can set the repo cost, too.

>  
>  Next, mirror configuration:
>  
>     smart mirror --add
> http://koji.fedoraproject.org/static-repos/dist-olpc3-build-current/i386/
> http://koji.fedoraproject.org/packages/
>  
>  This command essentially lets me perform url-rewriting as needed. This ability
>  is not _necessary_ for my purpose, but it sure is convenient from time to time. 


>  >> priority system, 
>  >available in a yum plugin - though questionably useful to begin with.
>  
>  Yum plugins seem, at first blush and in general rather than in specific, highly
>  inconvenient for my purpose because of issues like the:
>  
>    * lack of uniform quality control  
- So all features in smart have complete
uniform quality control? I've
read smart's bug reports. I don't think all features are tested
identically.

>    * absence of centralized documentation  
>    * potential lack of unified error handling conventions,
>    * perceived difficulty of installation and potential for version mismatch
>      between plugin and host,
>    * lack of a API stability guarantees 
What on earth? We've had a few bugs but the yum api and the plugin api
has been stable for quite a while, now.



>  Second, please acknowledge that I am, for the time being dedicated
> first to the
>  XO hardware and second to Fedora. This being what it is, and there lots of
>  other people out there who would like to get their pet distribution running on
>  the XO hardware, it's important to me to be able to collaborate with others
>  using the same tools that I (hope to soon) use during the rest of my day. Isn't
>  that going to benefit Fedora insofar as it helps make Fedora packages work
>  better on all 400k XOs we've shipped it on and insofar as it stimulates more
>  people to make useful contributions to upstream projects?
>   
>  Error-checking. Pilgrim was notorious for producing broken builds
> because
>  pilgrim failed or was unable to detect situations in which yum was unable to
>  install all the requested packages. Perhaps this is a documentation problem --
>  I note that the yum man page contains no information about what error codes yum
>  will or can be made to return (if any?).
>  
>  In particular, 
>  
>     yum -d 10 -e 10 install grhlkjoei
>  
>  returns 0. This needlessly complicates my life: I _DO NOT_ want to have to
>  parse the yum output in order to learn that a problem occurred. 
Then don't use shell! Use python.



>  Regarding the possibility of using the yum python modules directly: 
>  
>    * what documentation exists? 
pydoc has a fair bit in docstrings for yum. We have
lots of good example
code to work from, too. Both James and I have been posting common
examples to get people started writing their own utilities using the yum
modules.

>    * what guarantees of API stability exist? 
3.2.X is guaranteed api stable. If we've broken compat it is not
intentional.

>    * what sample code exists and is it sane or contorted? 
http://illiterat.livejournal.com/6254.html
http://fedoraproject.org/wiki/SethVidal/YumCodeSnippets

all of the yum-utils are pretty good code examples, too.


Finally, I'd like to mention that excluding the 'make it work with
random package formats' everything you've described above is wheel
reinvention. You've respun what we've already done and had available in
fedora, rhel, rhl for years. It sure feels like there's a lot of NIH at
work in OLPC development and that's not productive for anyone and
certainly not very collaborative.

-sv

[Dominik 'Rathann' Mierzejewski]

On Wednesday, 10 September 2008 at 20:42, Seth Vidal wrote:
>  On Wed, 2008-09-10 at 14:24 -0400, Doug Ledford wrote:
>  > On Wed, 2008-09-10 at 14:16 -0400, Seth Vidal wrote:
>  > > On Wed, 2008-09-10 at 14:14 -0400, Colin Walters wrote:
>  > > > On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal
> <skvidal(a)fedoraproject.org&gt; wrote:
>  > > > >
>  > > > > Here's a proof of concept plugin:
>  > > > >
>  > > > > http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  > > > 
>  > > > Ok cool - with one small fix to the plugin, this simple action:
>  > > > 
>  > > 
>  > > 
>  > > Thank you, I've applied this to the one I uploaded above.
>  > > 
>  > > So this seems to work - now the question is - should we push for this to
>  > > go into rpm instead?
>  > 
>  > Absolutely.  If it's going to exist at all, it needs to be in rpm.
>  > Otherwise this and the corresponding rpm changes fail the litmus test of
>  > "will my system be the same if I download the rpm(2) and run the
>  > transaction manually using rpm as it is if I use yum to install the
>  > rpm".  Any time you fail that litmus test, you've put the code in the
>  > wrong place (at least IMO).
>  
>  unless we bite the bullet and stop falling back on being able to do
>  everything from the rpm interface. 
-1

Fedora doesn't need to go where Suse already is.

Regards,
R.

-- 
Fedora http://fedoraproject.org/wiki/User:Rathann
Livna http://rpm.livna.org | MPlayer http://mplayerhq.hu
"Faith manages."
        -- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"

[Rahul Sundaram]

Dominik 'Rathann' Mierzejewski wrote:
>  On Wednesday, 10 September 2008 at 20:42, Seth Vidal wrote: 
> > unless we bite the bullet and stop falling back on being able to
> do
> > everything from the rpm interface.
>  
>  -1
>  
>  Fedora doesn't need to go where Suse already is. 
Instead of voting, it would be better, if you can explain why it is 
wrong in your opinion.  +1/-1 doesn't really convey much useful 
information to assign any weight in a debate.

Rahul

[Nicolas Mailhot]

Le Jeu 11 septembre 2008 08:41, Rahul Sundaram a écrit :
>  Dominik 'Rathann' Mierzejewski wrote:
> > On Wednesday, 10 September 2008 at 20:42, Seth Vidal wrote:
>
> >> unless we bite the bullet and stop falling back on being able to do
> >> everything from the rpm interface.
> >
> > -1
> >
> > Fedora doesn't need to go where Suse already is.
>
>  Instead of voting, it would be better, if you can explain why it is
>  wrong in your opinion.  +1/-1 doesn't really convey much useful
>  information to assign any weight in a debate. 
Yum has progressed a lot in past years but it is not significantly
better than competitors (as evidenced by the fact rpm distros have not
adopted in in mass) and thus entrenching yum by relying on
yum-specific features feels to me rather premature. Competition is
healthy.

Plus rpm.org has been resurected so there's no compelling reason not
to do stuff at this level.

-- 
Nicolas Mailhot

[Dominik 'Rathann' Mierzejewski]

On Thursday, 11 September 2008 at 08:41, Rahul Sundaram wrote:
>  Dominik 'Rathann' Mierzejewski wrote:
>  >On Wednesday, 10 September 2008 at 20:42, Seth Vidal wrote:
>  
>  >>unless we bite the bullet and stop falling back on being able to do
>  >>everything from the rpm interface.
>  >
>  >-1
>  >
>  >Fedora doesn't need to go where Suse already is.
>  
>  Instead of voting, it would be better, if you can explain why it is 
>  wrong in your opinion.  +1/-1 doesn't really convey much useful 
>  information to assign any weight in a debate. 
I believe it is the one who proposes the change that has to explain
why it is necessary and can't be done in any other way. It has already
been said why it is wrong in this thread: people still use rpm command
to install packages. Is there a very good reason to force them to use
yum instead? Yum is still slower than plain rpm. After all, it is
a frontend and I do not think forcing people to use frontends is a good
idea when the tool behind the frontend can do the job just as well.

Regards,
R.

-- 
Fedora http://fedoraproject.org/wiki/User:Rathann
Livna http://rpm.livna.org | MPlayer http://mplayerhq.hu
"Faith manages."
        -- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"

[Panu Matilainen]

On Wed, 10 Sep 2008, Seth Vidal wrote:

>  On Wed, 2008-09-10 at 14:14 -0400, Colin Walters wrote:
> > On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:
> >>
> >> Here's a proof of concept plugin:
> >>
> >> http://skvidal.fedorapeople.org/misc/post-transaction-actions/
> >
> > Ok cool - with one small fix to the plugin, this simple action:
>
>  Thank you, I've applied this to the one I uploaded above.
>
>  So this seems to work - now the question is - should we push for this to
>  go into rpm instead? 
Not instead, but to rpm *too*, yes.

Quite obviously an action that's currently done from say %post script 
should stay at rpm level, moving some things to be only done once per 
transaction is just an optimization really. But yum knows a lot more about 
the surrounding world than rpm does, such as which repository a package 
came from which is something rpm has no clue about, and thus it can do 
things in post-transaction that rpm simply could not do. I don't have an 
example use-case in mind atm but I'm sure somebody can come up with some 
relatively sane example ;)

 	- Panu -

[Dominik 'Rathann' Mierzejewski]

On Thursday, 11 September 2008 at 10:34, Panu Matilainen wrote:
>  On Wed, 10 Sep 2008, Seth Vidal wrote:
>  
>  >On Wed, 2008-09-10 at 14:14 -0400, Colin Walters wrote:
>  >>On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt; 
>  >>wrote:
>  >>>
>  >>>Here's a proof of concept plugin:
>  >>>
>  >>>http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  >>
>  >>Ok cool - with one small fix to the plugin, this simple action:
>  >
>  >Thank you, I've applied this to the one I uploaded above.
>  >
>  >So this seems to work - now the question is - should we push for this to
>  >go into rpm instead?
>  
>  Not instead, but to rpm *too*, yes.
>  
>  Quite obviously an action that's currently done from say %post script 
>  should stay at rpm level, moving some things to be only done once per 
>  transaction is just an optimization really. But yum knows a lot more about 
>  the surrounding world than rpm does, such as which repository a package 
>  came from which is something rpm has no clue about, and thus it can do 
>  things in post-transaction that rpm simply could not do. 
Granted, but why would it matter where a package came from? All packages
should be treated the same, regardless of their origin. I smell some
package racism brewing up here. ;)

Regards,
R.

-- 
Fedora http://fedoraproject.org/wiki/User:Rathann
Livna http://rpm.livna.org | MPlayer http://mplayerhq.hu
"Faith manages."
        -- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"

[Panu Matilainen]

On Thu, 11 Sep 2008, Dominik 'Rathann' Mierzejewski wrote:

>  On Thursday, 11 September 2008 at 10:34, Panu Matilainen wrote:
> > On Wed, 10 Sep 2008, Seth Vidal wrote:
> >
> >> On Wed, 2008-09-10 at 14:14 -0400, Colin Walters wrote:
> >>> On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt;
> >>> wrote:
> >>>>
> >>>> Here's a proof of concept plugin:
> >>>>
> >>>> http://skvidal.fedorapeople.org/misc/post-transaction-actions/
> >>>
> >>> Ok cool - with one small fix to the plugin, this simple action:
> >>
> >> Thank you, I've applied this to the one I uploaded above.
> >>
> >> So this seems to work - now the question is - should we push for this to
> >> go into rpm instead?
> >
> > Not instead, but to rpm *too*, yes.
> >
> > Quite obviously an action that's currently done from say %post script
> > should stay at rpm level, moving some things to be only done once per
> > transaction is just an optimization really. But yum knows a lot more about
> > the surrounding world than rpm does, such as which repository a package
> > came from which is something rpm has no clue about, and thus it can do
> > things in post-transaction that rpm simply could not do.
>
>  Granted, but why would it matter where a package came from? All packages
>  should be treated the same, regardless of their origin. I smell some
>  package racism brewing up here. ;) 
Where a package came from is just one example of things that yum knows 
that rpm doesn't. Whether there's any real use-case for that particular 
piece of information is entirely different question. The point is that yum 
is in position to do various things that rpm cannot possibly do because 
they operate on different levels and with different amount of information 
available to them.

 	- Panu -

[Ralf Corsepius]

On Thu, 2008-09-11 at 12:18 +0300, Panu Matilainen wrote:

>  Where a package came from is just one example of things that yum
> knows 
>  that rpm doesn't. Whether there's any real use-case for that particular 
>  piece of information is entirely different question. The point is that yum 
>  is in position to do various things that rpm cannot possibly do because 
>  they operate on different levels and with different amount of information 
>  available to them. 
I would turn this argument around: rpm missed its opportunities
"to do
various things" forcing people to circumvent rpm's limitations by
ruck-sacking rpm with add-ons such as yum, yast/ycl etc.

Ralf

[Seth Vidal]

On Thu, 2008-09-11 at 12:33 +0200, Ralf Corsepius wrote:
>  On Thu, 2008-09-11 at 12:18 +0300, Panu Matilainen wrote:
>  
>  > Where a package came from is just one example of things that yum knows 
>  > that rpm doesn't. Whether there's any real use-case for that particular 
>  > piece of information is entirely different question. The point is that yum 
>  > is in position to do various things that rpm cannot possibly do because 
>  > they operate on different levels and with different amount of information 
>  > available to them.
>  I would turn this argument around: rpm missed its opportunities "to do
>  various things" forcing people to circumvent rpm's limitations by
>  ruck-sacking rpm with add-ons such as yum, yast/ycl etc. 
okay - but how does the above change anything. Yes, there was about 3-4
years of relatively no major rpm devel. And people engineered around the
outside of it.

1. why is this bad?
2. why is this particularly surprising?

-sv

[Dominik 'Rathann' Mierzejewski]

On Thursday, 11 September 2008 at 16:54, Seth Vidal wrote:
>  On Thu, 2008-09-11 at 12:33 +0200, Ralf Corsepius wrote:
>  > On Thu, 2008-09-11 at 12:18 +0300, Panu Matilainen wrote:
>  > 
>  > > Where a package came from is just one example of things that yum knows 
>  > > that rpm doesn't. Whether there's any real use-case for that particular
>
>  > > piece of information is entirely different question. The point is that yum 
>  > > is in position to do various things that rpm cannot possibly do because 
>  > > they operate on different levels and with different amount of information 
>  > > available to them.
>  > I would turn this argument around: rpm missed its opportunities "to do
>  > various things" forcing people to circumvent rpm's limitations by
>  > ruck-sacking rpm with add-ons such as yum, yast/ycl etc.
>  
>  okay - but how does the above change anything. Yes, there was about 3-4
>  years of relatively no major rpm devel. And people engineered around the
>  outside of it.
>  
>  1. why is this bad? 
It's bad because of duplicated work. We now have a dozen of rpm frontends
all of which do mostly the same things using different code.

>  2. why is this particularly surprising? 
Nobody said it was surprising.

Regards,
R.

-- 
Fedora http://fedoraproject.org/wiki/User:Rathann
Livna http://rpm.livna.org | MPlayer http://mplayerhq.hu
"Faith manages."
        -- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"

[Callum Lerwick]

On Thu, 2008-09-11 at 12:33 +0200, Ralf Corsepius wrote:
>  I would turn this argument around: rpm missed its opportunities
> "to do
>  various things" forcing people to circumvent rpm's limitations by
>  ruck-sacking rpm with add-ons such as yum, yast/ycl etc. 
Oh come on. How is layered, loosely coupled, encapsulated separation of
concerns NOT good design?

The diversity of RPM front ends is a sign we're doing things right.

Do we want to drag all of Yum's deps (python, and so on) down in to RPM
itself? How about C++?

RPM should be kept as small and simple as possible. We should avoid
creeping features into it. We need to think carefully about what RPM's
job really is. If anything it needs to be simpler. There was talk of
ripping the URL-download "feature" out of RPM. Did that ever actually
happen?

Two things are getting completely confounded in this thread.

1) Prelinking
2) GNOME icon cache

Prelinking is a non-vital system-wide optimization. It does not need to
be in RPM.

The GNOME icon cache is a whole other bag of crackrock. Is it really
required for system functioning? I'm assuming since we're bending over
backward for it, it is required. Duplicate spec scriptlets stink, but I
think the real problem is in GNOME. Should the cache really be system
wide? Why is it not per-user, and automatically re-generated? The whole
thing stinks of bad design, we shouldn't have to be special-casing crap
like this in our package management in the first place. It doesn't
belong in RPM *or* yum.

http://en.wikipedia.org/wiki/Code_smell

[Ralf Corsepius]

On Mon, 2008-09-15 at 00:48 -0500, Callum Lerwick wrote:
>  On Thu, 2008-09-11 at 12:33 +0200, Ralf Corsepius wrote:
>  > I would turn this argument around: rpm missed its opportunities "to do
>  > various things" forcing people to circumvent rpm's limitations by
>  > ruck-sacking rpm with add-ons such as yum, yast/ycl etc.
>  
>  Oh come on. How is layered, loosely coupled, encapsulated separation of
>  concerns NOT good design? 
The point is: Any actions on rpm must be performed
through rpm (rsp.
corresponding library calls= only.

In other words: rpm -U|-i|-e must work. High end frontends such as
yum/smart/apt/yast must not performing additional hacks interfering into
the system (e.g. prelinking) or the rpmdata bases.

>  The diversity of RPM front ends is a sign we're doing things
> right. 
I disagree. Provided the history of rpm, I take the diversity of
highlevel RPM front ends, as a sign of diverging interest and of
influence of vendor hegemony.

>  Do we want to drag all of Yum's deps (python, and so on) down in
> to RPM
>  itself? 
C.f. above. 

>   How about C++? 
Implementation detail.

>  RPM should be kept as small and simple as possible. 
It should
perform what is necessary to make rpm -U|-i|-e etc.
deterministically working.

Ralf

[Behdad Esfahbod]

Callum Lerwick wrote:
>  The diversity of RPM front ends is a sign we're doing things
> right. 
And the fact that none reliably works...

>  Do we want to drag all of Yum's deps (python, and so on) down in
> to RPM
>  itself? How about C++? 
Since when C code can't do all the stuff Python and C++ can?

>  Two things are getting completely confounded in this thread.
>  
>  1) Prelinking
>  2) GNOME icon cache
>  
>  Prelinking is a non-vital system-wide optimization. It does not need to
>  be in RPM. 
You are completely missing the discussion.  No one suggested RPM should do
prelinking or update GNOME icon cache.  The discussion is how to get RPM
trigger those.

>  The GNOME icon cache is a whole other bag of crackrock. 
Design a better one if you can.

>  Is it really
>  required for system functioning? I'm assuming since we're bending over
>  backward for it, it is required. Duplicate spec scriptlets stink, but I
>  think the real problem is in GNOME. Should the cache really be system
>  wide? Why is it not per-user, and automatically re-generated? The whole
>  thing stinks of bad design, we shouldn't have to be special-casing crap
>  like this in our package management in the first place. It doesn't
>  belong in RPM *or* yum. 
Allowing flexible triggers such that GTK+ can add a trigger to be run whenever
an icon is installed belongs in RPM.

behdad

>  http://en.wikipedia.org/wiki/Code_smell
>

[Seth Vidal]

On Mon, 2008-09-15 at 02:14 -0400, Behdad Esfahbod wrote:
>  Callum Lerwick wrote:
>  > The diversity of RPM front ends is a sign we're doing things right.
>  
>  And the fact that none reliably works... 
Do you think that might reflect on the complexity of problem more than
it reflects the tools?

-sv

[Behdad Esfahbod]

Seth Vidal wrote:
>  On Mon, 2008-09-15 at 02:14 -0400, Behdad Esfahbod wrote:
> > Callum Lerwick wrote:
> >> The diversity of RPM front ends is a sign we're doing things right.
> > And the fact that none reliably works...
>  
>  Do you think that might reflect on the complexity of problem more than
>  it reflects the tools? 
I'm sure it is a very very complex problem.  I also hear that a lot of these
problems are caused by RPM having to retain a lot of baggage for the name of
compatibility.  Hence my conclusion.

The question is: if one could throw RPM away and design a new one, could one
do significantly better?  From what I hear I think the answer might be yes.
What does it mean to Fedora:  we need to kick RPM in the butt to catch up with
competition.

Lets look back at the problem at hand: we all agree that custom-installed
glob-matched post-transaction triggers are useful things.  I think I can also
say that we agree that it should be in the lowest-level package management
system.  What has been up to debate so far is whether that lowest-level is
RPM, or that RPM is a lost case and yum is considered the lowest-level.  The
answer to that one does not matter.  A few data points that put this
discussion in perspective however:

  - Michael K Johnson, the very first Fedora Project leader, thought he can
invent a better RPM, left Red Hat, and built Conary.  It does use Python
indeed, and it does support glob-matched scripts.

  - First Mandriva and then SuSE felt they needed to heavily customize RPM and
went down that road.  I wonder how longer we need to wait to reach for the
same conclusion.

  - Kristian Høgsberg built razor which can solve deps in fraction of a second
where yum simply takes way too long.  Is anyone in the RPM camp impressed?
No.  They just don't think it's possible to do with RPM.

  - RPM already supports %posttrans scripts which are a huge improvement over
%post, but when I desperately needed them to refresh fontconfig caches upon
upgrade (using %post the old package is still installed when the script is
run; ouch!), but was told to not use %posttrans as it's broken and eats
babies.  I'm not sure if that's accurate, but that was the recommendation I
got from very core Fedora developers.  This is the kind of limitation RPM
imposes right now.

Now I understand that using RPM is one of Fedora's biggest assets ("industry
standard" blah blah), so I'm not suggesting that we should move to something
significantly different.  Just to acknowledge that there is a problem here,
identify solutions, and fix them.

I'd love to be corrected if any of the above statements is wrong.

behdad

>  -sv

[Seth Vidal]

On Mon, 2008-09-15 at 13:02 -0400, Behdad Esfahbod wrote:

>  I'm sure it is a very very complex problem.  I also hear that a
> lot of these
>  problems are caused by RPM having to retain a lot of baggage for the name of
>  compatibility.  Hence my conclusion. 
have you talked to the rpm maintainers? Also - compatibility is
important, not just to people working on longer-term oses like
rhel/centos/sled but also to fedora package maintainers - remember a lot
of the fedora maintainers also work on epel.


>  The question is: if one could throw RPM away and design a new one,
> could one
>  do significantly better?  From what I hear I think the answer might be yes.
>  What does it mean to Fedora:  we need to kick RPM in the butt to catch up with
>  competition. 
Have you tracked the changes that have been going on currently or do you
consider anything other than 'throwing it all away' not rapid enough
movement? I think you underestimate the sheer volume of code that needs
to be protected/verified and reimplemented if we were to just 'throw it
all away'.


>  Lets look back at the problem at hand: we all agree that
> custom-installed
>  glob-matched post-transaction triggers are useful things.  I think I can also
>  say that we agree that it should be in the lowest-level package management
>  system.  What has been up to debate so far is whether that lowest-level is
>  RPM, or that RPM is a lost case and yum is considered the lowest-level.  The
>  answer to that one does not matter.  A few data points that put this
>  discussion in perspective however:
>  
>    - Michael K Johnson, the very first Fedora Project leader, thought he can
>  invent a better RPM, left Red Hat, and built Conary.  It does use Python
>  indeed, and it does support glob-matched scripts. 
The migration path to conary for fedora is measured in years - not to
mention breaking track considerably with everything that came before us
and the projects forked from us. And, of course, we've just swapped our
known bugs for a whole new set of unknown bugs. It'll be fun to come
back a year after a migration to conary to find a whole different set of
complaints. Hey, Maybe we'll be encouraged to throw it all away, again.
That'd be amusing.

>  
>    - First Mandriva and then SuSE felt they needed to heavily customize RPM and
>  went down that road.  I wonder how longer we need to wait to reach for the
>  same conclusion. 
Mandriva and SuSE are working with rpm.org and collaborating to make rpm
better. They did customize things and we're all progressing on a common
tree.

>  
>    - Kristian Høgsberg built razor which can solve deps in fraction of a second
>  where yum simply takes way too long.  Is anyone in the RPM camp impressed?
>  No.  They just don't think it's possible to do with RPM. 

Kristian built something which solves some dependencies. Mostly he wrote
a mmap'd repometadata format.

>    - RPM already supports %posttrans scripts which are a huge
> improvement over
>  %post, but when I desperately needed them to refresh fontconfig caches upon
>  upgrade (using %post the old package is still installed when the script is
>  run; ouch!), but was told to not use %posttrans as it's broken and eats
>  babies.  I'm not sure if that's accurate, but that was the recommendation I
>  got from very core Fedora developers.  This is the kind of limitation RPM
>  imposes right now. 
People use posttrans now - the issue is that posttrans scripts often
make some intriguing assumptions about what you're running on and how
you're being installed. Ditto with pretrans. And posttrans doesn't play
nicely with the rpmdb being accessed/modified right now, either.


>  Now I understand that using RPM is one of Fedora's biggest assets
> ("industry
>  standard" blah blah), so I'm not suggesting that we should move to something
>  significantly different.  Just to acknowledge that there is a problem here,
>  identify solutions, and fix them. 
Have you interacted, at all, with rpm.org - Panu, Florian, Jindrich
about what they're working on? Maybe reviewed the slides from brno's
fudcon? I think they're posted somewhere publicly. Checked out the git
changelog for rpm.org? Looked at the rpm feature proposals for fedora?

Simply put, Fedora-devel is not the best place for learning about rpm
development. rpm-maint(a)lists.rpm.org is a better place. Look at the rpm
wiki, too:

http://wiki.rpm.org/DevPriorities


If you want to talk about rpm issues that need work, or better yet, if
you want to work on them - come to the rpm-maint list. I can think of
one or two items which I'm sure the rpm devs would love some additional
thoughts on.

File fingerprinting comes to mind as non-trivial and excruciating all at
the same time. And, of course, eminently necessary for putting down
files.

-sv

[Kristian Kristensen]

On Mon, 2008-09-15 at 14:04 -0400, Seth Vidal wrote:
>  On Mon, 2008-09-15 at 13:02 -0400, Behdad Esfahbod wrote:
>  >   - Kristian Høgsberg built razor which can solve deps in fraction of a second
>  > where yum simply takes way too long.  Is anyone in the RPM camp impressed?
>  > No.  They just don't think it's possible to do with RPM.
>  
>  
>  Kristian built something which solves some dependencies. Mostly he wrote
>  a mmap'd repometadata format. 
Look, depsolving isn't hard and razor is on par with yum as far as
correctness goes and in a whole different class as far as performance
goes.  I'm sure it's convenient to write it off as just a different
repodata format, but we both know razor is much more than that.

Our current package deployment stack is a horrible tangle of different
scripting languages, different database formats and an rpm codebase that
we're afraid to touch.  A lof of that complexity is self-inflicted, the
rest is horrible implementation.  We could be ambitious here and try to
own the problem from laying the files down on disk all the way up to
doing depsolving in an instant.  Instead we have a small group of people
adding bandaids to yum and tip-toeing around the rpm complexity, while
telling everybody how hard this problem is and we're lucky yum isn't
slower with all the things it does.

I don't have the time or energy to jump into this discussion, and when I
do, despite my better judgement, I get frustrated, upset and lose my
motivation for a week.  Keep your pathetic little kingdom.

Kristian

[Rahul Sundaram]

Kristian Høgsberg wrote:
>  On Mon, 2008-09-15 at 14:04 -0400, Seth Vidal wrote:
> > On Mon, 2008-09-15 at 13:02 -0400, Behdad Esfahbod wrote:
> >>   - Kristian Høgsberg built razor which can solve deps in fraction of a second
> >> where yum simply takes way too long.  Is anyone in the RPM camp impressed?
> >> No.  They just don't think it's possible to do with RPM.
> >
> > Kristian built something which solves some dependencies. Mostly he wrote
> > a mmap'd repometadata format.
>  
>  Look, depsolving isn't hard and razor is on par with yum as far as
>  correctness goes and in a whole different class as far as performance
>  goes.  
Is it in a usable state for day to day work or atleast for testing? I 
would like to see it packaged and more easily consumable in that case 
atleast for rawhide.

Rahul

[Richard Hughes]

On Tue, 2008-09-16 at 02:02 +0530, Rahul Sundaram wrote:
>  Is it in a usable state for day to day work or atleast for testing? I
>
>  would like to see it packaged and more easily consumable in that case 
>  atleast for rawhide. 
I've got local rpms of razor on my system, but I'm not sure they are
ready to inflict even on developers yet.

Razor is cool to play with and hack on, but I think it needs some more
work before it's even ready to release a first version.

But, my god, it's quick.

Richard.

[Mark]

On Tue, Sep 16, 2008 at 11:17 AM, Richard Hughes <hughsient(a)gmail.com&gt; wrote:
>  On Tue, 2008-09-16 at 02:02 +0530, Rahul Sundaram wrote:
> > Is it in a usable state for day to day work or atleast for testing? I
> > would like to see it packaged and more easily consumable in that case
> > atleast for rawhide.
>
>  I've got local rpms of razor on my system, but I'm not sure they are
>  ready to inflict even on developers yet.
>
>  Razor is cool to play with and hack on, but I think it needs some more
>  work before it's even ready to release a first version.
>
>  But, my god, it's quick.
>
>  Richard.
>
>
>  --
>  fedora-devel-list mailing list
>  fedora-devel-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-devel-list
>  
The discussion here isn't really going about the 5 second boot anymore..
I hope we can go back on that now since i really want to know how that's done.

[James Antill]

On Mon, 2008-09-15 at 13:02 -0400, Behdad Esfahbod wrote:

>  The question is: if one could throw RPM away and design a new one,
> could one
>  do significantly better? 
 Of course one could, the relevant question is _how long_ would it take.
And could it be done faster by just fixing rpm, and I've not seen any
compelling arguments that it would be faster to throw away what we have.

 It is also wide believed as true in the software field that "re-write
from scratch" is a last resort, and will always take longer and be more
expensive etc.
 There are cases where the subset of problems people care about is
smaller than those solved by the original program, "everyone" dislikes
the original program and a small group of good programmers are willing
to spend/invest a _lot_ of time to create a replacement. But I can only
think of a handful of examples here, and there's a reason for that.

>  Lets look back at the problem at hand: we all agree that
> custom-installed
>  glob-matched post-transaction triggers are useful things.  I think I can also
>  say that we agree that it should be in the lowest-level package management
>  system.  What has been up to debate so far is whether that lowest-level is
>  RPM, or that RPM is a lost case and yum is considered the lowest-level. 
 That is a severe mis-reading of the discussion, the question is given
that rpm+yum are currently how all Fedora users manage their system. Do
we want to still require that all packaging problems should be solved at
the rpm layer, or should we try to move up and allow some more of the
problems to be solved at the yum layer.
 There are many advantages to doing this, including just plain ease of
implementation. The only real disadvantage is that apt/smart/zypp/etc.
will become even more of a second class citizen in Fedora than they
already are (although I'm confident that the change proposed by Seth
could easily be ported to work in all of the above).

-- 
James Antill <james.antill(a)redhat.com&gt;
Red Hat

[Behdad Esfahbod]

James Antill wrote:
>  On Mon, 2008-09-15 at 13:02 -0400, Behdad Esfahbod wrote:
>  
> > The question is: if one could throw RPM away and design a new one, could one
> > do significantly better?
>  
>   Of course one could, the relevant question is _how long_ would it take.
>  And could it be done faster by just fixing rpm, and I've not seen any
>  compelling arguments that it would be faster to throw away what we have. 
I made it very clear in my mail that I'm not suggesting that we should throw
RPM away.  The question I asked is supposed to give one an idea of how much of
the problem is caused by the current implementation as opposed to the inherent
complexity of the problem.

> > Lets look back at the problem at hand: we all agree that
> custom-installed
> > glob-matched post-transaction triggers are useful things.  I think I can also
> > say that we agree that it should be in the lowest-level package management
> > system.  What has been up to debate so far is whether that lowest-level is
> > RPM, or that RPM is a lost case and yum is considered the lowest-level.
>  
>   That is a severe mis-reading of the discussion, the question is given
>  that rpm+yum are currently how all Fedora users manage their system. Do
>  we want to still require that all packaging problems should be solved at
>  the rpm layer, or should we try to move up and allow some more of the
>  problems to be solved at the yum layer.
>   There are many advantages to doing this, including just plain ease of
>  implementation. The only real disadvantage is that apt/smart/zypp/etc.
>  will become even more of a second class citizen in Fedora than they
>  already are (although I'm confident that the change proposed by Seth
>  could easily be ported to work in all of the above). 
What exactly in my paragraph you quotes is "a severe mis-reading of the
discussion"?  You say the exact same thing as I did except that instead of
calling RPM a lost case as I did, you call this approach making
apt/smart/zypp/etc. even more of a second class citizen.  End result is the
same pig: use yum or be screwed.

You and Seth seem to be on the camp that thinks it's no big deal if one uses
RPM directly and gets into a suboptimal state.  Some of us think that that's
unacceptable.  We disagree and that's fine.  Whoever ends up doing the work
decides what to do.  I fully understand that and don't mean to imply what
people should be working on.  Hope I've made myself clear this time.


behdad

[Seth Vidal]

On Mon, 2008-09-15 at 19:11 -0400, James Antill wrote:
>  On Mon, 2008-09-15 at 13:02 -0400, Behdad Esfahbod wrote:
>  
>  > The question is: if one could throw RPM away and design a new one, could one
>  > do significantly better?
>  
>   Of course one could, the relevant question is _how long_ would it take.
>  And could it be done faster by just fixing rpm, and I've not seen any
>  compelling arguments that it would be faster to throw away what we have.
>  
>   It is also wide believed as true in the software field that "re-write
>  from scratch" is a last resort, and will always take longer and be more
>  expensive etc.
>   There are cases where the subset of problems people care about is
>  smaller than those solved by the original program, "everyone" dislikes
>  the original program and a small group of good programmers are willing
>  to spend/invest a _lot_ of time to create a replacement. But I can only
>  think of a handful of examples here, and there's a reason for that.
>  
>  > Lets look back at the problem at hand: we all agree that custom-installed
>  > glob-matched post-transaction triggers are useful things.  I think I can also
>  > say that we agree that it should be in the lowest-level package management
>  > system.  What has been up to debate so far is whether that lowest-level is
>  > RPM, or that RPM is a lost case and yum is considered the lowest-level.
>  
>   That is a severe mis-reading of the discussion, the question is given
>  that rpm+yum are currently how all Fedora users manage their system. Do
>  we want to still require that all packaging problems should be solved at
>  the rpm layer, or should we try to move up and allow some more of the
>  problems to be solved at the yum layer.
>   There are many advantages to doing this, including just plain ease of
>  implementation. The only real disadvantage is that apt/smart/zypp/etc.
>  will become even more of a second class citizen in Fedora than they
>  already are (although I'm confident that the change proposed by Seth
>  could easily be ported to work in all of the above).
>   
s/easily/trivially/

it's a colon-delimited file format with pretty simple rules.

-sv

[Colin Walters]

2008/9/15 Callum Lerwick <seg(a)haxxed.com&gt;:
>  On Thu, 2008-09-11 at 12:33 +0200, Ralf Corsepius wrote:
> > I would turn this argument around: rpm missed its opportunities "to do
> > various things" forcing people to circumvent rpm's limitations by
> > ruck-sacking rpm with add-ons such as yum, yast/ycl etc.
>
>  Oh come on. How is layered, loosely coupled, encapsulated separation of
>  concerns NOT good design?
>
>  The diversity of RPM front ends is a sign we're doing things right. 
No.  If even 1/10 of the energy that goes into all these alternate
frontends went into fixing yum, we would have one better frontend
instead of multiples with different sets of bugs.

>  Do we want to drag all of Yum's deps (python, and so on) down in
> to RPM
>  itself? 
If you have a lot of C code, adding a scripting language on top is a
*very* sensible way to write an application.

>  The GNOME icon cache is a whole other bag of crackrock. Is it really
>  required for system functioning? 
Look, building an actual operating system turns out to be harder than
just wget and untar and call it done.  There are going to be other
things that will need global knowledge outside of an individual piece
of software.

>  I'm assuming since we're bending over
>  backward for it, it is required. Duplicate spec scriptlets stink, but I
>  think the real problem is in GNOME. Should the cache really be system
>  wide? Why is it not per-user, and automatically re-generated? 
That would be immensely wasteful on the kinds of servers that have
multiple users.  I would think this would be obvious and not require
explanation..

[Seth Vidal]

On Mon, 2008-09-15 at 07:12 -0400, Colin Walters wrote:

>  No.  If even 1/10 of the energy that goes into all these alternate
>  frontends went into fixing yum, we would have one better frontend
>  instead of multiples with different sets of bugs.
>   
Really? Then 1/10th of the energy? I'm intrigued to hear what you
believe is broken with yum, now. Better yet, I'm interested to see what
bugs you've filed.

I don't think yum is flaw free but I am curious what all things you
perceive as wrong that you also seem to believe can be fixed with 1/10th
the energy being spent on the various pkg managers.

-sv

[Colin Walters]

On Mon, Sep 15, 2008 at 7:53 AM, Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:
>  On Mon, 2008-09-15 at 07:12 -0400, Colin Walters wrote:
>
> > No.  If even 1/10 of the energy that goes into all these alternate
> > frontends went into fixing yum, we would have one better frontend
> > instead of multiples with different sets of bugs.
> >
>
>  Really? Then 1/10th of the energy? I'm intrigued to hear what you
>  believe is broken with yum, now. Better yet, I'm interested to see what
>  bugs you've filed. 
I personally don't have any real issues.  About the only thing that
annoys me is that it prompts "Y/n" for installing single packages, but
I've just gotten in the habit of using -y.  Apparently other people do
have issues, but I don't know much about it.

I think you perceived my message as being negative towards yum - it
was not intended to be so, quite the opposite.  When I said "fixing
yum" I meant that in the general way of "all software has flaws and
can be improved".

[Ralf Corsepius]

On Thu, 2008-09-11 at 11:34 +0300, Panu Matilainen wrote:
>  On Wed, 10 Sep 2008, Seth Vidal wrote:
>  
>  > On Wed, 2008-09-10 at 14:14 -0400, Colin Walters wrote:
>  >> On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt;
> wrote:
>  >>>
>  >>> Here's a proof of concept plugin:
>  >>>
>  >>> http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  >>
>  >> Ok cool - with one small fix to the plugin, this simple action:
>  >
>  > Thank you, I've applied this to the one I uploaded above.
>  >
>  > So this seems to work - now the question is - should we push for this to
>  > go into rpm instead?
>  
>  Not instead, but to rpm *too*, yes. 
Rpm only!

Yum a wrapper on-top of rpm, not a replacement for rpm nor a wrapper to
remedy packing bugs inside of rpm-packages.

Ralf

[Seth Vidal]

On Thu, 2008-09-11 at 11:34 +0300, Panu Matilainen wrote:
>  > So this seems to work - now the question is - should we push for
> this to
>  > go into rpm instead?
>  
>  Not instead, but to rpm *too*, yes.
>  
>  Quite obviously an action that's currently done from say %post script 
>  should stay at rpm level, moving some things to be only done once per 
>  transaction is just an optimization really. But yum knows a lot more about 
>  the surrounding world than rpm does, such as which repository a package 
>  came from which is something rpm has no clue about, and thus it can do 
>  things in post-transaction that rpm simply could not do. I don't have an 
>  example use-case in mind atm but I'm sure somebody can come up with some 
>  relatively sane example ;) 
Recording info on what happened in each transaction in an external
database and wanting to include the info on where a package came from.

I was thinking of adding a set of variables you could pass to external
scripts. Something like:
 1. pkg nevra
 2. pkg repo (if pkg being installed/updated)
 3. action being taken on package (install, update, erase, etc)
 4. package checksum (if pkg being installed/updated)

Afaict all but 2 should be doable inside rpm. I'm just a little blurry
on how easy it will be to implement in rpm.

If you want it in rpm, though, that's fine by me. I'm holding off
checking this plugin into yum-utils until after this is discussed a bit.

-sv

[Behdad Esfahbod]

Colin Walters wrote:
>  On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal
> <skvidal(a)fedoraproject.org&gt; wrote:
> > Here's a proof of concept plugin:
> >
> > http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  
>  Ok cool - with one small fix to the plugin, this simple action:
>  
>  # Update the GTK+ icon cache when an icon is installed.
>  # See: http://www.gtk.org/api/2.6/gtk/gtk-update-icon-cache.html
>  /usr/share/icons/hicolor/*:any:gtk-update-icon-cache --quiet --force
>  /usr/share/icons/hicolor 
Err, shouldn't update cache for the entire /usr/share/icons/hicolor.  Just the
directory changed...


behdad

[Colin Walters]

On Wed, Sep 10, 2008 at 2:19 PM, Behdad Esfahbod <behdad(a)behdad.org&gt; wrote:
>  Colin Walters wrote:
> > On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:
> >> Here's a proof of concept plugin:
> >>
> >> http://skvidal.fedorapeople.org/misc/post-transaction-actions/
> >
> > Ok cool - with one small fix to the plugin, this simple action:
> >
> > # Update the GTK+ icon cache when an icon is installed.
> > # See: http://www.gtk.org/api/2.6/gtk/gtk-update-icon-cache.html
> > /usr/share/icons/hicolor/*:any:gtk-update-icon-cache --quiet --force
> > /usr/share/icons/hicolor
>
>  Err, shouldn't update cache for the entire /usr/share/icons/hicolor.  Just the
>  directory changed... 
Yeah, true.  Perhaps the simplest way to do this is for the yum plugin
to pass down the paths that matched?  Another alternative would be for
the the actions themselves to be Python code, and we just do the
equivalent of:

for plugin in plugin:
  plugin.run(package_name, changed_files_list)

Something to figure out for when this gets pushed into RPM.

[Matthias Clasen]

On Wed, 2008-09-10 at 14:23 -0400, Colin Walters wrote:
>  On Wed, Sep 10, 2008 at 2:19 PM, Behdad Esfahbod
> <behdad(a)behdad.org&gt; wrote:
>  > Colin Walters wrote:
>  >> On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt;
> wrote:
>  >>> Here's a proof of concept plugin:
>  >>>
>  >>> http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  >>
>  >> Ok cool - with one small fix to the plugin, this simple action:
>  >>
>  >> # Update the GTK+ icon cache when an icon is installed.
>  >> # See: http://www.gtk.org/api/2.6/gtk/gtk-update-icon-cache.html
>  >> /usr/share/icons/hicolor/*:any:gtk-update-icon-cache --quiet --force
>  >> /usr/share/icons/hicolor
>  >
>  > Err, shouldn't update cache for the entire /usr/share/icons/hicolor.  Just the
>  > directory changed...
>  
>  Yeah, true.   
Err, there's only a single cache per theme. Nothing per-directory

[Behdad Esfahbod]

Matthias Clasen wrote:
>  On Wed, 2008-09-10 at 14:23 -0400, Colin Walters wrote:
> > On Wed, Sep 10, 2008 at 2:19 PM, Behdad Esfahbod <behdad(a)behdad.org&gt; wrote:
> >> Colin Walters wrote:
> >>> On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal <skvidal(a)fedoraproject.org&gt;
> wrote:
> >>>> Here's a proof of concept plugin:
> >>>>
> >>>> http://skvidal.fedorapeople.org/misc/post-transaction-actions/
> >>> Ok cool - with one small fix to the plugin, this simple action:
> >>>
> >>> # Update the GTK+ icon cache when an icon is installed.
> >>> # See: http://www.gtk.org/api/2.6/gtk/gtk-update-icon-cache.html
> >>> /usr/share/icons/hicolor/*:any:gtk-update-icon-cache --quiet --force
> >>> /usr/share/icons/hicolor
> >> Err, shouldn't update cache for the entire /usr/share/icons/hicolor.  Just
> the
> >> directory changed...
> > Yeah, true.  
>  
>  Err, there's only a single cache per theme. Nothing per-directory 
Umm, I almost wrote "(this may not be an issue for icon theme cache, but
definitely will be for font cache)" but removed it.  Anyway, said now.

behdad

[Karel Zak]

On Wed, Sep 10, 2008 at 02:14:52PM -0400, Colin Walters wrote:
>  On Tue, Sep 9, 2008 at 5:33 PM, Seth Vidal
> <skvidal(a)fedoraproject.org&gt; wrote:
>  >
>  > Here's a proof of concept plugin:
>  >
>  > http://skvidal.fedorapeople.org/misc/post-transaction-actions/
>  
>  Ok cool - with one small fix to the plugin, this simple action:
>  
>  # Update the GTK+ icon cache when an icon is installed.
>  # See: http://www.gtk.org/api/2.6/gtk/gtk-update-icon-cache.html
>  /usr/share/icons/hicolor/*:any:gtk-update-icon-cache --quiet --force
>  /usr/share/icons/hicolor
>  
>  seems to work.  Now we can apply hundreds (thousand?) of patches of
>  the following form, and probably in many cases entirely eliminate the
>  single %post and %postun we've been copying and pasting around.  It
>  will be awesome. 
 +1; ...triggers make much more sense than hard coded (and duplicate)
 scripts.

    Karel

-- 
 Karel Zak  <kzak(a)redhat.com&gt;

[Tom London]

On Sat, Sep 6, 2008 at 3:52 AM, Harald Hoyer <harald(a)redhat.com&gt; wrote:
>  Hello fellow Fedora developers,
>
>  recently readahead was modified to adapt to system file changes and to start
>  very early in the boot process via upstart.
>
>  I would like to encourage you to test readahead-1.4.5-3.fc10 from rawhide
>  (even possible on F9), which I built some minutes ago. It may take a day to
>  reach your local mirror.
>
>  # yum --enablerepo=rawhide install readahead
>  or
>  # yum --enablerepo=rawhide update readahead
>
>  With the next reboot readahead-collector runs and collects the information
>  which files are used during the boot process. The next reboot then,
>  readahead read ahead those files and the boot process (from init start to
>  gdm login screen) should be approx. 10% faster.
>
>  Hope it speeds up your boot process a little bit. Please report any changes
>  in your boot time (can be measured with a stop clock or bootchart).
>
>  You can modify /etc/sysconfig/readahead to turn readahead on/off. 
Running latest rawhide on Thinkpad X60, measured with stopwatch:

Prior to installing readahead:
     time from start of boot to gdm login: 56 seconds
     time from entering password to gnome applets appearing in upper
right panel: 23 seconds

First boot after installing readahead:
     time from start of boot to gdm login: 61 seconds
     time from password to gnome: 22 seconds

Second boot after installing readahead
     time from start of boot to gdm login: 50 seconds
     time from password to gnome: 23 seconds

Appears 10% faster, as promised.... :-D

tom
-- 
Tom London

[Mark]

On Sat, Sep 6, 2008 at 9:03 PM, Tom London <selinux(a)gmail.com&gt; wrote:
>  On Sat, Sep 6, 2008 at 3:52 AM, Harald Hoyer
> <harald(a)redhat.com&gt; wrote:
> > Hello fellow Fedora developers,
> >
> > recently readahead was modified to adapt to system file changes and to start
> > very early in the boot process via upstart.
> >
> > I would like to encourage you to test readahead-1.4.5-3.fc10 from rawhide
> > (even possible on F9), which I built some minutes ago. It may take a day to
> > reach your local mirror.
> >
> > # yum --enablerepo=rawhide install readahead
> > or
> > # yum --enablerepo=rawhide update readahead
> >
> > With the next reboot readahead-collector runs and collects the information
> > which files are used during the boot process. The next reboot then,
> > readahead read ahead those files and the boot process (from init start to
> > gdm login screen) should be approx. 10% faster.
> >
> > Hope it speeds up your boot process a little bit. Please report any changes
> > in your boot time (can be measured with a stop clock or bootchart).
> >
> > You can modify /etc/sysconfig/readahead to turn readahead on/off.
>
>  Running latest rawhide on Thinkpad X60, measured with stopwatch:
>
>  Prior to installing readahead:
>      time from start of boot to gdm login: 56 seconds
>      time from entering password to gnome applets appearing in upper
>  right panel: 23 seconds
>
>  First boot after installing readahead:
>      time from start of boot to gdm login: 61 seconds
>      time from password to gnome: 22 seconds
>
>  Second boot after installing readahead
>      time from start of boot to gdm login: 50 seconds
>      time from password to gnome: 23 seconds
>
>  Appears 10% faster, as promised.... :-D
>
>  tom
>  --
>  Tom London
>
>  --
>  fedora-devel-list mailing list
>  fedora-devel-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-devel-list
>  
O wow good to see that readahead is finally getting worth something.
What i wonder is if the following are alose read ahead:
- The default icon theme is also read ahead?
- The GTK parts so that for example gnome-panel is open faster or any
gtk using app
- Are the most used user applications (Specially nautilus and firefox)
fully read ahead so that they start FAST (will greatly improve user
experience)

I hope the performance of readahead can go up a bit more (25%?)

Anyway good numbers so far!
Keep up posted if you want.

[Ville Skyttä]

Good stuff.

I see slightly more than 10% speedup with this on F-9 from end of grub to 
until the KDE splash screen disappears.  Ditto to until all apps that I have 
on autostart after a small modification.

The modifications I made to /etc/event.d/readahead-collector.event were 
commenting out the [ -f /.autorelabel ] test (no selinux here) as well as 
increasing the sleep time in pre-stop script to 50 so that all the apps I 
have on autostart get collected.

[Harald Hoyer]

Ville Skyttä schrieb:
>  Good stuff.
>  
>  I see slightly more than 10% speedup with this on F-9 from end of grub to 
>  until the KDE splash screen disappears.  Ditto to until all apps that I have 
>  on autostart after a small modification.
>  
>  The modifications I made to /etc/event.d/readahead-collector.event were 
>  commenting out the [ -f /.autorelabel ] test (no selinux here) as well as 
>  increasing the sleep time in pre-stop script to 50 so that all the apps I 
>  have on autostart get collected.
>   
.autorelabel check should be fixed in readahead-1.4.6 now and the time to sleep in
pre-stop can be configured 
now in /etc/sysconfig/readahead

[Arjan van de Ven]

On Sat, 06 Sep 2008 12:52:56 +0200
Harald Hoyer <harald(a)redhat.com&gt; wrote:

>  Hello fellow Fedora developers,
>  
>  recently readahead was modified to adapt to system file changes and
>  to start very early in the boot process via upstart.
>  
>  I would like to encourage you to test readahead-1.4.5-3.fc10 from
>  rawhide (even possible on F9), which I built some minutes ago. It may
>  take a day to reach your local mirror.
>  
>  # yum --enablerepo=rawhide install readahead
>  or
>  # yum --enablerepo=rawhide update readahead
>  
>  With the next reboot readahead-collector runs and collects the
>  information which files are used during the boot process. The next
>  reboot then, readahead read ahead those files and the boot process
>  (from init start to gdm login screen) should be approx. 10% faster.
>   
for those who are going to the plumbers conference... Auke Kok and I
will present there on how to make a Fedora based system boot in 5
seconds...

and yes a form of readahead was needed to achieve that ;-)


-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[David Nielsen]

2008/9/7 Arjan van de Ven <arjan(a)infradead.org&gt;

>  for those who are going to the plumbers conference... Auke Kok and I
>  will present there on how to make a Fedora based system boot in 5
>  seconds...
>
>  and yes a form of readahead was needed to achieve that ;-) 

5 SECS??

I sincerely hope they will be recording the talks either as audio or even
better as video because this I have got to see.

- David

[Harald Hoyer]

David Nielsen schrieb:
>  
>  
>  2008/9/7 Arjan van de Ven <arjan(a)infradead.org <mailto:[e-mail]>>
>  
>  
>      for those who are going to the plumbers conference... Auke Kok and I
>      will present there on how to make a Fedora based system boot in 5
>      seconds...
>  
>      and yes a form of readahead was needed to achieve that ;-)
>  
>  
>  5 SECS??
>  
>  I sincerely hope they will be recording the talks either as audio or 
>  even better as video because this I have got to see.
>  
>  - David
>   
standard kernel + initrd alone takes more time than 12s on my system, so no chance to come
to 5s without 
building my own kernel with all modules compiled in.

[Christopher Brown]

2008/9/7 Harald Hoyer <harald(a)redhat.com&gt;:
>  David Nielsen schrieb:
> >
> >
> > 2008/9/7 Arjan van de Ven <arjan(a)infradead.org
> > <mailto:[e-mail]>>
> >
> >
> >    for those who are going to the plumbers conference... Auke Kok and I
> >    will present there on how to make a Fedora based system boot in 5
> >    seconds...
> >
> >    and yes a form of readahead was needed to achieve that ;-)
> >
> >
> > 5 SECS??
> >
> > I sincerely hope they will be recording the talks either as audio or even
> > better as video because this I have got to see.
> >
> > - David
> >
>
>  standard kernel + initrd alone takes more time than 12s on my system, so no
>  chance to come to 5s without building my own kernel with all modules
>  compiled in. 
Is this on spinning plates or solid state?

-- 
Christopher Brown

http://www.chruz.com

[drago01]

On Sun, Sep 7, 2008 at 6:21 AM, Arjan van de Ven <arjan(a)infradead.org&gt; wrote:
>  On Sat, 06 Sep 2008 12:52:56 +0200
>  Harald Hoyer <harald(a)redhat.com&gt; wrote:
>
> > Hello fellow Fedora developers,
> >
> > recently readahead was modified to adapt to system file changes and
> > to start very early in the boot process via upstart.
> >
> > I would like to encourage you to test readahead-1.4.5-3.fc10 from
> > rawhide (even possible on F9), which I built some minutes ago. It may
> > take a day to reach your local mirror.
> >
> > # yum --enablerepo=rawhide install readahead
> > or
> > # yum --enablerepo=rawhide update readahead
> >
> > With the next reboot readahead-collector runs and collects the
> > information which files are used during the boot process. The next
> > reboot then, readahead read ahead those files and the boot process
> > (from init start to gdm login screen) should be approx. 10% faster.
> >
>
>  for those who are going to the plumbers conference... Auke Kok and I
>  will present there on how to make a Fedora based system boot in 5
>  seconds...
>
>  and yes a form of readahead was needed to achieve that ;-) 
wtf?
without a custom kernel I doubt that this is possible....

[John Reiser]

drago01 wrote:
>  On Sun, Sep 7, 2008 at 6:21 AM, Arjan van de Ven
> <arjan(a)infradead.org&gt; wrote: 
   [snip]
> > for those who are going to the plumbers conference... Auke Kok
> and I
> > will present there on how to make a Fedora based system boot in 5
> > seconds...
> >
> > and yes a form of readahead was needed to achieve that ;-)
>  
>  wtf?
>  without a custom kernel I doubt that this is possible.... 
Shut down and boot an Apple Macintosh running current OS X 10.5.4.
As part of shutting down, it builds a RAM disk of what will be needed
at next boot, then saves the RAM disk to hard disk.  (I have not seen
the work by Arjan+Auke, but I boot many kinds of systems.)

--

[Vasile Gaburici]

On Mon, Sep 8, 2008 at 1:26 AM, John Reiser <jreiser(a)bitwagon.com&gt; wrote:
>  drago01 wrote:
> > On Sun, Sep 7, 2008 at 6:21 AM, Arjan van de Ven <arjan(a)infradead.org&gt; wrote:
>    [snip]
> >> for those who are going to the plumbers conference... Auke Kok and I
> >> will present there on how to make a Fedora based system boot in 5
> >> seconds...
> >>
> >> and yes a form of readahead was needed to achieve that ;-)
> >
> > wtf?
> > without a custom kernel I doubt that this is possible....
>
>  Shut down and boot an Apple Macintosh running current OS X 10.5.4.
>  As part of shutting down, it builds a RAM disk of what will be needed
>  at next boot, then saves the RAM disk to hard disk.  (I have not seen
>  the work by Arjan+Auke, but I boot many kinds of systems.) 
Interesting, so it's almost a hibernate in disguise. (I haven't
touched a Mac since 10.4)

>  --
>
>  --
>  fedora-devel-list mailing list
>  fedora-devel-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-devel-list
>
>

[Michel Salim]

On Sun, Sep 7, 2008 at 6:38 PM, Vasile Gaburici <vgaburici(a)gmail.com&gt; wrote:
>  On Mon, Sep 8, 2008 at 1:26 AM, John Reiser
> <jreiser(a)bitwagon.com&gt; wrote:
> > drago01 wrote:
> >> On Sun, Sep 7, 2008 at 6:21 AM, Arjan van de Ven <arjan(a)infradead.org&gt;
> wrote:
> >   [snip]
> >>> for those who are going to the plumbers conference... Auke Kok and I
> >>> will present there on how to make a Fedora based system boot in 5
> >>> seconds...
> >>>
> >>> and yes a form of readahead was needed to achieve that ;-)
> >>
> >> wtf?
> >> without a custom kernel I doubt that this is possible....
> >
> > Shut down and boot an Apple Macintosh running current OS X 10.5.4.
> > As part of shutting down, it builds a RAM disk of what will be needed
> > at next boot, then saves the RAM disk to hard disk.  (I have not seen
> > the work by Arjan+Auke, but I boot many kinds of systems.)
>
>  Interesting, so it's almost a hibernate in disguise. (I haven't
>  touched a Mac since 10.4) 
More like a "JIT"-optimized initrd, perhaps.

10.5 has some interesting optimizations, though the stricter-checking
fsck is really annoying. But that's not a discussion to have here.

-- 
Michel Salim
http://hircus.jaiku.com/

[Mark]

On Mon, Sep 8, 2008 at 12:26 AM, John Reiser <jreiser(a)bitwagon.com&gt; wrote:
>  drago01 wrote:
> > On Sun, Sep 7, 2008 at 6:21 AM, Arjan van de Ven <arjan(a)infradead.org&gt; wrote:
>    [snip]
> >> for those who are going to the plumbers conference... Auke Kok and I
> >> will present there on how to make a Fedora based system boot in 5
> >> seconds...
> >>
> >> and yes a form of readahead was needed to achieve that ;-)
> >
> > wtf?
> > without a custom kernel I doubt that this is possible....
>
>  Shut down and boot an Apple Macintosh running current OS X 10.5.4.
>  As part of shutting down, it builds a RAM disk of what will be needed
>  at next boot, then saves the RAM disk to hard disk.  (I have not seen
>  the work by Arjan+Auke, but I boot many kinds of systems.)
>
>  --
>
>  --
>  fedora-devel-list mailing list
>  fedora-devel-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-devel-list
>  
I've been wondering if that was possible or not. Now since you told it
is (for mac) i wonder if something like it exists for linux..
I've been looking for something like: "boot from ramdisk" but nothing
useful so far.

@Arjan,
Can you tell if you made code adjustments to readahead? or didn't you
use that (because you said "a form of")..
And any other code adjustments..

Men.. just tell us how you did this ^_^

[Doug Ledford]

On Mon, 2008-09-08 at 00:52 +0200, Mark wrote:
>  @Arjan,
>  Can you tell if you made code adjustments to readahead? or didn't you
>  use that (because you said "a form of")..
>  And any other code adjustments..
>  
>  Men.. just tell us how you did this ^_^ 
Hmm...well, first off the best way to speed up the kernel portion of the
boot process would be to identify all the kernel modules you load, and
load them in parallel.  You might need to tweak some of them to do
scheduled sleeps instead of busy loops when waiting for initialization
tasks to complete.  Then of course upstart and parallel service starts.

The biggest time waster in the boot process is serial initialization
delay waits.  Maybe you have a SCSI controller (the old parallel
kind ;-) that needs to scan for devices.  There is a 256ms timeout
(nominally) for each device *not* present on the bus.  If you have a
single disk on a wide bus, that's 14 timeouts you need to wait through
at 1/4 second each.  If you can have your net controllers, your USB
controllers, and other modules all doing this kind of stuff
simultaneously, you greatly speed up the boot process.  Thanks to udev,
persistent device naming, and being able to do things like set the eth
number by MAC address instead of order that the ethernet drivers are
loaded, you can really dump a lot of module loads into the kernel in
parallel and then just wait for them all to complete.  In that scenario
your total time wait is the time it takes your slowest module to
initialize (roughly).  Similarly with user space services.  Do the same
basic thing.

Readahead can be a tremendous help if you can actually start it before
all of the kernel modules are done initializing.  For instance, USB
devices have a settle delay before they are scanned, so starting
readahead as soon as the SATA driver and drive are up means the system
can be doing something useful while USB is finishing up.  However, I
suspect Arjan's system doesn't wait on any USB devices, so at most he
has host controllers but nothing slow to initialize.  Even just some USB
disks might be enough to blow that 5 second load time.

Am I close Arjan?

-- 
Doug Ledford <dledford(a)redhat.com&gt;
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

[Arjan van de Ven]

On Sun, 07 Sep 2008 21:11:56 -0400
Doug Ledford <dledford(a)redhat.com&gt; wrote:

>  Hmm...well, first off the best way to speed up the kernel portion of
>  the boot process would be to identify all the kernel modules you
>  load, and load them in parallel. 
close but no cigar; my fastboot git tree is used yes but not for modules
(we don't use modules for the devices in the system; they're so bog
standard that there's no value in using modules for them, and modules
are just too slow)

>   You might need to tweak some of
>  them to do scheduled sleeps instead of busy loops when waiting for
>  initialization tasks to complete.  Then of course upstart and
>  parallel service starts. 
yeah we don't use upstart; it made things slower not faster.

>  
>  The biggest time waster in the boot process is serial initialization
>  delay waits.  Maybe you have a SCSI controller (the old parallel
>  kind ;-) that needs to scan for devices.   
this is where kernel fastboot comes in.
>  
>  Readahead can be a tremendous help if you can actually start it before
>  all of the kernel modules are done initializing.  For instance, USB
>  devices have a settle delay before they are scanned, so starting
>  readahead as soon as the SATA driver and drive are up means the system
>  can be doing something useful while USB is finishing up.  
we only start readahead after the kernel is done... (but otherwise very
early)

(it's not the fedora readahead but a totally newly written one)


-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Linus Walleij]

mån 2008-09-08 klockan 00:52 +0200 skrev Mark:

>  > Shut down and boot an Apple Macintosh running current OS X
> 10.5.4.
>  > As part of shutting down, it builds a RAM disk of what will be needed
>  > at next boot, then saves the RAM disk to hard disk. 
>  I've been wondering if that was possible or not. Now since you
> told it
>  is (for mac) i wonder if something like it exists for linux.. 
At the Ottawa Linux Symposium the following presentation was
done:
http://tree.celinuxforum.org/CelfPubWiki/OttawaLinuxSymposium2006?action=...

The name is "snapshot boot" and it works well for systems with
well-written drivers. (Like some embedded...)

Linus

[Mark]

On Sun, Sep 7, 2008 at 6:21 AM, Arjan van de Ven <arjan(a)infradead.org&gt; wrote:
>  On Sat, 06 Sep 2008 12:52:56 +0200
>  Harald Hoyer <harald(a)redhat.com&gt; wrote:
>
> > Hello fellow Fedora developers,
> >
> > recently readahead was modified to adapt to system file changes and
> > to start very early in the boot process via upstart.
> >
> > I would like to encourage you to test readahead-1.4.5-3.fc10 from
> > rawhide (even possible on F9), which I built some minutes ago. It may
> > take a day to reach your local mirror.
> >
> > # yum --enablerepo=rawhide install readahead
> > or
> > # yum --enablerepo=rawhide update readahead
> >
> > With the next reboot readahead-collector runs and collects the
> > information which files are used during the boot process. The next
> > reboot then, readahead read ahead those files and the boot process
> > (from init start to gdm login screen) should be approx. 10% faster.
> >
>
>  for those who are going to the plumbers conference... Auke Kok and I
>  will present there on how to make a Fedora based system boot in 5
>  seconds...
>
>  and yes a form of readahead was needed to achieve that ;-)
>
>
>  --
>  If you want to reach me at my work email, use arjan(a)linux.intel.com
>  For development, discussion and tips for power savings,
>  visit http://www.lesswatts.org
>
>  --
>  fedora-devel-list mailing list
>  fedora-devel-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-devel-list
>  
Please!! make a video of that and full documentation on how to get
that done! i'm gona try that out as well when there are docs on how to
do it.

I sure hope your right in a 5 second boot.
I hope it's still with:
- Grub (or grub 2)
- that new rhgb replacement
- gdm/kdm/slim/xdm

Or do you make somekind of a boot dump and start that the next time you boot?

More info please! this is interesting.

[Arjan van de Ven]

On Sun, 7 Sep 2008 15:35:34 +0200
Mark <markg85(a)gmail.com&gt; wrote:

>  Please!! make a video of that and full documentation on how to get
>  that done! i'm gona try that out as well when there are docs on how to
>  do it.
>  
>  I sure hope your right in a 5 second boot.
>  I hope it's still with:
>  - Grub (or grub 2)
>  - that new rhgb replacement
>  - gdm/kdm/slim/xdm
>  
>  Or do you make somekind of a boot dump and start that the next time
>  you boot?
>   
without spoiling the presentation.. no we don't do fakes like
suspend/resume etc.

we start counting at the grub screen and stop counting when the GUI is
on the screen and the CPU and disk are idle.

we're not using rhgb (why would we? it's on the screen maybe at 8
seconds.. we're done at 5)


-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Eric Sandeen]

Arjan van de Ven wrote:
>  On Sun, 7 Sep 2008 15:35:34 +0200
>  Mark <markg85(a)gmail.com&gt; wrote:
>  
> > Please!! make a video of that and full documentation on how to get
> > that done! i'm gona try that out as well when there are docs on how to
> > do it.
> >
> > I sure hope your right in a 5 second boot.
> > I hope it's still with:
> > - Grub (or grub 2)
> > - that new rhgb replacement
> > - gdm/kdm/slim/xdm
> >
> > Or do you make somekind of a boot dump and start that the next time
> > you boot?
> >
>  
>  without spoiling the presentation..  
While the presentation is well and good, I hope that delaying it for the
presentation won't make it miss F10, if that would otherwise be a
possibility? :)

-Eric

>  no we don't do fakes like
>  suspend/resume etc.
>  
>  we start counting at the grub screen and stop counting when the GUI is
>  on the screen and the CPU and disk are idle.
>  
>  we're not using rhgb (why would we? it's on the screen maybe at 8
>  seconds.. we're done at 5)
>  
>

[Pádraig Brady]

Arjan van de Ven wrote:
>  for those who are going to the plumbers conference... Auke Kok and I
>  will present there on how to make a Fedora based system boot in 5
>  seconds...
>  
>  and yes a form of readahead was needed to achieve that ;-) 
There is some public info on this now:
http://lwn.net/Articles/299483/

[Rahul Sundaram]

Pádraig Brady wrote:
>  Arjan van de Ven wrote:
> > for those who are going to the plumbers conference... Auke Kok and I
> > will present there on how to make a Fedora based system boot in 5
> > seconds...
> >
> > and yes a form of readahead was needed to achieve that ;-)
>  
>  There is some public info on this now:
>  http://lwn.net/Articles/299483/ 
You are late. Check

https://bugzilla.redhat.com/show_bug.cgi?id=464045

Rahul

[Karel Zak]

On Thu, Oct 02, 2008 at 05:20:56PM +0530, Rahul Sundaram wrote:
>  Pádraig Brady wrote:
> > Arjan van de Ven wrote:
> >> for those who are going to the plumbers conference... Auke Kok and I
> >> will present there on how to make a Fedora based system boot in 5
> >> seconds...
> >>
> >> and yes a form of readahead was needed to achieve that ;-)
> >
> > There is some public info on this now:
> > http://lwn.net/Articles/299483/
>
>  You are late. Check
>
>  https://bugzilla.redhat.com/show_bug.cgi?id=464045 
 Well, on first glance the package is not ready for real distribution
 (for example see how filelist.c checks argv[1]...).

 I have a few questions,

  * sreadahead depends on Arjan's EXT3_IOC_INODE_JIFFIES ioctl, does
    Fedora kernel support this ioctl?

  * what generates the list of files?

  * shouldn't be better to add EXT3_IOC_INODE_JIFFIES support to
    standard Fedora readahead?

        Karel

-- 
 Karel Zak  <kzak(a)redhat.com&gt;