Nightly attempts of live image creation

[Jesse Keating]

I'm working on some scripting that will attempt to make the Live images
each night, and show some output.  Here is a first stab at what and
where one could find the output:

http://alt.fedoraproject.org/pub/alt/stage/development-live/Live/i686/

I'm no good with html, so when it comes time to script this up, I'll be
looking for some help to make this page a bit more useful, and also I'm
looking for input on what you'd like to see on these pages.

as an aside, we've got a snapshot due this Thursday/Friday and we're
still oversize.  The durastic space saving changes (package removals)
were only done on the beta branch because we were hoping to fix things
with better deps/builds for later.  Please see the above for what kind
of shape we're in.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Matthias Clasen]

On Tue, 2008-10-07 at 17:26 -0700, Jesse Keating wrote:
>  I'm working on some scripting that will attempt to make the Live
> images
>  each night, and show some output.  Here is a first stab at what and
>  where one could find the output:
>  
>  http://alt.fedoraproject.org/pub/alt/stage/development-live/Live/i686/
>  
>  I'm no good with html, so when it comes time to script this up, I'll be
>  looking for some help to make this page a bit more useful, and also I'm
>  looking for input on what you'd like to see on these pages.
>  
>  as an aside, we've got a snapshot due this Thursday/Friday and we're
>  still oversize.  The durastic space saving changes (package removals)
>  were only done on the beta branch because we were hoping to fix things
>  with better deps/builds for later.  Please see the above for what kind
>  of shape we're in.
>   
Frustrating. Again a lot of low-percentage growth, but what is really
killing us is the new packages that keep getting pulled in:

new package OpenEXR-libs: 823896
new package libv4l: 170205
new package gegl: 2161070
new package sane-backends-libs: 6404932
new package gnome-games: 18683521
new package babl: 341948
new package sgml-common: 135667
new package python-setuptools: 988965
new package pygtkglext: 368259
new package hplip-libs: 312821
new package sane-backends: 4246863
new package net-snmp-libs: 4707665
new package WebKit-gtk: 12024292
new package ilmbase: 408114
new package freeglut: 428017
new package lzma: 207155
new package PackageKit-udev-helper: 113030
new package PyOpenGL: 5647635
new package libsane-hpaio: 77784
new package iwl5000-firmware: 352062
new package guile: 3473157
new package libggz: 348426
new package PackageKit-gstreamer-plugin: 123694
new package PackageKit-yum-plugin: 114822
new package gnome-session-xsession: 4701
new package gnome-python2-gnomeprint: 443188
new package plymouth-plugin-solar: 512770
new package SDL: 502851
new package libbeagle: 99087
new package evolution-perl: 7823
new package docbook-dtds: 4568267
new package gnuchess: 3959589
new package lohit-fonts-marathi: 97286
new package lm_sensors: 500370
new package gnome-doc-utils-stylesheets: 1090771
new package libicu: 17202512
new package lzma-libs: 92737
new package gtkglext-libs: 484178
new package libieee1284: 69512
new package ggz-client-libs: 614106
new package yelp: 3333760

I assume a good chunk of these is because we need to reinstate the removal of gnome-games
and yelp, but there
some which are not explainable that way. E.g the new gimp now pulls in WebKit-gtk.

[Martin Sourada]

On Tue, 2008-10-07 at 23:45 -0400, Matthias Clasen wrote:
>  I assume a good chunk of these is because we need to reinstate the
> removal of gnome-games and yelp, but there
>  some which are not explainable that way. E.g the new gimp now pulls in WebKit-gtk.
>   
I'd say it's rather unfortunate that the switch from xulrunner to webkit
is not synchronised in upstream gnome projects, e.g. epiphany stayed for
one another release with xulrunner as its default backend (because of
still lacking features in webkit) and now I see, gimp already
switched :-/ For future it's probably better, because we will be able to
drop xulrunner and firefox in favour of smaller, but feature comparable
(when webkit-gtk implements some missing functionality), epiphany and
webkit-gtk, but the transition state will be pretty hard, since we'll
need both during that time...

Martin

[Nicu Buculei]

Martin Sourada wrote:
>  
>  I'd say it's rather unfortunate that the switch from xulrunner to webkit
>  is not synchronised in upstream gnome projects, e.g. epiphany stayed for 
Hate to bring this, but Gimp is not a GNOME project.

>  one another release with xulrunner as its default backend (because
> of
>  still lacking features in webkit) and now I see, gimp already
>  switched :-/ For future it's probably better, because we will be able to
>  drop xulrunner and firefox in favour of smaller, but feature comparable
>  (when webkit-gtk implements some missing functionality), epiphany and
>  webkit-gtk, but the transition state will be pretty hard, since we'll
>  need both during that time... 
 From my understanding, for the foreseeable future Firefox will remain 
the default browser in Fedora even if we also get a WebKit powered 
Epiphany (pretty much for the same reason we have not used a Gecko 
powered Epiphany as default - brand recognition).

-- 
nicu :: http://nicubunu.ro :: http://nicubunu.blogspot.com
Cool Fedora wallpapers: http://fedora.nicubunu.ro/wallpapers/
Open Clip Art Library: http://www.openclipart.org
my Fedora stuff: http://fedora.nicubunu.ro

[Nils Philippsen]

On Wed, 2008-10-08 at 10:25 +0300, Nicu Buculei wrote:

>  Hate to bring this, but Gimp is not a GNOME project. 
http://www.gnome.org/projects/ seems to suggest the opposite... ;-)

Nils
-- 
Nils Philippsen      "Those who would give up Essential Liberty to purchase 
Red Hat               a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com       nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:      C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Nicu Buculei]

Nils Philippsen wrote:
>  On Wed, 2008-10-08 at 10:25 +0300, Nicu Buculei wrote:
>  
> > Hate to bring this, but Gimp is not a GNOME project.
>  
>  http://www.gnome.org/projects/ seems to suggest the opposite... ;-) 
Well, that is a list of "GNOME-related projects".
OTOH, no mention of GNOME here: http://gimp.org/about/introduction.html


-- 
nicu :: http://nicubunu.ro :: http://nicubunu.blogspot.com
Cool Fedora wallpapers: http://fedora.nicubunu.ro/wallpapers/
Open Clip Art Library: http://www.openclipart.org
my Fedora stuff: http://fedora.nicubunu.ro

[Nils Philippsen]

On Wed, 2008-10-08 at 14:18 +0300, Nicu Buculei wrote:
>  Nils Philippsen wrote:
>  > On Wed, 2008-10-08 at 10:25 +0300, Nicu Buculei wrote:
>  > 
>  >> Hate to bring this, but Gimp is not a GNOME project.
>  > 
>  > http://www.gnome.org/projects/ seems to suggest the opposite... ;-)
>  
>  Well, that is a list of "GNOME-related projects".
>  OTOH, no mention of GNOME here: http://gimp.org/about/introduction.html 
I guess this comes from GIMP pre-dating GNOME by some time and boils
down to the question of what makes something a "GNOME project" -- it
uses GNOME infrastructure FWIW.

In the short time I spent on it I've not found a list of components that
are officially considered "GNOME" when put together, though.

Nils
-- 
Nils Philippsen      "Those who would give up Essential Liberty to purchase 
Red Hat               a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com       nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:      C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Nicu Buculei]

Nils Philippsen wrote:
>  On Wed, 2008-10-08 at 14:18 +0300, Nicu Buculei wrote:
> > Nils Philippsen wrote:
> >> On Wed, 2008-10-08 at 10:25 +0300, Nicu Buculei wrote:
> >>> Hate to bring this, but Gimp is not a GNOME project.
> >> http://www.gnome.org/projects/ seems to suggest the opposite... ;-)
> > Well, that is a list of "GNOME-related projects".
> > OTOH, no mention of GNOME here: http://gimp.org/about/introduction.html
>  
>  I guess this comes from GIMP pre-dating GNOME by some time and boils
>  down to the question of what makes something a "GNOME project" -- it
>  uses GNOME infrastructure FWIW.
>  
>  In the short time I spent on it I've not found a list of components that
>  are officially considered "GNOME" when put together, though. 
I know is pointless to continue with this, but I can't stop myself 
(found a better link): http://www.gimp.org/docs/userfaq.html#Gnome

Is true that GIMP uses a lot of GNOME resources, but it still retains 
its independence, this is why is not likely to expect them to have 
synchronized  library dependencies (like WebKit) or release dates.

-- 
nicu :: http://nicubunu.ro :: http://nicubunu.blogspot.com
Cool Fedora wallpapers: http://fedora.nicubunu.ro/wallpapers/
Open Clip Art Library: http://www.openclipart.org
my Fedora stuff: http://fedora.nicubunu.ro

[Nils Philippsen]

On Wed, 2008-10-08 at 15:57 +0300, Nicu Buculei wrote:
>  Nils Philippsen wrote:
>  > On Wed, 2008-10-08 at 14:18 +0300, Nicu Buculei wrote:
>  >> Nils Philippsen wrote:
>  >>> On Wed, 2008-10-08 at 10:25 +0300, Nicu Buculei wrote:
>  >>>> Hate to bring this, but Gimp is not a GNOME project.
>  >>> http://www.gnome.org/projects/ seems to suggest the opposite... ;-)
>  >> Well, that is a list of "GNOME-related projects".
>  >> OTOH, no mention of GNOME here: http://gimp.org/about/introduction.html
>  > 
>  > I guess this comes from GIMP pre-dating GNOME by some time and boils
>  > down to the question of what makes something a "GNOME project" -- it
>  > uses GNOME infrastructure FWIW.
>  > 
>  > In the short time I spent on it I've not found a list of components that
>  > are officially considered "GNOME" when put together, though.
>  
>  I know is pointless to continue with this, but I can't stop myself 
>  (found a better link): http://www.gimp.org/docs/userfaq.html#Gnome 
Point taken ;-). I seem to remember though that once upon a time GIMP
was listed as part of GNOME (when a new GNOME version was announced, by
the GNOME people -- maybe there's a different POV on this from both
projects...).

>  Is true that GIMP uses a lot of GNOME resources, but it still retains
>
>  its independence, this is why is not likely to expect them to have 
>  synchronized  library dependencies (like WebKit) or release dates. 
Agreed.

Nils
-- 
Nils Philippsen      "Those who would give up Essential Liberty to purchase 
Red Hat               a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com       nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:      C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Richard Hughes]

On Tue, 2008-10-07 at 23:45 -0400, Matthias Clasen wrote:
>  new package PackageKit-udev-helper: 113030 
Needed for the auto firmware install thing.

>  new package PackageKit-gstreamer-plugin: 123694 
Needed for the auto codec install thing.

>  new package PackageKit-yum-plugin: 114822 
The new name of yum-packagekit, needed so PackageKit can use yum as a
backend, and removable as SMART can be used instead.

Richard.

[Nils Philippsen]

On Tue, 2008-10-07 at 23:45 -0400, Matthias Clasen wrote:
>  On Tue, 2008-10-07 at 17:26 -0700, Jesse Keating wrote:
>  > I'm working on some scripting that will attempt to make the Live images
>  > each night, and show some output.  Here is a first stab at what and
>  > where one could find the output:
>  > 
>  > http://alt.fedoraproject.org/pub/alt/stage/development-live/Live/i686/
>  > 
>  > I'm no good with html, so when it comes time to script this up, I'll be
>  > looking for some help to make this page a bit more useful, and also I'm
>  > looking for input on what you'd like to see on these pages.
>  > 
>  > as an aside, we've got a snapshot due this Thursday/Friday and we're
>  > still oversize.  The durastic space saving changes (package removals)
>  > were only done on the beta branch because we were hoping to fix things
>  > with better deps/builds for later.  Please see the above for what kind
>  > of shape we're in.
>  > 
>  
>  Frustrating. Again a lot of low-percentage growth, but what is really
>  killing us is the new packages that keep getting pulled in:
>  
>  new package OpenEXR-libs: 823896
>  new package libv4l: 170205
>  new package gegl: 2161070
>  new package sane-backends-libs: 6404932
>  new package gnome-games: 18683521
>  new package babl: 341948
>  new package sgml-common: 135667
>  new package python-setuptools: 988965
>  new package pygtkglext: 368259
>  new package hplip-libs: 312821
>  new package sane-backends: 4246863
>  new package net-snmp-libs: 4707665
>  new package WebKit-gtk: 12024292
>  new package ilmbase: 408114
>  new package freeglut: 428017
>  new package lzma: 207155
>  new package PackageKit-udev-helper: 113030
>  new package PyOpenGL: 5647635
>  new package libsane-hpaio: 77784
>  new package iwl5000-firmware: 352062
>  new package guile: 3473157
>  new package libggz: 348426
>  new package PackageKit-gstreamer-plugin: 123694
>  new package PackageKit-yum-plugin: 114822
>  new package gnome-session-xsession: 4701
>  new package gnome-python2-gnomeprint: 443188
>  new package plymouth-plugin-solar: 512770
>  new package SDL: 502851
>  new package libbeagle: 99087
>  new package evolution-perl: 7823
>  new package docbook-dtds: 4568267
>  new package gnuchess: 3959589
>  new package lohit-fonts-marathi: 97286
>  new package lm_sensors: 500370
>  new package gnome-doc-utils-stylesheets: 1090771
>  new package libicu: 17202512
>  new package lzma-libs: 92737
>  new package gtkglext-libs: 484178
>  new package libieee1284: 69512
>  new package ggz-client-libs: 614106
>  new package yelp: 3333760
>  
>  I assume a good chunk of these is because we need to reinstate the removal of gnome-games
> and yelp, but there
>  some which are not explainable that way. E.g the new gimp now pulls in WebKit-gtk.
I can deactivate the internal help browser in GIMP, it currently has
stability issues anyway. Then it should use the external default browser
for displaying help (I don't know if we ship gimp-help in the live media
at all). Alternatively, I can split off the internal help browser so we
can leave that out on live media -- it should be installed on normal
installations though because (once it's not crashing) it's much faster
at startup than say firefox.

Nils
-- 
Nils Philippsen      "Those who would give up Essential Liberty to purchase 
Red Hat               a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com       nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:      C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Jeremy Katz]

On Wed, 2008-10-08 at 12:39 +0200, Nils Philippsen wrote:
>  I can deactivate the internal help browser in GIMP, it currently has
>  stability issues anyway. Then it should use the external default browser
>  for displaying help (I don't know if we ship gimp-help in the live media
>  at all). Alternatively, I can split off the internal help browser so we
>  can leave that out on live media -- it should be installed on normal
>  installations though because (once it's not crashing) it's much faster
>  at startup than say firefox. 
Is it faster than a new tab?  Since while it may be faster to start
without firefox already running, I suspect that most people already hvae
firefox running most of the time

Jeremy

[Nils Philippsen]

On Wed, 2008-10-08 at 09:02 -0400, Jeremy Katz wrote:
>  On Wed, 2008-10-08 at 12:39 +0200, Nils Philippsen wrote:
>  > I can deactivate the internal help browser in GIMP, it currently has
>  > stability issues anyway. Then it should use the external default browser
>  > for displaying help (I don't know if we ship gimp-help in the live media
>  > at all). Alternatively, I can split off the internal help browser so we
>  > can leave that out on live media -- it should be installed on normal
>  > installations though because (once it's not crashing) it's much faster
>  > at startup than say firefox.
>  
>  Is it faster than a new tab?  Since while it may be faster to start
>  without firefox already running, I suspect that most people already hvae
>  firefox running most of the time 
With a running firefox it doesn't make much difference, but an external
web browser likely doesn't have the table-of-contents tree beside the
actual help content that the built-in browser has.

Users that prefer an external browser can configure GIMP that way.

Nils
-- 
Nils Philippsen      "Those who would give up Essential Liberty to purchase 
Red Hat               a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com       nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:      C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Matthias Clasen]

On Wed, 2008-10-08 at 12:39 +0200, Nils Philippsen wrote:

>  >  E.g the new gimp now pulls in WebKit-gtk.
>  
>  I can deactivate the internal help browser in GIMP, it currently has
>  stability issues anyway. Then it should use the external default browser
>  for displaying help (I don't know if we ship gimp-help in the live media
>  at all). Alternatively, I can split off the internal help browser so we
>  can leave that out on live media -- it should be installed on normal
>  installations though because (once it's not crashing) it's much faster
>  at startup than say firefox. 
Splitting the help browser off sounds like a great idea, since we are
not shipping the gimp-help on the live cd anyway, as you say.

[Nils Philippsen]

On Wed, 2008-10-08 at 10:13 -0400, Matthias Clasen wrote:
>  On Wed, 2008-10-08 at 12:39 +0200, Nils Philippsen wrote:
>  
>  > >  E.g the new gimp now pulls in WebKit-gtk.
>  > 
>  > I can deactivate the internal help browser in GIMP, it currently has
>  > stability issues anyway. Then it should use the external default browser
>  > for displaying help (I don't know if we ship gimp-help in the live media
>  > at all). Alternatively, I can split off the internal help browser so we
>  > can leave that out on live media -- it should be installed on normal
>  > installations though because (once it's not crashing) it's much faster
>  > at startup than say firefox.
>  
>  Splitting the help browser off sounds like a great idea, since we are
>  not shipping the gimp-help on the live cd anyway, as you say.  
2.6.0-3 has the help browser splitt off into gimp-help-browser. I've
added obsolete lines so that package upgrades from older versions will
pull in the browser subpackage, as well as adding this to comps for F10.

Nils
-- 
Nils Philippsen      "Those who would give up Essential Liberty to purchase 
Red Hat               a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com       nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:      C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Rahul Sundaram]

Jesse Keating wrote:
>  I'm working on some scripting that will attempt to make the Live
> images
>  each night, and show some output.  Here is a first stab at what and
>  where one could find the output:
>  
>  http://alt.fedoraproject.org/pub/alt/stage/development-live/Live/i686/ 
Where are the scripts? Are they posted anywhere publicly?

Rahul

[Jesse Keating]

On Wed, 2008-10-08 at 09:23 +0530, Rahul Sundaram wrote:
>  Where are the scripts? Are they posted anywhere publicly? 
There are none, because they haven't been written yet.  This is the
result of doing some things by hand, to get a feel for what the script
needs to have in it, and what kind of output we want.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Richard Hughes]

On Tue, 2008-10-07 at 17:26 -0700, Jesse Keating wrote:
>  I'm working on some scripting that will attempt to make the Live
>  images each night, and show some output.  Here is a first stab at what
>  and where one could find the output: 
Cool, good work. Would it be possible to actually run the images and see
if if they boot? All it would need is starting the image in qemu and
then getting the image to ping/send a file/ssh the host, and then we can
kill the image. Potentially we could run this script nightly on a while
load of different architectures.

If you post the scripts, I'll see what I can do.

Richard.

[Jesse Keating]

On Wed, 2008-10-08 at 09:18 +0100, Richard Hughes wrote:
>  Cool, good work. Would it be possible to actually run the images and
> see
>  if if they boot? All it would need is starting the image in qemu and
>  then getting the image to ping/send a file/ssh the host, and then we can
>  kill the image. Potentially we could run this script nightly on a while
>  load of different architectures.
>  
>  If you post the scripts, I'll see what I can do. 
There are no scripts yet since I'm still doing things by hand until
we're happy with the results.

Attempting to boot them is going to be a little harder.  These compose
attempts will be running within a Xen guest, so I wouldn't be able to
make use of say KVM to boot them, and we don't have a lab of test
machines in the PHX infrastructure to do QA like tasks like this.  We
have a long term goal for this, but it's just not there right now.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Matthias Clasen]

On Tue, 2008-10-07 at 17:26 -0700, Jesse Keating wrote:

>  
>  I'm no good with html, so when it comes time to script this up, I'll be
>  looking for some help to make this page a bit more useful, and also I'm
>  looking for input on what you'd like to see on these pages. 
Things I would like to see are:

- full list of packages with sizes
- lists of new/removed packages compared to the beta/last snapshot
- list of size changes for all packages (including shrinkage) compare to
last snapshot. This list would imo be more useful, if the size change
was given in absolute terms and the list were sorted by size change. The
relative change you are showing now is not really helpful in finding the
big offenders, since 50% of a tiny package is still irrelevant.

[Jesse Keating]

On Wed, 2008-10-08 at 11:18 -0400, Matthias Clasen wrote:

at http://alt.fedoraproject.org/pub/alt/stage/development-live/Live/i686/

>  - full list of packages with sizes 
We have this, just not sorted in a good way just yet.  That'll be easy
to fix.

>  - lists of new/removed packages compared to the beta/last snapshot
That is there now, for both beta and the last day's attempt.

>  - list of size changes for all packages (including shrinkage) compare
> to
>  last snapshot.  
There.

>  This list would imo be more useful, if the size change
>  was given in absolute terms and the list were sorted by size change. The
>  relative change you are showing now is not really helpful in finding the
>  big offenders, since 50% of a tiny package is still irrelevant. 
The growth and shrinkage are sorted by the absolute amount of growth or
shrinkage.  The adds and removals are sorted by the size of the package.

Hoping this helps!

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Matthias Clasen]

On Wed, 2008-10-08 at 17:46 -0700, Jesse Keating wrote:
>  On Wed, 2008-10-08 at 11:18 -0400, Matthias Clasen wrote:
>  
>  at http://alt.fedoraproject.org/pub/alt/stage/development-live/Live/i686/
>  
>  > - full list of packages with sizes
>  
>  We have this, just not sorted in a good way just yet.  That'll be easy
>  to fix.
>  
>  > - lists of new/removed packages compared to the beta/last snapshot
>  
>  That is there now, for both beta and the last day's attempt.
>  
>  > - list of size changes for all packages (including shrinkage) compare to
>  > last snapshot. 
>  
>  There.
>  
>  > This list would imo be more useful, if the size change
>  > was given in absolute terms and the list were sorted by size change. The
>  > relative change you are showing now is not really helpful in finding the
>  > big offenders, since 50% of a tiny package is still irrelevant.
>  
>  The growth and shrinkage are sorted by the absolute amount of growth or
>  shrinkage.  The adds and removals are sorted by the size of the package.
>  
>  Hoping this helps!
>   
Very nice, thanks Jesse. Of course, the next step is that we need a
script to do a similar comparison for the files in an individual
package. Or does some rpmutils package already ship such a tool ?

[Jesse Keating]

On Wed, 2008-10-08 at 21:28 -0400, Matthias Clasen wrote:
>  Very nice, thanks Jesse. Of course, the next step is that we need a
>  script to do a similar comparison for the files in an individual
>  package. Or does some rpmutils package already ship such a tool ?
>   
Hrm, I don't think so, but we should ask around.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating