[ANNOUNCE]: thetango SRPM dependency system

[Prarit Bhargava]

The problems associated with bootstrapping an OS on a new architecture have
been around for a long time.  The issues with doing so are numerous -- how
to build, what order should the packages in the OS be built in, how to
identify build dependencies, etc.  Determining a build map, or build tree,
would go a long way in decreasing the time to entirely rebuild or bootstrap
an architecture.

As a starting point, a brute force build method of repeatedly traversing
the F7 SRPM list alphabetically was done.  This resulted in 
approximately 2500
packages built over a week -- far short of the complete list of SRPMS.
The frequency of successful builds slowed down dramatically after this 
number
of SRPMS were built.  The perl-* SRPMS are particularily nasty ...

thetango attempts to resolve the build problem by taking a set of SRPMS and
building build trees by evaluating the dependencies between the potential
resulting binary RPMs, and determining the build tree for individual SRPMS.

* I have not put this in source control yet.  The TODO section has an 
entry about this and that is something I hope to complete in the next week.

http://home.comcast.net/~prarit/thetango/

Prarit Bhargava
prarit(a)redhat.com
oftc.net: #ia64

: thetango SRPM dependency system

[Axel Thimm]

On Sun, Jul 01, 2007 at 08:16:02AM -0400, Prarit Bhargava wrote:
>  The problems associated with bootstrapping an OS on a new
> architecture have
>  been around for a long time.  The issues with doing so are numerous -- how
>  to build, what order should the packages in the OS be built in, how to
>  identify build dependencies, etc.  Determining a build map, or build tree,
>  would go a long way in decreasing the time to entirely rebuild or bootstrap
>  an architecture.
>  
>  As a starting point, a brute force build method of repeatedly traversing
>  the F7 SRPM list alphabetically was done.  This resulted in 
>  approximately 2500
>  packages built over a week -- far short of the complete list of SRPMS.
>  The frequency of successful builds slowed down dramatically after this 
>  number
>  of SRPMS were built.  The perl-* SRPMS are particularily nasty ... 
How can thetango know that perl-Foo will be providing perl(Bar) which
perl-Baz BuildRequires?

E.g. how can any automatic provides be guessed by thetango? A
not-too-far away goal is to have a perl()-like system for python and
other languages, which means more and more BRs depending on virtual
automatic provides that you can only detect after a build, not before.

I guess that if thetango cannot compute the depdency then it probably
gets discarded to at least get some better ordering, but the final
build will still need a brute force iterative approach.

>  thetango attempts to resolve the build problem by taking a set of
> SRPMS and
>  building build trees by evaluating the dependencies between the potential
>  resulting binary RPMs, and determining the build tree for individual SRPMS.
>  
>  * I have not put this in source control yet.  The TODO section has an 
>  entry about this and that is something I hope to complete in the next week.
>  
>  http://home.comcast.net/~prarit/thetango/
>  
>  Prarit Bhargava
>  prarit(a)redhat.com
>  oftc.net: #ia64
>   
-- 
Axel.Thimm at ATrpms.net

: thetango SRPM dependency system

[Prarit Bhargava]

Hi Axel,
>  How can thetango know that perl-Foo will be providing perl(Bar) which
>  perl-Baz BuildRequires?
>     
Good question, and I knew someone was going to ask how I got around that 
particular issue.  perl-* was a real pain to deal with.

/usr/lib/rpm/perl.prov is run on every source package to generate a 
close-to-complete list of perl(Bar::Foo) includes.

I explode the source, but do not compile, and run perl.prov on it.  This 
generates the perl(Bar::Foo) package entries.

For example perl-AppConfig-1.64-1.fc7.src.rpm provides:

perl-AppConfig
perl(AppConfig)
perl(AppConfig::Args)
perl(AppConfig::CGI)
perl(AppConfig::File)
perl(AppConfig::Getopt)
perl(AppConfig::State)
perl(AppConfig::Sys)

>  E.g. how can any automatic provides be guessed by thetango? A
>  not-too-far away goal is to have a perl()-like system for python and
>  other languages, which means more and more BRs depending on virtual
>  automatic provides that you can only detect after a build, not before.
>
>     
IIRC, there were only a handful of these.  I haven't decided what to do 
with these packages.  It maybe useful to have an additional list 
(perl-prov?) that is queried and perl.prov is run on them if they are in 
the list.

perl(mod_perl2) mod_perl-2.0.3-7.src.rpm
perl(Apache2::Cookie) libapreq2-2.09-0.rc2.4.fc7.src.rpm
perl(Apache2::Request) libapreq2-2.09-0.rc2.4.fc7.src.rpm
perl(Apache::Test) mod_perl-2.0.3-7.src.rpm
perl(Config) perl-5.8.8-18.fc7.src.rpm
perl(Data::UUID) uuid-1.5.1-3.fc7.src.rpm
perl(Errno) perl-5.8.8-18.fc7.src.rpm
perl(Image::Magick) ImageMagick-6.3.2.9-3.fc7.src.rpm
perl(Mail::SpamAssassin) spamassassin-3.2.0-1.fc7.src.rpm
perl(Newt) newt-0.52.6-3.fc7.src.rpm
perl(RRDs) rrdtool-1.2.23-3.fc7.src.rpm
perl(SVN::Core) subversion-1.4.3-4.src.rpm
perl(XML::SAX::Base) perl-XML-SAX-0.15-1.noarch.rpm
perl(XML::SAX::Exception) perl-XML-SAX-0.15-1.noarch.rpm
php-pear(PEAR) php-pear-PEAR-Command-Packaging-0.1.2-5.fc6.src.rpm

As I point out in the code, this same process needs to be done for python.

P.



>  I guess that if thetango cannot compute the depdency then it
> probably
>  gets discarded to at least get some better ordering, but the final
>  build will still need a brute force iterative approach.
>
>     
Nope.  This absolutely is 100 times better than a brute force iterative 
approach.  IMO of course :)

P.

: thetango SRPM dependency system

[Axel Thimm]

Hi,

On Sun, Jul 01, 2007 at 07:47:20PM -0400, Prarit Bhargava wrote:
>  >How can thetango know that perl-Foo will be providing perl(Bar)
> which
>  >perl-Baz BuildRequires?
>  
>  Good question, and I knew someone was going to ask how I got around that 
>  particular issue.  perl-* was a real pain to deal with.
>  
>  /usr/lib/rpm/perl.prov is run on every source package to generate a 
>  close-to-complete list of perl(Bar::Foo) includes. 
I see, good cheating ;)

You will probably have a couple of false positives from code that is
not being installed.

>  > I guess that if thetango cannot compute the depdency then it
> probably
>  > gets discarded to at least get some better ordering, but the final
>  > build will still need a brute force iterative approach.
>  
>  Nope.  This absolutely is 100 times better than a brute force iterative 
>  approach.  IMO of course :) 
It's surely better, what I mean is you cannot avoid the final
iterative approach, you can only provide a better ordered package list
to start with.

For example even when you can cheat a simulated build to find perl
dependecnies, as well as many other script language depedencies, you
will not be able to detect stuff needed for BuildRequires:
%{_libdir}/libfoo.so, which as brain-daed as it looks it is sometimes
needed when packages transition for not having *-devel to having
one. I even have submitted such a package.

But I agree, having an almost fully sorted filelist is 100x better than a
random one.

How about the following addition idea: Does restating thetango on
cached results with a couple different provides take up too much time?
Probably not. In that case you could append such an incremental
thetango call after each package is built and dynamically adjust the
ordering while building getting an almost 100% correct ordering. But
maybe it is already 95% good and the final 5% will be using brute
force, so it would be not worth while to do dynamical adjustments.
-- 
Axel.Thimm at ATrpms.net

: thetango SRPM dependency system

[Prarit Bhargava]

Axel Thimm wrote:
>  I see, good cheating ;)
>
>     
:o)

>  You will probably have a couple of false positives from code that is
>  not being installed.
>
>     
Interesting.  I'll keep that in mind.

> >> I guess that if thetango cannot compute the depdency then it
> probably
> >> gets discarded to at least get some better ordering, but the final
> >> build will still need a brute force iterative approach.
> >>       
> > Nope.  This absolutely is 100 times better than a brute force iterative 
> > approach.  IMO of course :)
> >     
>
>  It's surely better, what I mean is you cannot avoid the final
>  iterative approach, you can only provide a better ordered package list
>  to start with.
>
>  For example even when you can cheat a simulated build to find perl
>  dependecnies, as well as many other script language depedencies, you
>  will not be able to detect stuff needed for BuildRequires:
>  %{_libdir}/libfoo.so, which as brain-daed as it looks it is sometimes
>  needed when packages transition for not having *-devel to having
>  one. I even have submitted such a package.
>
>     
Isn't there also a libtool that would figure that out as well?  As the 
code mentions I have not implemented it yet, but I intend to do so.

>  But I agree, having an almost fully sorted filelist is 100x better
> than a
>  random one.
>     
I'm going to return to  the system that I did the brute force build on 
this week.  It is only a 64p/128G box, and I'm going to write code that 
uses a bottom-up tree algorithm to build the RPMS.  I'd like to see how 
much faster having the sorted list is.

>  How about the following addition idea: Does restating thetango on
>  cached results with a couple different provides take up too much time?
>  Probably not. In that case you could append such an incremental
>  thetango call after each package is built and dynamically adjust the
>  ordering while building getting an almost 100% correct ordering. But
>  maybe it is already 95% good and the final 5% will be using brute
>  force, so it would be not worth while to do dynamical adjustments.
>    
>     
Hmmm ... I haven't even considered that.  I'll add that to the TODO list. :)

P.