Make buildSRPMFromSCM faster?

[Richard W.M. Jones]

The first step of most Koji builds is buildSRPMFromSCM, where a
.src.rpm file is built from the git repo.

Currently this involves completely building a mock buildroot
containing all the BuildRequires, and running `rpmbuild -bs'.  This
takes many minutes (especially when arm is chosen as a builder).

It seems the reason for this is because the spec file has to be fully
parsed in order to work out the Source lines.  Since Source lines
might depend on RPM macros which might depend on any BuildRequire'd
package, every BR package must be installed in the mock root.
`rpmbuild -bs' takes seconds because it just bundles all the Source
files with the spec file into an SRPM.

Is this really necessary?

Two shortcuts seem possible:

(1) Limit the use of macros in Source lines, so that only a simple,
standard, perhaps pre-cached buildroot can be used.

(2) Perhaps uglier: Just build an SRPM that contains everything in
dist-git + everything in the lookaside cache, and hope for the best ...

Thoughts?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Richard Shaw]

How about not rebuilding the chroot every time... It's not like you have to
worry about leftover BR's from building another package.

Richard

[Eric Smith]

On Sat, Jul 19, 2014 at 7:26 AM, Richard Shaw <hobbes1069(a)gmail.com&gt; wrote:
>  How about not rebuilding the chroot every time... It's not like
> you have to
>  worry about leftover BR's from building another package. 
That could lead to packages that happen to build some of the time due
to an undetected dependency on something that isn't in the BR but
happens to be in a cached buildroot.

[Al Dunsmuir]

On Saturday, July 19, 2014, 8:20:30 PM, Eric Smith wrote:
>  On Sat, Jul 19, 2014 at 7:26 AM, Richard Shaw
> <hobbes1069(a)gmail.com&gt; wrote:
> > How about not rebuilding the chroot every time... It's not like you have to
> > worry about leftover BR's from building another package. 
>  That could lead to packages that happen to build some of the time
> due
>  to an undetected dependency on something that isn't in the BR but
>  happens to be in a cached buildroot. 
Then you would at least get failure or a working package.

Just  because  a package builds without failure doesn't mean it is the
intended build. A less desirable case would be a package with optional
build  dependencies,  that  gets  built  with different configurations
based  on  the  presence  of  extra packages that happen to be in that
cached buildroot.

Having the required packages for build and no more is important.

[Richard Shaw]

On Sat, Jul 19, 2014 at 7:20 PM, Eric Smith <spacewar(a)gmail.com&gt; wrote:

>  On Sat, Jul 19, 2014 at 7:26 AM, Richard Shaw
> <hobbes1069(a)gmail.com&gt;
>  wrote:
>  > How about not rebuilding the chroot every time... It's not like you have
>  to
>  > worry about leftover BR's from building another package.
>
>  That could lead to packages that happen to build some of the time due
>  to an undetected dependency on something that isn't in the BR but
>  happens to be in a cached buildroot. 

I fail to see why this would occur building an SRPM, the dependencies
should be identical for all packages, we're not talking about actually
building the package, just building the SRPM.

For that matter, instead of rebuilding a chroot every time, why not cache
one (tar?) so you get rid of the transaction overhead. It can still be
re-created on a regular basis in case any of the packages in the chroot get
updated, you just don't do it every time for every package.

Richard

[Eric Smith]

On Sat, Jul 19, 2014 at 7:12 PM, Richard Shaw <hobbes1069(a)gmail.com&gt; wrote:
>  On Sat, Jul 19, 2014 at 7:20 PM, Eric Smith
> <spacewar(a)gmail.com&gt; wrote:
> > That could lead to packages that happen to build some of the time due
> > to an undetected dependency on something that isn't in the BR but
> > happens to be in a cached buildroot.
>  I fail to see why this would occur building an SRPM, the dependencies should 
Right. A "thinko" on my part.  Sorry.

[Sérgio Basto]

On Sáb, 2014-07-19 at 12:30 +0100, Richard W.M. Jones wrote: 
>  The first step of most Koji builds is buildSRPMFromSCM, where a
>  .src.rpm file is built from the git repo.
>  
>  Currently this involves completely building a mock buildroot
>  containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  takes many minutes (especially when arm is chosen as a builder). 
If I understood the question, I use 
fedpkg switch-branch f20 , fedpkg switch-branch master etc and fedpkg
srpm .
>  It seems the reason for this is because the spec file has to be
> fully
>  parsed in order to work out the Source lines.  Since Source lines
>  might depend on RPM macros which might depend on any BuildRequire'd
>  package, every BR package must be installed in the mock root.
>  `rpmbuild -bs' takes seconds because it just bundles all the Source
>  files with the spec file into an SRPM.
>  
>  Is this really necessary? 
Just if we have "Source lines might depend on RPM macros which might
depend on any BuildRequire'd" :) 
have you any example , Source lines ? Source lines should be immutable 
I remember put a ifdef in a source line for i386 and src.rpm wasn't
equal for i386 and x86_64, but should, because we only export one of
them. 


>  Two shortcuts seem possible:
>  
>  (1) Limit the use of macros in Source lines, so that only a simple,
>  standard, perhaps pre-cached buildroot can be used.
>  
>  (2) Perhaps uglier: Just build an SRPM that contains everything in
>  dist-git + everything in the lookaside cache, and hope for the best ...
>  
>  Thoughts?
>  
>  Rich.
>  
>  -- 
>  Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
>  Read my programming and virtualization blog: http://rwmj.wordpress.com
>  Fedora Windows cross-compiler. Compile Windows programs, test, and
>  build Windows installers. Over 100 libraries supported.
>  http://fedoraproject.org/wiki/MinGW 
-- 
Sérgio M. B.

[Richard W.M. Jones]

On Sat, Jul 19, 2014 at 03:37:15PM +0100, Sérgio Basto wrote:
>  
>  
>  On Sáb, 2014-07-19 at 12:30 +0100, Richard W.M. Jones wrote: 
>  > The first step of most Koji builds is buildSRPMFromSCM, where a
>  > .src.rpm file is built from the git repo.
>  > 
>  > Currently this involves completely building a mock buildroot
>  > containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  > takes many minutes (especially when arm is chosen as a builder).
>  
>  If I understood the question, I use 
>  fedpkg switch-branch f20 , fedpkg switch-branch master etc and fedpkg
>  srpm . 
The question is related to what Koji does, not what local developers
do with 'fedpkg'.

>  Just if we have "Source lines might depend on RPM macros which
> might
>  depend on any BuildRequire'd" :) 
>  have you any example , Source lines ? Source lines should be immutable 
>  I remember put a ifdef in a source line for i386 and src.rpm wasn't
>  equal for i386 and x86_64, but should, because we only export one of
>  them.  
Well something like (made up example):

  Source0: %{lang_download_url}/%{name}-%{version}.tar.gz

where %lang_download_url is an RPM macro defined in some
lang-specific package that's not part of a normal build root.

I have no idea if there are packages which do this, which is kind of
my point.  We spend probably an extra 10 minutes on each build on a
pretty obscure corner case.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Richard W.M. Jones]

On Sat, Jul 19, 2014 at 04:38:22PM +0100, Richard W.M. Jones wrote:
>  Well something like (made up example):
>  
>    Source0: %{lang_download_url}/%{name}-%{version}.tar.gz
>  
>  where %lang_download_url is an RPM macro defined in some
>  lang-specific package that's not part of a normal build root.
>  
>  I have no idea if there are packages which do this, which is kind of
>  my point.  We spend probably an extra 10 minutes on each build on a
>  pretty obscure corner case. 
This is wrong actually -- see my other reply.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[drago01]

On Sat, Jul 19, 2014 at 1:30 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  The first step of most Koji builds is buildSRPMFromSCM, where a
>  .src.rpm file is built from the git repo.
>
>  Currently this involves completely building a mock buildroot
>  containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  takes many minutes (especially when arm is chosen as a builder).
>
>  It seems the reason for this is because the spec file has to be fully
>  parsed in order to work out the Source lines.  Since Source lines
>  might depend on RPM macros which might depend on any BuildRequire'd
>  package,  [...] 
For most packages this is not the case (i.e you do not need BR to
build the srpm) ... maybe we should have some way to flag a package as
such?
Maybe just by adding a file with a specific name to the repo?

[Eric Smith]

On Sat, Jul 19, 2014 at 5:30 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  (1) Limit the use of macros in Source lines, so that only a simple,
>  standard, perhaps pre-cached buildroot can be used. 
How about just trying to parse the Source lines, and if that fails due
to use of macros that can't be evaluated in a simpleminded way, revert
to the method used now?

[Mattias Ellert]

lör 2014-07-19 klockan 12:30 +0100 skrev Richard W.M. Jones:
>  The first step of most Koji builds is buildSRPMFromSCM, where a
>  .src.rpm file is built from the git repo.
>  
>  Currently this involves completely building a mock buildroot
>  containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  takes many minutes (especially when arm is chosen as a builder). 
The buildroot for the buildSRPMFromSCM step is not even the full default
buildroot for the package build step and it doesn't install the BRs.

On my recent koji builds 'group install "srpm-build"' results in
"Install 10 Packages (+226 Dependent packages)" and no additional
packages are installed during the buildSRPMFromSCM step.

For the build step the 'group install "build"' results in "Install
24
Packages (+141 Dependent packages)" after which the BRs and their
dependencies are installed.

(These numbers are from rawhide.)

>  It seems the reason for this is because the spec file has to be
> fully
>  parsed in order to work out the Source lines.  Since Source lines
>  might depend on RPM macros which might depend on any BuildRequire'd
>  package, every BR package must be installed in the mock root.
>  `rpmbuild -bs' takes seconds because it just bundles all the Source
>  files with the spec file into an SRPM. 
If there are such lines they will break since the BRs are not installed
during the buildSRPMFromSCM step.

>  Is this really necessary?
>  
>  Two shortcuts seem possible:
>  
>  (1) Limit the use of macros in Source lines, so that only a simple,
>  standard, perhaps pre-cached buildroot can be used.
>  
>  (2) Perhaps uglier: Just build an SRPM that contains everything in
>  dist-git + everything in the lookaside cache, and hope for the best ...
>  
>  Thoughts?
>  
>  Rich. 
	Mattias

[Richard W.M. Jones]

On Sun, Jul 20, 2014 at 07:10:40AM +0200, Mattias Ellert wrote:
>  lör 2014-07-19 klockan 12:30 +0100 skrev Richard W.M. Jones:
>  > The first step of most Koji builds is buildSRPMFromSCM, where a
>  > .src.rpm file is built from the git repo.
>  > 
>  > Currently this involves completely building a mock buildroot
>  > containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  > takes many minutes (especially when arm is chosen as a builder).
>  
>  The buildroot for the buildSRPMFromSCM step is not even the full default
>  buildroot for the package build step and it doesn't install the BRs.
>  
>  On my recent koji builds 'group install "srpm-build"' results in
>  "Install 10 Packages (+226 Dependent packages)" and no additional
>  packages are installed during the buildSRPMFromSCM step.
>  
>  For the build step the 'group install "build"' results in "Install
> 24
>  Packages (+141 Dependent packages)" after which the BRs and their
>  dependencies are installed.
>  
>  (These numbers are from rawhide.) 
So indeed this is true.

However it still takes 6 minutes to perform this step on the ARM
builders, which for some reason are usually chosen for
buildSRPMFromSCM.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Ville Skyttä]

On Sun, Jul 20, 2014 at 1:16 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  However it still takes 6 minutes to perform this step on the ARM
>  builders, which for some reason are usually chosen for
>  buildSRPMFromSCM. 
ISTR when ARM wasn't yet in the mix but PPC was, it was the PPC
builders that did buildSRPMFromSCMs. And the PPC builders were the
slowest back then too. Maybe koji chooses a builder based on
alphabetical sorting of the names of available ones, dunno. If that's
so, a quick "fix" would be to use reverse sort instead :)

[Adam Jackson]

On Sat, 2014-07-19 at 12:30 +0100, Richard W.M. Jones wrote:
>  The first step of most Koji builds is buildSRPMFromSCM, where a
>  .src.rpm file is built from the git repo.
>  
>  Currently this involves completely building a mock buildroot
>  containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  takes many minutes (especially when arm is chosen as a builder). 
We really ought to just set up a channel populated only by actually
performant hardware that only accepts buildSRPMfromSCM jobs.

And then also fix https://fedorahosted.org/koji/ticket/276 so that the
job's output is deterministically good for all architectures, thus
avoiding the "but you should make sure it works if you do build the srpm
on arm" argument.

- ajax

[Dennis Gilmore]

On Sat, 19 Jul 2014 12:30:38 +0100
"Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote:

>  
>  The first step of most Koji builds is buildSRPMFromSCM, where a
>  .src.rpm file is built from the git repo.
>  
>  Currently this involves completely building a mock buildroot
>  containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  takes many minutes (especially when arm is chosen as a builder).
>  
>  It seems the reason for this is because the spec file has to be fully
>  parsed in order to work out the Source lines.  Since Source lines
>  might depend on RPM macros which might depend on any BuildRequire'd
>  package, every BR package must be installed in the mock root.
>  `rpmbuild -bs' takes seconds because it just bundles all the Source
>  files with the spec file into an SRPM.
>  
>  Is this really necessary? 
This is not at all true. we have a minimal buildroot that
is installed
for every single build, the buildroot packages are the same across all
builds. there is no BuildRequires pulled in until the buildArch task


>  Two shortcuts seem possible:
>  
>  (1) Limit the use of macros in Source lines, so that only a simple,
>  standard, perhaps pre-cached buildroot can be used.
>  
>  (2) Perhaps uglier: Just build an SRPM that contains everything in
>  dist-git + everything in the lookaside cache, and hope for the
>  best ... 
This is what we do now.

Dennis

[Dennis Gilmore]

On Mon, 21 Jul 2014 13:58:49 -0500
Dennis Gilmore <dennis(a)ausil.us&gt; wrote:

>  On Sat, 19 Jul 2014 12:30:38 +0100
>  "Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote:
>  
>  > 
>  > The first step of most Koji builds is buildSRPMFromSCM, where a
>  > .src.rpm file is built from the git repo.
>  > 
>  > Currently this involves completely building a mock buildroot
>  > containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  > takes many minutes (especially when arm is chosen as a builder).
>  > 
>  > It seems the reason for this is because the spec file has to be
>  > fully parsed in order to work out the Source lines.  Since Source
>  > lines might depend on RPM macros which might depend on any
>  > BuildRequire'd package, every BR package must be installed in the
>  > mock root. `rpmbuild -bs' takes seconds because it just bundles all
>  > the Source files with the spec file into an SRPM.
>  > 
>  > Is this really necessary?
>  This is not at all true. we have a minimal buildroot that is installed
>  for every single build, the buildroot packages are the same across all
>  builds. there is no BuildRequires pulled in until the buildArch task
>  
>  
>  > Two shortcuts seem possible:
>  > 
>  > (1) Limit the use of macros in Source lines, so that only a simple,
>  > standard, perhaps pre-cached buildroot can be used.
>  > 
>  > (2) Perhaps uglier: Just build an SRPM that contains everything in
>  > dist-git + everything in the lookaside cache, and hope for the
>  > best ...
>  This is what we do now.
>  
>  Dennis 
we could make it faster by switching to a fedpkg wrapper that was just
a shell script to get the sources. fedpkg has a lot of deps, we would
make what needs installed smaller.

Dennis

[Richard W.M. Jones]

On Mon, Jul 21, 2014 at 01:58:49PM -0500, Dennis Gilmore wrote:
>  On Sat, 19 Jul 2014 12:30:38 +0100
>  "Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote:
>  
>  > 
>  > The first step of most Koji builds is buildSRPMFromSCM, where a
>  > .src.rpm file is built from the git repo.
>  > 
>  > Currently this involves completely building a mock buildroot
>  > containing all the BuildRequires, and running `rpmbuild -bs'.  This
>  > takes many minutes (especially when arm is chosen as a builder).
>  > 
>  > It seems the reason for this is because the spec file has to be fully
>  > parsed in order to work out the Source lines.  Since Source lines
>  > might depend on RPM macros which might depend on any BuildRequire'd
>  > package, every BR package must be installed in the mock root.
>  > `rpmbuild -bs' takes seconds because it just bundles all the Source
>  > files with the spec file into an SRPM.
>  > 
>  > Is this really necessary?
>  This is not at all true. we have a minimal buildroot that is installed
>  for every single build, the buildroot packages are the same across all
>  builds. there is no BuildRequires pulled in until the buildArch task 
Right, see my other reply.  However it still takes 6 minutes to build
the chroot.

>  > Two shortcuts seem possible:
>  > 
>  > (1) Limit the use of macros in Source lines, so that only a simple,
>  > standard, perhaps pre-cached buildroot can be used.
>  > 
>  > (2) Perhaps uglier: Just build an SRPM that contains everything in
>  > dist-git + everything in the lookaside cache, and hope for the
>  > best ...
>
>  This is what we do now. 
My suggestion with (2) is that we shortcut building the .src.rpm by
taking everything in dist-git + the files mentioned in sources (from
the lookaside cache), then build something that looks like a .src.rpm.
I doubt you can actually do this using current rpm tools.  This should
be very quick - seconds not minutes.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v