Remove gcc, gcc-c++ and make from minimal build root

[Dan Horák]

sorry for top-post, but one more thing to consider, if caching would be
used, are secondary arches - how it would affect their ability to build
the buildroots


		Dan

On Sat, 17 Jan 2015 11:07:53 -0600
Dennis Gilmore <dennis at ausil.us> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Sat, 17 Jan 2015 03:44:10 +0000
> Peter Robinson <pbrobinson at gmail.com> wrote:
> 
> > >> >>> > > that all being said. koji doesn't use any caching and
> > >> >>> > > will not use the lvm plugin. we make every buildroot from
> > >> >>> > > scratch using a fully clean environment to help with
> > >> >>> > > ensuring reproducability.
> > >> >> >
> > >> >> > You can cache and still preserve reproducability. What I'm
> > >> >> > planning for Copr is to do (every week/month) for chroot in
> > >> >> > fedora-20-x86_64 fedora-21_86_64 ... ; do mock --init
> > >> >> > $chroot done
> > >> >> > take snapshot of that. I plan to do that on VM level.
> > >> >> > And when new task come, I will just restore from that
> > >> >> > snapshot. And mock will start with already populated cache.
> > >> >> > So I will have better caching and yet reproducability.
> > >> > you really can't.  you would need to make a new cache any time
> > >> > one of the packages in the minimal buildroot changes.
> > >>
> > >> That is why mock run 'yum update' on minimal buildroot before
> > >> proceeding further. So if one package in buildroot changes, you
> > >> will just download and install just that one package and all
> > >> other comes from cache.
> > >
> > > That invalidates being able to reproduce the build root exactly.
> > > as you would need to know and reproduce the package update. This
> > > is something that has undergone a lot of thought and
> > > consideration and there is very very good reasons things are the
> > > way they are. as i showed elsewhere in this thread most repos are
> > > used only once or twice at most on any given host.
> > 
> > The only way you could cache a base build root effectively is if you
> > track the packages and NVRs and when one of them bump you regenerate
> > the buildroot, likely pausing all builds until a new one is there.
> > there's likely not that much churn in the packages contained in the
> > root buildroot (you'd want to make sure you had everything for
> > building .src.rpms too) and in some cases you could likely get days
> > on the same image without having to rebuild it, other days you
> > might have to do it a few times a time.
> 
> As I showed earlier in the thread most repos are only ever used once
> or twice on a host. but we do not know if something in the minimal
> buildroot has changed. there is a lot of added complexity needed to
> attempt to use caching. which adds many more chances for something to
> go wrong. I honestly do not know if its quicker to build the minimal
> buildroot from scratch or to unpack the tarball. there is costs both
> ways. with the added complexity in tracking things to try and make
> caching worthwhile.  in the Fedora case we would need to have
> somewhere in the order of 6-12 cached buildroots on each builder
> depending on sidetags etc in use at different times.  but there is
> instances of koji where that number would be in the hundreds. we need
> to make sure that it works and scales not only for our use cases but
> also other users of koji.
> 
> > Whether the effort to implement out weights the effort remains to be
> > seen. I think, given the low IOPs on the builds with an underlying
> > COP image it might well give a reasonable reduction in building
> > especially during mass rebuilds where the builders are very active
> > and as it's in a side tag the new build root packages don't end up
> > in the build root until tagged. Using fedmsg it's likely not hard
> > either to monitor a package subset for changes and regen the image.
> 
> There is a massive effort to figure out if the effort to implement any
> caching is worth while. If someone wanted to take the time to figure
> that out, making sure to gather the needs of different koji users to
> make sure that we did not make things worse for them, then by all
> means that person is welcome to do so and I will gladly answer any
> and all questions that they had. However I have no interest in going
> down that rabbit hole. we can not rely on koji users using fedmsg so
> that is something that would likely need to be ruled out of the design
> decision. fedmsg also does not guarantee delivery of messages and so
> can not be relied on for anything where we have to be 100% sure that
> things are done given a trigger.
> 
> Dennis
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
> 
> iQIcBAEBAgAGBQJUupbqAAoJEH7ltONmPFDR+iQP/ietNky5y2yhtbUZ/MMGP1er
> PDMyHz5MD9v78mstXXjx6WsBrOjc2a51WCufBV2drlPuwyTrstwJShbOhm1XfOld
> 1HSI9etIkwsVxBFgRfh6HUxbLGZWSHDZviY34KocxLNuuaRLqc5RHjTTZmzQwKl+
> 4EA4o6HJAYPsVz0Fu2/sJAy9cCclfCPy8Mcpstd1NIHOPhtNTpFiEQBl4zpXRqQs
> nlmlNh1MhZepv84k+EvhRZCJlmfn0yLBLlNJfNh6jtdy6zv+lNf2oClrHDR9o5aL
> nRCEJhRl/LX+BNf2mV5MULxAliXMME1qoYrm//BgsQk45t3XvVJ2fpPEVs8gcr49
> gZ0yoJ20fcSW/UJCRMhoI9dtp1MYQmF3h3LHfNRROv+Ga9Uumw9OmQyFqFJwbF2o
> doLOvDxtpEXWsiXcTUSqcl3CBGyT50AiAbRvv5yrQ5TU1Z8TplcUVUbJ/zifBoid
> GexFUETsN3E1P0byvfbEiJJgaAYkWTaho2SwO+gRY/iieHlWyEbK6BwAiw7BsJOM
> Tm5P6EjHyqk1kPiycoG9Zz70iiBhrFraMgzWwtrjNid2n8vvV6pdEpXpiaDzOAgS
> SdVpHH2R8tu8GjnbONazSt6GpG7oEqnS5Nz2YZT6MMTcNuCvxY5bVoRfAl4DlDTM
> X/98RH2w7xXM/fejAZN3
> =Vjxf
> -----END PGP SIGNATURE-----
> -- 
> devel mailing list
> devel at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct