A smart cache-y proposal for composes

[Kevin Fenzi]

So, I had an idea of one method of attack for composes and lifecycles
that I thought I would toss out there to see what others thought of it.

Instead of redesigning and reimplementing our current build/compose/test
pipeline we improve it by making the current process cache as much of
everything as it possibly can and listening and operating on any changes
in inputs. So, instead of gathering everything all the time from all the
inputs every time we only do 'full' composes every so often (weekly?)
and then less and less often as our caching is fixed.

At each place we currently gather or query an input, we instead query it
and check it against what we already have. If they are the same, go on,
if they are different, cache it and change only those things that are
affected by that input.

Some examples:

a new xfce4-session package is built. The composer sees this and goes
through all things affected by 'new package':

* Does it pass CI?
* remove old package from package caches/repos and update them.
* is it in a deliverable? yes, a Xfce live and a xfce arm image
* make those (just those) and test
* If everything passes, release those deliverables (a updated repo where
you just have the one package updated and new repodata, some images and
checksum files)

A new kernel is built:

* Does it pass CI?
* remove old package from package caches/repos and update them.
* is it in a deliverable? yes, almost all of them.
* build and test each deliverable in order of importance.
* If they all pass, release those deliverables (new repos, new images,
new checksums, etc).

This of course really only works for rawhide, and in practice (at least
at first) we would swamp it pretty easily (ie, kernel and coreutils and
dnf update nearly at the same time could result in tons of activity),
but it could be nice down the road as it would mean we are always ready
for a release anytime we choose to do so.

There will of course be cases of caches not updating, or updating, but
not building all the affected outputs. It would also require a lot of
mapping of input to outputs, but almost any approach we have will need
that.

Thoughts?
or better yet: Other concrete ideas how to approach this re-tooling?

kevin

[Randy Barlow]

"NOOOOO" -adamw

[Adam Williamson]

On Thu, 2018-11-29 at 16:01 -0500, Randy Barlow wrote:
>  "NOOOOO" -adamw 
=)

More seriously, to me this is kinda a natural consequence of what we've
already talked about quite a bit with associating inputs and outputs.
But I dunno if I'd think of it as *caching*, exactly, just more along
the lines of 'only generate outputs when the relevant inputs change'.
But, I mean, it's fine to think of it as caching too, I guess, it just
hadn't occurred to me to look at it that way.

Like all caching, it's fine so long as the implementation is correct,
and a giant PITA when it isn't :P
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net