[Modularity] Module build proposal

[Petr Šabata]

Hi all,

as promised on the last Modularity WG meeting [0], here's
a module build proposal we've been pondering for a while.

As always, this is nowhere near final and there are still many
open questions.  The purpose of this mail is to get some early
feedback.  Feel free to comment, especially if you see anything
obviously wrong or have ideas how to make the whole thing better.

Note the last module metadata proposal is a little outdated and
will have to be adapted to make this possible.

Key points:
- building a module means building the components it contains
  and composing a module deliverable (a repository, an image, other)
- the build must be reproducible
- the module's buildroot is defined by "buildrequiring" other,
  already existing modules
- modules should be tracked in a VCS, such as dist-git [1]

Build proposal, step by step:
- we have a module, let's name it foo
- the module definition says it's composed of RPM packages
  bar and baz
- the module definition also says where to find the sources
  for these packages, e.g. a link to a git repository, the
  commit hash and a link to the lookaside cache [2]
- the build service clones bar's and baz's repositories and
  checks out the specified commits [3]
- the build service prepares the buildroot using the modules
  listed as its build dependencies
- the build service builds bar and baz in the prepared buildroot;
  some modules might require buildroot overrides and build their
  packages in a particular order
- for whatever reason, we might not always want all the built
  packages in our module; we can filter some out at this point
- the service optionally includes (well, bundles) dependencies
  present in the buildroot but not present in the module's runtime
  environment in the currently built module
- the service gathers any extra metadata it needs and records
  them in the module -- this could include automatically generated
  lists of component licenses, builder IDs, vendor tags, timestamps,
  module life cycle data and more
- the build service composes the deliverable(s)

Challenges:
- we need to ensure the module sources will be available for the
  entire lifetime of the module (e.g. SPEC files, tarballs)
- plenty more, point them out

Most of this can be done with the tools we already have or
with minimal changes to them.  This proposal doesn't address
parallel installations of conflicting modules but that's a
somewhat different topic and may be solved on a different level.

Again, share your comments and feel free to ask any questions.

Cheers,
P

[0]
https://meetbot.fedoraproject.org/teams/modularity-wg/modularity-wg.2016-...
[1] Working on that; our staging dist-git and pkgdb already
    include support for this.
[2] Obviously some of these couldn't be set by the module author
    when building official Fedora modules; our build service should
    always use our dist-git and our cache.  It is still useful for
    development, testing and third-party module vendors.
[3] We've considered using git tags but it's more trouble than it's
    worth.