MBI (Playground 2.0)

[Igor Gnatenko]

MayBe I …(can do something useful)?

Hello,

We've been discussing some (hopefully) nice idea with Mikolaj, Neal and
Jakub how we could improve packager (and user) experience and we have some
proposal which will be described below.

We would like to ask you to read it, understand it and ask us any questions
you have. From the Fedora Infrastructure we would like to ask for some
machines to implement this idea (you can find some more information in
"Implementation details" part).

I would like to apologize for HTML email, but it is much easier to have it
that way to have better visibility and reading easiness.

Feel free to reply to this email or comment in google doc (there is a link
on the bottom).
Proposal Owners

   -

   Mikolaj Izdebski (mizdebsk) <mizdebsk(a)redhat.com&gt; - Java SIG, Fedora
   infrastructure
   -

   Igor Gnatenko (ignatenkobrain) <ignatenkobrain(a)fedoraproject.org&gt; - Rust
   SIG, Golang SIG, Neuro SIG, RPM and libsolv contributor
   -

   Neal Gompa (ngompa) <ngompa13(a)gmail.com&gt; - Rust SIG, Golang SIG, RPM
   contributor
   -

   Jakub Čajka (jcajka) <jcajka(a)fedoraproject.org&gt; - Golang SIG, Container
   SIG


This proposal aligns with the objective of improving the packager experience
<https://fedoraproject.org/wiki/Objectives/Packager_Experience> by
developing a platform whereby the proposal owners and others can experiment
with improvements that can be funneled back into the official production
infrastructure.
ProblemsProblem №1: Build-only packages

Some ecosystems have many build-only packages (packages which are used to
build other packages, without having them installed on end-user systems).
Those ecosystems include Java, Rust and Go.

It is extremely hard to keep up maintaining them due to lack of
time/people. Upstreams are usually changing quickly (sometimes when you
update just one such package, you’d need to update tens of the
dependencies). Few facts about such packages:

   -

   They are almost always outdated in released versions of distribution;
   -

   They are often FTBFS (e.g. because there was compiler update but not
   package update, broken deps, broken APIs due to deps rebases, …).


In Rust ecosystem, we abuse Rawhide to build and store such packages there
and then generate end-user application (e.g. ripgrep) which uses some of
those packages and produces binary for all supported releases. Those
applications have high quality and are supported.

Rawhide gating makes this much more complicated because builds appear in
buildroot slower, updating group of packages would need side tags and it’s
just painful to work with.

https://pagure.io/fesco/issue/2068

And, after all, those packages shouldn’t be shipped to users.
Problem №2: Testing of new rpm/koji/mock features/configuration

When developing new features in RPM (e.g. rich dependencies) or testing
different Koji configuration (e.g. removing gcc/gcc-c++ from the
buildroot), it is required to make custom configuration and try building
things.
Problem №3: Developing modules

Modules are built in MBS as a single unit. It is hard to develop big
modules by progressively improving individual packages or small package
groups. Scratch builds for modules are not implemented. Local builds work
differently from official module builds, they don’t scale and don’t allow
groups of people to work collaboratively. All these problems can be solved
by first developing a flat repository of packages in a shared environment
and then generating modulemd from such package set.
Problem №4: Testing things before push

Primary Fedora Koji and dist-git are not suited for packaging
experimentation. Packagers are expected to experiment on their own systems
and push changes of successful experiments only. But this approach doesn’t
scale with number of maintained packages. Often you can find commits like
“d’oh, forgot to upload sources” or “let’s try with this settings”. People
need to build things somewhere quickly, do development and testing. And
only after that, they should push commit(s) to Fedora.
Solution

   -

   Separate Koji + Koschei deployed in Fedora infrastructure cloud;
   -

   Builders are optimized for the best performance (see below
  
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
   );
   -

   People can have their own targets where they can break things as they
   wish without affecting others;
   -

   Package changes are eventually contributed to Fedora proper by merging
   changes to Fedora dist-git and rebuilding packages in official Fedora Koji;
   -

   All improvements can eventually be contributed back to official Fedora
   infra.

Ideas

All ideas which you’ll find below are just an ideas and do not have to be
implemented.
Idea №1: Automated legal review

openSUSE released their review app called Cavil
<https://github.com/openSUSE/cavil> which we could try running in automated
way.
Idea №2: Automated package review

Currently review process is burden and we could run automated legal review,
create a repo, run set of checks and notify person. Somewhat similar to
fresque <https://github.com/fedora-infra/fresque>;. In future might be
integrated with approval process and auto-imported into Fedora.
Idea №3: Building packages for multiple distributions

In Rust ecosystem, we managed to get have same packaging guidelines in
openSUSE, Mageia and Fedora. We could automatically build some packages on
multiple distributions and be kinda upstream.
Idea №4: Building custom images with user content

Deploy (or build) a tool(s) to enable user-built install, appliance and
container images with their content (modulo restrictions similar to COPR)
on top of Fedora.
Implementation details

   -

   Koji and Koschei deployed in fedorainfracloud.org
   -

   A few builders constantly running, with a possibility to add more
   builders as needed and as available cloud resources allow
   -

   Deployed and maintained by proposal owners - not supported by Fedora
   infrastructure
   -

   Other contributors can have access granted by proposal owners (for the
   time being only users in “packagers” group will be eligible to get access)
   -

   Possibility to have some builders running outsides of Fedora
   infrastructure - contributed by proposal owners
   -

   Koji builds can be done from SCM (src.fp.o, pagure.io, github, …) or
   from SRPM upload

FAQWhy not COPR?

   -

   COPR has been starved of resources for years, which has impaired its
   ability to provide reliable service at scale. Fedora Infrastructure refuses
   to consider supporting it officially and Fedora Release Engineering seems
   to have some undefined issues with COPR.
   -

   It is not official build system of Fedora which is not helping with Problem
   №2: Testing of new rpm/koji/mock features/configuration
  
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
   .
   -

   COPR is not extensible enough, more specifically:
   -

      No query API (e.g. it is not possible to find out from which SCM
      commit the package has been built)
      -

      Builds always have access to all previous builds in a repository
      (i.e. not possible to control when/how repo is created)
      -

      GC doesn’t exist
      -

      No scratch builds

Why not staging Koji?

   -

   Staging Koji is meant for testing Koji itself and not for package
   development.
   -

   Staging Koji is often in broken state where it is not possible to build
   anything.
   -

   No one can touch staging Koji, so it’s pointless for offering a useful
   service.

How can you make Koji builds faster?

   -

   By tuning Koji for performance, not correctness and reliability.
   -

   By building only on a single, fast architecture.
   -

   By extensive caching. Files like RPM packages, repodata and files in
   lookaside cache don’t change after they are initially stored, so builders
   can cache them aggressively.
   -

   By keeping build repositories small. Some package sets don’t need to be
   built against full Fedora, but can use a minimal subset of Fedora. Such
   repositories can be generated by Koji in seconds, not minutes.

Why not the Open Build Service (OBS)?

   -

   OBS is not yet packaged for Fedora officially. The upstream code lacks
   adaptations necessary to deploy and run on Fedora and in Fedora
   Infrastructure.
   -

   OBS is not official build system of Fedora, which does not help with Problem
   №2: Testing of new rpm/koji/mock features/configuration
  
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
   .

What does MBI stand for?

   -

   M for middlestream, module, mizdebsk, …
   -

   B for build, bootstrap, …
   -

   I for infrastructure, initiative, ignatenkobrain, …
   -

   MBI might be pronounced as “maybe I …(can make something cool)?”
   -

   MBI is also initials of mizdebsk (Mikolaj Bozydar Izdebski)
   -

   MBI is also IBM spelled backwards :)

Is it somehow related to Fedora Playground
<https://fedoraproject.org/wiki/Playground>?

Yes, it is. Although it is more developer-focused. Users could enrol for
some specific things like experimental Java/Rust packages.


 MBI (Playground 2.0)
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...