MBI (Playground 2.0)

[Igor Gnatenko]

MayBe I …(can do something useful)?

Hello,

We've been discussing some (hopefully) nice idea with Mikolaj, Neal and
Jakub how we could improve packager (and user) experience and we have some
proposal which will be described below.

We would like to ask you to read it, understand it and ask us any questions
you have. From the Fedora Infrastructure we would like to ask for some
machines to implement this idea (you can find some more information in
"Implementation details" part).

I would like to apologize for HTML email, but it is much easier to have it
that way to have better visibility and reading easiness.

Feel free to reply to this email or comment in google doc (there is a link
on the bottom).
Proposal Owners

   -

   Mikolaj Izdebski (mizdebsk) <mizdebsk(a)redhat.com&gt; - Java SIG, Fedora
   infrastructure
   -

   Igor Gnatenko (ignatenkobrain) <ignatenkobrain(a)fedoraproject.org&gt; - Rust
   SIG, Golang SIG, Neuro SIG, RPM and libsolv contributor
   -

   Neal Gompa (ngompa) <ngompa13(a)gmail.com&gt; - Rust SIG, Golang SIG, RPM
   contributor
   -

   Jakub Čajka (jcajka) <jcajka(a)fedoraproject.org&gt; - Golang SIG, Container
   SIG


This proposal aligns with the objective of improving the packager experience
<https://fedoraproject.org/wiki/Objectives/Packager_Experience> by
developing a platform whereby the proposal owners and others can experiment
with improvements that can be funneled back into the official production
infrastructure.
ProblemsProblem №1: Build-only packages

Some ecosystems have many build-only packages (packages which are used to
build other packages, without having them installed on end-user systems).
Those ecosystems include Java, Rust and Go.

It is extremely hard to keep up maintaining them due to lack of
time/people. Upstreams are usually changing quickly (sometimes when you
update just one such package, you’d need to update tens of the
dependencies). Few facts about such packages:

   -

   They are almost always outdated in released versions of distribution;
   -

   They are often FTBFS (e.g. because there was compiler update but not
   package update, broken deps, broken APIs due to deps rebases, …).


In Rust ecosystem, we abuse Rawhide to build and store such packages there
and then generate end-user application (e.g. ripgrep) which uses some of
those packages and produces binary for all supported releases. Those
applications have high quality and are supported.

Rawhide gating makes this much more complicated because builds appear in
buildroot slower, updating group of packages would need side tags and it’s
just painful to work with.

https://pagure.io/fesco/issue/2068

And, after all, those packages shouldn’t be shipped to users.
Problem №2: Testing of new rpm/koji/mock features/configuration

When developing new features in RPM (e.g. rich dependencies) or testing
different Koji configuration (e.g. removing gcc/gcc-c++ from the
buildroot), it is required to make custom configuration and try building
things.
Problem №3: Developing modules

Modules are built in MBS as a single unit. It is hard to develop big
modules by progressively improving individual packages or small package
groups. Scratch builds for modules are not implemented. Local builds work
differently from official module builds, they don’t scale and don’t allow
groups of people to work collaboratively. All these problems can be solved
by first developing a flat repository of packages in a shared environment
and then generating modulemd from such package set.
Problem №4: Testing things before push

Primary Fedora Koji and dist-git are not suited for packaging
experimentation. Packagers are expected to experiment on their own systems
and push changes of successful experiments only. But this approach doesn’t
scale with number of maintained packages. Often you can find commits like
“d’oh, forgot to upload sources” or “let’s try with this settings”. People
need to build things somewhere quickly, do development and testing. And
only after that, they should push commit(s) to Fedora.
Solution

   -

   Separate Koji + Koschei deployed in Fedora infrastructure cloud;
   -

   Builders are optimized for the best performance (see below
  
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
   );
   -

   People can have their own targets where they can break things as they
   wish without affecting others;
   -

   Package changes are eventually contributed to Fedora proper by merging
   changes to Fedora dist-git and rebuilding packages in official Fedora Koji;
   -

   All improvements can eventually be contributed back to official Fedora
   infra.

Ideas

All ideas which you’ll find below are just an ideas and do not have to be
implemented.
Idea №1: Automated legal review

openSUSE released their review app called Cavil
<https://github.com/openSUSE/cavil> which we could try running in automated
way.
Idea №2: Automated package review

Currently review process is burden and we could run automated legal review,
create a repo, run set of checks and notify person. Somewhat similar to
fresque <https://github.com/fedora-infra/fresque>;. In future might be
integrated with approval process and auto-imported into Fedora.
Idea №3: Building packages for multiple distributions

In Rust ecosystem, we managed to get have same packaging guidelines in
openSUSE, Mageia and Fedora. We could automatically build some packages on
multiple distributions and be kinda upstream.
Idea №4: Building custom images with user content

Deploy (or build) a tool(s) to enable user-built install, appliance and
container images with their content (modulo restrictions similar to COPR)
on top of Fedora.
Implementation details

   -

   Koji and Koschei deployed in fedorainfracloud.org
   -

   A few builders constantly running, with a possibility to add more
   builders as needed and as available cloud resources allow
   -

   Deployed and maintained by proposal owners - not supported by Fedora
   infrastructure
   -

   Other contributors can have access granted by proposal owners (for the
   time being only users in “packagers” group will be eligible to get access)
   -

   Possibility to have some builders running outsides of Fedora
   infrastructure - contributed by proposal owners
   -

   Koji builds can be done from SCM (src.fp.o, pagure.io, github, …) or
   from SRPM upload

FAQWhy not COPR?

   -

   COPR has been starved of resources for years, which has impaired its
   ability to provide reliable service at scale. Fedora Infrastructure refuses
   to consider supporting it officially and Fedora Release Engineering seems
   to have some undefined issues with COPR.
   -

   It is not official build system of Fedora which is not helping with Problem
   №2: Testing of new rpm/koji/mock features/configuration
  
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
   .
   -

   COPR is not extensible enough, more specifically:
   -

      No query API (e.g. it is not possible to find out from which SCM
      commit the package has been built)
      -

      Builds always have access to all previous builds in a repository
      (i.e. not possible to control when/how repo is created)
      -

      GC doesn’t exist
      -

      No scratch builds

Why not staging Koji?

   -

   Staging Koji is meant for testing Koji itself and not for package
   development.
   -

   Staging Koji is often in broken state where it is not possible to build
   anything.
   -

   No one can touch staging Koji, so it’s pointless for offering a useful
   service.

How can you make Koji builds faster?

   -

   By tuning Koji for performance, not correctness and reliability.
   -

   By building only on a single, fast architecture.
   -

   By extensive caching. Files like RPM packages, repodata and files in
   lookaside cache don’t change after they are initially stored, so builders
   can cache them aggressively.
   -

   By keeping build repositories small. Some package sets don’t need to be
   built against full Fedora, but can use a minimal subset of Fedora. Such
   repositories can be generated by Koji in seconds, not minutes.

Why not the Open Build Service (OBS)?

   -

   OBS is not yet packaged for Fedora officially. The upstream code lacks
   adaptations necessary to deploy and run on Fedora and in Fedora
   Infrastructure.
   -

   OBS is not official build system of Fedora, which does not help with Problem
   №2: Testing of new rpm/koji/mock features/configuration
  
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
   .

What does MBI stand for?

   -

   M for middlestream, module, mizdebsk, …
   -

   B for build, bootstrap, …
   -

   I for infrastructure, initiative, ignatenkobrain, …
   -

   MBI might be pronounced as “maybe I …(can make something cool)?”
   -

   MBI is also initials of mizdebsk (Mikolaj Bozydar Izdebski)
   -

   MBI is also IBM spelled backwards :)

Is it somehow related to Fedora Playground
<https://fedoraproject.org/wiki/Playground>?

Yes, it is. Although it is more developer-focused. Users could enrol for
some specific things like experimental Java/Rust packages.


 MBI (Playground 2.0)
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...

[Josh Boyer]

On Thu, Jan 31, 2019 at 7:28 AM Igor Gnatenko
<ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  MayBe I …(can do something useful)?
>
>  Hello,
>
>  We've been discussing some (hopefully) nice idea with Mikolaj, Neal and Jakub how we
> could improve packager (and user) experience and we have some proposal which will be
> described below.
>
>  We would like to ask you to read it, understand it and ask us any questions you have.
> From the Fedora Infrastructure we would like to ask for some machines to implement this
> idea (you can find some more information in "Implementation details" part).
>
>  I would like to apologize for HTML email, but it is much easier to have it that way to
> have better visibility and reading easiness.
>
>  Feel free to reply to this email or comment in google doc (there is a link on the
> bottom).
>
>  Proposal Owners
>
>  Mikolaj Izdebski (mizdebsk) - Java SIG, Fedora infrastructure
>
>  Igor Gnatenko (ignatenkobrain) - Rust SIG, Golang SIG, Neuro SIG, RPM and libsolv
> contributor
>
>  Neal Gompa (ngompa) - Rust SIG, Golang SIG, RPM contributor
>
>  Jakub Čajka (jcajka) - Golang SIG, Container SIG
>
>
>  This proposal aligns with the objective of improving the packager experience by
> developing a platform whereby the proposal owners and others can experiment with
> improvements that can be funneled back into the official production infrastructure.
>
>  Problems
>
>  Problem №1: Build-only packages
>
>  Some ecosystems have many build-only packages (packages which are used to build other
> packages, without having them installed on end-user systems). Those ecosystems include
> Java, Rust and Go.
>
>
>  It is extremely hard to keep up maintaining them due to lack of time/people. Upstreams
> are usually changing quickly (sometimes when you update just one such package, you’d need
> to update tens of the dependencies). Few facts about such packages:
>
>  They are almost always outdated in released versions of distribution;
>
>  They are often FTBFS (e.g. because there was compiler update but not package update,
> broken deps, broken APIs due to deps rebases, …).
>
>
>  In Rust ecosystem, we abuse Rawhide to build and store such packages there and then
> generate end-user application (e.g. ripgrep) which uses some of those packages and
> produces binary for all supported releases. Those applications have high quality and are
> supported.
>
>  Rawhide gating makes this much more complicated because builds appear in buildroot
> slower, updating group of packages would need side tags and it’s just painful to work
> with.
>
>  https://pagure.io/fesco/issue/2068
>
>
>  And, after all, those packages shouldn’t be shipped to users.
>
>  Problem №2: Testing of new rpm/koji/mock features/configuration
>
>  When developing new features in RPM (e.g. rich dependencies) or testing different Koji
> configuration (e.g. removing gcc/gcc-c++ from the buildroot), it is required to make
> custom configuration and try building things.
>
>  Problem №3: Developing modules
>
>  Modules are built in MBS as a single unit. It is hard to develop big modules by
> progressively improving individual packages or small package groups. Scratch builds for
> modules are not implemented. Local builds work differently from official module builds,
> they don’t scale and don’t allow groups of people to work collaboratively. All these
> problems can be solved by first developing a flat repository of packages in a shared
> environment and then generating modulemd from such package set.
>
>  Problem №4: Testing things before push
>
>  Primary Fedora Koji and dist-git are not suited for packaging experimentation. Packagers
> are expected to experiment on their own systems and push changes of successful experiments
> only. But this approach doesn’t scale with number of maintained packages. Often you can
> find commits like “d’oh, forgot to upload sources” or “let’s try with this settings”.
> People need to build things somewhere quickly, do development and testing. And only after
> that, they should push commit(s) to Fedora.
>
>  Solution
>
>  Separate Koji + Koschei deployed in Fedora infrastructure cloud; 
Why does this need to be deployed in the fedora infrastructure cloud?
Seems like you could stand it up in AWS or somewhere else.

josh

[Mikolaj Izdebski]

On Thu, Jan 31, 2019 at 1:36 PM Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  Why does this need to be deployed in the fedora infrastructure
> cloud?
>  Seems like you could stand it up in AWS or somewhere else. 
Because we (Fedora contributors) don't have budget to pay AWS bills.
If someone is willing to sponsor this then AWS would work well.

--
Mikolaj

[Brian (bex) Exelbierd]

On Thu, Jan 31, 2019 at 1:40 PM Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  On Thu, Jan 31, 2019 at 1:36 PM Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  > Why does this need to be deployed in the fedora infrastructure cloud?
>  > Seems like you could stand it up in AWS or somewhere else.
>
>  Because we (Fedora contributors) don't have budget to pay AWS bills.
>  If someone is willing to sponsor this then AWS would work well. 
I encourage you to specify what you need and then the Project can
figure out if and how it wants to provide it.  So if you need a
specific kind of access, note that.  If you just need machines, note
that.  If you need sysadmin work, note that, etc.

regards,

bex

>  --
>  Mikolaj
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...


-- 
Brian (bex) Exelbierd | bexelbie(a)redhat.com | bex(a)pobox.com
Fedora Community Action & Impact Coordinator
@bexelbie | http://www.winglemeyer.org

[Ben Cotton]

Igor,

This is great. It seems like it would fit in really well with the Packager
Experience objective proposal[1] that Ben Rosser was working on. I know you
weighed in on that thread at one point. Is this a part of that proposed
objective or is it separate?

[1]
https://lists.fedoraproject.org/archives/list/[e-mail].o...

-- 
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis

[Stephen John Smoogen]

On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <
ignatenkobrain(a)fedoraproject.org&gt; wrote:

>  Solution
>
>     -
>
>     Separate Koji + Koschei deployed in Fedora infrastructure cloud;
>
>  
>     -
>
>     FAQ
>
>  Why not COPR?
>
>     -
>
>     COPR has been starved of resources for years, which has impaired its
>     ability to provide reliable service at scale. Fedora Infrastructure refuses
>     to consider supporting it officially and Fedora Release Engineering seems
>     to have some undefined issues with COPR.
>     -
>
>
>
>  
OK I would say that putting this into the very place where the other build
system is starved of resources does not sound like it is going to work any
better. You will need to price out what AWS is and find a budget payer for
it. There are several groups who should be able to do so if they can find a
real budget for it.

-- 
Stephen J Smoogen.

[Kevin Fenzi]

On 1/31/19 4:24 AM, Igor Gnatenko wrote:

>  ProblemsProblem №1: Build-only packages
>  
>  Some ecosystems have many build-only packages (packages which are used to
>  build other packages, without having them installed on end-user systems).
>  Those ecosystems include Java, Rust and Go.
>  
>  It is extremely hard to keep up maintaining them due to lack of
>  time/people. Upstreams are usually changing quickly (sometimes when you
>  update just one such package, you’d need to update tens of the
>  dependencies). Few facts about such packages:
>  
>     -
>  
>     They are almost always outdated in released versions of distribution;
>     -
>  
>     They are often FTBFS (e.g. because there was compiler update but not
>     package update, broken deps, broken APIs due to deps rebases, …).
>  
>  
>  In Rust ecosystem, we abuse Rawhide to build and store such packages there
>  and then generate end-user application (e.g. ripgrep) which uses some of
>  those packages and produces binary for all supported releases. Those
>  applications have high quality and are supported.
>  
>  Rawhide gating makes this much more complicated because builds appear in
>  buildroot slower, updating group of packages would need side tags and it’s
>  just painful to work with. 
Note that the rawhide gating plans include self service side tags, so
you can make one anytime you want to update a collection of packages.

This of course doesn't solve the above problem, but it's worth
mentioning I think.

>  https://pagure.io/fesco/issue/2068
>  
>  And, after all, those packages shouldn’t be shipped to users. 
This of course means however that users who want to build their own
verions of the applications or whatever can't start from our repos, they
have to use whatever setup we use for developing them. I don't know any
answers here but it seems to me we keep making it harder for people to
use Fedora to develop applications.

...snip...

>  Solution
>  
>     -
>  
>     Separate Koji + Koschei deployed in Fedora infrastructure cloud; 
Turns out we are going to be retiring our cloud, so no, this is not a
place to put it. :)

>     -
>  
>     Builders are optimized for the best performance (see below
>    
> <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
>     );
>     -
>  
>     People can have their own targets where they can break things as they
>     wish without affecting others; 
Would that be entirely self service? Or ?

>     -
>  
>     Package changes are eventually contributed to Fedora proper by merging
>     changes to Fedora dist-git and rebuilding packages in official Fedora Koji; 
Where is the source for the changes from? Would this need it's own
pagure/src/dist-git? Or would it pull from PR's in prod dist git? Or ?

>     -
>  
>     All improvements can eventually be contributed back to official Fedora
>     infra. 
Sounds great!

>  
>  Ideas
>  
>  All ideas which you’ll find below are just an ideas and do not have to be
>  implemented.
>  Idea №1: Automated legal review
>  
>  openSUSE released their review app called Cavil
>  <https://github.com/openSUSE/cavil> which we could try running in automated
>  way. 
I guess this would be up to legal to look at and tell us if it's worth
doing.

>  Idea №2: Automated package review
>  
>  Currently review process is burden and we could run automated legal review,
>  create a repo, run set of checks and notify person. Somewhat similar to
>  fresque <https://github.com/fedora-infra/fresque>;. In future might be
>  integrated with approval process and auto-imported into Fedora. 
Sounds great if someone has time to build the app(s)

>  Idea №3: Building packages for multiple distributions
>  
>  In Rust ecosystem, we managed to get have same packaging guidelines in
>  openSUSE, Mageia and Fedora. We could automatically build some packages on
>  multiple distributions and be kinda upstream. 
So, this setup will need to distribute things... would it need mirrors
as well? or ?

>  Idea №4: Building custom images with user content
>  
>  Deploy (or build) a tool(s) to enable user-built install, appliance and
>  container images with their content (modulo restrictions similar to COPR)
>  on top of Fedora. 
Yeah, this has been attempted about 3 times, but if someone has cycles
to work on a app/tool again, great!

>  Implementation details
>  
>     -
>  
>     Koji and Koschei deployed in fedorainfracloud.org 
Nope. There are options of course, but the cloud won't be one. ;)
We could do a remote cloud provider possibly, or we are considering
another openshift cluster with kubevirt or perhaps just some simple
virthost/ansible controlled thing. In any case I think if it's important
we can get resources.
>     -
>  
>     A few builders constantly running, with a possibility to add more
>     builders as needed and as available cloud resources allow 
koji doesn't have much way to dynamically deal with builders...

>     -
>  
>     Deployed and maintained by proposal owners - not supported by Fedora
>     infrastructure
>     -
>  
>     Other contributors can have access granted by proposal owners (for the
>     time being only users in “packagers” group will be eligible to get access)
>     -
>  
>     Possibility to have some builders running outsides of Fedora
>     infrastructure - contributed by proposal owners 
I'd be carefull of this, as it can cause a lot of issues, but of course
up to you as you are doing the work. :)

>     -
>  
>     Koji builds can be done from SCM (src.fp.o, pagure.io, github, …) or
>     from SRPM upload 
Ah, so it would not have it's own git/pagure?

>  
>  FAQWhy not COPR?
>  
>     -
>  
>     COPR has been starved of resources for years, which has impaired its
>     ability to provide reliable service at scale. Fedora Infrastructure refuses
>     to consider supporting it officially and Fedora Release Engineering seems
>     to have some undefined issues with COPR. 
Thats kind of glossing over a lot. The reason infrastructure doesn't
want to support it is that its running on a ancient version of openstack
thats very hard to fix when broken. The only issues I know otherwise is
that copr is just designed differently from koji. koji is set to record
everything and keep anything needed to rebuild things, but cope is much
more free form.

In any case with our cloud retirement there will be changes to copr.
Not determined yet, but hopefully it will have more resources and a
solid base to run on.

For the case of this proposal, I agree copr might not be a good fit, as
you are trying to fold all the changes back to koji, so best to just run
a koji to make it as close as possible.

>     -
>  
>     It is not official build system of Fedora which is not helping with Problem
>     №2: Testing of new rpm/koji/mock features/configuration
>    
> <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
Note that if you also are testing new things you could break packagers
that are trying to do other work...

...snip...

>  How can you make Koji builds faster?
>  
>     -
>  
>     By tuning Koji for performance, not correctness and reliability.
>     -
>  
>     By building only on a single, fast architecture. 
This may well bite you when you merge back to koji and build for all
arches.

...snip...

Overall sounds very nice... we will have to work out details if everyone
is on board with it. Do note that it's going to be a lot of work for you
all... :)

kevin

[Mikolaj Izdebski]

On Thu, Jan 31, 2019 at 11:16 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  > https://pagure.io/fesco/issue/2068
>  >
>  > And, after all, those packages shouldn’t be shipped to users.
>
>  This of course means however that users who want to build their own
>  verions of the applications or whatever can't start from our repos, they
>  have to use whatever setup we use for developing them. I don't know any
>  answers here but it seems to me we keep making it harder for people to
>  use Fedora to develop applications. 
All packages built in MBI are meant to be eventually contributed back
to Fedora. Just like developing a package on your own system and
building in local mock. Eventually you push the code to dist-git and
rebuild packages in Fedora Koji. With MBI the difference is that
people can collaboratively devlop changes and they have scalable build
infrastructure.

>  >    Separate Koji + Koschei deployed in Fedora infrastructure
> cloud;
>
>  Turns out we are going to be retiring our cloud, so no, this is not a
>  place to put it. :) 
Cloud doesn't necessarily mean OpenStack. There are other options. For
example, a baremetal, out-of-warranty virthost in an isolated network
would work - better that OpenStack.

MBI was ran this way (baremetal server with KVM guests) for some time.
But it was a private setup, usably only by a few people. Among other
things it was used for GCC removal from buildroot change - all
packages were mass-rebuilt several times and then build logs were
scanned to see which packages failed due to missing gcc/g++ and would
need to have BuildRequires added. The change was eventually pushed to
Fedora.

>  >    People can have their own targets where they can break things
> as they
>  >    wish without affecting others;
>
>  Would that be entirely self service? Or ? 
No, I would be setting up these tags/targets myself, manually.

>  >    Package changes are eventually contributed to Fedora proper
> by merging
>  >    changes to Fedora dist-git and rebuilding packages in official Fedora Koji;
>
>  Where is the source for the changes from? Would this need it's own
>  pagure/src/dist-git? Or would it pull from PR's in prod dist git? Or ? 
Changes would be kept somewhere, depending on packager preferences.
Ideally in a publicly-available git repository where people can
collaborate. Ideally in a fork on src.fedoraproject.org to make sure
that contributors have FPCA signed. Giving myself as example, I have
forked all my packages and created "mbi" branches for them, for
example: https://src.fedoraproject.org/fork/mizdebsk/rpms/ant/commits/mbi

>  > Idea №3: Building packages for multiple distributions
>  >
>  > In Rust ecosystem, we managed to get have same packaging guidelines in
>  > openSUSE, Mageia and Fedora. We could automatically build some packages on
>  > multiple distributions and be kinda upstream.
>
>  So, this setup will need to distribute things... would it need mirrors
>  as well? or ? 
Nothing is distributed to end users. This system is for use by Fedora
contributors - packagers/QA/etc. Of course anyone interested is able
to download these packages, but they are not meant for end users. Just
like Koji scratch builds are not meant for end users, but anyone can
download them and use in production, if they like.

>  >    A few builders constantly running, with a possibility to add
> more
>  >    builders as needed and as available cloud resources allow
>
>  koji doesn't have much way to dynamically deal with builders... 
Koji builders can be added and removed dynamically very easily. Adding
a builder in public cloud is as easy as spawning some VM, installing
koji-builder, putting config file in place and starting kojid - of
course automated using Ansible. Removing a builder only requires
terminating the VM, nothing more. Adding a builder that is installed
on your laptop is as easy as booting up the VM containing in, that's
it.

>  >    Possibility to have some builders running outsides of Fedora
>  >    infrastructure - contributed by proposal owners
>
>  I'd be carefull of this, as it can cause a lot of issues, but of course
>  up to you as you are doing the work. :) 
The idea was to have just a few builders (eg. 3) running constantly.
If someone wants to do massive amounts of builds (and have them
complete quickly) then they can plug in their own hardware or
instances in public cloud that they would pay for, which would be
added to a separate Koji channel. Then their builds would use their
builders. Builders have very low requirements - in particular they
don't need public IP, can be behind firewall/NAT and only need to talk
to hub over https. Even someone's personal laptop could be used as a
builder.

>  >    Koji builds can be done from SCM (src.fp.o, pagure.io,
> github, …) or
>  >    from SRPM upload
>
>  Ah, so it would not have it's own git/pagure? 
No. Builds would be allowed from any configured SCM - it could be from
forks on src.fedoraproject.org, from repositories on pagure.io,
repositories on other git hosting services. It's up to people using
the system to choose the workflow that suits them best. They could
even build from SRPM uploads.

>  >    It is not official build system of Fedora which is not
> helping with Problem
>  >    №2: Testing of new rpm/koji/mock features/configuration
>  >   
> <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
>
>  Note that if you also are testing new things you could break packagers
>  that are trying to do other work... 
Testing new features would be done in separate tags. So no, others
would not be affected.

>  >    By building only on a single, fast architecture.
>
>  This may well bite you when you merge back to koji and build for all
>  arches. 
I don't see this as a problem.Most of the packages that we are
planning to build in MBI (Java/Rust/Go) are noarch anyway.

>  Overall sounds very nice... we will have to work out details if
> everyone
>  is on board with it. Do note that it's going to be a lot of work for you
>  all... :) 
Actually there's not that much work. Most of the work is already done.

I've developed and I'm running MBI setup myself in AWS, using my
personal account. Others liked this workflow and asked whether they
could use it. I kindly refused as I can't pay for all that myself.
Then we come up with a proposal to have this hosted by Fedora.

--
Mikolaj

[Kevin Fenzi]

On 2/1/19 4:06 AM, Mikolaj Izdebski wrote:

...snip...

> >>    Separate Koji + Koschei deployed in Fedora infrastructure
> cloud;
> >
> > Turns out we are going to be retiring our cloud, so no, this is not a
> > place to put it. :)
>  
>  Cloud doesn't necessarily mean OpenStack. There are other options. For
>  example, a baremetal, out-of-warranty virthost in an isolated network
>  would work - better that OpenStack. 
Well, I was responsing to "Fedora Infrastructure cloud" which completely
does mean openstack. ;) But yes, as I mentioned we can come up with
other options if it's desired.

...snip...

>  Koji builders can be added and removed dynamically very easily.
> Adding
>  a builder in public cloud is as easy as spawning some VM, installing
>  koji-builder, putting config file in place and starting kojid - of
>  course automated using Ansible. Removing a builder only requires
>  terminating the VM, nothing more. Adding a builder that is installed
>  on your laptop is as easy as booting up the VM containing in, that's
>  it. 
Sure, except they exist in the database by that name forever.

> >>    Possibility to have some builders running outsides of
> Fedora
> >>    infrastructure - contributed by proposal owners
> >
> > I'd be carefull of this, as it can cause a lot of issues, but of course
> > up to you as you are doing the work. :)
>  
>  The idea was to have just a few builders (eg. 3) running constantly.
>  If someone wants to do massive amounts of builds (and have them
>  complete quickly) then they can plug in their own hardware or
>  instances in public cloud that they would pay for, which would be
>  added to a separate Koji channel. Then their builds would use their
>  builders. Builders have very low requirements - in particular they
>  don't need public IP, can be behind firewall/NAT and only need to talk
>  to hub over https. Even someone's personal laptop could be used as a
>  builder. 
Sure, note there would be some setup here for you... adding channel,
keytabs or certs for builders, etc.

> >>    Koji builds can be done from SCM (src.fp.o, pagure.io,
> github, …) or
> >>    from SRPM upload
> >
> > Ah, so it would not have it's own git/pagure?
>  
>  No. Builds would be allowed from any configured SCM - it could be from
>  forks on src.fedoraproject.org, from repositories on pagure.io,
>  repositories on other git hosting services. It's up to people using
>  the system to choose the workflow that suits them best. They could
>  even build from SRPM uploads.
>  
> >>    It is not official build system of Fedora which is not helping with Problem
> >>    №2: Testing of new rpm/koji/mock features/configuration
> >>   
> <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
> >
> > Note that if you also are testing new things you could break packagers
> > that are trying to do other work...
>  
>  Testing new features would be done in separate tags. So no, others
>  would not be affected. 
I'm not sure how you could test a new koji hub or rpm on the hub with
tags, but sure, you could test some things by having different builders.

>  
> >>    By building only on a single, fast architecture.
> >
> > This may well bite you when you merge back to koji and build for all
> > arches.
>  
>  I don't see this as a problem.Most of the packages that we are
>  planning to build in MBI (Java/Rust/Go) are noarch anyway. 
ok. Unless some other folks start using it.

> > Overall sounds very nice... we will have to work out details if
> everyone
> > is on board with it. Do note that it's going to be a lot of work for you
> > all... :)
>  
>  Actually there's not that much work. Most of the work is already done. 
great.

>  I've developed and I'm running MBI setup myself in AWS, using
> my
>  personal account. Others liked this workflow and asked whether they
>  could use it. I kindly refused as I can't pay for all that myself.
>  Then we come up with a proposal to have this hosted by Fedora. 
Sure, makes sense.

Thanks for the answers!

kevin

[Mikolaj Izdebski]

On Fri, Feb 1, 2019 at 4:48 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  >>>    It is not official build system of Fedora which is
> not helping with Problem
>  >>>    №2: Testing of new rpm/koji/mock features/configuration
>  >>>   
> <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
>  >>
>  >> Note that if you also are testing new things you could break packagers
>  >> that are trying to do other work...
>  >
>  > Testing new features would be done in separate tags. So no, others
>  > would not be affected.
>
>  I'm not sure how you could test a new koji hub or rpm on the hub with
>  tags, but sure, you could test some things by having different builders. 
Testing new rpm features, global macros etc. can be done by tagging a
build (of rpm, redhat-rpm-config etc.) into one tag without affecting
other tags.
Testing new kojid/mock configuration can be done on builders in
non-default channels.

For testing new Koji releases we have staging setup.

>  >>>    By building only on a single, fast architecture.
>  >>
>  >> This may well bite you when you merge back to koji and build for all
>  >> arches.
>  >
>  > I don't see this as a problem.Most of the packages that we are
>  > planning to build in MBI (Java/Rust/Go) are noarch anyway.
>
>  ok. Unless some other folks start using it. 
Then we can reconsider more architectures later. Or they can
contribute their builders of any architectures, even not supported by
Fedora Koji.

--
Mikolaj

[Stephen John Smoogen]

On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <
ignatenkobrain(a)fedoraproject.org&gt; wrote:

>  Problem №1: Build-only packages
>
>
>  Rawhide gating makes this much more complicated because builds appear in
>  buildroot slower, updating group of packages would need side tags and it’s
>  just painful to work with.
>
>  https://pagure.io/fesco/issue/2068
>
>  And, after all, those packages shouldn’t be shipped to users.
>  
My main problem with this is the above. Yes Joe Desktop User isn't going to
see/need that package.. but we have a LOT of packagers who take our stuff
and rebuild it for themselves for various reasons. I find it hard to put
together the proposal which is supposed to make developing/packaging easier
with making developing/packaging harder. Whether you want it to or not,
this comes across as "If you want to use Go or Rust, you will need our
special set of tools which we keep hidden."


>  

-- 
Stephen J Smoogen.

[Neal Gompa]

On Thu, Jan 31, 2019 at 6:58 PM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <ignatenkobrain(a)fedoraproject.org&gt;
> wrote:
> >
> > Problem №1: Build-only packages
> >
> > Rawhide gating makes this much more complicated because builds appear in buildroot
> slower, updating group of packages would need side tags and it’s just painful to work
> with.
> >
> > https://pagure.io/fesco/issue/2068
> >
> > And, after all, those packages shouldn’t be shipped to users.
>
>
>  My main problem with this is the above. Yes Joe Desktop User isn't going to see/need
> that package.. but we have a LOT of packagers who take our stuff and rebuild it for
> themselves for various reasons. I find it hard to put together the proposal which is
> supposed to make developing/packaging easier with making developing/packaging harder.
> Whether you want it to or not, this comes across as "If you want to use Go or Rust,
> you will need our special set of tools which we keep hidden."
>  
This is actually something I really don't like either. But the Fedora
leadership has pushed very hard on the concept of having packages that
aren't available to "normies", and require special tooling to be able
to leverage (that for various reasons, I can't even use as a third
party packager!). That's a big part of the core to how Modularity is
being used. The CRB in RHEL 8 is another example of this. But fixing
this is very hard when few others see it as a problem. There was even
talk of not publishing SRPMs of packages that make up modules a while
ago. That idea died very quickly thankfully, but I don't know how
people think of these things anymore.

I'm tired of fighting... Our tools haven't improved enough to handle
our new challenges in the Fedora way, because it's hard for people in
the community to experiment and explore in this manner. My hope is
that with something like MBI, we can finally put the power to
experiment and develop tooling that actually makes sense for Fedora
(rather than clearly being designed for RHEL and shoehorned into
Fedora) without all the pain and agony of people imposing RHELish
requirements. There's no way we can evolve and meet the needs of our
communities without some way for people to "play".

COPR was supposed to be that outlet, but no one gives a damn about it.
Everyone complains that the service is "bad" and that the design is
"bad" but no one wants to actually constructively improve it. The
quality of service on COPR has fallen due to lack of care and
unwillingness to invest, so what are we supposed to do? The horrible
thing is that COPR is successful *despite* that. COPR is an
interesting and cool service, and I would have loved to see an
eventual merger of the Koji and COPR systems (because there are
awesome attributes to both and we should have a large and strong team
actually supporting the literal core of the distribution), but it
won't happen because everyone thinks the COPR system is terrible even
though it's not really.

I've been arguing for literally years about our shortcomings. I've
built tools for working around pitfalls in the Fedora workflow for my
personal use. I've personally explored how other people and other
distributions do things. I got my hands dirty to learn about other
ways of doing things. I've used that knowledge and expertise to avoid
those mistakes when deploying package and image construction
infrastructure for $DAYJOB. But it deeply saddens me that I have not
been able to make any meaningful progress in convincing anyone that it
was something that we should invest in.

I still believe that Modularity, as designed, is a mistake. But if I
don't have a way to prove a better way to do things, there's no way
anything will improve. The folks running the show like what they have
now, so it'll take a lot to trigger change.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Kevin Fenzi]

On 1/31/19 4:52 PM, Neal Gompa wrote:

...snip...

>  COPR was supposed to be that outlet, but no one gives a damn about
> it.
>  Everyone complains that the service is "bad" and that the design is
>  "bad" but no one wants to actually constructively improve it. The
>  quality of service on COPR has fallen due to lack of care and
>  unwillingness to invest, so what are we supposed to do? The horrible 
I've heard you and some others say this, but can you perhaps expand on
it? How has quality of service of copr fallen?

There are times when a bunch of builds are dumped on it that it takes a
bit to catch up, but it's usually like an hour not days or anything.
Is it the build time you refer to? Or something(s) else?

kevin

[Mikolaj Izdebski]

On Fri, Feb 1, 2019 at 2:28 AM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On 1/31/19 4:52 PM, Neal Gompa wrote:
>
>  ...snip...
>
>  > COPR was supposed to be that outlet, but no one gives a damn about it.
>  > Everyone complains that the service is "bad" and that the design is
>  > "bad" but no one wants to actually constructively improve it. The
>  > quality of service on COPR has fallen due to lack of care and
>  > unwillingness to invest, so what are we supposed to do? The horrible
>
>  I've heard you and some others say this, but can you perhaps expand on
>  it? How has quality of service of copr fallen? 
There are some of current/recent major problems with COPR that did not
exist initially:

- builds failing due to failure to download packages from official
Fedora mirror dl.fedoraproject.org
- builds failing due to problems with keygen (was not a problem before
keygen was introduced)
- builds failing to import to COPR distgit (was not a problem before
dist-git was introduced)
- outages caused by read-only filesystem after reboot - copr is
unusable until someone remounts it rw
- multi-day long outages caused by out of disk space
- multi-hour or even day- long build queues
- outdated SOP preventing non-COPR people like me from fixing COPR outages

--
Mikolaj

[Nicolas Mailhot]

Le 2019-02-01 13:21, Mikolaj Izdebski a écrit :
>  On Fri, Feb 1, 2019 at 2:28 AM Kevin Fenzi <kevin(a)scrye.com&gt;
> wrote:
> > 
> > On 1/31/19 4:52 PM, Neal Gompa wrote:
> > 
> > ...snip...
> > 
> > > COPR was supposed to be that outlet, but no one gives a damn about it.
> > > Everyone complains that the service is "bad" and that the design is
> > > "bad" but no one wants to actually constructively improve it. The
> > > quality of service on COPR has fallen due to lack of care and
> > > unwillingness to invest, so what are we supposed to do? The horrible
> > 
> > I've heard you and some others say this, but can you perhaps expand on
> > it? How has quality of service of copr fallen?
>  
>  There are some of current/recent major problems with COPR that did not
>  exist initially:
>  
>  - builds failing due to failure to download packages from official
>  Fedora mirror dl.fedoraproject.org
>  - builds failing due to problems with keygen (was not a problem before
>  keygen was introduced)
>  - builds failing to import to COPR distgit (was not a problem before
>  dist-git was introduced) 
And builds failing is just an annoyance when you are building a single 
leaf package. When your build execution plan covers hundreds of 
interdependent packages, with bootstrapping phases, having something 
failing in the middle is not just an annoyance, it often means 
restarting the full plan from scratch, because trying to salvage the 
result manually is just too much work

Regards,

-- 
Nicolas Mailhot

[Miroslav Suchý]

Dne 01. 02. 19 v 13:21 Mikolaj Izdebski napsal(a):
>  - builds failing due to failure to download packages from official
>  Fedora mirror dl.fedoraproject.org 
This is not first time I hear this. So I will open discussion to (again) alter
builder's mock config to point to PHX
location, which is just rack away.

>  - builds failing to import to COPR distgit (was not a problem before
>  dist-git was introduced) 
This is actually new to me. Can you point me to some report?

>  - outages caused by read-only filesystem after reboot - copr is
>  unusable until someone remounts it rw 
This was actually recently fixed.

>  - multi-day long outages caused by out of disk space
>  - multi-hour or even day- long build queues 
That is because I actually care about the resources. It is very easy to submit a proposal
and state that "it will run in
Fedora Cloud", but the cloud is not infinite. I can easily increase the soft limits
in Cloud dashboard. But we are
either hitting hard limits (storage) or we would drastically limit other users using the
same cloud (vCPU in case more
builders).
Copr is not designed for spikes, it is designed for the average traffic. Which is handling
pretty well. You can check it
here:
https://copr.fedorainfracloud.org/status/stats/
If you submit 300+ build today (like happened today) you will simply have to wait some
time. At the same time, it should
not affect (too much) other users. This is IMO big difference from Koji which is IIRC
strictly FIFO.


>  - outdated SOP preventing non-COPR people like me from fixing COPR
> outages 
Feel free to raise any issue. Last update in this document was in October 2018. Checking
it right now I see outdated
part about ppc builders, but otherwise it should be ok.
Thou, I will review that document after weekend.

Miroslav

[Mikolaj Izdebski]

On Fri, Feb 1, 2019 at 3:37 PM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 01. 02. 19 v 13:21 Mikolaj Izdebski napsal(a):
>  > - builds failing to import to COPR distgit (was not a problem before
>  > dist-git was introduced)
>
>  This is actually new to me. Can you point me to some report? 
I will try to find a reference.

>  > - outdated SOP preventing non-COPR people like me from fixing
> COPR outages
>
>  Feel free to raise any issue. Last update in this document was in October 2018. Checking
> it right now I see outdated
>  part about ppc builders, but otherwise it should be ok.
>  Thou, I will review that document after weekend. 
Already reported in RFR ticket:
https://pagure.io/fedora-infrastructure/issue/5166#comment-550297

--
Mikolaj

[Neal Gompa]

On Sun, Feb 3, 2019 at 6:19 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 01. 02. 19 v 13:21 Mikolaj Izdebski napsal(a):
>  > - builds failing due to failure to download packages from official
>  > Fedora mirror dl.fedoraproject.org
>
>  This is not first time I hear this. So I will open discussion to (again) alter
> builder's mock config to point to PHX
>  location, which is just rack away.
>  
Wouldn't it make sense to maintain a local mirror of the distributions
enabled in COPR and just have all the mock configs point to that? That
eliminates all the network pressure and the issues we keep seeing
there when trying to use it.

>  > - builds failing to import to COPR distgit (was not a problem
> before
>  > dist-git was introduced)
>
>  This is actually new to me. Can you point me to some report?
>  
I can't say I've experienced this issue. However, I do wish we could
interact with COPR Dist-Git the same way we do with the official
Fedora Dist-Git, with a Pagure instance and everything...

>  > - outages caused by read-only filesystem after reboot - copr is
>  > unusable until someone remounts it rw
>
>  This was actually recently fixed.
>
>  > - multi-day long outages caused by out of disk space
>  > - multi-hour or even day- long build queues
>
>  That is because I actually care about the resources. It is very easy to submit a proposal
> and state that "it will run in
>  Fedora Cloud", but the cloud is not infinite. I can easily increase the soft limits
> in Cloud dashboard. But we are
>  either hitting hard limits (storage) or we would drastically limit other users using the
> same cloud (vCPU in case more
>  builders).
>  Copr is not designed for spikes, it is designed for the average traffic. Which is
> handling pretty well. You can check it
>  here:
>  https://copr.fedorainfracloud.org/status/stats/
>  If you submit 300+ build today (like happened today) you will simply have to wait some
> time. At the same time, it should
>  not affect (too much) other users. This is IMO big difference from Koji which is IIRC
> strictly FIFO.
>  
Would it be possible to extend COPR to support multiple locations for
builders? For example, in addition to an OpenStack system, builders
could be hosted on an oVirt system, or AWS, or GCP, and so on? That
way it can support on-premise deployments and cloudy deployments too.

Perhaps supporting even some limited form of auto-scaling for when it
needs to "burst" to support more build traffic using clouds or
hypervisors? For example, you could use AWS spot instances to do it on
the cheap for the time a builder needs to run, and then have it go
away afterward. I've done builds this way with buildbot and you can do
thousands of builds for really cheap (on the order of hundreds of
dollars each year).

>  > - outdated SOP preventing non-COPR people like me from fixing COPR outages
>
>  Feel free to raise any issue. Last update in this document was in October 2018. Checking
> it right now I see outdated
>  part about ppc builders, but otherwise it should be ok.
>  Thou, I will review that document after weekend.
>  
I don't know anything about this...


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Stephen John Smoogen]

On Sun, 3 Feb 2019 at 16:43, Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Sun, Feb 3, 2019 at 6:19 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  >
>  > Dne 01. 02. 19 v 13:21 Mikolaj Izdebski napsal(a):
>  > > - builds failing due to failure to download packages from official
>  > > Fedora mirror dl.fedoraproject.org
>  >
>  > This is not first time I hear this. So I will open discussion to (again) alter
> builder's mock config to point to PHX
>  > location, which is just rack away.
>  > 
So problem 1 is that dl.fedoraproject.org is 1 rack away. There is
some sort of problem with the setup of the dl servers which is no
longer using memory efficiently. I think it needs a local varnish or
similar caching server but i haven't had time to analyze.

>  Wouldn't it make sense to maintain a local mirror of the distributions
>  enabled in COPR and just have all the mock configs point to that? That
>  eliminates all the network pressure and the issues we keep seeing
>  there when trying to use it.
>  
COPR doesn't have the disk space to mirror all that at speed. [Yes you
can cheaply stick them on slow large drives but trying to run the
several dozen builders would be worse than going across the rack to
the other network server. So you then need a lot of slow large drives
and then you are no longer cheap and it seems like you have a lot of
empty disk space that isn't being used so you might as well go with
smaller expensive SAS/SSD. ]

>  Would it be possible to extend COPR to support multiple locations for
>  builders? For example, in addition to an OpenStack system, builders
>  could be hosted on an oVirt system, or AWS, or GCP, and so on? That
>  way it can support on-premise deployments and cloudy deployments too.
>  
I expect it is possible but it is going to come down to a
credential/funding problem.  Going out to the cloud like GCP/AWS
requires funding and setup to make sure that the systems aren't
compromised. [We have had several cloud offers from smaller vendors
but they then wanted our root password and other credentials.. just to
remind you that this isn't your hardware and they can look in it any
time you want.]

There would also be the speed issue of getting the data from the
on-premise to the cloud. That takes time and sometimes seems to take
longer than waiting in queue locally. Then there is the mirroring part
of the data. [That seems to be slower than molasses at times on the
order of a day. I expect it is partially that we are doing something
naive and would be better dealt with elsewhere.]

>  Perhaps supporting even some limited form of auto-scaling for when
> it
>  needs to "burst" to support more build traffic using clouds or
>  hypervisors? For example, you could use AWS spot instances to do it on
>  the cheap for the time a builder needs to run, and then have it go
>  away afterward. I've done builds this way with buildbot and you can do
>  thousands of builds for really cheap (on the order of hundreds of
>  dollars each year).
>  
That would be useful.



-- 
Stephen J Smoogen.

[Pavel Raiskup]

On Sunday, February 3, 2019 9:45:06 PM CET Neal Gompa wrote:
>  Would it be possible to extend COPR to support multiple locations
> for
>  builders? For example, in addition to an OpenStack system, builders
>  could be hosted on an oVirt system, or AWS, or GCP, and so on? That way
>  it can support on-premise deployments and cloudy deployments too. 
I have implemented something along those lines in the Red Hat internal
Copr deployment.  There's something like middleware between Copr and VM
providers which allocates the VMs a flexible way on multiple places.
This feature is tracked in https://bugzilla.redhat.com/1334701 .

This would be really trivial to deploy for Fedora Copr - if there was use
for it (are there any resources which could rise throughput in Fedora
Copr?).  That said, this is not much of technical problem to me, but
rather policy/funding problem nowadays.

>  Perhaps supporting even some limited form of auto-scaling for when
> it
>  needs to "burst" to support more build traffic using clouds or
>  hypervisors? For example, you could use AWS spot instances to do it on
>  the cheap for the time a builder needs to run, and then have it go away
>  afterward. I've done builds this way with buildbot and you can do
>  thousands of builds for really cheap (on the order of hundreds of
>  dollars each year). 
This is really interesting idea...

Pavel

[Stephen John Smoogen]

On Fri, 1 Feb 2019 at 08:17, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:

>  On Fri, Feb 1, 2019 at 2:28 AM Kevin Fenzi <kevin(a)scrye.com&gt;
> wrote:
>  >
>  > On 1/31/19 4:52 PM, Neal Gompa wrote:
>  >
>  > ...snip...
>  >
>  > > COPR was supposed to be that outlet, but no one gives a damn about it.
>  > > Everyone complains that the service is "bad" and that the design is
>  > > "bad" but no one wants to actually constructively improve it. The
>  > > quality of service on COPR has fallen due to lack of care and
>  > > unwillingness to invest, so what are we supposed to do? The horrible
>  >
>  > I've heard you and some others say this, but can you perhaps expand on
>  > it? How has quality of service of copr fallen?
>
>  There are some of current/recent major problems with COPR that did not
>  exist initially:
>
>  - builds failing due to failure to download packages from official
>  Fedora mirror dl.fedoraproject.org 

This one is on us in infrastructure.  The problem is mainly because dl was
never scoped or built to handle the load that COPR brings. It is meant to
mostly deal with yum updates of last resort and mirror traffic. When a
bunch of COPR builds happen it gets overloaded and will drop those. It does
need to be fixed for this... but the MBI or similar tools will run into
this also.

The others I can't really speak for (beyond the disk space where
Infrastructure provides an equallogics and it would need a budget increase
to get more space.)


>  - builds failing due to problems with keygen (was not a problem
> before
>  keygen was introduced)
>  - builds failing to import to COPR distgit (was not a problem before
>  dist-git was introduced)
>  - outages caused by read-only filesystem after reboot - copr is
>  unusable until someone remounts it rw
>  - multi-day long outages caused by out of disk space
>  - multi-hour or even day- long build queues
>  - outdated SOP preventing non-COPR people like me from fixing COPR outages
>
>  --
>  Mikolaj
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 
Stephen J Smoogen.

[Neal Gompa]

On Mon, Feb 4, 2019 at 5:14 AM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On 1/31/19 4:52 PM, Neal Gompa wrote:
>
>  ...snip...
>
>  > COPR was supposed to be that outlet, but no one gives a damn about it.
>  > Everyone complains that the service is "bad" and that the design is
>  > "bad" but no one wants to actually constructively improve it. The
>  > quality of service on COPR has fallen due to lack of care and
>  > unwillingness to invest, so what are we supposed to do? The horrible
>
>  I've heard you and some others say this, but can you perhaps expand on
>  it? How has quality of service of copr fallen?
>
>  There are times when a bunch of builds are dumped on it that it takes a
>  bit to catch up, but it's usually like an hour not days or anything.
>  Is it the build time you refer to? Or something(s) else?
>  
Aside from the times when it falls over for various reasons, I've had
entire days where I wait for a build to even start, because people who
use it for doing things like building KDE, chromium, or the Linux
kernel occupy literally all the available builder slots for a long
time. There aren't that many slots and it's easy to fill that up.
There's usually a large queue of packages to build, but not enough
builders to allow them to get done.

That indicates two things:
1. The builders are weak and so builds take a long time (which means
slots are held up longer)
2. The demand and popularity of the service isn't being handled
appropriately (i.e. it should get more builders provisioned).

I don't do things like build kernels often, but when I do, it usually
doesn't take all day. But stuff like Chromium is hard to build
locally, so I appreciate that we have somewhere to build and publish.

But, as of right now, there are 16 tasks running, with 85 tasks
waiting for a builder.

I wish we had visualizations like the ones OBS has[1][2], so that we
have an idea of how stuff is occupied and know at a glance that we're
over capacity.

All I know right now is that it's easy to see that COPR gets into a
state where I just wind up waiting for builds to even start.

[1]: https://build.opensuse.org/monitor
[2]: https://build.opensuse.org/monitor/old


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Kevin Fenzi]

On 2/4/19 4:03 PM, Neal Gompa wrote:

>  Aside from the times when it falls over for various reasons, I've
> had
>  entire days where I wait for a build to even start, because people who
>  use it for doing things like building KDE, chromium, or the Linux
>  kernel occupy literally all the available builder slots for a long
>  time. There aren't that many slots and it's easy to fill that up.
>  There's usually a large queue of packages to build, but not enough
>  builders to allow them to get done. 
Well, thats not supposed to be the case. Each user is given a limited
amount of slots, so if say I dumped 100 kernels on it, it would only do
a few of those at a time. So, sounds like a bug in allowing people more
slots than they should have?
>  That indicates two things:
>  1. The builders are weak and so builds take a long time (which means
>  slots are held up longer) 
Could be indeed. Could look at increasing the size...

>  2. The demand and popularity of the service isn't being handled
>  appropriately (i.e. it should get more builders provisioned). 
Well, there's another issue here that seems to be a copr bug (see below)

>  I don't do things like build kernels often, but when I do, it
> usually
>  doesn't take all day. But stuff like Chromium is hard to build
>  locally, so I appreciate that we have somewhere to build and publish.
>  
>  But, as of right now, there are 16 tasks running, with 85 tasks
>  waiting for a builder. 
Yeah, but copr is allocated the following:

150 instances
200 vcps
488 GB ram

So, either copr is loosing track of it's builders or openstack is doing
something odd and not really providing that. I know openstack has
accounting issues, but I do see about 65 or so builder instances, so I
think this is on the copr side. So, fixing this would go a long way to
helping out.

Additionally, sometimes jobs finish, but copr shows them as still
running. For example, the oldest job now:

https://copr.fedorainfracloud.org/coprs/omos/kernel-testing/

which copr shows running for 11 hours, the logs show it's done (I don't
know how long ago as there's no timestamps).

>  I wish we had visualizations like the ones OBS has[1][2], so that we
>  have an idea of how stuff is occupied and know at a glance that we're
>  over capacity.
>  
>  All I know right now is that it's easy to see that COPR gets into a
>  state where I just wind up waiting for builds to even start.
>  
>  [1]: https://build.opensuse.org/monitor
>  [2]: https://build.opensuse.org/monitor/old 
Well, we do have:
https://copr.fedorainfracloud.org/status/stats/
but some of it doesn't match up with osbs, since copr builders are dynamic.

kevin

[Fabio Valentini]

FWIW, it looks like COPR's connectivity / download issues when
installing the buildroot have gotten even worse over the past two
days;
The last 13 builds for my elementary-nightly repository **all** failed
due to some download issue in root.log, and the situation was similar
yesterday morning.

At this point, COPR is becoming completely useless to catch upstream
issues, since builds fail for COPR related issues 99% of the time, and
only 1% are real issues.
I usually try to re-submit builds until they either succeed or fail
for real, but at this rate (and with the slow web interface), this
isn't really feasible anymore.

Fabio

On Tue, Feb 5, 2019 at 6:07 AM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>
> On 2/4/19 4:03 PM, Neal Gompa wrote:
>
> > Aside from the times when it falls over for various reasons, I've had
> > entire days where I wait for a build to even start, because people who
> > use it for doing things like building KDE, chromium, or the Linux
> > kernel occupy literally all the available builder slots for a long
> > time. There aren't that many slots and it's easy to fill that up.
> > There's usually a large queue of packages to build, but not enough
> > builders to allow them to get done.
>
> Well, thats not supposed to be the case. Each user is given a limited
> amount of slots, so if say I dumped 100 kernels on it, it would only do
> a few of those at a time. So, sounds like a bug in allowing people more
> slots than they should have?
> > That indicates two things:
> > 1. The builders are weak and so builds take a long time (which means
> > slots are held up longer)
>
> Could be indeed. Could look at increasing the size...
>
> > 2. The demand and popularity of the service isn't being handled
> > appropriately (i.e. it should get more builders provisioned).
>
> Well, there's another issue here that seems to be a copr bug (see below)
>
> > I don't do things like build kernels often, but when I do, it usually
> > doesn't take all day. But stuff like Chromium is hard to build
> > locally, so I appreciate that we have somewhere to build and publish.
> >
> > But, as of right now, there are 16 tasks running, with 85 tasks
> > waiting for a builder.
>
> Yeah, but copr is allocated the following:
>
> 150 instances
> 200 vcps
> 488 GB ram
>
> So, either copr is loosing track of it's builders or openstack is doing
> something odd and not really providing that. I know openstack has
> accounting issues, but I do see about 65 or so builder instances, so I
> think this is on the copr side. So, fixing this would go a long way to
> helping out.
>
> Additionally, sometimes jobs finish, but copr shows them as still
> running. For example, the oldest job now:
>
> https://copr.fedorainfracloud.org/coprs/omos/kernel-testing/
>
> which copr shows running for 11 hours, the logs show it's done (I don't
> know how long ago as there's no timestamps).
>
> > I wish we had visualizations like the ones OBS has[1][2], so that we
> > have an idea of how stuff is occupied and know at a glance that we're
> > over capacity.
> >
> > All I know right now is that it's easy to see that COPR gets into a
> > state where I just wind up waiting for builds to even start.
> >
> > [1]: https://build.opensuse.org/monitor
> > [2]: https://build.opensuse.org/monitor/old
>
> Well, we do have:
> https://copr.fedorainfracloud.org/status/stats/
> but some of it doesn't match up with osbs, since copr builders are dynamic.
>
> kevin
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Stephen John Smoogen]

On Thu, 7 Feb 2019 at 07:34, Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  FWIW, it looks like COPR's connectivity / download issues when
>  installing the buildroot have gotten even worse over the past two
>  days;
>  The last 13 builds for my elementary-nightly repository **all** failed
>  due to some download issue in root.log, and the situation was similar
>  yesterday morning.
>
>  At this point, COPR is becoming completely useless to catch upstream
>  issues, since builds fail for COPR related issues 99% of the time, and
>  only 1% are real issues.
>  I usually try to re-submit builds until they either succeed or fail
>  for real, but at this rate (and with the slow web interface), this
>  isn't really feasible anymore.
>  
I need links and data to go with for any troubleshooting. I don't know
where these reports you have are and without that I can't know which
download servers are causing a problem.


-- 
Stephen J Smoogen.

[Fabio Valentini]

On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Thu, 7 Feb 2019 at 07:34, Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  >
>  > FWIW, it looks like COPR's connectivity / download issues when
>  > installing the buildroot have gotten even worse over the past two
>  > days;
>  > The last 13 builds for my elementary-nightly repository **all** failed
>  > due to some download issue in root.log, and the situation was similar
>  > yesterday morning.
>  >
>  > At this point, COPR is becoming completely useless to catch upstream
>  > issues, since builds fail for COPR related issues 99% of the time, and
>  > only 1% are real issues.
>  > I usually try to re-submit builds until they either succeed or fail
>  > for real, but at this rate (and with the slow web interface), this
>  > isn't really feasible anymore.
>  >
>
>  I need links and data to go with for any troubleshooting. I don't know
>  where these reports you have are and without that I can't know which
>  download servers are causing a problem. 
Sorry, forgot to add an explicit link:
https://copr.fedorainfracloud.org/coprs/decathorpe/elementary-nightly/bui...

As you can see, *all* builds since about 14 hours ago failed with
either "Error: Failed to synchronize cache for repo 'fedora'"  (or the
same message for the "updates" repo).

Fabio

> --
> Stephen J Smoogen.
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Stephen John Smoogen]

On Thu, 7 Feb 2019 at 09:21, Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  >
>  > On Thu, 7 Feb 2019 at 07:34, Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  > >
>  > > FWIW, it looks like COPR's connectivity / download issues when
>  > > installing the buildroot have gotten even worse over the past two
>  > > days;
>  > > The last 13 builds for my elementary-nightly repository **all** failed
>  > > due to some download issue in root.log, and the situation was similar
>  > > yesterday morning.
>  > >
>  > > At this point, COPR is becoming completely useless to catch upstream
>  > > issues, since builds fail for COPR related issues 99% of the time, and
>  > > only 1% are real issues.
>  > > I usually try to re-submit builds until they either succeed or fail
>  > > for real, but at this rate (and with the slow web interface), this
>  > > isn't really feasible anymore.
>  > >
>  >
>  > I need links and data to go with for any troubleshooting. I don't know
>  > where these reports you have are and without that I can't know which
>  > download servers are causing a problem.
>
>  Sorry, forgot to add an explicit link:
>  https://copr.fedorainfracloud.org/coprs/decathorpe/elementary-nightly/bui...
>
>  As you can see, *all* builds since about 14 hours ago failed with
>  either "Error: Failed to synchronize cache for repo 'fedora'"  (or the
>  same message for the "updates" repo).
>  
The page was dynamic and gave me all good ones when I went there. I
think this failed one shows the problem:

https://copr-be.cloud.fedoraproject.org/results/decathorpe/elementary-nig...

2019-02-07T00:01:52Z DEBUG fedora: using metadata from Wed 24 Oct 2018
10:20:15 PM UTC.
2019-02-07T00:01:52Z DDEBUG repo: downloading from remote: updates,
_Handle: metalnk:
https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...,
mlist: None, urls [].
2019-02-07T00:02:29Z DEBUG Cannot download
'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&arch=x86_64':
Cannot prepare internal mirrorlist: Curl error (28): Timeout was
reached for
https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...
[Operation too slow. Less than 1000 bytes/sec transferred the last 30
seconds].
2019-02-07T00:02:29Z DDEBUG Cleaning up.
2019-02-07T00:02:29Z SUBDEBUG
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 230, in _perform
    return super(_Handle, self).perform(result)
  File "/usr/lib64/python3.6/site-packages/librepo/__init__.py", line
1522, in perform
    _librepo.Handle.perform(self, result)
librepo.LibrepoException: (8, 'Cannot prepare internal mirrorlist:
Curl error (28): Timeout was reached for
https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...
[Operation too slow. Less than 1000 bytes/sec transferred the last 30
seconds]', 'An Curl handle error')

That says the proxy it was trying to get to was too slow.. I will see
if I can find which one it was at that timezone and see if I can
address.


>  Fabio
>
>  > --
>  > Stephen J Smoogen.
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail] 


-- 
Stephen J Smoogen.

[Stephen John Smoogen]

On Thu, 7 Feb 2019 at 09:40, Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Thu, 7 Feb 2019 at 09:21, Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  >
>  > On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  > >
>  > > On Thu, 7 Feb 2019 at 07:34, Fabio Valentini <decathorpe(a)gmail.com&gt;
> wrote:
>  > > > 
>  librepo.LibrepoException: (8, 'Cannot prepare internal
> mirrorlist:
>  Curl error (28): Timeout was reached for
>  https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...
>  [Operation too slow. Less than 1000 bytes/sec transferred the last 30
>  seconds]', 'An Curl handle error')
>
>  That says the proxy it was trying to get to was too slow.. I will see
>  if I can find which one it was at that timezone and see if I can
>  address.
>  
OK the problem does not look like it is mirrorlist related but
something to do with the copr host/network at that time

209.132.184.33 - - [07/Feb/2019:00:01:59 +0000] "GET
/metalink?repo=updates-released-f29&arch=x86_64 HTTP/1.1" 200 18343
"-" "dnf/2.7.5"

says it sent the data but it looks like something that the client was
looking for timed out. I will work with the copr team to diagnose
further.


-- 
Stephen J Smoogen.

[Fabio Valentini]

On Thu, Feb 7, 2019, 18:29 Stephen John Smoogen <smooge(a)gmail.com wrote:

>  On Thu, 7 Feb 2019 at 09:40, Stephen John Smoogen
> <smooge(a)gmail.com&gt;
>  wrote:
>  >
>  > On Thu, 7 Feb 2019 at 09:21, Fabio Valentini <decathorpe(a)gmail.com&gt;
>  wrote:
>  > >
>  > > On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen <smooge(a)gmail.com&gt;
>  wrote:
>  > > >
>  > > > On Thu, 7 Feb 2019 at 07:34, Fabio Valentini <decathorpe(a)gmail.com&gt;
>  wrote:
>  > > > >
>
>  > librepo.LibrepoException: (8, 'Cannot prepare internal mirrorlist:
>  > Curl error (28): Timeout was reached for
>  >
>  https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...
>  > [Operation too slow. Less than 1000 bytes/sec transferred the last 30
>  > seconds]', 'An Curl handle error')
>  >
>  > That says the proxy it was trying to get to was too slow.. I will see
>  > if I can find which one it was at that timezone and see if I can
>  > address.
>  >
>
>  OK the problem does not look like it is mirrorlist related but
>  something to do with the copr host/network at that time
>
>  209.132.184.33 - - [07/Feb/2019:00:01:59 +0000] "GET
>  /metalink?repo=updates-released-f29&arch=x86_64 HTTP/1.1" 200 18343
>  "-" "dnf/2.7.5"
>
>  says it sent the data but it looks like something that the client was
>  looking for timed out. I will work with the copr team to diagnose
>  further.
>  
Thank you!

Fabio


>  --
>  Stephen J Smoogen.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Adam Williamson]

On Thu, 2019-02-07 at 12:21 -0500, Stephen John Smoogen wrote:
>  On Thu, 7 Feb 2019 at 09:40, Stephen John Smoogen
> <smooge(a)gmail.com&gt; wrote:
>  > On Thu, 7 Feb 2019 at 09:21, Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  > > On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen <smooge(a)gmail.com&gt;
> wrote:
>  > > > On Thu, 7 Feb 2019 at 07:34, Fabio Valentini <decathorpe(a)gmail.com&gt;
> wrote:
>  > librepo.LibrepoException: (8, 'Cannot prepare internal mirrorlist:
>  > Curl error (28): Timeout was reached for
>  >
> https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...
>  > [Operation too slow. Less than 1000 bytes/sec transferred the last 30
>  > seconds]', 'An Curl handle error')
>  > 
>  > That says the proxy it was trying to get to was too slow.. I will see
>  > if I can find which one it was at that timezone and see if I can
>  > address.
>  > 
>  
>  OK the problem does not look like it is mirrorlist related but
>  something to do with the copr host/network at that time
>  
>  209.132.184.33 - - [07/Feb/2019:00:01:59 +0000] "GET
>  /metalink?repo=updates-released-f29&arch=x86_64 HTTP/1.1" 200 18343
>  "-" "dnf/2.7.5"
>  
>  says it sent the data but it looks like something that the client was
>  looking for timed out. I will work with the copr team to diagnose
>  further. 
FWIW, openQA tests have been failing relatively frequently with
timeouts while refreshing repos or 503s from mirrormanager for the last
few weeks. I'm pretty sure this didn't used to be anything like as
common...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Stephen John Smoogen]

On Fri, 8 Feb 2019 at 14:31, Adam Williamson <adamwill(a)fedoraproject.org&gt; wrote:
>  On Thu, 2019-02-07 at 12:21 -0500, Stephen John Smoogen wrote:
>  > On Thu, 7 Feb 2019 at 09:40, Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  > > On Thu, 7 Feb 2019 at 09:21, Fabio Valentini <decathorpe(a)gmail.com&gt;
> wrote:
>  > > > On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen
> <smooge(a)gmail.com&gt; wrote:
>  > > > > On Thu, 7 Feb 2019 at 07:34, Fabio Valentini
> <decathorpe(a)gmail.com&gt; wrote:
>  > > librepo.LibrepoException: (8, 'Cannot prepare internal mirrorlist:
>  > > Curl error (28): Timeout was reached for
>  > >
> https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...
>  > > [Operation too slow. Less than 1000 bytes/sec transferred the last 30
>  > > seconds]', 'An Curl handle error')
>  > >
>  > > That says the proxy it was trying to get to was too slow.. I will see
>  > > if I can find which one it was at that timezone and see if I can
>  > > address.
>  > >
>  >
>  > OK the problem does not look like it is mirrorlist related but
>  > something to do with the copr host/network at that time
>  >
>  > 209.132.184.33 - - [07/Feb/2019:00:01:59 +0000] "GET
>  > /metalink?repo=updates-released-f29&arch=x86_64 HTTP/1.1" 200 18343
>  > "-" "dnf/2.7.5"
>  >
>  > says it sent the data but it looks like something that the client was
>  > looking for timed out. I will work with the copr team to diagnose
>  > further.
>
>  FWIW, openQA tests have been failing relatively frequently with
>  timeouts while refreshing repos or 503s from mirrormanager for the last
>  few weeks. I'm pretty sure this didn't used to be anything like as
>  common... 
I am checking if we are seeing an increase and when. There are
generally some every hour when mirrormanager changes out pkls but
sometimes it grows.



-- 
Stephen J Smoogen.

[Stephen John Smoogen]

On Fri, 8 Feb 2019 at 16:00, Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Fri, 8 Feb 2019 at 14:31, Adam Williamson <adamwill(a)fedoraproject.org&gt; wrote:
>  >
>  > On Thu, 2019-02-07 at 12:21 -0500, Stephen John Smoogen wrote:
>  > > On Thu, 7 Feb 2019 at 09:40, Stephen John Smoogen <smooge(a)gmail.com&gt;
> wrote:
>  > > > On Thu, 7 Feb 2019 at 09:21, Fabio Valentini <decathorpe(a)gmail.com&gt;
> wrote:
>  > > > > On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen
> <smooge(a)gmail.com&gt; wrote:
>  > > > > > On Thu, 7 Feb 2019 at 07:34, Fabio Valentini
> <decathorpe(a)gmail.com&gt; wrote:
>  > > > librepo.LibrepoException: (8, 'Cannot prepare internal mirrorlist:
>  > > > Curl error (28): Timeout was reached for
>  > > >
> https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...
>  > > > [Operation too slow. Less than 1000 bytes/sec transferred the last 30
>  > > > seconds]', 'An Curl handle error')
>  > > >
>  > > > That says the proxy it was trying to get to was too slow.. I will see
>  > > > if I can find which one it was at that timezone and see if I can
>  > > > address.
>  > > >
>  > >
>  > > OK the problem does not look like it is mirrorlist related but
>  > > something to do with the copr host/network at that time
>  > >
>  > > 209.132.184.33 - - [07/Feb/2019:00:01:59 +0000] "GET
>  > > /metalink?repo=updates-released-f29&arch=x86_64 HTTP/1.1" 200 18343
>  > > "-" "dnf/2.7.5"
>  > >
>  > > says it sent the data but it looks like something that the client was
>  > > looking for timed out. I will work with the copr team to diagnose
>  > > further.
>  >
>  > FWIW, openQA tests have been failing relatively frequently with
>  > timeouts while refreshing repos or 503s from mirrormanager for the last
>  > few weeks. I'm pretty sure this didn't used to be anything like as
>  > common...
>
>  I am checking if we are seeing an increase and when. There are
>  generally some every hour when mirrormanager changes out pkls but
>  sometimes it grows. 
OK looking at the general stats for the last 2 months, we were seeing
an average of  0.2% of all connections generating a 503 (Out of
22114356 connections on 2018/12/03 only 41263 were 503 and the rest
are 200, similar to all days). There doesn't seem to be any large
uptick in them that stands out but I am not ruling out specific
mirrors, times of requests or clients yet that could make these more
likely.




--
Stephen J Smoogen.

[Fabio Valentini]

On Sun, Feb 10, 2019 at 5:18 PM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Fri, 8 Feb 2019 at 16:00, Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  >
>  > On Fri, 8 Feb 2019 at 14:31, Adam Williamson <adamwill(a)fedoraproject.org&gt;
> wrote:
>  > >
>  > > On Thu, 2019-02-07 at 12:21 -0500, Stephen John Smoogen wrote:
>  > > > On Thu, 7 Feb 2019 at 09:40, Stephen John Smoogen <smooge(a)gmail.com&gt;
> wrote:
>  > > > > On Thu, 7 Feb 2019 at 09:21, Fabio Valentini
> <decathorpe(a)gmail.com&gt; wrote:
>  > > > > > On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen
> <smooge(a)gmail.com&gt; wrote:
>  > > > > > > On Thu, 7 Feb 2019 at 07:34, Fabio Valentini
> <decathorpe(a)gmail.com&gt; wrote:
>  > > > > librepo.LibrepoException: (8, 'Cannot prepare internal
> mirrorlist:
>  > > > > Curl error (28): Timeout was reached for
>  > > > >
> https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&...
>  > > > > [Operation too slow. Less than 1000 bytes/sec transferred the last
> 30
>  > > > > seconds]', 'An Curl handle error')
>  > > > >
>  > > > > That says the proxy it was trying to get to was too slow.. I will
> see
>  > > > > if I can find which one it was at that timezone and see if I can
>  > > > > address.
>  > > > >
>  > > >
>  > > > OK the problem does not look like it is mirrorlist related but
>  > > > something to do with the copr host/network at that time
>  > > >
>  > > > 209.132.184.33 - - [07/Feb/2019:00:01:59 +0000] "GET
>  > > > /metalink?repo=updates-released-f29&arch=x86_64 HTTP/1.1" 200
> 18343
>  > > > "-" "dnf/2.7.5"
>  > > >
>  > > > says it sent the data but it looks like something that the client was
>  > > > looking for timed out. I will work with the copr team to diagnose
>  > > > further.
>  > >
>  > > FWIW, openQA tests have been failing relatively frequently with
>  > > timeouts while refreshing repos or 503s from mirrormanager for the last
>  > > few weeks. I'm pretty sure this didn't used to be anything like as
>  > > common...
>  >
>  > I am checking if we are seeing an increase and when. There are
>  > generally some every hour when mirrormanager changes out pkls but
>  > sometimes it grows.
>
>  OK looking at the general stats for the last 2 months, we were seeing
>  an average of  0.2% of all connections generating a 503 (Out of
>  22114356 connections on 2018/12/03 only 41263 were 503 and the rest
>  are 200, similar to all days). There doesn't seem to be any large
>  uptick in them that stands out but I am not ruling out specific
>  mirrors, times of requests or clients yet that could make these more
>  likely. 
Either I am unreasonably unlucky, or your statistics are wrong:
Looking at [0], 35 of the last 44 build tasks failed due to network
issues (not counting all sub-tasks), which is about 80%.

[0]: https://copr.fedorainfracloud.org/coprs/decathorpe/elementary-nightly/bui...

Fabio

>
> --
> Stephen J Smoogen.
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Nicolas Mailhot]

Le 2019-02-11 13:51, Fabio Valentini a écrit :

>  Either I am unreasonably unlucky, or your statistics are wrong:
>  Looking at [0], 35 of the last 44 build tasks failed due to network
>  issues (not counting all sub-tasks), which is about 80%. 
You're not unreasonably unlucky, and the stats are not wrong. You're 
just not talking about the same things.

A repo sync that works will results in hundreds or more of requests just 
to get the various packages and files. So it will easily account for 
hundreds or more of successful requests in stats. OTOH a failing sync 
will account for at most 4-5 retries before dnf gives up. So the raw 
stat number is heavily skewed against counting actual copr build 
failures.

-- 
Nicolas Mailhot

[Kevin Fenzi]

On 2/11/19 5:46 AM, Nicolas Mailhot wrote:
>  Le 2019-02-11 13:51, Fabio Valentini a écrit :
>  
> > Either I am unreasonably unlucky, or your statistics are wrong:
> > Looking at [0], 35 of the last 44 build tasks failed due to network
> > issues (not counting all sub-tasks), which is about 80%.
>  
>  You're not unreasonably unlucky, and the stats are not wrong. You're
>  just not talking about the same things.
>  
>  A repo sync that works will results in hundreds or more of requests just
>  to get the various packages and files. So it will easily account for
>  hundreds or more of successful requests in stats. OTOH a failing sync
>  will account for at most 4-5 retries before dnf gives up. So the raw
>  stat number is heavily skewed against counting actual copr build failures. 
This is made worse by:

https://bugzilla.redhat.com/show_bug.cgi?id=1643281

basically dnf gets a mirror, downloads packages from it, but if it gets
a 503, it just retries the same mirror and gives up, it doesn't go to
the next mirror. ;(

kevin

[Dridi Boukelmoune]

>  FWIW, openQA tests have been failing relatively frequently with
>  timeouts while refreshing repos or 503s from mirrormanager for the last
>  few weeks. I'm pretty sure this didn't used to be anything like as
>  common... 
Whether it's dnf upgrade of my system or mock builds, both problems
have been increasingly frustrating me by orders of magnitude.
Especially for mock build, I had to make sure to --offline a given
build after a successful repo refresh & rpm download because it became
ridiculously hard to get mock --buildsrpm and mock --rebuild to
succeed in a row!

And my internet access has been flaky for "the last few weeks" too so
I thought it was causing the failures, but then even on a good
connection it kept happening.

Dridi

[Fabio Valentini]

On Sat, Feb 9, 2019 at 9:29 AM Dridi Boukelmoune
<dridi.boukelmoune(a)gmail.com&gt; wrote:
>  > FWIW, openQA tests have been failing relatively frequently with
>  > timeouts while refreshing repos or 503s from mirrormanager for the last
>  > few weeks. I'm pretty sure this didn't used to be anything like as
>  > common...
>
>  Whether it's dnf upgrade of my system or mock builds, both problems
>  have been increasingly frustrating me by orders of magnitude.
>  Especially for mock build, I had to make sure to --offline a given
>  build after a successful repo refresh & rpm download because it became
>  ridiculously hard to get mock --buildsrpm and mock --rebuild to
>  succeed in a row!
>
>  And my internet access has been flaky for "the last few weeks" too so
>  I thought it was causing the failures, but then even on a good
>  connection it kept happening. 
That's exactly my experience, as well.

First I've put it off as "my internet connection being unreliable".
But, since almost everything works fine, but mock builds continue to fail
(mostly because of "Failed to synchronize cache for repo 'foo'", where
foo is either fedora, updates, or rawhide),
I now think this really is some infra problem.

Also, I think some of the problems that happen in COPR are caused by
its very burst-y behavior:

https://copr.fedorainfracloud.org/status/stats/

For example, today, some of my builds were pending for over 7 hours,
during which time almost no running build completed, but pending tasks
blew up from the average 0-10 to over 140, since no pending tasks
moved to running at all, during that time, from what I can tell.
Then, a lot of builds (about 35) were moved to the "running" state at
once, and some of them failed with connectivity issues, again - which
doesn't surprise me, when there are possibly hundreds of mock builds
triggered by COPR at about the same time, obviously overwhelming the
system serving the repositories.

Fabio


> Dridi
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Kevin Fenzi]

On 2/10/19 3:09 AM, Fabio Valentini wrote:
>  
>  That's exactly my experience, as well.
>  
>  First I've put it off as "my internet connection being unreliable".
>  But, since almost everything works fine, but mock builds continue to fail
>  (mostly because of "Failed to synchronize cache for repo 'foo'", where
>  foo is either fedora, updates, or rawhide),
>  I now think this really is some infra problem.
>  
>  Also, I think some of the problems that happen in COPR are caused by
>  its very burst-y behavior:
>  
>  https://copr.fedorainfracloud.org/status/stats/
>  
>  For example, today, some of my builds were pending for over 7 hours,
>  during which time almost no running build completed, but pending tasks
>  blew up from the average 0-10 to over 140, since no pending tasks
>  moved to running at all, during that time, from what I can tell.
>  Then, a lot of builds (about 35) were moved to the "running" state at
>  once, and some of them failed with connectivity issues, again - which
>  doesn't surprise me, when there are possibly hundreds of mock builds
>  triggered by COPR at about the same time, obviously overwhelming the
>  system serving the repositories. 
Well, that was caused by copr backend becoming unresponsive.

So, no jobs were processed until a copr maintainer was able to reboot it.

The repo issues seem to be our mirrorlist containers sometimes throwing
a 503 when they shouldn't. ;( This is made worse by dnf never retrying
them, so if it happens once thats it. We are investigating and trying to
fix whatever is causing this.

kevin

[Adam Williamson]

On Mon, 2019-02-11 at 10:23 -0800, Kevin Fenzi wrote:
>  The repo issues seem to be our mirrorlist containers sometimes
> throwing
>  a 503 when they shouldn't. ;( This is made worse by dnf never retrying
>  them, so if it happens once thats it. We are investigating and trying to
>  fix whatever is causing this. 
Note for the record this is causing spurious openQA test failures too,
so it'd be really nice if it could be fixed!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Kevin Fenzi]

On 2/11/19 10:26 AM, Adam Williamson wrote:
>  On Mon, 2019-02-11 at 10:23 -0800, Kevin Fenzi wrote:
> > The repo issues seem to be our mirrorlist containers sometimes throwing
> > a 503 when they shouldn't. ;( This is made worse by dnf never retrying
> > them, so if it happens once thats it. We are investigating and trying to
> > fix whatever is causing this.
>  
>  Note for the record this is causing spurious openQA test failures too,
>  so it'd be really nice if it could be fixed!
>   
I've filed
https://pagure.io/fedora-infrastructure/issue/7553
to track this.

kevin

[Dridi Boukelmoune]

>  I've filed
>  https://pagure.io/fedora-infrastructure/issue/7553
>  to track this. 
FWIW: https://github.com/rpm-software-management/dnf/pull/1109

[Nicolas Mailhot]

Le jeudi 31 janvier 2019 à 19:52 -0500, Neal Gompa a écrit :
>  On Thu, Jan 31, 2019 at 6:58 PM Stephen John Smoogen
> <smooge(a)gmail.com
>  > wrote:
>  > On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <
>  > ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  > > Problem №1: Build-only packages
>  > > 
>  > > Rawhide gating makes this much more complicated because builds
>  > > appear in buildroot slower, updating group of packages would need
>  > > side tags and it’s just painful to work with.
>  > > 
>  > > https://pagure.io/fesco/issue/2068
>  > > 
>  > > And, after all, those packages shouldn’t be shipped to users.
>  > 
>  > My main problem with this is the above. Yes Joe Desktop User isn't
>  > going to see/need that package.. but we have a LOT of packagers who
>  > take our stuff and rebuild it for themselves for various reasons. I
>  > find it hard to put together the proposal which is supposed to make
>  > developing/packaging easier with making developing/packaging harder.
>  > Whether you want it to or not, this comes across as "If you want to
>  > use Go or Rust, you will need our special set of tools which we keep
>  > hidden."
>  > 
>  
>  This is actually something I really don't like either. But the Fedora
>  leadership has pushed very hard on the concept of having packages that
>  aren't available to "normies", and require special tooling to be able
>  to leverage (that for various reasons, I can't even use as a third
>  party packager!).  
If MBI is about hiding build packages I don't see the point and I'm not
interested.

What I *am* interested in is being able to import in one go the
hundred(s) of srpms involved in building the complex highly modularized
stuff package projects output nowadays, without the two months of review
getting each spec audited separately would entail.

Our main infra and main administrative processes just don't scale.

-- 
Nicolas Mailhot

[Mikolaj Izdebski]

On Fri, Feb 1, 2019 at 8:46 AM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net&gt; wrote:
>  If MBI is about hiding build packages I don't see the point and
> I'm not
>  interested. 
It's the opposite - it's about opening build process to others and
allowing for collaboration. Instead of doing development by yourself
in private, you use a public, shared environment. But if you are not
interested then no one is forcing you to use it, you can keep
developing packages your current way.

--
Mikolaj

[Neal Gompa]

On Fri, Feb 1, 2019 at 6:59 AM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net&gt; wrote:
>  Le jeudi 31 janvier 2019 à 19:52 -0500, Neal Gompa a écrit :
>  > On Thu, Jan 31, 2019 at 6:58 PM Stephen John Smoogen <smooge(a)gmail.com
>  > > wrote:
>  > > On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <
>  > > ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  > > > Problem №1: Build-only packages
>  > > >
>  > > > Rawhide gating makes this much more complicated because builds
>  > > > appear in buildroot slower, updating group of packages would need
>  > > > side tags and it’s just painful to work with.
>  > > >
>  > > > https://pagure.io/fesco/issue/2068
>  > > >
>  > > > And, after all, those packages shouldn’t be shipped to users.
>  > >
>  > > My main problem with this is the above. Yes Joe Desktop User isn't
>  > > going to see/need that package.. but we have a LOT of packagers who
>  > > take our stuff and rebuild it for themselves for various reasons. I
>  > > find it hard to put together the proposal which is supposed to make
>  > > developing/packaging easier with making developing/packaging harder.
>  > > Whether you want it to or not, this comes across as "If you want to
>  > > use Go or Rust, you will need our special set of tools which we keep
>  > > hidden."
>  > >
>  >
>  > This is actually something I really don't like either. But the Fedora
>  > leadership has pushed very hard on the concept of having packages that
>  > aren't available to "normies", and require special tooling to be able
>  > to leverage (that for various reasons, I can't even use as a third
>  > party packager!).
>
>  If MBI is about hiding build packages I don't see the point and I'm not
>  interested.
>  
MBI isn't about that. That's what modularity is (partly) about.

>  What I *am* interested in is being able to import in one go the
>  hundred(s) of srpms involved in building the complex highly modularized
>  stuff package projects output nowadays, without the two months of review
>  getting each spec audited separately would entail.
>
>  Our main infra and main administrative processes just don't scale.
>  
Indeed. Those were ideas that we put into the proposal of things we
want to improve.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Stephen John Smoogen]

On Fri, 1 Feb 2019 at 03:26, Nicolas Mailhot <nicolas.mailhot(a)laposte.net&gt;
wrote:

>  Le jeudi 31 janvier 2019 à 19:52 -0500, Neal Gompa a écrit :
>  > On Thu, Jan 31, 2019 at 6:58 PM Stephen John Smoogen <smooge(a)gmail.com
>  > > wrote:
>  > > On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <
>  > > ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  > > > Problem №1: Build-only packages
>  > > >
>  > > > Rawhide gating makes this much more complicated because builds
>  > > > appear in buildroot slower, updating group of packages would need
>  > > > side tags and it’s just painful to work with.
>  > > >
>  > > > https://pagure.io/fesco/issue/2068
>  > > >
>  > > > And, after all, those packages shouldn’t be shipped to users.
>  > >
>  > > My main problem with this is the above. Yes Joe Desktop User isn't
>  > > going to see/need that package.. but we have a LOT of packagers who
>  > > take our stuff and rebuild it for themselves for various reasons. I
>  > > find it hard to put together the proposal which is supposed to make
>  > > developing/packaging easier with making developing/packaging harder.
>  > > Whether you want it to or not, this comes across as "If you want to
>  > > use Go or Rust, you will need our special set of tools which we keep
>  > > hidden."
>  > >
>  >
>  > This is actually something I really don't like either. But the Fedora
>  > leadership has pushed very hard on the concept of having packages that
>  > aren't available to "normies", and require special tooling to be able
>  > to leverage (that for various reasons, I can't even use as a third
>  > party packager!).
>
>  If MBI is about hiding build packages I don't see the point and I'm not
>  interested.
>
>  
Actually I don't think it is part of MBI but comes up in some Rust/Go
packages which do regular chain-bootstrapping to be bleeding edge. I will
let someone more versed in it speak on the specifics as I usually see red
on hidden packages and am not a disinterested party.


>  
-- 
Stephen J Smoogen.

[Josh Boyer]

On Thu, Jan 31, 2019 at 8:19 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Thu, Jan 31, 2019 at 6:58 PM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  >
>  > On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <ignatenkobrain(a)fedoraproject.org&gt;
> wrote:
>  >>
>  >> Problem №1: Build-only packages
>  >>
>  >> Rawhide gating makes this much more complicated because builds appear in
> buildroot slower, updating group of packages would need side tags and it’s just painful to
> work with.
>  >>
>  >> https://pagure.io/fesco/issue/2068
>  >>
>  >> And, after all, those packages shouldn’t be shipped to users.
>  >
>  >
>  > My main problem with this is the above. Yes Joe Desktop User isn't going to
> see/need that package.. but we have a LOT of packagers who take our stuff and rebuild it
> for themselves for various reasons. I find it hard to put together the proposal which is
> supposed to make developing/packaging easier with making developing/packaging harder.
> Whether you want it to or not, this comes across as "If you want to use Go or Rust,
> you will need our special set of tools which we keep hidden."
>  >
>
>  This is actually something I really don't like either. But the Fedora
>  leadership has pushed very hard on the concept of having packages that
>  aren't available to "normies", and require special tooling to be able
>  to leverage (that for various reasons, I can't even use as a third
>  party packager!). That's a big part of the core to how Modularity is
>  being used. The CRB in RHEL 8 is another example of this. But fixing 
I'd like to understand why you see RHEL 8 Code Ready Linux Builder as
an example of what you're talking about, but I think that's for a
different email.

>  this is very hard when few others see it as a problem. There was
> even
>  talk of not publishing SRPMs of packages that make up modules a while
>  ago. That idea died very quickly thankfully, but I don't know how
>  people think of these things anymore.
>
>  I'm tired of fighting... Our tools haven't improved enough to handle
>  our new challenges in the Fedora way, because it's hard for people in
>  the community to experiment and explore in this manner. My hope is
>  that with something like MBI, we can finally put the power to
>  experiment and develop tooling that actually makes sense for Fedora
>  (rather than clearly being designed for RHEL and shoehorned into
>  Fedora) without all the pain and agony of people imposing RHELish
>  requirements. There's no way we can evolve and meet the needs of our
>  communities without some way for people to "play". 
I don't understand what is preventing anyone from building such tools
or communities.  Why can't you do what you want?

>  COPR was supposed to be that outlet, but no one gives a damn about
> it.
>  Everyone complains that the service is "bad" and that the design is
>  "bad" but no one wants to actually constructively improve it. The
>  quality of service on COPR has fallen due to lack of care and
>  unwillingness to invest, so what are we supposed to do? The horrible
>  thing is that COPR is successful *despite* that. COPR is an
>  interesting and cool service, and I would have loved to see an
>  eventual merger of the Koji and COPR systems (because there are
>  awesome attributes to both and we should have a large and strong team
>  actually supporting the literal core of the distribution), but it
>  won't happen because everyone thinks the COPR system is terrible even
>  though it's not really.
>
>  I've been arguing for literally years about our shortcomings. I've
>  built tools for working around pitfalls in the Fedora workflow for my
>  personal use. I've personally explored how other people and other
>  distributions do things. I got my hands dirty to learn about other
>  ways of doing things. I've used that knowledge and expertise to avoid
>  those mistakes when deploying package and image construction
>  infrastructure for $DAYJOB. But it deeply saddens me that I have not
>  been able to make any meaningful progress in convincing anyone that it
>  was something that we should invest in. 
You've used the word "invest" twice in your email, so perhaps the
answer to my above question is "because I/we don't have enough funds
and/or time to make it happen"?  If that's the case, and I'm thinking
it likely is, then I have no good answer for you.  People choose to
invest their money and time in the things they are interested in or
see long-term value out of.  Corporations (which are not people)
choose to invest resources (machine, monetary, human) in things they
see as providing value to their long-term business interests.

It can be frustrating to not get that critical mass needed to move an
idea forward.  I'm in a similar boat with many (minor) things in
Fedora right now.  I can empathize with you on that front.

>  I still believe that Modularity, as designed, is a mistake. But if I
>  don't have a way to prove a better way to do things, there's no way
>  anything will improve. The folks running the show like what they have
>  now, so it'll take a lot to trigger change. 
Here is my issue with many of the ideas/proposals I've seen lately in
Fedora.  Someone describes a problem (good!), they come up with a plan
(ok!), that plan immediately requires a significant investment in
resources and time of *other people*.  There is no evolution from
existing tools, there is no small prototype they've done on the side
to prove things out or tackle some of the issues.  There's no evidence
the idea is even going to pay off at all.  It's an immediate "here's
my idea!" with a hand out to get started.

Fedora is a project, not a venture capital firm.  Things get done
because people do them, prove they work, get buy-in, and evolve over
time.  If that means ideas or projects start small and outside
existing Fedora infrastructure, that's OK!

For the record, I think Modularity started much the same way.
"Everything will be a module!" without proving any of it out in a
small scale hurt the adoption and messaging around why it's useful.
When it was reset and rescoped, it started making more progress and
continues to gain steam.  I am not blind to the fact that Modularity
did not suffer from lack of investment, but it should serve to
illustrate that even if you have a corporate backer your plan or idea
still might not succeed at first and will need to evolve over time.

josh

[Nicolas Mailhot]

Le 2019-02-01 13:17, Josh Boyer a écrit :

>  Fedora is a project, not a venture capital firm.  Things get done
>  because people do them, prove they work, get buy-in, and evolve over
>  time.  If that means ideas or projects start small and outside
>  existing Fedora infrastructure, that's OK! 
I think there is a misunderstanding of what the "Friends" Fedora value 
actually means. A lot of @RH contributors are developers at core and 
only see Friends in a dev context: sharing source code with others, and 
to hell with distro packages, and Fedora as a binary package 
distribution in general.

But, there are already lots of instances where you can collaborate on 
code (the Apache Foundation, etc). Fedora isn't one of those no matter 
how devs push for it to become one (sometimes our desktop friends seem 
to see Fedora as some form of GNOME Foundation sugar daddy nothing 
more).

What "Friends" mean in a Fedora context, is being to collaborate on 
binary packages (rpms), and a best of breed integrated system, no matter 
what your team size is, no matter if you do it as a hobbyist or as part 
of some job, no matter what level of the software stack you're 
interested in.

Whenever Fedora chooses to deploy infra solutions that are unmanageable 
except @RH, it's hurting its core Friends value. Whenever Fedora 
separates its offerings in special Editions (basically declaring "we 
don't care about those other use cases") it is hurting its Friends 
value. Whenever Fedora invests in deployment solutions, which can only 
be applied on the desktop, only applied on openshift, or whatever, it is 
hurting its Friends value. Whenever the Fedora main sponsor, chooses to 
segment EL in unmanageable optional channels and modules that require 
some friendly commercial help to be navigated, it's hurting its 
"Friends" value (and it *is* hurting Fedora when EPEL rules are chosen 
to accommodate unfriendly @RH policy).

And, why am I writing all this? Because the old proprietary behemoths 
like Oracle, that justified investing in the Fedora/Centos/RHEL 
ecosystem, are slowly fading away.

That is going to make Debian a direct Fedora competitor to attract 
contributions and getting things done, when before anyone saddled with 
Oracle or SAP or whatever wouldn't even look at Debian. And, because 
Debian is a competent distribution, and because Debian has the same 
software reach (or more) than Fedora, a huge part of the choice is going 
to be made on the Friends axis.

So yes Fedora is not a venture capital firm. That does not mean it is 
not competing. It's just competing for contributions, not money.

Thus stating clearly how initiatives like MBI, contribute to "Friends" 
excellence, is not some form of weird request, it's a survival 
requirement for Fedora as a project. That is, as long as Friends is 
supposed to be a core value and a core differentiator of the Fedora 
proposition.

-- 
Nicolas Mailhot

[Josh Boyer]

On Fri, Feb 1, 2019 at 9:00 AM Nicolas Mailhot
<nicolas.mailhot(a)laposte.net&gt; wrote:
>  Le 2019-02-01 13:17, Josh Boyer a écrit :
>
>  > Fedora is a project, not a venture capital firm.  Things get done
>  > because people do them, prove they work, get buy-in, and evolve over
>  > time.  If that means ideas or projects start small and outside
>  > existing Fedora infrastructure, that's OK!
>
>  I think there is a misunderstanding of what the "Friends" Fedora value
>  actually means. A lot of @RH contributors are developers at core and
>  only see Friends in a dev context: sharing source code with others, and
>  to hell with distro packages, and Fedora as a binary package
>  distribution in general.
>
>  But, there are already lots of instances where you can collaborate on
>  code (the Apache Foundation, etc). Fedora isn't one of those no matter
>  how devs push for it to become one (sometimes our desktop friends seem
>  to see Fedora as some form of GNOME Foundation sugar daddy nothing
>  more). 
I agree.  Fedora is not a typical developer project.  It is a
distribution project.

>  What "Friends" mean in a Fedora context, is being to
> collaborate on
>  binary packages (rpms), and a best of breed integrated system, no matter
>  what your team size is, no matter if you do it as a hobbyist or as part
>  of some job, no matter what level of the software stack you're
>  interested in. 
Maybe so.  I'm not convinced that's really what "Friends" means, but
the sentiment around collaboration isn't really wrong.

>  Whenever Fedora chooses to deploy infra solutions that are
> unmanageable
>  except @RH, it's hurting its core Friends value. Whenever Fedora
>  separates its offerings in special Editions (basically declaring "we
>  don't care about those other use cases") it is hurting its Friends
>  value. Whenever Fedora invests in deployment solutions, which can only 
I disagree that is what Editions is doing.

>  be applied on the desktop, only applied on openshift, or whatever, it
> is
>  hurting its Friends value. Whenever the Fedora main sponsor, chooses to
>  segment EL in unmanageable optional channels and modules that require
>  some friendly commercial help to be navigated, it's hurting its
>  "Friends" value (and it *is* hurting Fedora when EPEL rules are chosen
>  to accommodate unfriendly @RH policy). 
I can't comment on EPEL.  I'm only a user, not a contributor.  It
works well for my uses though.

>  And, why am I writing all this? Because the old proprietary
> behemoths
>  like Oracle, that justified investing in the Fedora/Centos/RHEL
>  ecosystem, are slowly fading away. 
Really?  Can you illustrate that with data?  I don't mean to be
argumentative, but there is a very old, large, and blue behemoth that
just decided to plunk down a $34B investment on all of those things.

>  That is going to make Debian a direct Fedora competitor to attract
>  contributions and getting things done, when before anyone saddled with
>  Oracle or SAP or whatever wouldn't even look at Debian. And, because
>  Debian is a competent distribution, and because Debian has the same
>  software reach (or more) than Fedora, a huge part of the choice is going
>  to be made on the Friends axis. 
"going to"?  If you believe that Fedora and Debian are not *already*
in competition for contributions with other distros or projects then
I'm somewhat confused.

>  So yes Fedora is not a venture capital firm. That does not mean it
> is
>  not competing. It's just competing for contributions, not money. 
OK?  I'm missing your point on this one.

>  Thus stating clearly how initiatives like MBI, contribute to
> "Friends"
>  excellence, is not some form of weird request, it's a survival
>  requirement for Fedora as a project. That is, as long as Friends is
>  supposed to be a core value and a core differentiator of the Fedora
>  proposition. 
I didn't say it was a weird request.  I didn't even say it was a bad
idea!  I said it's unproven and requires *further* investment on the
part of the project as a whole, which seems premature.

I can appreciate your perspective and focus on Friends to argue your
position, but it can easily be turned around as well.  I would rather
see things mature in Fedora on technical merit and not because of
pitch around a social construct that frankly doesn't exist.  The
Fedora project is friendly and welcoming and respectful.  That doesn't
mean we're all friends nor does it mean that everyone's ideas get
equal investment.

josh

[Neal Gompa]

On Sat, Feb 2, 2019 at 10:37 AM Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  On Thu, Jan 31, 2019 at 8:19 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > On Thu, Jan 31, 2019 at 6:58 PM Stephen John Smoogen <smooge(a)gmail.com&gt;
> wrote:
>  > >
>  > > On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko
> <ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  > >>
>  > >> Problem №1: Build-only packages
>  > >>
>  > >> Rawhide gating makes this much more complicated because builds appear in
> buildroot slower, updating group of packages would need side tags and it’s just painful to
> work with.
>  > >>
>  > >> https://pagure.io/fesco/issue/2068
>  > >>
>  > >> And, after all, those packages shouldn’t be shipped to users.
>  > >
>  > >
>  > > My main problem with this is the above. Yes Joe Desktop User isn't going to
> see/need that package.. but we have a LOT of packagers who take our stuff and rebuild it
> for themselves for various reasons. I find it hard to put together the proposal which is
> supposed to make developing/packaging easier with making developing/packaging harder.
> Whether you want it to or not, this comes across as "If you want to use Go or Rust,
> you will need our special set of tools which we keep hidden."
>  > >
>  >
>  > This is actually something I really don't like either. But the Fedora
>  > leadership has pushed very hard on the concept of having packages that
>  > aren't available to "normies", and require special tooling to be able
>  > to leverage (that for various reasons, I can't even use as a third
>  > party packager!). That's a big part of the core to how Modularity is
>  > being used. The CRB in RHEL 8 is another example of this. But fixing
>
>  I'd like to understand why you see RHEL 8 Code Ready Linux Builder as
>  an example of what you're talking about, but I think that's for a
>  different email.
>  
The problem is that it's a symptom of a greater issue, which is that
RHEL is no longer self-consistent as a distribution. Wiring up RHEL 8
into my build process is functionally impossible without
rebootstrapping and building all the module components as "normal"
packages. But there are a class of packages that simply don't exist in
any form: packages that are used exclusively build-time. We started
seeing this problem come about with RHEL 7 over time as CentOS 7
rebuilt new point releases. RHEL 8 takes that to a whole new level.
And the CRB wasn't even published as part of the RHEL 8 beta public
repos, which made it impossible to build anything using it.

Fedora is starting to do the same thing. One of the major points of
contention I've had with Igor in the Rust SIG was the choice to stop
publishing Rust crate packages that were used to build applications
themselves when we switched to modules. It's essentially vendoring all
in but name. But as we've talked about it, I realized that we
basically have no choice, because the people in control of the tooling
have not been receptive to caring about our problems and working with
us to improve it.

But it means that downstream people cannot easily take our stuff,
modify it, rebuild it, etc.

That's a huge loss from my point of view.

>  > this is very hard when few others see it as a problem. There was
> even
>  > talk of not publishing SRPMs of packages that make up modules a while
>  > ago. That idea died very quickly thankfully, but I don't know how
>  > people think of these things anymore.
>  >
>  > I'm tired of fighting... Our tools haven't improved enough to handle
>  > our new challenges in the Fedora way, because it's hard for people in
>  > the community to experiment and explore in this manner. My hope is
>  > that with something like MBI, we can finally put the power to
>  > experiment and develop tooling that actually makes sense for Fedora
>  > (rather than clearly being designed for RHEL and shoehorned into
>  > Fedora) without all the pain and agony of people imposing RHELish
>  > requirements. There's no way we can evolve and meet the needs of our
>  > communities without some way for people to "play".
>
>  I don't understand what is preventing anyone from building such tools
>  or communities.  Why can't you do what you want?
>  
Right now, it doesn't matter what I want to do. I don't have the
resources to host infrastructure and I don't have the buy-in from
people to consider new ideas that *I want to implement*.

>  > COPR was supposed to be that outlet, but no one gives a damn
> about it.
>  > Everyone complains that the service is "bad" and that the design is
>  > "bad" but no one wants to actually constructively improve it. The
>  > quality of service on COPR has fallen due to lack of care and
>  > unwillingness to invest, so what are we supposed to do? The horrible
>  > thing is that COPR is successful *despite* that. COPR is an
>  > interesting and cool service, and I would have loved to see an
>  > eventual merger of the Koji and COPR systems (because there are
>  > awesome attributes to both and we should have a large and strong team
>  > actually supporting the literal core of the distribution), but it
>  > won't happen because everyone thinks the COPR system is terrible even
>  > though it's not really.
>  >
>  > I've been arguing for literally years about our shortcomings. I've
>  > built tools for working around pitfalls in the Fedora workflow for my
>  > personal use. I've personally explored how other people and other
>  > distributions do things. I got my hands dirty to learn about other
>  > ways of doing things. I've used that knowledge and expertise to avoid
>  > those mistakes when deploying package and image construction
>  > infrastructure for $DAYJOB. But it deeply saddens me that I have not
>  > been able to make any meaningful progress in convincing anyone that it
>  > was something that we should invest in.
>
>  You've used the word "invest" twice in your email, so perhaps the
>  answer to my above question is "because I/we don't have enough funds
>  and/or time to make it happen"?  If that's the case, and I'm thinking
>  it likely is, then I have no good answer for you.  People choose to
>  invest their money and time in the things they are interested in or
>  see long-term value out of.  Corporations (which are not people)
>  choose to invest resources (machine, monetary, human) in things they
>  see as providing value to their long-term business interests.
>  
While money is helpful, the real problem is that people don't see it
as a priority to work with the community to resolve these issues. I'm
happy to help with doing the work, but I can't do it alone. This is
one of those things I *literally* cannot do alone because I don't have
the power, resources, or time to do alone.

But as an example of something for COPR that I'd been working on is an
autorebuilder service that is dependency-aware[1]. Basically, when the
repositories configured for your COPR project update, it would trigger
rebuilds of your packages in the correct order. This is very similar
to the build scheduler present in the Open Build Service. It wouldn't
require anything more than you to configure the rebuilder service on
which projects and packages to watch, and an API key for COPR.

Ideally, such a thing would be integrated into COPR itself and done
with a bit more finesse, but I wanted to build this out to support my
own use-case: rebuilding kmod packages automatically when new kernels
became available. But the idea would be trivial to extend to anything.

I hadn't gotten the initial code working yet, so there's only a
skeleton pushed, but it's something I have been hacking on
periodically.

[1]: https://pagure.io/copr-autorebuilder

>  It can be frustrating to not get that critical mass needed to move
> an
>  idea forward.  I'm in a similar boat with many (minor) things in
>  Fedora right now.  I can empathize with you on that front.
>
>  > I still believe that Modularity, as designed, is a mistake. But if I
>  > don't have a way to prove a better way to do things, there's no way
>  > anything will improve. The folks running the show like what they have
>  > now, so it'll take a lot to trigger change.
>
>  Here is my issue with many of the ideas/proposals I've seen lately in
>  Fedora.  Someone describes a problem (good!), they come up with a plan
>  (ok!), that plan immediately requires a significant investment in
>  resources and time of *other people*.  There is no evolution from
>  existing tools, there is no small prototype they've done on the side
>  to prove things out or tackle some of the issues.  There's no evidence
>  the idea is even going to pay off at all.  It's an immediate "here's
>  my idea!" with a hand out to get started.
>  
That has *never* been the way I roll with things. It doesn't take much
to see that I'm fairly involved in a wide variety of projects.
However, an important litmus for me is that the people in control of
the project have to be receptive and willing to work with me. I do not
have the mental bandwidth or time to sometimes carry out an idea to
its full conclusion. I can build small-scale things to prove out the
concept, and maybe help with scaling the idea up.

But this is not my job, this is my passion I work on in nights and
weekends. If I do it alone, it'll take a long time to get there. I
*know* I need help, and I'm happy to accept it. Perversely, it also
means if there's a fundamental disagreement that can't be worked past,
I probably am more likely to quit rather than fork the project and
keep going with it. I just simply don't have the ability to drive
something alone.

>  Fedora is a project, not a venture capital firm.  Things get done
>  because people do them, prove they work, get buy-in, and evolve over
>  time.  If that means ideas or projects start small and outside
>  existing Fedora infrastructure, that's OK!
>
>  For the record, I think Modularity started much the same way.
>  "Everything will be a module!" without proving any of it out in a
>  small scale hurt the adoption and messaging around why it's useful.
>  When it was reset and rescoped, it started making more progress and
>  continues to gain steam.  I am not blind to the fact that Modularity
>  did not suffer from lack of investment, but it should serve to
>  illustrate that even if you have a corporate backer your plan or idea
>  still might not succeed at first and will need to evolve over time.
>  
The problem with Modularity today is that it's still working from
several biases that will hurt the overall Fedora ecosystem. It assumes
people don't want to do local builds of packages reliably. It assumes
that everyone has a Koji with an MBS scheduler. It assumes that Mock
is used build-side and DNF is used for runtime. It assumes you never
use PackageKit. It does not even account for ecosystem assumptions
about repository metadata, which is extremely painful for me
personally.

Many of the improvements in MBS should not be module-exclusive. They
should be part of Koji and usable by regular packages too. But there's
no opportunity because the Koji team is incredibly unresponsive. To
the point that even basic, trivial patches go without response for
months. And doing PR reviews for them doesn't seem to help either.

I'm incredibly frustrated and dismayed because it's Fedora's (and
maybe even Red Hat's) own fault that our tooling is in such poor
shape. To me, it seems like we don't actually want to be better.

I haven't given up yet because I care very deeply about Fedora as a
whole. The Project, the distribution, and the people. I want us to be
the best we can be.

I just don't know what to do anymore...



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Miroslav Suchý]

Dne 31. 01. 19 v 13:24 Igor Gnatenko napsal(a):
>    *
>  
>      COPR has been starved of resources for years, which has impaired its ability to
> provide reliable service at scale.
>      Fedora Infrastructure refuses to consider supporting it officially and Fedora Release
> Engineering seems to have some
>      undefined issues with COPR. 
That is because Copr has been always more popular than we had had resources. We are adding
more resources over time, but
we are a bit behind Copr's popularity.
If you want to put your service into Fedora Cloud than you are talking about the same
(shared) resource as Copr has.

>      It is not official build system of Fedora which is not helping
> with Problem №2: Testing of new rpm/koji/mock
>      features/configuration
>     
> <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...;.
1) First we are working on making it officially supported.
2) When I released new version of Mock, then Copr is usually the first build system to
use. And we always use the latest
version. Compare that to Koji, which until recently used very old version of Mock. This
changed few weeks ago.
But I generally try to dogfood Mock in Copr first so I hit potential issues before it can
hit Koji, which is more important.
3) I do not think that allowing completely custom Mock configs in any build system is wise
solution.

>          No query API (e.g. it is not possible to find out from which
> SCM commit the package has been built) 
Every time, we got a reasonable request backed by good user-story we implemented it in
reasonable time frame.
Feel free to submit an issue to our Pagure.

>          GC doesn’t exist 
There is a garbage collection. Albeit it remove just the build artifacts, not the DB
entries, which I seen as advantage
in past. Most people does not share the same opinion, so we are working right now on
option to remove the DB entries as
well so it does not poulute WebUI.
At the same time we are finishing with GC of outdated chroots, which should allow us to
reclaim huge amount of storage.
Expect an announce about it pretty soon.

>        o
>  
>          No scratch builds 
Yes. Because everything in Copr is basicaly a scratch build. You can delete the build. Or
you can crate new project
which will use the first project. And after a build you can delete the new project. It is
easy and very cheap.

Miroslav

[Dan Čermák]

This idea sounds pretty good to me.

As a less active contributor: How can I help out? Do you have a task
list? (Sorry if this got already answered in one of the follow up
emails.)


Cheers,

Dan

Igor Gnatenko <ignatenkobrain(a)fedoraproject.org&gt; writes:

> MayBe I …(can do something useful)?
>
> Hello,
>
> We've been discussing some (hopefully) nice idea with Mikolaj, Neal and
> Jakub how we could improve packager (and user) experience and we have some
> proposal which will be described below.
>
> We would like to ask you to read it, understand it and ask us any questions
> you have. From the Fedora Infrastructure we would like to ask for some
> machines to implement this idea (you can find some more information in
> "Implementation details" part).
>
> I would like to apologize for HTML email, but it is much easier to have it
> that way to have better visibility and reading easiness.
>
> Feel free to reply to this email or comment in google doc (there is a link
> on the bottom).
> Proposal Owners
>
>    -
>
>    Mikolaj Izdebski (mizdebsk) <mizdebsk(a)redhat.com&gt; - Java SIG, Fedora
>    infrastructure
>    -
>
>    Igor Gnatenko (ignatenkobrain) <ignatenkobrain(a)fedoraproject.org&gt; - Rust
>    SIG, Golang SIG, Neuro SIG, RPM and libsolv contributor
>    -
>
>    Neal Gompa (ngompa) <ngompa13(a)gmail.com&gt; - Rust SIG, Golang SIG, RPM
>    contributor
>    -
>
>    Jakub Čajka (jcajka) <jcajka(a)fedoraproject.org&gt; - Golang SIG, Container
>    SIG
>
>
> This proposal aligns with the objective of improving the packager experience
> <https://fedoraproject.org/wiki/Objectives/Packager_Experience> by
> developing a platform whereby the proposal owners and others can experiment
> with improvements that can be funneled back into the official production
> infrastructure.
> ProblemsProblem №1: Build-only packages
>
> Some ecosystems have many build-only packages (packages which are used to
> build other packages, without having them installed on end-user systems).
> Those ecosystems include Java, Rust and Go.
>
> It is extremely hard to keep up maintaining them due to lack of
> time/people. Upstreams are usually changing quickly (sometimes when you
> update just one such package, you’d need to update tens of the
> dependencies). Few facts about such packages:
>
>    -
>
>    They are almost always outdated in released versions of distribution;
>    -
>
>    They are often FTBFS (e.g. because there was compiler update but not
>    package update, broken deps, broken APIs due to deps rebases, …).
>
>
> In Rust ecosystem, we abuse Rawhide to build and store such packages there
> and then generate end-user application (e.g. ripgrep) which uses some of
> those packages and produces binary for all supported releases. Those
> applications have high quality and are supported.
>
> Rawhide gating makes this much more complicated because builds appear in
> buildroot slower, updating group of packages would need side tags and it’s
> just painful to work with.
>
> https://pagure.io/fesco/issue/2068
>
> And, after all, those packages shouldn’t be shipped to users.
> Problem №2: Testing of new rpm/koji/mock features/configuration
>
> When developing new features in RPM (e.g. rich dependencies) or testing
> different Koji configuration (e.g. removing gcc/gcc-c++ from the
> buildroot), it is required to make custom configuration and try building
> things.
> Problem №3: Developing modules
>
> Modules are built in MBS as a single unit. It is hard to develop big
> modules by progressively improving individual packages or small package
> groups. Scratch builds for modules are not implemented. Local builds work
> differently from official module builds, they don’t scale and don’t allow
> groups of people to work collaboratively. All these problems can be solved
> by first developing a flat repository of packages in a shared environment
> and then generating modulemd from such package set.
> Problem №4: Testing things before push
>
> Primary Fedora Koji and dist-git are not suited for packaging
> experimentation. Packagers are expected to experiment on their own systems
> and push changes of successful experiments only. But this approach doesn’t
> scale with number of maintained packages. Often you can find commits like
> “d’oh, forgot to upload sources” or “let’s try with this settings”. People
> need to build things somewhere quickly, do development and testing. And
> only after that, they should push commit(s) to Fedora.
> Solution
>
>    -
>
>    Separate Koji + Koschei deployed in Fedora infrastructure cloud;
>    -
>
>    Builders are optimized for the best performance (see below
>   
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
>    );
>    -
>
>    People can have their own targets where they can break things as they
>    wish without affecting others;
>    -
>
>    Package changes are eventually contributed to Fedora proper by merging
>    changes to Fedora dist-git and rebuilding packages in official Fedora Koji;
>    -
>
>    All improvements can eventually be contributed back to official Fedora
>    infra.
>
> Ideas
>
> All ideas which you’ll find below are just an ideas and do not have to be
> implemented.
> Idea №1: Automated legal review
>
> openSUSE released their review app called Cavil
> <https://github.com/openSUSE/cavil> which we could try running in automated
> way.
> Idea №2: Automated package review
>
> Currently review process is burden and we could run automated legal review,
> create a repo, run set of checks and notify person. Somewhat similar to
> fresque <https://github.com/fedora-infra/fresque>;. In future might be
> integrated with approval process and auto-imported into Fedora.
> Idea №3: Building packages for multiple distributions
>
> In Rust ecosystem, we managed to get have same packaging guidelines in
> openSUSE, Mageia and Fedora. We could automatically build some packages on
> multiple distributions and be kinda upstream.
> Idea №4: Building custom images with user content
>
> Deploy (or build) a tool(s) to enable user-built install, appliance and
> container images with their content (modulo restrictions similar to COPR)
> on top of Fedora.
> Implementation details
>
>    -
>
>    Koji and Koschei deployed in fedorainfracloud.org
>    -
>
>    A few builders constantly running, with a possibility to add more
>    builders as needed and as available cloud resources allow
>    -
>
>    Deployed and maintained by proposal owners - not supported by Fedora
>    infrastructure
>    -
>
>    Other contributors can have access granted by proposal owners (for the
>    time being only users in “packagers” group will be eligible to get access)
>    -
>
>    Possibility to have some builders running outsides of Fedora
>    infrastructure - contributed by proposal owners
>    -
>
>    Koji builds can be done from SCM (src.fp.o, pagure.io, github, …) or
>    from SRPM upload
>
> FAQWhy not COPR?
>
>    -
>
>    COPR has been starved of resources for years, which has impaired its
>    ability to provide reliable service at scale. Fedora Infrastructure refuses
>    to consider supporting it officially and Fedora Release Engineering seems
>    to have some undefined issues with COPR.
>    -
>
>    It is not official build system of Fedora which is not helping with Problem
>    №2: Testing of new rpm/koji/mock features/configuration
>   
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
>    .
>    -
>
>    COPR is not extensible enough, more specifically:
>    -
>
>       No query API (e.g. it is not possible to find out from which SCM
>       commit the package has been built)
>       -
>
>       Builds always have access to all previous builds in a repository
>       (i.e. not possible to control when/how repo is created)
>       -
>
>       GC doesn’t exist
>       -
>
>       No scratch builds
>
> Why not staging Koji?
>
>    -
>
>    Staging Koji is meant for testing Koji itself and not for package
>    development.
>    -
>
>    Staging Koji is often in broken state where it is not possible to build
>    anything.
>    -
>
>    No one can touch staging Koji, so it’s pointless for offering a useful
>    service.
>
> How can you make Koji builds faster?
>
>    -
>
>    By tuning Koji for performance, not correctness and reliability.
>    -
>
>    By building only on a single, fast architecture.
>    -
>
>    By extensive caching. Files like RPM packages, repodata and files in
>    lookaside cache don’t change after they are initially stored, so builders
>    can cache them aggressively.
>    -
>
>    By keeping build repositories small. Some package sets don’t need to be
>    built against full Fedora, but can use a minimal subset of Fedora. Such
>    repositories can be generated by Koji in seconds, not minutes.
>
> Why not the Open Build Service (OBS)?
>
>    -
>
>    OBS is not yet packaged for Fedora officially. The upstream code lacks
>    adaptations necessary to deploy and run on Fedora and in Fedora
>    Infrastructure.
>    -
>
>    OBS is not official build system of Fedora, which does not help with Problem
>    №2: Testing of new rpm/koji/mock features/configuration
>   
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
>    .
>
> What does MBI stand for?
>
>    -
>
>    M for middlestream, module, mizdebsk, …
>    -
>
>    B for build, bootstrap, …
>    -
>
>    I for infrastructure, initiative, ignatenkobrain, …
>    -
>
>    MBI might be pronounced as “maybe I …(can make something cool)?”
>    -
>
>    MBI is also initials of mizdebsk (Mikolaj Bozydar Izdebski)
>    -
>
>    MBI is also IBM spelled backwards :)
>
> Is it somehow related to Fedora Playground
> <https://fedoraproject.org/wiki/Playground>?
>
> Yes, it is. Although it is more developer-focused. Users could enrol for
> some specific things like experimental Java/Rust packages.
>
>
>  MBI (Playground 2.0)
>
<https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]