Alternative buildroot as a development tool

[Aleksandra Fedorova]

Hi, all,

This topic goes along the lines of Matthew’s Operating System Factory
discussion[1], but with a slightly different angle. It also is the
generalization of the Change we have proposed recently [2]

So let me start with the problem:

In Fedora now we have a well known and established process of how to
deal with updates that bring new content through packages. There are
ways to package new content side by side with the old one, to perform
a non-disruptive testing, iterate and upgrade.

What I think is missing is the way to safely iterate on the process,
which happens after the content is packaged. Examples are: changes to
buildroot configuration as in [2], changes in comps files, for example
required for Minimization Objective [3], changes in compose building
process, changes in Mass Rebuild and so on.

For such changes we have to use an all-or-nothing approach, which
means we either implement the change too early or postpone it for too
long.

To add flexibility to the Fedora process, I’d like to propose the
concept of alternative buildroots.

------
Note: The naming is hard here, and while I tend to call it
“buildroot”, it actually needs to be called “alternative everything,
except the srpms”.

I think we shouldn’t use the word “variant”, “spin”, “flavour” or
something like that to describe it, as it is strictly the shared
development playground linked to the latest Fedora Rawhide state and
focused on the build and compose process. It is definitely not the
alternative Fedora. It shouldn’t have releases on its own, it has no
branches in the source code and it doesn’t target end-users.
------

The good thing is that, with the distributed CI approach we have
implemented, we are ready to consume the feedback from such
alternative setups in a non-blocking but informative way. I think we
can extend our use of CI machinery so that it simplifies the
development process for everyone, removing redundant heads-ups but
increasing the targeted collaboration in places where it is actually
needed.

### Use cases ###

The first example of such a buildroot is provided by the CPU baseline
testing [1].

There will be more, since we may want to work on comps files for
Minimization Objective.

There is also one particularly important case of the RHEL bootstrap:
RHEL tries hard to inherit as much as possible from Fedora, but since
both Fedora and RHEL have historically different build and compose
configurations, it is often problematic to deal with dependency and
build issues which arise from it. To the point that RHEL sometimes
just can’t use Fedora as an origin, and needs to find its way around.

The alternative buildroot targeting CentOS and RHEL would provide an
open shared development environment. There we can work on converging
the RHEL process to Fedora, and also on improving Fedora process with
things, which RHEL and CentOS have discovered.

### Proposal ###

This idea doesn’t quite fit into the Fedora Changes framework, as
there is no actual change to Fedora. And we don't even have a concept
of an Infrastructure Change at this point.

But there are several items one can think about:

1) Though as I said above the “buildroot” term is slightly misleading
but it is a start. And the change [2] is a first step, the prototype,
which goes into that direction. Its main focus is the compiler
parameters rather than anything else.

2) The second part of the proposal is to setup a CentOS/RHEL-targeting
buildroot and compose, with a focus on comps, dependency chains and
compose differences. Which I think overlaps in a certain way with the
Minimization Objective.

3) And the third part is to generally develop the alternative
buildroot as a concept. It should be our toolbox item, something
similar to the alternative architectures approach. So that it can be
requested by anyone based on the existence of a SIG or working group,
which would own and maintain it.

[1]
https://lists.fedoraproject.org/archives/list/[e-mail].o...
[2]
https://lists.fedoraproject.org/archives/list/[e-mail].o...
[3]
https://lists.fedoraproject.org/archives/list/[e-mail].o...

--
Aleksandra Fedorova
bookwar

[Brendan Conoboy]

Many of us deeply involved in RHEL's production are excited about this
idea.  We are looking for ways to do more of the RHEL 9 creation
activities in public and this could help serve that purpose.
Something along these lines would be fantastic because identical
Fedora sources could be used to produce differentiated builds, where
only inherited rpm macros would differ.  Perhaps even better is that
it looks like it would give Fedora the ability to do more to serve its
own mission, by allowing customization without forking sources.
Fedora would be able to grow beyond the traditional
one-set-of-defaults-must-fit-all model.


On Mon, Jan 13, 2020 at 8:45 AM Aleksandra Fedorova
<afedorova(a)redhat.com&gt; wrote:
>  Hi, all,
>
>  This topic goes along the lines of Matthew’s Operating System Factory
>  discussion[1], but with a slightly different angle. It also is the
>  generalization of the Change we have proposed recently [2]
>
>  So let me start with the problem:
>
>  In Fedora now we have a well known and established process of how to
>  deal with updates that bring new content through packages. There are
>  ways to package new content side by side with the old one, to perform
>  a non-disruptive testing, iterate and upgrade.
>
>  What I think is missing is the way to safely iterate on the process,
>  which happens after the content is packaged. Examples are: changes to
>  buildroot configuration as in [2], changes in comps files, for example
>  required for Minimization Objective [3], changes in compose building
>  process, changes in Mass Rebuild and so on.
>
>  For such changes we have to use an all-or-nothing approach, which
>  means we either implement the change too early or postpone it for too
>  long.
>
>  To add flexibility to the Fedora process, I’d like to propose the
>  concept of alternative buildroots.
>
>  ------
>  Note: The naming is hard here, and while I tend to call it
>  “buildroot”, it actually needs to be called “alternative everything,
>  except the srpms”.
>
>  I think we shouldn’t use the word “variant”, “spin”, “flavour” or
>  something like that to describe it, as it is strictly the shared
>  development playground linked to the latest Fedora Rawhide state and
>  focused on the build and compose process. It is definitely not the
>  alternative Fedora. It shouldn’t have releases on its own, it has no
>  branches in the source code and it doesn’t target end-users.
>  ------
>
>  The good thing is that, with the distributed CI approach we have
>  implemented, we are ready to consume the feedback from such
>  alternative setups in a non-blocking but informative way. I think we
>  can extend our use of CI machinery so that it simplifies the
>  development process for everyone, removing redundant heads-ups but
>  increasing the targeted collaboration in places where it is actually
>  needed.
>
>  ### Use cases ###
>
>  The first example of such a buildroot is provided by the CPU baseline
>  testing [1].
>
>  There will be more, since we may want to work on comps files for
>  Minimization Objective.
>
>  There is also one particularly important case of the RHEL bootstrap:
>  RHEL tries hard to inherit as much as possible from Fedora, but since
>  both Fedora and RHEL have historically different build and compose
>  configurations, it is often problematic to deal with dependency and
>  build issues which arise from it. To the point that RHEL sometimes
>  just can’t use Fedora as an origin, and needs to find its way around.
>
>  The alternative buildroot targeting CentOS and RHEL would provide an
>  open shared development environment. There we can work on converging
>  the RHEL process to Fedora, and also on improving Fedora process with
>  things, which RHEL and CentOS have discovered.
>
>  ### Proposal ###
>
>  This idea doesn’t quite fit into the Fedora Changes framework, as
>  there is no actual change to Fedora. And we don't even have a concept
>  of an Infrastructure Change at this point.
>
>  But there are several items one can think about:
>
>  1) Though as I said above the “buildroot” term is slightly misleading
>  but it is a start. And the change [2] is a first step, the prototype,
>  which goes into that direction. Its main focus is the compiler
>  parameters rather than anything else.
>
>  2) The second part of the proposal is to setup a CentOS/RHEL-targeting
>  buildroot and compose, with a focus on comps, dependency chains and
>  compose differences. Which I think overlaps in a certain way with the
>  Minimization Objective.
>
>  3) And the third part is to generally develop the alternative
>  buildroot as a concept. It should be our toolbox item, something
>  similar to the alternative architectures approach. So that it can be
>  requested by anyone based on the existence of a SIG or working group,
>  which would own and maintain it.
>
>  [1]
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  [2]
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  [3]
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>
>  --
>  Aleksandra Fedorova
>  bookwar
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail] 


--
Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc.

[Brian (bex) Exelbierd]

I am chiming in with a slightly long form of +1 because of the lack of
comments.

I think this is HUGE.  To be able to connect to new use cases and ideas and
have the bonus of helping a downstream stay more closely connected feels
like a big win!

regards,

bex

On Tue, Jan 14, 2020 at 5:54 AM Brendan Conoboy <blc(a)redhat.com&gt; wrote:

>  Many of us deeply involved in RHEL's production are excited about
> this
>  idea.  We are looking for ways to do more of the RHEL 9 creation
>  activities in public and this could help serve that purpose.
>  Something along these lines would be fantastic because identical
>  Fedora sources could be used to produce differentiated builds, where
>  only inherited rpm macros would differ.  Perhaps even better is that
>  it looks like it would give Fedora the ability to do more to serve its
>  own mission, by allowing customization without forking sources.
>  Fedora would be able to grow beyond the traditional
>  one-set-of-defaults-must-fit-all model.
>
>
>  On Mon, Jan 13, 2020 at 8:45 AM Aleksandra Fedorova
>  <afedorova(a)redhat.com&gt; wrote:
>  >
>  > Hi, all,
>  >
>  > This topic goes along the lines of Matthew’s Operating System Factory
>  > discussion[1], but with a slightly different angle. It also is the
>  > generalization of the Change we have proposed recently [2]
>  >
>  > So let me start with the problem:
>  >
>  > In Fedora now we have a well known and established process of how to
>  > deal with updates that bring new content through packages. There are
>  > ways to package new content side by side with the old one, to perform
>  > a non-disruptive testing, iterate and upgrade.
>  >
>  > What I think is missing is the way to safely iterate on the process,
>  > which happens after the content is packaged. Examples are: changes to
>  > buildroot configuration as in [2], changes in comps files, for example
>  > required for Minimization Objective [3], changes in compose building
>  > process, changes in Mass Rebuild and so on.
>  >
>  > For such changes we have to use an all-or-nothing approach, which
>  > means we either implement the change too early or postpone it for too
>  > long.
>  >
>  > To add flexibility to the Fedora process, I’d like to propose the
>  > concept of alternative buildroots.
>  >
>  > ------
>  > Note: The naming is hard here, and while I tend to call it
>  > “buildroot”, it actually needs to be called “alternative everything,
>  > except the srpms”.
>  >
>  > I think we shouldn’t use the word “variant”, “spin”, “flavour” or
>  > something like that to describe it, as it is strictly the shared
>  > development playground linked to the latest Fedora Rawhide state and
>  > focused on the build and compose process. It is definitely not the
>  > alternative Fedora. It shouldn’t have releases on its own, it has no
>  > branches in the source code and it doesn’t target end-users.
>  > ------
>  >
>  > The good thing is that, with the distributed CI approach we have
>  > implemented, we are ready to consume the feedback from such
>  > alternative setups in a non-blocking but informative way. I think we
>  > can extend our use of CI machinery so that it simplifies the
>  > development process for everyone, removing redundant heads-ups but
>  > increasing the targeted collaboration in places where it is actually
>  > needed.
>  >
>  > ### Use cases ###
>  >
>  > The first example of such a buildroot is provided by the CPU baseline
>  > testing [1].
>  >
>  > There will be more, since we may want to work on comps files for
>  > Minimization Objective.
>  >
>  > There is also one particularly important case of the RHEL bootstrap:
>  > RHEL tries hard to inherit as much as possible from Fedora, but since
>  > both Fedora and RHEL have historically different build and compose
>  > configurations, it is often problematic to deal with dependency and
>  > build issues which arise from it. To the point that RHEL sometimes
>  > just can’t use Fedora as an origin, and needs to find its way around.
>  >
>  > The alternative buildroot targeting CentOS and RHEL would provide an
>  > open shared development environment. There we can work on converging
>  > the RHEL process to Fedora, and also on improving Fedora process with
>  > things, which RHEL and CentOS have discovered.
>  >
>  > ### Proposal ###
>  >
>  > This idea doesn’t quite fit into the Fedora Changes framework, as
>  > there is no actual change to Fedora. And we don't even have a concept
>  > of an Infrastructure Change at this point.
>  >
>  > But there are several items one can think about:
>  >
>  > 1) Though as I said above the “buildroot” term is slightly misleading
>  > but it is a start. And the change [2] is a first step, the prototype,
>  > which goes into that direction. Its main focus is the compiler
>  > parameters rather than anything else.
>  >
>  > 2) The second part of the proposal is to setup a CentOS/RHEL-targeting
>  > buildroot and compose, with a focus on comps, dependency chains and
>  > compose differences. Which I think overlaps in a certain way with the
>  > Minimization Objective.
>  >
>  > 3) And the third part is to generally develop the alternative
>  > buildroot as a concept. It should be our toolbox item, something
>  > similar to the alternative architectures approach. So that it can be
>  > requested by anyone based on the existence of a SIG or working group,
>  > which would own and maintain it.
>  >
>  > [1]
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > [2]
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > [3]
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  >
>  > --
>  > Aleksandra Fedorova
>  > bookwar
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>
>
>
>  --
>  Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 
Brian "bex" Exelbierd (he/him/his)
RHEL Product Management & Community Architect
@bexelbie | http://www.winglemeyer.org
bexelbie(a)redhat.com

[Adam Samalik]

I really like this proposal. I feel like it's something we needed for a
long time.
More comments inline!

On Mon, Jan 13, 2020 at 5:45 PM Aleksandra Fedorova <afedorova(a)redhat.com&gt;
wrote:

>  Hi, all,
>
>  This topic goes along the lines of Matthew’s Operating System Factory
>  discussion[1], but with a slightly different angle. It also is the
>  generalization of the Change we have proposed recently [2]
>
>  So let me start with the problem:
>
>  In Fedora now we have a well known and established process of how to
>  deal with updates that bring new content through packages. There are
>  ways to package new content side by side with the old one, to perform
>  a non-disruptive testing, iterate and upgrade.
>
>  What I think is missing is the way to safely iterate on the process,
>  which happens after the content is packaged. Examples are: changes to
>  buildroot configuration as in [2], changes in comps files, for example
>  required for Minimization Objective [3], changes in compose building
>  process, changes in Mass Rebuild and so on.
>  
Yes! I felt that a few times in the past.

We know how to make our output evolve, how to introduce new RPM
versions and make other output-related changes. We test all updates
thanks to Bodhi, try and test bigger changes in side tags, etc. But we don't
really have a way to safely experiment with what makes it all
happen — the Fedora build system. And by "build system", I mean it in
the broadest sense, including Koji, compose, configurations, and
many other parts).

And that's exactly where things like improving packager experience in
an actual significant way (that requires some testing and iterations), or
performance tuning that affects compatibility — requiring all those
technical changes you've listed — happen. Or would happen!


>  For such changes we have to use an all-or-nothing approach, which
>  means we either implement the change too early or postpone it for too
>  long.
>  
... or don't have space to experiment with bigger changes, potentially
limiting
our ability to innovate. And when we try to, there is a high risk of
negatively
impacting our contributors, because, well, production is where the testing
happens.


>  To add flexibility to the Fedora process, I’d like to propose the
>  concept of alternative buildroots.
>  
Alternative buildroots are a good start.

>  ------
>  Note: The naming is hard here, and while I tend to call it
>  “buildroot”, it actually needs to be called “alternative everything,
>  except the srpms”.
>  
Oh! Ok, that's exactly what I was hoping for and that's even a better start!
(Or, you know, all we need :D)


>  I think we shouldn’t use the word “variant”, “spin”, “flavour” or
>  something like that to describe it, as it is strictly the shared
>  development playground linked to the latest Fedora Rawhide state and
>  focused on the build and compose process. It is definitely not the
>  alternative Fedora. It shouldn’t have releases on its own, it has no
>  branches in the source code and it doesn’t target end-users.
>  ------
>
>  The good thing is that, with the distributed CI approach we have
>  implemented, we are ready to consume the feedback from such
>  alternative setups in a non-blocking but informative way. I think we
>  can extend our use of CI machinery so that it simplifies the
>  development process for everyone, removing redundant heads-ups but
>  increasing the targeted collaboration in places where it is actually
>  needed.
>
>  ### Use cases ###
>
>  The first example of such a buildroot is provided by the CPU baseline
>  testing [1].
>
>  There will be more, since we may want to work on comps files for
>  Minimization Objective.
>  
As part of the objective, we avoided such changes as they're quite
disruptive.
I was actually thinking about trying those "somewhere" on the side, but
this proposal sounds like enabling this, done properly as part of our infra,
for everyone. Cool!


>  There is also one particularly important case of the RHEL bootstrap:
>  RHEL tries hard to inherit as much as possible from Fedora, but since
>  both Fedora and RHEL have historically different build and compose
>  configurations, it is often problematic to deal with dependency and
>  build issues which arise from it. To the point that RHEL sometimes
>  just can’t use Fedora as an origin, and needs to find its way around.
>
>  The alternative buildroot targeting CentOS and RHEL would provide an
>  open shared development environment. There we can work on converging
>  the RHEL process to Fedora, and also on improving Fedora process with
>  things, which RHEL and CentOS have discovered.
>
>  ### Proposal ###
>
>  This idea doesn’t quite fit into the Fedora Changes framework, as
>  there is no actual change to Fedora. And we don't even have a concept
>  of an Infrastructure Change at this point. 

Infra Changes are an interesting concept for this.

One thing that struck my mind was... Testing alternative approaches
on the side without affecting people is not really a change. That said,
Changes we do for content, implemented more or less in the main space,
force us get there faster. Sure, bigger changes are properly tested in
side tags for example, but some are done directly (while being properly
announced of course).

What I'm trying to say is... I think there is a balance of
speed vs. having a safe space to innovate on bigger things. And we
definitely need both for infra.


>  But there are several items one can think about:
>
>  1) Though as I said above the “buildroot” term is slightly misleading
>  but it is a start. And the change [2] is a first step, the prototype,
>  which goes into that direction. Its main focus is the compiler
>  parameters rather than anything else.
>  
I think this is a good start that will touch many places we'll need to
tweak to make them flexible enough to handle multiple build configs and,
of course, multiple outputs of those.


>  2) The second part of the proposal is to setup a
> CentOS/RHEL-targeting
>  buildroot and compose, with a focus on comps, dependency chains and
>  compose differences. Which I think overlaps in a certain way with the
>  Minimization Objective.
>  
Will this be a general thing, that will allow others to build such
additional
pipelines for testing? Or will this be solely for the CentOS/RHEL-targeting
part?

>  3) And the third part is to generally develop the alternative
>  buildroot as a concept. It should be our toolbox item, something
>  similar to the alternative architectures approach. So that it can be
>  requested by anyone based on the existence of a SIG or working group,
>  which would own and maintain it.
>  
I think this answer my question — we'll test it on that one thing, learn
some
lessons, and based on that we'll come up with a generic way for everyone.
And that actually sounds much better than trying to come up with a generic
thing first. Did I get that right?

----

I'm actually really excited about this effort. Definitely interested in
helping
with some of that!


>  [1]
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  [2]
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  [3]
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>
>  --
>  Aleksandra Fedorova
>  bookwar
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 

Adam Šamalík
---------------------------
Senior Software Engineer
Red Hat

[Kevin Fenzi]

On Mon, Jan 13, 2020 at 05:43:26PM +0100, Aleksandra Fedorova wrote:
...snip...

>  ------
>  Note: The naming is hard here, and while I tend to call it
>  “buildroot”, it actually needs to be called “alternative everything,
>  except the srpms”.
>  
>  I think we shouldn’t use the word “variant”, “spin”, “flavour” or
>  something like that to describe it, as it is strictly the shared
>  development playground linked to the latest Fedora Rawhide state and
>  focused on the build and compose process. It is definitely not the
>  alternative Fedora. It shouldn’t have releases on its own, it has no
>  branches in the source code and it doesn’t target end-users.
>  ------ 
petri dish? too hard to say...fields (like planting a test crop in
another field)? Naming is hard. :) 
>  
>  The good thing is that, with the distributed CI approach we have
>  implemented, we are ready to consume the feedback from such
>  alternative setups in a non-blocking but informative way. I think we
>  can extend our use of CI machinery so that it simplifies the
>  development process for everyone, removing redundant heads-ups but
>  increasing the targeted collaboration in places where it is actually
>  needed.
>  
>  ### Use cases ###
>  
>  The first example of such a buildroot is provided by the CPU baseline
>  testing [1].
>  
>  There will be more, since we may want to work on comps files for
>  Minimization Objective. 
Yeah, but comps files are used in composes, so we would have to do
composes to see changes there. I guess the CI part would do the
composing in that case? But we would also need a different comps (so we
didn't change the default one). 

>  There is also one particularly important case of the RHEL bootstrap:
>  RHEL tries hard to inherit as much as possible from Fedora, but since
>  both Fedora and RHEL have historically different build and compose
>  configurations, it is often problematic to deal with dependency and
>  build issues which arise from it. To the point that RHEL sometimes
>  just can’t use Fedora as an origin, and needs to find its way around.
>  
>  The alternative buildroot targeting CentOS and RHEL would provide an
>  open shared development environment. There we can work on converging
>  the RHEL process to Fedora, and also on improving Fedora process with
>  things, which RHEL and CentOS have discovered. 
+1 !
 
>  ### Proposal ###
>  
>  This idea doesn’t quite fit into the Fedora Changes framework, as
>  there is no actual change to Fedora. And we don't even have a concept
>  of an Infrastructure Change at this point. 
Well, infrastructure has a Request for resources process. 
However, it's a bit dated these days, not taking into account OpenShift
or the like. The CPE team has been wanting to move to a higher level
process for this kind of thing... but yeah, nothing is really clear, I'd
say just get buyin from stakeholders/fesco.
>  
>  But there are several items one can think about:
>  
>  1) Though as I said above the “buildroot” term is slightly misleading
>  but it is a start. And the change [2] is a first step, the prototype,
>  which goes into that direction. Its main focus is the compiler
>  parameters rather than anything else. 
Yep.

>  2) The second part of the proposal is to setup a
> CentOS/RHEL-targeting
>  buildroot and compose, with a focus on comps, dependency chains and
>  compose differences. Which I think overlaps in a certain way with the
>  Minimization Objective. 
So, this would have composes? Or they would be test composes in CI/ODCS?

>  3) And the third part is to generally develop the alternative
>  buildroot as a concept. It should be our toolbox item, something
>  similar to the alternative architectures approach. So that it can be
>  requested by anyone based on the existence of a SIG or working group,
>  which would own and maintain it. 
Yeah, although I think we should still examine if a thing is ready for
testing this way. I don't think we should just add anything that anyone
can think of. It does use resources. That said, I think if the concept
for testing is reasonable we could add it. 

Thanks for working on this and I'm eager to see how well we can get it
working! :) 

kevin

[Brendan Conoboy]

On Wed, Jan 15, 2020 at 8:31 AM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
[snip]
>  > ### Use cases ###
>  >
>  > The first example of such a buildroot is provided by the CPU baseline
>  > testing [1].
>  >
>  > There will be more, since we may want to work on comps files for
>  > Minimization Objective.
>
>  Yeah, but comps files are used in composes, so we would have to do
>  composes to see changes there. I guess the CI part would do the
>  composing in that case? But we would also need a different comps (so we
>  didn't change the default one). 
You would need a different comps, but that also means you would get to
have a different comps to experiment with...

[snip]
>  > But there are several items one can think about:
>  >
>  > 1) Though as I said above the “buildroot” term is slightly misleading
>  > but it is a start. And the change [2] is a first step, the prototype,
>  > which goes into that direction. Its main focus is the compiler
>  > parameters rather than anything else.
>
>  Yep.
>
>  > 2) The second part of the proposal is to setup a CentOS/RHEL-targeting
>  > buildroot and compose, with a focus on comps, dependency chains and
>  > compose differences. Which I think overlaps in a certain way with the
>  > Minimization Objective.
>
>  So, this would have composes? Or they would be test composes in CI/ODCS? 
One potential output of this change would be to have editions choose
what buildroot they are based on.  As an example, Fedora Server, which
has struggled to find its element, could actually be built in ways
more conducive to server use cases: different compiler flags, kernel
parameters, baselines, etc.

-- 
Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc.

[Kevin Fenzi]

On Thu, Jan 16, 2020 at 11:27:36AM -0800, Brendan Conoboy wrote:
>  On Wed, Jan 15, 2020 at 8:31 AM Kevin Fenzi <kevin(a)scrye.com&gt;
> wrote:
>  [snip]
>  > > ### Use cases ###
>  > >
>  > > The first example of such a buildroot is provided by the CPU baseline
>  > > testing [1].
>  > >
>  > > There will be more, since we may want to work on comps files for
>  > > Minimization Objective.
>  >
>  > Yeah, but comps files are used in composes, so we would have to do
>  > composes to see changes there. I guess the CI part would do the
>  > composing in that case? But we would also need a different comps (so we
>  > didn't change the default one).
>  
>  You would need a different comps, but that also means you would get to
>  have a different comps to experiment with... 
Sure. 
>  
>  [snip]
>  > > But there are several items one can think about:
>  > >
>  > > 1) Though as I said above the “buildroot” term is slightly misleading
>  > > but it is a start. And the change [2] is a first step, the prototype,
>  > > which goes into that direction. Its main focus is the compiler
>  > > parameters rather than anything else.
>  >
>  > Yep.
>  >
>  > > 2) The second part of the proposal is to setup a CentOS/RHEL-targeting
>  > > buildroot and compose, with a focus on comps, dependency chains and
>  > > compose differences. Which I think overlaps in a certain way with the
>  > > Minimization Objective.
>  >
>  > So, this would have composes? Or they would be test composes in CI/ODCS?
>  
>  One potential output of this change would be to have editions choose
>  what buildroot they are based on.  As an example, Fedora Server, which
>  has struggled to find its element, could actually be built in ways
>  more conducive to server use cases: different compiler flags, kernel
>  parameters, baselines, etc. 
Possibly, but also splitting things means they have less people using
them, and also there may be less compromise to come up with something
everyone can share. 

But I'm perfectly happy to give this a try and see where it leads us. :) 

kevin

[Brendan Conoboy]

On Thu, Jan 16, 2020 at 5:14 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Thu, Jan 16, 2020 at 11:27:36AM -0800, Brendan Conoboy wrote:
>  > One potential output of this change would be to have editions choose
>  > what buildroot they are based on.  As an example, Fedora Server, which
>  > has struggled to find its element, could actually be built in ways
>  > more conducive to server use cases: different compiler flags, kernel
>  > parameters, baselines, etc.
>
>  Possibly, but also splitting things means they have less people using
>  them, and also there may be less compromise to come up with something
>  everyone can share. 
This may be true, but if the number of people grows as a result, that
might be worth it.  No need to decide right nwo of course.  Part of
what is good about Aleksandra's proposal is that it's incremental-
each step along the way has a specific benefit.  The long term
potential for this is exciting, but the short and medium term benefits
seem good too.  Change can happen a little at a time then be measured
and adjusted as needed.

>  But I'm perfectly happy to give this a try and see where it leads
> us. :) 
Cool :-)

-- 
Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc.

[Neal Gompa]

On Thu, Jan 16, 2020 at 9:30 PM Brendan Conoboy <blc(a)redhat.com&gt; wrote:
>  On Thu, Jan 16, 2020 at 5:14 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  > On Thu, Jan 16, 2020 at 11:27:36AM -0800, Brendan Conoboy wrote:
>  > > One potential output of this change would be to have editions choose
>  > > what buildroot they are based on.  As an example, Fedora Server, which
>  > > has struggled to find its element, could actually be built in ways
>  > > more conducive to server use cases: different compiler flags, kernel
>  > > parameters, baselines, etc.
>  >
>  > Possibly, but also splitting things means they have less people using
>  > them, and also there may be less compromise to come up with something
>  > everyone can share.
>
>  This may be true, but if the number of people grows as a result, that
>  might be worth it.  No need to decide right nwo of course.  Part of
>  what is good about Aleksandra's proposal is that it's incremental-
>  each step along the way has a specific benefit.  The long term
>  potential for this is exciting, but the short and medium term benefits
>  seem good too.  Change can happen a little at a time then be measured
>  and adjusted as needed.
>
>  > But I'm perfectly happy to give this a try and see where it leads us. :)
>
>  Cool :-)
>  
I don't think I've made any secret that I would love to have the
ability to do awesome large-scale improvements in Fedora more easily.
I've mentioned in various avenues that I'd love for us to have that to
give people the ability to build an awesome distro and take it in
crazy directions to see where we can truly go.

I've also said that I don't think we can handle it as our
infrastructure currently stands. Our build system tooling has suffered
from a decade of neglect, and attempts to reinvent or improve that
have been rebuffed or failed in other ways. We have not learned from
what other communities are or are not doing, either out of hubris or
ignorance (I'm really hoping it's the latter, personally). Some of
these problems have been broadly exposed as part of the Modularity
bringup, but there's an underlying general problem that it is okay to
hobble along because we've hobbled along for so long. If anyone wants
to know details, feel free to ask me. I don't want to fill this with
the sordid details...

I've had the pleasure of becoming involved in many other communities
over the past several years, and have had exposure to how others have
solved problems we've had. Consequently, I've been spoiled at $DAYJOB
where some of these issues we've been talking about for a couple of
years simply don't exist because our tooling and infrastructure has
effectively solved it for us. That's not to say it's perfect and we
have no problems at all. There are still issues and quirks that I have
to work around just like Fedora does with what the project has. I'm
slowly working through trying to make what I use for $DAYJOB available
within Fedora so members of the Fedora community can use it for their
own purposes. After all, the stuff I use is open source too (except
for the stuff I wrote, which I'm trying to get that ready to be opened
up too...).

I can't believe I'm saying this, but I think "alternative buildroots"
is not enough. It's a step in the right direction in terms of thinking
about how to make developing bigger and more interesting changes. But
we need to be thinking more holistically about packager and developer
workflows within Fedora. Some of this is happening under the remit of
the Fedora CI work, and that's awesome! But why are these things
happening in haphazard pieces where we will likely wind up with more
half-implemented solutions or things that people don't really like but
stomach because some overlord is forcing it?

Incrementalism has its place, but this is something where once we
implement the "just enough increment", we'll stop again and everything
will just be uncomfortable and somewhat unhelpful, in much of the same
way the pkgdb to pagure transition was handled. The reason why it'll
happen is the same reason why it happened then: this stuff is
legitimately difficult to improve and it's easy to pull the brakes
once you reach a certain level of usefulness.

So... I guess I'm trying to say is "think bigger and go for broke",
because I think we're really only going to have one chance to do this
for the decade. Think like the people who made stuff like Koji and
Dist-Git a reality over nearly 15 years ago, when *those* ideas were
crazy and new! Just because the project is approaching being old
enough to drink in the US doesn't mean we can't do these kinds of big
things still!


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Nicolas Mailhot]

Le jeudi 16 janvier 2020 à 22:24 -0500, Neal Gompa a écrit :

Hi Neal,

>  I've also said that I don't think we can handle it as our
>  infrastructure currently stands. Our build system tooling has
>  suffered from a decade of neglect, and attempts to reinvent or
>  improve that have been rebuffed or failed in other ways.  
…


>  there's an underlying general problem that it is okay to
>  hobble along because we've hobbled along for so long. 
I agree 100%, thanks for posting this, it’s not easy be the messenger
for bad news.

>  Incrementalism has its place, but this is something where once we
>  implement the "just enough increment", we'll stop again and
>  everything will just be uncomfortable and somewhat unhelpful,  
And here I disagree. Beware of the “let’s rewrite everything better”
trap. If anything our incremental fixes failed more often than not
because key players waited on big bang magical solutions (for example,
modularity).

We need to implement more incremental fixes. Incremental does not mean
risk free, it means *managed* risk. Project members need to accept
incremental changes need to go in, and they need to accept incremental
changes may break things or require some amount of project-wide work. 

And, the Fedora leadership needs to do its leadership job. Leadership
does not mean promoting fuzzy long term objectives that no one can
disagree on because of the fuzziness, and that will explode on someone
else’s turn. Leadership means decisions for today, not just tomorrow.

Big bang initiatives have their places, but only when incrementalism
has hit a technical impasse. The Boeing 737 is an example where
incrementalism hit its limits. But, the 70 years of some 737 components
show how long incrementalism can be successful, as long as it it
managed intelligently.

Regards,

-- 
Nicolas Mailhot

[Stephen John Smoogen]

On Fri, 17 Jan 2020 at 04:33, Nicolas Mailhot via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Le jeudi 16 janvier 2020 à 22:24 -0500, Neal Gompa a écrit :
>
>  Hi Neal,
>
>  > I've also said that I don't think we can handle it as our
>  > infrastructure currently stands. Our build system tooling has
>  > suffered from a decade of neglect, and attempts to reinvent or
>  > improve that have been rebuffed or failed in other ways.
>
>  …
>
>
>  > there's an underlying general problem that it is okay to
>  > hobble along because we've hobbled along for so long.
>
>  I agree 100%, thanks for posting this, it’s not easy be the messenger
>  for bad news.
>
>  > Incrementalism has its place, but this is something where once we
>  > implement the "just enough increment", we'll stop again and
>  > everything will just be uncomfortable and somewhat unhelpful,
>
>  And here I disagree. Beware of the “let’s rewrite everything better”
>  trap. If anything our incremental fixes failed more often than not
>  because key players waited on big bang magical solutions (for example,
>  modularity).
>
>  We need to implement more incremental fixes. Incremental does not mean
>  risk free, it means *managed* risk. Project members need to accept
>  incremental changes need to go in, and they need to accept incremental
>  changes may break things or require some amount of project-wide work.
>
>  And, the Fedora leadership needs to do its leadership job. Leadership
>  does not mean promoting fuzzy long term objectives that no one can
>  disagree on because of the fuzziness, and that will explode on someone
>  else’s turn. Leadership means decisions for today, not just tomorrow.
>
>  Big bang initiatives have their places, but only when incrementalism
>  has hit a technical impasse. The Boeing 737 is an example where
>  incrementalism hit its limits. But, the 70 years of some 737 components
>  show how long incrementalism can be successful, as long as it it
>  managed intelligently.
>  
My 'take' on the 737MAX problems is that they decided that
incrementalism wasn't going 'fast' enough but they also didn't want to
add a lot of training costs or 'change aversion' items. So they put in
a lot of new things which would have made the plane a different model
but then tried to sell it that they hadn't changed much so you didn't
need to send your pilots to retrain over the fact that various
switches do the opposite of what they did before.. Boeing then further
'cut' costs by removing a lot of places where a pilot team could
over-ride the computers or have a backup piece in case the first
failed.

All of these things could have still been done with incrementalism,
but it would have probably taken 10 more years that they felt they
didn't need but that they also didn't want to lose their current
customer base by making a 7337 or a 797 plane which told everyone this
plane is different. One problem with rewrite everything is where you
try to keep your existing base because it is safe but also try to do
something completely new. If you are going to do it.. be clear about
it and realize that the people you had will have little interest in it
because they are happy with what they have.

-- 
Stephen J Smoogen.