Fedora 33 System-Wide Change proposal: Use %make_build and %make_install macros

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro

== Summary ==
This change will update all spec files in Fedora that use make and replace
the make invocations with either the %make_build or %make_install macros.

== Owner ==
* Name: [[User:tstellar| Tom Stellard]]
* Email: <tstellar(a)redhat.com&gt;


== Detailed Description ==

The goal of this change is to standardize the usage of make in Fedora.  All
make invocations in spec files that don't use the install target will be
modified to use the %make_build macro, and all make invocations that use
the install target will be updated to use the %make_install macro.  Any
additional arguments to make that are not included in either the
%make_build and %make_install will be preserved.

The %make_build macro enables parallel make builds using the -j option.  In
case a package does not build correctly with parallel make, then parallel
make will be disabled for that package by redefining the %_smp_mflags macro
like this:

  %global _smp_mflags -j1

All changes will be submitted to dist-git repositories via pull requests.
Pull requests will be merged after 1 week if there are no objections or
earlier if approved by the package maintainers.

A script will be used to apply the necessary changes to the spec files, and
the result will be manually inspected before being merged.

All packages updated by this change request will be rebuilt after the spec
file changes are merged.

Some packages already use the %make_build and %make_install macros.  These
packages will be left unchanged.

== Benefit to Fedora ==
* Reduced build times: Using the %make_build macros enables parallel make
builds which will reduce build times for Fedora packages.

* This will make it easier to enforce consistent build flag usage across
all of Fedora.

== Scope ==
* Proposal owners: Update spec files and submit pull requests and do new
package builds.  Optional: Merge pull requests (Proposal Owner would need
to request to be added to provenpackagers group)

* Other developers: Package owners may merge pull requests and submit new
builds if they want, but this is not required for them.  A member of the
provenpackagers group will be needed to merge pull requests.
* Release engineering: [https://pagure.io/releng/issues/9533 #9533]

* Policies and guidelines: Package guidelines already specify that packages
should use these macros when possible.  Documentation will be updated to
clarify that %make_build should be used for all make invocations (besides
make install), and also with instructions for packages that fail to build
with parallel make.
* Trademark approval: N/A (not needed for this Change)


== Upgrade/compatibility impact ==
No impact.

== How To Test ==
End-users will not notice any changes.

== Dependencies ==
No real dependencies, each package can be updated independently.

== Contingency Plan ==
* Contingency mechanism: None.  If not all packages are updated in time,
then the work can continue into the next release.
* Contingency deadline: All work will be done in the rawhide branch, and
will not be backported into the f33 branch once it is created, so whatever
gets done before the branch date will make it into the release.
* Blocks release? No

== Documentation ==
The packaging guidelines will be updated as described in the Scope Section.



-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis

[Tomasz Kłoczko]

On Fri, 19 Jun 2020 at 22:20, Ben Cotton <bcotton(a)redhat.com&gt; wrote:
[..]

>  The %make_build macro enables parallel make builds using the -j
> option.
>  In case a package does not build correctly with parallel make, then
>  parallel make will be disabled for that package by redefining the
>  %_smp_mflags macro like this:
>
>    %global _smp_mflags -j1
>  
Sometimes only %build or %install or %check is failing.
Any -j1 tweaks should be *only temporary* so IMO formalising this kind of
changes is pointless.
All parallel build issues should be treated *as critical bugs*
 which should be *ASAP fixed*.

kloczek
-- 
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH

[Kamil Dudka]

On Saturday, June 20, 2020 12:17:26 AM CEST Tomasz Kłoczko wrote:
>  All parallel build issues should be treated *as critical bugs*
>   which should be *ASAP fixed*.
>  
>  kloczek 
Seriously, are you saying that bugs causing slower build are as important
as bugs causing data lost or security vulnerabilities at run time?

If they are so important for you, please help us to get them fixed!

I tried to parallelize the build of nss and failed:

https://src.fedoraproject.org/rpms/nss/c/6e689ce0
https://src.fedoraproject.org/rpms/nss/c/3c27dc24

I would really appreciate your help on this.

Kamil

[Dominik 'Rathann' Mierzejewski]

On Saturday, 20 June 2020 at 09:20, Kamil Dudka wrote:
>  On Saturday, June 20, 2020 12:17:26 AM CEST Tomasz Kłoczko wrote:
>  > All parallel build issues should be treated *as critical bugs*
>  >  which should be *ASAP fixed*.
>  > 
>  > kloczek
>  
>  Seriously, are you saying that bugs causing slower build are as important
>  as bugs causing data lost or security vulnerabilities at run time?
>  
>  If they are so important for you, please help us to get them fixed!
>  
>  I tried to parallelize the build of nss and failed:
>  
>  https://src.fedoraproject.org/rpms/nss/c/6e689ce0
>  https://src.fedoraproject.org/rpms/nss/c/3c27dc24
>  
>  I would really appreciate your help on this. 
Same for INN:
https://bugzilla.redhat.com/show_bug.cgi?id=1558586

Regards,
Dominik
-- 
Fedora   https://getfedora.org  |  RPM Fusion  http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
        -- from "Collected Sayings of Muad'Dib" by the Princess Irulan

[Tom Stellard]

On 06/19/2020 03:17 PM, Tomasz Kłoczko wrote:
>  On Fri, 19 Jun 2020 at 22:20, Ben Cotton <bcotton(a)redhat.com
> <mailto:[e-mail]>> wrote:
>  [..]
>  
>      The %make_build macro enables parallel make builds using the -j option.  In case a
> package does not build correctly with parallel make, then parallel make will be disabled
> for that package by redefining the %_smp_mflags macro like this:
>  
>        %global _smp_mflags -j1
>  
>  
>  Sometimes only %build or %install or %check is failing. 
That's a good point.  What if as an alternative as part of
this change, we used %make_build -j1 in the case where
parallel building does not work?

>  Any -j1 tweaks should be *only temporary* so IMO formalising this
> kind of changes is pointless.
>  All parallel build issues should be treated *as critical bugs* which should be *ASAP
> fixed*. 

So are you saying that we should not document the %make_build -j1 convention
and instead update the documentation to state that parallel build failures must be
fixed?

Even if we don't document this new convention, I still thing it should
be used for this change, because I think having %make_build -j1 makes it
obvious something is broken and more likely to get fixed.

-Tom

>  
>  kloczek
>  -- 
>  Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
>  
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Miro Hrončok]

On 19. 06. 20 23:11, Ben Cotton wrote:
>  All make invocations in spec files that don't use the install
> target will be 
>  modified to use the %make_build macro 
Many Python packages build Sphinx documentation with variant of "make html". 
Such invocation will always be just 1 job. Hence there is no benefit of parallel 
make. Replacing it with %make_build will only make it harder to read.

Can we exclude such cases? Or do we want all make invocations to be mecronized, 
even if there is no benefit?

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Andy Mender]

On Sat, 20 Jun 2020 at 00:38, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:

>  On 19. 06. 20 23:11, Ben Cotton wrote:
>  > All make invocations in spec files that don't use the install target
>  will be
>  > modified to use the %make_build macro
>
>  Many Python packages build Sphinx documentation with variant of "make
>  html".
>  Such invocation will always be just 1 job. Hence there is no benefit of
>  parallel
>  make. Replacing it with %make_build will only make it harder to read.
>
>  Can we exclude such cases? Or do we want all make invocations to be
>  mecronized,
>  even if there is no benefit?
>  
The way I understand it is that %make_build would be a replacement for the
"make all" target with N jobs, which is called by default when you just run
"make". That would be the all-in-1 scenario when you want to build the main
binaries, docs and examples. If there is only 1 target (html) and you run
"make" with N jobs, there is no downside, because only that single target
gets picked up.

I really like the proposal and kind of sort of agree with Tomasz that if a
build requires -j1 explicitly and fails with -jN, it means the target
dependency tree was not defined properly and should be fixed. The only time
I ever run "make" with -j1 is when debugging to get a clean, sequential log
output. So the way one can handle target resolution issues is with the
proposed %global _smp_mflags -j1 or with a unified patch + submission to
upstream to fix the bug.

~Andy

[Miro Hrončok]

On 20. 06. 20 14:47, Andy Mender wrote:
>  On Sat, 20 Jun 2020 at 00:38, Miro Hrončok <mhroncok(a)redhat.com 
>  <mailto:[e-mail]>> wrote:
>  
>      On 19. 06. 20 23:11, Ben Cotton wrote:
>       > All make invocations in spec files that don't use the install target will
> be
>       > modified to use the %make_build macro
>  
>      Many Python packages build Sphinx documentation with variant of "make
> html".
>      Such invocation will always be just 1 job. Hence there is no benefit of
>      parallel
>      make. Replacing it with %make_build will only make it harder to read.
>  
>      Can we exclude such cases? Or do we want all make invocations to be mecronized,
>      even if there is no benefit?
>  
>  
>  The way I understand it is that %make_build would be a replacement for the "make 
>  all" target with N jobs, which is called by default when you just run
> "make".  
The proposal says all make invocations.

>  That would be the all-in-1 scenario when you want to build the main
> binaries, 
>  docs and examples. If there is only 1 target (html) and you run "make" with N 
>  jobs, there is no downside, because only that single target gets picked up. 
No downside when building, but downside when reading the spec file.

>  I really like the proposal and kind of sort of agree with Tomasz that
> if a build 
>  requires -j1 explicitly and fails with -jN, it means the target dependency tree 
>  was not defined properly and should be fixed. 
That is not the case here.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Björn Persson]

Miro Hrončok wrote:
>  On 19. 06. 20 23:11, Ben Cotton wrote:
>  > All make invocations in spec files that don't use the install target will be 
>  > modified to use the %make_build macro  
>  
>  Many Python packages build Sphinx documentation with variant of "make html". 
>  Such invocation will always be just 1 job. Hence there is no benefit of parallel 
>  make. 
That's also the case in several Ada packages, where the makefile just
assembles the command line for a single invocation of GPRbuild, and
parallel compilation is handled by GPRbuild. I expect that there are
many other more or less special cases.

>  Replacing it with %make_build will only make it harder to read.
Especially in cases like "make check", where the purpose isn't to build
anything but to run a testsuite. The word "make" is already less than
ideal in those cases, and replacing it with the less well-known
"make_build" will make it more confusing.

>  Can we exclude such cases? Or do we want all make invocations to be
> mecronized, 
>  even if there is no benefit? 
So I too want that question answered.


Also: Is it set in stone that "make_build" means "make in parallel"
and
nothing else? If so, why isn't the macro called "parallel_make"?

Or is it the case that "make_build" means "the typical make command to
use in a build stage", and a future version of the macro may include
other parameters that are considered useful in a build stage but may
not be appropriate for other use cases? In that case the macro should
only be applied in the %build section, and any make invocation that
looks less than typical should be left alone.

Björn Persson

[Miro Hrončok]

On 20. 06. 20 23:18, Björn Persson wrote:
>  Also: Is it set in stone that "make_build" means "make
> in parallel" and
>  nothing else? If so, why isn't the macro called "parallel_make"?
>  
>  Or is it the case that "make_build" means "the typical make command to
>  use in a build stage", and a future version of the macro may include
>  other parameters that are considered useful in a build stage but may
>  not be appropriate for other use cases? In that case the macro should
>  only be applied in the %build section, and any make invocation that
>  looks less than typical should be left alone. 
Excellent point, Björn!

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Tom Stellard]

On 06/20/2020 02:18 PM, Björn Persson wrote:
>  Miro Hrončok wrote:
> > On 19. 06. 20 23:11, Ben Cotton wrote:
> >> All make invocations in spec files that don't use the install target will be
>
> >> modified to use the %make_build macro  
> >
> > Many Python packages build Sphinx documentation with variant of "make
> html". 
> > Such invocation will always be just 1 job. Hence there is no benefit of parallel 
> > make.
>  
>  That's also the case in several Ada packages, where the makefile just
>  assembles the command line for a single invocation of GPRbuild, and
>  parallel compilation is handled by GPRbuild. I expect that there are
>  many other more or less special cases.
>  
> > Replacing it with %make_build will only make it harder to read.
>  
>  Especially in cases like "make check", where the purpose isn't to build
>  anything but to run a testsuite. The word "make" is already less than
>  ideal in those cases, and replacing it with the less well-known
>  "make_build" will make it more confusing.
>  
> > Can we exclude such cases? Or do we want all make invocations to be mecronized, 
> > even if there is no benefit?
>  
>  So I too want that question answered.
>  
>  
>  Also: Is it set in stone that "make_build" means "make in parallel"
> and
>  nothing else? If so, why isn't the macro called "parallel_make"?
>  
>  Or is it the case that "make_build" means "the typical make command to
>  use in a build stage", and a future version of the macro may include
>  other parameters that are considered useful in a build stage but may
>  not be appropriate for other use cases? In that case the macro should
>  only be applied in the %build section, and any make invocation that
>  looks less than typical should be left alone. 
This is correct, my assumption is that in the future the macro may include
other parameters.  This is what I was referring to in the "Benefit to Fedora"
section when I said: 

This will make it easier to enforce consistent build flag usage across all of Fedora.

One possible use case in the future would be to add CFLAGS and CXXFLAGS as arguments,
this would ensure that projects using make without a build system generator
like autoconf or cmake would use the correct build flags.

Another use case could be to set the CC and CXX variables to make explicit the use
of the Fedora system compiler and also ensure that packages using a different compiler
are doing so in the correct way.

This change is about more than just enabling parallel builds, so I do think
we should keep the macro name the same.

As for using this macro in the %check section.  There are 2 advantages to doing this:

1. Some packages build code when running `make check`, so parallelizing this will still
be a benefit.

2. This will ensure that code built using `make check` uses the same compiler/compiler
flags
that was used for the default make target.

-Tom
 

>  
>  Björn Persson
>  
>  
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Tom Stellard]

On 06/19/2020 03:37 PM, Miro Hrončok wrote:
>  On 19. 06. 20 23:11, Ben Cotton wrote:
> > All make invocations in spec files that don't use the install target will be
> modified to use the %make_build macro
>  
>  Many Python packages build Sphinx documentation with variant of "make html".
> Such invocation will always be just 1 job. Hence there is no benefit of parallel make.
> Replacing it with %make_build will only make it harder to read.
>  
>  Can we exclude such cases? Or do we want all make invocations to be mecronized, even if
> there is no benefit?
>   
The reason I put in the proposal that all make invocations would be updated,
is because this is easier to script and it would be hard for someone who
doesn't know the package to make the call about which invocations don't
need to use parallel make.

However, this issue is also part of the reason I included a 1 week review period,
for each change.  I think it would be reasonable for a maintainer to
reject a change for something like `make html` if they felt it was making
the spec file worse with no benefit.

What if I updated the proposed documentation change to say that
%make_build must be used for 'code building targets', check, and install,
but is optional otherwise?

-Tom

[Miro Hrončok]

On 22. 06. 20 17:21, Tom Stellard wrote:
>  On 06/19/2020 03:37 PM, Miro Hrončok wrote:
> > On 19. 06. 20 23:11, Ben Cotton wrote:
> >> All make invocations in spec files that don't use the install target will be
> modified to use the %make_build macro
> >
> > Many Python packages build Sphinx documentation with variant of "make
> html". Such invocation will always be just 1 job. Hence there is no benefit of
> parallel make. Replacing it with %make_build will only make it harder to read.
> >
> > Can we exclude such cases? Or do we want all make invocations to be mecronized, even
> if there is no benefit?
> >
>  
>  The reason I put in the proposal that all make invocations would be updated,
>  is because this is easier to script and it would be hard for someone who
>  doesn't know the package to make the call about which invocations don't
>  need to use parallel make.
>  
>  However, this issue is also part of the reason I included a 1 week review period,
>  for each change.  I think it would be reasonable for a maintainer to
>  reject a change for something like `make html` if they felt it was making
>  the spec file worse with no benefit. 
And that works, however I am trying to save you some work as well :)

>  What if I updated the proposed documentation change to say that
>  %make_build must be used for 'code building targets', check, and install,
>  but is optional otherwise? 
That would certainly %make_build more sense to me, thanks.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Björn Persson]

Tom Stellard wrote:
>  The reason I put in the proposal that all make invocations would be
> updated,
>  is because this is easier to script and it would be hard for someone who
>  doesn't know the package to make the call about which invocations don't
>  need to use parallel make. 
It's often tempting to do the easy thing instead of the right thing.

Björn Persson

[Tom Stellard]

On 06/22/2020 10:00 AM, Björn Persson wrote:
>  Tom Stellard wrote:
> > The reason I put in the proposal that all make invocations would be updated,
> > is because this is easier to script and it would be hard for someone who
> > doesn't know the package to make the call about which invocations don't
> > need to use parallel make.
>  
>  It's often tempting to do the easy thing instead of the right thing.
>   
What do you think would be the right thing to do in this case?

-Tom

>  Björn Persson
>  
>  
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Björn Persson]

Tom Stellard wrote:
>  On 06/22/2020 10:00 AM, Björn Persson wrote:
>  > Tom Stellard wrote:  
>  >> The reason I put in the proposal that all make invocations would be updated,
>  >> is because this is easier to script and it would be hard for someone who
>  >> doesn't know the package to make the call about which invocations don't
>  >> need to use parallel make.  
>  > 
>  > It's often tempting to do the easy thing instead of the right thing.
>  
>  What do you think would be the right thing to do in this case? 
To study each make invocation, determine whether it's parallelizable
and judge whether future build flags will be appropriate – which is
obviously much more work for the person pushing the change, but avoids
making work for package maintainers.

Björn Persson

[Tom Stellard]

On 06/22/2020 10:54 AM, Björn Persson wrote:
>  Tom Stellard wrote:
> > On 06/22/2020 10:00 AM, Björn Persson wrote:
> >> Tom Stellard wrote:  
> >>> The reason I put in the proposal that all make invocations would be updated,
> >>> is because this is easier to script and it would be hard for someone who
> >>> doesn't know the package to make the call about which invocations
> don't
> >>> need to use parallel make.  
> >>
> >> It's often tempting to do the easy thing instead of the right thing.
> >
> > What do you think would be the right thing to do in this case?
>  
>  To study each make invocation, determine whether it's parallelizable
>  and judge whether future build flags will be appropriate – which is
>  obviously much more work for the person pushing the change, but avoids
>  making work for package maintainers.
>   
I think that even if the change owner does the analysis themselves, the
package maintainer is still going to want to review the changes.  So,
I don't know if doing it this way would actually save maintainers work.

-Tom

>  Björn Persson
>  
>  
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Dan Čermák]

Ben Cotton <bcotton(a)redhat.com&gt; writes:

> https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro
>
> == Summary ==
> This change will update all spec files in Fedora that use make and replace
> the make invocations with either the %make_build or %make_install macros.
>
> == Owner ==
> * Name: [[User:tstellar| Tom Stellard]]
> * Email: <tstellar(a)redhat.com&gt;
>
>
> == Detailed Description ==
>
> The goal of this change is to standardize the usage of make in Fedora.  All
> make invocations in spec files that don't use the install target will be
> modified to use the %make_build macro, and all make invocations that use
> the install target will be updated to use the %make_install macro.  Any
> additional arguments to make that are not included in either the
> %make_build and %make_install will be preserved.
>
> The %make_build macro enables parallel make builds using the -j option.  In
> case a package does not build correctly with parallel make, then parallel
> make will be disabled for that package by redefining the %_smp_mflags macro
> like this:
>
>   %global _smp_mflags -j1
>
> All changes will be submitted to dist-git repositories via pull requests.
> Pull requests will be merged after 1 week if there are no objections or
> earlier if approved by the package maintainers.
>
> A script will be used to apply the necessary changes to the spec files, and
> the result will be manually inspected before being merged.
>
> All packages updated by this change request will be rebuilt after the spec
> file changes are merged.
>
> Some packages already use the %make_build and %make_install macros.  These
> packages will be left unchanged.
>
> == Benefit to Fedora ==
> * Reduced build times: Using the %make_build macros enables parallel make
> builds which will reduce build times for Fedora packages.
>
> * This will make it easier to enforce consistent build flag usage across
> all of Fedora.
>
> == Scope ==
> * Proposal owners: Update spec files and submit pull requests and do new
> package builds.  Optional: Merge pull requests (Proposal Owner would need
> to request to be added to provenpackagers group)
>
> * Other developers: Package owners may merge pull requests and submit new
> builds if they want, but this is not required for them.  A member of the
> provenpackagers group will be needed to merge pull requests.
> * Release engineering: [https://pagure.io/releng/issues/9533 #9533]
>
> * Policies and guidelines: Package guidelines already specify that packages
> should use these macros when possible.  Documentation will be updated to
> clarify that %make_build should be used for all make invocations (besides
> make install), and also with instructions for packages that fail to build
> with parallel make.
> * Trademark approval: N/A (not needed for this Change)
>
>
> == Upgrade/compatibility impact ==
> No impact.
>
> == How To Test ==
> End-users will not notice any changes.
>
> == Dependencies ==
> No real dependencies, each package can be updated independently.
>
> == Contingency Plan ==
> * Contingency mechanism: None.  If not all packages are updated in time,
> then the work can continue into the next release.
> * Contingency deadline: All work will be done in the rawhide branch, and
> will not be backported into the f33 branch once it is created, so whatever
> gets done before the branch date will make it into the release.
> * Blocks release? No
>
> == Documentation ==
> The packaging guidelines will be updated as described in the Scope Section.
>
>
>
> -- 
> Ben Cotton
> He / Him / His
> Senior Program Manager, Fedora & CentOS Stream
> Red Hat
> TZ=America/Indiana/Indianapolis
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Dan Čermák]

Sorry about the empty email, I've hit send too fast…

Anyway, on the topic of parallel builds: what is everyone's opinion on
adding the %limit_build macro from openSUSE (see:
https://build.opensuse.org/package/view_file/network:chromium/memory-cons...

tl;dr; %limit_build -m 1500 will set the number of parallel processes so
that if all of them consume 1500 MB of RAM at most, that they will not
OOM your worker (especially handy on workers with many cores, but not
much RAM).


Cheers,

Dan

Dan Čermák <dan.cermak(a)cgc-instruments.com&gt; writes:

>  Ben Cotton <bcotton(a)redhat.com&gt; writes:
>
> > https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro
> >
> > == Summary ==
> > This change will update all spec files in Fedora that use make and replace
> > the make invocations with either the %make_build or %make_install macros.
> >
> > == Owner ==
> > * Name: [[User:tstellar| Tom Stellard]]
> > * Email: <tstellar(a)redhat.com&gt;
> >
> >
> > == Detailed Description ==
> >
> > The goal of this change is to standardize the usage of make in Fedora.  All
> > make invocations in spec files that don't use the install target will be
> > modified to use the %make_build macro, and all make invocations that use
> > the install target will be updated to use the %make_install macro.  Any
> > additional arguments to make that are not included in either the
> > %make_build and %make_install will be preserved.
> >
> > The %make_build macro enables parallel make builds using the -j option.  In
> > case a package does not build correctly with parallel make, then parallel
> > make will be disabled for that package by redefining the %_smp_mflags macro
> > like this:
> >
> >   %global _smp_mflags -j1
> >
> > All changes will be submitted to dist-git repositories via pull requests.
> > Pull requests will be merged after 1 week if there are no objections or
> > earlier if approved by the package maintainers.
> >
> > A script will be used to apply the necessary changes to the spec files, and
> > the result will be manually inspected before being merged.
> >
> > All packages updated by this change request will be rebuilt after the spec
> > file changes are merged.
> >
> > Some packages already use the %make_build and %make_install macros.  These
> > packages will be left unchanged.
> >
> > == Benefit to Fedora ==
> > * Reduced build times: Using the %make_build macros enables parallel make
> > builds which will reduce build times for Fedora packages.
> >
> > * This will make it easier to enforce consistent build flag usage across
> > all of Fedora.
> >
> > == Scope ==
> > * Proposal owners: Update spec files and submit pull requests and do new
> > package builds.  Optional: Merge pull requests (Proposal Owner would need
> > to request to be added to provenpackagers group)
> >
> > * Other developers: Package owners may merge pull requests and submit new
> > builds if they want, but this is not required for them.  A member of the
> > provenpackagers group will be needed to merge pull requests.
> > * Release engineering: [https://pagure.io/releng/issues/9533 #9533]
> >
> > * Policies and guidelines: Package guidelines already specify that packages
> > should use these macros when possible.  Documentation will be updated to
> > clarify that %make_build should be used for all make invocations (besides
> > make install), and also with instructions for packages that fail to build
> > with parallel make.
> > * Trademark approval: N/A (not needed for this Change)
> >
> >
> > == Upgrade/compatibility impact ==
> > No impact.
> >
> > == How To Test ==
> > End-users will not notice any changes.
> >
> > == Dependencies ==
> > No real dependencies, each package can be updated independently.
> >
> > == Contingency Plan ==
> > * Contingency mechanism: None.  If not all packages are updated in time,
> > then the work can continue into the next release.
> > * Contingency deadline: All work will be done in the rawhide branch, and
> > will not be backported into the f33 branch once it is created, so whatever
> > gets done before the branch date will make it into the release.
> > * Blocks release? No
> >
> > == Documentation ==
> > The packaging guidelines will be updated as described in the Scope Section.
> >
> >
> >
> > -- 
> > Ben Cotton
> > He / Him / His
> > Senior Program Manager, Fedora & CentOS Stream
> > Red Hat
> > TZ=America/Indiana/Indianapolis
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Tomáš Popela]

On Sat, Jun 20, 2020 at 11:24 AM Dan Čermák <dan.cermak(a)cgc-instruments.com&gt;
wrote:

>  Sorry about the empty email, I've hit send too fast…
>
>  Anyway, on the topic of parallel builds: what is everyone's opinion on
>  adding the %limit_build macro from openSUSE (see:
>
>
> https://build.opensuse.org/package/view_file/network:chromium/memory-cons...
>  )?
>
>  tl;dr; %limit_build -m 1500 will set the number of parallel processes so
>  that if all of them consume 1500 MB of RAM at most, that they will not
>  OOM your worker (especially handy on workers with many cores, but not
>  much RAM).
>  
We are being hit by this in WebKitGTK and in Firefox/Thunderbird from time
to time. There is an RFE for RPM to solve it -
https://bugzilla.redhat.com/show_bug.cgi?id=1118734.

Bye,
Tom

[Tom Stellard]

On 06/20/2020 02:23 AM, Dan Čermák wrote:
>  Sorry about the empty email, I've hit send too fast…
>  
>  Anyway, on the topic of parallel builds: what is everyone's opinion on
>  adding the %limit_build macro from openSUSE (see:
>
> https://build.opensuse.org/package/view_file/network:chromium/memory-cons...
>  
>  tl;dr; %limit_build -m 1500 will set the number of parallel processes so
>  that if all of them consume 1500 MB of RAM at most, that they will not
>  OOM your worker (especially handy on workers with many cores, but not
>  much RAM).
>   
I would be in favor of this.  We run into this issue a lot when building
the LLVM packages.

-Tom

>  
>  Cheers,
>  
>  Dan
>  
>  Dan Čermák <dan.cermak(a)cgc-instruments.com&gt; writes:
>  
> > Ben Cotton <bcotton(a)redhat.com&gt; writes:
> >
> >> https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro
> >>
> >> == Summary ==
> >> This change will update all spec files in Fedora that use make and replace
> >> the make invocations with either the %make_build or %make_install macros.
> >>
> >> == Owner ==
> >> * Name: [[User:tstellar| Tom Stellard]]
> >> * Email: <tstellar(a)redhat.com&gt;
> >>
> >>
> >> == Detailed Description ==
> >>
> >> The goal of this change is to standardize the usage of make in Fedora.  All
> >> make invocations in spec files that don't use the install target will be
> >> modified to use the %make_build macro, and all make invocations that use
> >> the install target will be updated to use the %make_install macro.  Any
> >> additional arguments to make that are not included in either the
> >> %make_build and %make_install will be preserved.
> >>
> >> The %make_build macro enables parallel make builds using the -j option.  In
> >> case a package does not build correctly with parallel make, then parallel
> >> make will be disabled for that package by redefining the %_smp_mflags macro
> >> like this:
> >>
> >>   %global _smp_mflags -j1
> >>
> >> All changes will be submitted to dist-git repositories via pull requests.
> >> Pull requests will be merged after 1 week if there are no objections or
> >> earlier if approved by the package maintainers.
> >>
> >> A script will be used to apply the necessary changes to the spec files, and
> >> the result will be manually inspected before being merged.
> >>
> >> All packages updated by this change request will be rebuilt after the spec
> >> file changes are merged.
> >>
> >> Some packages already use the %make_build and %make_install macros.  These
> >> packages will be left unchanged.
> >>
> >> == Benefit to Fedora ==
> >> * Reduced build times: Using the %make_build macros enables parallel make
> >> builds which will reduce build times for Fedora packages.
> >>
> >> * This will make it easier to enforce consistent build flag usage across
> >> all of Fedora.
> >>
> >> == Scope ==
> >> * Proposal owners: Update spec files and submit pull requests and do new
> >> package builds.  Optional: Merge pull requests (Proposal Owner would need
> >> to request to be added to provenpackagers group)
> >>
> >> * Other developers: Package owners may merge pull requests and submit new
> >> builds if they want, but this is not required for them.  A member of the
> >> provenpackagers group will be needed to merge pull requests.
> >> * Release engineering: [https://pagure.io/releng/issues/9533 #9533]
> >>
> >> * Policies and guidelines: Package guidelines already specify that packages
> >> should use these macros when possible.  Documentation will be updated to
> >> clarify that %make_build should be used for all make invocations (besides
> >> make install), and also with instructions for packages that fail to build
> >> with parallel make.
> >> * Trademark approval: N/A (not needed for this Change)
> >>
> >>
> >> == Upgrade/compatibility impact ==
> >> No impact.
> >>
> >> == How To Test ==
> >> End-users will not notice any changes.
> >>
> >> == Dependencies ==
> >> No real dependencies, each package can be updated independently.
> >>
> >> == Contingency Plan ==
> >> * Contingency mechanism: None.  If not all packages are updated in time,
> >> then the work can continue into the next release.
> >> * Contingency deadline: All work will be done in the rawhide branch, and
> >> will not be backported into the f33 branch once it is created, so whatever
> >> gets done before the branch date will make it into the release.
> >> * Blocks release? No
> >>
> >> == Documentation ==
> >> The packaging guidelines will be updated as described in the Scope Section.
> >>
> >>
> >>
> >> -- 
> >> Ben Cotton
> >> He / Him / His
> >> Senior Program Manager, Fedora & CentOS Stream
> >> Red Hat
> >> TZ=America/Indiana/Indianapolis
> >> _______________________________________________
> >> devel mailing list -- devel(a)lists.fedoraproject.org
> >> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> >>
> >>
> >> _______________________________________________
> >> devel mailing list -- devel(a)lists.fedoraproject.org
> >> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Zbigniew Jędrzejewski-Szmek]

On Fri, Jun 19, 2020 at 05:11:43PM -0400, Ben Cotton wrote:
>  The %make_build macro enables parallel make builds using the -j
> option.  In
>  case a package does not build correctly with parallel make, then parallel
>  make will be disabled for that package by redefining the %_smp_mflags macro
>  like this:
>  
>    %global _smp_mflags -j1 
That is rather unwiely, in particular because it's quite common for
just *some* make invocations to fail in parallel mode. (E.g. the main
build works, but doc build fails, or install, etc.)

Why not specify the override for the individual invocation:
%make_build -j1
%make_install -j1
%make_build -C docs -j1

Zbyszek

[Neal Gompa]

On Sat, Jun 20, 2020 at 2:16 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Fri, Jun 19, 2020 at 05:11:43PM -0400, Ben Cotton wrote:
>  > The %make_build macro enables parallel make builds using the -j option.  In
>  > case a package does not build correctly with parallel make, then parallel
>  > make will be disabled for that package by redefining the %_smp_mflags macro
>  > like this:
>  >
>  >   %global _smp_mflags -j1
>
>  That is rather unwiely, in particular because it's quite common for
>  just *some* make invocations to fail in parallel mode. (E.g. the main
>  build works, but doc build fails, or install, etc.)
>
>  Why not specify the override for the individual invocation:
>  %make_build -j1
>  %make_install -j1
>  %make_build -C docs -j1
>  
I think I'd be more comfortable with making all "make %{?_smp_mflags}"
calls switch to "%make_build" and leaving the rest alone.

As for "make DESTDIR=%{buildroot} install" to "%make_install", I'm
fine with it.

-- 
真実はいつも一つ！/ Always, there's only one truth!

[Tom Stellard]

On 06/20/2020 11:20 AM, Neal Gompa wrote:
>  On Sat, Jun 20, 2020 at 2:16 PM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
> >
> > On Fri, Jun 19, 2020 at 05:11:43PM -0400, Ben Cotton wrote:
> >> The %make_build macro enables parallel make builds using the -j option.  In
> >> case a package does not build correctly with parallel make, then parallel
> >> make will be disabled for that package by redefining the %_smp_mflags macro
> >> like this:
> >>
> >>   %global _smp_mflags -j1
> >
> > That is rather unwiely, in particular because it's quite common for
> > just *some* make invocations to fail in parallel mode. (E.g. the main
> > build works, but doc build fails, or install, etc.)
> >
> > Why not specify the override for the individual invocation:
> > %make_build -j1
> > %make_install -j1
> > %make_build -C docs -j1
> >
>  
>  I think I'd be more comfortable with making all "make %{?_smp_mflags}"
>  calls switch to "%make_build" and leaving the rest alone.
>   
Is this because you are concerned about build failures caused by
enabling parallel builds for packages that aren't currently using
that?

-Tom

>  As for "make DESTDIR=%{buildroot} install" to
> "%make_install", I'm fine with it.
>

[Neal Gompa]

On Mon, Jun 22, 2020 at 11:51 AM Tom Stellard <tstellar(a)redhat.com&gt; wrote:
>  On 06/20/2020 11:20 AM, Neal Gompa wrote:
>  > On Sat, Jun 20, 2020 at 2:16 PM Zbigniew Jędrzejewski-Szmek
>  > <zbyszek(a)in.waw.pl&gt; wrote:
>  >>
>  >> On Fri, Jun 19, 2020 at 05:11:43PM -0400, Ben Cotton wrote:
>  >>> The %make_build macro enables parallel make builds using the -j option.  In
>  >>> case a package does not build correctly with parallel make, then parallel
>  >>> make will be disabled for that package by redefining the %_smp_mflags macro
>  >>> like this:
>  >>>
>  >>>   %global _smp_mflags -j1
>  >>
>  >> That is rather unwiely, in particular because it's quite common for
>  >> just *some* make invocations to fail in parallel mode. (E.g. the main
>  >> build works, but doc build fails, or install, etc.)
>  >>
>  >> Why not specify the override for the individual invocation:
>  >> %make_build -j1
>  >> %make_install -j1
>  >> %make_build -C docs -j1
>  >>
>  >
>  > I think I'd be more comfortable with making all "make
> %{?_smp_mflags}"
>  > calls switch to "%make_build" and leaving the rest alone.
>  >
>
>  Is this because you are concerned about build failures caused by
>  enabling parallel builds for packages that aren't currently using
>  that?
>  
Yes, and because Make is often used for non-compilation things and
those aren't necessarily multi-threaded, and passing mflags there can
break things.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Tom Stellard]

On 06/22/2020 10:14 AM, Neal Gompa wrote:
>  On Mon, Jun 22, 2020 at 11:51 AM Tom Stellard
> <tstellar(a)redhat.com&gt; wrote:
> >
> > On 06/20/2020 11:20 AM, Neal Gompa wrote:
> >> On Sat, Jun 20, 2020 at 2:16 PM Zbigniew Jędrzejewski-Szmek
> >> <zbyszek(a)in.waw.pl&gt; wrote:
> >>>
> >>> On Fri, Jun 19, 2020 at 05:11:43PM -0400, Ben Cotton wrote:
> >>>> The %make_build macro enables parallel make builds using the -j option. 
> In
> >>>> case a package does not build correctly with parallel make, then
> parallel
> >>>> make will be disabled for that package by redefining the %_smp_mflags
> macro
> >>>> like this:
> >>>>
> >>>>   %global _smp_mflags -j1
> >>>
> >>> That is rather unwiely, in particular because it's quite common for
> >>> just *some* make invocations to fail in parallel mode. (E.g. the main
> >>> build works, but doc build fails, or install, etc.)
> >>>
> >>> Why not specify the override for the individual invocation:
> >>> %make_build -j1
> >>> %make_install -j1
> >>> %make_build -C docs -j1
> >>>
> >>
> >> I think I'd be more comfortable with making all "make
> %{?_smp_mflags}"
> >> calls switch to "%make_build" and leaving the rest alone.
> >>
> >
> > Is this because you are concerned about build failures caused by
> > enabling parallel builds for packages that aren't currently using
> > that?
> >
>  
>  Yes, and because Make is often used for non-compilation things and
>  those aren't necessarily multi-threaded, and passing mflags there can
>  break things.
>  
>   
If replacing 'make %{?_smp_mflags}' with %make_build is the only change
that is done, then I don't think this change proposal would be worth
doing.  This would eliminate one of the benefits (increased build parallelism),
and also not help that much with the other stated benefit of enforcing
consistent build flag usage, since many packages would sill be using just 'make'.

Another possible solution to the issues you raise would be to restrict the
scope of this proposal to only make 'build', check, and install targets.

-Tom

[Panu Matilainen]

On 6/20/20 9:14 PM, Zbigniew Jędrzejewski-Szmek wrote:
>  On Fri, Jun 19, 2020 at 05:11:43PM -0400, Ben Cotton wrote:
> > The %make_build macro enables parallel make builds using the -j option.  In
> > case a package does not build correctly with parallel make, then parallel
> > make will be disabled for that package by redefining the %_smp_mflags macro
> > like this:
> >
> >    %global _smp_mflags -j1 
Please use this instead:

%global _smp_ncpus_max 1

That allows the same thing to be abstracted to whatever make system 
might be running underneath, some of which have different semantics wrt 
-j arguments or use a different switch altogether etc.

>  
>  That is rather unwiely, in particular because it's quite common for
>  just *some* make invocations to fail in parallel mode. (E.g. the main
>  build works, but doc build fails, or install, etc.)
>  
>  Why not specify the override for the individual invocation:
>  %make_build -j1
>  %make_install -j1
>  %make_build -C docs -j1 
This kind of thing gets easily lost in the noise, whereas a macro 
definition is easy to spot and grep for.

	- Panu -

>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Tom Stellard]

On 06/20/2020 11:14 AM, Zbigniew Jędrzejewski-Szmek wrote:
>  On Fri, Jun 19, 2020 at 05:11:43PM -0400, Ben Cotton wrote:
> > The %make_build macro enables parallel make builds using the -j option.  In
> > case a package does not build correctly with parallel make, then parallel
> > make will be disabled for that package by redefining the %_smp_mflags macro
> > like this:
> >
> >   %global _smp_mflags -j1
>  
>  That is rather unwiely, in particular because it's quite common for
>  just *some* make invocations to fail in parallel mode. (E.g. the main
>  build works, but doc build fails, or install, etc.)
>  
>  Why not specify the override for the individual invocation:
>  %make_build -j1
>  %make_install -j1
>  %make_build -C docs -j1
>   
I agree, this would be better.  I am going to update the proposal to
include this.

-Tom

>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Kevin Kofler]

Ben Cotton wrote:
>  == Summary ==
>  This change will update all spec files in Fedora that use make and replace
>  the make invocations with either the %make_build or %make_install macros.
>  
>  == Owner ==
>  * Name: [[User:tstellar| Tom Stellard]]
>  * Email: <tstellar(a)redhat.com&gt; 
I am opposed to this change.

Unlike other distros, it has always been Fedora policy to avoid this kind of 
pointless macros and just spell things out, in the name of readability. I 
think that should still be the way to go, also considering that:

>  The %make_build macro enables parallel make builds using the -j
> option. 
>  In case a package does not build correctly with parallel make, then
>  parallel make will be disabled for that package by redefining the
>  %_smp_mflags macro like this:
>  
>    %global _smp_mflags -j1 
This does not work when some make targets (typically, the default "all" 
target) support parallel make and others don't.

Additionally, how do you determine that "a package does not build correctly 
with parallel make"? Parallel make failures are race conditions and thus not 
reproducible. The builds can fail or succeed at random. The only safe 
approach is to assume that a package that does not use %{_smp_mflags} at 
this time probably omits it for a reason.

>  All changes will be submitted to dist-git repositories via pull
> requests.
>  Pull requests will be merged after 1 week if there are no objections or
>  earlier if approved by the package maintainers. 
What is the point of making such a purely cosmetic change as a mass change 
to all packages, with the potential of breaking things?

>  == Benefit to Fedora ==
>  * Reduced build times: Using the %make_build macros enables parallel make
>  builds which will reduce build times for Fedora packages. 
That is not true. The guidelines and templates clearly specify use of 
%{_smp_mflags}, so IMHO it is safe to assume that those packages that 
actually support parallel build already use %{_smp_mflags} and hence will 
not start building any faster.

As for those packages that do not currently use parallel build, see above: 
you have to assume that they omit it for a reason. It is unsafe to enable it 
behind the maintainer's back (giving them only 1 week to object, if they 
even realize that %make_build invisibly enables %{_smp_mflags} to begin with 
– IMHO, such hidden flags must be considered harmful).

>  * This will make it easier to enforce consistent build flag usage
> across
>  all of Fedora. 
What kind of build flags? Compiler flags? The valid way to set those depends 
on the build system used. Passing make CFLAGS="…" with no knowledge of what 
the makefile looks like is inherently unsafe: it can override required flags 
set by the build system or a handwritten makefile, it can also end up being 
ignored entirely. Usually, the right way to set flags is before the build 
system or configure script invocation, not in the make step at all. (The 
build system then hardcodes the flags in the makefile, so you do not have to 
pass them again in the make step.)

        Kevin Kofler

[Tom Stellard]

On 06/20/2020 05:11 PM, Kevin Kofler wrote:
>  Ben Cotton wrote:
> > == Summary ==
> > This change will update all spec files in Fedora that use make and replace
> > the make invocations with either the %make_build or %make_install macros.
> >
> > == Owner ==
> > * Name: [[User:tstellar| Tom Stellard]]
> > * Email: <tstellar(a)redhat.com&gt;
>  
>  I am opposed to this change.
>  
>  Unlike other distros, it has always been Fedora policy to avoid this kind of 
>  pointless macros and just spell things out, in the name of readability. I 
>  think that should still be the way to go, also considering that:
>   
The benefits for using the macro are mentioned in the proposal, so this is not
just a cosmetic change.  I think making it easier to standardize the build
flags in Fedora is an important benefit as we currently have a gap in the
project where packages that use make without a build system generator 
like autoconf or cmake do not have the correct build flags added automatically.

> > The %make_build macro enables parallel make builds using the -j
> option. 
> > In case a package does not build correctly with parallel make, then
> > parallel make will be disabled for that package by redefining the
> > %_smp_mflags macro like this:
> >
> >   %global _smp_mflags -j1
>  
>  This does not work when some make targets (typically, the default "all" 
>  target) support parallel make and others don't.
>   
I agree with this, and I will update the proposal to use %make_build -j1 instead.

>  Additionally, how do you determine that "a package does not
> build correctly 
>  with parallel make"? Parallel make failures are race conditions and thus not 
>  reproducible. The builds can fail or succeed at random. The only safe 
>  approach is to assume that a package that does not use %{_smp_mflags} at 
>  this time probably omits it for a reason.
>   
Packages will be tested using the koji-simple-ci job, which does a scratch
build when a pull request is submitted.  You are correct that this is not
a 100% guarantee to catch parallel build issues.  However, the downside
of missing a parallel build issue is very limited as package maintainers
can easily undo this change.

> > All changes will be submitted to dist-git repositories via pull
> requests.
> > Pull requests will be merged after 1 week if there are no objections or
> > earlier if approved by the package maintainers.
>  
>  What is the point of making such a purely cosmetic change as a mass change 
>  to all packages, with the potential of breaking things?
>  
> > == Benefit to Fedora ==
> > * Reduced build times: Using the %make_build macros enables parallel make
> > builds which will reduce build times for Fedora packages.
>  
>  That is not true. The guidelines and templates clearly specify use of 
>  %{_smp_mflags}, so IMHO it is safe to assume that those packages that 
>  actually support parallel build already use %{_smp_mflags} and hence will 
>  not start building any faster.
>   
I have not found this to be the case.  I have implemented the proposed change
in some packages already and I found packages that were not using %{_smp_mflags}
that did benefit from parallel builds.


>  As for those packages that do not currently use parallel build, see
> above: 
>  you have to assume that they omit it for a reason. It is unsafe to enable it 
>  behind the maintainer's back (giving them only 1 week to object, if they 
>  even realize that %make_build invisibly enables %{_smp_mflags} to begin with 
>  – IMHO, such hidden flags must be considered harmful).
>  
> > * This will make it easier to enforce consistent build flag usage across
> > all of Fedora.
>  
>  What kind of build flags? Compiler flags? The valid way to set those depends 
>  on the build system used. Passing make CFLAGS="…" with no knowledge of what 
>  the makefile looks like is inherently unsafe: it can override required flags 
>  set by the build system or a handwritten makefile, it can also end up being 
>  ignored entirely. Usually, the right way to set flags is before the build 
>  system or configure script invocation, not in the make step at all. (The 
>  build system then hardcodes the flags in the makefile, so you do not have to 
>  pass them again in the make step.) 
I'm thinking about compiler flags, like CFLAGS or even possibly compiler
variables, like CC and CXX.  And they would not necessarily be passed as
arguments directly to make.  One option would be to have %make_build
call the %set_build_flags macro which sets up environment variables with
the correct flags.

-Tom

>  
>          Kevin Kofler
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Kevin Kofler]

Tom Stellard wrote:
>  The benefits for using the macro are mentioned in the proposal, so
> this is
>  not just a cosmetic change. 
I have addressed those points further down in my message: I do not believe 
that the proposed change will achieve the mentioned goals, and I think the 
side effects are worse than what it can fix. See my detailed replies in the 
remainder of both my previous message and this message.

>  I think making it easier to standardize the build flags in Fedora is
> an
>  important benefit as we currently have a gap in the project where packages
>  that use make without a build system generator like autoconf or cmake do
>  not have the correct build flags added automatically. 
But there is no way RPM macros can address this problem, because there is no 
standardized way to pass build flags to a handwritten makefile:
* The makefile may or may not honor environment variables. And passing -e to
  make to try to force it may or may not work. (It can cause more problems
  than it solves. In particular, -e is always detrimental for packages that
  actually do use a makefile generator.)
* The makefile may or may not use standardized variable names such as
  CFLAGS. I have seen makefiles hardcode the flags entirely or use arbitrary
  names, such as COMPILER_FLAGS, GCC_FLAGS, COMPILER_OPTIONS or whatever.
  (Or worse, they may use standard variable names with subtly different
  semantics, e.g., CPP_FLAGS for C++ compiler flags.)
* Required flags may or may not be defined within CFLAGS. If they are, then
  just overriding CFLAGS with the distro defaults will likely break the
  package's build.

Handwritten makefiles are exactly the case where it is NOT safe to use a 
stock macro such as %make_build. Such standardized macros can only possibly 
work for packages using a popular build system, and for those, the configure 
step is the right place to apply build flags.

>  Packages will be tested using the koji-simple-ci job, which does a
> scratch
>  build when a pull request is submitted.  You are correct that this is not
>  a 100% guarantee to catch parallel build issues.  However, the downside
>  of missing a parallel build issue is very limited as package maintainers
>  can easily undo this change. 
Sorry, I do not consider it acceptable to mass-break packages by default and 
to require maintainers to explicitly unbreak them.

The only safe way is to assume that packages that do not use %{_smp_mflags} 
omit it for a reason.

>  I have not found this to be the case.  I have implemented the
> proposed
>  change in some packages already and I found packages that were not using
>  %{_smp_mflags} that did benefit from parallel builds. 
Then file bugs against those packages asking them to add %{_smp_mflags}. The 
guidelines explicitly state that it should always be used if it works. 

But don't be surprised if many of those bugs get closed as NOTABUG because 
the %{_smp_mflags} were omitted or removed for a reason that you were not 
aware of.

>  I'm thinking about compiler flags, like CFLAGS or even possibly
> compiler
>  variables, like CC and CXX.  And they would not necessarily be passed as
>  arguments directly to make.  One option would be to have %make_build
>  call the %set_build_flags macro which sets up environment variables with
>  the correct flags. 
Most handwritten makefiles ignore environment variables entirely. Something 
like:
CFLAGS=-foo
or even:
CFLAGS+=-foo
takes precedence over environment variables. (Unless you pass make -e, which 
will make it ignore even the += lines, and which can also cause other 
problems.)

Only ?= honors environment variables, and many handwritten makefiles do not 
use it, or at least not consistently.

And handwritten makefiles also do not necessarily use the conventional names 
(CFLAGS, CC, CXX, etc.). They can use arbitrary variable names, or no 
variables at all (i.e., everything hardcoded).

        Kevin Kofler

[Petr Pisar]

On Fri, Jun 19, 2020 at 05:11:43PM -0400, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro
>  
>  == Summary ==
>  This change will update all spec files in Fedora that use make and replace
>  the make invocations with either the %make_build or %make_install macros.
>   
This a great oportunity to speak about an implicit dependency on make covered
by RPM macros. What package should define the dependency explicitly? A spec
file that calls %make_build, or rpm package that defines the macro?

-- Petr

[Vít Ondruch]

I wonder if there is PR with the implementation somewhere or is this
just dry theoretical discussion O:-)



Vít


Dne 19. 06. 20 v 23:11 Ben Cotton napsal(a):
>  https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro
>
>  == Summary ==
>  This change will update all spec files in Fedora that use make and
>  replace the make invocations with either the %make_build or
>  %make_install macros.
>
>  == Owner ==
>  * Name: [[User:tstellar| Tom Stellard]]
>  * Email: <tstellar(a)redhat.com <mailto:[e-mail]>>
>
>
>  == Detailed Description ==
>
>  The goal of this change is to standardize the usage of make in
>  Fedora.  All make invocations in spec files that don't use the install
>  target will be modified to use the %make_build macro, and all make
>  invocations that use the install target will be updated to use the
>  %make_install macro.  Any additional arguments to make that are not
>  included in either the %make_build and %make_install will be preserved.
>
>  The %make_build macro enables parallel make builds using the -j
>  option.  In case a package does not build correctly with parallel
>  make, then parallel make will be disabled for that package by
>  redefining the %_smp_mflags macro like this:
>
>    %global _smp_mflags -j1
>
>  All changes will be submitted to dist-git repositories via pull
>  requests.  Pull requests will be merged after 1 week if there are no
>  objections or earlier if approved by the package maintainers.
>
>  A script will be used to apply the necessary changes to the spec
>  files, and the result will be manually inspected before being merged.
>
>  All packages updated by this change request will be rebuilt after the
>  spec file changes are merged.
>
>  Some packages already use the %make_build and %make_install macros. 
>  These packages will be left unchanged.
>
>  == Benefit to Fedora ==
>  * Reduced build times: Using the %make_build macros enables parallel
>  make builds which will reduce build times for Fedora packages.
>
>  * This will make it easier to enforce consistent build flag usage
>  across all of Fedora.
>
>  == Scope ==
>  * Proposal owners: Update spec files and submit pull requests and do
>  new package builds.  Optional: Merge pull requests (Proposal Owner
>  would need to request to be added to provenpackagers group)
>
>  * Other developers: Package owners may merge pull requests and submit
>  new builds if they want, but this is not required for them.  A member
>  of the provenpackagers group will be needed to merge pull requests.
>  * Release engineering: [https://pagure.io/releng/issues/9533 #9533]
>
>  * Policies and guidelines: Package guidelines already specify that
>  packages should use these macros when possible.  Documentation will be
>  updated to clarify that %make_build should be used for all make
>  invocations (besides make install), and also with instructions for
>  packages that fail to build with parallel make.
>  * Trademark approval: N/A (not needed for this Change)
>
>
>  == Upgrade/compatibility impact ==
>  No impact.
>
>  == How To Test ==
>  End-users will not notice any changes.
>
>  == Dependencies ==
>  No real dependencies, each package can be updated independently.
>
>  == Contingency Plan ==
>  * Contingency mechanism: None.  If not all packages are updated in
>  time, then the work can continue into the next release. 
>  * Contingency deadline: All work will be done in the rawhide branch,
>  and will not be backported into the f33 branch once it is created, so
>  whatever gets done before the branch date will make it into the release. 
>  * Blocks release? No
>
>  == Documentation ==
>  The packaging guidelines will be updated as described in the Scope
>  Section.
>
>
>
>  -- 
>  Ben Cotton
>  He / Him / His
>  Senior Program Manager, Fedora & CentOS Stream
>  Red Hat
>  TZ=America/Indiana/Indianapolis
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Tom Stellard]

On 06/24/2020 08:25 AM, Vít Ondruch wrote:
>  I wonder if there is PR with the implementation somewhere or is this
> just dry theoretical discussion O:-)
>  
>   
I have been experimenting with these changes over the past few months,
so there are a number of PRs where I have implemented these changes.  I can
provide a list if you are interested.

-Tom

>  
>  Vít
>  
>  
>  Dne 19. 06. 20 v 23:11 Ben Cotton napsal(a):
> > https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro
> >
> > == Summary ==
> > This change will update all spec files in Fedora that use make and replace the make
> invocations with either the %make_build or %make_install macros.
> >
> > == Owner ==
> > * Name: [[User:tstellar| Tom Stellard]]
> > * Email: <tstellar(a)redhat.com <mailto:[e-mail]>>
> >
> >
> > == Detailed Description ==
> >
> > The goal of this change is to standardize the usage of make in Fedora.  All make
> invocations in spec files that don't use the install target will be modified to use
> the %make_build macro, and all make invocations that use the install target will be
> updated to use the %make_install macro.  Any additional arguments to make that are not
> included in either the %make_build and %make_install will be preserved.
> >
> > The %make_build macro enables parallel make builds using the -j option.  In case a
> package does not build correctly with parallel make, then parallel make will be disabled
> for that package by redefining the %_smp_mflags macro like this:
> >
> >   %global _smp_mflags -j1
> >
> > All changes will be submitted to dist-git repositories via pull requests.  Pull
> requests will be merged after 1 week if there are no objections or earlier if approved by
> the package maintainers.
> >
> > A script will be used to apply the necessary changes to the spec files, and the
> result will be manually inspected before being merged.
> >
> > All packages updated by this change request will be rebuilt after the spec file
> changes are merged.
> >
> > Some packages already use the %make_build and %make_install macros.  These packages
> will be left unchanged.
> >
> > == Benefit to Fedora ==
> > * Reduced build times: Using the %make_build macros enables parallel make builds
> which will reduce build times for Fedora packages.
> >
> > * This will make it easier to enforce consistent build flag usage across all of
> Fedora.
> >
> > == Scope ==
> > * Proposal owners: Update spec files and submit pull requests and do new package
> builds.  Optional: Merge pull requests (Proposal Owner would need to request to be added
> to provenpackagers group)
> >
> > * Other developers: Package owners may merge pull requests and submit new builds if
> they want, but this is not required for them.  A member of the provenpackagers group will
> be needed to merge pull requests.
> > * Release engineering: [https://pagure.io/releng/issues/9533 #9533]
> >
> > * Policies and guidelines: Package guidelines already specify that packages should
> use these macros when possible.  Documentation will be updated to clarify that %make_build
> should be used for all make invocations (besides make install), and also with instructions
> for packages that fail to build with parallel make.
> > * Trademark approval: N/A (not needed for this Change)
> >
> >
> > == Upgrade/compatibility impact ==
> > No impact.
> >
> > == How To Test ==
> > End-users will not notice any changes.
> >
> > == Dependencies ==
> > No real dependencies, each package can be updated independently.
> >
> > == Contingency Plan ==
> > * Contingency mechanism: None.  If not all packages are updated in time, then the
> work can continue into the next release. 
> > * Contingency deadline: All work will be done in the rawhide branch, and will not be
> backported into the f33 branch once it is created, so whatever gets done before the branch
> date will make it into the release. 
> > * Blocks release? No
> >
> > == Documentation ==
> > The packaging guidelines will be updated as described in the Scope Section.
> >
> >
> >
> > -- 
> > Ben Cotton
> > He / Him / His
> > Senior Program Manager, Fedora & CentOS Stream
> > Red Hat
> > TZ=America/Indiana/Indianapolis
> >
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>