RFC: Feature macros (aka USE flags)

[Petr Šabata]

Based on the recent discussions around %fedora/%rhel macros and ELN,
and %bcond generally being confusing to work with, I came up with a
distribution-wide feature that defines generic feature keywords and
associated helper macros that packages can check in build-time
conditionals.

The key advantage here is the defaults are defined by the buildroot,
not the package. The package is just a building block.

I'd like some input to improve this and unless this turns out to be a
really bad idea, I intend to submit it as a change proposal. Even
though the more packages use it the more beneficial it gets, it's, of
course, perfectly optional.

Details in the gist:
https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408

Cheers,
P

[Neal Gompa]

On Mon, Apr 27, 2020 at 7:21 AM Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  Based on the recent discussions around %fedora/%rhel macros and ELN,
>  and %bcond generally being confusing to work with, I came up with a
>  distribution-wide feature that defines generic feature keywords and
>  associated helper macros that packages can check in build-time
>  conditionals.
>
>  The key advantage here is the defaults are defined by the buildroot,
>  not the package. The package is just a building block.
>
>  I'd like some input to improve this and unless this turns out to be a
>  really bad idea, I intend to submit it as a change proposal. Even
>  though the more packages use it the more beneficial it gets, it's, of
>  course, perfectly optional.
>
>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  
This looks pretty good! Have you considered submitting it into
upstream RPM so that we don't wind up having so many catch-22s with
its usability, though?



--
真実はいつも一つ！/ Always, there's only one truth!

[Petr Šabata]

On Mon, Apr 27, 2020 at 1:24 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Mon, Apr 27, 2020 at 7:21 AM Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  >
>  > Based on the recent discussions around %fedora/%rhel macros and ELN,
>  > and %bcond generally being confusing to work with, I came up with a
>  > distribution-wide feature that defines generic feature keywords and
>  > associated helper macros that packages can check in build-time
>  > conditionals.
>  >
>  > The key advantage here is the defaults are defined by the buildroot,
>  > not the package. The package is just a building block.
>  >
>  > I'd like some input to improve this and unless this turns out to be a
>  > really bad idea, I intend to submit it as a change proposal. Even
>  > though the more packages use it the more beneficial it gets, it's, of
>  > course, perfectly optional.
>  >
>  > Details in the gist:
>  > https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  >
>
>  This looks pretty good! Have you considered submitting it into
>  upstream RPM so that we don't wind up having so many catch-22s with
>  its usability, though? 
That would be an option, even though it would have to work somehow
differently, I presume.

P

[Daniel P. Berrangé]

On Mon, Apr 27, 2020 at 01:19:29PM +0200, Petr Šabata wrote:
>  Based on the recent discussions around %fedora/%rhel macros and ELN,
>  and %bcond generally being confusing to work with, I came up with a
>  distribution-wide feature that defines generic feature keywords and
>  associated helper macros that packages can check in build-time
>  conditionals.
>  
>  The key advantage here is the defaults are defined by the buildroot,
>  not the package. The package is just a building block. 
IMHO it is valuable having the package self-contained as it is
today, as both maintainers & users are able to see and control
exactly what features are intended, from a single place.

With this proposal they'd have to read the global.yml and the
local.$PKG.yml and the $PKG.spec file to figure out what is going
to be built. Some features will need to vary per-architecture too,
so this will need a global-$ARCH.yml  and a local-$ARCH.$PKG.yml file
too for each Fedora arch, or the global.yml file will need to ship
different content on each arch somehow. So that's potentially 3-5
files that need to be consulted to figure out what the enabled
features are for a given package build.

>  I'd like some input to improve this and unless this turns out to
> be a
>  really bad idea, I intend to submit it as a change proposal. Even
>  though the more packages use it the more beneficial it gets, it's, of
>  course, perfectly optional. 
>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408 
IIUC, the global.yml file is intended to live in the use-macros
package. It wasn't clear though where the local-$PKG.yml file is
intended to live ? Is it for the use-macros too, or in the
per-package dist-git ?

I'd be concerned about the per-package yml file living in "use-macros"
because that would means when package maintainers need to rebase to a
newer release, they potentially have to wait for any "use-macros" update
to be approved & built before they can update their specfile in Fedora
and do a build based on those features. This could also be an impact for
users trying to build new upstream releases in Copr, if the features for
the new upstream release ned to be different from those in the existing
release for that Fedora release stream.

On the point about trying to maintain compat for existing distros which
lack %use macros. I think the example shown is not the route I would take.

Instead I'd just define a %{with_XXX} macro for the feature upfront, based
on the %{use XXX} macro value, and then not use the %use macros at all.
In fact I might be inclined to do this even ignoring the old distro compat
question, because it makes it easy to override the %{use} global defaults
in the package.

%define with_foo %{?use:%{use foo}}%{!?use:1}

%if %{with_foo}
BuildRequires:  foo-devel

% define foo_configure_arg --with-foo
% define foo_test_arg -Dfoo
%endif

%prep
%configure %{?foo_configure_arg}

%check
make test %{?foo_test_arg}


NB The "%{use_enable ...}"  macro is targetting autoconf syntax, but
autotools is not the only build system. Should this aim for a
consistent approach - either provide macros for all build systems,
or for none. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Vít Ondruch]

Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):
>  Based on the recent discussions around %fedora/%rhel macros and ELN,
>  and %bcond generally being confusing to work with, I came up with a
>  distribution-wide feature that defines generic feature keywords and
>  associated helper macros that packages can check in build-time
>  conditionals. 

The most confusing part of the %bcond is the definition itself. The rest
is fine IMO. Therefore, I somehow don't understand why would you like to
replace:


```

%if %{with ssl}
BuildRequires:  openssl-devel
%endif

```


by


```

%if %{use ssl}
BuildRequires:  openssl-devel
%endif

```


Also I don't understand, why there is exposed some underscore macro,
such as `make test %{?_use_ssl:-DSSL}`. Shouldn't the underscore macro
be just implementation detail? For `%bcond`s, the `_with` macros are
discouraged, so why would you encourage them?


Vít


>  The key advantage here is the defaults are defined by the buildroot,
>  not the package. The package is just a building block.
>
>  I'd like some input to improve this and unless this turns out to be a
>  really bad idea, I intend to submit it as a change proposal. Even
>  though the more packages use it the more beneficial it gets, it's, of
>  course, perfectly optional.
>
>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>
>  Cheers,
>  P
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Robin Lee]

On Mon, Apr 27, 2020 at 7:21 PM Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  Based on the recent discussions around %fedora/%rhel macros and ELN,
>  and %bcond generally being confusing to work with, I came up with a
>  distribution-wide feature that defines generic feature keywords and
>  associated helper macros that packages can check in build-time
>  conditionals.
>
>  The key advantage here is the defaults are defined by the buildroot,
>  not the package. The package is just a building block.
>
>  I'd like some input to improve this and unless this turns out to be a
>  really bad idea, I intend to submit it as a change proposal. Even
>  though the more packages use it the more beneficial it gets, it's, of
>  course, perfectly optional.
>
>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408 
Should the use
flags be automatically recorded in Provides of binary packages?
Package A "Provides: A(+feature)"
Then package B may 'Requires: A(+feature)"

That semantics is implemented in Gentoo.
>
> Cheers,
> P
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Petr Pisar]

On Mon, Apr 27, 2020 at 01:19:29PM +0200, Petr Šabata wrote:
>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  
I'm very interested in overriding the global settings:

(1) Is it possible to override them from a modulemd when building a module?

I guess the asnswer is define %_with_ and %_without_ macros in
a buildopts/rpms/macros section of the module. Could elaborate more
the "Compatibility with RPM's --with & --without options" section?

Please note hat the RPM options and the macros have three states (true, false,
undefined) and %_with_foo 0 does not turn %bcond_without foo into false.
I don't ask about preserving a compatibility with this silly semantics, but
some clarification will be needed if this proposal becomes approved.

(2) Is it possible to override them on a per-package basis?

E.g. I have ncurses in global.yaml:

    - name: ncurses
      description: Add support for ncurses.
      enabled: true

and I have plenty of packages that use the ncurses feature in my module. What
should I write to my modulemd so that "ncurses" feature for "pcre"
package is
disabled, but all the other packages have it enabled? Or is it a completelly
illed request to have the same feature enabled at one package and disabled on
another one?

-- Petr

[Petr Šabata]

On Mon, Apr 27, 2020 at 2:15 PM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Mon, Apr 27, 2020 at 01:19:29PM +0200, Petr Šabata wrote:
>  > Based on the recent discussions around %fedora/%rhel macros and ELN,
>  > and %bcond generally being confusing to work with, I came up with a
>  > distribution-wide feature that defines generic feature keywords and
>  > associated helper macros that packages can check in build-time
>  > conditionals.
>  >
>  > The key advantage here is the defaults are defined by the buildroot,
>  > not the package. The package is just a building block.
>
>  IMHO it is valuable having the package self-contained as it is
>  today, as both maintainers & users are able to see and control
>  exactly what features are intended, from a single place.
>
>  With this proposal they'd have to read the global.yml and the
>  local.$PKG.yml and the $PKG.spec file to figure out what is going
>  to be built. Some features will need to vary per-architecture too,
>  so this will need a global-$ARCH.yml  and a local-$ARCH.$PKG.yml file
>  too for each Fedora arch, or the global.yml file will need to ship
>  different content on each arch somehow. So that's potentially 3-5
>  files that need to be consulted to figure out what the enabled
>  features are for a given package build. 
Yes, there is this additional complexity of changing it / looking at
two different places but that's the price paid for having the package
itself independent and the features set by the environment. These
approaches are not exclusive, people can use whichever method,
although I obviously prefer the latter. Hence this proposal.

You're correct about the architecture-specific needs. I wanted to keep
this simple and have people check for that in their packages. However,
that might be contradicting the point above. Architecture-specific
macros would solve this but be more complex to navigate and maintain.

>  > I'd like some input to improve this and unless this turns
> out to be a
>  > really bad idea, I intend to submit it as a change proposal. Even
>  > though the more packages use it the more beneficial it gets, it's, of
>  > course, perfectly optional.
>
>  > Details in the gist:
>  > https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>
>  IIUC, the global.yml file is intended to live in the use-macros
>  package. It wasn't clear though where the local-$PKG.yml file is
>  intended to live ? Is it for the use-macros too, or in the
>  per-package dist-git ? 
Both global and local macros live in the use-macros package. The YAMLs
are sources and generate a single file with all the macros defined in
it. Again, there's nothing binding the distribution local YAMLs to the
packages themselves; they could be built in completely different
buildroots with different macros set from exactly the same commit.

>  I'd be concerned about the per-package yml file living in
> "use-macros"
>  because that would means when package maintainers need to rebase to a
>  newer release, they potentially have to wait for any "use-macros" update
>  to be approved & built before they can update their specfile in Fedora
>  and do a build based on those features. This could also be an impact for
>  users trying to build new upstream releases in Copr, if the features for
>  the new upstream release ned to be different from those in the existing
>  release for that Fedora release stream. 
It's true this requires some coordination or uglier package checks
(assume the originally submitted default value if undefined and
simplify it later).

>  On the point about trying to maintain compat for existing distros
> which
>  lack %use macros. I think the example shown is not the route I would take.
>
>  Instead I'd just define a %{with_XXX} macro for the feature upfront, based
>  on the %{use XXX} macro value, and then not use the %use macros at all.
>  In fact I might be inclined to do this even ignoring the old distro compat
>  question, because it makes it easy to override the %{use} global defaults
>  in the package.
>
>  %define with_foo %{?use:%{use foo}}%{!?use:1}
>
>  %if %{with_foo}
>  BuildRequires:  foo-devel
>
>  % define foo_configure_arg --with-foo
>  % define foo_test_arg -Dfoo
>  %endif
>
>  %prep
>  %configure %{?foo_configure_arg}
>
>  %check
>  make test %{?foo_test_arg} 
You should also explicitly define --without-foo here. Sure, you could
do it this way if you think it's nicer.

>  NB The "%{use_enable ...}"  macro is targetting autoconf
> syntax, but
>  autotools is not the only build system. Should this aim for a
>  consistent approach - either provide macros for all build systems,
>  or for none. 
I started with these two helpers because autotools are extremely
common and because the source of inspiration, Gentoo, also defines
these. I have nothing against defining more for other build systems.

P

[Petr Šabata]

On Mon, Apr 27, 2020 at 3:01 PM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):
>  > Based on the recent discussions around %fedora/%rhel macros and ELN,
>  > and %bcond generally being confusing to work with, I came up with a
>  > distribution-wide feature that defines generic feature keywords and
>  > associated helper macros that packages can check in build-time
>  > conditionals.
>
>
>  The most confusing part of the %bcond is the definition itself. The rest
>  is fine IMO. Therefore, I somehow don't understand why would you like to
>  replace:
>
>
>  ```
>
>  %if %{with ssl}
>  BuildRequires:  openssl-devel
>  %endif
>
>  ```
>
>
>  by
>
>
>  ```
>
>  %if %{use ssl}
>  BuildRequires:  openssl-devel
>  %endif
>
>  ``` 
The difference here is %use defaults are defined by the buildroot
while %with %bconds are defined by the package.

>  Also I don't understand, why there is exposed some underscore
> macro,
>  such as `make test %{?_use_ssl:-DSSL}`. Shouldn't the underscore macro
>  be just implementation detail? For `%bcond`s, the `_with` macros are
>  discouraged, so why would you encourage them? 
Yes, this should be an internal detail. I'd like to replace this with
%use_defined or something similar.

P

[Petr Šabata]

On Mon, Apr 27, 2020 at 3:42 PM Robin Lee <cheeselee(a)fedoraproject.org&gt; wrote:
>  On Mon, Apr 27, 2020 at 7:21 PM Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  >
>  > Based on the recent discussions around %fedora/%rhel macros and ELN,
>  > and %bcond generally being confusing to work with, I came up with a
>  > distribution-wide feature that defines generic feature keywords and
>  > associated helper macros that packages can check in build-time
>  > conditionals.
>  >
>  > The key advantage here is the defaults are defined by the buildroot,
>  > not the package. The package is just a building block.
>  >
>  > I'd like some input to improve this and unless this turns out to be a
>  > really bad idea, I intend to submit it as a change proposal. Even
>  > though the more packages use it the more beneficial it gets, it's, of
>  > course, perfectly optional.
>  >
>  > Details in the gist:
>  > https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  Should the use flags be automatically recorded in Provides of binary packages?
>  Package A "Provides: A(+feature)"
>  Then package B may 'Requires: A(+feature)"
>
>  That semantics is implemented in Gentoo. 
Yes, I think that's a good idea. I already have that in the Ideas
section of the gist, although with a more verbose format.
P

[Petr Šabata]

On Mon, Apr 27, 2020 at 4:04 PM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  On Mon, Apr 27, 2020 at 01:19:29PM +0200, Petr Šabata wrote:
>  > Details in the gist:
>  > https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  >
>  I'm very interested in overriding the global settings:
>
>  (1) Is it possible to override them from a modulemd when building a module?
>
>  I guess the asnswer is define %_with_ and %_without_ macros in
>  a buildopts/rpms/macros section of the module. Could elaborate more
>  the "Compatibility with RPM's --with & --without options" section?
Yes, you can, and that's exactly how you would do that.

Currently the macros as defined with bconds, in the basic form for
enabled by default:

%_with_foo %{bcond_without foo}%{with foo}

>  Please note hat the RPM options and the macros have three states
> (true, false,
>  undefined) and %_with_foo 0 does not turn %bcond_without foo into false.
>  I don't ask about preserving a compatibility with this silly semantics, but
>  some clarification will be needed if this proposal becomes approved. 
It might not work if with/without is set with these macros directly as
it is written right now. I'll have to test that.

>  (2) Is it possible to override them on a per-package basis?
>
>  E.g. I have ncurses in global.yaml:
>
>      - name: ncurses
>        description: Add support for ncurses.
>        enabled: true
>
>  and I have plenty of packages that use the ncurses feature in my module. What
>  should I write to my modulemd so that "ncurses" feature for "pcre"
> package is
>  disabled, but all the other packages have it enabled? Or is it a completelly
>  illed request to have the same feature enabled at one package and disabled on
>  another one? 
It is and that's actually how the local is implemented. It extends the
basic definitions with %{name} checks like this:

%_use_ncurses %{lua:
if rpm.expand("%{name}") == "yourpackage1"
  or rpm.expand("%{name}") == "yourpackage2" then
  print(rpm.expand("%{bcond_with foo}%{with foo}"))
else
  print(rpm.expand("%{bcond_without foo}%{with foo}"))
end
}

I know it's not very user friendly. Maybe there's a better way that
doesn't blow up on recursive macro definition.

P

[Daniel Mach]

I'm wondering if this is your personal initiative or if you're sync with 
ELN people. I emailed them in January about the very same idea (and I 
used the very same name; we both seem to like Gentoo), we exchanged 
couple emails, but never got an answer if this is the way to go. Since I 
have a lot of problems of my own (dnf, rpm, modularity), I did not want 
to start this as my personal initiative.

I'm really glad that someone's looking into this finally.
BTW, the new libdnf spec is using this approach already: 
https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf...

Since I'm part of RPM team too, I hope they won't mind if I'll speak for 
them :) Don't you rather want to work with us on extending the existing 
with/without macros? I'd prefer to improve the existing approach over 
creating something brand new. We could also reuse existing rpmbuild 
--with/--without arguments and ideally remain backwards compatible.

Besides this, +1 from me


Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):
>  Based on the recent discussions around %fedora/%rhel macros and ELN,
>  and %bcond generally being confusing to work with, I came up with a
>  distribution-wide feature that defines generic feature keywords and
>  associated helper macros that packages can check in build-time
>  conditionals.
>  
>  The key advantage here is the defaults are defined by the buildroot,
>  not the package. The package is just a building block.
>  
>  I'd like some input to improve this and unless this turns out to be a
>  really bad idea, I intend to submit it as a change proposal. Even
>  though the more packages use it the more beneficial it gets, it's, of
>  course, perfectly optional.
>  
>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  
>  Cheers,
>  P

[Daniel Mach]

I'm wondering if this is your personal initiative or if you're sync with 
ELN people. I emailed them in January about the very same idea (and I 
used the very same name; we both seem to like Gentoo), we exchanged 
couple emails, but never got an answer if this is the way to go. Since I 
have a lot of problems of my own (dnf, rpm, modularity), I did not want 
to start this as my personal initiative.

I'm really glad that someone's looking into this finally.
BTW, the new libdnf spec is using this approach already: 
https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf...

Since I'm part of RPM team too, I hope they won't mind if I'll speak for 
them :) Don't you rather want to work with us on extending the existing 
with/without macros? I'd prefer to improve the existing approach over 
creating something brand new. We could also reuse existing rpmbuild 
--with/--without arguments and ideally remain backwards compatible.

Besides this, +1 from me


Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):
>  Based on the recent discussions around %fedora/%rhel macros and ELN,
>  and %bcond generally being confusing to work with, I came up with a
>  distribution-wide feature that defines generic feature keywords and
>  associated helper macros that packages can check in build-time
>  conditionals.
>  
>  The key advantage here is the defaults are defined by the buildroot,
>  not the package. The package is just a building block.
>  
>  I'd like some input to improve this and unless this turns out to be a
>  really bad idea, I intend to submit it as a change proposal. Even
>  though the more packages use it the more beneficial it gets, it's, of
>  course, perfectly optional.
>  
>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  
>  Cheers,
>  P

[Vít Ondruch]

Dne 27. 04. 20 v 16:25 Petr Šabata napsal(a):
>  On Mon, Apr 27, 2020 at 3:01 PM Vít Ondruch
> <vondruch(a)redhat.com&gt; wrote:
> >
> > Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):
> >> Based on the recent discussions around %fedora/%rhel macros and ELN,
> >> and %bcond generally being confusing to work with, I came up with a
> >> distribution-wide feature that defines generic feature keywords and
> >> associated helper macros that packages can check in build-time
> >> conditionals.
> >
> > The most confusing part of the %bcond is the definition itself. The rest
> > is fine IMO. Therefore, I somehow don't understand why would you like to
> > replace:
> >
> >
> > ```
> >
> > %if %{with ssl}
> > BuildRequires:  openssl-devel
> > %endif
> >
> > ```
> >
> >
> > by
> >
> >
> > ```
> >
> > %if %{use ssl}
> > BuildRequires:  openssl-devel
> > %endif
> >
> > ```
>  The difference here is %use defaults are defined by the buildroot
>  while %with %bconds are defined by the package. 

Looking at the provided example and the binary RPM and what not, I am
still confused and it is not clear to me what you actually want to achieve.

I think the biggest issue I have is that you seems to propose to use the
`%{use ssl}`, `%{use_enable ssl openssl}` and `%{?_use_ssl:-DSSL}` in
places, where it would make more sense to use the well established
`with/without` rpm(build) macros.

Also, this `%{use_enable ssl openssl}` for example looks like you have
some distribution wide `ssl` feature, but you want to locally implement
it via openssl and moreover it seems you are trying to address the case
where `configure` does not accept with/without but enable/disable
instead (which was probably not an issue so far, otherwise rpmbuild
would probably provide enable/disable options alongside with/without).

IOW it could be probably better if there was something like `%{use ssl
openssl}` call on top of the .spec file, which would enable use of
standard RPM macros instead of introducing set of new macros. This would
also make the `general availability` section much shorter, because there
would be probably need to check the macro existence on a single place.


Vít



> > Also I don't understand, why there is exposed some underscore macro,
> > such as `make test %{?_use_ssl:-DSSL}`. Shouldn't the underscore macro
> > be just implementation detail? For `%bcond`s, the `_with` macros are
> > discouraged, so why would you encourage them?
>  Yes, this should be an internal detail. I'd like to replace this with
>  %use_defined or something similar.
>
>  P
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Stephen Gallagher]

On Mon, Apr 27, 2020 at 10:47 AM Daniel Mach <dmach(a)redhat.com&gt; wrote:
>  I'm wondering if this is your personal initiative or if you're sync with
>  ELN people. I emailed them in January about the very same idea (and I
>  used the very same name; we both seem to like Gentoo), we exchanged
>  couple emails, but never got an answer if this is the way to go. Since I
>  have a lot of problems of my own (dnf, rpm, modularity), I did not want
>  to start this as my personal initiative.
>  
Yes, Petr is in fact ELN people and he's working on this at least
partly within that context (though the benefits are not exclusive to
ELN).

We looked at your approach, but it seemed like you were advocating for
dealing with this somehow at the libdnf layer, which we didn't think
was the right place. If we misunderstood, that's on us.

>  I'm really glad that someone's looking into this finally.
>  BTW, the new libdnf spec is using this approach already:
>  https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf...
>  
Well, not exactly. But it's similar. The main advantage here is that
we can define some things globally for all packages.a

>  Since I'm part of RPM team too, I hope they won't mind if
> I'll speak for
>  them :) Don't you rather want to work with us on extending the existing
>  with/without macros? I'd prefer to improve the existing approach over
>  creating something brand new. We could also reuse existing rpmbuild
>  --with/--without arguments and ideally remain backwards compatible. 
Could you explain a bit more what this means to you? I'm not sure what
you would want to do in RPM itself.

[Petr Šabata]

On Mon, Apr 27, 2020 at 6:38 PM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Dne 27. 04. 20 v 16:25 Petr Šabata napsal(a):
>  > On Mon, Apr 27, 2020 at 3:01 PM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  >>
>  >> Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):
>  >>> Based on the recent discussions around %fedora/%rhel macros and ELN,
>  >>> and %bcond generally being confusing to work with, I came up with a
>  >>> distribution-wide feature that defines generic feature keywords and
>  >>> associated helper macros that packages can check in build-time
>  >>> conditionals.
>  >>
>  >> The most confusing part of the %bcond is the definition itself. The rest
>  >> is fine IMO. Therefore, I somehow don't understand why would you like to
>  >> replace:
>  >>
>  >>
>  >> ```
>  >>
>  >> %if %{with ssl}
>  >> BuildRequires:  openssl-devel
>  >> %endif
>  >>
>  >> ```
>  >>
>  >>
>  >> by
>  >>
>  >>
>  >> ```
>  >>
>  >> %if %{use ssl}
>  >> BuildRequires:  openssl-devel
>  >> %endif
>  >>
>  >> ```
>  > The difference here is %use defaults are defined by the buildroot
>  > while %with %bconds are defined by the package.
>
>
>  Looking at the provided example and the binary RPM and what not, I am
>  still confused and it is not clear to me what you actually want to achieve.
>
>  I think the biggest issue I have is that you seems to propose to use the
>  `%{use ssl}`, `%{use_enable ssl openssl}` and `%{?_use_ssl:-DSSL}` in
>  places, where it would make more sense to use the well established
>  `with/without` rpm(build) macros.
>
>  Also, this `%{use_enable ssl openssl}` for example looks like you have
>  some distribution wide `ssl` feature, but you want to locally implement
>  it via openssl and moreover it seems you are trying to address the case
>  where `configure` does not accept with/without but enable/disable
>  instead (which was probably not an issue so far, otherwise rpmbuild
>  would probably provide enable/disable options alongside with/without). 
Yes, this is exactly what I'm trying to achieve -- to have
distribution-wide generic keywords that control default build options
in packages. In the example SSL support is enabled via the openssl
package.

The well-established with/without options and their defaults are bound
to the package, not the buildroot, which is the key idea behind this,
not the exact helper scripts. Considering it's a repeated point in
this thread, I should have done better job emphasizing it :)

>  IOW it could be probably better if there was something like `%{use
> ssl
>  openssl}` call on top of the .spec file, which would enable use of
>  standard RPM macros instead of introducing set of new macros. This would
>  also make the `general availability` section much shorter, because there
>  would be probably need to check the macro existence on a single place. 
Something like that could be done and would be an option if no one
cared for the helper macros (currently limited to autotools but the
list could grow). We could actually make this nice and SPECs more
readable.

P

[Daniel Mach]

Dne 27. 04. 20 v 19:00 Stephen Gallagher napsal(a):
>  On Mon, Apr 27, 2020 at 10:47 AM Daniel Mach <dmach(a)redhat.com&gt;
> wrote:
> >
> > I'm wondering if this is your personal initiative or if you're sync with
> > ELN people. I emailed them in January about the very same idea (and I
> > used the very same name; we both seem to like Gentoo), we exchanged
> > couple emails, but never got an answer if this is the way to go. Since I
> > have a lot of problems of my own (dnf, rpm, modularity), I did not want
> > to start this as my personal initiative.
> >
>  
>  Yes, Petr is in fact ELN people and he's working on this at least
>  partly within that context (though the benefits are not exclusive to
>  ELN).
>  
>  We looked at your approach, but it seemed like you were advocating for
>  dealing with this somehow at the libdnf layer, which we didn't think
>  was the right place. If we misunderstood, that's on us. 
Most likely. It had
nothing to do with libdnf, it was a generic concept 
based on centralized macros defined in the build roots.

>  
> > I'm really glad that someone's looking into this finally.
> > BTW, the new libdnf spec is using this approach already:
> > https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf...
> >
>  
>  Well, not exactly. But it's similar. The main advantage here is that
>  we can define some things globally for all packages.a 
Yes, that was exactly the
point in my proposal.
Building for another use case would mean just setting build root macros 
and building the package differently, no spec change needed.

>  
> > Since I'm part of RPM team too, I hope they won't mind if I'll speak for
> > them :) Don't you rather want to work with us on extending the existing
> > with/without macros? I'd prefer to improve the existing approach over
> > creating something brand new. We could also reuse existing rpmbuild
> > --with/--without arguments and ideally remain backwards compatible.
>  
>  Could you explain a bit more what this means to you? I'm not sure what
>  you would want to do in RPM itself. 
rpmbuild supports --with and --without options
for setting the %with 
macros already. That allows you to make a local build with custom %with 
values. I suppose you'll need something similar for the new feature. 
 From my perspective it would be better to re-use the existing options 
rather than introducing --use/--dont-use or something similar.

>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Nicolas Mailhot]

Le lundi 27 avril 2020 à 19:12 +0200, Petr Šabata a écrit :
>  
>  Yes, this is exactly what I'm trying to achieve -- to have
>  distribution-wide generic keywords that control default build options
>  in packages 
It can be done at the rpm layer and in fact there are already quite a
lot of such tunables in the distribution that packagers rely on every
day without realising it.

The basic integration pattern is:
1. you define a set of nice useful macros (in redhat-rpm-config or in
your own foo-rpm-macros project)
2. you make their behaviour dependant on the value of nicely named
variables
3. you ship default distro-wide values for those variables in a
separate macro file
4. you document the whole thing, get it reviewed, added to Fedora
guidelines by FPC, and added to the default buidroot
5. you spend ~1 year helping people use it (after that there are enough
examples in the distro to cut & paste, that packagers feel autonomous)

From this point on you can control the default behaviour of all the
spec files that used you macros distro-wide by changing the default
value of your tunables. Downstreams can change the behaviour by
changing the default tunable values in their version of the central
macro package. Packagers can override the default in individual spec
files just by setting a variable in their spec file if the default does
not work for their project, etc.

It’s a huge amount of design and review work however. Once you play
with the default behaviour of just a few % of distribution packages,
you need to be extra careful. 

-- 
Nicolas Mailhot

[clime]

On Mon, 27 Apr 2020 at 13:21, Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  Based on the recent discussions around %fedora/%rhel macros and ELN,
>  and %bcond generally being confusing to work with, I came up with a
>  distribution-wide feature that defines generic feature keywords and
>  associated helper macros that packages can check in build-time
>  conditionals.
>
>  The key advantage here is the defaults are defined by the buildroot,
>  not the package. The package is just a building block.
>
>  I'd like some input to improve this and unless this turns out to be a
>  really bad idea, I intend to submit it as a change proposal. Even
>  though the more packages use it the more beneficial it gets, it's, of
>  course, perfectly optional.
>
>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408 
Ad. Dan Mach's presented spec file:

> https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf... 
I really like how all the switches are defined almost on top of the
spec file so I think lots of inspiration can be drawn from that.

Now I also think that introduction of the new %use scheme for build
options instead of the current with/without scheme might be an
overkill.

I think lines like these:

%bcond_with html
%bcond_without man

could be replaced by:

%with_option_env html 0
%with_option_env man 1

i.e. %with_option_env <option_name> <default_value>

%with_option_env could be looking at an environmental buildroot
setting with the possibility to be overridden on command-line by
--with and --without as usual.

There could be also:
%with_option <option_name> <default_value>

which wouldn't look at environment settings so it would be the same
thing as %bcond_with/%bcond_without but less confusing.

So with this, you wouldn't need to change conditions like:

%if %{with foo}
    ...
%endif

and

%if %{without foo}
    ...
%endif

...so fewer changes and less work. I also think these conditions are
quite fine (except relying on with_foo being either defined or
undefined, which is quite funky but I don't think it is a reason to
replace them).

---

The yml files and translation from them into the actual macro files
are nice but I would consider if the hw dependent default values can
be added in future as a feature.

The local_<pkg>.yml file seems somewhat out of place to me. I think it
could be rather kept as an idea for future and for now we could just
start with only buildroot configs?

Best regards!
clime

>
> Cheers,
> P
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[clime]

Few more notes:

I think an idea that build system could be simply passing
--with/--without options on mock's command-line should be
considered...then basically no change in spec files is needed (i.e. no
syntax change).

It would be great if rpm remembered either the rpmbuild's command-line
in case the solution above is chosen or the set of with options (and
the final values for them) that were used during build if the
buildroot config approach is chosen. I think that would be very useful
for debugging and inspection of rpms.

...that's probably it from me.

clime

On Tue, 28 Apr 2020 at 03:06, clime <clime(a)fedoraproject.org&gt; wrote:
>
> On Mon, 27 Apr 2020 at 13:21, Petr Šabata <contyk(a)redhat.com&gt; wrote:
> >
> > Based on the recent discussions around %fedora/%rhel macros and ELN,
> > and %bcond generally being confusing to work with, I came up with a
> > distribution-wide feature that defines generic feature keywords and
> > associated helper macros that packages can check in build-time
> > conditionals.
> >
> > The key advantage here is the defaults are defined by the buildroot,
> > not the package. The package is just a building block.
> >
> > I'd like some input to improve this and unless this turns out to be a
> > really bad idea, I intend to submit it as a change proposal. Even
> > though the more packages use it the more beneficial it gets, it's, of
> > course, perfectly optional.
> >
> > Details in the gist:
> > https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>
> Ad. Dan Mach's presented spec file:
>
> > https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf...
>
> I really like how all the switches are defined almost on top of the
> spec file so I think lots of inspiration can be drawn from that.
>
> Now I also think that introduction of the new %use scheme for build
> options instead of the current with/without scheme might be an
> overkill.
>
> I think lines like these:
>
> %bcond_with html
> %bcond_without man
>
> could be replaced by:
>
> %with_option_env html 0
> %with_option_env man 1
>
> i.e. %with_option_env <option_name> <default_value>
>
> %with_option_env could be looking at an environmental buildroot
> setting with the possibility to be overridden on command-line by
> --with and --without as usual.
>
> There could be also:
> %with_option <option_name> <default_value>
>
> which wouldn't look at environment settings so it would be the same
> thing as %bcond_with/%bcond_without but less confusing.
>
> So with this, you wouldn't need to change conditions like:
>
> %if %{with foo}
>     ...
> %endif
>
> and
>
> %if %{without foo}
>     ...
> %endif
>
> ...so fewer changes and less work. I also think these conditions are
> quite fine (except relying on with_foo being either defined or
> undefined, which is quite funky but I don't think it is a reason to
> replace them).
>
> ---
>
> The yml files and translation from them into the actual macro files
> are nice but I would consider if the hw dependent default values can
> be added in future as a feature.
>
> The local_<pkg>.yml file seems somewhat out of place to me. I think it
> could be rather kept as an idea for future and for now we could just
> start with only buildroot configs?
>
> Best regards!
> clime
>
> >
> > Cheers,
> > P
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Petr Pisar]

On Mon, Apr 27, 2020 at 04:33:52PM +0200, Petr Šabata wrote:
>  On Mon, Apr 27, 2020 at 4:04 PM Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  > (2) Is it possible to override them on a per-package basis?
>  >
>  > E.g. I have ncurses in global.yaml:
>  >
>  >     - name: ncurses
>  >       description: Add support for ncurses.
>  >       enabled: true
>  >
>  > and I have plenty of packages that use the ncurses feature in my module. What
>  > should I write to my modulemd so that "ncurses" feature for
> "pcre" package is
>  > disabled, but all the other packages have it enabled? Or is it a completelly
>  > illed request to have the same feature enabled at one package and disabled on
>  > another one?
>  
>  It is and that's actually how the local is implemented. It extends the
>  basic definitions with %{name} checks like this:
>  
>  %_use_ncurses %{lua:
>  if rpm.expand("%{name}") == "yourpackage1"
>    or rpm.expand("%{name}") == "yourpackage2" then
>    print(rpm.expand("%{bcond_with foo}%{with foo}"))
>  else
>    print(rpm.expand("%{bcond_without foo}%{with foo}"))
>  end
>  }
>  
>  I know it's not very user friendly. Maybe there's a better way that
>  doesn't blow up on recursive macro definition.
>   
Do I understand it correctly that modules should reimplement the %_use_ncurses
macro? That's really clumsy and I'd like to avoid it. Not speaking about the
issues with recursion you are aware and I was hoping you found a solution.
Modules would have to simply redefine the macro covering all packages built in
the module.

Since you use Lua (because RPM conditions are not allowed there), wouldn't
be better to export the mapping from a package name to a feature as a Lua
associative array? Modules would then just added/rewrote a tuple there and
than call a Lua function to regenerate the %_use macros?

Actually if the generation of the macros was postponed to a spec file, there
would not have to exist any local.yaml file. That way the spec file would be
be self-contained. I agree with others that separating the local overrides
into local.yaml maintained in a different package is not handy and slows the
packager's work flow.

-- Petr

[Vít Ondruch]

Dne 27. 04. 20 v 19:40 Daniel Mach napsal(a):
>  Dne 27. 04. 20 v 19:00 Stephen Gallagher napsal(a):
> > On Mon, Apr 27, 2020 at 10:47 AM Daniel Mach <dmach(a)redhat.com&gt; wrote:
>
> >
> >> Since I'm part of RPM team too, I hope they won't mind if I'll speak
> >> for
> >> them :) Don't you rather want to work with us on extending the existing
> >> with/without macros? I'd prefer to improve the existing approach over
> >> creating something brand new. We could also reuse existing rpmbuild
> >> --with/--without arguments and ideally remain backwards compatible.
> >
> > Could you explain a bit more what this means to you? I'm not sure what
> > you would want to do in RPM itself.
>  rpmbuild supports --with and --without options for setting the %with
>  macros already. That allows you to make a local build with custom
>  %with values. I suppose you'll need something similar for the new
>  feature. From my perspective it would be better to re-use the existing
>  options rather than introducing --use/--dont-use or something similar. 

+1

This is the same concern I shared on different place of this thread.


Vít

[Nicolas Mailhot]

Le mardi 28 avril 2020 à 08:43 +0200, Petr Pisar a écrit :
>  On Mon, Apr 27, 2020 at 04:33:52PM +0200, Petr Šabata wrote:
>  > 
>  > %_use_ncurses %{lua:
>  > if rpm.expand("%{name}") == "yourpackage1"
>  >   or rpm.expand("%{name}") == "yourpackage2" then
>  >   print(rpm.expand("%{bcond_with foo}%{with foo}"))
>  > else
>  >   print(rpm.expand("%{bcond_without foo}%{with foo}"))
>  > end
>  > } 
%{name} use in macro logic is unsafe because %{name} does not exist in
the preamble before the Name: line, and the Name: line does not exist
before you start declaring package headers.

Once you’re in package
header declaration mode rpm parser behaviour will prevent you from
doing logic between headers blocks, so it all needs interleaving, which
ends un not possible sanely in any semi-complex spec file.

The rpm parser will now insult you with messages like
warning: undefined macro(s) in %{_sourcedir}: /var/lib/builder/rpmbuild/SOURCES/%{name}

if you try to use %{name} at the srpm level.

This is an rpm 4.15 change
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/83

Regards,

-- 
Nicolas Mailhot

[Panu Matilainen]

On 4/27/20 5:33 PM, Petr Šabata wrote:
>  On Mon, Apr 27, 2020 at 4:04 PM Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
> >
> > On Mon, Apr 27, 2020 at 01:19:29PM +0200, Petr Šabata wrote:
> >> Details in the gist:
> >> https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
> >>
> > I'm very interested in overriding the global settings:
> >
> > (1) Is it possible to override them from a modulemd when building a module?
> >
> > I guess the asnswer is define %_with_ and %_without_ macros in
> > a buildopts/rpms/macros section of the module. Could elaborate more
> > the "Compatibility with RPM's --with & --without options" section?
>  
>  Yes, you can, and that's exactly how you would do that.
>  
>  Currently the macros as defined with bconds, in the basic form for
>  enabled by default:
>  
>  %_with_foo %{bcond_without foo}%{with foo}
>  
> > Please note hat the RPM options and the macros have three states (true, false,
> > undefined) and %_with_foo 0 does not turn %bcond_without foo into false.
> > I don't ask about preserving a compatibility with this silly semantics, but
> > some clarification will be needed if this proposal becomes approved.
>  
>  It might not work if with/without is set with these macros directly as
>  it is written right now. I'll have to test that.
>  
> > (2) Is it possible to override them on a per-package basis?
> >
> > E.g. I have ncurses in global.yaml:
> >
> >      - name: ncurses
> >        description: Add support for ncurses.
> >        enabled: true
> >
> > and I have plenty of packages that use the ncurses feature in my module. What
> > should I write to my modulemd so that "ncurses" feature for
> "pcre" package is
> > disabled, but all the other packages have it enabled? Or is it a completelly
> > illed request to have the same feature enabled at one package and disabled on
> > another one?
>  
>  It is and that's actually how the local is implemented.  
Not entirely sure if we're talking about the same "local" here but I 
spotted this in the use-macros.spec:

# %%global local

for source in string.gmatch(rpm.expand("%{?local}"), "[%w_-]+")

Please avoid taking over such broadly generic names as "local" in the 
global macro namespace, rpm might well want to use %local as a macro 
primitive (opposite of %global) one day.

>  It extends the
>  basic definitions with %{name} checks like this:
>  
>  %_use_ncurses %{lua:
>  if rpm.expand("%{name}") == "yourpackage1"
>     or rpm.expand("%{name}") == "yourpackage2" then
>     print(rpm.expand("%{bcond_with foo}%{with foo}"))
>  else
>     print(rpm.expand("%{bcond_without foo}%{with foo}"))
>  end
>  } 
Danger Will Robinson.

Such checks will on %{name} will only work after the Name: tag in the 
spec, which is not obvious from the outset, and there's a large set of 
packages in Fedora that only declare the name towards the end of the 
spec preamble.

	- Panu -

>  I know it's not very user friendly. Maybe there's a better
> way that
>  doesn't blow up on recursive macro definition.
>  
>  P
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Panu Matilainen]

On 4/28/20 10:53 AM, Nicolas Mailhot via devel wrote:
>  Le mardi 28 avril 2020 à 08:43 +0200, Petr Pisar a écrit :
> > On Mon, Apr 27, 2020 at 04:33:52PM +0200, Petr Šabata wrote:
> >>
> >> %_use_ncurses %{lua:
> >> if rpm.expand("%{name}") == "yourpackage1"
> >>    or rpm.expand("%{name}") == "yourpackage2" then
> >>    print(rpm.expand("%{bcond_with foo}%{with foo}"))
> >> else
> >>    print(rpm.expand("%{bcond_without foo}%{with foo}"))
> >> end
> >> }
>  
>  %{name} use in macro logic is unsafe because %{name} does not exist in
>  the preamble before the Name: line, and the Name: line does not exist
>  before you start declaring package headers. 
Just to clarify, this is true of all versions of rpm.

>  Once you’re in package
>  header declaration mode rpm parser behaviour will prevent you from
>  doing logic between headers blocks, so it all needs interleaving, which
>  ends un not possible sanely in any semi-complex spec file.
>  
>  The rpm parser will now insult you with messages like
>  warning: undefined macro(s) in %{_sourcedir}: /var/lib/builder/rpmbuild/SOURCES/%{name}
>  
>  if you try to use %{name} at the srpm level.
>  
>  This is an rpm 4.15 change
>  https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/83
>   
And that's a redhat-rpm-config pull-request, which doesn't really 
explain a thing.

The actual background is in this bug: 
https://bugzilla.redhat.com/show_bug.cgi?id=1820349

	- Panu -

[Miroslav Suchý]

Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):
>  Based on the recent discussions around %fedora/%rhel macros and ELN,
>  and %bcond generally being confusing to work with, I came up with a
>  distribution-wide feature that defines generic feature keywords and
>  associated helper macros that packages can check in build-time
>  conditionals. 
Can you provide the Suse guidelines please? The words "use", "macros"
are terrible for searching.
I would like to see, how and where your proposal differ. Or if there are any differences
at all.


-- 
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

[Miroslav Suchý]

Dne 29. 04. 20 v 18:07 Miroslav Suchý napsal(a):
>  Dne 27. 04. 20 v 13:19 Petr Šabata napsal(a):
> > Based on the recent discussions around %fedora/%rhel macros and ELN,
> > and %bcond generally being confusing to work with, I came up with a
> > distribution-wide feature that defines generic feature keywords and
> > associated helper macros that packages can check in build-time
> > conditionals.
>  
>  Can you provide the Suse guidelines please? The words "use", "macros"
> are terrible for searching.
>  I would like to see, how and where your proposal differ. Or if there are any differences
> at all. 

Sorry, I am blind. It is there
  https://devmanual.gentoo.org/general-concepts/use-flags/index.html
and it is inspired by Gentoo and not Suse.
-- 
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

[Colin Walters]

On Mon, Apr 27, 2020, at 7:19 AM, Petr Šabata wrote:

>  Details in the gist:
>  https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408 
How about s/use/globalbuildopt/ ?

[Petr Pisar]

On Wed, Apr 29, 2020 at 03:39:20PM -0400, Colin Walters wrote:
>  On Mon, Apr 27, 2020, at 7:19 AM, Petr Šabata wrote:
>  
>  > Details in the gist:
>  > https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  
>  How about s/use/globalbuildopt/ ?
>   
It's awfully long concatenation of two words and an abbreviation. No.

-- Petr

[Petr Šabata]

On Tue, Apr 28, 2020 at 3:40 AM clime <clime(a)fedoraproject.org&gt; wrote:
>  Few more notes:
>
>  I think an idea that build system could be simply passing
>  --with/--without options on mock's command-line should be
>  considered...then basically no change in spec files is needed (i.e. no
>  syntax change). 
Yes but I'd say such a change would be more complex and invasive. You
would still need a place to store these values and make sure the
systems (koji, mocks, rpmbuilds on  the system, many more) would read
the correct configs and pass the values.

>  It would be great if rpm remembered either the rpmbuild's
> command-line
>  in case the solution above is chosen or the set of with options (and
>  the final values for them) that were used during build if the
>  buildroot config approach is chosen. I think that would be very useful
>  for debugging and inspection of rpms.
>
>  ...that's probably it from me.
>
>  clime
>
>  On Tue, 28 Apr 2020 at 03:06, clime <clime(a)fedoraproject.org&gt; wrote:
>  >
>  > On Mon, 27 Apr 2020 at 13:21, Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  > >
>  > > Based on the recent discussions around %fedora/%rhel macros and ELN,
>  > > and %bcond generally being confusing to work with, I came up with a
>  > > distribution-wide feature that defines generic feature keywords and
>  > > associated helper macros that packages can check in build-time
>  > > conditionals.
>  > >
>  > > The key advantage here is the defaults are defined by the buildroot,
>  > > not the package. The package is just a building block.
>  > >
>  > > I'd like some input to improve this and unless this turns out to be a
>  > > really bad idea, I intend to submit it as a change proposal. Even
>  > > though the more packages use it the more beneficial it gets, it's, of
>  > > course, perfectly optional.
>  > >
>  > > Details in the gist:
>  > > https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>  >
>  > Ad. Dan Mach's presented spec file:
>  >
>  > > https://github.com/rpm-software-management/libdnf/blob/dnf-5-devel/libdnf...
>  >
>  > I really like how all the switches are defined almost on top of the
>  > spec file so I think lots of inspiration can be drawn from that.
>  >
>  > Now I also think that introduction of the new %use scheme for build
>  > options instead of the current with/without scheme might be an
>  > overkill.
>  >
>  > I think lines like these:
>  >
>  > %bcond_with html
>  > %bcond_without man
>  >
>  > could be replaced by:
>  >
>  > %with_option_env html 0
>  > %with_option_env man 1
>  >
>  > i.e. %with_option_env <option_name> <default_value>
>  >
>  > %with_option_env could be looking at an environmental buildroot
>  > setting with the possibility to be overridden on command-line by
>  > --with and --without as usual.
>  >
>  > There could be also:
>  > %with_option <option_name> <default_value>
>  >
>  > which wouldn't look at environment settings so it would be the same
>  > thing as %bcond_with/%bcond_without but less confusing.
>  >
>  > So with this, you wouldn't need to change conditions like:
>  >
>  > %if %{with foo}
>  >     ...
>  > %endif
>  >
>  > and
>  >
>  > %if %{without foo}
>  >     ...
>  > %endif
>  >
>  > ...so fewer changes and less work. I also think these conditions are
>  > quite fine (except relying on with_foo being either defined or
>  > undefined, which is quite funky but I don't think it is a reason to
>  > replace them). 
I was trying to keep SPEC files as brief as possible. The first time
you call any of these macros, the feature is initialized even without
any preamble definitions. These are already compatible with
--with/--without so you can override the defaults with them.
Technically you could do just #%{use foo} in your preamble and then
work with %with and %without, which is pretty much what you're
suggesting. However, combining the two approaches feels more ugly and
confusing to me than sticking to one.


>  > The yml files and translation from them into the actual macro
> files
>  > are nice but I would consider if the hw dependent default values can
>  > be added in future as a feature.
>  >
>  > The local_<pkg>.yml file seems somewhat out of place to me. I think it
>  > could be rather kept as an idea for future and for now we could just
>  > start with only buildroot configs? 
It's a possibility for packages that want or need it. I wouldn't
expect many if any local files in the beginning.

P

[Petr Šabata]

On Tue, Apr 28, 2020 at 8:55 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  On Mon, Apr 27, 2020 at 04:33:52PM +0200, Petr Šabata wrote:
>  > On Mon, Apr 27, 2020 at 4:04 PM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  > > (2) Is it possible to override them on a per-package basis?
>  > >
>  > > E.g. I have ncurses in global.yaml:
>  > >
>  > >     - name: ncurses
>  > >       description: Add support for ncurses.
>  > >       enabled: true
>  > >
>  > > and I have plenty of packages that use the ncurses feature in my module. What
>  > > should I write to my modulemd so that "ncurses" feature for
> "pcre" package is
>  > > disabled, but all the other packages have it enabled? Or is it a completelly
>  > > illed request to have the same feature enabled at one package and disabled on
>  > > another one?
>  >
>  > It is and that's actually how the local is implemented. It extends the
>  > basic definitions with %{name} checks like this:
>  >
>  > %_use_ncurses %{lua:
>  > if rpm.expand("%{name}") == "yourpackage1"
>  >   or rpm.expand("%{name}") == "yourpackage2" then
>  >   print(rpm.expand("%{bcond_with foo}%{with foo}"))
>  > else
>  >   print(rpm.expand("%{bcond_without foo}%{with foo}"))
>  > end
>  > }
>  >
>  > I know it's not very user friendly. Maybe there's a better way that
>  > doesn't blow up on recursive macro definition.
>  >
>  Do I understand it correctly that modules should reimplement the %_use_ncurses
>  macro? That's really clumsy and I'd like to avoid it. Not speaking about the
>  issues with recursion you are aware and I was hoping you found a solution.
>  Modules would have to simply redefine the macro covering all packages built in
>  the module. 
Yes, it is clumsy and you're correct here.

MBS also just extends the macro definitions so I'm not sure how to
work around the recursion problem even if we introduced a new modulemd
section just for this (which wouldn't feel right either).  Would you
have any suggestions?

>  Since you use Lua (because RPM conditions are not allowed there),
> wouldn't
>  be better to export the mapping from a package name to a feature as a Lua
>  associative array? Modules would then just added/rewrote a tuple there and
>  than call a Lua function to regenerate the %_use macros? 
Yes, that would be nicer. I'll look into that.

>  Actually if the generation of the macros was postponed to a spec
> file, there
>  would not have to exist any local.yaml file. That way the spec file would be
>  be self-contained. I agree with others that separating the local overrides
>  into local.yaml maintained in a different package is not handy and slows the
>  packager's work flow. 
But it also defeats the idea of these being set by the system, leaving
package repos untouched.

P

[Petr Šabata]

On Tue, Apr 28, 2020 at 9:55 AM Nicolas Mailhot via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Le mardi 28 avril 2020 à 08:43 +0200, Petr Pisar a écrit :
>  > On Mon, Apr 27, 2020 at 04:33:52PM +0200, Petr Šabata wrote:
>  > >
>  > > %_use_ncurses %{lua:
>  > > if rpm.expand("%{name}") == "yourpackage1"
>  > >   or rpm.expand("%{name}") == "yourpackage2" then
>  > >   print(rpm.expand("%{bcond_with foo}%{with foo}"))
>  > > else
>  > >   print(rpm.expand("%{bcond_without foo}%{with foo}"))
>  > > end
>  > > }
>
>  %{name} use in macro logic is unsafe because %{name} does not exist in
>  the preamble before the Name: line, and the Name: line does not exist
>  before you start declaring package headers.
>
>  Once you’re in package
>  header declaration mode rpm parser behaviour will prevent you from
>  doing logic between headers blocks, so it all needs interleaving, which
>  ends un not possible sanely in any semi-complex spec file. 
You're correct but I can't think of any solution besides not using the
macros before you define Name, just like in your example below. It's
certainly a drawback.

P

>  The rpm parser will now insult you with messages like
>  warning: undefined macro(s) in %{_sourcedir}: /var/lib/builder/rpmbuild/SOURCES/%{name}
>
>  if you try to use %{name} at the srpm level.
>
>  This is an rpm 4.15 change
>  https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/83
>
>  Regards,
>
>  --
>  Nicolas Mailhot

[Petr Šabata]

On Wed, Apr 29, 2020 at 9:43 PM Colin Walters <walters(a)verbum.org&gt; wrote:
>  On Mon, Apr 27, 2020, at 7:19 AM, Petr Šabata wrote:
>
>  > Details in the gist:
>  > https://gist.github.com/contyk/0aaaaf0585c57976ca18a293b3566408
>
>  How about s/use/globalbuildopt/ ? 
Please, no :)

The reason I went with "use" is because this is reimplementing a
Gentoo concept and is virtually identical. I wanted to keep the
terminology as many people are already familiar with it.

P

[Petr Pisar]

On Thu, Apr 30, 2020 at 10:43:51AM +0200, Petr Šabata wrote:
>  On Tue, Apr 28, 2020 at 8:55 AM Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  >
>  > On Mon, Apr 27, 2020 at 04:33:52PM +0200, Petr Šabata wrote:
>  > > On Mon, Apr 27, 2020 at 4:04 PM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  > > > (2) Is it possible to override them on a per-package basis?
>  > > >
>  > > > E.g. I have ncurses in global.yaml:
>  > > >
>  > > >     - name: ncurses
>  > > >       description: Add support for ncurses.
>  > > >       enabled: true
>  > > >
>  > > > and I have plenty of packages that use the ncurses feature in my module.
> What
>  > > > should I write to my modulemd so that "ncurses" feature for
> "pcre" package is
>  > > > disabled, but all the other packages have it enabled? Or is it a
> completelly
>  > > > illed request to have the same feature enabled at one package and disabled
> on
>  > > > another one?
>  > >
>  > > It is and that's actually how the local is implemented. It extends the
>  > > basic definitions with %{name} checks like this:
>  > >
>  > > %_use_ncurses %{lua:
>  > > if rpm.expand("%{name}") == "yourpackage1"
>  > >   or rpm.expand("%{name}") == "yourpackage2" then
>  > >   print(rpm.expand("%{bcond_with foo}%{with foo}"))
>  > > else
>  > >   print(rpm.expand("%{bcond_without foo}%{with foo}"))
>  > > end
>  > > }
>  > >
>  > > I know it's not very user friendly. Maybe there's a better way that
>  > > doesn't blow up on recursive macro definition.
>  > >
>  > Do I understand it correctly that modules should reimplement the %_use_ncurses
>  > macro? That's really clumsy and I'd like to avoid it. Not speaking about
> the
>  > issues with recursion you are aware and I was hoping you found a solution.
>  > Modules would have to simply redefine the macro covering all packages built in
>  > the module.
>  
>  Yes, it is clumsy and you're correct here.
>  
>  MBS also just extends the macro definitions so I'm not sure how to
>  work around the recursion problem even if we introduced a new modulemd
>  section just for this (which wouldn't feel right either).  Would you
>  have any suggestions?
>   
Unfortunately I have no suggestion. I agree that the new modulemd section
would be an overkill.

>  > Actually if the generation of the macros was postponed to a spec
> file, there
>  > would not have to exist any local.yaml file. That way the spec file would be
>  > be self-contained. I agree with others that separating the local overrides
>  > into local.yaml maintained in a different package is not handy and slows the
>  > packager's work flow.
>  
>  But it also defeats the idea of these being set by the system, leaving
>  package repos untouched.
>   
You are right. I forgot this use case.

Probably because local.yaml is a misnomer. local.yaml is a distribution-wide
configation like global.yaml. It's only per-package configuration. In Gentoo,
"local" use flags are indeed defined next to the ebuild. But in Fedora their
definition would be separated from the packages. local.yaml is more alike to
a Gentoo profile.

Technically local.yaml (it's actually a set of locale.%{name}.yaml files)
could be merged into the global.yaml file. But the syntax would be horrible
because there would be a tension whether to collate them by a feature and then
by a package name, or vice versa. Not speaking about an inflation by
the descriptions. I miss the Gentoo brevity:

* ncurses
pcre -ncurses

-- Petr

[Nicolas Mailhot]

Le jeudi 30 avril 2020 à 10:49 +0200, Petr Šabata a écrit :
>  On Tue, Apr 28, 2020 at 9:55 AM Nicolas Mailhot via devel
>  <devel(a)lists.fedoraproject.org&gt; wrote:
>  > Le mardi 28 avril 2020 à 08:43 +0200, Petr Pisar a écrit :
>  > > On Mon, Apr 27, 2020 at 04:33:52PM +0200, Petr Šabata wrote:
>  > > > %_use_ncurses %{lua:
>  > > > if rpm.expand("%{name}") == "yourpackage1"
>  > > >   or rpm.expand("%{name}") == "yourpackage2" then
>  > > >   print(rpm.expand("%{bcond_with foo}%{with foo}"))
>  > > > else
>  > > >   print(rpm.expand("%{bcond_without foo}%{with foo}"))
>  > > > end
>  > > > }
>  > 
>  > %{name} use in macro logic is unsafe because %{name} does not exist
>  > in
>  > the preamble before the Name: line, and the Name: line does not
>  > exist
>  > before you start declaring package headers.
>  > 
>  > Once you’re in package
>  > header declaration mode rpm parser behaviour will prevent you from
>  > doing logic between headers blocks, so it all needs interleaving,
>  > which
>  > ends un not possible sanely in any semi-complex spec file.
>  
>  You're correct but I can't think of any solution besides not using
>  the macros before you define Name, just like in your example below.  
One positive result of all the ugliness in
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/83

(which is what happens, when people like you and me try to do
processing on %{name}, and rpm maintainers feel otherwise)

is a generic %new_package wrapper in redhat-rpm-config.

The PR, should it be merged, will get you several things:

1. a real %{source_name} variable in spec files (not just
%{source_name} emulation in dnf) that you can set and do things with
before entering package header land

2. a single entry point within each header, *after* %{source_name} and
%{name} are set, where you can do generic pre-header processing such as
the one you propose. And where you can choose between processing the
subpackage name or the SRPM name, because they are exposed in different
variables.

And it only requires, besides merging the PR, to replace Name: and
%package lines in a spec file with the corresponding %new_package line.

If the packager did not set %{source_name}, the first %new_package will
set it and own the SRPM (usual case).

rpm will sadly probably insist that the srpm header comes first, so you
can not use %{source_name} to shuffle package headers in interesting
ways. %new_package solves the following ordering problems only:
1. need to do things with the srpm name before entering dangerous
header land (your case)
2. need to know, for macros that create package headers, if they are
supposed to takeover the SRPM or not. The PR removes the need for
complex inter-macro syncronization, a macro can just use %new_package,
and that will check if source_name if free to use, matches the package
header name, or matches something else

I coded it for my own automation macro needs (in fonts and go packages)
but it also marches your use case nicely

Regards,
-- 
Nicolas Mailhot

[clime]

On Thu, 30 Apr 2020 at 10:40, Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  On Tue, Apr 28, 2020 at 3:40 AM clime <clime(a)fedoraproject.org&gt; wrote:
>  >
>  > Few more notes:
>  >
>  > I think an idea that build system could be simply passing
>  > --with/--without options on mock's command-line should be
>  > considered...then basically no change in spec files is needed (i.e. no
>  > syntax change).
>
>  Yes but I'd say such a change would be more complex and invasive. You
>  would still need a place to store these values and make sure the
>  systems (koji, mocks, rpmbuilds on  the system, many more) would read
>  the correct configs and pass the values. 
Well, in the end, it would probably mean rpmbuild would have a new
option to consume build-config.yml with all the with/withouts, which
would be cool because the config could be serialized in its headers -
at least for those options that were actually used from the spec file.
A disadvantage is that bare/default rpmbuild would be using spec
defined defaults for the with options so you would need to add the new
--build-config option explicitly on command-line to actually build the
same rpms that are being built in mock or a build system (both could
do that by default). This seems quite alright to me, nevertheless. I
like that the method to influence a build is explicit and the config
can be imprinted into the built rpms. I think this will be otherwise a
problem if the .yml configs are translated into system-wide rpm
macros.

>  I was trying to keep SPEC files as brief as possible. The first time
>  you call any of these macros, the feature is initialized even without
>  any preamble definitions. These are already compatible with
>  --with/--without so you can override the defaults with them.
>  Technically you could do just #%{use foo} in your preamble and then
>  work with %with and %without, which is pretty much what you're
>  suggesting. However, combining the two approaches feels more ugly and
>  confusing to me than sticking to one. 
I was rather suggesting using just a dedicated %with related preamble like:
%wIth_option_env to explicitly state: "hey, I am taking something from
the environment here."

It would give you a quick list in spec file what you can configure,
which is nice.

Anyway, it's quite hard to image %use and %with to co-exist. Ability
to affect %use by --with/--without options seems confusing so while I
like this initiative, it probably tries to change more than is really
required to change. Idk, I don't really have a strong opinion here. I
am glad you have built a cool prototype and showed it to the community
for review. I think that's an important thing here.

clime

>
>
> > > The yml files and translation from them into the actual macro files
> > > are nice but I would consider if the hw dependent default values can
> > > be added in future as a feature.
> > >
> > > The local_<pkg>.yml file seems somewhat out of place to me. I think
it
> > > could be rather kept as an idea for future and for now we could just
> > > start with only buildroot configs?
>
> It's a possibility for packages that want or need it. I wouldn't
> expect many if any local files in the beginning.
>
> P
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]