Proposal: Remove BuildRequires: exceptions from the guidelines

[Jason L Tibbitts III]

TL;DR: There's a proposal for nuking the list of BuildRequires
exceptions from the packaging guidelines.

Important note: this is just an idea.  We're not currently changing the
guidelines based on this.  Nothing is set in stone.  It may end up that
nothing happens.  Don't panic.

A couple of times recently, people have floated proposals to the FPC
about removing certain things from the buildroot.  (Specifically, perl
and gcc*.)  And the bottom line for these is that while FPC tries to
maintain a list of BuildRequires: exceptions, this is really up to
releng and whatever rpm happens to pull in at any particular time.

This means that whatever list FPC maintains, it's going to get outdated
eventually, and giving packagers the idea that they don't have to
actually specify all of their dependencies restricts what releng can do
and, perhaps, what dependencies the RPM package can drop without
breaking builds.

So, the generally stated proposal:

I would like to get FPC out of the business of specifying this list of
exceptions, and instead just indicate that packagers should completely
specify their dependencies.

My specific proposal can be seen in
https://fedorahosted.org/fpc/ticket/497; the draft is at
https://fedoraproject.org/wiki/User:Tibbs/BuildReqDraft2.  You can use
the history tab to see the differences between that and the current
guidelines.

Now, the real issue is in what packagers can depend upon.  Obviously you
have RPM and an environment necessary to build a package (which means
you have to have a shell to execute the scripts that make up the RPM
sections, and redhat-rpm-config, and whatever RPM happens to use to
execute %patch, which I guess could be some internal library if the RPM
devs wanted).  But what else?  That's the open question.

The draft currently says:

---
You may assume that you have everything necessary for RPM to function
and process your spec file (so of course RPM is present, along with
redhat-rpm-config and what is necessary for RPM to apply patches, unpack
archives, and run the shell scripts which make up the spec file
sections.) You should not assume any other packages are present, as RPM
dependencies and anything brought into the buildroot by the build system
may change over time.
---

Honestly, I'm not completely satisfied with that but I can't come up
with anything better.  Polite discussion is welcomed and encouraged.

 - J<

[Vít Ondruch]

The draft is still a bit wordy IMO.

For example, I don't understand why is there mentioned
redhat-rpm-config, since most people have no idea what this package is
good for (you can more or less count me in this group as well ;).
Patching stuff etc, you don't need to care who provides the
functionality, since it is hidden behind RPM macros. Most of the
dependencies from the current exception list are pulled in via RPM anyway.

Also this sentence "but even then keep in mind that you should still
double check your BuildRequires: because your package may still rely on
the particular makeup of the build environment." is just beating a dead
horse.

Otherwise big +1 for this change.


Vít


Dne 12.2.2015 v 17:38 Jason L Tibbitts III napsal(a):
>  TL;DR: There's a proposal for nuking the list of BuildRequires
>  exceptions from the packaging guidelines.
>
>  Important note: this is just an idea.  We're not currently changing the
>  guidelines based on this.  Nothing is set in stone.  It may end up that
>  nothing happens.  Don't panic.
>
>  A couple of times recently, people have floated proposals to the FPC
>  about removing certain things from the buildroot.  (Specifically, perl
>  and gcc*.)  And the bottom line for these is that while FPC tries to
>  maintain a list of BuildRequires: exceptions, this is really up to
>  releng and whatever rpm happens to pull in at any particular time.
>
>  This means that whatever list FPC maintains, it's going to get outdated
>  eventually, and giving packagers the idea that they don't have to
>  actually specify all of their dependencies restricts what releng can do
>  and, perhaps, what dependencies the RPM package can drop without
>  breaking builds.
>
>  So, the generally stated proposal:
>
>  I would like to get FPC out of the business of specifying this list of
>  exceptions, and instead just indicate that packagers should completely
>  specify their dependencies.
>
>  My specific proposal can be seen in
>  https://fedorahosted.org/fpc/ticket/497; the draft is at
>  https://fedoraproject.org/wiki/User:Tibbs/BuildReqDraft2.  You can use
>  the history tab to see the differences between that and the current
>  guidelines.
>
>  Now, the real issue is in what packagers can depend upon.  Obviously you
>  have RPM and an environment necessary to build a package (which means
>  you have to have a shell to execute the scripts that make up the RPM
>  sections, and redhat-rpm-config, and whatever RPM happens to use to
>  execute %patch, which I guess could be some internal library if the RPM
>  devs wanted).  But what else?  That's the open question.
>
>  The draft currently says:
>
>  ---
>  You may assume that you have everything necessary for RPM to function
>  and process your spec file (so of course RPM is present, along with
>  redhat-rpm-config and what is necessary for RPM to apply patches, unpack
>  archives, and run the shell scripts which make up the spec file
>  sections.) You should not assume any other packages are present, as RPM
>  dependencies and anything brought into the buildroot by the build system
>  may change over time.
>  ---
>
>  Honestly, I'm not completely satisfied with that but I can't come up
>  with anything better.  Polite discussion is welcomed and encouraged.
>
>   - J<
>  --
>  packaging mailing list
>  packaging(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/packaging

[Jason L Tibbitts III]

It's been a long time since I first proposed this, but I had a bit of
time so I took the feedback from this list and updated my draft for chopping
down the BuildRequires: section in the packaging guidelines.  Basically,
I'm now at a single paragraph:


It is important that your package list all necessary build dependencies
using the BuildRequires: tag. You may assume that enough of an
environment exists for RPM to function and execute basic shell scripts,
but you should not assume any other packages are present as RPM
dependencies and anything brought into the buildroot by the build system
may change over time.


Comments are welcome.

[Dominik 'Rathann' Mierzejewski]

On Tuesday, 05 May 2015 at 01:34, Jason L Tibbitts III wrote:
[...]
>  It is important that your package list all necessary build
> dependencies
>  using the BuildRequires: tag. You may assume that enough of an
>  environment exists for RPM to function and execute basic shell scripts,
>  but you should not assume any other packages are present as RPM
>  dependencies and anything brought into the buildroot by the build system
>  may change over time. 
+1. Thanks for working on this.

Regards,
Dominik
-- 
Fedora http://fedoraproject.org/wiki/User:Rathann
RPMFusion http://rpmfusion.org
"Faith manages."
        -- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"

[Ville Skyttä]

5.5.2015 2.34 "Jason L Tibbitts III" <tibbs(a)math.uh.edu&gt; kirjoitti:
>  You may assume that enough of an
>  environment exists for RPM to function and execute basic shell scripts 
s/RPM/rpmbuild/, and in addition to running shell scripts it is in a state
where it can unpack common archive types and apply patches?

[Vít Ondruch]

Dne 6.5.2015 v 12:34 Ville Skyttä napsal(a):
>  5.5.2015 2.34 "Jason L Tibbitts III" <tibbs(a)math.uh.edu
>  <mailto:[e-mail]>> kirjoitti:
>  >
>  > You may assume that enough of an
>  > environment exists for RPM to function and execute basic shell scripts
>
>  s/RPM/rpmbuild/, and in addition to running shell scripts it is in a
>  state where it can unpack common archive types and apply patches?
>
>  
"You may assume that enough of an environment exists for rpmbuild to
function" might be enough. Not sure if you really want to go into details.


# dnf repoquery --disablerepo=* --enablerepo=rawhide --requires rpm-build
Using metadata from Wed May  6 13:13:11 2015
rtld(GNU_HASH)
/bin/sh
libm.so.6()(64bit)
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)
libdl.so.2()(64bit)
libz.so.1()(64bit)
/bin/bash
libc.so.6(GLIBC_2.17)(64bit)
libbz2.so.1()(64bit)
libnss3.so()(64bit)
libpopt.so.0()(64bit)
libdb-5.3.so()(64bit)
liblzma.so.5()(64bit)
libpopt.so.0(LIBPOPT_0)(64bit)
libcap.so.2()(64bit)
grep
sed
libacl.so.1()(64bit)
liblua-5.3.so()(64bit)
librpm.so.3()(64bit)
gawk
librpmio.so.3()(64bit)
bzip2
gzip
libmagic.so.1()(64bit)
tar
libelf.so.1()(64bit)
xz
binutils
findutils
libelf.so.1(ELFUTILS_1.0)(64bit)
unzip
diffutils
file
cpio
librpmbuild.so.3()(64bit)
perl-generators
system-rpm-config
/usr/bin/gdb-add-index
rpm = 4.12.0.1-14.fc23
elfutils >= 0.128
patch >= 2.5
pkgconfig >= 1:0.24


Vít

[Jason L Tibbitts III]

> >>>> "VO" == Vít Ondruch
> <vondruch(a)redhat.com&gt; writes: 
VO> # dnf repoquery --disablerepo=* --enablerepo=rawhide --requires rpm-build

Those dependencies could change at any time.  I would like for them to
be able to change without the guidelines having to change with them.
Obviously any change would break some package somewhere, but at least it
would get one committee out of the process.

 - J<

[Ralf Corsepius]

On 05/06/2015 07:12 PM, Jason L Tibbitts III wrote:
> >>>>> "VO" == Vít Ondruch
> <vondruch(a)redhat.com&gt; writes:
>
>  VO> # dnf repoquery --disablerepo=* --enablerepo=rawhide --requires rpm-build
>
>  Those dependencies could change at any time.  I would like for them to
>  be able to change without the guidelines having to change with them.
>  Obviously any change would break some package somewhere, but at least it
>  would get one committee out of the process. 
As much as I welcome this effort, I think we need a detailed and fixed 
per-fedora-release package list, to be able to give packagers some 
helpful guidance.

That said, why can't we have a link to the list being used by mock
(The packages being listed in mock's "buildsys-build") inside the FPG?

Ralf

[Vít Ondruch]

Dne 6.5.2015 v 19:48 Ralf Corsepius napsal(a):
>  On 05/06/2015 07:12 PM, Jason L Tibbitts III wrote:
> >>>>>> "VO" == Vít Ondruch <vondruch(a)redhat.com&gt;
> writes:
> >
> > VO> # dnf repoquery --disablerepo=* --enablerepo=rawhide --requires
> > rpm-build
> >
> > Those dependencies could change at any time.  I would like for them to
> > be able to change without the guidelines having to change with them.
> > Obviously any change would break some package somewhere, but at least it
> > would get one committee out of the process.
>
>  As much as I welcome this effort, I think we need a detailed and fixed
>  per-fedora-release package list, to be able to give packagers some
>  helpful guidance.
>
>  That said, why can't we have a link to the list being used by mock
>  (The packages being listed in mock's "buildsys-build") inside the FPG?
>  
What would be the point of link to comps when most of the dependencies
are defined by rpm-build package? Actually, the buildsys-build should
contain just rpm-build and nothing more (or it could be abandoned
entirely, since it would loose its purpose this way).

Moreover, if the buildsys-build contains components which are already
required by rpm-build, they should be cleaned up immediately. I am
speaking about bzip2, gzip, tar, xz, sed, patch, grep, gawk, findutils,
diffutils, cpio, bash, these are all duplicated.


Vít

[Ralf Corsepius]

On 05/07/2015 09:21 AM, Vít Ondruch wrote:
>  Dne 6.5.2015 v 19:48 Ralf Corsepius napsal(a):
> > On 05/06/2015 07:12 PM, Jason L Tibbitts III wrote:
> >>>>>>> "VO" == Vít Ondruch <vondruch(a)redhat.com&gt;
> writes:
> >>
> >> VO> # dnf repoquery --disablerepo=* --enablerepo=rawhide --requires
> >> rpm-build
> >>
> >> Those dependencies could change at any time.  I would like for them to
> >> be able to change without the guidelines having to change with them.
> >> Obviously any change would break some package somewhere, but at least it
> >> would get one committee out of the process.
> >
> > As much as I welcome this effort, I think we need a detailed and fixed
> > per-fedora-release package list, to be able to give packagers some
> > helpful guidance.
> >
> > That said, why can't we have a link to the list being used by mock
> > (The packages being listed in mock's "buildsys-build") inside the FPG?
> >
>
>  What would be the point of link to comps when most of the dependencies
>  are defined by rpm-build package? 
With the FPG being changed to the proposal, I am expecting
- length discussions (esp. in reviews) about whether tools "x" is 
guaranteed to be present in build-roots: "Is touch, ls, zip, etc. in 
build-root or not? (Been there, seen that many times :) )

- broken builds, because vanishing implicit BR's can trigger different 
sets of build conditions and thus break packages.
(Bugs/discussions along the lines package A in fcX had "feature A", 
fcX+1 lacks it).


>  Actually, the buildsys-build should
>  contain just rpm-build and nothing more (or it could be abandoned
>  entirely, since it would loose its purpose this way). 
I do not consider this to be
workable, because rpm-build's deps are just 
arbitrary requirements and _not_ a well defined fundation of tools.

That said, I do not consider "rpm-build" to be something to be featured.

Instead, we need an explict well defined set of tools which are 
guaranteed to be present throughout the life-time of a release.

Implementation-wise, this could be implemented as explict BRs of 
rpm-build, an independent package or a package group.
IMO, the appropriate party to define this set of packages is those 
people who define the set of packages in mock.

>  Moreover, if the buildsys-build contains components which are
> already
>  required by rpm-build, they should be cleaned up immediately. I am
>  speaking about bzip2, gzip, tar, xz, sed, patch, grep, gawk, findutils,
>  diffutils, cpio, bash, these are all duplicated. 
IMO, here, you are repeating my
reasoning above in different words: 
rpm-build's deps are arbitrary.


Ralf

[Vít Ondruch]

Dne 7.5.2015 v 09:48 Ralf Corsepius napsal(a):
>  On 05/07/2015 09:21 AM, Vít Ondruch wrote:
> > Dne 6.5.2015 v 19:48 Ralf Corsepius napsal(a):
> >> On 05/06/2015 07:12 PM, Jason L Tibbitts III wrote:
> >>>>>>>> "VO" == Vít Ondruch <vondruch(a)redhat.com&gt;
> writes:
> >>>
> >>> VO> # dnf repoquery --disablerepo=* --enablerepo=rawhide --requires
> >>> rpm-build
> >>>
> >>> Those dependencies could change at any time.  I would like for them to
> >>> be able to change without the guidelines having to change with them.
> >>> Obviously any change would break some package somewhere, but at
> >>> least it
> >>> would get one committee out of the process.
> >>
> >> As much as I welcome this effort, I think we need a detailed and fixed
> >> per-fedora-release package list, to be able to give packagers some
> >> helpful guidance.
> >>
> >> That said, why can't we have a link to the list being used by mock
> >> (The packages being listed in mock's "buildsys-build") inside the
> FPG?
> >>
> >
> > What would be the point of link to comps when most of the dependencies
> > are defined by rpm-build package?
>
>  With the FPG being changed to the proposal, I am expecting
>  - length discussions (esp. in reviews) about whether tools "x" is
>  guaranteed to be present in build-roots: "Is touch, ls, zip, etc. in
>  build-root or not? (Been there, seen that many times :) )
>  
No guarantee is the best guarantee.

>  - broken builds, because vanishing implicit BR's can trigger
> different
>  sets of build conditions and thus break packages.
>  (Bugs/discussions along the lines package A in fcX had "feature A",
>  fcX+1 lacks it). 
You pretend like this does not happen on any level just a bit deeper
then what is defined by buildsys-build or rpm-build. But the bad news is
that it happens anywhere any time.

> > Actually, the buildsys-build should
> > contain just rpm-build and nothing more (or it could be abandoned
> > entirely, since it would loose its purpose this way).
>  I do not consider this to be workable, because rpm-build's deps are
>  just arbitrary requirements and _not_ a well defined fundation of tools.
>
>  That said, I do not consider "rpm-build" to be something to be featured.
>
>  Instead, we need an explict well defined set of tools which are
>  guaranteed to be present throughout the life-time of a release.
>
>  Implementation-wise, this could be implemented as explict BRs of
>  rpm-build, an independent package or a package group.
>  IMO, the appropriate party to define this set of packages is those
>  people who define the set of packages in mock.
>
> > Moreover, if the buildsys-build contains components which are already
> > required by rpm-build, they should be cleaned up immediately. I am
> > speaking about bzip2, gzip, tar, xz, sed, patch, grep, gawk, findutils,
> > diffutils, cpio, bash, these are all duplicated.
>  IMO, here, you are repeating my reasoning above in different words:
>  rpm-build's deps are arbitrary.
>  
As well as any defined group is arbitrary. We have probably disagree here.


Vít

[Nico Kadel-Garcia]

On Thu, May 7, 2015 at 3:56 AM, Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Dne 7.5.2015 v 09:48 Ralf Corsepius napsal(a):
> > On 05/07/2015 09:21 AM, Vít Ondruch wrote:
> >> Dne 6.5.2015 v 19:48 Ralf Corsepius napsal(a):
> >>> On 05/06/2015 07:12 PM, Jason L Tibbitts III wrote:
> >>>>>>>>> "VO" == Vít Ondruch
> <vondruch(a)redhat.com&gt; writes:
> >>>>
> >>>> VO> # dnf repoquery --disablerepo=* --enablerepo=rawhide --requires
> >>>> rpm-build
> >>>>
> >>>> Those dependencies could change at any time.  I would like for them to
> >>>> be able to change without the guidelines having to change with them.
> >>>> Obviously any change would break some package somewhere, but at
> >>>> least it
> >>>> would get one committee out of the process.
> >>>
> >>> As much as I welcome this effort, I think we need a detailed and fixed
> >>> per-fedora-release package list, to be able to give packagers some
> >>> helpful guidance.
> >>>
> >>> That said, why can't we have a link to the list being used by mock
> >>> (The packages being listed in mock's "buildsys-build") inside
> the FPG?
> >>>
> >>
> >> What would be the point of link to comps when most of the dependencies
> >> are defined by rpm-build package?
> >
> > With the FPG being changed to the proposal, I am expecting
> > - length discussions (esp. in reviews) about whether tools "x" is
> > guaranteed to be present in build-roots: "Is touch, ls, zip, etc. in
> > build-root or not? (Been there, seen that many times :) )
> >
>
>  No guarantee is the best guarantee. 
Stability and clarity help. One of the classic dependencies that gets
mucked up, right now, is 'mysql-libs'', with different MySQL variants
each having their own version of it, with a different name, and
different interwoven dependencies. Another is when there are two
parallel versions of the same component, such as 'samba' and 'samba4'
had for a while.

> >> Moreover, if the buildsys-build contains components which are
> already
> >> required by rpm-build, they should be cleaned up immediately. I am
> >> speaking about bzip2, gzip, tar, xz, sed, patch, grep, gawk, findutils,
> >> diffutils, cpio, bash, these are all duplicated.
> > IMO, here, you are repeating my reasoning above in different words:
> > rpm-build's deps are arbitrary.
>
>  As well as any defined group is arbitrary. We have probably disagree here. 
One reason to be more explicit about some of these is backwardsa
compatibility. If a particular ".spec" file is still stable for Fedora
or RHEL releases before the current rpmbuild contained these
particular components, "zip", in particular, is not required by 'mock'
baed rpmbuild environments, but some source bundles are now using it
because of GPG signed git tag exports for source code, rather than
published tarballs. I'm actually fairly horrified by how Red Hat is
publishing their RHEL source code these days, and really wish they'd
switch to using such tags rather then the very funky "read the git
logs and look for the word 'import'" to denote what everyone else in
the world would make a git tag, even a signed git tag these days.

[Vít Ondruch]

Dne 6.5.2015 v 19:12 Jason L Tibbitts III napsal(a):
> >>>>> "VO" == Vít Ondruch
> <vondruch(a)redhat.com&gt; writes:
>  VO> # dnf repoquery --disablerepo=* --enablerepo=rawhide --requires rpm-build
>
>  Those dependencies could change at any time.  I would like for them to
>  be able to change without the guidelines having to change with them.
>  Obviously any change would break some package somewhere, but at least it
>  would get one committee out of the process.
>
>   - J< 
Actually that was my point, why not speak about shell, unpacking
archives, applying patches etc and hence the proposed abbreviated
version  of guideline you leaved out from the quote :)


Vít

[Jason L Tibbitts III]

> >>>> "VO" == Vít Ondruch
> <vondruch(a)redhat.com&gt; writes: 
VO> Actually that was my point, why not speak about shell, unpacking
VO> archives, applying patches etc and hence the proposed abbreviated
VO> version of guideline you leaved out from the quote :)

I know, I was reinforcing your point, not refuting it.

 - J<

[Jason L Tibbitts III]

> >>>> "VS" == Ville Skyttä
> <ville.skytta(a)iki.fi&gt; writes: 
VS> s/RPM/rpmbuild/, and in addition to running shell scripts it is in a
VS> state where it can unpack common archive types and apply patches?

RPM can do that, but you should not assume how.  It could link in
libraries to do that without actually installing any executables it
could call.  Or it could call things which are not in $PATH.

 - J<

[Ville Skyttä]

On Wed, May 6, 2015 at 8:10 PM, Jason L Tibbitts III <tibbs(a)math.uh.edu&gt; wrote:
> >>>>> "VS" == Ville Skyttä
> <ville.skytta(a)iki.fi&gt; writes:
>
>  VS> s/RPM/rpmbuild/, and in addition to running shell scripts it is in a
>  VS> state where it can unpack common archive types and apply patches?
>
>  RPM can do that, but you should not assume how.  It could link in
>  libraries to do that without actually installing any executables it
>  could call.  Or it could call things which are not in $PATH. 
Sure (well except that it's still rpmbuild, not rpm...). I wasn't
making any assumptions how it does that, actually pretty much the
opposite. I just listed its features/state that can be assumed so that
people won't start adding build dependencies e.g. for tar and gzip
whenever they have a tgz source tarball unpacked by %setup or one for
patch when they have patches to apply with %patch etc.

[Dave Johansen]

On Thu, Feb 12, 2015 at 9:38 AM, Jason L Tibbitts III <tibbs(a)math.uh.edu&gt;
wrote:

>  The draft currently says:
>
>  ---
>  You may assume that you have everything necessary for RPM to function
>  and process your spec file (so of course RPM is present, along with
>  redhat-rpm-config and what is necessary for RPM to apply patches, unpack
>  archives, and run the shell scripts which make up the spec file
>  sections.) You should not assume any other packages are present, as RPM
>  dependencies and anything brought into the buildroot by the build system
>  may change over time.
>  ---
>  
I personally love this idea and hope a solution can be found. In my mind,
there are two main goals here that are closely related (and probably even a
bit intertwined/competing):
1) Reduce the number of programs that packagers "assume" is available, and
2) Give RPM maximum latitude to change implementation without breaking
anything.

I think that changing the wording to say that RPM will have "capabilities"
but not "any specific program to accomplish those capabilities" will
satisfy the above goals.

My proposed wording would be something like this:

You may assume that RPM has the ability to function
and process your spec file. This means that RPM is present it can perform
the fundamental tasks (like apply patches, unpack
archives, and run the shell scripts which make up the spec file
sections), but how these tasks are performed (programs/libraries/etc)
*cannot* be relied on.
In other words, you should not assume any other packages/programs/libraries
are present, as RPM
dependencies and anything brought into the buildroot by the build system
may change over time.