Let's standardize the way to disable tests during RPM build?

[Tomas Orsava]

Hi,
I think it would be useful to have a standard way of disabling the 
running of tests during RPM build (in the %check section of a spec file).

I see a lot of packages already having %bcond's or other macro 
definitions to archieve this, but each package has their own way, 
there's no real standard. Thus you have to first look into the spec, 
locate the appropriate %bcond or macro name and only then you can 
disable the tests.

I would like to propose two approaches:

(a) Add a *SHOULD* rule to the guidelines that specifies what is the 
preferred way to conditionalize the tests.

(b) Or, if that's too strong, mention in the guidelines the common 
methods that are being used (e.g. %bcond tests and %bcond check) so that 
new packagers have something to use.

What do you think?
Tomas

[Richard W.M. Jones]

On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
>  Hi,
>  I think it would be useful to have a standard way of disabling the
>  running of tests during RPM build (in the %check section of a spec
>  file).
>  
>  I see a lot of packages already having %bcond's or other macro
>  definitions to archieve this, but each package has their own way,
>  there's no real standard. Thus you have to first look into the spec,
>  locate the appropriate %bcond or macro name and only then you can
>  disable the tests.
>  
>  I would like to propose two approaches:
>  
>  (a) Add a *SHOULD* rule to the guidelines that specifies what is the
>  preferred way to conditionalize the tests.
>  
>  (b) Or, if that's too strong, mention in the guidelines the common
>  methods that are being used (e.g. %bcond tests and %bcond check) so
>  that new packagers have something to use. 
What's the motivation for disabling tests globally?

I have some packages where tests fail on particular architectures at
particular times, and what I do there is (a) file a BZ (b) surround
the %check section with %ifarch/%ifnarch and a comment linking to the
bug, and this seems to me a practical and lightweight approach that
requires no special support in the toolchain.

Also rpmbuild itself can happily disable tests, just add the --nocheck
flag.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Fedora-packaging] Let's standardize the way to disable tests during RPM build?

[Paul Wouters]

Or just a new option to rpmbuild that skips %check ? 

Sent from my iPhone

>  On Jun 5, 2020, at 10:11, Tomas Orsava <torsava(a)redhat.com&gt;
> wrote:
>  
>  Hi,
>  I think it would be useful to have a standard way of disabling the running of tests
> during RPM build (in the %check section of a spec file).
>  
>  I see a lot of packages already having %bcond's or other macro definitions to
> archieve this, but each package has their own way, there's no real standard. Thus you
> have to first look into the spec, locate the appropriate %bcond or macro name and only
> then you can disable the tests.
>  
>  I would like to propose two approaches:
>  
>  (a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to
> conditionalize the tests.
>  
>  (b) Or, if that's too strong, mention in the guidelines the common methods that are
> being used (e.g. %bcond tests and %bcond check) so that new packagers have something to
> use.
>  
>  What do you think?
>  Tomas
>  _______________________________________________
>  packaging mailing list -- packaging(a)lists.fedoraproject.org
>  To unsubscribe send an email to packaging-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...

[Miro Hrončok]

On 05. 06. 20 16:26, Richard W.M. Jones wrote:
>  On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
> > Hi,
> > I think it would be useful to have a standard way of disabling the
> > running of tests during RPM build (in the %check section of a spec
> > file).
> >
> > I see a lot of packages already having %bcond's or other macro
> > definitions to archieve this, but each package has their own way,
> > there's no real standard. Thus you have to first look into the spec,
> > locate the appropriate %bcond or macro name and only then you can
> > disable the tests.
> >
> > I would like to propose two approaches:
> >
> > (a) Add a *SHOULD* rule to the guidelines that specifies what is the
> > preferred way to conditionalize the tests.
> >
> > (b) Or, if that's too strong, mention in the guidelines the common
> > methods that are being used (e.g. %bcond tests and %bcond check) so
> > that new packagers have something to use.
>  
>  What's the motivation for disabling tests globally? 
Bootstrapping mostly.

>  I have some packages where tests fail on particular architectures at
>  particular times, and what I do there is (a) file a BZ (b) surround
>  the %check section with %ifarch/%ifnarch and a comment linking to the
>  bug, and this seems to me a practical and lightweight approach that
>  requires no special support in the toolchain.
>  
>  Also rpmbuild itself can happily disable tests, just add the --nocheck
>  flag. 
However rpmbuild itself doesn't support "CheckRequires", so the bcond often

looks like this:


BuildRequires: python3-devel
BuildRequires: python3-setuptools
%if %{with tests}
BuildRequires: python3-pytest
BuildRequires: python3-hypothesis
%endif

...

%if %{with tests}
%check
%pytest
%endif


-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Fedora-packaging] Let's standardize the way to disable tests during RPM build?

[Igor Raits]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote:
>  Hi,
>  I think it would be useful to have a standard way of disabling the 
>  running of tests during RPM build (in the %check section of a spec
>  file).
>  
>  I see a lot of packages already having %bcond's or other macro 
>  definitions to archieve this, but each package has their own way, 
>  there's no real standard. Thus you have to first look into the spec, 
>  locate the appropriate %bcond or macro name and only then you can 
>  disable the tests.
>  
>  I would like to propose two approaches:
>  
>  (a) Add a *SHOULD* rule to the guidelines that specifies what is the 
>  preferred way to conditionalize the tests.
>  
>  (b) Or, if that's too strong, mention in the guidelines the common 
>  methods that are being used (e.g. %bcond tests and %bcond check) so
>  that 
>  new packagers have something to use.
>  
>  What do you think? 
I'd like to have this finally be implemented in
https://github.com/rpm-software-management/rpm/issues/316. That way it
would be simply rpmbuild --nocheck or define %_without_check 1 which
would skip %check section entirely.

For now, all Rust crates just have `%bcond_without check` so using `--
without check` works just fine there.

Since this would be more generic thing to the RPM ecosystem, adding
rpm-ecosystem@ to the copy.

>  Tomas
>  _______________________________________________
>  packaging mailing list -- packaging(a)lists.fedoraproject.org
>  To unsubscribe send an email to  
>  packaging-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:  
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines:  
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:  
>  https://lists.fedoraproject.org/archives/list/[e-mail]... 
-
-- 
Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7aWQwACgkQEV1auJxc
> Hh6dyBAAmbJSCU0wtuET7vuXVVIeg7BeosaQF25/VoMwTSYGH3h36S9Gci9BRBgs
> yuque1uGnBaUQ74fsxBIMgGzapd73TvEY1M8PNnzHF3Miz0i0FgVhwnw3S9jvrTT
> aGqln2rE3L5jH0alII6pNOIqA67yPlYfb5+JtRazeO0KTarZuGOdemJsp6ONEKQS
> 5doQid6yrQvaUj90Xl2VpRY6goXx5FOQLDPb9DlaWlQDvBcVBJz5oaJ/VyxqCnC2
> ObyLjMB9AXq+pBiot/50QDLTUCxKOkro1siBPxfswNCjpwRy6vDp6dyczHyQkhJ8
> zFAHJQPWAr870WU3FMO/FirTv9yAqY6Je8jB+3EdxjzNuyBMTOT6Iq6r8Su/yxeq
> FcvDvUhlJ0OtWM8PfiIkaKpiSB9rzpuuM5hagPYqznLbqu5AeuTqAKojSyLbkK7Z
> 7fS+qABernfYqAVOlq7DkTaETh/0sAuIxhtwWXbbhz7vFPpbnsPdnyfUUGzFoIdT
> LBFnMOBQF0q4woTAhQRHez+VEH4ndiqZQGdYL8AJ9FtKeMZwwWmvl/r3ki/Hr5Yf
> bqETizKe4XBu5DxPRNN3+0RSi+TIXX11VeHtxIWeuGGErgdqq4EZkBfnmZlTb3/N
> gA8/3DUl9B4XRzGjnzq0AahOfIW1wNObh4pzlmoGN2jrG5odKPM=
> =JCw+
> -----END PGP SIGNATURE-----

[Fedora-packaging] Let's standardize the way to disable tests during RPM build?

[Richard W.M. Jones]

On Fri, Jun 05, 2020 at 10:28:39AM -0400, Paul Wouters wrote:
>  Or just a new option to rpmbuild that skips %check ?  
It exists already: rpmbuild --nocheck.

It's not wired into the rest of the stack - eg. you cannot start a
Koji build with checks disabled.  IMHO that's a good thing, although
when we first started doing the RISC-V bootstrap we initially and
briefly used this option to disable the tests, for convenience of
getting packages built.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Tomas Orsava]

On 6/5/20 4:26 PM, Richard W.M. Jones wrote:
>  On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
> > Hi,
> > I think it would be useful to have a standard way of disabling the
> > running of tests during RPM build (in the %check section of a spec
> > file).
> >
> > I see a lot of packages already having %bcond's or other macro
> > definitions to archieve this, but each package has their own way,
> > there's no real standard. Thus you have to first look into the spec,
> > locate the appropriate %bcond or macro name and only then you can
> > disable the tests.
> >
> > I would like to propose two approaches:
> >
> > (a) Add a *SHOULD* rule to the guidelines that specifies what is the
> > preferred way to conditionalize the tests.
> >
> > (b) Or, if that's too strong, mention in the guidelines the common
> > methods that are being used (e.g. %bcond tests and %bcond check) so
> > that new packagers have something to use.
>  What's the motivation for disabling tests globally? 
For example bootstrapping.

And this doesn't only benefit us on a global level, it also lowers the 
cognitive load when you're working with a random package, for example 
doing a PR.

>  I have some packages where tests fail on particular architectures at
>  particular times, and what I do there is (a) file a BZ (b) surround
>  the %check section with %ifarch/%ifnarch and a comment linking to the
>  bug, and this seems to me a practical and lightweight approach that
>  requires no special support in the toolchain.
>
>  Also rpmbuild itself can happily disable tests, just add the --nocheck
>  flag. 
Indeed, but it's not supported by Koji, for example.

[Fedora-packaging] Let's standardize the way to disable tests during RPM build?

[Tomas Orsava]

On 6/5/20 4:39 PM, Richard W.M. Jones wrote:
>  On Fri, Jun 05, 2020 at 10:28:39AM -0400, Paul Wouters wrote:
> > Or just a new option to rpmbuild that skips %check ?
>  It exists already: rpmbuild --nocheck.
>
>  It's not wired into the rest of the stack - eg. you cannot start a
>  Koji build with checks disabled.  IMHO that's a good thing, although
>  when we first started doing the RISC-V bootstrap we initially and
>  briefly used this option to disable the tests, for convenience of
>  getting packages built. 
It might be a good thing for regular builds, but it's sorely missed in 
scratch builds.

[Richard W.M. Jones]

On Fri, Jun 05, 2020 at 04:38:03PM +0200, Miro Hrončok wrote:
>  On 05. 06. 20 16:26, Richard W.M. Jones wrote:
>  >On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
>  >>Hi,
>  >>I think it would be useful to have a standard way of disabling the
>  >>running of tests during RPM build (in the %check section of a spec
>  >>file).
>  >>
>  >>I see a lot of packages already having %bcond's or other macro
>  >>definitions to archieve this, but each package has their own way,
>  >>there's no real standard. Thus you have to first look into the spec,
>  >>locate the appropriate %bcond or macro name and only then you can
>  >>disable the tests.
>  >>
>  >>I would like to propose two approaches:
>  >>
>  >>(a) Add a *SHOULD* rule to the guidelines that specifies what is the
>  >>preferred way to conditionalize the tests.
>  >>
>  >>(b) Or, if that's too strong, mention in the guidelines the common
>  >>methods that are being used (e.g. %bcond tests and %bcond check) so
>  >>that new packagers have something to use.
>  >
>  >What's the motivation for disabling tests globally?
>  
>  Bootstrapping mostly. 
For the RISC-V bootstrap we used rpmbuild directly (before Koji and
its dependencies had been ported), and added --nocheck.  However once
Koji was working we built packages properly with checks enabled.

How often do we bootstrap Fedora from scratch?  Wholly new
architectures are rare.  Are there other events that require
bootstrapping from scratch?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Tomas Orsava]

On 6/5/20 4:46 PM, Richard W.M. Jones wrote:
>  On Fri, Jun 05, 2020 at 04:38:03PM +0200, Miro Hrončok wrote:
> > On 05. 06. 20 16:26, Richard W.M. Jones wrote:
> >> On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
> >>> Hi,
> >>> I think it would be useful to have a standard way of disabling the
> >>> running of tests during RPM build (in the %check section of a spec
> >>> file).
> >>>
> >>> I see a lot of packages already having %bcond's or other macro
> >>> definitions to archieve this, but each package has their own way,
> >>> there's no real standard. Thus you have to first look into the spec,
> >>> locate the appropriate %bcond or macro name and only then you can
> >>> disable the tests.
> >>>
> >>> I would like to propose two approaches:
> >>>
> >>> (a) Add a *SHOULD* rule to the guidelines that specifies what is the
> >>> preferred way to conditionalize the tests.
> >>>
> >>> (b) Or, if that's too strong, mention in the guidelines the common
> >>> methods that are being used (e.g. %bcond tests and %bcond check) so
> >>> that new packagers have something to use.
> >> What's the motivation for disabling tests globally?
> > Bootstrapping mostly.
>  For the RISC-V bootstrap we used rpmbuild directly (before Koji and
>  its dependencies had been ported), and added --nocheck.  However once
>  Koji was working we built packages properly with checks enabled.
>
>  How often do we bootstrap Fedora from scratch?  Wholly new
>  architectures are rare.  Are there other events that require
>  bootstrapping from scratch? 
Not necessarily bootstrapping from scratch, this is useful for 
bootstrapping of anything in Fedora.

Fod example, Python now releases on a yearly schedule, and bootstrapping 
it is a huge undertaking involving thousands of components.


And most importantly, the reason tests are disabled during bootstrapping 
is missing dependencies. Those have to be conditionalized by some %bcond 
or macro, and `--nocheck` doesn't help.

Tomas

[Nicolas Mailhot]

Le vendredi 05 juin 2020 à 15:46 +0100, Richard W.M. Jones a écrit :
>  
>  For the RISC-V bootstrap we used rpmbuild directly (before Koji and
>  its dependencies had been ported), and added --nocheck.  However once
>  Koji was working we built packages properly with checks enabled.
>  
>  How often do we bootstrap Fedora from scratch?  Wholly new
>  architectures are rare.  Are there other events that require
>  bootstrapping from scratch? 
Some language ecosystems have very low quality unit tests, any mass
rebuild (every release) basically works in two passes, make packages
build with test disabled, then redo-it with tests and sift through test
results to see what is actual breakage and what is broken testing code

The people who release poor unit tests also change their dep graph at
high speed, poor unit tests go hand in hand with regular re-bootstraps

-- 
Nicolas Mailhot

[Fedora-packaging] Let's standardize the way to disable tests during RPM build?

[Miro Hrončok]

On 05. 06. 20 16:45, Tomas Orsava wrote:
>  On 6/5/20 4:39 PM, Richard W.M. Jones wrote:
> > On Fri, Jun 05, 2020 at 10:28:39AM -0400, Paul Wouters wrote:
> >> Or just a new option to rpmbuild that skips %check ?
> > It exists already: rpmbuild --nocheck.
> >
> > It's not wired into the rest of the stack - eg. you cannot start a
> > Koji build with checks disabled.  IMHO that's a good thing, although
> > when we first started doing the RISC-V bootstrap we initially and
> > briefly used this option to disable the tests, for convenience of
> > getting packages built.
>  
>  It might be a good thing for regular builds, but it's sorely missed in scratch 
>  builds. 
Generally, the ability to flip bconds in scratchbuilds would be a huge help over 
uploading the entire SRPM. Something like:

     $ fedpkg build --scratch --without tests --without optimizations

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Fedora-packaging] Let's standardize the way to disable tests during RPM build?

[Florian Festi]

May be https://github.com/rpm-software-management/rpm/pull/1256 does the
trick. Comments welcome!

Florian

On 6/5/20 4:39 PM, Igor Raits wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA512
>  
>  On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote:
> > Hi,
> > I think it would be useful to have a standard way of disabling the 
> > running of tests during RPM build (in the %check section of a spec
> > file).
> >
> > I see a lot of packages already having %bcond's or other macro 
> > definitions to archieve this, but each package has their own way, 
> > there's no real standard. Thus you have to first look into the spec, 
> > locate the appropriate %bcond or macro name and only then you can 
> > disable the tests.
> >
> > I would like to propose two approaches:
> >
> > (a) Add a *SHOULD* rule to the guidelines that specifies what is the 
> > preferred way to conditionalize the tests.
> >
> > (b) Or, if that's too strong, mention in the guidelines the common 
> > methods that are being used (e.g. %bcond tests and %bcond check) so
> > that 
> > new packagers have something to use.
> >
> > What do you think?
>  
>  I'd like to have this finally be implemented in
>  https://github.com/rpm-software-management/rpm/issues/316. That way it
>  would be simply rpmbuild --nocheck or define %_without_check 1 which
>  would skip %check section entirely.
>  
>  For now, all Rust crates just have `%bcond_without check` so using `--
>  without check` works just fine there.
>  
>  Since this would be more generic thing to the RPM ecosystem, adding
>  rpm-ecosystem@ to the copy.
>  
> > Tomas
> > _______________________________________________
> > packaging mailing list -- packaging(a)lists.fedoraproject.org
> > To unsubscribe send an email to  
> > packaging-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:  
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:  
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:  
> > https://lists.fedoraproject.org/archives/list/[e-mail]...
>  - -- 
>  Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> >  -----BEGIN PGP SIGNATURE-----
> >  
> >  iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7aWQwACgkQEV1auJxc
> >  Hh6dyBAAmbJSCU0wtuET7vuXVVIeg7BeosaQF25/VoMwTSYGH3h36S9Gci9BRBgs
> >  yuque1uGnBaUQ74fsxBIMgGzapd73TvEY1M8PNnzHF3Miz0i0FgVhwnw3S9jvrTT
> >  aGqln2rE3L5jH0alII6pNOIqA67yPlYfb5+JtRazeO0KTarZuGOdemJsp6ONEKQS
> >  5doQid6yrQvaUj90Xl2VpRY6goXx5FOQLDPb9DlaWlQDvBcVBJz5oaJ/VyxqCnC2
> >  ObyLjMB9AXq+pBiot/50QDLTUCxKOkro1siBPxfswNCjpwRy6vDp6dyczHyQkhJ8
> >  zFAHJQPWAr870WU3FMO/FirTv9yAqY6Je8jB+3EdxjzNuyBMTOT6Iq6r8Su/yxeq
> >  FcvDvUhlJ0OtWM8PfiIkaKpiSB9rzpuuM5hagPYqznLbqu5AeuTqAKojSyLbkK7Z
> >  7fS+qABernfYqAVOlq7DkTaETh/0sAuIxhtwWXbbhz7vFPpbnsPdnyfUUGzFoIdT
> >  LBFnMOBQF0q4woTAhQRHez+VEH4ndiqZQGdYL8AJ9FtKeMZwwWmvl/r3ki/Hr5Yf
> >  bqETizKe4XBu5DxPRNN3+0RSi+TIXX11VeHtxIWeuGGErgdqq4EZkBfnmZlTb3/N
> >  gA8/3DUl9B4XRzGjnzq0AahOfIW1wNObh4pzlmoGN2jrG5odKPM=
> >  =JCw+
> >  -----END PGP SIGNATURE----- 
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>   

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Laurie Krebs, Michael O'Neill,
Thomas Savage

[Fedora-packaging] Let's standardize the way to disable tests during RPM build?

[Michal Schorm]

On Fri, Jun 5, 2020 at 5:48 PM Nicolas Mailhot via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Some language ecosystems have very low quality unit tests ... 
I'd discourage the "some test suites are bad, so let's disable them
all '' attitude.
Some very small and very stable testsuites may still be beneficial to
be run every time.

Though the "small" and "stable" again relies on the maintainer. (But
what does not, anyway ...)

On Fri, Jun 5, 2020 at 4:39 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  However rpmbuild itself doesn't support
> "CheckRequires", so the bcond often looks like this:
>  %if %{with tests}
>  BuildRequires: python3-pytest
>  %endif 
I fully agree. If the testsuite is not required to be run, sometimes
dozens of packages which are only test dependencies are not required
to be in the buildroot - thus they shouldn't be there.

---

Here's a little insight to huge MariaDB and MySQL packages:

MariaDB & MySQL have about 5000 tests in their testsuites nowadays
from which we run more than 4000.
Many of them scratch the whole DB and re-deploy it again, which *is*
time consuming.

While the build of the DB generally takes hours, the testsuite may
take ten times longer.
Historically on some slow-ass architectures, one build took about 2
days (looking at you 32-bit ARM)

Now if you add randomly failing tests - some because of the testsuite,
some because of the build machine (not enough ports available, or a
sudden heavy load triggered the test timeout, ...),
there's a huge need for decreasing the build time.

Currently, I implemented a macro which holds the "last tested version" value.
Once the maintainer (well, me), goes through the full testsuite for
all architectures, he can bump the macro value (to match the current
DB release version), and from then on, all later re-builds will run
only a very basic "sanity" testsuite which is short and stable, while
still check if the DB works at all.
Until a new release of the package is packed and the "last tested
version" value becomes lower than the current version and the full
testsuite needs to be run again.

This really helps anyone doing rebuilds and PRs.

Of course, that's not all. There are also implemented macros for "do
not run testsuite at all"; "run the full testsuite even though only
sanity check is required"; "if the testsuite will be run, ignore the
results (= always pass)", and so on.

Some may say such a huge testsuite shouldn't be run at the build time,
but well, this is just the upstream set of tests.
We have several other test frameworks with a lot of tests which are
triggered on each build. (e.g. installability tests, Red Hat internal
security regression tests, ... ) And the whole test suite again to
assure it passes in a real environment outside the buildroot.

--

Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat

--

On Mon, Jun 8, 2020 at 10:58 AM Florian Festi <ffesti(a)redhat.com&gt; wrote:
>  May be https://github.com/rpm-software-management/rpm/pull/1256 does the
>  trick. Comments welcome!
>
>  Florian
>
>  On 6/5/20 4:39 PM, Igor Raits wrote:
>  > -----BEGIN PGP SIGNED MESSAGE-----
>  > Hash: SHA512
>  >
>  > On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote:
>  >> Hi,
>  >> I think it would be useful to have a standard way of disabling the
>  >> running of tests during RPM build (in the %check section of a spec
>  >> file).
>  >>
>  >> I see a lot of packages already having %bcond's or other macro
>  >> definitions to archieve this, but each package has their own way,
>  >> there's no real standard. Thus you have to first look into the spec,
>  >> locate the appropriate %bcond or macro name and only then you can
>  >> disable the tests.
>  >>
>  >> I would like to propose two approaches:
>  >>
>  >> (a) Add a *SHOULD* rule to the guidelines that specifies what is the
>  >> preferred way to conditionalize the tests.
>  >>
>  >> (b) Or, if that's too strong, mention in the guidelines the common
>  >> methods that are being used (e.g. %bcond tests and %bcond check) so
>  >> that
>  >> new packagers have something to use.
>  >>
>  >> What do you think?
>  >
>  > I'd like to have this finally be implemented in
>  > https://github.com/rpm-software-management/rpm/issues/316. That way it
>  > would be simply rpmbuild --nocheck or define %_without_check 1 which
>  > would skip %check section entirely.
>  >
>  > For now, all Rust crates just have `%bcond_without check` so using `--
>  > without check` works just fine there.
>  >
>  > Since this would be more generic thing to the RPM ecosystem, adding
>  > rpm-ecosystem@ to the copy.
>  >
>  >> Tomas
>  >> _______________________________________________
>  >> packaging mailing list -- packaging(a)lists.fedoraproject.org
>  >> To unsubscribe send an email to
>  >> packaging-leave(a)lists.fedoraproject.org
>  >> Fedora Code of Conduct:
>  >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  >> List Guidelines:
>  >> https://fedoraproject.org/wiki/Mailing_list_guidelines
>  >> List Archives:
>  >> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  > - --
>  > Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> >  > -----BEGIN PGP SIGNATURE-----
> >  >
> >  > iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7aWQwACgkQEV1auJxc
> >  > Hh6dyBAAmbJSCU0wtuET7vuXVVIeg7BeosaQF25/VoMwTSYGH3h36S9Gci9BRBgs
> >  > yuque1uGnBaUQ74fsxBIMgGzapd73TvEY1M8PNnzHF3Miz0i0FgVhwnw3S9jvrTT
> >  > aGqln2rE3L5jH0alII6pNOIqA67yPlYfb5+JtRazeO0KTarZuGOdemJsp6ONEKQS
> >  > 5doQid6yrQvaUj90Xl2VpRY6goXx5FOQLDPb9DlaWlQDvBcVBJz5oaJ/VyxqCnC2
> >  > ObyLjMB9AXq+pBiot/50QDLTUCxKOkro1siBPxfswNCjpwRy6vDp6dyczHyQkhJ8
> >  > zFAHJQPWAr870WU3FMO/FirTv9yAqY6Je8jB+3EdxjzNuyBMTOT6Iq6r8Su/yxeq
> >  > FcvDvUhlJ0OtWM8PfiIkaKpiSB9rzpuuM5hagPYqznLbqu5AeuTqAKojSyLbkK7Z
> >  > 7fS+qABernfYqAVOlq7DkTaETh/0sAuIxhtwWXbbhz7vFPpbnsPdnyfUUGzFoIdT
> >  > LBFnMOBQF0q4woTAhQRHez+VEH4ndiqZQGdYL8AJ9FtKeMZwwWmvl/r3ki/Hr5Yf
> >  > bqETizKe4XBu5DxPRNN3+0RSi+TIXX11VeHtxIWeuGGErgdqq4EZkBfnmZlTb3/N
> >  > gA8/3DUl9B4XRzGjnzq0AahOfIW1wNObh4pzlmoGN2jrG5odKPM=
> >  > =JCw+
> >  > -----END PGP SIGNATURE----- 
>  >
> _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  >
>
>
>  --
>  Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
>  Commercial register: Amtsgericht Muenchen, HRB 153243,
>  Managing Directors: Charles Cachera, Laurie Krebs, Michael O'Neill,
>  Thomas Savage
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Fedora-packaging] Let's standardize the way to disable tests during RPM build?

[Tomas Orsava]

Oooh, that would be perfect!

Tomas

On 6/8/20 10:57 AM, Florian Festi wrote:
>  May be https://github.com/rpm-software-management/rpm/pull/1256 does
> the
>  trick. Comments welcome!
>
>  Florian
>
>  On 6/5/20 4:39 PM, Igor Raits wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> >
> > On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote:
> >> Hi,
> >> I think it would be useful to have a standard way of disabling the
> >> running of tests during RPM build (in the %check section of a spec
> >> file).
> >>
> >> I see a lot of packages already having %bcond's or other macro
> >> definitions to archieve this, but each package has their own way,
> >> there's no real standard. Thus you have to first look into the spec,
> >> locate the appropriate %bcond or macro name and only then you can
> >> disable the tests.
> >>
> >> I would like to propose two approaches:
> >>
> >> (a) Add a *SHOULD* rule to the guidelines that specifies what is the
> >> preferred way to conditionalize the tests.
> >>
> >> (b) Or, if that's too strong, mention in the guidelines the common
> >> methods that are being used (e.g. %bcond tests and %bcond check) so
> >> that
> >> new packagers have something to use.
> >>
> >> What do you think?
> > I'd like to have this finally be implemented in
> > https://github.com/rpm-software-management/rpm/issues/316. That way it
> > would be simply rpmbuild --nocheck or define %_without_check 1 which
> > would skip %check section entirely.
> >
> > For now, all Rust crates just have `%bcond_without check` so using `--
> > without check` works just fine there.
> >
> > Since this would be more generic thing to the RPM ecosystem, adding
> > rpm-ecosystem@ to the copy.
> >
> >> Tomas
> >> _______________________________________________
> >> packaging mailing list -- packaging(a)lists.fedoraproject.org
> >> To unsubscribe send an email to
> >> packaging-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines:
> >> https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >> https://lists.fedoraproject.org/archives/list/[e-mail]...
> > - -- 
> > Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> > > -----BEGIN PGP SIGNATURE-----
> > >
> > > iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7aWQwACgkQEV1auJxc
> > > Hh6dyBAAmbJSCU0wtuET7vuXVVIeg7BeosaQF25/VoMwTSYGH3h36S9Gci9BRBgs
> > > yuque1uGnBaUQ74fsxBIMgGzapd73TvEY1M8PNnzHF3Miz0i0FgVhwnw3S9jvrTT
> > > aGqln2rE3L5jH0alII6pNOIqA67yPlYfb5+JtRazeO0KTarZuGOdemJsp6ONEKQS
> > > 5doQid6yrQvaUj90Xl2VpRY6goXx5FOQLDPb9DlaWlQDvBcVBJz5oaJ/VyxqCnC2
> > > ObyLjMB9AXq+pBiot/50QDLTUCxKOkro1siBPxfswNCjpwRy6vDp6dyczHyQkhJ8
> > > zFAHJQPWAr870WU3FMO/FirTv9yAqY6Je8jB+3EdxjzNuyBMTOT6Iq6r8Su/yxeq
> > > FcvDvUhlJ0OtWM8PfiIkaKpiSB9rzpuuM5hagPYqznLbqu5AeuTqAKojSyLbkK7Z
> > > 7fS+qABernfYqAVOlq7DkTaETh/0sAuIxhtwWXbbhz7vFPpbnsPdnyfUUGzFoIdT
> > > LBFnMOBQF0q4woTAhQRHez+VEH4ndiqZQGdYL8AJ9FtKeMZwwWmvl/r3ki/Hr5Yf
> > > bqETizKe4XBu5DxPRNN3+0RSi+TIXX11VeHtxIWeuGGErgdqq4EZkBfnmZlTb3/N
> > > gA8/3DUl9B4XRzGjnzq0AahOfIW1wNObh4pzlmoGN2jrG5odKPM=
> > > =JCw+
> > > -----END PGP SIGNATURE----- 
> >
> _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> >
>

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Vít Ondruch]

Dne 05. 06. 20 v 17:24 Tomas Orsava napsal(a):
>  On 6/5/20 4:46 PM, Richard W.M. Jones wrote:
> > On Fri, Jun 05, 2020 at 04:38:03PM +0200, Miro Hrončok wrote:
> >> On 05. 06. 20 16:26, Richard W.M. Jones wrote:
> >>> On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
> >>>> Hi,
> >>>> I think it would be useful to have a standard way of disabling the
> >>>> running of tests during RPM build (in the %check section of a spec
> >>>> file).
> >>>>
> >>>> I see a lot of packages already having %bcond's or other macro
> >>>> definitions to archieve this, but each package has their own way,
> >>>> there's no real standard. Thus you have to first look into the spec,
> >>>> locate the appropriate %bcond or macro name and only then you can
> >>>> disable the tests.
> >>>>
> >>>> I would like to propose two approaches:
> >>>>
> >>>> (a) Add a *SHOULD* rule to the guidelines that specifies what is the
> >>>> preferred way to conditionalize the tests.
> >>>>
> >>>> (b) Or, if that's too strong, mention in the guidelines the common
> >>>> methods that are being used (e.g. %bcond tests and %bcond check) so
> >>>> that new packagers have something to use.
> >>> What's the motivation for disabling tests globally?
> >> Bootstrapping mostly.
> > For the RISC-V bootstrap we used rpmbuild directly (before Koji and
> > its dependencies had been ported), and added --nocheck.  However once
> > Koji was working we built packages properly with checks enabled.
> >
> > How often do we bootstrap Fedora from scratch?  Wholly new
> > architectures are rare.  Are there other events that require
> > bootstrapping from scratch?
>
>  Not necessarily bootstrapping from scratch, this is useful for
>  bootstrapping of anything in Fedora. 

Just FTR, we have bootstrapping guidelines:


https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping


Vít



>  Fod example, Python now releases on a yearly schedule, and
>  bootstrapping it is a huge undertaking involving thousands of components.
>
>
>  And most importantly, the reason tests are disabled during
>  bootstrapping is missing dependencies. Those have to be
>  conditionalized by some %bcond or macro, and `--nocheck` doesn't help.
>
>  Tomas
>  _______________________________________________
>  packaging mailing list -- packaging(a)lists.fedoraproject.org
>  To unsubscribe send an email to packaging-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Nicolas Mailhot]

Le mardi 09 juin 2020 à 12:08 +0200, Vít Ondruch a écrit :
>  
>  Just FTR, we have bootstrapping guidelines  
>  https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
>   
Those suffer from
1. the horrible bcond logic inversion that trips pretty much everyone
all the time.
2. the fact you can not ask koji or mock for a bootstrapped build, you
have to change the spec manually

-- 
Nicolas Mailhot

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Vít Ondruch]

Dne 09. 06. 20 v 12:12 Nicolas Mailhot napsal(a):
>  Le mardi 09 juin 2020 à 12:08 +0200, Vít Ondruch a écrit :
> > Just FTR, we have bootstrapping guidelines  
> > https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
> >
>  Those suffer from
>  1. the horrible bcond logic inversion that trips pretty much everyone
>  all the time. 

That won't be different for what was the original question here, i.e.
conditionally disable tests. bconds are what we have for better or worse.

And really, this seems about bootstrapping not disabling tests, which
are not completely different, but nobody can objects bootstrapping,
while disabling tests might be good just to improve build speed and
nothing else. That should never happen in production environment IMO.


>  2. the fact you can not ask koji or mock for a bootstrapped build,
> you
>  have to change the spec manually
>  
You can set them in modules and I think Koji can set them:

https://pagure.io/koji/issue/416

Not mentioning that there is almost always way to provide some macro
file, e.g. there is no reason for python bootstrapping all the packages
to not ship some macro in `/usr/lib/macros.d/macros.python-bootstrap`.


Vít

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Nicolas Mailhot]

Le mardi 09 juin 2020 à 12:21 +0200, Vít Ondruch a écrit :
>  Dne 09. 06. 20 v 12:12 Nicolas Mailhot napsal(a):
>  > Le mardi 09 juin 2020 à 12:08 +0200, Vít Ondruch a écrit :
>  > > Just FTR, we have bootstrapping guidelines  
>  > >  
>  > > https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
>  > > 
>  > Those suffer from
>  > 1. the horrible bcond logic inversion that trips pretty much
>  > everyone
>  > all the time.
>  
>  
>  That won't be different for what was the original question here, i.e.
>  conditionally disable tests. bconds are what we have for better or
>  worse. 
bconds had adoption problems from day one and will continue to have
them as long as they are not fixed to use a human-friendly syntax

>  And really, this seems about bootstrapping not disabling tests,
> which
>  are not completely different, but nobody can objects bootstrapping,
>  while disabling tests might be good just to improve build speed and
>  nothing else. That should never happen in production environment IMO. 
That depends entirely on upstream’s test quality. FYI some upstream
tests will attempt reconfiguring the system as root, or download random
unchecked stuff drom the internet, or communicate with an internal
server of the company that wrote the tets for example. There are many
many shades or gray out there

>  You can set them in modules and I think Koji can set them:
>  https://pagure.io/koji/issue/416 
It would be nice if it has been fixed now

>  Not mentioning that there is almost always way to provide some macro
>  file 
That’s the kind of manual workaround that looks nice on paper for
people who do not have to do it, and does not scale at all for people
who actually have to do it for thousands of packages while rushing to
meet release dealines

Regards,

-- 
Nicolas Mailhot

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Miro Hrončok]

On 09. 06. 20 12:21, Vít Ondruch wrote:
>  That won't be different for what was the original question here,
> i.e.
>  conditionally disable tests. bconds are what we have for better or worse.
>  
>  And really, this seems about bootstrapping not disabling tests, which
>  are not completely different, but nobody can objects bootstrapping,
>  while disabling tests might be good just to improve build speed and
>  nothing else. That should never happen in production environment IMO. 
FTR the discussion here is about packages that already have a bcond/macro to 
disable tests -- Tomáš proposed a common way of doing it. This discussion is not 
about adding new conditionals to packages that don't have them.

Whether or not disabling tests has legitimate use cases is out of scope here. It 
happens. We just want it to be more predictable when dealing with packaging in bulk.

As a metaphor (arguably not a very good one), imagine combustion motor vehicles. 
They pollute the environment. We are proposing to introduce colored emission 
stickers. You are discussing whether we should have such vehicles at all. While 
such discussion is certainly legitimate, it is out of scope. Sure, if we discard 
all gasoline- and diesel-powered cars and switch to electric or bicycles or 
perpetuum mobile, we don't have to put the energy into the emission stickers 
project. But how likely is that?


-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Vít Ondruch]

Dne 09. 06. 20 v 13:33 Miro Hrončok napsal(a):
>  On 09. 06. 20 12:21, Vít Ondruch wrote:
> > That won't be different for what was the original question here, i.e.
> > conditionally disable tests. bconds are what we have for better or
> > worse.
> >
> > And really, this seems about bootstrapping not disabling tests, which
> > are not completely different, but nobody can objects bootstrapping,
> > while disabling tests might be good just to improve build speed and
> > nothing else. That should never happen in production environment IMO.
>
>  FTR the discussion here is about packages that already have a
>  bcond/macro to disable tests -- Tomáš proposed a common way of doing
>  it. This discussion is not about adding new conditionals to packages
>  that don't have them.
>
>  Whether or not disabling tests has legitimate use cases is out of
>  scope here. It happens. We just want it to be more predictable when
>  dealing with packaging in bulk.
>
>  As a metaphor (arguably not a very good one), imagine combustion motor
>  vehicles. They pollute the environment. We are proposing to introduce
>  colored emission stickers. 

While we have already some other kind of stickers which could be reused.

The proposal was to optionally disable test. When somebody asked why,
the answer was bootstrapping. But we know how to handle bootstrapping.
So shouldn't somebody spend time changing the test conditionals to
bootstrapping conditionals, because that seems to be the use case?

Or if you have different use case, then you probably want to explain it.


Vít


>  You are discussing whether we should have such vehicles at all.
> While
>  such discussion is certainly legitimate, it is out of scope. Sure, if
>  we discard all gasoline- and diesel-powered cars and switch to
>  electric or bicycles or perpetuum mobile, we don't have to put the
>  energy into the emission stickers project. But how likely is that?
>
>

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Nicolas Mailhot]

Le mardi 09 juin 2020 à 14:35 +0200, Vít Ondruch a écrit :
>  
>  The proposal was to optionally disable test. When somebody asked why,
>  the answer was bootstrapping. But we know how to handle
>  bootstrapping.
>  So shouldn't somebody spend time changing the test conditionals to
>  bootstrapping conditionals, because that seems to be the use case? 
One use case is bootstrapping. Another is just getting things to build
till you have the time to investigate if a new test failure is an
actual problem or upstream being careless as usual. There are probably
other use cases out there

-- 
Nicolas Mailhot

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Nicolas Mailhot]

Le mardi 09 juin 2020 à 14:56 +0200, Nicolas Mailhot a écrit :
>  Le mardi 09 juin 2020 à 14:35 +0200, Vít Ondruch a écrit :
>  > 
>  > The proposal was to optionally disable test. When somebody asked
>  > why,
>  > the answer was bootstrapping. But we know how to handle
>  > bootstrapping.
>  > So shouldn't somebody spend time changing the test conditionals to
>  > bootstrapping conditionals, because that seems to be the use case?
>  
>  One use case is bootstrapping. Another is just getting things to
>  build
>  till you have the time to investigate if a new test failure is an
>  actual problem or upstream being careless as usual. There are
>  probably
>  other use cases out there 
Another fun case: someone broke the dep of a component used in unit
tests. Fixing the component requires rebuilding the dep. Except, the
dep uses the component itself in its own unit tests…

There are boundless possibilities for fun and profit there (well
profit, not so sure actually)

-- 
Nicolas Mailhot

[Fedora-packaging] Re: Let's standardize the way to disable tests during RPM build?

[Tomas Orsava]

On 6/9/20 3:01 PM, Nicolas Mailhot via devel wrote:
>  Le mardi 09 juin 2020 à 14:56 +0200, Nicolas Mailhot a écrit :
> > Le mardi 09 juin 2020 à 14:35 +0200, Vít Ondruch a écrit :
> >> The proposal was to optionally disable test. When somebody asked
> >> why,
> >> the answer was bootstrapping. But we know how to handle
> >> bootstrapping.
> >> So shouldn't somebody spend time changing the test conditionals to
> >> bootstrapping conditionals, because that seems to be the use case?
> > One use case is bootstrapping. Another is just getting things to
> > build
> > till you have the time to investigate if a new test failure is an
> > actual problem or upstream being careless as usual. There are
> > probably
> > other use cases out there
>  Another fun case: someone broke the dep of a component used in unit
>  tests. Fixing the component requires rebuilding the dep. Except, the
>  dep uses the component itself in its own unit tests…
>
>  There are boundless possibilities for fun and profit there (well
>  profit, not so sure actually)
>  
Another common one for me is rapid development in the spec file.

Overall, bootstrapping is definitely a common reason for disabling 
tests, but it's not the only one.
Using bootstrapping conditional for non-bootstrapping purposes would be 
even more confusing than the status quo.

Therefore people will want to create macros that disable tests. I think 
we should follow the example of the bootstrapping macro, and recommend 
one macro that disables tests.

Tomas