May I file 1000 bugs aka upstream test suite tracking

[Aleksandar Kurtakov]

----- Original Message -----
> From: "Josh Boyer" <jwboyer at fedoraproject.org>
> To: "Daniel P. Berrange" <berrange at redhat.com>, "Development discussions related to Fedora"
> <devel at lists.fedoraproject.org>
> Sent: Friday, February 21, 2014 4:53:03 PM
> Subject: Re: May I file 1000 bugs aka upstream test suite tracking
> 
> On Fri, Feb 21, 2014 at 9:50 AM, Daniel P. Berrange <berrange at redhat.com>
> wrote:
> > On Fri, Feb 21, 2014 at 09:38:56AM -0500, Josh Boyer wrote:
> >> > That being said, a lot of packages in Fedora are simply that: packaged
> >> > upstreams. Many (most?) package maintainers are not developers of that
> >> > package and as such are probably not equipped to add tests to their
> >> > systems.
> >> >
> >> > A better case here would be to find a way to identify those packages
> >> > whose upstreams have tests that are not being run in %check. That
> >> > probably *would* be considered a bug.
> >>
> >> Unless there's a decree from FESCo or FPC about requiring this, it's
> >> going to be up to the maintainer as to whether using %check to run
> >> testsuites is required.  A lot of testsuites require external network
> >> access and won't work when run under koji.  Also, it increases build
> >> time and can bloat BuildRequires.  It is *not* a clear-cut bug.
> >>
> >> Personally, I don't think %check is a good idea at all.
> >
> > I think the benefit depends on the level of patching the Fedora maintainer
> > is doing. If they are shipping just vanilla upstream tar.gz then they can
> > have a moderate level of confidence in the functionality of their package
> > without tests, since you can assume upstream ran their test before release.
> > Running the test suite would however confirm that the package has not
> > been broken due to a change in an new version of an external library/tool
> > it depends on, so is a pretty good idea to enable on balance.
> >
> > If the maintainer is including any non-trivial patches that I think that
> > enabling %check should almost be mandatory to ensure they are not causing
> > regressions through their patches.
> 
> I don't disagree that testsuites should be run.  I just disagree they
> should be run in %check.  It's a poor replacement for things that
> should be run via AutoQA or some other test framework after the build.

In java land tests are usually run as part of the %build and if they not succeed the build fails. Running tests in %check will be harder to do as one would have to patch them to not run in %build so buildtimes do not grow too much. I also think that purely looking for tests in %check would not give correct data and packages that run tests as part of the regular build procedure should be considered (it's better practice IMHO).
Regarding tests being run after the build - this is yet another story as it requires packagers to work on having tests available in some form (subpackage?) for running after the build and many test suites are not intended to be run post build so making them run that way is non trivial amount of work. We (eclipse) work on having tests subpackages available for later run but this is slow and tedious process as it requires actually fixing upstream test suites.

Alexander Kurtakov
Red Hat Eclipse team

> josh
> --
> devel mailing list
> devel at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct