May I file 1000 bugs aka upstream test suite tracking

[Josh Boyer]

On Fri, Feb 21, 2014 at 9:50 AM, Daniel P. Berrange <berrange at redhat.com> wrote:
> On Fri, Feb 21, 2014 at 09:38:56AM -0500, Josh Boyer wrote:
>> > That being said, a lot of packages in Fedora are simply that: packaged
>> > upstreams. Many (most?) package maintainers are not developers of that
>> > package and as such are probably not equipped to add tests to their
>> > systems.
>> >
>> > A better case here would be to find a way to identify those packages
>> > whose upstreams have tests that are not being run in %check. That
>> > probably *would* be considered a bug.
>>
>> Unless there's a decree from FESCo or FPC about requiring this, it's
>> going to be up to the maintainer as to whether using %check to run
>> testsuites is required.  A lot of testsuites require external network
>> access and won't work when run under koji.  Also, it increases build
>> time and can bloat BuildRequires.  It is *not* a clear-cut bug.
>>
>> Personally, I don't think %check is a good idea at all.
>
> I think the benefit depends on the level of patching the Fedora maintainer
> is doing. If they are shipping just vanilla upstream tar.gz then they can
> have a moderate level of confidence in the functionality of their package
> without tests, since you can assume upstream ran their test before release.
> Running the test suite would however confirm that the package has not
> been broken due to a change in an new version of an external library/tool
> it depends on, so is a pretty good idea to enable on balance.
>
> If the maintainer is including any non-trivial patches that I think that
> enabling %check should almost be mandatory to ensure they are not causing
> regressions through their patches.

I don't disagree that testsuites should be run.  I just disagree they
should be run in %check.  It's a poor replacement for things that
should be run via AutoQA or some other test framework after the build.

josh