Packages with missing %check

[Josh Boyer]

On Thu, Feb 27, 2014 at 4:49 AM, Richard W.M. Jones <rjones at redhat.com> wrote:
> On Thu, Feb 27, 2014 at 12:12:59AM -0800, Adam Williamson wrote:
>> On Thu, 2014-02-27 at 08:09 +0000, Richard W.M. Jones wrote:
>> > On Wed, Feb 26, 2014 at 06:50:56PM -0800, Adam Williamson wrote:
>> > > On Wed, 2014-02-26 at 21:53 +0000, Richard W.M. Jones wrote:
>> > > >   It happens too often in
>> > > > Rawhide, and a simple test (in %check or elsewhere) could fix it.
>> > >
>> > > You can't really test a system boot in a package's %check. That's very
>> > > definitely not what it's for.
>> >
>> > It turns out you can.  Read the link I posted:
>> >
>> > http://people.redhat.com/~rjones/qemu-sanity-check/
>>
>> I've read it. I'm not exactly convinced.
>
> If there is a specific problem, please mention what it is.  Otherwise
> there is no action I can take.
>
>> > Yes, it doesn't catch every possible problem, but the kernel is
>> > sometimes pushed out with major problems that could be caught with
>> > simple testing.
>>
>> IIRC you had one example for this, and the bug turned out to be racy and
>> hardware-dependent, which is why the kernel devs didn't see it in the
>> tests they *do* actually do.
>
> *One*?  There's loads of them.  Regularly the kernel doesn't boot on
> qemu, most recently just two days ago (although that "fixed itself"
> before I managed to get around to filing a bug).
>
> JUST the bugs in the kernel & qemu filed or fixed in 2013/2014:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=872831
> https://bugzilla.redhat.com/show_bug.cgi?id=1033971
> https://bugzilla.redhat.com/show_bug.cgi?id=1034433
> https://bugzilla.redhat.com/show_bug.cgi?id=998722
> https://bugzilla.redhat.com/show_bug.cgi?id=991808
> https://bugzilla.redhat.com/show_bug.cgi?id=857026
> https://bugzilla.redhat.com/show_bug.cgi?id=962079
> https://bugzilla.redhat.com/show_bug.cgi?id=863978
>
> non-x86:
> https://bugzilla.redhat.com/show_bug.cgi?id=1066581
> https://bugzilla.redhat.com/show_bug.cgi?id=1036742
>
> non-Fedora:
> https://bugzilla.redhat.com/show_bug.cgi?id=998065
>
> In one sense none of this is surprising at all.  The kernel
> (especially) and qemu are pushed out with next to no testing.  They
> are huge complex pieces of software.  The fact they work some of the
> time is a miracle really.

The kernels that are built are tested at a minimum on 3 machines
before I even put them to git.  After they are built in koji we have
an autotest framework that pulls down the build and runs a set of
tests on them in multiple KVM guests.

I appreciate that you do find kernel bugs and you're very responsible
in reporting and pushing upstream where appropriate.  Thank you for
that.  However, please don't make assumptions and disparage the kernel
maintainers because you found a few bugs using QEMU.

Also, if you'd like to point out the testing aspect and some of the
deficiencies we have around how rawhide works, please make sure you're
speaking about rawhide.  The release update kernels are much more
thoroughly tested because we have a framework that prevents builds
from landing in a repo there.  If we had a manageable way to do that
in rawhide, I'd be happy to use it.

As for running the sanity check in %check, it would bloat the
BuildRequires for the kernel significantly and would increase already
long build times.

josh