Regression testing

[Justin Forbes]

As some of you have seen in our community meeting and minutes, we are
working on actively testing the kernel builds.  The short version is a
series of regression tests which reside in either the kernel git tree
and can be run with 'make tests' or possibly a control script in a
kernel-tests subpackage.  This will allow any user to run the regression
tests against their running kernel on real hardware, and hopefully
improve the test base considerably.  In addition we will have autotest
hooked up to do this plus more extensive testing.  More details can be
found at https://fedoraproject.org/wiki/KernelTestingInitiative

So here is the part where we really want to get community involved.  We
have a wiki page created to list the regression tests that need to be
written, and even sign up to write some if you are so inclined.  We need
your ideas for regression tests.  Tests should be simple pass/fail
checks.  Tests against specific modules are allowed and encouraged.  For
users testing where they module is not loaded, the test will simply be
skipped.  Destructive testing is possible, though it will not be
included in the standard test run (a user should know and explicitly
sign up for a risk of data loss).  Please take a look and add your
ideas: https://fedoraproject.org/wiki/KernelRegressionTests

Thanks,
Justin

[CAI Qian]

Hello,

----- Original Message -----
> From: "Justin M. Forbes" 
> To: [e-mail]
> Sent: Friday, April 20, 2012 11:05:22 PM
> Subject: Regression testing
>
> As some of you have seen in our community meeting and minutes, we are
> working on actively testing the kernel builds.  The short version is
> a
> series of regression tests which reside in either the kernel git tree
> and can be run with 'make tests' or possibly a control script in a
> kernel-tests subpackage.  This will allow any user to run the
> regression
> tests against their running kernel on real hardware, and hopefully
> improve the test base considerably.  In addition we will have
> autotest
> hooked up to do this plus more extensive testing.  More details can
> be
> found at https://fedoraproject.org/wiki/KernelTestingInitiative
>
> So here is the part where we really want to get community involved.
>  We
> have a wiki page created to list the regression tests that need to be
> written, and even sign up to write some if you are so inclined.  We
> need
> your ideas for regression tests.  Tests should be simple pass/fail
> checks.  Tests against specific modules are allowed and encouraged.
>  For
> users testing where they module is not loaded, the test will simply
> be
> skipped.  Destructive testing is possible, though it will not be
> included in the standard test run (a user should know and explicitly
> sign up for a risk of data loss).  Please take a look and add your
> ideas: https://fedoraproject.org/wiki/KernelRegressionTests

I am actually interesting in this efforts since my group is doing kernel
testing for a similar enterprise product. The natural thing to think is
to adopt tests that we used there for Fedora where possible. Does that
align with the community expectation here?

Regards,
CAI Qian
> Thanks,
> Justin
>
> kernel mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/kernel

[Josh Boyer]

On Thu, May 03, 2012 at 03:22:55AM -0400, CAI Qian wrote:
> > your ideas for regression tests.  Tests should be simple pass/fail
> > checks.  Tests against specific modules are allowed and encouraged.
> >  For
> > users testing where they module is not loaded, the test will simply
> > be
> > skipped.  Destructive testing is possible, though it will not be
> > included in the standard test run (a user should know and explicitly
> > sign up for a risk of data loss).  Please take a look and add your
> > ideas: https://fedoraproject.org/wiki/KernelRegressionTests
>
> I am actually interesting in this efforts since my group is doing kernel
> testing for a similar enterprise product. The natural thing to think is
> to adopt tests that we used there for Fedora where possible. Does that
> align with the community expectation here?

If you have tests that can be split out into easily run, stand-alone
test cases, then yes!  As you can see from the wiki page, we're somewhat
lacking for testcase ideas at the moment.

josh

[CAI Qian]

----- Original Message -----
> From: "Josh Boyer" 
> To: "CAI Qian" 
> Cc: [e-mail], "Justin M. Forbes" 
> Sent: Thursday, May 3, 2012 8:55:33 PM
> Subject: Re: Regression testing
>
> On Thu, May 03, 2012 at 03:22:55AM -0400, CAI Qian wrote:
> > > your ideas for regression tests.  Tests should be simple
> > > pass/fail
> > > checks.  Tests against specific modules are allowed and
> > > encouraged.
> > >  For
> > > users testing where they module is not loaded, the test will
> > > simply
> > > be
> > > skipped.  Destructive testing is possible, though it will not be
> > > included in the standard test run (a user should know and
> > > explicitly
> > > sign up for a risk of data loss).  Please take a look and add
> > > your
> > > ideas: https://fedoraproject.org/wiki/KernelRegressionTests
> >
> > I am actually interesting in this efforts since my group is doing
> > kernel
> > testing for a similar enterprise product. The natural thing to
> > think is
> > to adopt tests that we used there for Fedora where possible. Does
> > that
> > align with the community expectation here?
>
> If you have tests that can be split out into easily run, stand-alone
> test cases, then yes!  As you can see from the wiki page, we're
> somewhat
> lacking for testcase ideas at the moment.

OK, will kick of some ideas there. Do you have any guideline on how to
implement those tests? Will it be written using AutoQA or Beaker? Where
to submit and commit those tests? I am also CC'ing Adam here to see what's
his opinion.

Regards,
CAI Qian
> josh
> _______________________________________________
> kernel mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/kernel