Re: third-party tests *now* ?

[Tim Flink]

On Tue, 16 Aug 2011 08:32:28 -0400 (EDT)
Kamil Paral <kparal(a)redhat.com&gt; wrote:

>  There is this discussion in test list [1] where Steve Grubb posted
>  some security testing scripts. He asked these scripts to be included
>  in AutoQA and executed regularly. I would like to discuss it with the
>  whole team because I believe we don't have a clear idea what our
>  current stance is.
>  
>  I'd like to propose that we currently don't accept any third-party
>  tests. It is tempting and I'd love to have more tests inside AutoQA.
>  But we don't have the manpower to maintain them. Every single extra
>  test case drains power from implementing the AutoQA infrastructure.
>  We already have some test cases we (I) don't understand. And it would
>  get worse. 
I definitely see where you're coming from here and I agree that we're
not ready to start taking in third party tests.

What do we need to do in order to be ready to take third-party tests?
Where does this fit in the priority list?

The first things that come to my mind for stuff we would need are:
 - Separate the tests from the framework for packaging and
   distribution purposes

 - Define and implement a common external API
   * For test execution
   * For reporting results

 - Support disposable or at least separate test clients
   * having unknown scripts run on the same clients that we're using
     for our existing tests seems like a bad idea to me.

>  I believe we need to implement proper AutoQA infrastructure and then
>  allow people to run and maintain their tests, without our
>  babysitting. Different approach will consume our resources. 
I think that there might be more of a middle ground at first but
overall, I agree that there aren't enough of us to support an
ever-growing battery of tests.

>  What do you think? 
I'm thinking a compromise for now. AutoQA isn't ready for third party
tests but these sound worth running. I think that working with the QA
community to start running them manually for now would be a good step
forward.