RFC: Multipath testing as a (optional?) criteria for RC?

[Phil Knirsch]

On 05/02/2013 01:10 AM, Adam Williamson wrote:
> On Tue, 2013-04-30 at 17:01 +0200, Phil Knirsch wrote:
>> Hi guys.
>>
>> I'd like to open up the discussion about getting multipath testing as a
>> (optional) criteria for RC.
>>
>> I understand that the typical Fedora user most likely will never
>> encounter or use multipath, but as we do have server users for Fedora
>> there might be some merit to it.
>>
>> We've ran into this on the ppc64 side the last few releases where we did
>> test this during our alpha, beta and rc checks and worked with the
>> respective maintainers to fix the issues we found.
>>
>> So if there is an interest to make this a general test case we'd
>> certainly be able to provide a QE template with all the necessary steps.
>>
>> And as mentioned, even if it's not going to be a blocking criteria maybe
>> having it as included as an optional test case might still be worth it.
>>
>> Let me know what you guys think.
>
> So a couple of things: there's the test cases and the criteria. The test
> cases enforce the criteria, the criteria back the test cases.

Understood. :)

> So there's no such thing as 'optional criteria' - the criteria are
> always mandatory =) We do have 'optional test cases', though: these are
> test cases that don't enforce any particular criterion and hence don't
> have to pass, but that it seems like a good idea to test. Those are the
> things listed as 'Optional' at
> https://fedoraproject.org/wiki/QA:Fedora_19_Install_Results_Template .

Excellent, thats what i was looking for, thanks!

> I don't think there'd be any objections at all to adding an optional
> test case for multipath, certainly. So that would be the first thing to
> do, and if you want to go ahead and contribute one, that would be
> awesome. Really, all you need to do is go ahead and write the test case,
> edit it into
> https://fedoraproject.org/wiki/QA:Fedora_19_Install_Results_Template in
> the appropriate spot (probably with the other 'optional' tests, at the
> bottom), and let the list know you've done it - we can review it.
>

Cool, great. Once i have all the info to define the test template i'll 
work with Tommy on creating that.

> Adding a release criterion is a more significant action and that
> requires discussion / consensus for sure. I'm not sure if we want to
> block a Fedora release on multipath functionality, but I don't really
> have a lot of data for that, it's just a feeling. Does anyone have good
> arguments for/against covering multipath in the criteria? Johann has a
> good point about hardware availability; it should usually be the case
> that the QA folks who work for Red Hat should be able to get access to
> multipath hardware for testing one way or another, but that may not be
> enough for us to feel comfortable.
>
> If we did decide to cover multipath functionality in the criteria, all
> we'd have to do to the test case is change it from 'Optional' to 'Final'
> in the template at that point. No problem at all. So definitely the
> first step is just to write the test case and add that in.

100% agree. I didn't want to come across as "we must have this as a 
criteria for releases!!11!!" as i clearly understand that the 
availability of required hardware in Fedora is rather limited and that 
the typical use case for Fedora users is most likely not going to 
include multipath.

So we'll be working on defining the test case, add it as an optional 
test template. And as you said, the decision whether this should be a 
blocking criteria for releases is a separate from that and a first step 
in the test case anyway.

Thanks!

Regards, Phil

-- 
Philipp Knirsch              | Tel.:  +49-711-96437-470
Manager Core Services        | Fax.:  +49-711-96437-111
Red Hat GmbH                 | Email: Phil Knirsch <pknirsch at redhat.com>
Wankelstrasse 5              | Web:   http://www.redhat.com/
D-70563 Stuttgart, Germany