Draft Fedora 21 Test Plan

[Adam Williamson]

On Tue, 2014-05-27 at 07:58 -0400, Matthew Miller wrote:
> On Fri, May 23, 2014 at 06:02:07PM -0700, Adam Williamson wrote:
> > Hi, folks! So, I quickly bashed out that draft F21 Test Plan I've been
> > threatening to write for the last month or so.
> > https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan
> > So, what's the idea here?
> 
> Psshh -- so much for your claim of not doing any real work. This looks
> great! I'll put discussing it on our next Cloud WG meeting.

Awesome, thanks.

> > * There are several practical implications from the test plan - just
> > Work We Need To Do. Most obviously, we need to draw up release criteria
> > and supporting test cases for the Fedora.next Products. We also will
> > need to adjust the
> 
> Mike has some drafted up for cloud at
> <https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Test_Plan#Test_Pass.2FFail_Criteria>.
> It'll need a bit more by way of specifics.

hah, I missed those somehow. That's a good start. My mental model was
actually going to be that we'd keep the 'generic' release criteria as
something shared between all Products and then just have additive
Product-specific release criteria pages (they could also amend the base
criteria slightly if necessary, as long as the delta didn't get too
crazy), but the approach of having multiple 'complete' sets of release
criteria may work out better, we'll have to think it through. I'm
planning to kick off some threads about this work this week.

> > * Responsibilities! Particularly, in this *draft* Test Plan, I've
> > suggested that creating the Product-specific criteria and test cases
> > should be the responsibility of the relevant Working Groups, with
> > assistance from the QA team. They would also be jointly responsible,
> 
> This seems like the right way to me, too.

Cool, glad everyone seems to be agreeing on that.

> > for Fedora 21 testing *overall* - i.e. it's the QA team's responsibility
> > to make sure the WGs do the stuff assigned to them in the plan. If that
> > makes sense. Discussion welcome!
> 
> Yeah. If something isn't working, do you think the best path is QA->WG
> directly, or should it be QA says something to FESCo, FESCo works with WG?

As with all the others who've weighed in so far, I agree that the
optimal path is just for us all to talk to each other. I generally
prefer workflows that assume everyone's happy to work positively
together to a consensus solution over ones that assume everything will
be oppositional and must therefore go through a higher authority. So,
yep, IMO we should just talk to each other, and FESCo should only be
invoked to resolve irreconcilable disputes (of which we hope there won't
be any).

> Since parts of this are new to all of us, and other parts new to a lot of
> us, maybe it'd be good to put suggested/reasonable deadlines on some parts
> of the responsibilities? That helps put in focus what needs to be done.

Indeed, I didn't put it in the plan but it would obviously be helpful.

We should really aim to have the process nailed down and at least the
Alpha criteria and test cases (and any required automated testing stuff)
ready by the date of Alpha TC1. Ideally we'd have Beta and Final
criteria and test cases prepared at that point too, but we _could_ delay
those until those actual milestones if we have to.

The actual release process is heavily tied into the release criteria,
validation testing, and blocker bug stuff; if we don't have that in
place it's very difficult to work through a milestone release in any
meaningful way which isn't just "let's chuck something over the wall and
hope for the best".

Ideally I'd like to work fairly rapidly and get, say, the initial
release criteria ready by the end of next week, and then get the
required test cases and matrices written between then and branching
(which is 'no earlier than July 8' on current schedule).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net