Draft desktop validation test matrix

[Adam Williamson]

On Tue, 2010-01-12 at 07:48 -0500, James Laska wrote:
> On Tue, 2010-01-12 at 09:53 +0000, Adam Williamson wrote:
> > On Tue, 2010-01-12 at 17:10 +0800, He Rui wrote:
> > > Hi Adam,
> > > 
> > > Nice matrix!
> > > 
> > > I have a question. Is release level 'Alpha', 'Beta', 'Final' the same as
> > > priority 'tier1', 'tier2', 'tier3'? For example, if Beta Candidate is
> > > under testing, will the cases of Alpha release level be tested again? 
> > > 
> > > If all the cases should be tested on Final candidate, and many people
> > > attend the tests, then both the installation and this desktop matrix
> > > would be long. Then I prefer a separate page. 
> > 
> > Yes, all the tests will be performed at all stages.
> > 
> > Note that there are some other criteria we should probably have tests
> > for, which don't exactly fall into the category of installation or
> > desktop (e.g. 'all services should start successfully in a default
> > install'). I'm not sure where to put those. 
> 
> This topic came up yesterday in some discussion with John Poelstra and
> Jesse Keating.  The idea was something around an [alpha|beta|final]
> release checklist.  It might be a slightly different focus from what
> you're talking about, but I think we could adapt it to fit.  Things it
> might include ...
> 
>       * All services start successfully in a default install (as you
>         mentioned)
>       * No SELinux AVC denials on initial boot or subsequent login in a
>         default install
>       * No ABRT crash notifications on initial boot or subsequent login
>         in a default install

These two I've already written up as a 'desktop' test case and included
in the matrix.

>       * The BetaNag has been removed from the installer
>       * Fedora-release package contains final version
>       * Kernel no longer compiled with debugging enabled
>       * All packages signed with the appropriate key?
>       * Anything about release notes?
>       * What else?
> 
> Thoughts?

That definitely sounds like something we should do to avoid the
always-possible releng brown paper bag situations, but doesn't quite
address what I'm worrying about for the QA validation tests :)

> 
> > If anyone has some awesomely
> > clever idea for combining all validation testing in a convenient way
> > which isn't one gigantically long page...we're all ears :)
> 
> We can play games with a read-only master page which transcludes all
> other [[Test_Result]] content for easier test and result management.  

That sounds like a sensible approach - let's work on it after I get back
home.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net