Fedora 21 Alpha validation test work

Adam Williamson adamwill at fedoraproject.org
Tue Jul 8 05:46:55 UTC 2014

Hi, folks.

So, we're scheduled for Alpha TC1 tomorrow. We had a nice happy
co-operative plan where QA and the WGs would collaborate on revising the
release validation test process for Fedora.next...

...which, well, didn't really happen. As of this morning we were nowhere
near having a viable validation process. So I went for plan B: I spent
today more or less pulling the entire thing out of my ass.

It's a bit rough around the edges, but I think we more or less have
something workable now. I have skipped the draft stage for a lot of
documents just in the interest of having something vaguely workable in
time for TC1; of course, the pages can be revised as much as necessary
as we work with them.

It's a bit hard to remember everything I've done, but we now have a
draft Alpha Release Criteria page which should cover all
release-blocking media, which for now I'm assuming includes the
Workstation live media, Server minimal and offline install media, Cloud
and ARM disk images, and possibly some kind of generic network install
image. That draft is at
and is based on the stuff from
https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria , https://fedoraproject.org/wiki/User:Adamwill/Draft_workstation_release_criteria and https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Release_Criteria , plus some adjustments to the templates that handle the preamble.

We have a new validation matrix, for Server product-specific tests:

The Desktop matrix has been adjusted to cover - not quite elegantly, but
at least cover - both the Workstation product and the KDE spin:

The Base matrix has been extended to add a couple of new test cases that
came out of the Product criteria drafting process, but actually aren't
really product specific, and has had its columns adjusted to be

The installation matrix has similarly had a couple of new criteria
wedged in, but much more importantly, I ripped the netinst, DVD and live
image 'sanity test' sections and replaced them with an ARM-style table
where a single 'generic' test case is run against several images on
several platforms - that's the "Default boot and install" section, so
please cast an eye over it:

and I made a small change to the release validation test event SOP to
list the server matrix as one to create:


Here are links to all (I think) of the new test cases I had to write as
I went along:


The following test cases already existed, but are newly included in the
release validation process (they were written for test days):


There are still quite a few i's to dot and t's to cross. There are some
release criteria and test cases that explicitly reference 'the DVD'
image that will need to be adjusted. We need to apply the 'associated
release criterion' template to all the new test cases, and probably
clean up some categorizations. Various other process documentation pages
may need to be updated, we'll have to check through all of them. But I
think now we at least have the broad strokes of what's needed for .next
validation testing. 

All feedback on the above changes is of course welcome! Please do cast
your eye over and point out anything I missed, anything that looks
silly, any possible improvements and so on. Remember, though, this is
really *test process* design: we're not actually doing product design
here, if you think there are issues with the Fedora.next changes
themselves, that goes to the WGs or FESCo. As far as this work is
concerned, we're just trying to test the new Products as they're
designed (all of the above is based off of the Product PRDs and tech

I'll take time tomorrow to do some polishing, and of course look at any
and all feedback on the stuff I did today.
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net

More information about the server mailing list