New Anaconda UI initial testing

[Adam Williamson]

Hey, folks. So, I've been keeping in touch with Chris about testing the
new anaconda UI, and we think it's at the point now where QA can
probably contribute some useful testing. However, it still requires
handling with kid gloves. So I've tried to set something up that'll work
out.

0. DO NOT DO ANY KIND OF TESTING WITHIN TEN MILES OF ANY DATA YOU CARE
ABOUT
1. SEE 0
2. SEE 1
3. SEE 2

We're really serious about those three points. The newUI is very very
early code. It doesn't just eat babies, it laughs while it does it. If
you want to participate in this testing, do it on a disposable machine
or in a VM. Do _NOT_ do it anywhere near any kind of production system.
Related:

4. Do NOT send or link these images to anyone who doesn't really know
what they're doing, and only spread them to people who want to help with
this testing. We want to keep this testing relatively quiet, not for any
sinister reasons but just because the anaconda team doesn't consider the
code yet in any kind of shape for wide distribution. This isn't like a
late RC or something that you can link your friends to for a quick sneak
peek of Fedora 18. It's very very raw code that should only be used for
testing. Even if you happen to get a successful install, if all you want
is a successful install of Rawhide, you can get it a lot more safely by
running the regular installer or (even better) installing F17 and yum
updating. So don't go posting these test images on forums and stuff. We
don't want angry people complaining about their babies that got eaten.
They're _just for testing the installer_. They don't contain anything
magic or special besides the new anaconda UI. If you are some kind of
news organization reading this, _please_ don't grab the test images and
write a story about how horrendously buggy the new UI is. We know it is.
It's not done yet. This is just part of the process of building it.

5. Do NOT file bugs for any failures you hit in this testing. The
anaconda team doesn't want Bugzilla stuffed up with transient errors in
the new UI code. There's instructions further down on how to report
problems.

6. Alrighty, we're done with the warnings! Here's the good stuff. Go
check out
https://fedoraproject.org/wiki/Test_Results:Fedora_18_newUI_Install .
It's a highly modified installation validation matrix which I've tried
to design in a way that'll support testing of the new UI snapshots. I've
removed a lot of tests that aren't relevant to the new UI testing or
that cover functionality not yet working in the new UI (mainly custom
partitioning and upgrades). The page also contains links to the
snapshots for testing (so far there's only one). I've set up the results
somewhat differently - we can ignore EFI vs. BIOS and 32-bit vs. 64-bit
for UI testing, and that means each snapshot needs only a single results
column. That means we can use one page for multiple snapshots, which
will make it nice and easy to compare results across snapshots and saves
us making so many damn pages. Right now there's just one results column
for the single snapshot that's available; I'll add new columns to the
page as new snapshots come (tentatively, we're planning to do them
weekly).

7. Do some tests, and fill in your results - just as for regular
validation testing, except remember as mentioned above, DON'T file any
bugs. If you hit a failure, mark it as 'fail', with a reference. Please
keep the reference _brief_. If you can't describe the failure in a
sentence or two, make the reference a summary, and mail the list to
describe the problem you found. We might change this process in future,
it's just what Chris and I came up with so far. The most important thing
is to mark the test as fail and include your name, then we know who to
ask for more details.

8. Of course, we can expect there will be quite a few:
   a) Failures not related to the new UI code
   b) new UI problems not covered by test cases

In the case of a), please don't report them as fails on the matrix, or
discuss on the list. If you can, please confirm that they're also broken
in regular Rawhide, and if so, just go ahead and file a bug as normal.
The point of this testing is to check out the new UI code; things that
aren't related to that should just be handled through the regular
Bugzilla process.

In the case of b), please *do* report them on the wiki page somehow, or
to the list. But FIRST, check
https://fedoraproject.org/wiki/Anaconda/TestableUI , which explains
which things in the new UI are known to be working (and by implication,
which are known _not_ to be working). If your problem comes in an area
not listed as expected to be working, you can probably ignore it; if you
want to be sure, check in #anaconda on IRC (ideally with clumens, but
anyone should be able to help). If you find a problem in an area that's
expected to be working but isn't covered directly by a test case, that's
just great: please do report it to the list and/or the wiki page (or
directly to #anaconda IRC, if someone's around to take the report).

Chris, if you have any amendments to the above or the list of tests on
the Wiki page, please let us know! Thanks. If anyone's unclear about
this whole thing, which I pretty much just threw together at 10pm after
a couple of beers, do ping me, here or on IRC.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net