Fedora 16 QA Retrospective: recommendations filed, tickets up for grabs!

[Adam Williamson]

Hi, folks. I wanted to let everyone know that I've gone through all the
issues raised in the Fedora 16 QA Retrospective:

https://fedoraproject.org/wiki/Fedora_16_QA_Retrospective

and taken action on each item where possible.

For most items I've added a recommendation to the Recommendations
section:

https://fedoraproject.org/wiki/Fedora_16_QA_Retrospective#Recommendations

so that section now lists the trac tickets (in most cases) and mailing
list threads relevant to each item. I've also brought forward some
still-outstanding action items from the F15 retrospective, listed under
'Brought forward from Fedora_15_QA_Retrospective' sub-sections within
the Recommendations.

Here's a list of the specific tasks that are now outstanding in QA trac
as a direct result of the Retrospective: it would be great if people
could volunteer to pick these up. Please, if you'd like to help out,
just assign one of the tickets to yourself and go do it! There are no
hoops to jump through. I'll try and add links in each ticket to the
relevant processes for creating test cases and so on.

* http://fedorahosted.org/fedora-qa/ticket/256
  "Add EFI data to installation / base results matrices"

* http://fedorahosted.org/fedora-qa/ticket/257
  "Create and maintain F17 install test plan"

* http://fedorahosted.org/fedora-qa/ticket/258
  "Create EC2 test cases and integrate into validation matrices"

* http://fedorahosted.org/fedora-qa/ticket/259
  "Review Fedora 17 feature list and co-ordinate with owners of
significant features to create test plans"

* http://fedorahosted.org/fedora-qa/ticket/260
  "Cover USB-written images better in installation validation testing"

* http://fedorahosted.org/fedora-qa/ticket/261
  "Revise upgrade test case set"

There were a couple of items where the action didn't really fit
comfortably into the 'Recommendations' section, so for the record, I'll
list those here:

"adamw - when there are showstoppers in anaconda early we tend to just
sit around and wait for them to get fixed, but without much urgency;
which means we're getting no idea of what bugs are lurking behind the
showstoppers, and we wind up trying to fix a lot of blockers in a small
amount of time once the showstoppers are finally fixed"

"adamw - when we hit an obvious showstopper we tend to focus in on it
exclusively until it's fixed, iterate, and then be surprised when there
are bugs hidden behind it; we should work harder to try and workaround
showstoppers in a way that has as small an impact as possible, and
continue testing, to avoid this problem. e.g. RHBZ #730863 hid behind
RHBZ #729563, but we could have exposed it by
editing /etc/selinux/config in the installed system prior to rebooting
from the installer"

"adamw - need to do more direct personal pinging of maintainers who seem
unresponsive to blockers; some respond when contacted directly but do
not appear to place a high priority on bug reports"

There isn't really any specific action to be taken on these right now,
they're more things to keep in mind while validating F17.

items 5-8 are just notes of what bugs blocked Alpha RC spins, no
specific action was really needed in relation to any of those

"adamw - it wasn't an A+ idea for both rpm maintainers to be on vacation
at the same time, with no cover in place, during a critical freeze
period"

The action I took here was fairly RH-internal, as this is really an RH
staffing issue: I poked Tom Callaway to ask about putting a cover policy
in place for RH-employed maintainers of key Fedora components

"adamw - NFS testing instructions may need to be more precise to avoid
pilot errors, and may need to account for NFSv4 quirks"

This one's pretty much already taken care of, we did tweak the NFS test
cases somewhat during the f16 cycle

"tflink - Maybe it was due to the high number of TC/RC spins that we did
for F16 but it would be awesome to have some more automation around
image generation for testing. I'm not talking about any rel-eng
processes but it would be nice to be able to request generation of a
boot.iso made from custom packages and packages in koji in order to do
testing. I can see this being helpful for test day coordination as well
but there are issues with complexity and hosting so this is a bit of a
"pony list" item."

"tflink - Since qemu/kvm VMs are all legacy BIOS emulation, it would be
awesome to do EFI emulation. There are some projects out there on using
EFI in qemu VMs but it is unknown whether they actually work. Either
way, it might be good to look into them in order to evaluate whether
their use is practical"

These are sort of 'blue sky' ideas from tflink that don't lead to any
immediate action items that anyone could take care of: I'm relying on
him to keep thinking about them and turn them into more concrete
proposals. I don't really want to file open-ended tickets for rough
ideas.