[Fedora QA] #227: Document SOP for acceptance test event

[Fedora QA]

#227: Document SOP for acceptance test event
--------------------+-------------------------------------------------------
  Reporter:  rhe    |       Owner:  wutao85  
      Type:  task   |      Status:  new      
  Priority:  major  |   Milestone:  Fedora 16
 Component:  Wiki   |     Version:           
Resolution:         |    Keywords:           
--------------------+-------------------------------------------------------
Comment (by jlaska):

 Replying to [comment:10 adamwill]:
 > I'm not sure just linking to the acceptance test plan is sufficient
 instruction on how to do the testing. It's a planning document, and it has
 "TODO: Automate these test cases" written in it. :) Just reading that
 page, I have no idea of the actual steps one has to perform to complete
 the testing: run this program, click this button, etc. You could do the
 testing *manually* by following each test case linked there, but that's
 not really the point.

 Just a note ... this isn't fully automated.  I'd classify the automation
 as a mature proof-of-concept.  In addition, I've rarely been satisfied
 with the automated results alone.  The automation helps me figure out how
 far up I need to roll up my sleeves for testing.  In my experience, this
 milestone calls for more hands on attention until the automation has
 proven itself.

 What I've discussed with twu is that this milestone needs to be revisited
 and possibly extended to cover more test blocking issues ... not just it's
 original mission of 'rawhide acceptance'.  The concept of rawhide has
 changed since this test plan was born.  While the need still exists, this
 plan doesn't go far enough to find the bugs that will block testing of the
 'test compose'.  What I'd like to see out of this test run is
 identification of bugs for items identified in the current acceptance
 plan, but also bugs that block significant portions of the release
 validation test matrices.  The one week between TC1 and RC1 is *not*
 sufficient time to identify, resolve, verify and refix those bugs.  I'm
 convinced this has contributed to slips in all of the previous ALpha slips
 for ~4 releases (if memory serves).

 What I'd recommend is to focus on defining the goal and documenting the
 process for this effort ... then we can follow-up with focused patching to
 the rats_* test automation.

 > What we really need are instructions by which someone who'd never done
 the acceptance testing before could successfully do the acceptance testing
 - automated, not manually. Thanks!

 Pet peeve of mine ... I recommend ensuring the tests themselves are
 documented first.  From there, we can fill-in the gaps with any automation
 that exists.  The problem I have with the automation is that it's
 insufficient for what this activity calls for.  It's a good start, it
 needs more work, and in it's current state, it can be a distraction from
 the goal of finding bugs that will prevent further release validation
 testing.  Let's definitely improve the automation, which twu is looking
 into, but track that separately.

 > (If it requires access to some Red Hat-only system, or something, then
 a) that sucks, but b) we should still write it down.)

 It doesn't.

-- 
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/227#comment:11>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance