Package Update Acceptance Test Plan - final call

[Kamil Paral]

----- "Jesse Keating" <jkeating at redhat.com> wrote:

> On Wed, 2010-04-21 at 10:13 -0400, Kamil Paral wrote:
> > Hello,
> > this is a final call for comments to our Package Update 
> > Acceptance Test Plan [1]. This test plan is tightly related
> > to Package Update Acceptance Criteria [2] approved by FESCo.
> > It basically says how we, the QA team, will decide whether
> > a package update should be accepted or rejected. It also
> > tries to map the requirements of the policy into an actionable
> > test plan.
> > 
> > If you have any comments about the test plan, please post
> > them now, so we can include them in the document.
> > 
> > 
> 
> I'm a bit concerned about the test environment part.  We have already
> determined in the work that WWoods is doing for depcheck, that we do
> need to consider all the pending updates when testing a particular
> pending update.  It would make sense to me to follow the same logic
> here.

The Test Environment section tries to say that we are mainly interested
in testing changes that would happen for users with fully updated 
system, because that is the most common case and we can't simply test
all the possibilities. Good example is Package Sanity Test, where
we want to test upgrading from the most recent package in 'updates'
to the package coming to 'updates-testing'. We are not really 
interested in upgrading from one package in 'updates-testing' to
another package in 'updates-testing', because that's not what's gonna
happen for most of our users.

Of course some tests have a little different scope. Repo sanity
(the depcheck) tests repository contents rather than packages
on a system. That means that the third bullet point not really
applies to it.

So it really depends on a particular test case, but I think generally
the idea of Test Environment should be valid as it is.

> 
> As far as Responsibilities and Permissions, as others have said we
> could
> collapse the QA and Releng into the proventesters group.  Also we
> should
> add the bodhi ticket author, as the person who owns the package, the
> person who did the build, and the person who submitted the update can
> all be different people.

Thanks, added.

> 
> As far as Schedule we should make it clear at what phase the test
> runs.
> I think we want it to run post-update-submission, and must be
> completed
> before the request to push to either -testing or stable is
> "accepted".
> We talked with Luke a bit about this and how to handle it within
> koji/bodhi and have a game plan.

Well I'm not really sure where's the best place to define this.
I'm not even sure the Schedule section should be part of the
test plan. Because things you talk about are also described
here:
https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_policy
(which is not approved yet, but this or something similar will
be in the future).
It seems to me like a duplication of places where the definition
is. Ideas?