QA's Package update policy proposal

[James Laska]

On Tue, 2010-03-09 at 19:50 +0100, Kevin Kofler wrote:
> Adam Williamson wrote:
> > The proposal isn't really about expanding testing activities, it's about
> > formally codifying how the updates process is actually supposed to work.
> > Right now, we don't in fact define how the Fedora update process is
> > supposed to work: how updates get submitted, how they get promoted and
> > how they get released, who has what responsibilities at what point, none
> > of these is written down. That's what the proposal submitted by Kamil is
> > about. It's not really about trying to impose additional testing or
> > restrictions on updates.
> 
> Then why does it codify things which are not current practice, and in fact 
> require Bodhi changes to be implementable at all?
> 
> And why is "updates get pushed to stable when the maintainer feels they are 
> stable" (which is basically the current practice) not sufficient? We should 
> trust maintainers. That's what we have maintainers for. Stuff like karma 
> should only be a tool to help the maintainer decide.

We'll make adjustments based on feedback so far.  But I want to point
out that one goal for this policy is to reach consensus on a set of
criteria that all [1] packages must adhere to in order to be accepted as
Fedora updates.  The important word for me is "accepted".  This comes
long before any functional or bug verification.  This is intended to
support the fundamentals of packaging software for Fedora that have
already been established and are used to evaluate all software upon
entry into Fedora [2].

Some basics I'd propose as a starting point for defining acceptance
criteria include:

     1. repoclosure/conflicts - no package update can introduce broken
        deps or conflicts.  I'd recommend we apply this to both
        'updates-testing' and 'updates' (but that's detailed below)
     2. Package sanity
              * No rpmlint failures
              * Is the Source properly defined
              * License review/examination (if possible)
              * Upstream Source match tarball
              * Package scriptlet syntax checks
     3. Package must be newer than previously released versions - can't
        ship newer package in N-1.
     4. Any additional MUST requirements folks would like to see covered
        from the package review requirements?

Perhaps we can adjust the proposed workflow to detail there are multiple
phases of testing package updates, the first of which is acceptance
testing packages.  Once packages have landed in 'updates-testing',
that's when defect and functional verification can start.

With regards to defining acceptance criteria for package updates, does
the following correctly capture the next steps?

     1. Determine the appropriate subset of acceptance criteria
     2. Determine the appropriate package subset to apply the criteria -
        @critpath or all
     3. Agree on when the criteria will be applied
              * Upon entry into 'updates-testing'
              * And upon entry into 'updates' - perhaps same set, or a
                more complete set of criteria

Am I missing content, or way off base there?

> > The most important thing is to have a framework explaining who does what
> > at what point in the process; exactly what criteria are used to accept or
> > reject updates is not the main thrust of this proposal. As Kamil wrote,
> > the numbers in there at the moment are essentially placeholders, they
> > could be changed entirely, and that's not the important thing about this
> > proposal.
> 
> The numbers aren't the only controversial part of that proposal. Even 
> changing all the numbers to 0 would still create a policy which is much more 
> restrictive than what we have now (e.g. direct stable pushes aren't in the 
> workflow at all; a direct stable push is different from requesting a push to 
> stable after 0 days of testing, which assumes that we allowed it to hit 
> testing in the first place, so it can only go to stable in the next push).

Good point.

Thanks,
James

[1] In light of earlier feedback, it seems reasonable to restrict this
to critpath packages at first.
[2]
https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_policy

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20100309/ec64f91b/attachment.bin