QA's Package update policy proposal

[Al Dunsmuir]

On Wednesday, March 10, 2010, 7:11:31 PM, Kevin Kofler wrote:

> Al Dunsmuir wrote:
>> The  update to an older stable release should be made widely available
>> in   that   release's   updates-testing   after  the  equivalent  (not
>> necessarily identical) fix has been widely tested in the latest stable
>> release.

> Uh, no, just no.

> They should go to updates-testing for both releases at the same time. 
> Anything else:
> 1. makes things harder for the maintainer, as he/she has to go through all
> the Bodhi procedures twice,
> 2. just delays the fix for users for no good reason.

> I can somewhat understand the argument that they should get separate testing
> (even though I disagree with it), or even that the stable pushes should be
> staged (even though I also disagree with that), but I really don't see what
> it hurts to have the update available in updates-testing right away. Testing
> is what updates-testing is for.

A lot of the conflict that I am seeing is based on two different basic
assumptions: 1) the update will be good and 2) the update will fail.

It  is  OK after sufficient unit testing to go with approach #1 on the
latest  release. If it turns out there are problems, they can be fixed
in one or more additional iterations.

For  older  releases,  the  presumption/requirement  for  stability is
higher.   To  meet that, I believe more people would prefer going with
approach  #2  for  these  releases.  Once the update works out OK with
no  regressions  in  the  latest  release,  you  have  a  much  higher
probability  that your the equivalent update to the older release will
also be stable and meet that release's stability requirement.

Throwing  both  into  their  respective updates-testing simultaneously
does  allow  for  wider  user  testing... but is limited by the actual
amount  of testing. Allowing both releases to be promoted to stable at
the  same  time  is  the real problem. If you do not (or can not) hold
back  the older (and in user expectations, more stable) release update
than  any problem will have a much higher perceived and actual impact.
If  you  don't have the resources to ensure that older releases remain
more  stable  than  newer  releases,  perhaps  you  do need to revisit
whether updates to both releases are a good idea.

>> This minimizes the risk that due to a different execution environment,
>> build  environment, configuration or whatever the seemingly equivalent
>> fix  does  not work but causes a regression. You may start at the same
>> place  in  the  older stable release, but may end up down and entirely
>> different rabbit hole.

> Is this really such a common issue that it makes it worth delaying all
> updates, including bugfixes, while waiting for testing that may never arrive
> (because those folks who like testing things tend to run the current stuff)?
>         Kevin Kofler

Problems  will  happen,  likely in the worst possible way at the worst
possible  time for your users. If you work on that assumption, it just
makes sense to take more care and time to roll out your fixes the more
stable the release is expected to be.

If  you  and  the  users  have  a  mismatch  regarding expectations of
stability,  it  doesn't  matter  whether  your  changes  are  fixes or
retrofitted  enhancement - there will be highly visible problem if you
try to cut corners in either time or effort.