Proposals for the Updates Testing Procedure

[Michael K. Johnson]

On Wed, Nov 12, 2003 at 09:48:15PM -1000, Warren Togami wrote:
> Along with each announced test update, I believe it would be crucial to 
> include a link to a corresponding Bugzilla report.  While longer 

This is probably a good idea, even if we need to create an
empty report for discussion.

> * Backport Patches v. Upgrade Version
> Most libraries, core system and server packages should always have 
> backport patches.  End-user applications where other packages do not 
> depend on them can possibly be upgraded in version after sufficient 
> testing in updates-testing.  There are always possible exceptions to the 
> rule, although the determination of backport verses upgrade version will 
> need to be decided on a case-by-case basis based upon how intrusive the 
> change would be. (Yeah this description sucks.  Reply with a better and 
> expanded proposal if you care about this.)

Here we have disagreement, at least in wording.  As documented at
http://fedora.redhat.com/about/objectives.html
one of Red Hat's prerequisites for participation is:

 o  Do as much of the development work as possible directly in the
    upstream packages. This includes updates; our default policy will
    be to upgrade to new versions for security as well as for bugfix
    and new feature update releases of packages.

This is the default; when there is a good reason to use a backport
patch it is of course OK to do so.  Here are examples of reasons:

 o  It's a critical security fix and the changes to a new upstream
    package are so large that they cannot reasonably be tested in
    time.

 o  Configuration file formats have changed.

 o  Library ABIs have changed.

Essentially, the one main reason that should not be used for doing
a backport is "it's the rule".  We do, however, recognize that the
maintainer of the package has expertise in that package and so we
expect to trust maintainers' judgement for when exceptions should
be made.

Practically speaking, I'm not sure how many real differences there
will be between our approaches, since the exceptions on both sides
are relatively complementary...

> * Time-limit to publish where no negative comments are posted within the 
> Bugzilla report.  Senior developers reserve the right to hold an update 
>   if a good technical reason can be stated.  (Insert more details here.)

I'd like to have something a little different here.

Let's step back and look at the goal, which is simple "plenty of testing
without finding unexpected regressions".

Time doesn't really tell us anything.  I'd like us to be able to get
some information on downloads, and then consider the time since some
reasonable number of downloads have been done.  The amount of testing
that is going to be needed will depend on the package, and I'd think
that the package maintainer is going to be a good judge of how much
testing is needed.  So I'd say we should come up with guidelines, but
be flexible.  We need to look into providing the data on which to
base good decisions.

Also, security updates are clearly going to need to be handled differently
from bugfix updates, and new feature updates also differently.  Security
updates need the most expeditious handling; bugfix less so, and new feature
updates will tend to need less-expeditious handling *and* a larger amount
of testing.

> Why GPG?  Wouldn't that be slow and a waste of time?
> ====================================================
> No.  The GPG clearsigned messages over time that collect within many 
> reports and mailing list posts help to build a mass of "historical 
> evidence" about the reliability and trustability of the advice given by 
> a contributor.

Well, I'd say that a bugzilla account does the same.  I know that it
is only password-protected, but someone could GPG-sign a message saying
that they weren't responsible for obnoxious bugzilla messages posted
from their account -- and ask that their bugzilla password be reset!

That said, Red Hat is operating under the assumption that all developers,
at least, will have GPG keys.

michaelkjohnson

 "He that composes himself is wiser than he that composes a book."
 Linux Application Development                     -- Ben Franklin
 http://people.redhat.com/johnsonm/lad/