Responsibility for rebuilding dependent components, was: F-16 Branched report: 20110920 changes

[Doug Ledford]

----- Original Message -----
> So when _is_ a good time to do binary-incompatible changes to
> libraries?
> 
> * It's not after beta freeze, because they are unwanted at that time
> 
> * It's not 14 days before beta freeze, because they won't get out of
> updates-testing in time
> 
> * It's not 14 days + 3 (4?) weeks before beta freeze - even if the
> library gets out of updates-testing in time, its users may not be
> rebuilt because the maintainer is on vacation.
> 
> * What if there are two layers of users that need to be rebuilt?
> 
> The delays just pile one upon another...
>    Mirek

I'd like to expand on my previous email (the one where I played devil's advocate) and pick up where Mirek left off here.

I'm fine with our new early branched methodology, to a point.  I think it's a good idea that we do an early branch and separate our upcoming release from rawhide.  However, I think the process we use to get packages from updates-testing into the actual GA candidate tree is the problem.

I think we are gating package updates on the wrong thing: testing.  I say this because the *real* testing happens with the alpha/beta/rc candidate install images, not with individual testers pulling packages from updates-testing.  And when trying to stabilize a product for GA, you want *everyone* testing the updates, not just a few.

Instead, I think we ought to revamp the process like this:

Maintainer A builds new package B
Maintainer A files a bodhi ticket for package B
In that ticket, the maintainer is responsible for list each item of change from the previous package already in the compose tree.  For example, did the upstream source get bumped, did any new patches get applied, did any old patches stop being applied, are the changes verified bug fixes as tested in rawhide, are the changes isolated or are there feature additions as well, will this update create dependency problems from things such as an soname bump, will other packages need to be rebuilt.
Finally, the bodhi update should be reviewed by people from release engineering, and if the ticket meets the requirements of a reasonable change at this late stage of the game, the ticket should be approved and the package pushed to stable.

The point of this process is that when stabilizing the product for GA, we want to minimize unnecessary or risky churn, and that doesn't need testing, that needs a human to make a judgement call.  Then, once the judgement call is made, we need as much testing as possible to make the release as good as possible.  Holding things up in updates-testing and then releasing them later throws away untold instances of testing, wasting those resources on a package that may never be on the final product.  We need to make that judgement call, put the package in if we are going to, test the hell out of it, and fix any breakage we find.  We need this iterative "test, report breakage, fix, retest" process to be as fast as possible, and our current process moves at the speed of a salt coated slug.

That's my proposed process for our early branched release.  Thoughts?

-- 
Doug Ledford <dledford at redhat.com>
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford