FESCo wants to ban direct stable pushes in Bodhi (urgent call for feedback)

Paul Wouters paul at xelerance.com
Fri Feb 26 15:55:23 UTC 2010


On Fri, 26 Feb 2010, Jesse Keating wrote:

>> +1. Not being able to push those out quickly would really suck.
>
> What sucks more is recent "hot-fixes" which were even more broken than
> the issue they were trying to fix.  They were pushed directly to stable
> and broke a significant number of systems because of a scenario the
> maintainer didn't imagine or test.

I can see both sides here, having been part of a big disaster recently
(which might have been the cause of this discussion at FESCO). Let me
describe what happened in this case, so people can see these issues can
get very complex.

Due to delays in getting a fedora package updated (purely my fault)
the dnssec-conf package had expired DNSSEC trust anchors. This caused
a denial of service attack by Fedora servers against various important
nameservers at RIPE. This change was not due to a package update, but
due to a missing update. This needed urgent fixing, as in someone called
me at 4am. I got together with the bind maintainer, prepared an update
and we both tested it. It worked for us. I had this tested on devel,
F-12 and EL-5.

I requested a direct push to stable. Which was denied. I was unhappy that
we would not stop a DOS attack within weeks (my packages hardly ever get
any karma feedback despite their obvious use, though I must say that did
change for dnssec-conf after it blew up). So I objected, and got my way.

A push to stable happened, and something went wrong. It turned out that
anyone who had upgraded from F-10/F-11 onwards had a configuration file
in an old location. Testing should have seen this, but we didn't.

In this case the result was bad, though not worse without the
update. Without the update, the bind nameserver would be DOSing RIPE
(and not work very well as a nameserver because it was too busy). After
the update the nameserver failed to start with an error in a few include
statements to non-existing files that was annoying but fairly easy to
fix even for non-developers.

The fix for that was pushed into testing. I did not dare to request
going straight to stable again, though objectively looking from a policy
statement, that's what should have happened to avoid people upgrading
to the broken release.

Looking back at this, the ideal solution would have been to involve a
few people in testing/approving the direct-to-stable push. I believe
this is what Fesco is thinking about with a Fesco/releng/QA. Whether you
call this an "expediated push from testing" or a "direct push to stable"
is really semantics. What's important is that there are resources that
can take on a "hot fix". I don't know where we can get those resources,
as Fesco seems pretty busy as it is.

Summary: Blocking direct-to-stable is fine, as long as requests to
override it are taken seriously and somehow the package gets some
additional attention from someone beside the packager.

It all comes down to human resources....

Paul


More information about the devel mailing list