[Fedora-packaging] critical path security update policy

[Michael Schwendt]

On Tue, 21 Apr 2015 08:42:58 +0200, Mario Torre wrote:

> >> Security has precedence over even backward compatibility.
> >
> > Said that and applied a _tested_ Yum update (a few positive votes in
> > bodhi) that resulted in Yum not working at all anymore. Worst-case
> > scenario for all users that rely on Yum for updating their installations.
> > It has been discussed quite a lot after it had happened. It's just an
> > example here. (WDYT why three installed kernels are kept?)
> 
> Well, I guess the yum maintainer is also a developer of yum, he should
> know better.
> 
> The case you mentioned may have happened (everybody screws up from
> time to time, that's ok), but I'm confident this is not always the
> case.

True. Still it's a reason why there are update policies and why package
maintainers are not fully trusted and may not ignore the policies.

It's not pretty.

Every once in a while a self-confident maintainer would love to rush out a
fix that's considered important and safe, and some days later it turns out
that everything is fine indeed, but taking the risks is not worth it. In
case there's breakage that slips through, everything is different.

Packages, which have been sitting in updates-testing for one week without
any feedback in their bodhi ticket, may be completely untested when the
maintainer pushes them into stable updates. Same for two weeks, four weeks,
and longer times of waiting for karma points. We implicitly assume the
package maintainer is happy with the update and doesn't need to self-vote
on them. Anyone else been using those updates? No idea! Only with more
users enabling updates-testing, there may be users among them who use
the packages actually. Once the same packages are pushed into stable
updates, it's too late.  [I'll try not to repeat that again ;)]

> And anyway, this is different really, a security patch generally can
> cause only minor areas of distress, and they should be "easily"
> tested, because when you apply security patches you don't also apply a
> bunch of extra new development things, you only apply (or should so!)
> the actual security fix.

It's not that I'm in the position to "block" policy changes or that I want
to fight them in discussions like this. It's just that you cannot know
whether a completely new build work flawlessly. This update of firefox
has been a new minor version. And for F21 there have been more testers.

Imagine the update submitter were ultimately trusted and could push an
update directly into stable updates. What to do if it broke for many
users? Try to fight the symptoms with even more hot-fix updates? How to
take responsibility and avoid repeating mistakes in the future?

> Perhaps we should teach people how to apply security patches instead then?
> 
> >> The maintainers should be ultimately responsible to ensure that the
> >> package they maintain is in a coherent state and in theory just
> >> backporting the security patches. I know that is is often easier said
> >> than done, but the general rule is security first.
> >
> > Cheap talk. ;)
> 
> It's not cheap talk, this is what we do for the Java packages, it's
> actual facts.

The "easier said than done" is the problem. That's what I've referred
to. Enough fixes for security vulnerabilities (and normal bugs) are not
backported. And even if they were backported, you do fresh builds in
a buildroot that may have changed in unknown ways, which invalidate all
past testing results.

> Software should be tested, granted. But a security errata, well
> widespread and understood, with reproducers out in the wild, and in a
> primary component like Desktop, Browser, Java, Kernel, *can't* linger
> for two weeks or so without being pushed.

The update submitter could give some background.

There is a policy. The automatic push to stable would have been triggered
quickly, if the update had not been submitted with a +3 karma threshold
instead of the minimum +2.

Firefox is a critpath package. Who would decide when/if it may ignore the
critpath update policy for a security-fix? What to do with other critpath
packages? Individual policies for each package? Services/computers that
may refuse to start/boot. Data loss, corrupted databases. Software updater
tools that would silently fail to download updates. One could come up with
different levels of importance for each package and an alarm-bell (and
a messaging system) that would try to raise awareness of testers. Perhaps
require dedicated testers for each package. Two people would have been
enough for Firefox this time.

> Of course, we can argue forever but the real question is how do we fix it?

Well, that's where "Jerry" has not suggested anything other than to build
packages earlier, release faster, do fully automated pushes.