Drawing lessons from fatal SELinux bug #1054350

Kevin Kofler kevin.kofler at chello.at
Sat Jan 25 18:03:00 UTC 2014

Adam Williamson wrote:
> The 'comment' field exists to allow people to express all these things,
> but as it's just a completely free-form text field, it's intrinsically
> impossible to really base any programmatic stuff or even policy on it.
> In theory maintainers could submit updates without using autokarma and
> then keep a careful eye on the feedback and 'tend' their updates
> manually, but I think it's pretty clear that in practice, this is not
> what happens: maintainers really want to be able to use the karma system
> as a 'helper', they want to farm out the evaluation process to Bodhi/the
> karma system. But our current system is too stupid to handle this
> perfectly, so we get these breakdowns.

It's not that they WANT to be able to do that, it's that the system is 
rigged to encourage that broken practice. Autokarma (ab)use was much lower 
before the enactment of the Update Policies. And really, autokarma needs to 
just go away entirely. Having an intelligent being interpret the free-form 
text field is the only way to make sane decisions (which also implies that 
we should not arbitrarily impose any restrictions that, by their nature, 
cannot take the free-form text into account).

> With a more flexible karma system we have a *lot* of opportunity to do
> much cleverer stuff. We can provide presets for all the above different
> things that are currently commonly expressed via +1 or -1 with a
> comment. This opens up possibilities at two different levels: the distro
> policy level, and the packager level. We can make the distro policy much
> more fine-grained, if we want to - we can require certain of the 'karma
> types' to be available in all updates, and for instance, block any
> update where X people pull the 'it's completely busted' or 'it
> introduces a security vulnerability' cord, regardless of how much
> broadly-categorized 'positive' karma it has. At the packager level, the
> packager gets the freedom to define a much more fine-grained policy for
> when they're happy that updates to their package are 'good to go', but
> they still don't have to sit there reading the emails and manually
> interpreting what people have written. You get to define the policy that
> makes the most sense for your package, within the confines of the
> distro-wide policy - if you have a good package-specific test suite, you
> can say to the auto-karma system 'don't send this update out until at
> least one person sets the "I ran the test suite and it passed" karma
> property.

To me, this just screams "OVERENGINEERED!!!". :-(

You are introducing a lot of complexity, that will ultimately always only be 
an approximation to reality. You just cannot reliably quantify all the 
details. E.g. "this introduces a regression, but the regression is that you 
sometimes have to click OK twice (instead of once) to format a 5.25" floppy 
in KFloppy, whereas it fixes a critical bug in KDE Plasma Desktop where all 
your data was sent to the NSA and then securely wiped from your hard disk". 
:-) (Yes, of course that is an exaggerated example. ;-) I sure hope we don't 
have bugs like that. ;-) ) If you only have "fixes a bug, but introduces a 
regression" as a feedback type, you probably end up making the wrong 
decision. If you try to get more fine-grained, then you again need numbers 
to quantify the severity of one issue vs. the other, and those will 
inherently be subjective. (Users always think THEIR bug is the end of the 
world whereas regressions that don't affect them are entirely unimportant.)

The complexity also means there are a lot more arbitrary parameters to deal 
with. The current stable/unstable thresholds are already bad enough, and 
often end up set to the wrong value. A decision process tends to be the 
worse the more arbitrary parameters it needs.

And of course, more complexity means less transparency. It becomes harder 
and harder to understand what really needs to be satisfied for an update to 
be allowed to go stable.

So I can only advocate for the KISS approach: The update is stable when the 
maintainer says so, period. We do not need any karma, be it a simple ±1 or a 
long (and inherently non-exhaustive) list of all the things that can 
possibly happen. So let's just do away with the 3 radio buttons and use a 
free-form text field only. We just need somebody able to read comments, and 
that is what we have maintainers for!

        Kevin Kofler

More information about the devel mailing list