Drawing lessons from fatal SELinux bug #1054350

[Adam Williamson]

On Fri, 2014-01-24 at 21:17 +0100, Michael Schwendt wrote:
> On Fri, 24 Jan 2014 11:14:50 -0800, Adam Williamson wrote:
> 
> > It's not at all
> > obvious to anyone that you ought to test update/install of another
> > package in order to validate an update to selinux-policy-targeted .
> > Hell, I don't do that.
> 
> Amazing.
> 
> https://admin.fedoraproject.org/updates/FEDORA-2014-0806/selinux-policy-3.12.1-116.fc20
> 
>  | selinux-policy-3.12.1-116.fc20 critical path bugfix update
> 
> 
> https://fedoraproject.org/wiki/Critical_path_package#Actions
> 
>  | Packages within the critical path are required to perform the
>  | most fundamental actions on a system. Those actions include: 
>  |
>  | [...]
>  | get updates 
>  | [...]
> 
> 
> How to understand that?

It also says:

     graphical network install
    post-install booting
    decrypt encrypted filesystems
    graphics
    login
    networking
    get updates
    minimal buildroot
    compose new trees
    compose live 

Are we to be expected to re-test every single one of those actions with
every single critical path update? That seems unreasonable. If that were
the approach, I think Kevin would have an actual apoplexy while waiting
for his updates to get released. :)

> Especially with regard to downloading builds from koji, installing them
> manually and voting +1 even before a test update has entered the repo.
> 
> The fast people, who do that regularly in addition to a daily yum update,
> could not escape from this bug. On the contrary, other users who don't
> update often, have skipped the bad selinux policy update.
> 
> I consider it likely that the testers would have noticed Yum/RPM update
> errors, if only they had used their updated systems normal for let's say
> one or two days and at least one reboot.
> 
> There's also
> 
>   fedora-easy-karma --installed-min-days=4
> 
> which is can very helpful, since you won't be asked for updates installed
> just a few minutes ago.

Yup, indeed. Of course, this is another area where we could improve the
tooling: it doesn't seem like it'd be difficult for maintainers to be
allowed to set a minimum timeframe before their update goes stable, but
at present this isn't possible.

> Also let's not forget, for testing an selinux-policy-targeted update,
> you ought to run with SELinux in enforcing mode.

This is already generally understood, I think.

> > The 'comment' field exists to allow people to express all these things,
> > but as it's just a completely free-form text field,
> 
> ... and even can be left empty :(  so a packager doesn't get any
> explicit feedback from the tester other than the +1.
>  
> > Those are just examples: the point is that what we badly need here is a
> > more expressive and flexible system. (As well, as I've said elsewhere in
> > the discussion, as a good automated test for this specific and
> > well-known category of 'delayed action' update problems).
> 
> Is it so hard for testers to slow down a bit until such a system will be
> available? ;-)

Well, at present, it kind of is, because we have this Bodhi-Bugzilla
interaction where when an update is submitted that claims to fix a given
bug, Bodhi posts a comment on the bug that says 'try this update and
leave positive karma if it fixes your bug'.

I mean, we could tweak that process, but I still think the Grand Bodhi
2.0 Vision is much more interesting, because if we have multiple karma
types we can still take and use fast 'this fixed my bug' feedback
however we feel it to be appropriate, while having _much_ more
flexibility to do 'valuation' of feedback on a 'type of feedback' and
'package being updated' basis. Again, hate to sound like a broken
record, but it's just hard to get enthusiastic about trying to twiddle
the edges of the process when the process is fundamentally inadequate.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net