Drawing lessons from fatal SELinux bug #1054350

[Adam Williamson]

On Tue, 2014-01-28 at 15:19 -0500, Matthew Miller wrote:
> On Tue, Jan 28, 2014 at 10:39:32AM +0800, Mathieu Bridon wrote:
> > > I could have sworn we already had something like this where bodhi would
> > > add a link to a wiki page for test plan on a package if that wiki page
> > > existed. I can't seem to find it now, so perhaps it was just something
> > > we talked about, but never implemented. 
> > Nope, you're right. :)
> > https://github.com/fedora-infra/bodhi/blob/develop/bodhi/model.py#L191
> > For example, the test case pages for the package 'foo' need to be added
> > to the 'Category:Package foo test cases' category.
> 
> Huh, lookit that. That's great! So I guess my suggestion changes to "make
> that more visible".
> 
> Also my head is exploding a little bit, because it's yet another case of the
> wiki doing a lot of different types of work.

That's because I came up with the design, and I'm an idiot monkey who
knows how wikis work and not much else. :P

Also, tbf, it's because we use the wiki as our TCMS (test case
management system - look, I'm using jargon to pretend I have some kind
of formal knowledge about my job!), and as long as we're doing that,
it's probably no *more* hideous to use wiki categories for this kind of
purpose as it is to weld on some kind of external method for
categorizing test cases in this way.

Using the wiki as a TCMS is of course a gross hack, but it comes with a
surprising number of advantages - see
https://fedoraproject.org/wiki/Tcms_Comparison . It's very
https://xkcd.com/1172/ , but goddamnit, my spacebar heating works. We're
not opposed to moving to some better in principle, but it would be a
moderate amount of work and we do at least want the 'better' thing to be
really and truly and concretely better and not just 'at least it's
actually called a TCMS' :P

>  I wonder if it would help both
> of these things to move the test plans to live in package git (right next to
> the future taskotron scripts?

Well, what would they look like, and how would you read them? What we're
calling a 'test plan' for these purposes is basically just a category.
Sure, we could stick some kind of metadata in some format in package git
that just included the list of test cases associated with that package,
but that doesn't seem like a fantastic design either, and it comes with
the rather substantial downside that you now need commit rights to a
package in order to write a test plan for it (or the co-operation of
someone who has those rights). Of the relatively few test plans we have
so far, most were written by QA folks, not package maintainers.

> And, while clearly helpful, I think it suffers from a little bit of tl;dr.
> Maybe we could embed summaries of each right into the Bodhi page?

I'm not quite clear what you mean here - what's tl? Summaries of what?

The *user experience* here is indeed that you just see the test cases
right in the Bodhi page (or in whatever other interface you're using to
view the update - easy-karma, gooey-karma, whatever). The documents
describing how the system works and how to create test plans etc are
somewhat long-winded, yeah, in the latter case because I wrote it. :P
Are you saying you'd like a summary of how to create a test plan right
in the Bodhi page for an update? that seems a bit inappropriate.

> The other thing is that I think that in addition to the per-package test
> cases, it would be good to encourage updates to include at least a line
> about what is of particular interest to test in _this_ release. Sometimes,
> that's the same as the update Details, but not always.

It would indeed be nice if maintainers did this, though of course it's
not relevant to the case we're currently considering (as the broken
content in the update was a mistake and the maintainer didn't even know
it was there, he'd hardly have been likely to highlight it for testing).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net