Drawing lessons from fatal SELinux bug #1054350

[Adam Williamson]

On Wed, 2014-01-29 at 09:26 -0500, Matthew Miller wrote:
> On Tue, Jan 28, 2014 at 06:42:52PM -0800, Adam Williamson wrote:
> > Using the wiki as a TCMS is of course a gross hack, but it comes with a
> > surprising number of advantages - see
> 
> Makes sense to me. I'm actually not opposed to using the wiki in this way,
> but I think it highlights the need for _something else_ to be our
> short-quick-easy documentation solution.
> 
> > >  I wonder if it would help both
> > > of these things to move the test plans to live in package git (right next to
> > > the future taskotron scripts?
> > Well, what would they look like, and how would you read them? What we're
> 
> They would be short text files. Possibly markdown. Something simple, though.
> 
> I hear you on the downside of having to have commit rights to the package in
> order to work on test plans in git. That applies to the taskotron tests too,
> though, doesn't it? Maybe we could use git submodules for this. (Where each
> fedora package has a submodule consisting of its tests, both automated and
> test plans.)

Yeah it does - there's an argument to be made that you need a lot more
(or, y'know, any at all) in the way of dev skills to write a taskotron
test than a manual test case, but still, you could be the world's
foremost expert on automated testing and not have provenpackager rights,
so it's still a problem. Submodules with broader access rights doesn't
seem like a bad design...but I guess if this all gets hooked up to
Bodhi, you should still need package commit privs to decide which
automated tests can cause an update to be rejected or whatever (or else
anyone can DoS a package, accidentally or intentionally, by adding a
broken automated test).

> But like I said I'm not really opposed to continuing to use the wiki. Maybe
> we just need an awareness campaign. (Some text in the bodhi.template from
> fedpkg update would be a good start, I think. And on the New Update page in
> the bodhi web interface. And definitely in
> https://fedoraproject.org/wiki/Package_update_HOWTO -- I'll do that, as soon
> as I have that coffee I didn't have when I said I was going to at the end of
> the last message I sent.)

OK, that makes sense - you're talking about 'publicising' this system in
the places *packagers* see when interacting with Bodhi. Absolutely, good
idea, no objections.

> > > And, while clearly helpful, I think it suffers from a little bit of tl;dr.
> > > Maybe we could embed summaries of each right into the Bodhi page?
> > I'm not quite clear what you mean here - what's tl? Summaries of what?
> 
> I mean the links to the test cases shown in bodhi. In the context of trying
> to get better feedback in bodhi, it would be nice if there were a summary of
> each test right in the bodhi screen (not just its title).

Ahh, I see. I'd worry that would make the Bodhi pages a bit long, with
the way test cases are laid out - they tend to be vertical space hungry
- but it'd be interesting to mock it up and see. 

>  Then, there could
> be a little dropdown to choose between "didn't try this aspect", "looked at
> the thing in the summary; seemed okay" and "actually went through this full
> test case". That would make it easy for people to be descriptive and honest
> about what they actually tried.

*This* bit is part of the Glorious Bodhi 2.0 Vision, yep - if you go and
find that post from forever ago in the archives, it describes something
very like this. It is Bodhi 2.0-dependent, though, I think.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net