Drawing lessons from fatal SELinux bug #1054350

[Matthew Miller]

On Fri, Jan 24, 2014 at 11:46:41PM +0100, Michael Schwendt wrote:
> Any ideas how to attract more testers?
> How to make the updates-testing repo more sexy?

* More automated smoke-tests, so that:

 a) you know the most boring tests have been handled so you
    can focus on more interesting apsects of the update
 a) it's less likely that something broken will slip through,
    so it's more safe to keep the testing repo active

* possibly adding a "what should users test?" field to the update info.

  I know that there's a "notes" field in the update, but maybe it'd help
  to explicitly include testing instructions?

  Each package in the pkgdb (or in git, or wherever) could have
  a standard list included in each update as the default (for example, for
  'calc', it might be to try `calc -q read /usr/share/calc/regress.cal`.
  That would duplicate a likely smoke-test, though, so maybe also "run
  interactively and make sure basic math works".

  Then, each update could also optionally (and this would be presented in
  bold if it were used) say something like "New release adds log() function;
  please test that it works", or "Severe bug where 1+1=3 corrected; please
  test that the answer now corresponds with consensus reality."

* badges! We already have this, but making them more visible would help.
  People on Stack Exchange get crazily obsessed with quality control all
  in exchange for digital gold stars. 

* In fact, this ties into the general problem that various bits of Fedora
  are disconnected and not particularly discoverable. You can find them if
  you're looking, but mostly out-of-site, out-of-mind. Unless one is already
  engaged, how would one even know that this is really an easy way to help?

* Present pending updates in a more informative way.

  Let's say I'm in the mood to be helpful and test some stuff and earn some
  badges. I go to https://admin.fedoraproject.org/updates/ (because I have
  that bookmarked; see the previous point). I see a list of package names,
  and update types. Maybe I recognize something, maybe I don't. I don't
  right now, it happens, so I think "okay, there's xflr5. Maybe that's
  interesting". I click on
  https://admin.fedoraproject.org/updates/xflr5-6.09.06-1.fc20, and am
  immediately shown that this is only a package I should care about if I
  already know what it is -- that is, there's no description at all. The
  update note is reasonably helpful as these things go (it's an update to a
  new upstream version), but the URL isn't hyperlinked, and when I go there,
  Sourceforge and makes me wait and then makes the file download instead of
  displaying it. And then I see that this program is very a technical
  airplane engineering program and I'm not qualified to test it. Okay, there
  went five minutes of my life. Shall I guess another one? Hmmm, maybe
  "alleyoop". Will there be cavemen? Okay, also no description, so I'll do
  "yum info" in a shell. Ah, okay, memory debugger GUI. Okay, I can test
  that one but it's not necessarily gonna be quick....

  Anyway. The list could be more informative. Maybe I could even star
  particular packages I'm interested in, and future updates would show up
  first. And after choosing something, the above idea of having a quick
  description of what to test would help here too.

* Silly, but... remember my login in bodhi longer, so there are fewer
  steps.

-- 
Matthew Miller    --   Fedora Project    --    <mattdm at fedoraproject.org>