A Glorious Vision of Our Shared Update Feedback Future (bodhi, karma, and proventesters, oh my)

[Adam Williamson]

On Tue, 2011-11-22 at 21:31 +0000, "Jóhann B. Guðmundsson" wrote:
> On 11/22/2011 09:03 PM, Adam Williamson wrote:
> > 2. Any update marked as 'critpath breaking' by a proven tester would be
> > blocked from being pushed stable at all - automatically or manually -
> > until the PT modified the feedback or it was overridden by someone with
> > appropriately godlike powers (TBD, but probably not just the maintainer)
> 
> Hum I thought the proven tester concept was being dropped anyway.

At present that seems like where we're going, but I ignored it for the
purposes of the Glorious Vision. The Glorious Vision does, I think,
provide a reasonable suggestion as to how PTs could be useful if we had
more detailed karma possibilities.

> A) Have maintainers for critical path component provided test cases for 
> proven tester to use to test to differentiate them from fas-tester and 
> the "I installed this update and continued to use my system as normal 
> and did not note any regressions'"?

In many cases no, but the nice thing about critpath is that in most
cases we don't really need that. For some tricky components it'd be nice
to have test cases that effectively explain exactly how that component
intersects with critpath, but for almost all critpath components, I
can't see that it would ever be wrong to hit the 'panic button' if you
installed the update, rebooted, and the system blew up, for instance.
The nice thing about a system based on *negative* feedback is that while
it can be less than optimally efficient, it's almost always useful to at
least some degree even when it's not perfect.

> B) Has the scalability problem be solved as in the required minimum 
> amount of active proven testers has been met for this to actually work?

See above: again the nice thing here is we don't _need_ a critical mass
for a mechanism based on negative rather than positive results. Of
course, the more testers we have the more issues we will likely
identify, but a panic button is useful even if it only ever gets pressed
once (as long as the press isn't a false one). Even if there were 99
other times when it _could_ have been pressed and wasn't, 1/100 success
rate is better than 0/100.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net