gathering useful statistics for QA team

["Jóhann B. Guðmundsson"]

  On 07/20/2010 01:13 PM, Kamil Paral wrote:
> Hello,
>
> we would like to help QA team to increase public participation in its
> activities. We believe that an important step in achieving that is in
> rewarding the most active participants with "fame" in different top
> tens, ladders and charts. Therefore we would like to extend the current
> Fedora Community website [1] with different statistics regarding user
> participation in different areas relevant to your team. We have called
> this project "Fedora Hall of Fame" [2]. This information can then be used
> in different newsletters (FWN, etc) to praise the best people and motivate
> the others.
>

Ah the carrot game here a couple of things you need to know when playing 
that game....

Firstly.

Wont this lead to less quality of testing/triaging as in 
reporters/triagers will end up competing amongst themselves to reach the 
"carrot" the "hall of fame", resulting in every reporter starts either 
filing bug for every error msg they come across and provide +karma for 
every package that goes through bodhi with similar behaviour in bugzilla 
from triagers.

Secondly.

How do you think new reporters/triagers that see those stats are going 
to react?

Thirdly.

How far back into time are you thinking about gathering those stats?

Fourthly.

How are you going to compare Red Hat and other company's employs that 
literally are paid to work at this vs those that are freely given what 
ever time they have to the project?

Fifthly

Do you think they will ever manage to be equally and or more productive 
than those that get full time pay to do this?

Sixthly

Do you believe that newcomers and those that give freely what ever 
limited time they have will look the hall of fame and think by 
themselves hey I think I can get my name up there?

Seventhly and perhaps one of the more devastating downside when playing 
the carrot game..

How do you suggest that the community handles those that have spent 
countless hours trying to get their name in "Hall of Fame" when the cold 
hard reality strikes them and they realize they cant and we risk loosing 
them and their valuable contribution from the project forever?

> The information we hope to receive from you is:
> 1. What are the most important tools you would like to have tracked?
> For example it can be your wiki, mailing lists, Bugzilla, Koji, Bodhi,
> Transifex, packages' source code, and so on.
> 2. What are the most important characteristics you would like to see
> gathered? For example: # of wiki edits, # of new bug reports, # of
> package updates released, etc. Some of our ideas are at [3].
>
> In short, we're looking for hints which statistical data would help
> your team most in evaluating your best contributors.
>

Here's an Idea..

How about starting from the right end and gather those information about 
components then about the maintainers then about QA and the rest.

We need to know components stats first and foremost so we can 
effectively focus the project resource where they are mostly needed.

So I can get on the same page what's the end game here as in what do you 
think can be achieved by this?

Was the solution of "playing the carrot game" the only solution on the 
table that could achieve that goal?

JBG