gathering useful statistics for QA team

[Kamil Paral]

----- "\"Jóhann B. Guðmundsson\"" <johannbg at gmail.com> wrote:

> On 07/20/2010 01:13 PM, Kamil Paral wrote:
> > Hello,
> >
> > we would like to help QA team to increase public participation in
> its
> > activities. We believe that an important step in achieving that is
> in
> > rewarding the most active participants with "fame" in different top
> > tens, ladders and charts. Therefore we would like to extend the
> current
> > Fedora Community website [1] with different statistics regarding
> user
> > participation in different areas relevant to your team. We have
> called
> > this project "Fedora Hall of Fame" [2]. This information can then be
> used
> > in different newsletters (FWN, etc) to praise the best people and
> motivate
> > the others.
> >
> 
> Ah the carrot game here a couple of things you need to know when
> playing 
> that game....

Hello,
thank you for your comments. I think you take it a little too seriously
and to the extreme. We certainly don't want to conclude some hard results
from these statistics - this person is the best, this person is the worst -
not at all. We suppose that a common sense will be used when reviewing
these stats. James has replied really well on your email, I really share
his point of view.

We suppose those statistics may be valuable for many team leads when 
thinking about their team's performance. They can use it when evaluating
which people from community do a really great job and act properly 
afterwards (for example praise their work in FWN or elsewhere). Not being
in the top ten certainly doesn't diminish your work.

I have participated in quite some number of communities and I can tell
you that even though I don't usually care about these kind of statistics
and I am not a boasting person, seeing your name mentioned in some article
or statistics can surely please you. I feel then more energetic and
I have higher motivation to continue. It works this way for many people.
When they devote their personal time for some common good, they like to
see some acknowledgment from time to time. People are not robots. When they
feel like a cog in a machine, they loose interest soon.

Maybe you have gained an impression this would be a big project. No, it
will not. There's a few of us and we have very limited time. We want to 
implement a few more statistics into Fedora Community site. But even though
this won't be big, we hope for a small difference in a positive direction.
That's why we ask what statistics would be most valuable for different teams.
How to use it - that's up to them. Again, we suspect common sense to be
around us, especially in open source communities.

> 
> Firstly.
> 
> Wont this lead to less quality of testing/triaging as in 
> reporters/triagers will end up competing amongst themselves to reach
> the 
> "carrot" the "hall of fame", resulting in every reporter starts either
> 
> filing bug for every error msg they come across and provide +karma for
> 
> every package that goes through bodhi with similar behaviour in
> bugzilla 
> from triagers.

Many people don't care about such ladders. I don't. But everyone is
susceptible to praise. They won't lower the quality, but will be delighted
if praised. Better mood, better quality. Win-win solution.

And then there will be surely some of them as you mention - sacrificing
quality for highest metrics. There are always people like that among us.
But, truthfully, do you expect to see many of them? I don't. And the
community has some ways how to solve these issues, as always. L10N team
members would surely know that someone posted 50 new translated sentences
and 50 different commits just to have metrics bumped. Is it a glory then
to be at the top of the ladder this way? Or rather shame?

> 
> Secondly.
> 
> How do you think new reporters/triagers that see those stats are going
> 
> to react?
> 
> Thirdly.
> 
> How far back into time are you thinking about gathering those stats?

I think we won't have the capacity to provide long-term metrics. I believe
it will be last week/last month, that's it.

> 
> Fourthly.
> 
> How are you going to compare Red Hat and other company's employs that
> 
> literally are paid to work at this vs those that are freely given what
> 
> ever time they have to the project?

Common sense again. It is expected that people from Red Hat that are paid
to do this job will be like to occupy some front lines. Does it diminish
the effort of those non-redhatters that share the top ten with them, or
quite the reverse - it emphasizes their effort much more, because they
were capable to even out with paid employees? Red Hat people could be even
somehow distinguished in the stats, if technically possible, good idea.

If some team decides to send some gifts to top contributors, it seems
obvious to me to send it to top community contributors, not to salaried
employees. Just seems reasonable to me, again.

> 
> Fifthly
> 
> Do you think they will ever manage to be equally and or more
> productive 
> than those that get full time pay to do this?

Some of that might, some of them may not. What does it matter? This
is not a car race, not a million dollar competition, this is a
community project.

> 
> Sixthly
> 
> Do you believe that newcomers and those that give freely what ever 
> limited time they have will look the hall of fame and think by 
> themselves hey I think I can get my name up there?

There are areas where we might be very surprised how few people
really do some work there.
"Hey, really just a three people do package reviews every month?
Maybe I could help out with that."

> 
> Seventhly and perhaps one of the more devastating downside when
> playing 
> the carrot game..
> 
> How do you suggest that the community handles those that have spent 
> countless hours trying to get their name in "Hall of Fame" when the
> cold 
> hard reality strikes them and they realize they cant and we risk
> loosing 
> them and their valuable contribution from the project forever?

Why would you quit? Just because you know there are people that
invest more time to the project that you can afford? Is there some
requirement you must work 8 hours/day? I don't think so.

You paint it black, but this is a really artificial problem. There
is always someone able to invest more time into it than you are,
we all know that. That doesn't render your work worthless.

> 
> > The information we hope to receive from you is:
> > 1. What are the most important tools you would like to have
> tracked?
> > For example it can be your wiki, mailing lists, Bugzilla, Koji,
> Bodhi,
> > Transifex, packages' source code, and so on.
> > 2. What are the most important characteristics you would like to
> see
> > gathered? For example: # of wiki edits, # of new bug reports, # of
> > package updates released, etc. Some of our ideas are at [3].
> >
> > In short, we're looking for hints which statistical data would help
> > your team most in evaluating your best contributors.
> >
> 
> Here's an Idea..
> 
> How about starting from the right end and gather those information
> about 
> components then about the maintainers then about QA and the rest.
> 
> We need to know components stats first and foremost so we can 
> effectively focus the project resource where they are mostly needed.
> 
> So I can get on the same page what's the end game here as in what do
> you 
> think can be achieved by this?
> 
> Was the solution of "playing the carrot game" the only solution on the
> 
> table that could achieve that goal?

We surely need more statistics. Our time is limited and we decided to
invest it into community participation related statistics, because we
feel our community management and motivation management seems to lag
behind a little.

Anyway, I don't want to write another long emails about that. Let's
keep it constructive, not negative-be-default. Those statistics already
exist and anyone can grab them, they are not gathered yet. Even though
I'm from QA team, I would like to see some suggestions which characteristics
could be useful from our point of view.