bodhi statistics

[Luke Macken]

On Wed, 2010-06-09 at 01:46 +0200, Kevin Kofler wrote:
> Luke Macken wrote:
> > This report definitely conveys the shortcomings in our testing, however,
> > it does show us improving with each release. 

By 'shortcomings in our testing', I mean, 'shortcomings in the process
by which we currently use bodhi for "testing" updates'.

> For Fedora 13, we implemented
> > the No Frozen Rawhide process with improved Critical Path policies, which
> > were definitely a success. With these enhanced procedures, along with the
> > upcoming implementation of AutoQA and the new Package update acceptance
> > criteria
> > (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
> > think we'll see these numbers drastically improve in the future.
> 
> Only because those numbers are taylored towards that very process (they 
> measure the exact same things that process is going to enforce) and do not 
> reflect the actual quality of the packages in any way.
> 
> You can make really anything a "success" by measuring the very symptoms of 
> the process and calling them a metric of "quality".

These metrics do not reflect the actual quality of our updates, the
policy or process behind them, or even the quality of testing that they
endured before being released.  It is simply a measurement of how we
have been interacting with this specific piece of infrastructure.

By "success" I mean that I felt we were successful in drafting,
implementing, deploying, and utilizing the mentioned policies as
expected, and the results show increased community engagement.
Attempting to quantify the quality of these community interactions is
another story.  As to whether or not the policies behind the processes
have helped or hindered the release quality overall, that has yet to be
determined.

> The reasons for which Bodhi karma (especially in its current incarnation) is 
> a completely broken indicator of quality have been pointed out in several 
> past threads.

I'm well aware, and I agree that the current bodhi karma implementation
does not convey an effective measurement of update quality.  The problem
is not a lack of good ideas for improvement, but rather a lack of
developers to help improve.

luke