bodhi statistics

[Luke Macken]

On Wed, 2010-06-09 at 09:35 +0200, Marcela Mašláňová wrote:
> On 06/08/2010 10:51 PM, Luke Macken wrote:
> > I recently wrote some code to generate detailed statistics of Fedora&  EPEL updates within bodhi. Eventually this will be auto-generated and exposed within bodhi itself, but for now here are the initial metrics.
> >
> > This report definitely conveys the shortcomings in our testing, however, it does show us improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process with improved Critical Path policies, which were definitely a success. With these enhanced procedures, along with the upcoming implementation of AutoQA and the new Package update acceptance criteria (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I think we'll see these numbers drastically improve in the future.
> >    

> I can't agree that this update policy is success (in any dictionary). 

If you can't agree that we, as a community, were successful in
accomplishing our intended purpose of implementing the No Frozen Rawhide
& Critical Path package processes that we drafted, then you're not
understanding the use of the word.

Again, the "update policy" that is currently rubber stamped does not
match what is enforced by the process.  The "package update acceptance
criteria"[0] has been "approved", but not yet implemented.  So, if you
feel like you have suggestions or some sort of constructive criticism to
offer, I recommend bringing it up with FESCo.

[0]: https://fedoraproject.org/wiki/Package_update_acceptance_criteria

> Since I'm forced to wait two weeks for pushing into stable

For EPEL updates, yes, that is their policy.  For Fedora, there is
nothing stopping you from pushing your non-critpath updates straight to
stable.

> Other thing is that I'm forgetting packages in bodhi, because I believed 
> until yesterday that updates will be pushed after two weeks 
> automatically. Not sure whether policy changed or it's a bug [1].

Bodhi has never done this, ever.

According to the new acceptance critera, updates will have to "spend
some minimum amount of time in updates-testing, currently one week".
Now, as to whether or not bodhi should auto-push after that week, that
I'm not quite sure.

> Karma in bodhi updates doesn't speak about quality of updates at all. At 
> least my packages which has more dependencies are pushed without 
> testing, and less important are "tested" even testers don't know what 
> does it do [2]. For example here were testers happy because it was 
> updated without any warning message, but they don't know if it's working 
> or not.

Right, karma does not equate quality.  I don't think anyone ever claimed
that it did?

> I'm looking forward to qa project that could really improve quality - 
> like testing updates, which fixed bugs with bugzilla and testers really 
> reproduce them before and not after. Otherwise is bodhi good only for 
> checking update by yum without errors.

Yes, I'm looking forward to AutoQA as well.  We all are.

The majority of karma is "this ran without exploding", which is far from
sufficient testing, but it's still valuable information.

luke