kernel component on bugzilla and call for "old bugs hunting season" to pen

[Tomasz Kłoczko]

Hi,

In last two years, I've created something between 10 to 20 tickets against
the kernel.
Today after reboot on latest rc9 I found OOPS in dmesg and as usually, my
first thought was "raise the ticket". Just after this was second one ..
"but none of my kernel tickets has been replied or even closed as DUP!!".

So I've done some search on Bugzilla for kernel tickets in NEW state.

https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&component=kern...

Above shows list of current kernel tickets in NEW state with older tickets
on top.
There are 691 such tickets and older is from 2009.

It would be good to clean this mess because in current state opening any
new tickets do not make to much sense.

I'm asking for helping anyone who has Bugzilla account.
It would be good to start "bugs hunting season" to close as much as
possible already outdated tickets.

All Fedora NEW tickets sorted from oldest as first is possible to display
by going to URL:

https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&list_id=834639...

There is now not closed and in NEW state *14474* such tickets.


1) Probably it would be good start posting one time a week summary stats
about Bugzilla tickets. Any volunteer to create such weekly report?

2) Q: do we need new kernel package maintainer or secondary co maintainers?

3) can someone crate stats highest flow of the tickets in Bugzilla per
package to find a help for some packages maintainers?

kloczek
PS. Please do not send comments why it happened or who can be blamed
personally for this.
-- 
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH

[Justin Forbes]

On Fri, Jan 26, 2018 at 4:03 PM, Tomasz Kłoczko
<kloczko.tomasz(a)gmail.com&gt; wrote:
>  Hi,
>
>  In last two years, I've created something between 10 to 20 tickets against
>  the kernel.
>  Today after reboot on latest rc9 I found OOPS in dmesg and as usually, my
>  first thought was "raise the ticket". Just after this was second one ..
> "but
>  none of my kernel tickets has been replied or even closed as DUP!!". 
Triage is a problem, we have pointed that out several times (including
flock last year). While I can guarantee that every bug filed against
the kernel is looked at by a maintainer, we typically do not take the
time to comment/reassign/etc.  There aren't many of us, and a lot to
do.  I read all of the bugs as they come in via email.  I am not
entirely sure about the workflow of jcline and labbott, but I would
expect it is similar.
We do typically send out a mass needinfo on kernel bugs shortly after
a rebase, which helps to close out bugs which have been fixed, get
more information on bugs that are still occurring, and after a period
of time. close out bugs where the reporter is unresponsive.  I did not
do this for the 4.14 rebases as it slipped my mind between vacation
and holidays. I expect Laura or Jeremy will do so after the 4.15
rebases.

>  So I've done some search on Bugzilla for kernel tickets in NEW state.
>
>
> https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&component=kern...
>
>  Above shows list of current kernel tickets in NEW state with older tickets
>  on top.
>  There are 691 such tickets and older is from 2009.
>
>  It would be good to clean this mess because in current state opening any new
>  tickets do not make to much sense.
>
>  I'm asking for helping anyone who has Bugzilla account.
>  It would be good to start "bugs hunting season" to close as much as possible
>  already outdated tickets.
>
>  All Fedora NEW tickets sorted from oldest as first is possible to display by
>  going to URL:
>
>
> https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&list_id=834639...
>
>  There is now not closed and in NEW state 14474 such tickets.
>
>
>  1) Probably it would be good start posting one time a week summary stats
>  about Bugzilla tickets. Any volunteer to create such weekly report? 
                26       27        rawhide
open:       243     191      288            (722)

Since 2017-12-27
opened:   7         69        15              (91)
closed:     7         36        13              (56)


This is the stats for the past 30 days, which are fairly bad because
1)  a *lot* of time has been taken to deal with Spectre/Meltdown and
2) Most kernel folks weren't working for the holidays, and half the
team was travelling last week. I will be happy to start sending out
such reports to the kernel list monthly.

>  2) Q: do we need new kernel package maintainer or secondary co maintainers? 
We currently have 2.5 maintainers, and I believe the .5 is ramping up
to a full time 3.   We have asked several times over the years for
people to help with triage, and even have a lovely step by step guide:
https://fedoraproject.org/wiki/KernelBugTriage Very few people seem
interested.

> 3) can someone crate stats highest flow of the tickets in Bugzilla per
> package to find a help for some packages maintainers?

[Tomasz Kłoczko]

On 27 January 2018 at 21:50, Justin Forbes <jmforbes(a)linuxtx.org&gt; wrote:
[..]

>  > 1) Probably it would be good start posting one time a week
> summary stats
>  > about Bugzilla tickets. Any volunteer to create such weekly report?
>
>                  26       27        rawhide
>  open:       243     191      288            (722)
>
>  Since 2017-12-27
>  opened:   7         69        15              (91)
>  closed:     7         36        13              (56)
>
>
>  This is the stats for the past 30 days, which are fairly bad because
>  1)  a *lot* of time has been taken to deal with Spectre/Meltdown and
>  2) Most kernel folks weren't working for the holidays, and half the
>  team was travelling last week. I will be happy to start sending out
>  such reports to the kernel list monthly.
>  
That is OK.
Problem is with very long tail of other still opened tickets.
Again: some of the tickets are in NEW state from 2009.

In last two days I've closed about 10-15 tickets which I found that
definitely are 100% outdated.
Every package maintainer can at least spent 1-15 min a month to try have
look is anything on the maintained packages list can be closed or not.

Among those tickets I found at least one class which should be not opened
against Fedora but exacts project which is producing at the end tar ball
used in Fedora package.
This class it is various RFE (Request For Enhance) tickets which none or
almost none of the Fedora packagers can implement.

Next: I think that it would be good to have obligatory rule that even if
ticket was opened longer 3 to 5 years such ticket should be automatically
closed,

Next: as I wrote I found tickets which are opened against packages which
are no lo longer part of the Fedora.
Someone who can make query against Bugzilla SQL backend should try to find
all those tickets and close them all.
Those packets should be removed from list of Fefora components as well.

One example I've mention already and it is ghdcpd ticket:
https://bugzilla.redhat.com/show_bug.cgi?id=502717
Second one is fr example xorg-x11-fonts:
https://bugzilla.redhat.com/show_bug.cgi?id=477486

I'm sure that list of such packets/components and tickets is way longer.
It is possible to close those tickets by compare with list of still
maintained Fedora packages.
This could be done by Biugzilla admin and cannot be done that way by anyone
else.

>  2) Q: do we need new kernel package maintainer or secondary co
>  maintainers?
>
>  We currently have 2.5 maintainers, and I believe the .5 is ramping up
>  to a full time 3.   We have asked several times over the years for
>  people to help with triage, and even have a lovely step by step guide:
>  https://fedoraproject.org/wiki/KernelBugTriage Very few people seem
>  interested.
>  
So I was simple unlucky (?) that in +1 years none of my +10 tickets with
details about versions and call traces NEVER had even single short msg
"We know about the issue. Closing ad duplicated ticket to <num>"?
Kernel is fast moving package end even within few days tickets with some
reported OOPSes is possible to close. Why it does not happen?
Is someone has time to update kernel package why the same person has no
time to close few recent tickers wit c&p comment:

"New kernel release is in rawhide repo.
Closing ticket.
Please reopen if issue still is present"

?
Is it really so hard?
After passing +100 still opened tickets per package I can understand that
no one wants to even look on new tickets.
This as result does not make any sense opening new tickets. Isn't it?
If current kernel (secondary) maintainers don't want to work on Bugzilla
tickets (again) maybe it would be better to find new co-maintainers or add
more co-maintainers?
Kernel package is very specific so treating a little differently should be
no big deal ..

kloczek
-- 
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH

[Josh Boyer]

On Sun, Jan 28, 2018 at 1:09 PM, Tomasz Kłoczko
<kloczko.tomasz(a)gmail.com&gt; wrote:
>  On 27 January 2018 at 21:50, Justin Forbes
> <jmforbes(a)linuxtx.org&gt; wrote:
>  [..]
> >
> > > 1) Probably it would be good start posting one time a week summary stats
> > > about Bugzilla tickets. Any volunteer to create such weekly report?
> >
> >                 26       27        rawhide
> > open:       243     191      288            (722)
> >
> > Since 2017-12-27
> > opened:   7         69        15              (91)
> > closed:     7         36        13              (56)
> >
> >
> > This is the stats for the past 30 days, which are fairly bad because
> > 1)  a *lot* of time has been taken to deal with Spectre/Meltdown and
> > 2) Most kernel folks weren't working for the holidays, and half the
> > team was travelling last week. I will be happy to start sending out
> > such reports to the kernel list monthly.
>
>
>  That is OK.
>  Problem is with very long tail of other still opened tickets.
>  Again: some of the tickets are in NEW state from 2009.
>
>  In last two days I've closed about 10-15 tickets which I found that
>  definitely are 100% outdated.
>  Every package maintainer can at least spent 1-15 min a month to try have
>  look is anything on the maintained packages list can be closed or not.
>
>  Among those tickets I found at least one class which should be not opened
>  against Fedora but exacts project which is producing at the end tar ball
>  used in Fedora package.
>  This class it is various RFE (Request For Enhance) tickets which none or
>  almost none of the Fedora packagers can implement.
>
>  Next: I think that it would be good to have obligatory rule that even if
>  ticket was opened longer 3 to 5 years such ticket should be automatically
>  closed,
>
>  Next: as I wrote I found tickets which are opened against packages which are
>  no lo longer part of the Fedora.
>  Someone who can make query against Bugzilla SQL backend should try to find
>  all those tickets and close them all.
>  Those packets should be removed from list of Fefora components as well.
>
>  One example I've mention already and it is ghdcpd ticket:
>  https://bugzilla.redhat.com/show_bug.cgi?id=502717
>  Second one is fr example xorg-x11-fonts:
>  https://bugzilla.redhat.com/show_bug.cgi?id=477486
>
>  I'm sure that list of such packets/components and tickets is way longer.
>  It is possible to close those tickets by compare with list of still
>  maintained Fedora packages.
>  This could be done by Biugzilla admin and cannot be done that way by anyone
>  else.
>
> > > 2) Q: do we need new kernel package maintainer or secondary co
> > > maintainers?
> >
> > We currently have 2.5 maintainers, and I believe the .5 is ramping up
> > to a full time 3.   We have asked several times over the years for
> > people to help with triage, and even have a lovely step by step guide:
> > https://fedoraproject.org/wiki/KernelBugTriage Very few people seem
> > interested.
>
>
>  So I was simple unlucky (?) that in +1 years none of my +10 tickets with
>  details about versions and call traces NEVER had even single short msg
>  "We know about the issue. Closing ad duplicated ticket to <num>"?
>  Kernel is fast moving package end even within few days tickets with some
>  reported OOPSes is possible to close. Why it does not happen?
>  Is someone has time to update kernel package why the same person has no time
>  to close few recent tickers wit c&p comment:
>
>  "New kernel release is in rawhide repo.
>  Closing ticket.
>  Please reopen if issue still is present"
>
>  ?
>  Is it really so hard? 
Yes.  Mostly because there's no guarantee the newer kernel fixes the
issue and it's really not known.  So then you get into a cycle of
CLOSED->REOPENED->CLOSED->REOPENED *every day* and that's a waste of
time and a horrible user experience.

>  After passing +100 still opened tickets per package I can understand
> that no
>  one wants to even look on new tickets. 
That's not the case for the kernel.

>  This as result does not make any sense opening new tickets. Isn't
> it? 
We need the new tickets.  The large number of tickets helps spot
patterns that would otherwise not be obvious given the kernel's
problem space.  If one user reports an issue, it might just be their
machine.  This happens frequently.  If 30 users report the same (or
very similar) issue, it's much more likely there is a bug we can hone
in on and fix.  We need the aggregate data.

>  If current kernel (secondary) maintainers don't want to work on
> Bugzilla
>  tickets (again) maybe it would be better to find new co-maintainers or add
>  more co-maintainers? 
There's nothing preventing anyone from doing this today.

josh