stale tags in koji

[Tim Flink]

All,

I happened to be going through the upgradepath results for F15 today and
I'm seeing a lot of builds with stale tags (~50 so far and I'm not
done). If we re-enable comments before these builds are taken care of,
we're going to be adding a lot more comments to old builds again.

This is why spot-checking (like I did before) isn't enough, I suppose.

Anyhow, I sent an email to lmacken with my initial list and I'm planning
to write a script to check for stale tags tomorrow.

Tim

[Kamil Paral]

>  All,
>  
>  I happened to be going through the upgradepath results for F15 today
>  and
>  I'm seeing a lot of builds with stale tags (~50 so far and I'm not
>  done). If we re-enable comments before these builds are taken care of,
>  we're going to be adding a lot more comments to old builds again. 
These are bad news. If lmacken doesn't take care of that soon, I'm afraid we will
have to postpone re-enabling of bodhi comments. The other approach is to write an
explanation to the announcement, explain how to recognize stale koji tag from some autoqa
bug and ask people to ignore comments for stable builds for now.

What do you think?

[Tim Flink]

On 04/08/2011 01:01 AM, Kamil Paral wrote:
> > All,
> > 
> > I happened to be going through the upgradepath results for F15
> > today and I'm seeing a lot of builds with stale tags (~50 so far
> > and I'm not done). If we re-enable comments before these builds are
> > taken care of, we're going to be adding a lot more comments to old
> > builds again.
>  
>  These are bad news. If lmacken doesn't take care of that soon, I'm
>  afraid we will have to postpone re-enabling of bodhi comments. The
>  other approach is to write an explanation to the announcement,
>  explain how to recognize stale koji tag from some autoqa bug and ask
>  people to ignore comments for stable builds for now.
>  
>  What do you think? 
I guess that I'm for a bit of both, more of the waiting as long as
reasonable to make sure that the tags are taken care of.

Obviously, I'd rather see the stale tags taken care of before we enable
bodhi comments again. Maybe the Fedora community is better about this,
but the thought process I've seen elsewhere is like "Why is AutoQA
commenting this old build? It must be a broken piece of SW, I'm not sure
that we can trust it" regardless of whether or not an explanation is
sent over email.

If we can keep the signal/noise ratio up as high as possible, that would
be awesome.

On the bright side, all of the builds that I've found so far are old
(most were pushed to stable 2011-03-06 to 2011-03-08) so I'm pretty sure
that the root cause has been taken care of in bodhi and koji. We just
need to make sure that everything was cleaned up.

For now, let's see what lmacken says and hold off on any decision to
enable or not enable comments until Monday. I assume that we'll hear
back from him today at some point and he has been very responsive to
this kind of stuff so far.

Tim

[Tim Flink]

On 04/08/2011 08:07 AM, Tim Flink wrote:
>  On 04/08/2011 01:01 AM, Kamil Paral wrote:
> >> All,
> >>
> >> I happened to be going through the upgradepath results for F15
> >> today and I'm seeing a lot of builds with stale tags (~50 so far
> >> and I'm not done). If we re-enable comments before these builds are
> >> taken care of, we're going to be adding a lot more comments to old
> >> builds again.
> >
> > These are bad news. If lmacken doesn't take care of that soon, I'm
> > afraid we will have to postpone re-enabling of bodhi comments. The
> > other approach is to write an explanation to the announcement,
> > explain how to recognize stale koji tag from some autoqa bug and ask
> > people to ignore comments for stable builds for now.
> >
> > What do you think?
>  
>  I guess that I'm for a bit of both, more of the waiting as long as
>  reasonable to make sure that the tags are taken care of.
>  
>  Obviously, I'd rather see the stale tags taken care of before we enable
>  bodhi comments again. Maybe the Fedora community is better about this,
>  but the thought process I've seen elsewhere is like "Why is AutoQA
>  commenting this old build? It must be a broken piece of SW, I'm not sure
>  that we can trust it" regardless of whether or not an explanation is
>  sent over email.
>  
>  If we can keep the signal/noise ratio up as high as possible, that would
>  be awesome.
>  
>  On the bright side, all of the builds that I've found so far are old
>  (most were pushed to stable 2011-03-06 to 2011-03-08) so I'm pretty sure
>  that the root cause has been taken care of in bodhi and koji. We just
>  need to make sure that everything was cleaned up.
>  
>  For now, let's see what lmacken says and hold off on any decision to
>  enable or not enable comments until Monday. I assume that we'll hear
>  back from him today at some point and he has been very responsive to
>  this kind of stuff so far.
>  
>  Tim 
As an update, I talked to lmacken today and he's looking into the stale
koji tags even though he's on PTO today.

Looking at the upgradecheck results, I don't think that all the tags
have been fixed yet but I will send out more information when I have it.

If anyone is interested in what I was running, the results and crappy
script I was running are up at http://tflink.fedorapeople.org/tag_reports/

When I last ran it, there were 106 builds with stale tags that would be
commented improperly if we re-enable bodhi comments.

Tim

[Luke Macken]

On Fri, Apr 08, 2011 at 04:17:59PM -0600, Tim Flink wrote:
>  On 04/08/2011 08:07 AM, Tim Flink wrote:
>  > On 04/08/2011 01:01 AM, Kamil Paral wrote:
>  >>> All,
>  >>>
>  >>> I happened to be going through the upgradepath results for F15
>  >>> today and I'm seeing a lot of builds with stale tags (~50 so far
>  >>> and I'm not done). If we re-enable comments before these builds are
>  >>> taken care of, we're going to be adding a lot more comments to old
>  >>> builds again.
>  >>
>  >> These are bad news. If lmacken doesn't take care of that soon, I'm
>  >> afraid we will have to postpone re-enabling of bodhi comments. The
>  >> other approach is to write an explanation to the announcement,
>  >> explain how to recognize stale koji tag from some autoqa bug and ask
>  >> people to ignore comments for stable builds for now.
>  >>
>  >> What do you think?
>  > 
>  > I guess that I'm for a bit of both, more of the waiting as long as
>  > reasonable to make sure that the tags are taken care of.
>  > 
>  > Obviously, I'd rather see the stale tags taken care of before we enable
>  > bodhi comments again. Maybe the Fedora community is better about this,
>  > but the thought process I've seen elsewhere is like "Why is AutoQA
>  > commenting this old build? It must be a broken piece of SW, I'm not sure
>  > that we can trust it" regardless of whether or not an explanation is
>  > sent over email.
>  > 
>  > If we can keep the signal/noise ratio up as high as possible, that would
>  > be awesome.
>  > 
>  > On the bright side, all of the builds that I've found so far are old
>  > (most were pushed to stable 2011-03-06 to 2011-03-08) so I'm pretty sure
>  > that the root cause has been taken care of in bodhi and koji. We just
>  > need to make sure that everything was cleaned up.
>  > 
>  > For now, let's see what lmacken says and hold off on any decision to
>  > enable or not enable comments until Monday. I assume that we'll hear
>  > back from him today at some point and he has been very responsive to
>  > this kind of stuff so far.
>  > 
>  > Tim
>  
>  As an update, I talked to lmacken today and he's looking into the stale
>  koji tags even though he's on PTO today. 
So, I'm pretty sure the last bodhi update fixed the underlying tagging
issues, but there are still many builds that are stable with unnecessary
testing/pending/candidate tags. The tag checking script that I wrote in
bodhi queries koji for the 'latest' packages in each tag, as opposed to
everything with that tag, and thus many went unnoticed. I updated the
script and am running it now to clean up the rest.

luke

[Luke Macken]

On Fri, Apr 08, 2011 at 06:25:47PM -0400, Luke Macken wrote:
>  On Fri, Apr 08, 2011 at 04:17:59PM -0600, Tim Flink wrote:
>  > On 04/08/2011 08:07 AM, Tim Flink wrote:
>  > > On 04/08/2011 01:01 AM, Kamil Paral wrote:
>  > >>> All,
>  > >>>
>  > >>> I happened to be going through the upgradepath results for F15
>  > >>> today and I'm seeing a lot of builds with stale tags (~50 so far
>  > >>> and I'm not done). If we re-enable comments before these builds
> are
>  > >>> taken care of, we're going to be adding a lot more comments to old
>  > >>> builds again.
>  > >>
>  > >> These are bad news. If lmacken doesn't take care of that soon, I'm
>  > >> afraid we will have to postpone re-enabling of bodhi comments. The
>  > >> other approach is to write an explanation to the announcement,
>  > >> explain how to recognize stale koji tag from some autoqa bug and ask
>  > >> people to ignore comments for stable builds for now.
>  > >>
>  > >> What do you think?
>  > > 
>  > > I guess that I'm for a bit of both, more of the waiting as long as
>  > > reasonable to make sure that the tags are taken care of.
>  > > 
>  > > Obviously, I'd rather see the stale tags taken care of before we enable
>  > > bodhi comments again. Maybe the Fedora community is better about this,
>  > > but the thought process I've seen elsewhere is like "Why is AutoQA
>  > > commenting this old build? It must be a broken piece of SW, I'm not sure
>  > > that we can trust it" regardless of whether or not an explanation is
>  > > sent over email.
>  > > 
>  > > If we can keep the signal/noise ratio up as high as possible, that would
>  > > be awesome.
>  > > 
>  > > On the bright side, all of the builds that I've found so far are old
>  > > (most were pushed to stable 2011-03-06 to 2011-03-08) so I'm pretty sure
>  > > that the root cause has been taken care of in bodhi and koji. We just
>  > > need to make sure that everything was cleaned up.
>  > > 
>  > > For now, let's see what lmacken says and hold off on any decision to
>  > > enable or not enable comments until Monday. I assume that we'll hear
>  > > back from him today at some point and he has been very responsive to
>  > > this kind of stuff so far.
>  > > 
>  > > Tim
>  > 
>  > As an update, I talked to lmacken today and he's looking into the stale
>  > koji tags even though he's on PTO today.
>  
>  So, I'm pretty sure the last bodhi update fixed the underlying tagging
>  issues, but there are still many builds that are stable with unnecessary
>  testing/pending/candidate tags. The tag checking script that I wrote in
>  bodhi queries koji for the 'latest' packages in each tag, as opposed to
>  everything with that tag, and thus many went unnoticed. I updated the
>  script and am running it now to clean up the rest. 
Ok, all of the stray tags in your list have been cleaned up.

I updated bodhi's tagcheck script to find all stale tags, and also to
utilize koji multiCalls to drastically speed things up.  I ran your
script again, and it doesn't complain about any stale tags, and the
'missing tags' are all false positives.

luke

[Tim Flink]

On 04/10/2011 05:20 PM, Luke Macken wrote:
>  On Fri, Apr 08, 2011 at 06:25:47PM -0400, Luke Macken wrote:
> > On Fri, Apr 08, 2011 at 04:17:59PM -0600, Tim Flink wrote:
> >> On 04/08/2011 08:07 AM, Tim Flink wrote:
> >>> On 04/08/2011 01:01 AM, Kamil Paral wrote:
> >>>>> All,
> >>>>>
> >>>>> I happened to be going through the upgradepath results for F15
> >>>>> today and I'm seeing a lot of builds with stale tags (~50 so far
> >>>>> and I'm not done). If we re-enable comments before these builds
> are
> >>>>> taken care of, we're going to be adding a lot more comments to
> old
> >>>>> builds again.
> >>>>
> >>>> These are bad news. If lmacken doesn't take care of that soon,
> I'm
> >>>> afraid we will have to postpone re-enabling of bodhi comments. The
> >>>> other approach is to write an explanation to the announcement,
> >>>> explain how to recognize stale koji tag from some autoqa bug and ask
> >>>> people to ignore comments for stable builds for now.
> >>>>
> >>>> What do you think?
> >>>
> >>> I guess that I'm for a bit of both, more of the waiting as long as
> >>> reasonable to make sure that the tags are taken care of.
> >>>
> >>> Obviously, I'd rather see the stale tags taken care of before we enable
> >>> bodhi comments again. Maybe the Fedora community is better about this,
> >>> but the thought process I've seen elsewhere is like "Why is AutoQA
> >>> commenting this old build? It must be a broken piece of SW, I'm not sure
> >>> that we can trust it" regardless of whether or not an explanation is
> >>> sent over email.
> >>>
> >>> If we can keep the signal/noise ratio up as high as possible, that would
> >>> be awesome.
> >>>
> >>> On the bright side, all of the builds that I've found so far are old
> >>> (most were pushed to stable 2011-03-06 to 2011-03-08) so I'm pretty sure
> >>> that the root cause has been taken care of in bodhi and koji. We just
> >>> need to make sure that everything was cleaned up.
> >>>
> >>> For now, let's see what lmacken says and hold off on any decision to
> >>> enable or not enable comments until Monday. I assume that we'll hear
> >>> back from him today at some point and he has been very responsive to
> >>> this kind of stuff so far.
> >>>
> >>> Tim
> >>
> >> As an update, I talked to lmacken today and he's looking into the stale
> >> koji tags even though he's on PTO today.
> >
> > So, I'm pretty sure the last bodhi update fixed the underlying tagging
> > issues, but there are still many builds that are stable with unnecessary
> > testing/pending/candidate tags. The tag checking script that I wrote in
> > bodhi queries koji for the 'latest' packages in each tag, as opposed to
> > everything with that tag, and thus many went unnoticed. I updated the
> > script and am running it now to clean up the rest.
>  
>  Ok, all of the stray tags in your list have been cleaned up.
>  
>  I updated bodhi's tagcheck script to find all stale tags, and also to
>  utilize koji multiCalls to drastically speed things up.  I ran your
>  script again, and it doesn't complain about any stale tags, and the
>  'missing tags' are all false positives.
>  
>  luke 
Thanks for taking care of that. We really appreciate it on such short
notice and especially when you were on PTO.

Looking through some of the upgradepath results, things are looking much
better to me.

Yeah, I wasn't sure if those were really missing tags or not but figured
that it was better to include information that might need to be ignored
than to leave it out.

I'll have to look into multiCalls, though. It feels like the koji calls
in autoqa take way too long when I'm running them locally. Or maybe I'm
just impatient :)

Thanks again,

Tim