autoqa-results@ results archive disabled

[James Laska]

Greetings gang,

Just a heads up ...

It looks like autoqa-results@ hasn't been doing its exercises.  It has become
so saturated with autoqa test result emails (~ 8-13k/day), that it's slowing
down mailman on fedorahosted.org, and causing delivery issues for other lists.
After discussing with the Fedora infrastructure team, it was determined
immediate action was needed to allow other lists on fedorahosted.org to
continue receiving mail.  We performed the following ...

 1) Disabled autoqa-results@ delivery on autoqa.fedoraproject.org immediately

     I commented out result_email in /etc/autoqa/autoqa.conf

 2) Increase the autoqa.fp.org results time from 30days to 60days (we have disk
    space to allow this)

    /etc/cron.d/autoqa
    0 3 * * * autotest /usr/sbin/tmpwatch -vv --dirmtime -umc -f 1440
/usr/share/autotest/results/ -X README

This should address the problem in the short term, but adjustments may be
needed on to recognize that autoqa-results@ archive no longer available.  It
seems that the volume+size of autoqa-results@ mails increased dramatically
recently.  I'm not clear why, so that might be interesting to diagnose.  This
also helps us determine whether autoqa-results@ is still the best option for
long-term test result storage, or if there are other options worth pursuing.

Thanks,
James

[Tim Flink]

On Fri, 30 Sep 2011 15:42:13 -0400
James Laska <jlaska(a)redhat.com&gt; wrote:

>  This should address the problem in the short term, but adjustments
>  may be needed on to recognize that autoqa-results@ archive no longer
>  available.  It seems that the volume+size of autoqa-results@ mails
>  increased dramatically recently.  I'm not clear why, so that might be
>  interesting to diagnose.  This also helps us determine whether
>  autoqa-results@ is still the best option for long-term test result
>  storage, or if there are other options worth pursuing. 
I think it has to do with a change that went in when we did pretty logs.

When we started splitting off the results pages for every package in a
depcheck or upgradepath result, we started sending emails for those
packages, too.

The net result is that we started sending one email per pacakge in a
depcheck run in addition to the email for every run. Personally, I
think that the email volume has gotten a little out of hand and would
like to see it toned down a bit. I was actually having problems with my
email clients (couldn't view entire folder, couldn't delete messages
from it) when I forgot to clean out my autoqa-results folder for a
while.

I don't think that it would be too hard to patch autoqa back to its
pre-split volume if there are no objections.

Tim

[Kamil Paral]

>  > This should address the problem in the short term, but
> adjustments
>  > may be needed on to recognize that autoqa-results@ archive no
>  > longer
>  > available.  It seems that the volume+size of autoqa-results@ mails
>  > increased dramatically recently.  I'm not clear why, so that might
>  > be
>  > interesting to diagnose.  This also helps us determine whether
>  > autoqa-results@ is still the best option for long-term test result
>  > storage, or if there are other options worth pursuing.
>  
>  I think it has to do with a change that went in when we did pretty
>  logs.
>  
>  When we started splitting off the results pages for every package in
>  a
>  depcheck or upgradepath result, we started sending emails for those
>  packages, too. 
I counted the results from 28th October:
https://fedorahosted.org/pipermail/autoqa-results/20110928/thread.html

Total: 11523 emails

The number of emails per test is:
   3745 rpmguard
   3743 rpmlint
   3718 upgradepath
    280 depcheck
     12 rats_sanity
     11 repoclosure
      6 conflicts
      4 anaconda_storage
      4 rats_install

I also counted .fcNN patterns:
   6183 fc16
   3331 fc17
   1270 fc15
    537 fc14

rpmlint and rpmguard numbers are basically doubled regarding to tests actually run,
because for every test run we send a pretty log link and a full log output (2 emails).

Upgradepath and depcheck send emails for every update in the currently pending set. The
reason for the high amount of upgradepath emails is that depcheck uses "previously
accepted updates" concept and doesn't send emails for those (confirmation needed,
but I believe my guess is pretty close). Upgradepath always checks all the pending updates
and always reports about all of them (and the bodhi library ensures duplicates removal).

>  
>  The net result is that we started sending one email per pacakge in a
>  depcheck run in addition to the email for every run. Personally, I
>  think that the email volume has gotten a little out of hand and would
>  like to see it toned down a bit. I was actually having problems with
>  my
>  email clients (couldn't view entire folder, couldn't delete messages
>  from it) when I forgot to clean out my autoqa-results folder for a
>  while.
>  
>  I don't think that it would be too hard to patch autoqa back to its
>  pre-split volume if there are no objections. 
I think we have multiple (non-exclusive) options:
1. keep emails disabled and wait for resultsdb (work, josef, work!)
  - not fully optimal, we won't detect crashed tests
2. disable pretty log link sending for rpmlint+rpmguard, that will save half of their
emails
  - but please remember that in case of some mass rebuild in f16 or f17 we will be flooded
anyway
3. disable rawhide (fc17) checking for rpmling+rpmguard
  - it also saves a lot of emails and is not used much (probably?)
4. don't send per-item results for depcheck+upgradepath
  - that will save considerable fraction of emails

The options may be combined.

Ideas?

[Tim Flink]

On Tue, 04 Oct 2011 10:36:52 -0400 (EDT)
Kamil Paral <kparal(a)redhat.com&gt; wrote:


>  I counted the results from 28th October:
>  https://fedorahosted.org/pipermail/autoqa-results/20110928/thread.html
>  
>  Total: 11523 emails
>  
>  The number of emails per test is:
>     3745 rpmguard
>     3743 rpmlint
>     3718 upgradepath
>      280 depcheck
>       12 rats_sanity
>       11 repoclosure
>        6 conflicts
>        4 anaconda_storage
>        4 rats_install
>  
>  I also counted .fcNN patterns:
>     6183 fc16
>     3331 fc17
>     1270 fc15
>      537 fc14
>  
>  rpmlint and rpmguard numbers are basically doubled regarding to tests
>  actually run, because for every test run we send a pretty log link
>  and a full log output (2 emails).
>  
>  Upgradepath and depcheck send emails for every update in the
>  currently pending set. The reason for the high amount of upgradepath
>  emails is that depcheck uses "previously accepted updates" concept
>  and doesn't send emails for those (confirmation needed, but I believe
>  my guess is pretty close). Upgradepath always checks all the pending
>  updates and always reports about all of them (and the bodhi library
>  ensures duplicates removal). 
I'm glad at least one of us is approaching this in a less gut-feeling
manner. It looks like my guess was quite wrong.

>  > 
>  > The net result is that we started sending one email per pacakge in a
>  > depcheck run in addition to the email for every run. Personally, I
>  > think that the email volume has gotten a little out of hand and
>  > would like to see it toned down a bit. I was actually having
>  > problems with my
>  > email clients (couldn't view entire folder, couldn't delete messages
>  > from it) when I forgot to clean out my autoqa-results folder for a
>  > while.
>  > 
>  > I don't think that it would be too hard to patch autoqa back to its
>  > pre-split volume if there are no objections.
>  
>  I think we have multiple (non-exclusive) options:
>  1. keep emails disabled and wait for resultsdb (work, josef, work!)
>    - not fully optimal, we won't detect crashed tests 
I think that it would be awesome to finally get resultsdb up and
running. We could eventually only email on failures (in the test or
reporting to resultsdb).

>  2. disable pretty log link sending for rpmlint+rpmguard, that will
>  save half of their emails
>    - but please remember that in case of some mass rebuild in f16 or
>      f17 we will be flooded anyway 
I think that this would be a good idea. I don't think that the issue
was a one-time flood of emails but the constant flood of emails
interfering with the normal operation of mailman.

If the fedora-admin people think that a one-time flood would be a big
problem, we could always disable emails for a day or two surrounding
the rebuild - they aren't very frequent and we should have at least
some warning before hand.

>  3. disable rawhide (fc17) checking for rpmling+rpmguard
>    - it also saves a lot of emails and is not used much (probably?) 
I don't think this is needed all that much if we're disabling emails
for rpmguard+rpmlint.

>  4. don't send per-item results for depcheck+upgradepath
>    - that will save considerable fraction of emails 
This also sounds like a good idea. Another option would be to tweak the
email settings such that we only send the per-item results to emails
that have been opted-in and only send the job email to autoqa-results@

>  The options may be combined.
>  
>  Ideas? 
I think that you about have it covered. I guess that my vote would be
to do the following:

1. resultsdb ASAP!
2. Disable emails for rpmlint+rpmguard (leave on failure, if possible)
3. Disable emails of per-item results to autoqa-results@ (allow for
   opt-in if possible)

Any other votes?

Tim

[Kamil Paral]

>  I think that you about have it covered. I guess that my vote would
> be
>  to do the following:
>  
>  1. resultsdb ASAP!
>  2. Disable emails for rpmlint+rpmguard (leave on failure, if
>  possible)
>  3. Disable emails of per-item results to autoqa-results@ (allow for
>     opt-in if possible)
>  
>  Any other votes? 
I've looked into the code. I don't see a clean and elegant solution how to disable
all emails for some tests (rpmlint, rpmguard) but keep full.log emails enabled for other
tests (upgradepath, depcheck). Of course we can hard-code it, but then we have to also
change our configuration files like this one:

# Comma-separated list of email addresses that will receive full test results
# of every completed test.
# (leave blank to disable)
result_email =

If we changed the code but left the configuration intact it would be very confusing for
anyone using/exploring AutoQA.

To do it cleanly, I see two major solutions:

1. Send only one email per test (but for every test). We can either send full.log or
pretty log created from the main TestDetail. The second option is probably better, because
the email is much smaller (we send just the header and a hyperlink).

On October 28th this would reduce the number of results from 11523 emails to ~ 4341 emails
(and considerably reduced email size, I suppose by >80%).

2. Send emails only for crashed tests. We will have to adjust "result_email"
option to correctly state it concerns only crashed tests. Alternatively we can introduce a
new option "send_only_crashed" and set it to True on production server.


I would go for the first option if the decreased load (of 3-4 thousand emails per day) is
acceptable by Fedora's mail server. If it is not, let's go for the second option.

Please note this does not concern opt-in emails at all. Opt-in emails do not cause any
problems and it should be possible to keep their behavior intact.

Thoughts?

[Kamil Paral]

>  I've looked into the code. I don't see a clean and elegant
> solution
>  how to disable all emails for some tests (rpmlint, rpmguard) but
>  keep full.log emails enabled for other tests (upgradepath,
>  depcheck). Of course we can hard-code it, but then we have to also
>  change our configuration files like this one:
>  
>  # Comma-separated list of email addresses that will receive full test
>  results
>  # of every completed test.
>  # (leave blank to disable)
>  result_email =
>  
>  If we changed the code but left the configuration intact it would be
>  very confusing for anyone using/exploring AutoQA.
>  
>  To do it cleanly, I see two major solutions:
>  
>  1. Send only one email per test (but for every test). We can either
>  send full.log or pretty log created from the main TestDetail. The
>  second option is probably better, because the email is much smaller
>  (we send just the header and a hyperlink).
>  
>  On October 28th this would reduce the number of results from 11523
>  emails to ~ 4341 emails (and considerably reduced email size, I
>  suppose by >80%).
>  
>  2. Send emails only for crashed tests. We will have to adjust
>  "result_email" option to correctly state it concerns only crashed
>  tests. Alternatively we can introduce a new option
>  "send_only_crashed" and set it to True on production server.
>  
>  
>  I would go for the first option if the decreased load (of 3-4
>  thousand emails per day) is acceptable by Fedora's mail server. If
>  it is not, let's go for the second option.
>  
>  Please note this does not concern opt-in emails at all. Opt-in emails
>  do not cause any problems and it should be possible to keep their
>  behavior intact.
>  
>  Thoughts? 
By the way, when going through my bookmarks I have found a long forgotten link:
http://autoqa.fedoraproject.org/tko/compose_query.cgi?columns=test&ro...

This is also an easy way how to spot crashed tests without autoqa-results archive. It
doesn't inform us about NEEDS_INSPECTION results though, just CRASHED.

[Kamil Paral]

>  I counted the results from 28th October:
>  https://fedorahosted.org/pipermail/autoqa-results/20110928/thread.html
>  
>  Total: 11523 emails
>  
>  The number of emails per test is:
>     3745 rpmguard
>     3743 rpmlint
>     3718 upgradepath
>      280 depcheck
>       12 rats_sanity
>       11 repoclosure
>        6 conflicts
>        4 anaconda_storage
>        4 rats_install
>  
>  I also counted .fcNN patterns:
>     6183 fc16
>     3331 fc17
>     1270 fc15
>      537 fc14 
I created more statistics today for the last 4 days. The first set of numbers denote the
number of tests executed. The second set of numbers denote the number of results (pretty
logs) created. If you count up both numbers for a particular test you'll receive the
number of emails we would have sent (because we used to send both pretty logs and full
logs).

Dec 20:

    161 rpmlint
    161 rpmguard
    117 depcheck
     30 upgradepath
      2 repoclosure
      2 rats_sanity
      2 conflicts

    681 upgradepath
    251 depcheck
    161 rpmlint
    161 rpmguard
      2 repoclosure
      2 rats_sanity
      2 conflicts

Dec 19:

    167 rpmlint
    166 rpmguard
    126 depcheck
     34 upgradepath
      2 repoclosure
      2 rats_sanity
      2 conflicts

    552 upgradepath
    185 depcheck
    167 rpmlint
    166 rpmguard
      2 repoclosure
      2 rats_sanity
      2 conflicts


Dec 18:

     65 rpmlint
     65 rpmguard
     59 depcheck
     15 upgradepath
     10 repoclosure
     10 rats_sanity
     10 conflicts

    180 upgradepath
     83 depcheck
     65 rpmlint
     65 rpmguard
     10 repoclosure
     10 rats_sanity
     10 conflicts


Dec 17:

     62 rpmlint
     62 rpmguard
     36 depcheck
     10 upgradepath
     10 repoclosure
     10 rats_sanity
     10 conflicts

     78 upgradepath
     62 rpmlint
     62 rpmguard
     54 depcheck
     10 repoclosure
     10 rats_sanity
     10 conflicts


After consulting with Josef I'll try to modify AutoQA to send emails only for full
logs (i.e. one for every test execution). That should be fairly easy and the total number
of emails for Dec 20 would be then around 450, which I find acceptable. This change will
not affect opt-in emails at all.

----

For future reference I used these commands:
test execution count:
(for DIR in {259531..259730}-autotest/*/*/results/; do echo $DIR | cut -d / -f3; done) |
sort | uniq -c | sort -nr
test results count:
(for DIR in {259531..259730}-autotest/*/*/results/; do TEST=`echo $DIR | cut -d / -f3`; if
[ "$TEST" == "*" ]; then continue; fi; for RESULT in `echo
$DIR/*.html`; do echo $TEST; done; done;) | sort | uniq -c | sort -nr