4:10 p.m.
Since this has almost nothing to do with the log output of depcheck
upgradepath, I'm splitting this conversation off into another thread.
As Kamil pointed out, my initial proposal was naive and needs more
thought before implementing anything. We will soon have the ability to
disable emails for comments in Bodhi and we need to figure out how to
best make use of that ability.
Lots of questions here, I'll save my thoughts for replying to myself.
Tim
How would we deal with tests that change state between PASS and FAIL?
- Either 'PASS -> FAIL' or 'FAIL -> PASS'
What emails do we want to get rid of?
- Just certain 'PASS' messages?
- All results messages?
How do we expect maintainers to learn of AutoQA results if we turn off
email notifications?
Can we do this in a way such that it is configurable by maintainer?
- If so, should we attempt this?
- What is the benefit? How big is it?
Can we just send a single email after all tests have passed?
- Again, if so, should we?
Is there another approach that would be better?
5:10 p.m.
On 04/29/2011 03:10 PM, Tim Flink wrote:
How would we deal with tests that change state between PASS and
FAIL?
- Either 'PASS -> FAIL' or 'FAIL -> PASS'
I think that both of these cases are important and I'm not sure how to
do them without at least some parsing and comprehension of update
comments as a whole.
This could get really complicated, really quick but I have some ideas
based on parsing comments, creating sequences of test result order and
making more holistic decisions.
I tried putting my thoughts into words, but it wasn't working. I'm not
convinced that my idea here is actually going to work with a reasonable
amount of work.
I'll see if I can spike something together for a rough concept. It might
not get done today, though.
What emails do we want to get rid of?
- Just certain 'PASS' messages?
- All results messages?
I hadn't put a whole lot of thought into the details of this, but my
initial thought on the most important messages we could send would be:
- Update PASS/FAIL status transition
- Package PASS/FAIL status transition
For now, I'm wondering if package state transition would be good enough.
How do we expect maintainers to learn of AutoQA results if we turn
off
email notifications?
I can't really think of anything here. I think that some pestering is a
good thing, just not too much and that turning off emails completely
would leave us with not enough pestering.
Can we do this in a way such that it is configurable by maintainer?
- If so, should we attempt this?
- What is the benefit? How big is it?
This is a hard one. I'm sure that we could do it but I'm more interested
in the should (at least for the near future).
It would be a decent amount of work and require some new concepts in
AutoQA - interfacing with FAS (at least indirectly), defining various
knobs to turn for update emails and looking that up before or during
commenting.
I'm thinking that this would be a cool feature but unless we start
getting requests for it, I'd vote for not now.
Can we just send a single email after all tests have passed?
- Again, if so, should we?
Is there a reasonable way that we could do this without ResultsDB? Sure,
we could code up some stuff to determine what packages are supposed to
pass what tests, but this seems to be quite a bit of overlap with PUATP
and ResultsDB to me.
I'm inclined to say wait on ResultsDB for this one. I think that it
might be a good idea to continue to use Bodhi comments to notify
maintainers of update test status after we implement ResultsDB but I
can't think of an easy way to do this until then.
Is there another approach that would be better?
I'm probably the wrong person to answer this question but I'm interested
in hearing what the alternatives would be.
I like this approach because it gives us the ability to determine when
emails are getting sent. Unless that functionality is already in bodhi
or elsewhere, I'm not sure it makes a whole lot of sense to do the
filtering outside AutoQA.
10:54 a.m.
> Can we just send a single email after all tests have passed?
> - Again, if so, should we?
Is there a reasonable way that we could do this without ResultsDB?
Sure,
we could code up some stuff to determine what packages are supposed to
pass what tests, but this seems to be quite a bit of overlap with
PUATP
and ResultsDB to me.
From what I have understood we're doing 0.5.0 exactly because we
don't have ResultDB (which will take a while) and we need some reasonable workaround
till that time. Of course some of that work will be nicely usable even for ResultDB (like
splitting one bug test output into multiple per-item logs). But some stuff is just
temporary fix. Therefore I wouldn't really be afraid of a few unclean hacks that will
be removed once we have ResultDB.
So I thought about this. Could we send an email only after all tests have finished? That
would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded definitions.
We know that we currently execute only two tests, depcheck and upgradepath. Upgradepath
runs always on a noarch arch. Depcheck runs on i386 and x86_64 in 99,9% [1].
So our code may default to email=False and change to email=True only when all tests have
been executed (counting also the current one that is to be sent). That alone decreases the
email load by 66%.
Let's go further. If all tests are completed and PASSED, I wouldn't send an email.
If everything went fine, why bother the maintainer?
Let's go further. Currently we re-send FAILED results after at least 3 days from the
first failure. I wouldn't send email notifications for them. The purpose is to provide
a fresh failure log, not the notification (the maintainer already knows about it).
Then there remains the question of PASS->FAIL and FAIL->PASS transitions. I would
probably send notifications for them. They seem rather important.
What do you think?
[1] I've examined F14 repository, it contains about 16800 packages. About 20 packages
is 32bit only:
http://fpaste.org/BRwA/
and 4 packages is 64bit only:
http://fpaste.org/WVBO/
I'm willing to sacrifice that, for now. We have much higher test crash ratio anyway.
12:34 p.m.
On 05/02/2011 09:54 AM, Kamil Paral wrote:
>> Can we just send a single email after all tests have passed?
-
>> Again, if so, should we?
>
> Is there a reasonable way that we could do this without ResultsDB?
> Sure, we could code up some stuff to determine what packages are
> supposed to pass what tests, but this seems to be quite a bit of
> overlap with PUATP and ResultsDB to me.
From what I have understood we're doing 0.5.0 exactly because we
don't have ResultDB (which will take a while) and we need some
reasonable workaround till that time. Of course some of that work
will be nicely usable even for ResultDB (like splitting one bug
test output into multiple per-item logs). But some stuff is just
temporary fix. Therefore I wouldn't really be afraid of a few
unclean hacks that will be removed once we have ResultDB.
I think that we're of the same mind on this. I'm not completely against
not-completely-clean hacks for the short term but I also think that it
would be wise to think about the level of effort we're putting in. In my
mind, we're shooting for somewhat short-term gains on some of this and
after a certain point, energy devoted to hacking stuff together could be
better directed towards finishing ResultsDB.
So I thought about this. Could we send an email only after all tests
have finished? That would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded
definitions.
We know that we currently execute only two tests, depcheck and
upgradepath. Upgradepath runs always on a noarch arch. Depcheck runs
on i386 and x86_64 in 99,9% [1].
If there are that few, we could hard code them into the bodhi commenting
code. Not sure that is the greatest idea ever, though. Another option
would be to document the issue and contact the affected maintainers.
So our code may default to email=False and change to email=True only
when all tests have been executed (counting also the current one that
is to be sent). That alone decreases the email load by 66%.
That works for me in the short term (the hardcoding part).
Let's go further. If all tests are completed and PASSED, I
wouldn't
send an email. If everything went fine, why bother the maintainer?
All tests completed and PASSED might be worth sending an email for as a
notification that everything is working but I see your point. We might
want to get some input from maintainers on that one.
Let's go further. Currently we re-send FAILED results after at
least
3 days from the first failure. I wouldn't send email notifications
for them. The purpose is to provide a fresh failure log, not the
notification (the maintainer already knows about it).
That makes sense to me unless we do really want to pester maintainers
with packages that are failing repeatedly. Maybe up it from 3 to 5 or 7
days?
Then there remains the question of PASS->FAIL and FAIL->PASS
transitions. I would probably send notifications for them. They seem
rather important.
At the very least, transitions are important. Agreed on this one.
What do you think?
[1] I've examined F14 repository, it contains about 16800 packages.
About 20 packages is 32bit only: http://fpaste.org/BRwA/ and 4
packages is 64bit only: http://fpaste.org/WVBO/ I'm willing to
sacrifice that, for now. We have much higher test crash ratio
anyway.
4:32 a.m.
So I thought about this. Could we send an email only after all tests
have finished? That would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded
definitions.
We know that we currently execute only two tests, depcheck and
upgradepath. Upgradepath runs always on a noarch arch. Depcheck runs
on i386 and x86_64 in 99,9% [1].
I just realized that upgradepath is not run for updates proposed into updates-testing.
Something to keep an eye on. Still it should be doable.
10:16 a.m.
On 04/29/2011 03:10 PM, Tim Flink wrote:
Since this has almost nothing to do with the log output of depcheck
upgradepath, I'm splitting this conversation off into another thread.
Are we good with what has been discussed so far or at least good enough
to say we're done with the initial planning? I'm looking to get started
on this today.
My plan would be to get some rough work done and propose it since there
are still some TBD aspects of this.
Tim
1:13 p.m.
Just as an aside, I wouldn't mind few emails from Bodhi, anyway. What I was thinking
is
if we could do a 'Daily Digest' if there has been any changes/comments in the
previous
24 hours, rather than receive an individual email every time. It can get annoying if I
get
five emails, and each one is essentially copied in the last.
Am I making sense here?
John Dulaney
jdulaney(a)fedoraproject.org
j_dulaney
Date: Wed, 11 May 2011 09:16:57 -0600
From: tflink(a)redhat.com
To: autoqa-devel(a)lists.fedorahosted.org
Subject: Re: Decreasing Bodhi Comment Emails From AutoQA
On 04/29/2011 03:10 PM, Tim Flink wrote:
Since this has almost nothing to do with the log output of depcheck
upgradepath, I'm splitting this conversation off into another thread.
Are we good with what has been discussed so far or at least good enough
to say we're done with the initial planning? I'm looking to get started
on this today.
My plan would be to get some rough work done and propose it since there
are still some TBD aspects of this.
Tim
_______________________________________________
autoqa-devel mailing list
autoqa-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/autoqa-devel
1:48 p.m.
On 05/11/2011 12:13 PM, John Dulaney wrote:
Just as an aside, I wouldn't mind few emails from Bodhi, anyway.
What I was thinking is
if we could do a 'Daily Digest' if there has been any changes/comments in the
previous
24 hours, rather than receive an individual email every time. It can get annoying if I
get
five emails, and each one is essentially copied in the last.
Am I making sense here?
I think so, but I'm wondering if getting emails immediately when a test
fails would be helpful instead of waiting until the next digest was sent.
Do you think that receiving emails as a digest would be better than
receiving emails deemed "important" as soon as they are generated and
getting far fewer "PASSED" noise emails?
Tim
2 p.m.
I think so, but I'm wondering if getting emails immediately when a
test
fails would be helpful instead of waiting until the next digest was sent.
Point.
Do you think that receiving emails as a digest would be better than
receiving emails deemed "important" as soon as they are generated and
getting far fewer "PASSED" noise emails?
Indeed. Getting six emails saying "Yes, this works." is rather useless, to
me, especially when the sixth contains all the info in the previous five.
Example:
The email bellow is for Anaconda. It contains everything that was in the
previous four emails I have received from Bodhi on this particular package.
Just the one would have been fine.
The following comment has been added to the anaconda-15.31-1.fc15 update:
autoqa - 2011-05-11 05:45:40 (karma: 0)
AutoQA: upgradepath test PASSED on noarch. Result log:
http://autoqa.fedoraproject.org/results/95379-autotest/qa02.c.fedoraproje...
(results are informative only)
To reply to this comment, please visit the URL at the bottom of this mail
================================================================================
anaconda-15.31-1.fc15
================================================================================
Update ID: FEDORA-2011-6638
Release: Fedora 15
Status: testing
Type: bugfix
Karma: 3
Request: stable
Bugs: 702603 - FormatCreateError: ('format failed: 1', '/dev/mapper
: /vg_kralik-LogVol01')
: 699923 - livecd doesn't have /var/log/dmesg
: 700085 - iSCSI login dialog allows unselecting all
: devices and continuing - unable to login to any iSCSI
: devices after
Notes: This update fixes the memory requirement dialog and enforces the
: minimum btrfs filesystem size on partitions of that
: type. This update fixes a problem with reporting bugs
: on live installs and a bug with iSCSI.
Submitter: clumens
Submitted: 2011-05-06 19:03:25
Comments: bodhi - 2011-05-06 19:03:49 (karma 0)
This update has been submitted for testing by clumens.
bodhi - 2011-05-07 15:05:02 (karma 0)
This update has been pushed to testing
filiperosset (proventesters) - 2011-05-07 23:33:45 (karma 0)
Not tested.
jlaska (proventesters) - 2011-05-09 19:33:12 (karma 1)
filiperosset - Note, no comment is needed if you don't
test an update. I tested the proposed update with
custom-built x86_64 boot.iso. Works good for
installation here.
adamwill (proventesters) - 2011-05-09 20:40:41 (karma 1)
straight through live install in a vm works.
bodhi - 2011-05-09 20:40:41 (karma 0)
Critical path update approved
jdulaney (proventesters) - 2011-05-09 22:00:15 (karma 0)
not tested
rhe (proventesters) - 2011-05-11 05:30:54 (karma 1)
Default install passed.
bodhi - 2011-05-11 05:30:57 (karma 0)
This update has reached the stable karma threshold and
will be pushed to the stable updates repository
autoqa - 2011-05-11 05:45:40 (karma 0)
AutoQA: upgradepath test PASSED on noarch. Result log:
http://autoqa.fedoraproject.org/results/95379-autotest
/qa02.c.fedoraproject.org/upgradepath/results/output.l
og (results are informative only)
https://admin.fedoraproject.org/updates/anaconda-15.31-1.fc15
2:31 p.m.
On 05/11/2011 01:00 PM, John Dulaney wrote:
> I think so, but I'm wondering if getting emails immediately
when a test
> fails would be helpful instead of waiting until the next digest was sent.
Point.
> Do you think that receiving emails as a digest would be better than
> receiving emails deemed "important" as soon as they are generated and
> getting far fewer "PASSED" noise emails?
Indeed. Getting six emails saying "Yes, this works." is rather useless, to
me, especially when the sixth contains all the info in the previous five.
I might be misunderstanding you, but it sounds like you're talking about
the emails generated when people add comments to an update. If I'm
right, that is part of Bodhi and something that we can't change. You
could file an enhancement request for Bodhi or bring it up on devel@,
though.
Example:
The email bellow is for Anaconda. It contains everything that was in the
previous four emails I have received from Bodhi on this particular package.
Just the one would have been fine.
I think that the current plans of sending emails when all tests are done
or when there has been a state change between PASS and FAIL would take
care of the extra email noise generated by AutoQA. More details are in
trac [1].
Are we missing a case where this wouldn't be true?
Tim
[1] https://fedorahosted.org/autoqa/wiki/Planning05xEmailReduction
6:56 p.m.
I might be misunderstanding you, but it sounds like you're
talking about
the emails generated when people add comments to an update. If I'm
right, that is part of Bodhi and something that we can't change. You
could file an enhancement request for Bodhi or bring it up on devel@,
though.
Exactly, and will do.
I think that the current plans of sending emails when all tests are
done
or when there has been a state change between PASS and FAIL would take
care of the extra email noise generated by AutoQA. More details are in
trac [1].
+ 1
Are we missing a case where this wouldn't be true?
Don't think so off the top of my head.
John Dulaney
jdulaney(a)fedoraproject.org
j_dulaney
4760
days inactive
4772
days old
autoqa-devel@lists.fedorahosted.org
10 comments
3 participants
participants (3)
-
John Dulaney -
Kamil Paral -
Tim Flink