Avoiding conditioning ignorance towards AutoQA

Thu Apr 28 06:34:10 UTC 2011

On 04/27/2011 03:00 PM, James Laska wrote:
> On Wed, 2011-04-27 at 21:03 +0200, Tim Niemueller wrote:
>> On 27.04.2011 15:38, Kamil Paral wrote:
>>>>
>>>> - Post only errors It is common, for example, in automated build or
>>>> continuous integration systems to send out emails only on errors.
>>>> Similar goes for Unix tools, which tend to be quiet if everything
>>>> is ok, and only bother you with output if something is not.
>>>> Therefore, I propose to have AutoQA messages posted only in case
>>>> that there has been an error.
>>>
>>> How can you then distinguish an update for which the tests have
>>> passed from an update for which the tests haven't yet been executed?
>>>
>>> Moreover, currently not all updates are tested. Sometimes our tests
>>> simply don't work properly. Not just the updates are being tested,
>>> the whole AutoQA is being tested (and developed) in this whole
>>> effort.
>>
>> Please don't force testing, fixing, and maintaining AutoQA on the rest
>> of us. 
> 
> Well, we don't force it, but like bodhi, koji and most other key
> infrastructure tools, it is software ... and software unfortunately has
> bugs.  We do our best to minimize those bugs, but we've found it very
> difficult to emulate a real-world bodhi+koji setup (with interesting
> data) in our test instance.
> 
>> Integrate it such that stuff is pushed to testing only after
>> AutoQA has been run, or have a flag display "tests ran". Or post the
>> "PASSED" messages, but make Bodhi not sent messages in the case the
>> tests passed.
> 
> Re: integration and only allowing updates to proceed to
> 'updates-testing' ... that's the goal.  Unfortunately, as with many
> things, arriving at a destination involves a journey.  We can't simply
> turn this "feature" on until we have confidence that the tests are
> correctly enforcing the package update policy [1].  Therefore, we are
> operating in a permissive mode at this time.
> 
> With regards to having a flag or having bodhi not send messages in
> certain scenarios.  Those are definitely options.  We'll need to raise
> those ideas the bodhi folks [2] for review, since AutoQA and bodhi are
> separate projects.  But your concerns are definitely worth following up
> on.
> 
> [1] https://fedoraproject.org/wiki/Updates_Policy
> [2] https://fedorahosted.org/mailman/listinfo/bodhi
>
>> Keeping the current way will just make me (and possibly others) add
>> filters to throw away messages from AutoQA. Please be aware of how much
>> contributor time you waste by making them hope through useless (because
>> the tests have passed and no information is gained) mails. I realize you
>> want to improve things, but at the current stage its consuming the most
>> valuable resource we have, packager/developer time.
> 
>>>> - Accumulate error messages An email is sent for every single
>>>
>>> We have that in plan, believe me.
>>>
>>>> Combined with the earlier proposal, the states for all platforms
>>>> should be collected by an intermediate node, and if and only if a
>>>> test failed on any of the platforms, one message with all status
>>>> messages is posted to the update.
>>>
>>> Sending Bodhi comments is just a quick way how to inform the
>>> maintainers. We are working on a results database with API that other
>>> Fedora services (Koji, Bodhi) could query and use the results as they
>>> seem fit. For most tests I expect it will be similar to what you
>>> describe. But that's future. Until that's implemented we can only
>>> either send comments to Bodhi or send no comments at all.
>>
>> I understand you like to have a quick and working solution for now, and
>> that great stuff is coming. But you cost a lot of time right now. Please
>> reconsider to make your development and testing time less intrusive for
>> others.
> 
> Thanks for your feedback.  We're actively working on ways to reduce the
> unnecessary emails generated by bodhi when posting feedback.  Obviously,
> the goal is to work *for* maintainers, not against them.  

I second the thanks for feedback. I'm not promising that we can please
everyone all of the time, but it's harder for us to fix pain points if
we don't know where they are. Like most developers, we like constructive
feedback!

>>>> On a related note: it'd be much appreciated if Bodhi would provide
>>>> an option to get a daily digest with all comments of all the
>>>> packages I'm involved with.
>>>
>>> Great idea, you can ask lmacken about that (or create ticket in its
>>> Trac). Or, you can filter your emails and check the relevant folder
>>> once a day :)
>>
>> That still means striving through many messages with lots of non-info
>> text. One concise email would make things much better.
> 
> The point Kamil was making is that bodhi and AutoQA are different tools.
> If you have a good idea for bodhi, please raise that on
> https://fedorahosted.org/mailman/listinfo/bodhi.
> 
>>>> I hope the fine folks of the AutoQA effort take these proposals
>>>> into account when proceeding in the development of the system and
>>>> help me to stop ignorance from taking over.
>>>
>>> It will take some time, but we see the deficiencies, same as you do. 
>>> We try to improve as fast as possible.
>>
>> Please try to find a way to do this without costing as much time as atm.
>> There must be ways, for example have a list of packages to use it for
>> that packagers can opt-in and make AutoQA developers the first to use it.
> 
> Opt-in support has been available for some time now.  I probably could
> communicate this better, but alas ...
> 
> https://jlaska.wordpress.com/2010/06/01/fedora-package-maintainers-want-test-results/

Correct me if I'm wrong, but I don't think that he was talking about the
raw results from AutoQA but the notification emails that get sent from
Bodhi as a result of the comments made by AutoQA.

>>> PS: We have a special mailing list for AutoQA: 
>>> https://fedorahosted.org/mailman/listinfo/autoqa-devel
>>
>> Sorry, to keep me involved in Fedora I have to make it a reasonable
>> effort, joining yet another project is out of my possibilities atm.

Again, I could be wrong here but I do believe that Kamil's suggestion
was to cc: autoqa-devel at lists.fedorahosted.org on topics related to
AutoQA to receive a faster response. It is by no means a requirement but
I know that I only check devel@ a couple of times a day. On the other
hand, if something shows up on the autoqa-devel list I read it as soon
as I notice it. Either way works, though.

> Understood.  There are only so many hours in a day.
> 
> Thanks,
> James
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 553 bytes
Desc: OpenPGP digital signature
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20110428/f084413a/attachment.bin 