Avoiding conditioning ignorance towards AutoQA

[Kamil Paral]

> Hi fellow Fedorans.
> 
> Recently, AutoQA has been introduced to catch typical problems early
> in
> the update process. In general, I appreciate that effort, but
> currently
> I find myself in a phase of conditioning ignorance towards AutoQA,
> essentially because it is drowning me in irrelevant information. The
> current case why I'm writing is
> https://admin.fedoraproject.org/updates/lua-wsapi-1.3.4-4.fc15.
> 
> Here are two ideas to make AutoQA relevant, less time-consuming, and
> more helpful. In short: good QA is always quiet, only if there is a
> problem it communicates.
> 
> - Post only errors
> It is common, for example, in automated build or continuous
> integration
> systems to send out emails only on errors. Similar goes for Unix
> tools,
> which tend to be quiet if everything is ok, and only bother you with
> output if something is not. Therefore, I propose to have AutoQA
> messages
> posted only in case that there has been an error.

How can you then distinguish an update for which the tests have passed from an update for which the tests haven't yet been executed?

Moreover, currently not all updates are tested. Sometimes our tests simply don't work properly. Not just the updates are being tested, the whole AutoQA is being tested (and developed) in this whole effort.

> 
> - Accumulate error messages
> An email is sent for every single comment to Bodhi. In the case of
> AutoQA, it causes one email per platform. It increases the load of
> email
> tremendously to deal with, which in turn makes me ignore it.
> Therefore,
> I propose to accumulate messages for all platforms. 

We have that in plan, believe me.

> Combined with the
> earlier proposal, the states for all platforms should be collected by
> an
> intermediate node, and if and only if a test failed on any of the
> platforms, one message with all status messages is posted to the
> update.

Sending Bodhi comments is just a quick way how to inform the maintainers. We are working on a results database with API that other Fedora services (Koji, Bodhi) could query and use the results as they seem fit. For most tests I expect it will be similar to what you describe. But that's future. Until that's implemented we can only either send comments to Bodhi or send no comments at all.

> 
> On a related note: it'd be much appreciated if Bodhi would provide an
> option to get a daily digest with all comments of all the packages I'm
> involved with.

Great idea, you can ask lmacken about that (or create ticket in its Trac).
Or, you can filter your emails and check the relevant folder once a day :)

> 
> I hope the fine folks of the AutoQA effort take these proposals into
> account when proceeding in the development of the system and help me
> to
> stop ignorance from taking over.

It will take some time, but we see the deficiencies, same as you do.
We try to improve as fast as possible.

> 
> Regards,
> Tim

Thanks,
Kamil


PS: We have a special mailing list for AutoQA:
https://fedorahosted.org/mailman/listinfo/autoqa-devel