Avoiding conditioning ignorance towards AutoQA

Wed Apr 27 21:00:11 UTC 2011

On Wed, 2011-04-27 at 21:03 +0200, Tim Niemueller wrote:
> On 27.04.2011 15:38, Kamil Paral wrote:
> >> 
> >> - Post only errors It is common, for example, in automated build or
> >> continuous integration systems to send out emails only on errors.
> >> Similar goes for Unix tools, which tend to be quiet if everything
> >> is ok, and only bother you with output if something is not.
> >> Therefore, I propose to have AutoQA messages posted only in case
> >> that there has been an error.
> > 
> > How can you then distinguish an update for which the tests have
> > passed from an update for which the tests haven't yet been executed?
> > 
> > Moreover, currently not all updates are tested. Sometimes our tests
> > simply don't work properly. Not just the updates are being tested,
> > the whole AutoQA is being tested (and developed) in this whole
> > effort.
> 
> Please don't force testing, fixing, and maintaining AutoQA on the rest
> of us. 

Well, we don't force it, but like bodhi, koji and most other key
infrastructure tools, it is software ... and software unfortunately has
bugs.  We do our best to minimize those bugs, but we've found it very
difficult to emulate a real-world bodhi+koji setup (with interesting
data) in our test instance.

> Integrate it such that stuff is pushed to testing only after
> AutoQA has been run, or have a flag display "tests ran". Or post the
> "PASSED" messages, but make Bodhi not sent messages in the case the
> tests passed.

Re: integration and only allowing updates to proceed to
'updates-testing' ... that's the goal.  Unfortunately, as with many
things, arriving at a destination involves a journey.  We can't simply
turn this "feature" on until we have confidence that the tests are
correctly enforcing the package update policy [1].  Therefore, we are
operating in a permissive mode at this time.

With regards to having a flag or having bodhi not send messages in
certain scenarios.  Those are definitely options.  We'll need to raise
those ideas the bodhi folks [2] for review, since AutoQA and bodhi are
separate projects.  But your concerns are definitely worth following up
on.

[1] https://fedoraproject.org/wiki/Updates_Policy
[2] https://fedorahosted.org/mailman/listinfo/bodhi

> Keeping the current way will just make me (and possibly others) add
> filters to throw away messages from AutoQA. Please be aware of how much
> contributor time you waste by making them hope through useless (because
> the tests have passed and no information is gained) mails. I realize you
> want to improve things, but at the current stage its consuming the most
> valuable resource we have, packager/developer time.

> >> - Accumulate error messages An email is sent for every single
> > 
> > We have that in plan, believe me.
> > 
> >> Combined with the earlier proposal, the states for all platforms
> >> should be collected by an intermediate node, and if and only if a
> >> test failed on any of the platforms, one message with all status
> >> messages is posted to the update.
> > 
> > Sending Bodhi comments is just a quick way how to inform the
> > maintainers. We are working on a results database with API that other
> > Fedora services (Koji, Bodhi) could query and use the results as they
> > seem fit. For most tests I expect it will be similar to what you
> > describe. But that's future. Until that's implemented we can only
> > either send comments to Bodhi or send no comments at all.
> 
> I understand you like to have a quick and working solution for now, and
> that great stuff is coming. But you cost a lot of time right now. Please
> reconsider to make your development and testing time less intrusive for
> others.

Thanks for your feedback.  We're actively working on ways to reduce the
unnecessary emails generated by bodhi when posting feedback.  Obviously,
the goal is to work *for* maintainers, not against them.  

> >> On a related note: it'd be much appreciated if Bodhi would provide
> >> an option to get a daily digest with all comments of all the
> >> packages I'm involved with.
> > 
> > Great idea, you can ask lmacken about that (or create ticket in its
> > Trac). Or, you can filter your emails and check the relevant folder
> > once a day :)
> 
> That still means striving through many messages with lots of non-info
> text. One concise email would make things much better.

The point Kamil was making is that bodhi and AutoQA are different tools.
If you have a good idea for bodhi, please raise that on
https://fedorahosted.org/mailman/listinfo/bodhi.

> >> I hope the fine folks of the AutoQA effort take these proposals
> >> into account when proceeding in the development of the system and
> >> help me to stop ignorance from taking over.
> > 
> > It will take some time, but we see the deficiencies, same as you do. 
> > We try to improve as fast as possible.
> 
> Please try to find a way to do this without costing as much time as atm.
> There must be ways, for example have a list of packages to use it for
> that packagers can opt-in and make AutoQA developers the first to use it.

Opt-in support has been available for some time now.  I probably could
communicate this better, but alas ...

https://jlaska.wordpress.com/2010/06/01/fedora-package-maintainers-want-test-results/

> > PS: We have a special mailing list for AutoQA: 
> > https://fedorahosted.org/mailman/listinfo/autoqa-devel
> 
> Sorry, to keep me involved in Fedora I have to make it a reasonable
> effort, joining yet another project is out of my possibilities atm.

Understood.  There are only so many hours in a day.

Thanks,
James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20110427/c0a85ddb/attachment.bin 