AutoQA: distro congestion?

[Axel Thimm]

On Sun, 2011-04-17 at 08:01 +0200, Kevin Kofler wrote:
> Axel Thimm wrote:
> > Are you sure? I requested a push of the packages to stable (some were in
> > testing for a week, others were security updates) and the message was
> > that it doesn't pass AutoQA, so it converted to request to push only to
> > testing and indeed bodhi has marked the request as to testing only.
> 
> It did that because you changed it.
> 
> If you had kept the request to stable, it would have been pushed to stable.
> 
> AutoQA is still in testing phase, there is no enforcement yet.

Maybe the bodhi messages confused me. When I logon to a.f.o/updates it
prominently displays:

                Bodhi is now enforcing the Package Update Acceptance
                Criteria across all Fedora releases.

And the conversion to testing was mentioned when I tried to push the
package, not later, when a human could process the request.

Currently the autoqa process is very muddy to me and probably other
packagers as well:

      * There are contradictory statements on whether autoqa will be
        enforced or informative only. bodhi claims it is in place, Tim
        claims it will be in place by F16 and other claim it will always
        be informative only. For me it looks like bodhi rejects my
        packages based on autoqa partly without any autoqa run output
        available to me (read below).
      * Currently bodhi behaves differently than before. For example the
        two package sets I'm worried about right now are fail2ban and
        mediawiki. ATM all packages are in rawhide and testing (f2b for
        f15 is even stable), so I would expect at least the next-highest
        Fedora packages to pass the autoqa. Still, trying to push any of
        these packages from testing to stable I get:
              * "This update has not yet met the minimum testing
                requirements defined in the Package Update Acceptance
                Criteria"
              * The package request field in bodhi is empty, e.g. for he
                packager at least it looks like no push attempt was ever
                made, I don't know how it looks on the push granting
                side, but I assume it looks the same.
              * In some cases I don't get any indication why the autoqa
                failed. For example for the fail2ban packages I either
                received positive autoqa results or none at all (for f14
                I never received any autoqa). Still the fail2ban
                packages are rejected (all but f15).
              * I also never received any autoqa info on
                mediawiki-1.16.4-57.fc15 even though bodhi/autoqa
                rejects it.
      * Most of the autoqa bits are accessible only through my mail
        folder. Given that some autoqa never made it there (the assumed
        fail2ban autoqa failures), nor the bodhi comments, I am
        currently at a loss what to do with the fail2ban packages.

I like the general idea of an automated test suite, but currently autoqa
is blocking/rejecting package pushes on a.f.o/updates, or at least bodhi
tells me so. And the reasons for doing so are non transparent.

Focusing on fail2ban, which is the more traditional update from the two
(mediawiki was fasttracked by another update, so let's keep this example
simpler by ignoring it):

* The packages were submitted to testing, accepted and stayed there for
  a while.
* packages pushed to stable are being rejected for f13/f14 w/o the
  packager understanding why (there is no (f14) or just positive
  (f13/f15) autoqa feedback), especially when f15 did get pushed
  through.
* My push request seem to get currently dropped, so my packages remain
  in a non-requesting testing-updates limbo, and probably no one
  notices.

Please help with these packages - it takes me much longer to get them
through bodhi than the actual packaging/updating/testing of the
packages.

Also please consider the following for autoqa:

* make a firm decision on whether, when and how autoqa will enforce its
  results. It isn't just informative ATM.
* place ALL autoqa results in bodhi, even repeated ones. Hunting the
  autoqa output though mails and bodhi isn't helping.
* Make autoqa always informative. If autoqa fails allow the packager to
  add a comment to the update for the pusher to evaluate whether the
  packages should be pushed regardless of the autoqa failures. Please
  don't automatically reset push requests!

Thanks!
-- 
http://thimm.gr/ - http://ATrpms.net/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20110419/8470ee1d/attachment.bin