AutoQA Testcase Stats

[Tim Flink]

I've been working to revamp the AutoQA infrastructure so that we're not
running completely on F17 clients and after building several new client
machines in staging, I was curious about how well the tests were
running on the new client machines.

I've included the numbers below but 2 things jump out at me:
 - #438 is causing problems for repoclosure
 - there is a problem with depcheck on f19. the only runs that didn't
   crash are the ones without any builds in them. The crash looks
   consistent but I don't know what the root cause is.

Note that these numbers reflect execution in staging autotest since I
rebuilt the clients (including the queued jobs that weren't run while
rebuilding), not pass/fail results and not from production. I was only
interested in numbers about execution of the tests (GOOD - executed
cleanly, reported results. ERROR - problems in execution, probably
didn't report results)

Tim


Global Stats
============
2077 Jobs
  -  535   fc17
  -  760   fc18
  -  781   fc19

Crashes
-------
227 Crashes
  -  0   fc17
  -  80   fc18
  -  147   fc19

Passes
-------
1849 Passes
  -  535   fc17
  -  680   fc18
  -  634   fc19

Test Specific Stats
===================

depcheck
   363 runs ( 167 GOOD, 196 ERROR )
    -  84 fc17 runs ( 84 GOOD, 0 ERROR )
    -  151 fc18 runs ( 75 GOOD, 76 ERROR )
    -  128 fc19 runs ( 8 GOOD, 120 ERROR )

repoclosure
   48 runs ( 27 GOOD, 21 ERROR )
    -  7 fc17 runs ( 7 GOOD, 0 ERROR )
    -  20 fc18 runs ( 20 GOOD, 0 ERROR )
    -  21 fc19 runs ( 0 GOOD, 21 ERROR )

rpmguard
   795 runs ( 790 GOOD, 5 ERROR )
    -  202 fc17 runs ( 202 GOOD, 0 ERROR )
    -  270 fc18 runs ( 268 GOOD, 2 ERROR )
    -  323 fc19 runs ( 320 GOOD, 3 ERROR )

rpmlint
   795 runs ( 790 GOOD, 5 ERROR )
    -  220 fc17 runs ( 220 GOOD, 0 ERROR )
    -  301 fc18 runs ( 299 GOOD, 2 ERROR )
    -  274 fc19 runs ( 271 GOOD, 3 ERROR )

upgradepath
   75 runs ( 75 GOOD, 0 ERROR )
    -  22 fc17 runs ( 22 GOOD, 0 ERROR )
    -  18 fc18 runs ( 18 GOOD, 0 ERROR )
    -  35 fc19 runs ( 35 GOOD, 0 ERROR )
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20130719/f1aa6b2f/attachment.sig>