wikitcms / relval 1.1: now with user-stats and testcase-stats

[Adam Williamson]

taskotron, schmaskotron, the really big news of the day is relval 1.1!

It's tagged in git, uploaded to
https://www.happyassassin.net/relval/releases/relval-1.1.tar.xz and
https://www.happyassassin.net/wikitcms/releases/wikitcms-1.1.tar.xz ,
and available from the repo at
https://www.happyassassin.net/wikitcms/repo/wikitcms.repo .

relval has two new sub-commands: user-stats and testcase-stats . They
re-implement stats-wiki.py and testcase_stats/testcase-stats.py from
fedora-qa.git stats/ , respectively. Both should be improvements on the
previous versions, as well as using wikitcms and dumping the direct MW
API calls.

testcase-stats especially got rather better. It considers 'unique
tests', not just test cases, now. It (along with wikitcms) considers
three things:

* test case page name
* test 'name' - the link text in the "Test Case" cell, if it is not just
the wiki page name
* the section of the wiki page in which the test appears

so results for the same test, but with different 'names' or appearing
multiple times on the same page in different sections (as with the Cloud
and Desktop pages), are not mashed up together, but properly tracked
separately.

The 'details' pages for unique tests now provide an accurate coverage
percentage for the test for each page on which it appeared, break down
the results by environment, and list bugs associated with 'pass' and
'warn' results as well as 'fail' results.

The summary page is split into tables according to the sections in the
underlying page (note: if a new section is added between two composes,
or a new test case added, or a section renamed or a test case moved,
relval will currently treat this as a 'new' test / section and render it
at the bottom).

The results 'bitmap' is smarter: it knows all the composes that exist in
the whole result set, and includes a completely empty column if the
'unique test' did not exist in that compose. This helps with tests that
got moved around or renamed or whatever, because all the bitmaps line up
- you can actually see which composes any given 'unique test' existed
in.

You can pass --milestone to limit the information to a specific
milestone, if you like (but you can still run on an entire release, and
the wikitcms version uses some wikitcms foo to make sure the results
don't include Rawhide and nightly and PowerPC and whatever pages).

Please do grab it, play around, and see if you can break it! I've run it
over Fedora 20, Fedora 21, Fedora 18 and Fedora 14 with good results.
You can see output for various releases here:

https://www.happyassassin.net/testcase_stats/

21's Install/Installation results are a bit messy because of the
extensive changes to that page during Alpha and Beta (it was renamed
from Install to Installation, for a start...); tomorrow I'll spend a bit
of time modifying the pre-Beta TC2 pages to match the current format, so
relval's results look nicer. I *did* do some mental design on a sort of
pre-processor function which would try to handle changes to the page
layout, but it got very complex in a hurry and I figured it makes more
sense to just clean up the pages when this sort of thing happens. The 20
results are fairly nice and clean, just a couple of table renames in
Desktop.

It just occurred to me that adding a column for 'last 100% coverage'
would be trivial...and might be useful.

If this looks good to folks, maybe we could put it into fedora-qa.git
and add it to Phabricator for issue tracking.

It would, at this point, not be very difficult to add a result reporting
CLI, either...I suppose I could do that if I get time, just for fun.

Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net