testcase-stats on wikitcms PoC

Sun Oct 12 10:17:39 UTC 2014

On Sat, 2014-10-11 at 02:37 -0700, Adam Williamson wrote:
> Hey, folks. Today I got a PoC of testcase-stats on top of wikitcms up
> and running. It's not complete yet as I haven't written the bits to
> provide the CLI and page selection stuff, but all the hard stuff is
> there. I'm attaching it. To test it, just stick it in the
> stats/testcase-stats directory of a fedora-qa git checkout, symlink an
> up-to-date checkout of the wikitcms 'results-types' branch into the same
> directory, and run it. It'll output to /tmp/tcstats_test , but you can
> easily change that if you like. You can also change the hard-coded list
> of pages it runs on.
> 
> Tomorrow I'll aim to clean it all up and make it a relval sub-command,
> then maybe release wikitcms 2.0. it may get interesting when I look at
> feeding in various possible groups of pages, but I'll burn that bridge
> when I come to it.

So I didn't quite get to the relval sub-command point yet, but I made it
a hell of a lot better today. If you want to play with my current state
of the art:

https://www.happyassassin.net/temp/tcstats_test/

is the output from a run on Fedora 20. It handles test cases that occur
multiple times both with different 'names' and with the same 'name' but
*in a different table* (it was rather badly thrown by the Desktop page
before I made wikitcms parse tables).

By default it displays the tests in more or less the order they appear
on the actual pages, with table separators - changes to the results or
tables between composes confuse it a bit, whenever a case shows up that
didn't exist in a previous compose it'll likely get stuck at the bottom
and may lead to table separator duplication.

It provides rather a lot more detail on the 'details' page: it counts
results per 'environment', lists bugs for passes and warns as well as
fails, and calculates the 'coverage' - the percentage of environments
for which a result was provided - for each page.

https://www.happyassassin.net/temp/tcstats_test/Desktop.html vs
http://testdays.qa.fedoraproject.org/testcase_stats/Category_Desktop_validation_testing.html makes an interesting comparison (mine looked a lot like Josef's before I taught it to parse tables) - you can see mine's more complex but also trickable, the name of the 'release blocking' table changed after Alpha TC1, but it's rather difficult to somehow catch that and merge the results with those for the renamed table from later composes. The easier fix in this and some similar cases is just to go to the wiki and make the pages consistent =)

Sorting is currently broken - that's more or less next on my list (after
somehow fixing "Install" / "Installation", which is confusing the 21
results slightly).

The current state of the code is attached for the curious, it still
needs some cleanups here and there. I made write_detail_page() a
function nested within the print_results_html() function because it
needs to be run for each test kind of in the middle of the summary page
generation loop, to make sure the links from the summary page and the
page names actually generated by the detail page writer stay consistent.
It also works as a top-level function you pass a whole *ton* of
variables into, but that seemed uglier. I might instead just make Test
class methods to generate the necessary bits and call into those, have
to try it later to see if that makes it nicer.

It still needs HEAD of wikitcms' results-types branch, I had to make
quite a lot of changes to wikitcms. All the improved result page parsing
went in there, of course.

Amongst other things, the page.ComposePage() class grew a couple of
slightly interesting new members.

from_wikipage() is a class method which you can feed an mwclient page
object (i.e. a wiki page) and it'll try and spit out the ComposePage()
object for the same page - newstats first gets all the mwclient page
objects for the release according to Release.milestone_pages(), then
feeds 'em into this method to get wikitcms page objects it can use.

sortname() is a property which returns a string that you can use to sort
ComposePage objects: for any given set of ComposePage objects, sorting
with sortname() as the key should give you the pages in the 'correct'
order, i.e. Fedora 13 Final RC1, Fedora 14 Alpha TC1, Fedora 14 Alpha
TC2, Fedora 14 Alpha RC1, Fedora 14 Beta TC1, Fedora 14 Final RC2,
Fedora 16 Alpha TC3... - the order the composes actually happened in. It
does this by join()ing the release with heavily-modified versions of the
milestone and compose that should always sort correctly...or, well, I
mean, as close as seemed reasonable to bother handling (we had some
really pretty wacky page names back before Fedora 14). newstats uses
this to sort the pages for each test type right before it gets their
result rows and turns them into newstats Test() objects. Before I made
it an instance property, I had this magnificently hideous hack instead,
which I wish to preserve for posterity:

pages.sort(key=lambda x: ' '.join((x.milestone, x.compose.replace('TC',
'AC'))))

which...actually works surprisingly well, since "Alpha", "Beta" and
"Final" happen to sort alphabetically in any case. count the
parentheses!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: newstats.py
Type: text/x-python
Size: 11722 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20141012/3796dd0a/attachment.py>