8:40 p.m.
With the ARM moving to primary and essentially cloud achieving the same
status (from a QA perspectice), we need a better way to track and
display test results for each release. The current method (1) is a wiki
page with a set of columns for each architecture and rows for test cases
grouped by test case type. While this is easily expandable, it quickly
becomes a less than optimal method to display and manage data.
What Adam and I were discussing is a web app to do this. The problem is
that neither of us are UI designers. We have some ideas as to how we
want information displayed, queried, etc, but that's it. Here are the
relevant irc logs.
16:08 < handsome_pirate> adamw: Also, it's probably too late for F20, but
I've been thinking about the huge number of columns in the test matrices issue
16:08 < handsome_pirate> adamw: Namely, I've been kicking around doing a web
app
16:09 <@dgilmore> adamw: generally pxe with kickstart, but pxe and interactive works
also
16:09 < adamw> handsome_pirate: yeah, too late for f20, but i'd love a more
capable replacement
16:09 < adamw> handsome_pirate: i have a plan for f20 that's much simpler: table
the other way around
16:09 < handsome_pirate> Indeed
16:10 < adamw> so for this test case, it'll be a table on the install matrix
page but with the *test case* as the columns and the platforms as the rows
16:10 < handsome_pirate> adamw: I'll see if I can get my programming instructor
to let me do it for class credit
16:10 < adamw> probably have multiple columns representing the same test case with
different images
16:10 < adamw> that was my next thing to do
16:10 < adamw> handsome_pirate: it might turn into kind of a big project, note
16:11 < handsome_pirate> adamw: Indeed
16:11 < adamw> handsome_pirate: if we're going to do something better then
we'd want to do something awesome, that hooks into the blocker bug webapp etc
16:11 < adamw> but don't let me project stop energy at you :P
16:11 < handsome_pirate> adamw: But, I'm not a GSoC person; I won't be
going anywhere
16:11 < handsome_pirate> adamw: That would be wicked cool
16:11 < handsome_pirate> adamw: But, for starters, I want to follow KISS
16:12 < handsome_pirate> adamw: I'll ping mizmo or emichan to see about making
it pretty
16:13 * handsome_pirate noms on steak and bacon tacos
16:15 < adamw> handsome_pirate: yeah, in fact, thinking about it, the matrices are
the most easily drop-in-replacable thing
16:15 < adamw> you could just do a webapp which represents that layout more flexibly
and that would drop right in and make things better
16:15 < adamw> good thinking
16:16 < handsome_pirate> Exactly what I was thinking to start out
16:17 < adamw> cool
16:17 < adamw> let's see, if i wrote a five minute spec for you...
16:17 < handsome_pirate> Aye
16:17 < adamw> we'd need a concept of tests, configurations, and releases
16:17 < adamw> and then a nice interface for querying the database based on those
concepts
16:18 -!- streeter [streeter@nat/redhat/x-asubidnuwbirqiyg] has quit [Quit: Leaving]
16:18 < adamw> oh, and test level of course...and that might need to vary depending
on the other concepts...hm
16:19 < adamw> handsome_pirate: so we need to express stuff like 'this test
needs to be run on this config at alpha, on this config at beta, can optionally be run on
this config, and is not applicable to this other one'
16:19 < adamw> and then some sort of awesome query engine so we can generate all
sorts of different views, for working from and for searching
16:19 < adamw> SOUNDS EASY RITE?!
16:19 < handsome_pirate> adamw: For db query, maybe have some way to select a test
case and then it shows a page with current results for that test case for all images
16:19 < handsome_pirate> heh
16:20 < adamw> oh and we need test category too
16:20 < handsome_pirate> Indeed
16:20 -!- tempus_fol [~tempus@gateway/tor-sasl/tempusfol] has joined #fedora-qa
16:20 < Cerlyn> Just don't try and do it within Mediawiki's own internal db
unless they've made it so you don't have to flush every page to see an update
16:20 < adamw> sure, but i'm thinking stuff like 'show me all the required
deployments tests for ARM that we haven't done since RC1'
16:20 < handsome_pirate> I've got something sort of floating around in my head
16:20 < adamw> Cerlyn: handsome's thinking of writing a webapp for it
16:20 < handsome_pirate> adamw: Aye
16:21 < Cerlyn> Yet another test case manager :/
16:21 < handsome_pirate> Cerlyn: Probably use something like sqllite
16:21 < handsome_pirate> Cerlyn: Do you think there's one out there we can
adopt?
16:21 -!- weld [~weld(a)p57A83BEF.dip0.t-ipconnect.de] has joined #fedora-qa
16:22 < Cerlyn> handsome_pirate: I'd have to look. At one point Fedora was
going to move to what RH was moving to (Nitrate) but that seem to have stopped
16:22 < handsome_pirate> adamw: Honestly, the hardest part for me is going to be
the UI
16:22 < adamw> handsome_pirate: we're growing more and more test cases and test
case sets and platforms, the wiki approach is definitely getting unwieldy
16:22 < Cerlyn> Mozilla was working on one which was supposed to be OSS & public
friendly
16:22 < handsome_pirate> adamw: Indeed
16:22 < adamw> handsome_pirate: actually the more i think about it the more this
would be awesome and we need it NOW
16:22 < handsome_pirate> adamw: Okay, I'll set to work
16:23 < adamw> Cerlyn: the neat thing here is we're just looking at a UI for the
*matrices*
16:23 < adamw> Cerlyn: we're doing an end run around the whole tcms problem
16:23 < adamw> Cerlyn: the test cases can still be dumb wiki pages; what we really
could do with is a much more flexible way of querying test case sets (and entering
results)
16:24 < handsome_pirate> Cerlyn: All the test case managers I've seen won't
do what we need to do
16:24 < Cerlyn> So your TCMS will import the Wiki data as a source
16:24 < adamw> Cerlyn: it doesn't need any data as a source. it's not a
tcms.
16:24 < adamw> it's a webapp for representing views on the set of test cases.
16:24 < adamw> all it needs to know is the URLs of the test case pages. it does not
care a jot about the contents.
16:25 < Cerlyn> then how does it know which test cases haven't been run yet?
16:25 < adamw> Cerlyn: we could build a simple one which just lets us look at groups
of test cases and even that would be interesting, but yeah, we need it to store that
16:26 < adamw> handsome_pirate: i've been using trac's query interface a lot
the last few days and it has its good points. i'm kinda thinking of something like
that - a nice quick GUI 'query builder' which lets you generate all kinds of
different views and save useful ones. we'd probably have a UI with some kind of
overview for the current test build as the default, a set of canned queries, and the ui
for building a custom one
16:27 -!- tyll [~till@fedora/tyll] has quit [Ping timeout: 246 seconds]
16:27 < adamw> Cerlyn: it shouldn't be _too_ hard to store that info, though. it
really just needs to store...let's see, user foo ran test bar on configuration moo for
release baa on (date)
16:28 < adamw> with result X, associated bug reports Y, freeform comment Z
16:28 < adamw> did I miss anything?
16:28 < handsome_pirate> adamw: I'm thinking that for the main overview (ie,
where you could input results, get a quick view, etc), have a list of current test cases
16:29 < handsome_pirate> Click on one, and it expands to show current results and a
place to input results
16:29 < handsome_pirate> Does that make sense?
16:29 < handsome_pirate> I'll follow trac's query page for setting up my
own
16:29 < adamw> you might be best asking design team, like you said :) but let me
think
16:29 < adamw> if we put ALL the test cases in one big list that's gonna be a
bit overwhelming...
16:30 < handsome_pirate> True
16:30 * adamw tries to think what a UI would look like from a completely white page and
is utterly terrified
16:30 * adamw never wants to be a designer
16:31 < handsome_pirate> I think the click on test case, it expands with info (at
the least, link to wiki), test results, and place to enter test results may be a starting
point
16:31 < handsome_pirate> I'll write up an email for the design list
16:31 < adamw> sure, if you just want to have a placeholder UI while you work on it
it sounds fine
16:31 -!- Cerlyn [~cerlyn(a)173-12-75-9-miami.hfc.comcastbusiness.net] has quit [Remote host
closed the connection]
16:32 < adamw> i suppose if you query the list in a certain way and then hit
'enter result' it should pre-fill the result thingy with your query restrictions
16:32 < adamw> (so if you query for ARM tests for F20 Alpha RC2, it should give you
a report form with that config and release selected...)
16:32 < adamw> zoiks, i need to get back to working on what we have now, hehe :)
i'll leave you to it
At this point, any help with figuring out how to make it look would be
much appreciated.
Thanks much,
John.
(1): Example: https://fedoraproject.org/wiki/Test_Results:Fedora_19_Alpha_RC1_Install