On Thu, 2010-09-30 at 13:44 +0200, Josef Skladanka wrote:
On 09/29/2010 10:38 PM, James Laska wrote:
> Thanks for summarizing Josef. It's been a while since we got to work
> with resultdb, so this help keeps things fresh!
>
> On Wed, 2010-09-29 at 09:10 -0400, Josef Skladanka wrote:
>> **skipped **
> Interesting, so the logic of how to interpret test results lives on the
> wiki, not in packaged source code. This is exciting and new! :)
Yes, my idea was to provide a 'meaningfull' metadata, so we can have a
'common basic frontend', which will show different testplans based on
the metadata in wiki. So you have the testplan semantics defined in the
wiki, results stored in the ResultsDB -> easy way to see, whether some
packages passes different testplans. Because you'll just provide the
testplan's URL, and (i.e.) package name, so the frontend can read the
data from ResultsDB, and visualise it based on the semantics provided on
the wiki.
Heh, this description feels a bit overcomplicated, but hopefully, you'll
get the idea.
> One thought, we seem to be mixing a type of structured syntax with wiki
> syntax (bullets). I wonder if that will lead to syntax-errors and
> confusion from test plan authors. For example, I imagine for some plan
> authors (installer) the metadata sections would be quite large and we'll
> want an easy way for the test plan maintainer to sanity-check their
> syntax. Having to manipulate a large chunk of this data might be rough.
> Being able to cut'n'paste it in one big section into a text editor might
> be useful? Perhaps nesting it all in<pre></pre> tags? Or perhaps
> there is a syntax parser for this stuff?
>
> This is a bit extreme, but is there a defined syntax?
The syntax on wiki (bullets & JSON) is used, because the library Will
wrote, just assumes this structure of data.
I see, great. I just wanted to make sure there was some standard format
whose documentation/semantics we could rely on.
Is this something we could use to help validate the JSON used?
The reason it's like that is IMHO, that we really need to have
only one
key/value pair per line, and bullets tend to provide the straightforward
way to visualise this requirement.
We sure can use/parse data from <pre> tags, but I'd rather talk about it
with Will first - because he might have other reasons to implement it,
as it is now.
Same as above. I think it would help us, and test plan authors, if we
can direct them to self-diagnose any JSON formatting issues on their
wiki plans.
If it's straight JSON, I wonder if it would keep folks sanity to show it
as straight JSON (not mixed with wiki elements). Even better, using a
mediawiki syntax highlighter (like
http://www.mediawiki.org/wiki/Extension:GeSHiHighlight) to display the
content. Not required, just thinking of ways to make it more clear what
the format+syntax is.
>> You noticed, that Rpmlint is both in the
"introspecition" and "advisory" groups, this is based on the current
state
>> of the
testplan<https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_u...
- I'm not really
>> sure how to base the "no errors" and "no warnings" into
that, based solely on the result, but that's not the real issue for now.
>>
>> This brought me to one more possible thing to think about: are the current
"result states" (RUNNING, PASSED,
>> INFO, FAILED, ABORTED, CRASHED, WAIVED, NEEDS_INSPECTION) enough?
>
> As listed, you have states for test case results, but it seems that test
> classes implicitly have a result as well? Do test cases and test
> classes use the same results?
>
> Or do test classes use one subset: PASSED, FAILED, WAIVED,
> NEEDS_INSPECTION
>
> And test cases use another subset (overlapping): PASSED, FAILED, WAIVED,
> INFO, CRASHED, ABORTED
Well, maybe I misdescribed it (or just do not understand your question
correctly) but.
I suppose that by "test classes" you mean the "mandatory",
"introspection" and "advisory" groups. If so, then these use the same
set of results, as the testcases do.
This is simply because the groups should define:
1) What is the 'allowed subset of results' for the testcases. I.E. if
all the testcases have their results in that subset, the overall
testplan result is PASSED.
2) If any of the tests, in the respective group, has result out of the
allowed subset, than the overall testplan result will be the one
specified by 'on_fail' value.
You described it fine. I was thinking about the uncertainty around
which values to use for different results/plans/classes meant that we
were trying to apply the same identifier, to different things.
For example, I was thinking how bugs have a status
(NEW,ASSIGNED,MODIFIED,ON_QA,VERIFIED,CLOSED...) and a resolution
(NOTABUG, WONTFIX, RAWHIDE,ERRATA,DUPLICATE...). The exact values don't
apply to this scenario, but I wasn't sure if there was an analogy here
between test results (bug status) and test plan/class result (bug
resolution).
Certainly not a big issue, just tossing that idea out for
discussion/debate.
>> AND - are we able to sort these according to the
>> 'urgency' or 'importance' or 'relevance' or how to call
it? Let's say
>> we have two tests in a testplan,
>> one is ABORTED and the second is CRASHED - what would you like to see
>> as 'overall result' Aborted or Crashed?
>> I'm not sure, if I was able to describe it clearly (I'm really hitting
>> the limits of my descriptive skills here :-D),
>> so feel free to ask as many questions, as needed.
>
> Honestly, I have no idea. I feel like I'd need to get some time behind
> the wheel on this in order to make a determination. You know, see it in
> action for a while. If I had to pick now, I'd say that CRASHED or
> ABORTED == FAILED.
Sure thing, I have quite the same feeling (even though i think, that the
result set is OK at the moment). It's just that Kamil and I have talked
about it, some time ago, so I felt like the rest of you should also have
a chance to comment it :)
Makes sense, thanks.
> If I'm understanding correctly, what seems tremendously
challenging in
> what you're creating is that you are designing a mechanism to allow plan
> authors to draft and implement policy on what results will be collected
> (wiki metadata), how results will be presented (front-ends) and what
> actions to take based on the results (bodhi karma etc...)? Is that
> accurate?
Not really a policy on "which results to collect" - because this is
mostly dependent on "what tests do we actually have" - but more like a
mechanism allowing plan authors to draft/describe the policy i a
'machine-readable' form, so you can immediately see, if this testplan
would pass/fail on given package(s). Because you already have the
results stored in the ResultsDB.
Maybe the best term to describe it would be a "view" - the metadata you
provide tell the aforementioned 'generic' frontend, which results should
it grab from DB, and how the individual test results affect the overall
testplan result.
So the aforementioned metadata are mainly describing the presentation
part, but are also supposed to provide a mechanism to deduce the
testplan's result from the individual testcase results.
Hope it's clearer now.
Please ask as many questions, as necessary!
If definitely is clearer, thank you. I love the idea, it's certainly
bold! I don't have experience with providing view-related metadata in a
different system as is being proposed. I do see the possible benefits
of having the separation between the presentation and the data.
The only guidance I can think of is, as we proceed, let's be mindful if
providing a generic mechanism for custom views is more difficult to
maintain that providing specific view implementations.
Full steam ahead! :)
Thanks,
James