On Wed, 2017-05-03 at 13:29 +1000, Dan Callaghan wrote:
What you have implemented in your Bodhi patch is actually to just
treat
the results as unique on item+testcase+scenario (without arbitrary extra
fields that vary per item). And the different combinations of OpenQA
installation test are smooshed together into the scenario string. Right?
More or less, yeah.
Your Bad News One is that ResultsDB is not efficient at finding the
latest result by item+testcase+scenario. But I am sure we can fix that
in ResultsDB if we want. Worst case, by special-casing "scenario" in the
db schema and promoting it out of extra_data into its own db column with
an index.
Well, all things are possible, but this would be a very significant
change to resultsdb, and I suspect Josef wouldn't be a big fan :) But
if we decide that scenario really is a sufficiently important concept
and we're sufficiently confident in our final implementation of it,
yes, this is a *possibility*.
Your Bad News Two is that you either display the scenario strings in
Bodhi, and they look kinda ugly and hard to read, or you *don't* display
them in Bodhi and then it looks like there are duplicate test results.
This problem already existed for the test case names though, right?
Those are also ugly, human-unfriendly strings.
Meh, I wouldn't say so. I think there's kind of an implication in rdb's
design that the test case name can / should be used as an identifier
for squishy humans, and indeed they frequently *are* used as such
(including by Bodhi).
The solution for the test
cases is that ResultsDB stores some extra metadata about them, with
a link off to a human-friendly page in `ref_url`. (Although I note that
Bodhi right now doesn't show that info anywhere...)
I don't *think* this is quite the idea. The URL associated with a test
case is intended to be a place you can go to find out more about the
test case if you would like to, but the test case's name is supposed to
be the primary human-readable identifier, I believe.
Could you do the same thing for scenarios? Is there an unreasonably
large number of different scenario combinations?
I haven't run the numbers, but I suspect the answer is 'yes', and in
general, the scenario concept is a bit too fluid for "hand write a
human-consumable description of every possible scenario identifier" to
be a good idea.
Kamil's suggestion was that rather than a result just providing a
single scenario value that's intended to be a not-very-human-readable
string that represents, in some sense, the entire 'scenario' for a
test, a result (let's say it's for the test update.base_selinux) would
provide something like this:
scenario_keys: [arch, machine, flavor, version, distri]
arch: x86_64
machine: uefi
flavor: server
version: 26
distri: fedora
Then if you're an rdb consumer and you wanted to be really sure you had
the entire 'description' of a given test you could represent it just by
displaying *all* those values, with or without the key name depending
on how it fit into your consumer's interface. But your consumer could
also think like this:
OK, I've got 50 results for this item. The arch, machine, version and
distri for *every* result are the same, so there's probably no need to
show those. But some of the results have different flavors. So I should
probably use 'testcase name - flavor' as the identifier, like so:
"update.base_selinux - server". Hey, that looks pretty good!
We *could* store the scenario_keys together with the test case rather
than with each result. I'm...unsure whether that's a good idea.
Do new ones appear
often?
They certainly can. For the openQA case, for instance, we might come up
with a new flavor or machine at any time, for one of several reasons.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net