On Tue, 2017-03-07 at 12:18 -0500, Kamil Paral wrote:
> Note if I'm being honest I have a practical reason for this,
as openQA
> defines the test case name as one of the 'scenario keys', so if we go
> with 'test name not in scenario value', in the openQA reporter we'll
> have to take the list of 'scenario keys' and then remove the test name
> from it. Obviously this is a trivial point, but just to be honest, it's
> the real reason why I initially assumed we'd put the test name in the
> scenario value: just cos that's how openQA is currently set up to do
> it, if you do it the easiest way :)
Ah, now we know! :-) Hmm, maybe we can take a different path here
then. We don't really need to standardize how scenario value looks.
We just need to say:
"the purpose of scenario key is to identify unique runs of your
test, when testcase+item combination is not enough. You should
include all extradata fields values that make this test run unique
(e.g. 'scenario=x86_64.uefi' for 'arch=x86_64' and
'firmware=uefi'
extradata)."
So the recommendation is clear. If you include the test name in
openqa results, it does not negatively affect the uniqueness
resolution. We just need to be clear in our documentation that in
order to recognize unique results, you always need to consider
testcase + item + scenario (if present).
Of course if someone takes inspiration from openqa results, he/she
might also include non-important strings (testcase name) in it, but
if we're clear in our documentation, that should not be a problem
(i.e. hopefully the consumer authors will read it). Would that work
for you?
I was kinda thinking the same thing, but at that point we're very close
to "it doesn't include the test case name". I really don't mind a lot
going with that, if you still think it's better. I can take the test
case name out of the openQA scenarios going forward easily enough.
> I can try and send a
> patch for Taskotron to do this as well, if you like, or would you
> rather do it?
I'll do it this week. Unless you want it in 24 hours, in that case
feel free to submit patches :)
This week is fine I think!
> > Btw, if you end up submitting patches to Bodhi and using
> > 'type=bodhi_update' queries, please also use the 'since='
argument
> > [2] and set it to the update's 'date_modified' timestamp. That
> > doesn't really implement the higher reliability of such results (as
> > we talked elsewhere), but it doesn't stress resultsdb so much, which
> > is always good (we can't do the same for type=koji_build easily, but
> > we can for type=bodhi_update).
>
> I'm planning to do this, but I did want to talk over with you guys what
> would be appropriate for the Taskotron results. Bodhi is definitely
> going to have to query for 'type=bodhi_update' results where the item
> is the update ID in order to find the openQA results. But I don't know
> if we should just make it find the Taskotron results in the same way,
> or if it's best to have it query for both koji_build and bodhi_update
> results and somehow deduplicate them (so it doesn't show Taskotron
> results which were reported against both the Koji build and the Bodhi
> update twice). But that's probably a separate thread.
Yep, let's talk about that separately.
Great, I'll ping you about it. There's also the question of whether we
'retrofit' scenarios into existing results, though...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net