> Another possibility would be to make it a convention to include
some
> kind of indication of the test 'scenarios' in the extradata for each
> result: a 'scenario' key, or something along those lines. This would
> make it much easier to include the 'item identifier' and 'test
> scenario' proper separately, and you could simply combine them when you
> needed the 'complete' scenario.
I'm not sure what do you mean exactly, but having a "scenario" key that
would
list all the other keys which are necessary to understand what makes this
scenario unique looks like a reasonable idea. For example:
scenario = [firmware_type, arch] # testcase name and item are implied
The downside is that task authors are required to provide this, and therefore
it's error-prone. I'm not sure how to do it better, though. We could set
some reasonable defaults for each type - so e.g. for koji_build type, we
know we compare testcase name + item (required to be nvr) + arch (if
present). For bodhi_update type, it would be testcase_name + item (required
to be Bodhi ID) + last_updated_timestamp. Etc. Anything above those defaults
would need to be in "scenario".
Also, I realized we could add the "scenario" key to the testcase object, not the
result object (if it's not possible now, we could add it). So it would be easy to just
retrieve the testcase info a figure out how to structure your query in order to get unique
results. Resultsdb would still stay a dumb storage, of course, this would be a consumer
logic.
>
> I'm trying to avoid consumers of ResultsDB data having to start
> learning about the details of individual test 'sources' in order to be
> able to perform this kind of de-duplication. It'd suck if releng had to
> learn the openQA 'scenario keys' concept directly, for instance, then
> learn corresponding smarts for any other system that submitted results.
Yes, that's definitely an important goal. Initially, I think, we didn't think
about this much. My idea was that each task would document its results
structure (it would become an API basically), and the tools would learn how
to consume that. I didn't expect to have too many important tasks that we
use for gating etc. But having common conventions is definitely easier.
Another use case I'm pondering right now is the new /latest endpoint in ResultsDB 2.0.
It should give you only the latest result for each available testcase, therefore sparing
you from performing expensive queries (either a lot of them, for each testcase, for
raising the time delta limit too much). Example:
http://taskotron-dev.fedoraproject.org/resultsdb_api/api/v2.0/results/lat...
But that currently breaks for depcheck, because it reports results separately for each
arch. So for some testcases, we need to be able to say "you must consider
'arch' when deciding uniqueness". You can of course add
"&arch=x86_64" to the /latest query, but a) how do you know all the possible
arch values? b) that won't give you any results which don't use 'arch'
keyword.
So, we need to come up with a way how to make this easy for consumers (so that they
don't need to know every single detail for each task, if possible), but still keep the
logic outside of the database (if we want to keep resultsdb data agnostic). I assume we
could be able to make that happen using reporting conventions, maybe even forcing them for
some (the important) tasks. Ideas?