10:13 p.m.
Hi folks! So rather than send a welcome mail I figured let's get right
into something real ;) I actually got the idea for this list because I
wrote up some thoughts on something, and realized there was nowhere
good to send it. So I invented this mailing list. And now I can send
it! Here it is.
I've been thinking some more about convenient consumption of ResultsDB
results. I'm thinking about the problem of generic handling of
duplicated / repeated tests.
Say Alice from release engineering wants to decide whether a given
deliverable is of release quality. With ResultsDB, Alice can query all
test results for that deliverable, regardless of where they came from.
Great. However, it's not uncommon for a test to have been repeated. Say
the test failed, the failure was investigated and determined to be a
bug in the test, and the test was repeated and passed. Both results
wind up in ResultsDB; we have a fail, then a pass.
How does Alice conveniently and *without special knowledge of the
system that submitted the result* identify the second result as being
for 'the same test' as the first, and thus know she can consider only
the second (most recent) result, and not worry about the failure? (I'm
expecting that to usually be the desired behaviour). There are also
other situations in which it's useful to be able to identify 'the same
test' for different executions; for instance, `check-compose` needs to
do this when it does its 'system information comparison' checks from
compose to compose.
I guess it's worth noting that this is somewhat related to the similar
question for test 'items' (the 'thing' being tested, in ResultsDB
parlance) - the question of uniquely identifying 'the same' item within
and across composes. At least for productmd 'images', myself and
lsedlar are currently discussing that in
https://pagure.io/pungi/issue/525 . Obviously it's more or less a
solved problem for RPMs.
I can think of two possible ways to handle this: via the extradata, or
via the test case name.
openQA has a useful concept here. It defines what combination of
metadata defines a unique test scenario like this, and calls it...well,
that - the 'scenario'. There's a constant definition called
SCENARIO_KEYS in openQA that you can use to discover the appropriate
keys. So I'm going to use the term 'scenario' for this from now on.
There's kinda two levels of scenario, now I think about it, depending
on whether you include 'item' identification in the scenario definition
or not. For identifying duplicates within the results for a single
item, you don't need to, but it doesn't hurt; for identifying the same
scenario across multiple composes, you do need to. I suppose someone
may have a case for identifying 'the same' test against different
items; for that purpose, you'd need the lighter 'scenario' definition
(not including the item identifier).
One thing we could do is make it a convention that each test case (and
/ or test case name?) indicates a test 'scenario' - such that all
results for the same test case for the same item should be considered
'duplicates' in this sense, and consumers can confidently count all
results for the same test case as results for the same test 'scenario'.
This seems to me like the simplest possibility, but I do see two
potential issues.
First, there's a possibility it may result in rather long and unwieldy
test case names in some situations. If we take the more complete
'scenario' definition and include sufficient information to uniquely
identify the item, The test case name for an openQA test that includes
sufficient information to uniquely identify the item under test may
look something like: `fedora.25.server-dvd-
iso.x86_64.install_default.uefi` (and that's with a fairly short test
name).
Second, it makes it difficult to handle the two different kinds of
'scenario' - i.e. it's not obvious how to split off the bits that
identify the 'item' from the bits that identify the 'test scenario'
proper. In this case the 'test scenario' is `install_default.uefi` and
the 'item identifier' is `fedora.25.server-dvd.iso.x86_64`, but there's
no real way to *know* that from the outside, unless we get into
defining separators, which always seems to be a losing game.
Another possibility would be to make it a convention to include some
kind of indication of the test 'scenarios' in the extradata for each
result: a 'scenario' key, or something along those lines. This would
make it much easier to include the 'item identifier' and 'test
scenario' proper separately, and you could simply combine them when you
needed the 'complete' scenario.
I'm trying to avoid consumers of ResultsDB data having to start
learning about the details of individual test 'sources' in order to be
able to perform this kind of de-duplication. It'd suck if releng had to
learn the openQA 'scenario keys' concept directly, for instance, then
learn corresponding smarts for any other system that submitted results.
Any thoughts on this? Any better ideas? Any existing work? Thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
8:36 a.m.
Hi folks! So rather than send a welcome mail I figured let's get
right
into something real ;) I actually got the idea for this list because I
wrote up some thoughts on something, and realized there was nowhere
good to send it. So I invented this mailing list. And now I can send
it! Here it is.
I've been thinking some more about convenient consumption of ResultsDB
results. I'm thinking about the problem of generic handling of
duplicated / repeated tests.
Say Alice from release engineering wants to decide whether a given
deliverable is of release quality. With ResultsDB, Alice can query all
test results for that deliverable, regardless of where they came from.
Great. However, it's not uncommon for a test to have been repeated. Say
the test failed, the failure was investigated and determined to be a
bug in the test, and the test was repeated and passed. Both results
wind up in ResultsDB; we have a fail, then a pass.
How does Alice conveniently and *without special knowledge of the
system that submitted the result* identify the second result as being
for 'the same test' as the first, and thus know she can consider only
the second (most recent) result, and not worry about the failure? (I'm
expecting that to usually be the desired behaviour).
Let's even complicate it more. Sometimes it's the same result repeated again (e.g.
new depcheck result with updated repo state), but sometimes it's a different tool
submitting the result for the "same" test case. For example, I assume this is
the reason why you chose "compose.base_selinux" testcase name instead of
"compose.openqa.base_selinux". The idea is that several tools can submit the
result for the same test case, so openqa, autocloud or even a manual tester can do it.
I'm not currently sold on this idea (sharing the testcase name instead of having
"compose.openqa.base_selinux", "compose.autocloud.base_selinux" and
"compose.manual.base_selinux"), but that seems to be the current state. So with
this, it's even harder to recognize whether we've received two results from openqa
(the latter superseding the former), or whether we received two results from two different
tools (and therefore we should consider both).
There are also
other situations in which it's useful to be able to identify 'the same
test' for different executions; for instance, `check-compose` needs to
do this when it does its 'system information comparison' checks from
compose to compose.
I guess it's worth noting that this is somewhat related to the similar
question for test 'items' (the 'thing' being tested, in ResultsDB
parlance) - the question of uniquely identifying 'the same' item within
and across composes. At least for productmd 'images', myself and
lsedlar are currently discussing that in
https://pagure.io/pungi/issue/525 .
We've discussed this with Josef a while back on qa-devel. We seemed to agree that item
should identify the thing under test well, even uniquely if possible, but stay simple. We
want to avoid having too many pieces of information concatenated into a single string,
just for the purpose of unique identification. Extra data should be used for that
(structured data, no string parsing). The tradeoff is that searching is a bit more
difficult (we'd need to allow users to also search by extra data in the frontend, and
they'd have to know what to search for).
For example, for git commits, we don't really like items like
"pagure#namespace/project#githash". Perhaps we could have just githash as item
(because it's almost unique identification even across many projects) and the rest as
extradata. This way we keep item simple, it's easy to search for manually, and
it's easy to search for automatically (no string parsing).
Obviously it's more or less a
solved problem for RPMs.
Almost. For upgradepath, yes, NVR uniquely identifies the result. For depcheck, NVR + arch
(from extradata) is unique identification.
I can think of two possible ways to handle this: via the extradata, or
via the test case name.
openQA has a useful concept here. It defines what combination of
metadata defines a unique test scenario like this, and calls it...well,
that - the 'scenario'. There's a constant definition called
SCENARIO_KEYS in openQA that you can use to discover the appropriate
keys. So I'm going to use the term 'scenario' for this from now on.
There's kinda two levels of scenario, now I think about it, depending
on whether you include 'item' identification in the scenario definition
or not. For identifying duplicates within the results for a single
item, you don't need to, but it doesn't hurt; for identifying the same
scenario across multiple composes, you do need to.
I don't follow here. In order to identify another execution of the same scenario, you
need at least testcase name and item to be exactly the same (and possibly also some
metadata). Do you have some examples to show it otherwise?
I suppose someone
may have a case for identifying 'the same' test against different
items; for that purpose, you'd need the lighter 'scenario' definition
(not including the item identifier).
I don't understand this at all, it seems to go against the intended meaning of
"item".
One thing we could do is make it a convention that each test case (and
/ or test case name?)
What's the difference between the two?
indicates a test 'scenario' - such that all
results for the same test case for the same item should be considered
'duplicates' in this sense, and consumers can confidently count all
results for the same test case as results for the same test 'scenario'.
That was our naive idea originally, until we got things that are not easily uniquely
identifiable via item only (if you don't want to make item a horrible compound of all
needed information).
This seems to me like the simplest possibility, but I do see two
potential issues.
First, there's a possibility it may result in rather long and unwieldy
test case names in some situations. If we take the more complete
'scenario' definition and include sufficient information to uniquely
identify the item, The test case name for an openQA test that includes
sufficient information to uniquely identify the item under test may
look something like: `fedora.25.server-dvd-
iso.x86_64.install_default.uefi` (and that's with a fairly short test
name).
Abomination! Let's not do that.
The purpose of testcase is to identify the steps that were taken to perform the test, or
the tool, or both. In case of rpmlint, both the tool and what gets done is the same
("the steps that rpmlint performs"), so the testcase name (excluding namespace)
is just "rpmlint". For openqa, it's a tool that performs many different test
cases, so it should create a separate testcase for each of them. For your example, it
should be "install_default", or better "openqa.install_default". I
wouldn't mind adding ".uefi" to the end, even though it's not clear
whether it's a sub-step result or part of the testcase (so maybe rather
"install_default_uefi"). But it would also make complete sense to make
"firmware_type" as part of the extradata and have just
"install_default".
We didn't want to impose any restrictions on what the tool produces (so everything
under "compose.openqa" is your playground, do whatever you wish), but of course
once the tool is important enough and we want to use it for gating or any other important
tasks, we need to recommend some approaches to make it report similarly to all other
important tasks, so that querying those results is not overly difficult. That could be
part of the resultsdb_conventions, I guess.
Second, it makes it difficult to handle the two different kinds of
'scenario' - i.e. it's not obvious how to split off the bits that
identify the 'item' from the bits that identify the 'test scenario'
proper. In this case the 'test scenario' is `install_default.uefi` and
the 'item identifier' is `fedora.25.server-dvd.iso.x86_64`, but there's
no real way to *know* that from the outside, unless we get into
defining separators, which always seems to be a losing game.
Yes, splitting strings is a bad idea.
Another possibility would be to make it a convention to include some
kind of indication of the test 'scenarios' in the extradata for each
result: a 'scenario' key, or something along those lines. This would
make it much easier to include the 'item identifier' and 'test
scenario' proper separately, and you could simply combine them when you
needed the 'complete' scenario.
I'm not sure what do you mean exactly, but having a "scenario" key that
would list all the other keys which are necessary to understand what makes this scenario
unique looks like a reasonable idea. For example:
scenario = [firmware_type, arch] # testcase name and item are implied
The downside is that task authors are required to provide this, and therefore it's
error-prone. I'm not sure how to do it better, though. We could set some reasonable
defaults for each type - so e.g. for koji_build type, we know we compare testcase name +
item (required to be nvr) + arch (if present). For bodhi_update type, it would be
testcase_name + item (required to be Bodhi ID) + last_updated_timestamp. Etc. Anything
above those defaults would need to be in "scenario".
I'm trying to avoid consumers of ResultsDB data having to start
learning about the details of individual test 'sources' in order to be
able to perform this kind of de-duplication. It'd suck if releng had to
learn the openQA 'scenario keys' concept directly, for instance, then
learn corresponding smarts for any other system that submitted results.
Yes, that's definitely an important goal. Initially, I think, we didn't think
about this much. My idea was that each task would document its results structure (it would
become an API basically), and the tools would learn how to consume that. I didn't
expect to have too many important tasks that we use for gating etc. But having common
conventions is definitely easier.
10:47 a.m.
> Another possibility would be to make it a convention to include
some
> kind of indication of the test 'scenarios' in the extradata for each
> result: a 'scenario' key, or something along those lines. This would
> make it much easier to include the 'item identifier' and 'test
> scenario' proper separately, and you could simply combine them when you
> needed the 'complete' scenario.
I'm not sure what do you mean exactly, but having a "scenario" key that
would
list all the other keys which are necessary to understand what makes this
scenario unique looks like a reasonable idea. For example:
scenario = [firmware_type, arch] # testcase name and item are implied
The downside is that task authors are required to provide this, and therefore
it's error-prone. I'm not sure how to do it better, though. We could set
some reasonable defaults for each type - so e.g. for koji_build type, we
know we compare testcase name + item (required to be nvr) + arch (if
present). For bodhi_update type, it would be testcase_name + item (required
to be Bodhi ID) + last_updated_timestamp. Etc. Anything above those defaults
would need to be in "scenario".
Also, I realized we could add the "scenario" key to the testcase object, not the
result object (if it's not possible now, we could add it). So it would be easy to just
retrieve the testcase info a figure out how to structure your query in order to get unique
results. Resultsdb would still stay a dumb storage, of course, this would be a consumer
logic.
>
> I'm trying to avoid consumers of ResultsDB data having to start
> learning about the details of individual test 'sources' in order to be
> able to perform this kind of de-duplication. It'd suck if releng had to
> learn the openQA 'scenario keys' concept directly, for instance, then
> learn corresponding smarts for any other system that submitted results.
Yes, that's definitely an important goal. Initially, I think, we didn't think
about this much. My idea was that each task would document its results
structure (it would become an API basically), and the tools would learn how
to consume that. I didn't expect to have too many important tasks that we
use for gating etc. But having common conventions is definitely easier.
Another use case I'm pondering right now is the new /latest endpoint in ResultsDB 2.0.
It should give you only the latest result for each available testcase, therefore sparing
you from performing expensive queries (either a lot of them, for each testcase, for
raising the time delta limit too much). Example:
http://taskotron-dev.fedoraproject.org/resultsdb_api/api/v2.0/results/lat...
But that currently breaks for depcheck, because it reports results separately for each
arch. So for some testcases, we need to be able to say "you must consider
'arch' when deciding uniqueness". You can of course add
"&arch=x86_64" to the /latest query, but a) how do you know all the possible
arch values? b) that won't give you any results which don't use 'arch'
keyword.
So, we need to come up with a way how to make this easy for consumers (so that they
don't need to know every single detail for each task, if possible), but still keep the
logic outside of the database (if we want to keep resultsdb data agnostic). I assume we
could be able to make that happen using reporting conventions, maybe even forcing them for
some (the important) tasks. Ideas?
11:42 a.m.
On Wed, 2017-03-01 at 11:47 -0500, Kamil Paral wrote:
Also, I realized we could add the "scenario" key to the
testcase
object, not the result object (if it's not possible now, we could add
it). So it would be easy to just retrieve the testcase info a figure
out how to structure your query in order to get unique results.
Resultsdb would still stay a dumb storage, of course, this would be a
consumer logic.
Yeah, I think that was one thing I was thinking of, but I'm not sure I
really like it. I think I kinda prefer just having an actual 'scenario'
*value* for each result.
Another use case I'm pondering right now is the new /latest
endpoint
in ResultsDB 2.0. It should give you only the latest result for each
available testcase, therefore sparing you from performing expensive
queries (either a lot of them, for each testcase, for raising the
time delta limit too much). Example:
http://taskotron-dev.fedoraproject.org/resultsdb_api/api/v2.0/results
/latest?item=netpbm-10.77.00-3.fc24
But that currently breaks for depcheck, because it reports results
separately for each arch. So for some testcases, we need to be able
to say "you must consider 'arch' when deciding uniqueness". You can
of course add "&arch=x86_64" to the /latest query, but a) how do you
know all the possible arch values? b) that won't give you any results
which don't use 'arch' keyword.
So, we need to come up with a way how to make this easy for consumers
(so that they don't need to know every single detail for each task,
if possible), but still keep the logic outside of the database (if we
want to keep resultsdb data agnostic). I assume we could be able to
make that happen using reporting conventions, maybe even forcing them
for some (the important) tasks. Ideas?
Well, yeah, this is really *exactly* the same case: you're just talking
about the 'scenario' for depcheck tests. My suggestion is that depcheck
tests should simply have an extradata value like:
scenario: depcheck.x86_64
Note that this kind of thing is *exactly* why openQA came up with the
'scenario' concept; they realized they kept reinventing this "well, is
job X 'the same as' job Y?" logic in different contexts (the web UI
code, API queries, a bunch of different places). They have exactly your
use case, among others - a /latest endpoint. So they realized the
concept needed to be defined and shared between all those places, and
came up with the 'scenario' naming.
I hope you can see that we're really in the same use case? With my
idea, your 'latest' endpoint would simply look for all results for the
specified item then de-duplicate them by scenario. It wouldn't need to
consider any other metadata at all, just the scenario would be enough.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
11:36 a.m.
On Wed, 2017-03-01 at 09:36 -0500, Kamil Paral wrote:
>
Let's even complicate it more. Sometimes it's the same result
repeated again (e.g. new depcheck result with updated repo state),
but sometimes it's a different tool submitting the result for the
"same" test case. For example, I assume this is the reason why you
chose "compose.base_selinux" testcase name instead of
"compose.openqa.base_selinux". The idea is that several tools can
submit the result for the same test case, so openqa, autocloud or
even a manual tester can do it. I'm not currently sold on this idea
(sharing the testcase name instead of having
"compose.openqa.base_selinux", "compose.autocloud.base_selinux" and
"compose.manual.base_selinux"), but that seems to be the current
state. So with this, it's even harder to recognize whether we've
received two results from openqa (the latter superseding the former),
or whether we received two results from two different tools (and
therefore we should consider both).
Well, I wasn't really thinking about having multiple test systems
running 'the same' test concurrently. I was only thinking about the
possibility of moving tests between systems. base_selinux is a fairly
good example, as it's a trivial test that isn't terribly tied to
openQA: 'boot a freshly installed system and check SELinux is enabled'.
We certainly might, at some point, move that test from openQA to some
other system. I was envisaging that if we can say with a high degree of
confidence that the new system is truly performing 'the same test', we
could just 'transfer' the name rather than naming it differently.
Of course, now I combine that thought with this one, it adds an
interesting wrinkle, because if we *do* move tests between systems in
this way, the 'scenario' is likely to differ between the systems. The
'scenario' for an openQA 'base_selinux' test probably wouldn't be the
same as the 'scenario' for a Taskotron 'base_selinux' test. Not sure if
that's a case it's worth worrying about, though.
It was, like, a five minute thing, though. I hadn't thought it through
that systematically. I'm actually not sure if even the 'compose.'
prefix was a good idea, now.
> There are also
> other situations in which it's useful to be able to identify 'the same
> test' for different executions; for instance, `check-compose` needs to
> do this when it does its 'system information comparison' checks from
> compose to compose.
>
> I guess it's worth noting that this is somewhat related to the similar
> question for test 'items' (the 'thing' being tested, in ResultsDB
> parlance) - the question of uniquely identifying 'the same' item within
> and across composes. At least for productmd 'images', myself and
> lsedlar are currently discussing that in
> https://pagure.io/pungi/issue/525 .
We've discussed this with Josef a while back on qa-devel. We seemed
to agree that item should identify the thing under test well, even
uniquely if possible, but stay simple. We want to avoid having too
many pieces of information concatenated into a single string, just
for the purpose of unique identification. Extra data should be used
for that (structured data, no string parsing). The tradeoff is that
searching is a bit more difficult (we'd need to allow users to also
search by extra data in the frontend, and they'd have to know what to
search for).
For example, for git commits, we don't really like items like
"pagure#namespace/project#githash". Perhaps we could have just
githash as item (because it's almost unique identification even
across many projects) and the rest as extradata. This way we keep
item simple, it's easy to search for manually, and it's easy to
search for automatically (no string parsing).
> Obviously it's more or less a
> solved problem for RPMs.
Almost. For upgradepath, yes, NVR uniquely identifies the result. For
depcheck, NVR + arch (from extradata) is unique identification.
>
> I can think of two possible ways to handle this: via the extradata, or
> via the test case name.
>
> openQA has a useful concept here. It defines what combination of
> metadata defines a unique test scenario like this, and calls it...well,
> that - the 'scenario'. There's a constant definition called
> SCENARIO_KEYS in openQA that you can use to discover the appropriate
> keys. So I'm going to use the term 'scenario' for this from now on.
>
> There's kinda two levels of scenario, now I think about it, depending
> on whether you include 'item' identification in the scenario definition
> or not. For identifying duplicates within the results for a single
> item, you don't need to, but it doesn't hurt; for identifying the same
> scenario across multiple composes, you do need to.
I don't follow here. In order to identify another execution of the
same scenario, you need at least testcase name and item to be exactly
the same (and possibly also some metadata). Do you have some examples
to show it otherwise?
Well, this is all about that "possibly also some metadata". *What*
metadata? How do you, some random releng (or whatever) person trying to
consume arbitrary ResultsDB data, know *what* "possibly also some"
metadata you need to look at to identify 'duplicate' results?
As of right now you have to come ask someone and we say "well, for
depcheck tests do foo, for upgradepath tests do bar, for openQA tests
do moo..."
I'm trying to fix that.
So concrete examples, okay. Here's two openQA test results:
https://taskotron.fedoraproject.org/resultsdb/results/12461058
https://taskotron.fedoraproject.org/resultsdb/results/12460886
they are both results for testcase 'compose.install_ext3' on item
'Fedora-Server-dvd-x86_64-Rawhide-20170228.n.0.iso' . In this case,
they even have the same arch - 'x86_64'. Does this mean they're dupes
(i.e. the test got restarted for some reason)? No. They're actually
different tests; one was run on a BIOS VM, one on a UEFI VM. In other
words, the 'machine' (in openQA terms) is part of the 'scenario' for
openQA tests. But this is hardly a universal rule; I can't just throw
openQA's 'machine' setting into the results and tell everyone trying to
de-duplicate ResultsDB results to look for the 'machine' value.
The idea is just to be able to say, not "you have to look for same test
case and item and possibly some metadata", but "you have to look for
same test case and item and 'scenario' metadata item".
> I suppose someone
> may have a case for identifying 'the same' test against different
> items; for that purpose, you'd need the lighter 'scenario' definition
> (not including the item identifier).
I don't understand this at all, it seems to go against the intended
meaning of "item".
I just mean, say you want to look at all the results for 'the same'
test but for different tested items; I want to look at the last three
weeks worth of all x86_64 BIOS compose.install_ext3 tests, or something
like that. In that case, your 'scenario' does not include the 'item'.
>
> One thing we could do is make it a convention that each test case (and
> / or test case name?)
What's the difference between the two?
I, uh, honestly don't remember what distinction I was trying to draw
there :/. I think it was about the fact that a 'test case' to ResultsDB
is a more complex item than just a name - it has a URL and stuff - so
the 'scenario' properties could possibly be included in something other
than just the test case name.
> Another possibility would be to make it a convention to include
some
> kind of indication of the test 'scenarios' in the extradata for each
> result: a 'scenario' key, or something along those lines. This would
> make it much easier to include the 'item identifier' and 'test
> scenario' proper separately, and you could simply combine them when you
> needed the 'complete' scenario.
I'm not sure what do you mean exactly, but having a "scenario" key
that would list all the other keys which are necessary to understand
what makes this scenario unique looks like a reasonable idea. For
example:
scenario = [firmware_type, arch] # testcase name and item are implied
Well, it's a simpler idea than that: just include a key that has all
the necessary *values*. The indirection of having a key that tells you
what other keys to go look up just seems unnecessarily complex. The
idea was simply that there'd be an item like this in the metadata:
scenario: fedora.Rawhide.Server-dvd-iso.x86_64.server_realmd_join_kickstart.64bit
That's an actual openQA scenario: DISTRI.VERSION.FLAVOR.ARCH.TESTCASE.MACHINE
Then you can look up 'all results for same item, same scenario' (for
de-duplication) or 'all results for same scenario' (to compare results
for "the same test" across different composes).
I have a diff in right now that would add this to the openQA reporter:
https://phab.qa.fedoraproject.org/D1155
since we kinda need it right now for the update stuff (so Bodhi can
find the correct results to display).
The downside is that task authors are required to provide this, and
therefore it's error-prone. I'm not sure how to do it better, though.
We could set some reasonable defaults for each type - so e.g. for
koji_build type, we know we compare testcase name + item (required to
be nvr) + arch (if present). For bodhi_update type, it would be
testcase_name + item (required to be Bodhi ID) +
last_updated_timestamp. Etc. Anything above those defaults would need
to be in "scenario".
Eh, I dunno about having defaults by type. Though I was thinking
that 'testcase name' is kinda the implicit default; if a result doesn't
have a 'scenario' item at all, then just assume you should use
'testcase name'. For any case where it's more complex than that, the
system that submits the results should provide the scenario info (so
Taskotron should add a 'scenario' key like 'TESTCASE_NAME.ARCH' for
Koji results.
Note I don't think we should include the 'item' in the scenario for all
the reasons discussed above; it should just be understood that for
different purposes you might want to query on "scenario + item" or just
"scenario".
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
9:16 a.m.
> Let's even complicate it more. Sometimes it's the same
result
> repeated again (e.g. new depcheck result with updated repo state),
> but sometimes it's a different tool submitting the result for the
> "same" test case. For example, I assume this is the reason why you
> chose "compose.base_selinux" testcase name instead of
> "compose.openqa.base_selinux". The idea is that several tools can
> submit the result for the same test case, so openqa, autocloud or
> even a manual tester can do it. I'm not currently sold on this idea
> (sharing the testcase name instead of having
> "compose.openqa.base_selinux", "compose.autocloud.base_selinux"
and
> "compose.manual.base_selinux"), but that seems to be the current
> state. So with this, it's even harder to recognize whether we've
> received two results from openqa (the latter superseding the former),
> or whether we received two results from two different tools (and
> therefore we should consider both).
Well, I wasn't really thinking about having multiple test systems
running 'the same' test concurrently. I was only thinking about the
possibility of moving tests between systems. base_selinux is a fairly
good example, as it's a trivial test that isn't terribly tied to
openQA: 'boot a freshly installed system and check SELinux is enabled'.
We certainly might, at some point, move that test from openQA to some
other system. I was envisaging that if we can say with a high degree of
confidence that the new system is truly performing 'the same test', we
could just 'transfer' the name rather than naming it differently.
It seems to me that you're envisioning a system where we all work together, we have a
shared definition of test cases (so that base_selinux means the same, whatever tool it is
implemented in), and we all know what we're doing (using the conventions, being
up-to-date to latest test structure and extradata fields, etc). I'm envisioning a
total chaos and a mess :) , with *some* groups sharing the same conventions and
approaches, and other groups or people using the system in their own way.
So let's say there's a person who has a script to check composes in some
interesting way, and later makes some conclusions or stats from it. As long as it is his
personal hobby, he'll probably use user.<fasname> namespace. If other people
find that results interesting, we might make it more official and move it to
`compose.<toolname>` namespace. The important thing is that his results don't
clash with our results, be it testcase names, or extradata, or whether he uses conventions
or not. Only once we decide to use his results for gating, we might need to enforce some
common conventions to be used. Until that time, he's free to use whatever results
structure he wishes, and the consumers need to accommodate. Free for all space.
That's why I'm not completely happy about "polluting" the top level
`compose` namespace, because it means it's now completely reserved to openqa (or other
tools tightly collaborating with it, sharing the test cases, etc). I'd rather create
`compose.openqa`, or maybe `compose.relval` or `compose.qa` (to have a generic name for
release validation performed by us), and everything in there would be openqa's +
related tools' playground. With this approach, we could create `compose.othertool`
namespace and not worry about naming clashes.
I don't really object to sharing testcase names, if it makes sense to you. So
`compose.qa.base_selinux` could be either submitted using openqa, or autocloud, or
manually, or all of them. An alternative approach is to have
`compose.openqa.base_selinux`, `compose.autocloud.base_selinux` and
`compose.manual.base_selinux`, which has different advantages and disadvantages. I
don't care that much, the important part for me is access separation, so that if I
decide to write a new task, I don't really need to care about what testcase names
other tools use, I don't need to sync up with anyone, I don't need to be worried
that I overwrite something important or screw something up, I have a space that is
exclusive to me.
> >
> > I can think of two possible ways to handle this: via the extradata, or
> > via the test case name.
> >
> > openQA has a useful concept here. It defines what combination of
> > metadata defines a unique test scenario like this, and calls it...well,
> > that - the 'scenario'. There's a constant definition called
> > SCENARIO_KEYS in openQA that you can use to discover the appropriate
> > keys. So I'm going to use the term 'scenario' for this from now
on.
> >
> > There's kinda two levels of scenario, now I think about it, depending
> > on whether you include 'item' identification in the scenario
definition
> > or not. For identifying duplicates within the results for a single
> > item, you don't need to, but it doesn't hurt; for identifying the same
> > scenario across multiple composes, you do need to.
>
> I don't follow here. In order to identify another execution of the
> same scenario, you need at least testcase name and item to be exactly
> the same (and possibly also some metadata). Do you have some examples
> to show it otherwise?
Well, this is all about that "possibly also some metadata". *What*
metadata? How do you, some random releng (or whatever) person trying to
consume arbitrary ResultsDB data, know *what* "possibly also some"
metadata you need to look at to identify 'duplicate' results?
As of right now you have to come ask someone and we say "well, for
depcheck tests do foo, for upgradepath tests do bar, for openQA tests
do moo..."
I'm trying to fix that.
Yes, that all sounds reasonable. I didn't get your last paragraph, because in my view
you always need 'item' to identify duplicates. But after reading it again, it
seems there's no difference, just choice of words. If you say "For identifying
duplicates within the results for a single item, you don't need to", you actually
just used 'item', so it matches my claim that you always need it.
So concrete examples, okay. Here's two openQA test results:
https://taskotron.fedoraproject.org/resultsdb/results/12461058
https://taskotron.fedoraproject.org/resultsdb/results/12460886
they are both results for testcase 'compose.install_ext3' on item
'Fedora-Server-dvd-x86_64-Rawhide-20170228.n.0.iso' . In this case,
they even have the same arch - 'x86_64'. Does this mean they're dupes
(i.e. the test got restarted for some reason)? No. They're actually
different tests; one was run on a BIOS VM, one on a UEFI VM. In other
words, the 'machine' (in openQA terms) is part of the 'scenario' for
openQA tests. But this is hardly a universal rule; I can't just throw
openQA's 'machine' setting into the results and tell everyone trying to
de-duplicate ResultsDB results to look for the 'machine' value.
You can, but it requires heavy logic at the consumer side, and it's pain to maintain.
We could supply a library to do that (here's a new thought). But I agree it would be
nice to make it simpler ootb, if we can.
The idea is just to be able to say, not "you have to look for same test
case and item and possibly some metadata", but "you have to look for
same test case and item and 'scenario' metadata item".
> > I suppose someone
> > may have a case for identifying 'the same' test against different
> > items; for that purpose, you'd need the lighter 'scenario'
definition
> > (not including the item identifier).
>
> I don't understand this at all, it seems to go against the intended
> meaning of "item".
I just mean, say you want to look at all the results for 'the same'
test but for different tested items; I want to look at the last three
weeks worth of all x86_64 BIOS compose.install_ext3 tests, or something
like that. In that case, your 'scenario' does not include the 'item'.
Do I need scenario in this case at all? If I know exactly what I want, why wouldn't I
query testcase=compose.install_ext3&arch=x86_64&firmware=bios ?
I understand you want to introduce scenario for cases when the consumer doesn't want
to have particular knowledge of that system. If the consumer has deep knowledge of this,
what is scenario good for?
> >
> > One thing we could do is make it a convention that each test case (and
> > / or test case name?)
>
> What's the difference between the two?
I, uh, honestly don't remember what distinction I was trying to draw
there :/. I think it was about the fact that a 'test case' to ResultsDB
is a more complex item than just a name - it has a URL and stuff - so
the 'scenario' properties could possibly be included in something other
than just the test case name.
> > Another possibility would be to make it a convention to include some
> > kind of indication of the test 'scenarios' in the extradata for each
> > result: a 'scenario' key, or something along those lines. This would
> > make it much easier to include the 'item identifier' and 'test
> > scenario' proper separately, and you could simply combine them when you
> > needed the 'complete' scenario.
>
> I'm not sure what do you mean exactly, but having a "scenario" key
> that would list all the other keys which are necessary to understand
> what makes this scenario unique looks like a reasonable idea. For
> example:
> scenario = [firmware_type, arch] # testcase name and item are implied
Well, it's a simpler idea than that: just include a key that has all
the necessary *values*. The indirection of having a key that tells you
what other keys to go look up just seems unnecessarily complex. The
idea was simply that there'd be an item like this in the metadata:
scenario:
fedora.Rawhide.Server-dvd-iso.x86_64.server_realmd_join_kickstart.64bit
That's an actual openQA scenario: DISTRI.VERSION.FLAVOR.ARCH.TESTCASE.MACHINE
Then you can look up 'all results for same item, same scenario' (for
de-duplication) or 'all results for same scenario' (to compare results
for "the same test" across different composes).
I guess I need query examples here. I can imagine three options:
a) query
`results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-Rawhide-20170228.n.0.iso`
and then go through all the results, make them unique by eliminating everything that has
the same 'scenario', and work with that
b) query
`results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-Rawhide-20170228.n.0.iso&unique_key=scenario`
and have the duplication filtering implemented inside resultsdb (this way it seems it can
be implemented in a generic fashion), work with that
c) query
`results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-Rawhide-20170228.n.0.iso&scenario=fedora.Rawhide.Server-dvd-iso.x86_64.server_realmd_join_kickstart.64bit`
and work with that. But in this case the consumer would again have to have a deep
knowledge of how the scenario is constructed, which would miss the whole point, I think.
So please specify how exactly you imagine this working from the consumer POV.
I have a diff in right now that would add this to the openQA reporter:
https://phab.qa.fedoraproject.org/D1155
since we kinda need it right now for the update stuff (so Bodhi can
find the correct results to display).
> The downside is that task authors are required to provide this, and
> therefore it's error-prone. I'm not sure how to do it better, though.
> We could set some reasonable defaults for each type - so e.g. for
> koji_build type, we know we compare testcase name + item (required to
> be nvr) + arch (if present). For bodhi_update type, it would be
> testcase_name + item (required to be Bodhi ID) +
> last_updated_timestamp. Etc. Anything above those defaults would need
> to be in "scenario".
Eh, I dunno about having defaults by type. Though I was thinking
that 'testcase name' is kinda the implicit default; if a result doesn't
have a 'scenario' item at all, then just assume you should use
'testcase name'. For any case where it's more complex than that, the
system that submits the results should provide the scenario info (so
Taskotron should add a 'scenario' key like 'TESTCASE_NAME.ARCH' for
Koji results.
Note I don't think we should include the 'item' in the scenario for all
the reasons discussed above; it should just be understood that for
different purposes you might want to query on "scenario + item" or just
"scenario".
I don't understand why you would ever want to put testcase name into the scenario
field. Both the item and the scenario are defined externally in a well known fields that
are mandatory to fill in. There's no guessing like with 'arch' or
'firmware' (whether it exists, what the allowed values are). So why wouldn't
you specify only the extra args that make the execution unique in the scenario field? If
you ever need to search through multiple testcases with the same scenario value (I
don't know why you would need that, but let's assume you do), you can always do
that either for a particular testcase like this:
results?testcases=compose.base_selinux&scenario=fedora.Rawhide.Server-dvd-iso.x86_64.64bit
or across many testcases by using a list or a wildcard:
results?testcases=compose.*&scenario=fedora.Rawhide.Server-dvd-iso.x86_64.64bit
I don't think you even want to do this query:
results?scenario=foobar
because you can *bet* that you'll eventually receive some results from
scratch.tmp.testthis.fake namespace or similar. You should always specify at least a
wildcarded testcase (compose.openqa.*). I'd like our system to allow in-development,
proof-of-concept and experimental tools to run and submit results. There will be errors,
mistakes, copycats, everything. But they will be constrained to their namespaces. And so
should you also limit your queries only to those namespaces that are known and maintained
with the level of quality you need for your purposes. You can imagine the concept to be
the same as e.g. github - you always work in your namespace, and there's no global
namespace, because that's just asking for trouble.
Of course, those are all just my visions of how the project should work. Josef often
disagrees with me. If you think some of that should work differently, please disagree with
me as well :)
8:08 p.m.
On Thu, 2017-03-02 at 10:16 -0500, Kamil Paral wrote:
> > It seems to me that you're envisioning a system where
we all work together,
<snip>
I'm gonna just cut off this part of the discussion, as it's really not
related to the 'scenario' topic at all. I don't really have a strong
opinion on how much test case names should be tied to test case systems
or anything like that, everything you said seems fine to me, and I
don't really mind adjusting the openQA test case names if we want to
change them.
Yes, that all sounds reasonable. I didn't get your last
paragraph,
because in my view you always need 'item' to identify duplicates. But
after reading it again, it seems there's no difference, just choice
of words. If you say "For identifying duplicates within the results
for a single item, you don't need to", you actually just used 'item',
so it matches my claim that you always need it.
Yes, if you're trying to identify duplicate tests, of course you need
the item. I was just noting that you can actually use the 'scenario'
concept for cases beyond simply identifying duplicates. You can also
use it to compare results from 'the same' test across multiple items.
>
> So concrete examples, okay. Here's two openQA test results:
>
> https://taskotron.fedoraproject.org/resultsdb/results/12461058
> https://taskotron.fedoraproject.org/resultsdb/results/12460886
>
> they are both results for testcase 'compose.install_ext3' on item
> 'Fedora-Server-dvd-x86_64-Rawhide-20170228.n.0.iso' . In this case,
> they even have the same arch - 'x86_64'. Does this mean they're dupes
> (i.e. the test got restarted for some reason)? No. They're actually
> different tests; one was run on a BIOS VM, one on a UEFI VM. In other
> words, the 'machine' (in openQA terms) is part of the 'scenario'
for
> openQA tests. But this is hardly a universal rule; I can't just throw
> openQA's 'machine' setting into the results and tell everyone trying to
> de-duplicate ResultsDB results to look for the 'machine' value.
You can, but it requires heavy logic at the consumer side, and it's
pain to maintain. We could supply a library to do that (here's a new
thought). But I agree it would be nice to make it simpler ootb, if we
can.
Yeah, a library is a possibility, but I think I kinda prefer the idea
of a scenario convention, at least until we find out it doesn't really
work, or something. :P
>
> The idea is just to be able to say, not "you have to look for same test
> case and item and possibly some metadata", but "you have to look for
> same test case and item and 'scenario' metadata item".
>
> > > I suppose someone
> > > may have a case for identifying 'the same' test against different
> > > items; for that purpose, you'd need the lighter 'scenario'
definition
> > > (not including the item identifier).
> >
> > I don't understand this at all, it seems to go against the intended
> > meaning of "item".
>
> I just mean, say you want to look at all the results for 'the same'
> test but for different tested items; I want to look at the last three
> weeks worth of all x86_64 BIOS compose.install_ext3 tests, or something
> like that. In that case, your 'scenario' does not include the
'item'.
Do I need scenario in this case at all? If I know exactly what I
want, why wouldn't I query
testcase=compose.install_ext3&arch=x86_64&firmware=bios ?
I understand you want to introduce scenario for cases when the
consumer doesn't want to have particular knowledge of that system. If
the consumer has deep knowledge of this, what is scenario good for?
Maybe you *don't* already know exactly what you want, though. Imagine
you happen to have come across some specific result and you're
interested in comparing it to results for 'the same' test for other
items, for instance. Or imagine you just want to take *all* the results
for, say, ten composes, and compare them all; the 'scenario' concept
lets you know which result from compose A you should compare with which
result from compose B.
> Well, it's a simpler idea than that: just include a key that
has all
> the necessary *values*. The indirection of having a key that tells you
> what other keys to go look up just seems unnecessarily complex. The
> idea was simply that there'd be an item like this in the metadata:
>
> scenario:
> fedora.Rawhide.Server-dvd-iso.x86_64.server_realmd_join_kickstart.64bit
>
> That's an actual openQA scenario: DISTRI.VERSION.FLAVOR.ARCH.TESTCASE.MACHINE
>
> Then you can look up 'all results for same item, same scenario' (for
> de-duplication) or 'all results for same scenario' (to compare results
> for "the same test" across different composes).
I guess I need query examples here. I can imagine three options:
a) query `results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
Rawhide-20170228.n.0.iso` and then go through all the results, make
them unique by eliminating everything that has the same 'scenario',
and work with that
This was one of the primary cases I was thinking of, yes. This is the
case that releng would be using to decide whether to release a compose,
for instance. It's also the case for listing openQA results in Bodhi:
we'll want Bodhi to grab all results for the update, then de-duplicate
by scenario (taking only the most recent result for each scenario).
b) query
`results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
Rawhide-20170228.n.0.iso&unique_key=scenario` and have the
duplication filtering implemented inside resultsdb (this way it seems
it can be implemented in a generic fashion), work with that
Oo, is that possible? If so, neat. But conceptually it's the same as
a), right? Just a case of where the filtering gets done.
c) query
`results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
Rawhide-20170228.n.0.iso&scenario=fedora.Rawhide.Server-dvd-
iso.x86_64.server_realmd_join_kickstart.64bit` and work with that.
But in this case the consumer would again have to have a deep
knowledge of how the scenario is constructed, which would miss the
whole point, I think.
Well, see the above examples - I was kinda envisioning that you'd
*discover* the scenario you're interested in from some kind of existing
result, or just query for all results from two *different* composes,
then use the 'scenario' key to compare the results for the two composes
(so you can say 'ok, scenario A passed in compose 1 but failed in
compose 2'). This is the sort of thing check-compose does - that's how
it produces a list of 'tests that failed today but passed yesterday',
etc. It doesn't *know* what the scenarios are before it queries the
results, but *once it has the results* it calculates the scenarios and
uses them for comparison.
> Note I don't think we should include the 'item' in
the scenario for all
> the reasons discussed above; it should just be understood that for
> different purposes you might want to query on "scenario + item" or just
> "scenario".
I don't understand why you would ever want to put testcase name into
the scenario field. Both the item and the scenario are defined
externally in a well known fields that are mandatory to fill in.
There's no guessing like with 'arch' or 'firmware' (whether it
exists, what the allowed values are). So why wouldn't you specify
only the extra args that make the execution unique in the scenario
field? If you ever need to search through multiple testcases with the
same scenario value (I don't know why you would need that, but let's
assume you do), you can always do that either for a particular
testcase like this:
results?testcases=compose.base_selinux&scenario=fedora.Rawhide.Server
-dvd-iso.x86_64.64bit
or across many testcases by using a list or a wildcard:
results?testcases=compose.*&scenario=fedora.Rawhide.Server-dvd-
iso.x86_64.64bit
I don't think you even want to do this query:
results?scenario=foobar
because you can *bet* that you'll eventually receive some results
from scratch.tmp.testthis.fake namespace or similar. You should
always specify at least a wildcarded testcase (compose.openqa.*). I'd
like our system to allow in-development, proof-of-concept and
experimental tools to run and submit results. There will be errors,
mistakes, copycats, everything. But they will be constrained to their
namespaces. And so should you also limit your queries only to those
namespaces that are known and maintained with the level of quality
you need for your purposes. You can imagine the concept to be the
same as e.g. github - you always work in your namespace, and there's
no global namespace, because that's just asking for trouble.
Well, the funny thing is, for all the same reasons I come to the other
conclusion :) I entirely agree that you'd never want to use the
scenario *without* the test case name...so in that case, saying "you
should always query for the test case name plus the scenario" seems
like pointless make-work, right? If we know that there's never any
point in just using the scenario values *without* the test case name,
why not just stuff the test case name right into the scenario value so
you only have to query for 'scenario', rather than always having to
query for both 'test case name' and 'scenario'?
But it's really a pretty minor point. The system would work fine either
way, it just slightly changes what consumers have to do.
Does anyone else have thoughts on this? I'd like to either land the
'scenario' change in the openQA reporter, or definitely decide we want
to do something else, tomorrow or early next week, so I can move
forward with the 'is this compose releasable?' script and displaying
openQA update test results in Bodhi...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
5:12 p.m.
On Thu, 2017-03-02 at 18:08 -0800, Adam Williamson wrote:
It's also the case for listing openQA results in Bodhi:
we'll want Bodhi to grab all results for the update, then de-duplicate
by scenario (taking only the most recent result for each scenario).
Just as a quick note on this - I actually got around to looking at what
Bodhi is *currently* doing here today, and the answer is it's pretty
much inventing the 'scenario' concept on the client end:
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/templates...
That is, it uses a nested dict for de-duplication, and assumes all the
results it cares about will have 'test name', 'arch' and 'item'
properties and effectively treats the combination of those three as the
'scenario' for its purposes.
Of course, that wouldn't be safe for openQA results (it'd wind up
considering UEFI and BIOS tests to be dupes, for instance), so if we
were to extend Bodhi to handle openQA results without introducing a
'scenario' concept, I'd have to basically teach Bodhi the openQA
scenario keys. Then if we wanted Bodhi to care about results from some
other system whose scenario was different, we'd have to teach it the
appropriate scenario values for *that* system too and so on. So, I
think this clearly demonstrates the value of the concept :)
If we implement a 'scenario' key, we can then just teach Bodhi to look
for that and combine it with the item, then maybe if it can't find one,
treat 'testname-arch' as the scenario (as a fallback for old results,
results from Taskotron until we teach it to write 'scenario' items into
its results, and a general fallback for cases where 'scenario' is
missing for some reason). That could be the general convention, I
guess: first look for a 'scenario' key, otherwise try and construct one
from very commonly available keys.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
9:25 a.m.
On Fri, Mar 03, 2017 at 03:12:22PM -0800, Adam Williamson wrote:
On Thu, 2017-03-02 at 18:08 -0800, Adam Williamson wrote:
> It's also the case for listing openQA results in Bodhi:
> we'll want Bodhi to grab all results for the update, then de-duplicate
> by scenario (taking only the most recent result for each scenario).
Just as a quick note on this - I actually got around to looking at what
Bodhi is *currently* doing here today, and the answer is it's pretty
much inventing the 'scenario' concept on the client end:
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/templates...
That is, it uses a nested dict for de-duplication, and assumes all the
results it cares about will have 'test name', 'arch' and 'item'
properties and effectively treats the combination of those three as the
'scenario' for its purposes.
Of course, that wouldn't be safe for openQA results (it'd wind up
considering UEFI and BIOS tests to be dupes, for instance), so if we
were to extend Bodhi to handle openQA results without introducing a
'scenario' concept, I'd have to basically teach Bodhi the openQA
scenario keys. Then if we wanted Bodhi to care about results from some
other system whose scenario was different, we'd have to teach it the
appropriate scenario values for *that* system too and so on. So, I
think this clearly demonstrates the value of the concept :)
If we implement a 'scenario' key, we can then just teach Bodhi to look
for that and combine it with the item, then maybe if it can't find one,
treat 'testname-arch' as the scenario (as a fallback for old results,
results from Taskotron until we teach it to write 'scenario' items into
its results, and a general fallback for cases where 'scenario' is
missing for some reason). That could be the general convention, I
guess: first look for a 'scenario' key, otherwise try and construct one
from very commonly available keys.
This reasoning makes a lot of sense to me.
11 a.m.
If we implement a 'scenario' key, we can then just teach
Bodhi to look
for that and combine it with the item, then maybe if it can't find one,
treat 'testname-arch' as the scenario (as a fallback for old results,
We don't have that many tests yet, and all of them are basically under our control, so
if we end up doing this, I think we should simply push a patch to all of them (should be
fairly easy) and avoid any legacy handling.
results from Taskotron until we teach it to write 'scenario'
items into
its results, and a general fallback for cases where 'scenario' is
missing for some reason). That could be the general convention, I
guess: first look for a 'scenario' key, otherwise try and construct one
from very commonly available keys.
I'd like to avoid the guessing game here. The system will be reliable only if it is
obvious and simple. If scenario turns out to work well for us, let's go all the way.
If there's no scenario, results should be unique when identified by testcase name +
item. If that's not enough, scenario needs to be there, period. Of course people can
submit whatever results they want, but once we want to employ them in gating, the results
need to adhere to the rules. We're open source, I believe we should send patches to
fix the task reporting if needed, rather than extending the guessing algorithms in our
consumers.
12:17 p.m.
On Mon, 2017-03-06 at 12:00 -0500, Kamil Paral wrote:
> If we implement a 'scenario' key, we can then just teach
Bodhi to look
> for that and combine it with the item, then maybe if it can't find one,
> treat 'testname-arch' as the scenario (as a fallback for old results,
We don't have that many tests yet, and all of them are basically
under our control, so if we end up doing this, I think we should
simply push a patch to all of them (should be fairly easy) and avoid
any legacy handling.
Well, the case I was thinking of is specifically Bodhi: even if we
adjust things so that all results *going forward* have a scenario key,
*existing* results will not, and often updates sit around in Bodhi for
a long time before going stable. If we make Bodhi only work if the
result has a 'scenario' key, it'll either break or not be able to
display test results for older updates, I think. (And I think Bodhi
shows results even for stable updates, so really, don't we need it to
be able to handle the existing results for all non-EOL releases?)
> results from Taskotron until we teach it to write
'scenario' items into
> its results, and a general fallback for cases where 'scenario' is
> missing for some reason). That could be the general convention, I
> guess: first look for a 'scenario' key, otherwise try and construct one
> from very commonly available keys.
I'd like to avoid the guessing game here. The system will be reliable
only if it is obvious and simple. If scenario turns out to work well
for us, let's go all the way. If there's no scenario, results should
be unique when identified by testcase name + item. If that's not
enough, scenario needs to be there, period. Of course people can
submit whatever results they want, but once we want to employ them in
gating, the results need to adhere to the rules. We're open source, I
believe we should send patches to fix the task reporting if needed,
rather than extending the guessing algorithms in our consumers.
I agree we should implement the 'scenario' reporting for all systems
ASAP (should I send a patch for taskotron, if I can figure it out?),
but I don't think that's actually sufficient, for the above reason.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
1:06 p.m.
On Mon, 2017-03-06 at 12:00 -0500, Kamil Paral wrote:
> > If we implement a 'scenario' key, we can then just teach Bodhi to look
> > for that and combine it with the item, then maybe if it can't find one,
> > treat 'testname-arch' as the scenario (as a fallback for old results,
>
> We don't have that many tests yet, and all of them are basically
> under our control, so if we end up doing this, I think we should
> simply push a patch to all of them (should be fairly easy) and avoid
> any legacy handling.
Well, the case I was thinking of is specifically Bodhi: even if we
adjust things so that all results *going forward* have a scenario key,
*existing* results will not, and often updates sit around in Bodhi for
a long time before going stable. If we make Bodhi only work if the
result has a 'scenario' key, it'll either break or not be able to
display test results for older updates, I think. (And I think Bodhi
shows results even for stable updates, so really, don't we need it to
be able to handle the existing results for all non-EOL releases?)
I was under the impression that Bodhi would consider the scenario key only if present,
otherwise consider all testcase+item-based results unique. So it should work both for
future and past results (with the slight exception of disregarding arch-specific results).
Am I missing something? Or are you hinting at the arch-specific results (depcheck)?
> > results from Taskotron until we teach it to write 'scenario' items
into
> > its results, and a general fallback for cases where 'scenario' is
> > missing for some reason). That could be the general convention, I
> > guess: first look for a 'scenario' key, otherwise try and construct
one
> > from very commonly available keys.
>
> I'd like to avoid the guessing game here. The system will be reliable
> only if it is obvious and simple. If scenario turns out to work well
> for us, let's go all the way. If there's no scenario, results should
> be unique when identified by testcase name + item. If that's not
> enough, scenario needs to be there, period. Of course people can
> submit whatever results they want, but once we want to employ them in
> gating, the results need to adhere to the rules. We're open source, I
> believe we should send patches to fix the task reporting if needed,
> rather than extending the guessing algorithms in our consumers.
I agree we should implement the 'scenario' reporting for all systems
ASAP (should I send a patch for taskotron, if I can figure it out?),
but I don't think that's actually sufficient, for the above reason.
I can definitely take care of adjusting the core taskotron once we decide to go full steam
ahead. From tasks, it seems only depcheck/rpmdeplint will be affected (nothing else uses
any extradata fields). Of course ideally we should also document this in libtaskotron docs
and/or resultsdb_conventions. But I'd leave that until we test it out and realize
whether it's working correctly or not.
1:19 p.m.
On Mon, 2017-03-06 at 14:06 -0500, Kamil Paral wrote:
> On Mon, 2017-03-06 at 12:00 -0500, Kamil Paral wrote:
> > > If we implement a 'scenario' key, we can then just teach Bodhi to
look
> > > for that and combine it with the item, then maybe if it can't find
one,
> > > treat 'testname-arch' as the scenario (as a fallback for old
results,
> >
> > We don't have that many tests yet, and all of them are basically
> > under our control, so if we end up doing this, I think we should
> > simply push a patch to all of them (should be fairly easy) and avoid
> > any legacy handling.
>
> Well, the case I was thinking of is specifically Bodhi: even if we
> adjust things so that all results *going forward* have a scenario key,
> *existing* results will not, and often updates sit around in Bodhi for
> a long time before going stable. If we make Bodhi only work if the
> result has a 'scenario' key, it'll either break or not be able to
> display test results for older updates, I think. (And I think Bodhi
> shows results even for stable updates, so really, don't we need it to
> be able to handle the existing results for all non-EOL releases?)
I was under the impression that Bodhi would consider the scenario key
only if present, otherwise consider all testcase+item-based results
unique. So it should work both for future and past results (with the
slight exception of disregarding arch-specific results). Am I missing
something? Or are you hinting at the arch-specific results
(depcheck)?
Sorry, yeah, that's exactly the case in point (and older openQA results
that don't include a 'scenario' key, I was planning to teach Bodhi how
to 'synthesize' it for that case). If we remove its existing ability to
'synthesize' the scenario for depcheck results, it'll display older
depcheck results with no scenario key improperly, right?
I was planning to clearly mark the 'synthesis' code as something that's
needed only so long as we care about older results without the scenario
key, so it can be removed in future.
Of course, an alternative would be to go into ResultsDB and amend all
existing older depcheck and openQA results to include a 'scenario' key.
I don't know if that's frowned-upon, though? I don't think it would be
very technically difficult to *do*, all the necessary information
should still be available (though for openQA results I think we may
have to query some of the values from openQA, I don't believe we
forward all the relevant 'scenario' key values to RDB at this
time...probably we should).
> I agree we should implement the 'scenario' reporting for
all systems
> ASAP (should I send a patch for taskotron, if I can figure it out?),
> but I don't think that's actually sufficient, for the above reason.
I can definitely take care of adjusting the core taskotron once we
decide to go full steam ahead. From tasks, it seems only
depcheck/rpmdeplint will be affected (nothing else uses any extradata
fields). Of course ideally we should also document this in
libtaskotron docs and/or resultsdb_conventions. But I'd leave that
until we test it out and realize whether it's working correctly or
not.
Agreed. Thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
11:29 a.m.
> I was under the impression that Bodhi would consider the
scenario key
> only if present, otherwise consider all testcase+item-based results
> unique. So it should work both for future and past results (with the
> slight exception of disregarding arch-specific results). Am I missing
> something? Or are you hinting at the arch-specific results
> (depcheck)?
Sorry, yeah, that's exactly the case in point (and older openQA results
that don't include a 'scenario' key, I was planning to teach Bodhi how
to 'synthesize' it for that case). If we remove its existing ability to
'synthesize' the scenario for depcheck results, it'll display older
depcheck results with no scenario key improperly, right?
Theoretically it would display only one depcheck result (based on the last arch tested)
instead of several per-arch results. However, we re-run depcheck almost constantly, so
once we include the scenario key, it will re-test and re-report all results for all
pending RPMs in a few hours (every new package pushed to -pending triggers it). So this
should be a non-issue, hopefully.
So the only problem would be the older OpenQA results, but... is that such a big deal?
It's few days old :) I'd personally avoid having extra code just for this.
I was planning to clearly mark the 'synthesis' code as something that's
needed only so long as we care about older results without the scenario
key, so it can be removed in future.
Of course, an alternative would be to go into ResultsDB and amend all
existing older depcheck and openQA results to include a 'scenario' key.
I don't know if that's frowned-upon, though? I don't think it would be
very technically difficult to *do*, all the necessary information
should still be available (though for openQA results I think we may
have to query some of the values from openQA, I don't believe we
forward all the relevant 'scenario' key values to RDB at this
time...probably we should).
Not worth the energy, I think. Modifying the production database is... worrying. Assuming
depcheck is not a problem, and that you insist on even having the older OpenQA results
showing up properly, maybe it would be easier to re-trigger the older OpenQA jobs?
Provided it's not very hard to do.
8:07 a.m.
I've had a long (three days long!) chat about this with Josef. The solution discussed
here (bodhi deduplicating results by scenario, if it exists) is usable temporarily, but
might not work well enough in the future, due to performance. The problem is that this
approach requires to search through most or all the database before you figure out that
you have every unique result available. For example, openqa result uses
"scenario" to either put "uefi" or "bios" in there. However,
Bodhi doesn't know all the possible values of scenario for this particular test case.
So even if it already found several results with scenario:uefi and several results with
scenario:bios, it needs to search further and further, until there are no more results to
be returned and you can be sure there's no other scenario value available.
There are some optimizations we can do, for example limit the search with a bodhi id
"last modified" timestamp (for bodhi_update results) or with the koji build
"build_time" timestamp (for koji_build results). However, that won't always
help (the koji build can be available for weeks or months, before you create a bodhi
update with it, and then you're again searching through most of the database).
We tried to brainstorm the best solution for this that wouldn't be overly performance
demanding, and we have some ideas. But first, let's clarify what the requirement is,
so that there's no confusion what we're solving and why. IIUIC, the requirement
is:
** We want to be able to retrieve all unique results for a particular item. Unique results
might be determined just by testcase+item+type combination, or they can be further
distinguished by some (but not necessarily all) additional extradata fields. Those fields
might be arbitrary, i.e. not restricted to "common fields" for that type (e.g.
"arch" for koji builds), but we can require to have them identified somehow
(e.g. scenario key, linking to them or duplicating them). We want to be able to do this
from a consumer that has no internal knowledge about the tasks it displays, e.g. Bodhi
showing all tasks results for a specific item, without knowing what fields differentiate
them and what are all the available values of those fields. **
Do you think that's a correct description? Also, do you see any other similar
requirement/use case that would need in the mid-term future (maybe for Bodhi, maybe for
some other consumer)? If you have any idea about further use cases, please state them
now.
Now, based on the requirement above, we thought of this:
1. We will use resultsdb_conventions to specify common fields for a particular item type.
For example for koji builds, it will be 'arch'. For git_commit, it might be
'url' (provided that commit hash is an item). For compose, it might be
'compose_id' (provided it won't be an item). Those fields will cover only
things are universal and make sense for all results for given type.
2. Those common fields will be mandatory to be filled in. Libtaskotron will be able to
emit warnings/fail early if any of them is missing. Resultsdb will be able to reject
results not meeting the conventions. Even if Resultsdb stores them, the consumers might
not display them properly or at all.
3. The purpose of these common fields is to ensure that all results for a given type has
enough basic information for the consumer to make proper queries, if needed.
4. If the result needs to distinguished further, it will need to add "scenario"
field and populate it with unique strings that will allow to distinguish the results.
5. The scenario field has the advantage (over adding the environment information to a
testcase name) that it serves solely for this purpose and doesn't force the user to
mix unrelated information (testcase and environment). It will also still allow the user to
make exact queries for a particular testcase (if environment was added to the testcase
name, the user would have to do wildcarded queries, which can be problematic in some
cases).
6. We will implement a "middleware" which will be tailored to Bodhi use case
(getting latest result of every unique result). Bodhi will be querying this middleware
instead of ResultsDB directly. The middleware will intercept resultsdb fedmsgs and store
only the latest unique results for a particular item. It will know the resultsdb
conventions (therefore know which fields are important and mandatory for each item type)
and for each item store a combination of testcase+type+common fields+scenario, and a
matching resultsdb result ID. For this reason, it should be much faster to find all unique
results because we won't have to search through all the database space, but instead we
will have all the unique result ids stored in a separate database. Josef wants to have
this middleware separated from resultsdb, because it's specific to a particular Fedora
workflow. If there are other similar use cases in the future, the same approach would be
used if possible (middleware separated from resultsdb).
7. Bodhi will query the middleware for an item, and it will receive back all existing
unique (common fields+scenario) results for all testcases. Bodhi won't need to have
any additional logic on its side. It will also be possible to futher restrict the query
(ask just for a specific testcase prefix, or for a particular common fields value), but
the filters will only narrow down the complete list of latest results. Specifying
additional filters (other than item+type) will not change the set of result, which would
be returned without those filters present. This is, in fact mostly a user-faced
convenience, since the post-process filtering can be easily implemented on the server
side. If you need to make a query for specific non-mandatory fields, you'll need to
ask resultsdb instead of the middleware, but it will be slower and you won't have
latest unique results automatically filtered, you'll need to do that yourself.
What do you think, does that sound reasonable and does it fulfill all the needs we can
envision at this moment?
For completeness, I add a solution B, which we considered inferior to the one above:
B1) Resultsdb conventions again specify mandatory common fields for specific types.
B2) There will be no scenario field.
B3) If you need to differentiate your result beyond the common fields, you need to append
that information into the testcase name (so e.g. autoqa.base_install.uefi).
B4) The consumer (e.g. Bodhi) needs to be aware of the resultsdb conventions, and needs to
do very specific directed queries. For example it knows that koji_build results have a
mandatory common field "arch", which can contain these 7 different values, and
needs to perform a separate query for each arch it is interested in.
Solution B doesn't require any middleware, but it requires consumers to have deeper
knowledge about what they're asking for, and it forces tasks with extra requirements
to mix up their testcase names with environment descriptions. That might make it difficult
to ask for a specific testcase result (for example, you can't ask for
openqa.base_install regardless of environment, because openqa.base_install* might also
select openqa.base_install.uefi.prepare and similar "sub-results").
If we're OK with these tradeoffs, however, this approach would again be reasonably
fast.
I know this has been a long email, but, feedback needed! :-)
Thanks,
Kamil
8:36 a.m.
Thx for taking the time to put our discussions together! You are a
gentleman and a scholar.
J.
On Wed, Mar 22, 2017 at 2:07 PM, Kamil Paral <kparal(a)redhat.com> wrote:
I've had a long (three days long!) chat about this with Josef.
The
solution discussed here (bodhi deduplicating results by scenario, if it
exists) is usable temporarily, but might not work well enough in the
future, due to performance. The problem is that this approach requires to
search through most or all the database before you figure out that you have
every unique result available. For example, openqa result uses "scenario"
to either put "uefi" or "bios" in there. However, Bodhi doesn't
know all
the possible values of scenario for this particular test case. So even if
it already found several results with scenario:uefi and several results
with scenario:bios, it needs to search further and further, until there are
no more results to be returned and you can be sure there's no other
scenario value available.
There are some optimizations we can do, for example limit the search with
a bodhi id "last modified" timestamp (for bodhi_update results) or with the
koji build "build_time" timestamp (for koji_build results). However, that
won't always help (the koji build can be available for weeks or months,
before you create a bodhi update with it, and then you're again searching
through most of the database).
We tried to brainstorm the best solution for this that wouldn't be overly
performance demanding, and we have some ideas. But first, let's clarify
what the requirement is, so that there's no confusion what we're solving
and why. IIUIC, the requirement is:
** We want to be able to retrieve all unique results for a particular
item. Unique results might be determined just by testcase+item+type
combination, or they can be further distinguished by some (but not
necessarily all) additional extradata fields. Those fields might be
arbitrary, i.e. not restricted to "common fields" for that type (e.g.
"arch" for koji builds), but we can require to have them identified somehow
(e.g. scenario key, linking to them or duplicating them). We want to be
able to do this from a consumer that has no internal knowledge about the
tasks it displays, e.g. Bodhi showing all tasks results for a specific
item, without knowing what fields differentiate them and what are all the
available values of those fields. **
Do you think that's a correct description? Also, do you see any other
similar requirement/use case that would need in the mid-term future (maybe
for Bodhi, maybe for some other consumer)? If you have any idea about
further use cases, please state them now.
Now, based on the requirement above, we thought of this:
1. We will use resultsdb_conventions to specify common fields for a
particular item type. For example for koji builds, it will be 'arch'. For
git_commit, it might be 'url' (provided that commit hash is an item). For
compose, it might be 'compose_id' (provided it won't be an item). Those
fields will cover only things are universal and make sense for all results
for given type.
2. Those common fields will be mandatory to be filled in. Libtaskotron
will be able to emit warnings/fail early if any of them is missing.
Resultsdb will be able to reject results not meeting the conventions. Even
if Resultsdb stores them, the consumers might not display them properly or
at all.
3. The purpose of these common fields is to ensure that all results for a
given type has enough basic information for the consumer to make proper
queries, if needed.
4. If the result needs to distinguished further, it will need to add
"scenario" field and populate it with unique strings that will allow to
distinguish the results.
5. The scenario field has the advantage (over adding the environment
information to a testcase name) that it serves solely for this purpose and
doesn't force the user to mix unrelated information (testcase and
environment). It will also still allow the user to make exact queries for a
particular testcase (if environment was added to the testcase name, the
user would have to do wildcarded queries, which can be problematic in some
cases).
6. We will implement a "middleware" which will be tailored to Bodhi use
case (getting latest result of every unique result). Bodhi will be querying
this middleware instead of ResultsDB directly. The middleware will
intercept resultsdb fedmsgs and store only the latest unique results for a
particular item. It will know the resultsdb conventions (therefore know
which fields are important and mandatory for each item type) and for each
item store a combination of testcase+type+common fields+scenario, and a
matching resultsdb result ID. For this reason, it should be much faster to
find all unique results because we won't have to search through all the
database space, but instead we will have all the unique result ids stored
in a separate database. Josef wants to have this middleware separated from
resultsdb, because it's specific to a particular Fedora workflow. If there
are other similar use cases in the future, the same approach would be used
if possible (middleware separated from resultsdb).
7. Bodhi will query the middleware for an item, and it will receive back
all existing unique (common fields+scenario) results for all testcases.
Bodhi won't need to have any additional logic on its side. It will also be
possible to futher restrict the query (ask just for a specific testcase
prefix, or for a particular common fields value), but the filters will only
narrow down the complete list of latest results. Specifying additional
filters (other than item+type) will not change the set of result, which
would be returned without those filters present. This is, in fact mostly a
user-faced convenience, since the post-process filtering can be easily
implemented on the server side. If you need to make a query for specific
non-mandatory fields, you'll need to ask resultsdb instead of the
middleware, but it will be slower and you won't have latest unique results
automatically filtered, you'll need to do that yourself.
What do you think, does that sound reasonable and does it fulfill all the
needs we can envision at this moment?
For completeness, I add a solution B, which we considered inferior to the
one above:
B1) Resultsdb conventions again specify mandatory common fields for
specific types.
B2) There will be no scenario field.
B3) If you need to differentiate your result beyond the common fields, you
need to append that information into the testcase name (so e.g.
autoqa.base_install.uefi).
B4) The consumer (e.g. Bodhi) needs to be aware of the resultsdb
conventions, and needs to do very specific directed queries. For example it
knows that koji_build results have a mandatory common field "arch", which
can contain these 7 different values, and needs to perform a separate query
for each arch it is interested in.
Solution B doesn't require any middleware, but it requires consumers to
have deeper knowledge about what they're asking for, and it forces tasks
with extra requirements to mix up their testcase names with environment
descriptions. That might make it difficult to ask for a specific testcase
result (for example, you can't ask for openqa.base_install regardless of
environment, because openqa.base_install* might also select
openqa.base_install.uefi.prepare and similar "sub-results").
If we're OK with these tradeoffs, however, this approach would again be
reasonably fast.
I know this has been a long email, but, feedback needed! :-)
Thanks,
Kamil
_______________________________________________
Resultsdb-users mailing list -- resultsdb-users(a)lists.fedoraproject.org
To unsubscribe send an email to resultsdb-users-leave@lists.
fedoraproject.org
11:30 a.m.
I like solution A, provided it will be robust enough for our use cases
(it definitely seems so).
Only problem I can think of is, if we are going to use "common fields"
for testing equality of two tests (for Bodhi), we should be careful
about what tools, that are reporting to ResultsDB, consider as same
tests. For example, if both openQA and Tunir are testing specific
compose, I can imagine that e. g. in openQA two tests with different
'arch' are two different tests, while Tunir doesn't care about 'arch'
at all, even though both are testing "compose" type. But I guess this
will be solved by using 'scenario' in openQA's tests. We just have to
be sure what we put into "common fields".
Jan
12:23 p.m.
On Wed, 2017-03-22 at 09:07 -0400, Kamil Paral wrote:
The problem is that this approach requires to search through most or
all the database before you figure out that you have every unique
result available.
Sorry, I'm really badly glazing over trying to read this mail. But let
me just zero in on the first thing I don't understand, which is this.
Later in the mail, you write this:
"IIUIC, the requirement is:
** We want to be able to retrieve all unique results for a particular
item. ..."
yes, that's the case I've mostly been considering. So where does this
"search through most or all of the database" thing come from? I've been
assuming all along that almost all uses are going to be limiting the
amount of searching that needs to be done quite heavily by specifying
at least an item, or multiple items, or a result group, or a test case
name, or some combination of all of those. I certainly am not expecting
many if any folks/tools to be just searching for "absolutely any
results with scenario X".
So where does this concern about "search[ing] through most or all of
the database" come from? I just don't get that, which I think is making
it harder to understand the rest of the proposal.
Thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
1:51 p.m.
As I understand it, if you'd ask for "latest test with item == X", it
would just go through the database sorted by the time and returned on
first item with "item == X" it would find. But, if you'd ask for
"latest tests with item == X for each unique value 'scenario'", it
would not stop on the first item with "item == X", it would have to go
through the whole DB, because what if there is some other item with
"item == X" but different scenario value? You would have to create a
new table with all possible "scenario" values, but that wouldn't be a
generic solution.
1:55 p.m.
On Wed, 2017-03-22 at 19:51 +0100, Jan Sedlak wrote:
As I understand it, if you'd ask for "latest test with item
== X", it
would just go through the database sorted by the time and returned on
first item with "item == X" it would find. But, if you'd ask for
"latest tests with item == X for each unique value 'scenario'", it
would not stop on the first item with "item == X", it would have to go
through the whole DB, because what if there is some other item with
"item == X" but different scenario value? You would have to create a
new table with all possible "scenario" values, but that wouldn't be a
generic solution.
I guess the usage I was kinda assuming was that you'd request 'all
tests with item == X' and then do the scenario de-duplication on the
client side. This is what Bodhi does now, for instance - it just kinda
invents the scenario concept on its end. It doesn't just ask for 'the
latest result for item X' from rdb.
Is asking for 'all results for item == X' gonna be too much work for
rdb to handle? Is that the problem?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
6:05 p.m.
On Wed, Mar 22, 2017 at 7:55 PM, Adam Williamson <adamwill(a)fedoraproject.org
wrote:
Is asking for 'all results for item == X' gonna be too much
work for
rdb to handle? Is that the problem?
Sure could be done that way, the only slight difference is through-output.
First up, be sure to have a look at the attached graph_data_count.png
representing the number of results per item (Y-axis). tl;dr;
- DEV contains data since 2015-03-18, and has 226428 distinct items
- PROD contains data since2017-01-10, and has 30913 distinct items
- DEV
- median = 16
- max = 16817
- outliers begin at about 140 results per item (cca 7%)
- PROD
- median = 17
- max = 2867
- outliers begin at about 65 results per item (cca 13%)
The bulk of the outliers is composed of depcheck results, taking
gnote-3.17.0-3.fc23 as an example:
testcase_name | count
------------------+-------
dist.rpmlint | 1
dist.depcheck | 16287
dist.upgradepath | 21
Let's get some stats going on.. I did some tests on DEV by timing various
parts of the process of querying data.(inside RedHat, outside Infra),
removing the outlier times to crop the IO bound problems with DB (at the
moment the DB server a bit overloaded, and data are routinely swapped in
and out of memory, thus creating order of magnitude delays for non-cahed
data). Roundtrip for a databse-disconnected, almost empty result <
http://taskotron-dev.fedoraproject.org/resultsdb_api/api/v2.0/> is about
500 ms, this time will most probably be better for Bodhi, as it is in the
same datacenter (I would think), so take the 'roundtrip' times with that in
mind
The times are split as such:
- database ~ the time to get the results from database (read the notes
section below)
- serialization ~ the time it takes to produce the serialized data (dict
to be jsonified) based on the data from database
- jsonify ~ the time it takes to jsonify the prepared dict
- server ~ sum of all the above
- roundtrip ~ time it took to get the data
All the test queries were run 25 times, first 5 were discarded (to force
the data in question into memory on the DB machine) and the times are
averaged.
= 20 results (median)=
`curl "http://taskotron-dev.fedoraproject.org/resultsdb_
api/api/v2.0/results?item=hplip-3.16.7-3.fc25&limit=20" -s /tmp/datafile`
"database":
0.0072879791259765625,
"jsonify": 0.0033910274505615234,
"serialization": 0.19698095321655273,
"server": 0.20765995979309082,
"roundtrip": 0.935
= 55 results (3rd quartile) =
`curl "http://taskotron-dev.fedoraproject.org/resultsdb_
api/api/v2.0/results?item=dolphin-15.12.3-1.fc24&limit=55" -s >
/tmp/datafile`
``
"database": 0.007297039031982422,
"jsonify": 0.008601188659667969,
"serialization": 0.4342458248138428,
"server": 0.45014405250549316,
"roundtrip": 1.547
= 140 results (data without outliers) =
`curl "http://taskotron-dev.fedoraproject.org/resultsdb_
api/api/v2.0/results?item=moodle-3.0.4-1.fc24&limit=140&callback=timeit" -s
/tmp/datafile`
"database": 0.015254974365234375,
"jsonify": 0.022364139556884766,
"serialization": 1.390178918838501,
"server": 1.4277980327606201,
"roundtrip": 3.020
= Notes =
== database vs. serializatoin times ==
The `databse` tiime does not account for loading the extra data, testcase
and group information, that is in the `serialization` time. This is mostly
due to the fact that I was not able to force the sqlalchemy to load all the
related data at once, to be able to profile the actual process of
serialization separately. The queries performed there are all only direct
index searches over numerical id, but three of them are performed per
result. If the webapp-database roundtrip takes noticeable amount of time
(and my gut feeling would be that they are not insignificant) , finding a
way to load the data all at once might help with the time, at the cost of
higher memory consumption. But the time to grab the 'raw' results is still
informative, as it shows the base time to even get the process started.
== DB IO ==
As I said earlier, these are the bestcase scenarios - most of the time, you
(Bodhi) won't be doing that N-th same query to get that benefit of having
data in memory, but the data will need to be loaded from the disk. This is
something Tim and I want to get sorted out (by splitting the various
databases from the one machine that currently handles them all). But as
this will be the "common" expected experience, consider the "slow"
times
for those same queries:
20 results:
"database": 0.19269204139709473,
"jsonify": 0.003347158432006836,
"serialization": 0.5247390270233154,
"server": 0.720778226852417
55 results:
"database": 0.2362060546875,
"jsonify": 0.008295059204101562,
"serialization": 0.8430919647216797,
"server": 1.0875930786132812
140 results:
"database": 0.013692140579223633,
"jsonify": 0.02150702476501465,
"serialization": 2.1643788814544678,
"server": 2.199578046798706
= Conclusion =
== Regarding ResultsDB ==
I will profile the serialization code itself, and investigate on a way to
force less sql queries so we can compare the results. My gut feeling is
that we could lower the "best" time focusing on these aspects
The actual problem, that needs solving though, is moving the resultsdb
production DB to a separate machine to get rid of that IO bound problems.
These cause/will cause the actual slowdown that can be observed by querying
ther resultsdb in the semi-random way bodhi does/will be doing.
== Regarding the Bodhi endpoint/middleware optimization ==
I'm not sure it is worth doing _right now_ but given the nature of what it
would do, and the way it would be implemented, there would be measurable
spead-up in the DB-part, as the database would be optimized for this one
specific use-case. It would also lower the Bodhi load, since the
de-duplication would be taken care of there, instead of in the consumer.
I still believe, that having designated DB server for resultsdb prod and
exploring the loading process will have bigger impact, but I can not take
care of the first part on my own.
Did this help to answer your questions?
Josef
6:50 p.m.
On Fri, 2017-03-24 at 00:05 +0100, Josef Skladanka wrote:
On Wed, Mar 22, 2017 at 7:55 PM, Adam Williamson
<adamwill(a)fedoraproject.org
> wrote:
> Is asking for 'all results for item == X' gonna be too much work for
> rdb to handle? Is that the problem?
>
Sure could be done that way, the only slight difference is through-output.
First up, be sure to have a look at the attached graph_data_count.png
representing the number of results per item (Y-axis). tl;dr;
- DEV contains data since 2015-03-18, and has 226428 distinct items
- PROD contains data since2017-01-10, and has 30913 distinct items
- DEV
- median = 16
- max = 16817
- outliers begin at about 140 results per item (cca 7%)
- PROD
- median = 17
- max = 2867
- outliers begin at about 65 results per item (cca 13%)
The bulk of the outliers is composed of depcheck results, taking
gnote-3.17.0-3.fc23 as an example:
testcase_name | count
------------------+-------
dist.rpmlint | 1
dist.depcheck | 16287
dist.upgradepath | 21
So, wait. Let me be totally sure I'm on the same page here. You mean
that dist.depcheck has been run on that single package *sixteen
thousand times*? So part of the problem is, if you just ask for 'all
test results for package X', occasionally you might get tens of
thousands of results?
If so, then that certainly helps me understand the issue. I simply
hadn't expected anything like that to be the case. Obviously openQA is
my usual point of reference, and if you ask for 'all openQA results for
compose X', you're going to get, at max, a few hundred results.
(Usually, in prod, a couple hundred). From what I knew of the package
tests, I was expecting something in the same range, or fewer, for
those. So I just wasn't expecting that we needed to worry a lot about
optimizing beyond 'get all the results for the item you want results
for, then filter from there'.
If I'm understanding correctly, that certainly helps me to understand
why there might be a problem, and I'll go back and read kparal's mail
again in that light.
Still, if we're mostly ignoring the outliers for now and thinking about
the 'normal' cases, I think I'd contend that getting 140 results at a
time should be a pretty 'normal' use case for something like resultsdb.
I'm not sure I'm a fan of doing something more complex than the
'scenario' key if the goal is purely to try and reduce the number of
results that actually need to be returned by the server in the first
place from, say, 200 to 50. That's just MHO, though. :)
= Conclusion =
== Regarding ResultsDB ==
I will profile the serialization code itself, and investigate on a way to
force less sql queries so we can compare the results. My gut feeling is
that we could lower the "best" time focusing on these aspects
The actual problem, that needs solving though, is moving the resultsdb
production DB to a separate machine to get rid of that IO bound problems.
These cause/will cause the actual slowdown that can be observed by querying
ther resultsdb in the semi-random way bodhi does/will be doing.
== Regarding the Bodhi endpoint/middleware optimization ==
I'm not sure it is worth doing _right now_ but given the nature of what it
would do, and the way it would be implemented, there would be measurable
spead-up in the DB-part, as the database would be optimized for this one
specific use-case. It would also lower the Bodhi load, since the
de-duplication would be taken care of there, instead of in the consumer.
I still believe, that having designated DB server for resultsdb prod and
exploring the loading process will have bigger impact, but I can not take
care of the first part on my own.
I *think* kparal's approach was an attempt to be more generic than just
tailored to Bodhi's use case, but I'll have to go back and read it
again.
Did this help to answer your questions?
Yes, thanks very much!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
3:46 a.m.
On Fri, Mar 24, 2017 at 12:50 AM, Adam Williamson <
adamwill(a)fedoraproject.org> wrote:
On Fri, 2017-03-24 at 00:05 +0100, Josef Skladanka wrote:
So, wait. Let me be totally sure I'm on the same page here. You mean
that dist.depcheck has been run on that single package *sixteen
thousand times*? So part of the problem is, if you just ask for 'all
test results for package X', occasionally you might get tens of
thousands of results?
....
If I'm understanding correctly, that certainly helps me to understand
why there might be a problem, and I'll go back and read kparal's mail
again in that light.
Yup, that's it. The other part of what kparal and I were discussing was
that if the "query all, deduplicate in consumer" usecase is the most
common, it would make sense to optimize for it. Not that it's necessary,
but convenient for both parties.
Still, if we're mostly ignoring the outliers for now and thinking
about
the 'normal' cases, I think I'd contend that getting 140 results at a
time should be a pretty 'normal' use case for something like resultsdb.
I'm not sure I'm a fan of doing something more complex than the
'scenario' key if the goal is purely to try and reduce the number of
results that actually need to be returned by the server in the first
place from, say, 200 to 50. That's just MHO, though. :)
I tend to agree, and the solution Kamil and I discussed is basically "just
that" - using scenario. On top of that, we wanted to have some conventions
on the "default scenario value" - say for koji_builds the "default"
scenario would be `arch`, and if you wanted to differentiate even more,
you'd just fill the actual scenario field.
The bonus, as we see it, is in fact the convention part of it, since it can
act as a point of reference. I'm sure that saying "do whatever, but we
handle stuff this way" is also a solution. I just feel like it contains
more of the "well, if you want to do this, then you should do that, but
lets first make sure you understand what we do, so you can base your
decisions on something relevant" conversations with devels, where the
sensible conventions give most of them the right solution out of the box.
The added value of those conventions (for 'minimal set of relevant fields
per item type') have the added benefit of being able to do data validation
in resultsdb.
I *think* kparal's approach was an attempt to be more generic than just
tailored to Bodhi's use case, but I'll have to go back and read it
again.
At the beginning of our three-days-long flamewar (in person) it certainly
was, but AFAIK the current goal is to have the Bodhi convenience (where
'bodhi convenience' is equivalent of 'common usecase' for us at the
moment).
Josef
5:56 a.m.
Still, if we're mostly ignoring the outliers for now and thinking
about
the 'normal' cases, I think I'd contend that getting 140 results at a
time should be a pretty 'normal' use case for something like resultsdb.
I'm not sure I'm a fan of doing something more complex than the
'scenario' key if the goal is purely to try and reduce the number of
results that actually need to be returned by the server in the first
place from, say, 200 to 50. That's just MHO, though. :)
We can try this and see. So if we decided to just get all results, and ignore those which
happen to have too many of them, this could be the reasonable approach from Bodhi:
1. Query /results?item=NVR
2. Add &since= based on last Bodhi update modification timestamp (for bodhi_update
type) or earliest Koji build timestamp (for koji_build type, if not difficult)
3. Add &limit= with some reasonable number based on our current db stats (probably
something between 50 and 100)
4. Stop loading additional pages if it seems there's too much data (e.g. give up after
&page=4 -> 500 results)
5. Deduplicate the results based on scenario in Bodhi (server or web browser)
6. For some items, Bodhi will not be able to display results. It should display some
explanation message instead (it can advise people to go to resultsdb_frontend and query
for a specific testcase result if they're interested).
We should on course still get a better hosting for our production DB. If the combination
of these doesn't help with performance, or the db data explode with depcheck-style
results, or we get more requests for /latest endpoint while understanding scenario,
we'd have to come up with the middleware mentioned before.
Does that sound reasonable?
10:03 a.m.
On Wed, 2017-03-22 at 09:07 -0400, Kamil Paral wrote:
I know this has been a long email, but, feedback needed! :-)
So I just read through this again.
Now I get the idea somewhat better. I think it's a viable idea. I'm not
sure it's a *good* idea. It seems to add an awful lot of complexity in
order to optimize some quite specific workflows, which I thought was
kinda what we decided to get out of the business of doing, when we made
ResultsDB really simple. Writing, testing, deploying and maintaining
this middleware layer seems like it'd be a pretty sizable chunk of
work, and we'd have that extra complexity to keep in mind and debug
forever after: any time we had some kind of quirk with discovering and
displaying results, we'd have to figure out whether the bug was in the
client, the middleware, or resultsdb...
It's only my feeling, I'm not saying 'nonono don't do this'. But my
inkling is it's not the best use of resources.
It might be a useful thing to use for leverage in asking for more
hardware for resultsdb, though ;) "Well, look, if you won't give us
some more capable hardware, we're going to have to spend a ton of our
valuable time building THIS pile of bits. Is that what you want to be
paying us to do?" :P
I'm wondering if we can do something simpler and within resultsdb,
though it still *is* adding complexity: basically, add de-duplication
by specified fields as a feature to resultsdb, and let things that use
it request de-duplicated results when querying. This is something
openQA does, for whatever that's worth: when querying openQA directly,
I can request that it give me only the 'latest' results, which
translates to it de-duplicating by scenario on the server side.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
11:13 a.m.
On Thu, Mar 30, 2017 at 5:03 PM, Adam Williamson <adamwill(a)fedoraproject.org
wrote:
On Wed, 2017-03-22 at 09:07 -0400, Kamil Paral wrote:
> I know this has been a long email, but, feedback needed! :-)
So I just read through this again.
I'm wondering if we can do something simpler and within resultsdb,
though it still *is* adding complexity: basically, add de-duplication
by specified fields as a feature to resultsdb
Sure, can be done. It is not "simple" though, and still requires traversing
all the data in database, plus it means more CPU and memory load on the web
server (as it will be doing the de-duplication) - this all meaning slower
response times than what the middleware could have. I'm not saying it's not
doable, just that having reasonable HW is becoming more of a hard
requirement, if we decide to go this way.
The key differences between OpenQA and ResultsDB here are
1) OpenQA operates on few orders of magnitude less data
2) OpenQA can be optimized for that one special use case - if the
de-duplication is done on one key, and always on that one key, you can
easily make a DB optimization to cater for that use-case. Since ResultsDB
is supposed to be data-agnostic, we can't really do that (well, we can,
actually, but I am strongly against having special-usecase-solving code
[like optimizing for deduplication based on one special non-required piece
of data basically is] in the codebase, so...). This is what the middleware
can do - have a DB structure + code optimized for that one special
use-case, plus it can have different data-retention policies than
ResultsDB, and so on (like you could easily prune data from a thing where
you know stuff older than 6 months does not need to be cared about).
Once again - not saying this necessarily needs to be done one way or
another. Just for the record - I really do believe that for the most cases,
doing it from the Bodhi's frontend, like Kamil described, and you initially
wanted to (IIRC), is a sane way. At least for now :)
J.
5:31 p.m.
On Thu, 2017-03-30 at 18:13 +0200, Josef Skladanka wrote:
On Thu, Mar 30, 2017 at 5:03 PM, Adam Williamson
<adamwill(a)fedoraproject.org
> wrote:
> On Wed, 2017-03-22 at 09:07 -0400, Kamil Paral wrote:
>
> > I know this has been a long email, but, feedback needed! :-)
>
> So I just read through this again.
>
> I'm wondering if we can do something simpler and within resultsdb,
> though it still *is* adding complexity: basically, add de-duplication
> by specified fields as a feature to resultsdb
>
Sure, can be done. It is not "simple" though, and still requires traversing
all the data in database, plus it means more CPU and memory load on the web
server (as it will be doing the de-duplication) - this all meaning slower
response times than what the middleware could have.
FWIW, I actually looked into doing this and now I understand what you
mean...I was trying to set up something that would work just like
`/latest` except it would get the latest result *per scenario*, not per
testcase...but indeed it turns out to be very tricky to do given how
the storage of the 'extradata' differs from the storage of the core
attributes for each result.
I kinda feel like it ought to be possible for someone with better
advanced-SQL-fu than me, though :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
5:44 p.m.
On Mon, 2017-03-06 at 14:06 -0500, Kamil Paral wrote:
I can definitely take care of adjusting the core taskotron once we
decide to go full steam ahead. From tasks, it seems only
depcheck/rpmdeplint will be affected (nothing else uses any extradata
fields). Of course ideally we should also document this in
libtaskotron docs and/or resultsdb_conventions. But I'd leave that
until we test it out and realize whether it's working correctly or
not.
I filed a couple of tickets for this, one each for task-depcheck and
task-rpmdeplint :
https://phab.qa.fedoraproject.org/T921
https://phab.qa.fedoraproject.org/T922
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
1:08 p.m.
On Mon, Mar 06, 2017 at 10:17:06AM -0800, Adam Williamson wrote:
On Mon, 2017-03-06 at 12:00 -0500, Kamil Paral wrote:
> > If we implement a 'scenario' key, we can then just teach Bodhi to look
> > for that and combine it with the item, then maybe if it can't find one,
> > treat 'testname-arch' as the scenario (as a fallback for old results,
>
> We don't have that many tests yet, and all of them are basically
> under our control, so if we end up doing this, I think we should
> simply push a patch to all of them (should be fairly easy) and avoid
> any legacy handling.
Well, the case I was thinking of is specifically Bodhi: even if we
adjust things so that all results *going forward* have a scenario key,
*existing* results will not, and often updates sit around in Bodhi for
a long time before going stable. If we make Bodhi only work if the
result has a 'scenario' key, it'll either break or not be able to
display test results for older updates, I think. (And I think Bodhi
shows results even for stable updates, so really, don't we need it to
be able to handle the existing results for all non-EOL releases?)
Could the old data in resultsdb be migrated to use the new scheme?
Or provide some sort of redirect?
Pierre
8:50 p.m.
On Mon, Mar 06, 2017 at 10:17:06AM -0800, Adam Williamson wrote:
On Mon, 2017-03-06 at 12:00 -0500, Kamil Paral wrote:
> > If we implement a 'scenario' key, we can then just teach Bodhi to look
> > for that and combine it with the item, then maybe if it can't find one,
> > treat 'testname-arch' as the scenario (as a fallback for old results,
>
> We don't have that many tests yet, and all of them are basically
> under our control, so if we end up doing this, I think we should
> simply push a patch to all of them (should be fairly easy) and avoid
> any legacy handling.
Well, the case I was thinking of is specifically Bodhi: even if we
adjust things so that all results *going forward* have a scenario key,
*existing* results will not, and often updates sit around in Bodhi for
a long time before going stable. If we make Bodhi only work if the
result has a 'scenario' key, it'll either break or not be able to
display test results for older updates, I think. (And I think Bodhi
shows results even for stable updates, so really, don't we need it to
be able to handle the existing results for all non-EOL releases?)
I'd vote "no" on this on worrying about Bodhi supporting both the new
and the old way. That's just visualization in Bodhi. If the results
disappeared from the UI for some updates for a short period of time
while we cut over to new scenario based keys, then no big loss. If we
really need the results for some audit reason, we can also go dig them
up directly in resultsdb. That would be an exceptional situation.
6:15 p.m.
On Fri, 2017-03-03 at 15:12 -0800, Adam Williamson wrote:
If we implement a 'scenario' key, we can then just teach
Bodhi to look
for that and combine it with the item, then maybe if it can't find one,
treat 'testname-arch' as the scenario (as a fallback for old results,
results from Taskotron until we teach it to write 'scenario' items into
its results, and a general fallback for cases where 'scenario' is
missing for some reason). That could be the general convention, I
guess: first look for a 'scenario' key, otherwise try and construct one
from very commonly available keys.
So I come back, a whole two months older and wiser, with a report from
the field on this ;)
In the end we went ahead with more or less my initial plan for the
'scenario' handling: openQA reports a single 'scenario' value in the
extradata for each report it submits, containing a string
representation of the 'scenario' (it does this by smooshing the values
of all the openQA 'scenario keys' together using periods, for the
record).
Then I got to teach Bodhi to handle openQA results. Happily, this
*does* involve dealing with the 'scenario problem' (though the initial
implementation we did in Bodhi ignores it, and winds up omitting some
openQA results). For each update, we run four of the tests twice, once
on a Server image, once on a Workstation image. In openQA's terms, this
is the same test run on a different 'flavor', and 'flavor' is one of
openQA's "scenario keys".
So for reference, here are a couple of sample results:
https://taskotron.fedoraproject.org/resultsdb/results/13573636
https://taskotron.fedoraproject.org/resultsdb/results/13573633
Same 'testcase', same 'item', same 'arch' even...but different
'scenario':
fedora.26.updates-workstation.x86_64.base_selinux.64bit
fedora.26.updates-server.x86_64.base_selinux.64bit
So, first the good news: fundamentally, the concept works. I can tell
Bodhi to query 'all bodhi_update results for the item FEDORA-2017-
4e95959f0d' and then display (only) the latest result for each
'scenario' that appears in the whole set of results.
However, there are two pieces of bad news.
Bad News One: As kparal pointed out in the original thread, the
`/latest` endpoint that was added to ResultsDB 2.0 and intended to help
optimize queries like this doesn't handle 'scenario' cases at all: it
only gives you the single 'latest' result for each test case in your
query, there is no option to ask for 'latest by scenario' rather than
'latest by test case'. As we discussed further in the thread,
implementing this in RDB would be difficult (though not impossible) and
not as efficient as latest-by-testcase, due to details of how RDB
actually stores data. So Bodhi has to either:
1. use `/latest` but then have special knowledge of the cases where it
needs to go and run supplemental queries to handle 'scenario' cases
(which is what it currently does for depcheck, and what my pull request
does for openQA for now)
2. just not use /latest and get *all* the results for the items, then
do the latest-by-scenario work itself. This works, and results in
cleaner Bodhi code, but is slower and makes RDB work harder
Bad News Two: The 'just provide a scenario string' approach fails to
solve a problem I didn't think about until I came to actually work from
the consumer end. Say we get all the openQA results for a given update,
then do the 'latest-by-scenario' filtering using the 'scenario' values.
That works, and now we have all the results we need to display. Great.
But...how do we (Bodhi) *display* them?
Currently Bodhi displays each result as a table row, with four columns
- an icon indicating the outcome; the test case name; the result
'note', if there is one; and the time the result was submitted.
But if we just do that, then you really can't tell the difference
between the results for the 'same test, different flavor' cases. And
the key point is, *RDB doesn't really help Bodhi here*. The ugly,
smooshed-together scenario string does the job I meant it to do - lets
Bodhi identify which results it should display - but it doesn't do a
job it ought to do, but which I didn't think about: it doesn't tell
Bodhi how it can cleanly display the results with the same test case
name so that a reader actually knows what the difference is. Bodhi
*could* just stuff the scenario name in the table somewhere (or use it
instead of the test case name), but it wasn't actually designed to be
read and looks pretty ugly.
So I wound up just having to do ugly special-case stuff again, and
effectively 'teach' Bodhi that the 'scenario' element that can actually
vary between openQA results at present is the 'flavor', and teach it to
parse the flavor out of the scenario value (stupidly, we don't actually
report a 'flavor' key in the extradata...I should fix that) and display
it as a label next to the test case name. (There is also pre-existing
similar 'special casing' to have it display the arch as a label, for
the dist.depcheck case we've talked about before.)
Addressing Bad News One is a bit difficult, as mentioned. But perhaps
we can address Bad News Two with a second try at implementing the
'scenario' concept in results. I think, somewhere back in the thread,
kparal suggested that each result could include each scenario-relevant
value in its extradata, and also indicate which keys were the 'scenario
keys' for the result. I thought that was over-complicated at the time,
but it'd actually help address Bad News Two. Bodhi would then have the
opportunity to, say, compare all the scenario values for the results
for the item, ignore ones that are the same for *every* result, and for
the values that actually do vary across some of the results, display
those values in the result table. That wouldn't require it to have
special knowledge.
Just in case this email isn't long enough, here's my Bodhi PR to
implement 'scenario' handling for openQA results, with a nice long
commit message:
https://github.com/fedora-infra/bodhi/pull/1496
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
7:09 p.m.
Excerpts from Adam Williamson's message of 2017-05-02 16:15 -07:00:
So, first the good news: fundamentally, the concept works. I can
tell
Bodhi to query 'all bodhi_update results for the item FEDORA-2017-
4e95959f0d' and then display (only) the latest result for each
'scenario' that appears in the whole set of results.
I think this might be the fundamental problem here. I think the
ResultsDB "item" should not be an update, but the set of packages in an
update. If someone edits the update to change the set of packages in it,
the previous results are invalidated and new tests are kicked off.
Putting aside the issue of how to actually make the "item" be a set of
packages -- I think this would eliminate all of these issues with OpenQA
scenarios.
OpenQA would simply report results with test cases like this, with all
your scenarios and flavours and so forth as different permutations in
the test case name:
graphical_installation.fedora.26.server.x86_64.blahblah
graphical_installation.fedora.26.workstation...
and the existing logic for latest-result-by-testcase will work fine with
no special cases needed in Bodhi. Bodhi would just query for the
ResultsDB "item" consisting of all packages currently in the update.
This also eliminates the problem where an update is edited, and Bodhi
queries for item FEDORA-2017-4e95959f0d but it sees older test results
based on the packages that were previously in the update because the new
test runs are not complete yet.
If you go down that route then it means the *real* problem to solve is
how to represent an "item" which is a set of packages rather than
a single package.
--
Dan Callaghan <dcallagh(a)redhat.com>
Senior Software Engineer, Products & Technologies Operations
Red Hat
7:45 p.m.
On Wed, 2017-05-03 at 10:09 +1000, Dan Callaghan wrote:
Excerpts from Adam Williamson's message of 2017-05-02 16:15
-07:00:
> So, first the good news: fundamentally, the concept works. I can tell
> Bodhi to query 'all bodhi_update results for the item FEDORA-2017-
> 4e95959f0d' and then display (only) the latest result for each
> 'scenario' that appears in the whole set of results.
I think this might be the fundamental problem here. I think the
ResultsDB "item" should not be an update, but the set of packages in an
update. If someone edits the update to change the set of packages in it,
the previous results are invalidated and new tests are kicked off.
Putting aside the issue of how to actually make the "item" be a set of
packages -- I think this would eliminate all of these issues with OpenQA
scenarios.
Uh, I think you're misunderstanding something, because it just wouldn't
do so at all. Even if openQA tested by package rather than by update,
it would still run the same four tests twice for each package, once
starting from a Server base image, once starting from a Workstation
base image. That change would make no difference whatsoever to this
problem.
OpenQA would simply report results with test cases like this, with
all
your scenarios and flavours and so forth as different permutations in
the test case name:
graphical_installation.fedora.26.server.x86_64.blahblah
graphical_installation.fedora.26.workstation...
This is an entirely different suggestion that has nothing to do with
your other suggestion: you're effectively suggesting that we simply
encode the scenario into the test case name. This was actually one of
the alternatives I floated in my very first mail on this topic, but we
didn't really like it for several reasons.
But unlike your other suggestion this one *would* actually work to help
Bodhi's use case; it would let Bodhi just go ahead and treat the test
case name as the 'scenario' with no need for special casing at all
(assuming we also changed the test case name for Taskotron depcheck
tests to something like 'dist.depcheck.i686' and
'dist.depcheck.x86_64').
and the existing logic for latest-result-by-testcase will work fine
with
no special cases needed in Bodhi. Bodhi would just query for the
ResultsDB "item" consisting of all packages currently in the update.
There is no necessary relationship at all between your two suggestions.
This also eliminates the problem where an update is edited, and Bodhi
queries for item FEDORA-2017-4e95959f0d but it sees older test results
based on the packages that were previously in the update because the new
test runs are not complete yet.
There is no such problem, because we tell Bodhi only to look for
results since the update was last edited.
If you go down that route then it means the *real* problem to solve
is
how to represent an "item" which is a set of packages rather than
a single package.
Sorry, but no, I don't think it does. I think you're misunderstanding
what the 'flavor' is, possibly? It's not about running the same test on
multiple packages within an update, it's about running the same test on
the same set of packages starting from a different base disk image
(which is what the 'flavor' represents, in this particular openQA
configuration).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10:29 p.m.
Excerpts from Adam Williamson's message of 2017-05-02 17:45 -07:00:
On Wed, 2017-05-03 at 10:09 +1000, Dan Callaghan wrote:
> Excerpts from Adam Williamson's message of 2017-05-02 16:15 -07:00:
> > So, first the good news: fundamentally, the concept works. I can tell
> > Bodhi to query 'all bodhi_update results for the item FEDORA-2017-
> > 4e95959f0d' and then display (only) the latest result for each
> > 'scenario' that appears in the whole set of results.
>
> I think this might be the fundamental problem here. I think the
> ResultsDB "item" should not be an update, but the set of packages in an
> update. If someone edits the update to change the set of packages in it,
> the previous results are invalidated and new tests are kicked off.
>
> Putting aside the issue of how to actually make the "item" be a set of
> packages -- I think this would eliminate all of these issues with OpenQA
> scenarios.
Uh, I think you're misunderstanding something,
Yes absolutely. :-)
I thought all the "scenario" complexity was because the item
FEDORA-2017-4e95959f0d could have more than one result per test case
*across time* (if an update was edited), but you have already solved
that problem by filtering on update modification timestamp. I didn't
realise that.
But actually the "scenario" is because the item FEDORA-2017-4e95959f0d
could have more than one result per test case because OpenQA tests
different things under the same test case name?
So you are right, I was actually suggesting two totally unrelated
things. Changing the item from being an update to being a set of
packages doesn't actually help anything.
because it just wouldn't
do so at all. Even if openQA tested by package rather than by update,
it would still run the same four tests twice for each package, once
starting from a Server base image, once starting from a Workstation
base image. That change would make no difference whatsoever to this
problem.
> OpenQA would simply report results with test cases like this, with all
> your scenarios and flavours and so forth as different permutations in
> the test case name:
>
> graphical_installation.fedora.26.server.x86_64.blahblah
> graphical_installation.fedora.26.workstation...
This is an entirely different suggestion that has nothing to do with
your other suggestion: you're effectively suggesting that we simply
encode the scenario into the test case name. This was actually one of
the alternatives I floated in my very first mail on this topic, but we
didn't really like it for several reasons.
But unlike your other suggestion this one *would* actually work to help
Bodhi's use case; it would let Bodhi just go ahead and treat the test
case name as the 'scenario' with no need for special casing at all
(assuming we also changed the test case name for Taskotron depcheck
tests to something like 'dist.depcheck.i686' and
'dist.depcheck.x86_64').
Right... so I think what I *actually* wanted to ask was, why not do
this? :-)
I have to admit that I got lost in this thread originally and stopped
following it. Re-reading it all again now, I see that Kamil wrote up
a good summary of the plan here:
https://lists.fedoraproject.org/archives/list/resultsdb-users@lists.fedor...
where option A was to add some new mandatory fields to distinguish the
same test run in different environments. So a ResultsDB result is not
unique on just item+testcase, but rather on
item+testcase+(some other fields that vary per item)+scenario.
Option B was basically what I chimed in with above: jam everything into
the test case name to make it unique.
What you have implemented in your Bodhi patch is actually to just treat
the results as unique on item+testcase+scenario (without arbitrary extra
fields that vary per item). And the different combinations of OpenQA
installation test are smooshed together into the scenario string. Right?
Your Bad News One is that ResultsDB is not efficient at finding the
latest result by item+testcase+scenario. But I am sure we can fix that
in ResultsDB if we want. Worst case, by special-casing "scenario" in the
db schema and promoting it out of extra_data into its own db column with
an index.
Your Bad News Two is that you either display the scenario strings in
Bodhi, and they look kinda ugly and hard to read, or you *don't* display
them in Bodhi and then it looks like there are duplicate test results.
This problem already existed for the test case names though, right?
Those are also ugly, human-unfriendly strings. The solution for the test
cases is that ResultsDB stores some extra metadata about them, with
a link off to a human-friendly page in `ref_url`. (Although I note that
Bodhi right now doesn't show that info anywhere...)
Could you do the same thing for scenarios? Is there an unreasonably
large number of different scenario combinations? Do new ones appear
often?
And if you *could* do that -- maybe it would make sense to just smoosh
it into the test case name, and go back to having results be unique on
item+testcase to keep things simple?
(BTW the reason I am interested in this -- and the reason why I am
trying to help come up with the simplest possible way to model it -- is
because whatever is decided on, the "policy engine" service we are
working on will have to know how to deal with it. Right now we were
assuming that results are unique on item+testcase but there are proving
to be lots of hairy edge cases around that.)
--
Dan Callaghan <dcallagh(a)redhat.com>
Senior Software Engineer, Products & Technologies Operations
Red Hat
11:08 p.m.
On Wed, 2017-05-03 at 13:29 +1000, Dan Callaghan wrote:
What you have implemented in your Bodhi patch is actually to just
treat
the results as unique on item+testcase+scenario (without arbitrary extra
fields that vary per item). And the different combinations of OpenQA
installation test are smooshed together into the scenario string. Right?
More or less, yeah.
Your Bad News One is that ResultsDB is not efficient at finding the
latest result by item+testcase+scenario. But I am sure we can fix that
in ResultsDB if we want. Worst case, by special-casing "scenario" in the
db schema and promoting it out of extra_data into its own db column with
an index.
Well, all things are possible, but this would be a very significant
change to resultsdb, and I suspect Josef wouldn't be a big fan :) But
if we decide that scenario really is a sufficiently important concept
and we're sufficiently confident in our final implementation of it,
yes, this is a *possibility*.
Your Bad News Two is that you either display the scenario strings in
Bodhi, and they look kinda ugly and hard to read, or you *don't* display
them in Bodhi and then it looks like there are duplicate test results.
This problem already existed for the test case names though, right?
Those are also ugly, human-unfriendly strings.
Meh, I wouldn't say so. I think there's kind of an implication in rdb's
design that the test case name can / should be used as an identifier
for squishy humans, and indeed they frequently *are* used as such
(including by Bodhi).
The solution for the test
cases is that ResultsDB stores some extra metadata about them, with
a link off to a human-friendly page in `ref_url`. (Although I note that
Bodhi right now doesn't show that info anywhere...)
I don't *think* this is quite the idea. The URL associated with a test
case is intended to be a place you can go to find out more about the
test case if you would like to, but the test case's name is supposed to
be the primary human-readable identifier, I believe.
Could you do the same thing for scenarios? Is there an unreasonably
large number of different scenario combinations?
I haven't run the numbers, but I suspect the answer is 'yes', and in
general, the scenario concept is a bit too fluid for "hand write a
human-consumable description of every possible scenario identifier" to
be a good idea.
Kamil's suggestion was that rather than a result just providing a
single scenario value that's intended to be a not-very-human-readable
string that represents, in some sense, the entire 'scenario' for a
test, a result (let's say it's for the test update.base_selinux) would
provide something like this:
scenario_keys: [arch, machine, flavor, version, distri]
arch: x86_64
machine: uefi
flavor: server
version: 26
distri: fedora
Then if you're an rdb consumer and you wanted to be really sure you had
the entire 'description' of a given test you could represent it just by
displaying *all* those values, with or without the key name depending
on how it fit into your consumer's interface. But your consumer could
also think like this:
OK, I've got 50 results for this item. The arch, machine, version and
distri for *every* result are the same, so there's probably no need to
show those. But some of the results have different flavors. So I should
probably use 'testcase name - flavor' as the identifier, like so:
"update.base_selinux - server". Hey, that looks pretty good!
We *could* store the scenario_keys together with the test case rather
than with each result. I'm...unsure whether that's a good idea.
Do new ones appear
often?
They certainly can. For the openQA case, for instance, we might come up
with a new flavor or machine at any time, for one of several reasons.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
3:11 a.m.
On Wed, May 3, 2017 at 6:08 AM, Adam Williamson <adamwill(a)fedoraproject.org>
wrote:
Kamil's suggestion was that rather than a result just providing
a
single scenario value that's intended to be a not-very-human-readable
string that represents, in some sense, the entire 'scenario' for a
test, a result (let's say it's for the test update.base_selinux) would
provide something like this:
scenario_keys: [arch, machine, flavor, version, distri]
arch: x86_64
machine: uefi
flavor: server
version: 26
distri: fedora
Then if you're an rdb consumer and you wanted to be really sure you had
the entire 'description' of a given test you could represent it just by
displaying *all* those values, with or without the key name depending
on how it fit into your consumer's interface. But your consumer could
also think like this:
OK, I've got 50 results for this item. The arch, machine, version and
distri for *every* result are the same, so there's probably no need to
show those. But some of the results have different flavors. So I should
probably use 'testcase name - flavor' as the identifier, like so:
"update.base_selinux - server". Hey, that looks pretty good!
Thinking about this...
If two results have the same scenario-related extra args, but differ in
"selinux: 0" vs "selinux: 1", the smart diff you proposed would just
display "update.base_selinux - 0" and "update.base_selinux - 1".
That's not
very helpful.
Of course you can say you'll display the key name as well, so
"update.base_selinux - selinux=0" and "update.base_selinux -
selinux=1".
That looks OK, but it assumes they key name is readable/understandable - it
doesn't have to. Also, if the results differ in several fields, the whole
line starts to be quite long, e.g.:
"flavor=server;machine=uefi;arch=x86_64;selinux=1". Compare this with
"server.uefi.x86_64.selinux-on".
What I'm trying to point out is that no matter what we do, this scenario
description can very easily become unreadable or too long. OpenQA results
scenario happens to be structured in some way, other systems will structure
it in a different way. If we intend to display scenario strings to
distinguish results in UI, we'll either need to live up with complex
identifiers, or ask people to use readable strings and not overpopulate
them with unnecessary data (or come up with a completely different
solution). And assuming we'll ask people to make scenario short and to the
point, it seems easier to me to use the scenario string in its current form
("server.uefi.x86_64.selinux-on") rather than display the full args
("flavor=server;machine=uefi;arch=x86_64;selinux=1").
But I don't have strong opinions here. All the approaches are usable,
provided task creators collaborate in making it readable. I just slightly
prefer the approach we already (at least partially) implemented rather than
re-doing it again.
9:23 a.m.
On Wed, May 3, 2017 at 1:15 AM, Adam Williamson <adamwill(a)fedoraproject.org>
wrote:
On Fri, 2017-03-03 at 15:12 -0800, Adam Williamson wrote:
> If we implement a 'scenario' key, we can then just teach Bodhi to look
> for that and combine it with the item, then maybe if it can't find one,
> treat 'testname-arch' as the scenario (as a fallback for old results,
> results from Taskotron until we teach it to write 'scenario' items into
> its results, and a general fallback for cases where 'scenario' is
> missing for some reason). That could be the general convention, I
> guess: first look for a 'scenario' key, otherwise try and construct one
> from very commonly available keys.
So I come back, a whole two months older and wiser, with a report from
the field on this ;)
And I again forgot all the details in the meantime :) So take my replies
with a grain of salt.
In the end we went ahead with more or less my initial plan for the
'scenario' handling: openQA reports a single 'scenario' value in the
extradata for each report it submits, containing a string
representation of the 'scenario' (it does this by smooshing the values
of all the openQA 'scenario keys' together using periods, for the
record).
Then I got to teach Bodhi to handle openQA results. Happily, this
*does* involve dealing with the 'scenario problem' (though the initial
implementation we did in Bodhi ignores it, and winds up omitting some
openQA results). For each update, we run four of the tests twice, once
on a Server image, once on a Workstation image. In openQA's terms, this
is the same test run on a different 'flavor', and 'flavor' is one of
openQA's "scenario keys".
So for reference, here are a couple of sample results:
https://taskotron.fedoraproject.org/resultsdb/results/13573636
https://taskotron.fedoraproject.org/resultsdb/results/13573633
Same 'testcase', same 'item', same 'arch' even...but different
'scenario':
fedora.26.updates-workstation.x86_64.base_selinux.64bit
fedora.26.updates-server.x86_64.base_selinux.64bit
So, first the good news: fundamentally, the concept works. I can tell
Bodhi to query 'all bodhi_update results for the item FEDORA-2017-
4e95959f0d' and then display (only) the latest result for each
'scenario' that appears in the whole set of results.
However, there are two pieces of bad news.
Bad News One: As kparal pointed out in the original thread, the
`/latest` endpoint that was added to ResultsDB 2.0 and intended to help
optimize queries like this doesn't handle 'scenario' cases at all: it
only gives you the single 'latest' result for each test case in your
query, there is no option to ask for 'latest by scenario' rather than
'latest by test case'. As we discussed further in the thread,
implementing this in RDB would be difficult (though not impossible) and
not as efficient as latest-by-testcase, due to details of how RDB
actually stores data. So Bodhi has to either:
1. use `/latest` but then have special knowledge of the cases where it
needs to go and run supplemental queries to handle 'scenario' cases
(which is what it currently does for depcheck, and what my pull request
does for openQA for now)
We already talked about this
https://github.com/fedora-infra/bodhi/issues/1471 and you didn't like my
proposed solution. But I still think it's a good way forward. We should
continue the discussion here.
My goal is be programatically able to determine whether additional queries
are needed or not, after receiving the /latest results. The simplest
approach is to insist that tasks should only fill 'scenario' key when they
need it - then we know we need to pull the whole history. I understand this
might require non-trivial changes in openqa code. But is it that
insurmountable?
I see only one potential problem here - if you were not able to determine
whether scenario is needed or not, neither during coding nor during
runtime. I can't imagine such a case right now. Do you have some example of
this? My current opinion is that we know well what we're going to test when
we write the task. For example initially you write an openqa test for
booting using bios only. No need for scenario. But later you extend it with
ability to test uefi as well. At that moment you add 'scenario: bios/uefi'
to both. Similarly let's say we have a taskotron task to check shell
variable expansion. First we target only bash, no scenario. Later we start
testing zsh as well, so we add 'scenario: bash/zsh'.
Actually, I just realized one such "don't know in advance" scenario - arch.
Supported arches to run on is likely to be a server-side configuration
value, and the task can't know whether it will be a single or multi value.
So arch-based tasks will always need to put arch in scenario. That's
unfortunate.
To answer your concern from github, what if the newly added results (with
scenario) would be for the same item, therefore clashing with the previous
results with no scenario? No problem, proper results will be available for
the next "item". Of course assuming you added scenario key even to the
original test case. If you want the new results for the current item, just
reschedule the task. /latest will yield results with scenario, the consumer
will pull the history, and will need to be to handle the case when some
historical results don't have scenario and the new ones do.
2. just not use /latest and get *all* the results for the items, then
do the latest-by-scenario work itself. This works, and results in
cleaner Bodhi code, but is slower and makes RDB work harder
/latest has two uses - it's efficient and convenient. It's based on the
premise that most test cases will not need to use scenario and therefore
deduplication is trivial. If there's a substantial part of test cases that
use scenarios, we might get rid of /latest "extra step" and pull all
results for all test cases, as we used to. But let's just be aware that
this is extremely slow for both the client and the server, as it requires
you to do hundreds/thousands of calls (query all test cases one by one).
It's a performance killer. I think this approach is not viable long-term.
If we don't do it with /latest, we'll have to do some optimizations
elsewhere - in the worst case by manually keeping a list of test cases and
their supported extra args values. Or by changing the database design
completely.
Bad News Two: The 'just provide a scenario string' approach fails to
solve a problem I didn't think about until I came to actually work from
the consumer end. Say we get all the openQA results for a given update,
then do the 'latest-by-scenario' filtering using the 'scenario' values.
That works, and now we have all the results we need to display. Great.
But...how do we (Bodhi) *display* them?
Currently Bodhi displays each result as a table row, with four columns
- an icon indicating the outcome; the test case name; the result
'note', if there is one; and the time the result was submitted.
But if we just do that, then you really can't tell the difference
between the results for the 'same test, different flavor' cases. And
the key point is, *RDB doesn't really help Bodhi here*. The ugly,
smooshed-together scenario string does the job I meant it to do - lets
Bodhi identify which results it should display - but it doesn't do a
job it ought to do, but which I didn't think about: it doesn't tell
Bodhi how it can cleanly display the results with the same test case
name so that a reader actually knows what the difference is. Bodhi
*could* just stuff the scenario name in the table somewhere (or use it
instead of the test case name), but it wasn't actually designed to be
read and looks pretty ugly.
The solution seems pretty easy here. Same as test cases should be quite
human readable, so that the reader can easily understand what's going on,
let's keep the scenario string also short and readable. There's no reason
(apart from technical ones on openqa side) to have these ugly scenarios:
fedora.26.updates-workstation.x86_64.base_selinux.64bit
fedora.26.updates-server.x86_64.base_selinux.64bit
Really, they should be just this instead:
workstation
server
Everything else in there is junk (duplicating info from elsewhere). So
let's just fix the scenario strings in openqa?
Of course, we can't forbid people using long and ugly scenarios. But UIs
have a way to deal with that. Just display 10 characters with an ellipsis
in the middle (and expand on click or mouse over). I believe we'll need to
have some level of quality for results shown in Bodhi anyway (in the
future), so we won't show every random task that somebody has coded up for
his personal pleasure. And for those high quality, useful results, we and
package maintainers will push their developers to keep things readable.
Honestly, people will not care about the results unless they fail. And for
the few failing results, even a longer scenario (as openqa currently
produces) wouldn't be that much of a problem, I feel.
So I wound up just having to do ugly special-case stuff again, and
effectively 'teach' Bodhi that the 'scenario' element that can actually
vary between openQA results at present is the 'flavor', and teach it to
parse the flavor out of the scenario value (stupidly, we don't actually
report a 'flavor' key in the extradata...I should fix that) and display
it as a label next to the test case name. (There is also pre-existing
similar 'special casing' to have it display the arch as a label, for
the dist.depcheck case we've talked about before.)
Let me use your own quote against you:
I dunno why you always want to make the scenario concept so
*complicated*.
-- adamw
And now you're parsing it! That is a road to hell :-)
(Even if you didn't parse it and had it as an extra field, that still
presents a lot of added complexity if you want to make it generic, and not
hardcoded to openqa results).
Addressing Bad News One is a bit difficult, as mentioned. But perhaps
we can address Bad News Two with a second try at implementing the
'scenario' concept in results. I think, somewhere back in the thread,
kparal suggested that each result could include each scenario-relevant
value in its extradata, and also indicate which keys were the 'scenario
keys' for the result. I thought that was over-complicated at the time,
but it'd actually help address Bad News Two. Bodhi would then have the
opportunity to, say, compare all the scenario values for the results
for the item, ignore ones that are the same for *every* result, and for
the values that actually do vary across some of the results, display
those values in the result table. That wouldn't require it to have
special knowledge.
To me, this seems to me like a lot of work and complexity for something
that isn't even a problem. Keeping the scenario string short and readable
is the simple solution here. And even in the worst case with long scenario
strings, I don't think it hurts so much.
Now, going back to Bad News One. I'm going back and forth on approaches to
that one. We have this triangle:
simple
/ \
fast -- generic
And it seems we can only get two vertices out of three, as usual. We have
the simple generic solution right now. The fast generic solution would be
the one with "just latest results" middleware, or alternatively lots of
latest/scenario optimization, possibly combined with keeping of
testcases+available extra args lists to be used by consumers. Lots of
complexity for everyone. The simple and fast solution is making the
database schema directly tailored to our Fedora use cases, making scenario
searches fast.
Now, regarding sacrificing generality. I think the fact that RHEL folks are
interested in ResultsDB is partly due to the fact that we decided to keep
the project generic in the first place. If we tied it heavily to Fedora,
maybe they wouldn't be able to use it internally. That being said, there
needs to be a reasonable limit on this. If we don't expect ResultsDB to be
used in other distributions any time soon, and we don't plan to use it for
other projects outside of release validation, maybe we can start tailoring
it towards the common denominator of Fedora+RHEL+CentOS use cases? This
would mean making scenario a mandatory, indexed field for fast searches.
Maybe having arch as a first-class field as well. And maybe some other
optimizations.
Another idea would be to revise what we actually expect from ResultsDB. Do
we really need it to hold every submitted result for all the history? I
don't think we really do, in Fedora. And we definitely don't need 16000
depcheck results for a NVR, that's bat-shit crazy. So we could make
ResultsDB prune the outdated results regularly (and here I mean pruning the
database itself, not standing up a middleware for this). The actual policy
can be very flexible, the pruning could happen on each new result, or
regularly once a day during quiet hours, and we could keep only the latest
result, or the last X results, or all the last results for the past X days
before the latest result, etc. This pruning would be done by a different
process, with its own logic (which would implement how to recognize unique
results), so the database itself would still be generic, unaware of the
pruning process and logic. If we did this, it would probably solve most of
our performance issues, and we would probably be able to use the simplest
approach "give me all of history" and process that in the consumer. But
here we might clash with the RHEL use cases - pruning outdated results
might be a no-go for them. And while it's true that they're able to throw
more hardware at it, to counteract lack of pruning, it might not scale good
enough. A good topic to discuss, if we decide to investigate this approach.
9:30 a.m.
On Wed, May 10, 2017 at 4:23 PM, Kamil Paral <kparal(a)redhat.com> wrote:
tl;dr; but:
... The simple and fast solution is making the database schema
directly
tailored to our Fedora use cases, making scenario searches fast....
Not really simple, honestly, especially if by "scenario searches" you mean
"retrieving a list of results deduplicated by scenario" - as it is the same
as that middleware thing. Only this would be embedded in ResultsDB, and
that one would be a separate thing (which is good, btw, and as long as
there is no rigorous specification, I'm not willing to put that stuff
directly in resultsdb, as it would only cause more trouble than gain in the
end).
2:39 a.m.
On Wed, May 10, 2017 at 4:30 PM, Josef Skladanka <jskladan(a)redhat.com>
wrote:
On Wed, May 10, 2017 at 4:23 PM, Kamil Paral <kparal(a)redhat.com> wrote:
tl;dr; but:
> ... The simple and fast solution is making the database schema directly
> tailored to our Fedora use cases, making scenario searches fast....
>
Not really simple, honestly, especially if by "scenario searches" you mean
"retrieving a list of results deduplicated by scenario" - as it is the same
as that middleware thing.
I didn't really mean that when I wrote it. I meant it would be fast to ask
for all scenarios available for $item, and then you could ask for the
latest result for each scenario. But I admit I don't know much about
databases, so...
as long as there is no rigorous specification,
Hahahah. This is not NASA, Josef, this is open source! :o) Even if you
received a rigorous specification, the requirements would change in 6
months :)
But I would like to know your thoughts on my last paragraph in my previous
email.
3:02 a.m.
On Thu, May 11, 2017 at 9:39 AM, Kamil Paral <kparal(a)redhat.com> wrote:
On Wed, May 10, 2017 at 4:30 PM, Josef Skladanka
<jskladan(a)redhat.com>
wrote:
>
> On Wed, May 10, 2017 at 4:23 PM, Kamil Paral <kparal(a)redhat.com> wrote:
> tl;dr; but:
>
>> ... The simple and fast solution is making the database schema directly
>> tailored to our Fedora use cases, making scenario searches fast....
>>
>
> Not really simple, honestly, especially if by "scenario searches" you
> mean "retrieving a list of results deduplicated by scenario" - as it is
the
> same as that middleware thing.
>
I didn't really mean that when I wrote it. I meant it would be fast to ask
for all scenarios available for $item, and then you could ask for the
latest result for each scenario. But I admit I don't know much about
databases, so...
Honestly, this is just a different angle on the same thing - if you want to
store item->ANY(scenario) mappings, then you can just go and store the
item->LATEST(scenario), and do what you wanted to do in the first place. I
can see why this can seem different to you, but it really is not. As I said
many times - each time you want to have "all of something" it means either
having very specific table structure, or just traversing the whole database.
>
> as long as there is no rigorous specification,
>
Hahahah. This is not NASA, Josef, this is open source! :o) Even if you
received a rigorous specification, the requirements would change in 6
months :)
No, it is not. But this sofware engineering - knowing _what_, and even more
importantly _why_ you want to do something is an integral part of the
development process. We sure can have a steaming pile of quick hacks and
workarounds, but if that is the desired/expected outcome, I don't really
see a point of having the "design conversations".
But I would like to know your thoughts on my last paragraph in my
previous
email.
Ad pruning - sure, why not. Even though I don't think that it is the right
way of tackling this, should we decide to do it, my preferred way would be
"move to archive" instead "delete".
But we are getting back to those NASA-like specifications and policies
here, so... You still need to define "what is unique", "what is
latest",
"how long do we keep stuff" - just like with any other solution. The only
difference here is, that instead of adding a layer on top of a datastore
(resultsdb), you decided to just prune the data in said data store.
I'm not trying to get personal, or petty, but in almost three months, we
were not able to agree even on what the "scenario" should be. I came from
"this is nonsense" to "OK, if used systematically, this can be
helpful",
Adam is on the path of switching from "Why do you guys need to make
everything so overcomplicated, this is but a string!" to "Well, some of
what you said actually makes sense, now that I think about it a bit more",
but that's about it. We do "something" now, but we don't do it in a
known,
defined way, so while this works for some cases, it does not for others.
And more than that - we are not even really sure where it does and does not
work.
Yes, this was slightly hyperbolic, but I hope you get the point. And I
might be a data-freak, but for some reason, I don't think it is a good idea
to sort "we have too much data" by deleting it haphazardly.
J.
3:13 a.m.
On Thu, May 11, 2017 at 10:02 AM, Josef Skladanka <jskladan(a)redhat.com wrote:
On Thu, May 11, 2017 at 9:39 AM, Kamil Paral
<kparal(a)redhat.com> wrote:
> On Wed, May 10, 2017 at 4:30 PM, Josef Skladanka <jskladan(a)redhat.com > wrote:
> >> On Wed, May 10, 2017 at
4:23 PM, Kamil Paral <kparal(a)redhat.com> wrote:
>> tl;dr; but:
> >>> ... The simple and fast solution is making the
database schema directly
>>> tailored to our Fedora use cases, making scenario searches fast....
>> > >>
Not really simple, honestly, especially if by "scenario searches" you
>> mean "retrieving a list of results deduplicated by scenario" - as it is
the
>> same as that middleware thing.
> > I didn't really mean that
when I wrote it. I meant it would be fast to
> ask for all scenarios available for $item, and then you could ask for the
> latest result for each scenario. But I admit I don't know much about
> databases, so...
Honestly, this is just a different angle on the same thing - if you want
to store item->ANY(scenario) mappings, then you can just go and store the
item->LATEST(scenario), and do what you wanted to do in the first place. I
can see why this can seem different to you, but it really is not. As I said
many times - each time you want to have "all of something" it means either
having very specific table structure, or just traversing the whole database.
> >> as long as there is no
rigorous specification,
> > Hahahah. This is not NASA,
Josef, this is open source! :o) Even if you
> received a rigorous specification, the requirements would change in 6
> months :)
No, it is not. But this sofware
engineering - knowing _what_, and even
more importantly _why_ you want to do something is an integral part of the
development process. We sure can have a steaming pile of quick hacks and
workarounds, but if that is the desired/expected outcome, I don't really
see a point of having the "design conversations".
> But I would like to know your thoughts on my last paragraph in my
> previous email.
Ad pruning - sure, why not. Even though I don't think that it is the right
way of tackling this, should we decide to do it, my preferred way would be
"move to archive" instead "delete".
But we are getting back to those NASA-like specifications and policies
here, so... You still need to define "what is unique", "what is
latest",
"how long do we keep stuff" - just like with any other solution. The only
difference here is, that instead of adding a layer on top of a datastore
(resultsdb), you decided to just prune the data in said data store.
I'm not trying to get personal, or petty, but in almost three months, we
were not able to agree even on what the "scenario" should be. I came from
"this is nonsense" to "OK, if used systematically, this can be
helpful",
Adam is on the path of switching from "Why do you guys need to make
everything so overcomplicated, this is but a string!" to "Well, some of
what you said actually makes sense, now that I think about it a bit more",
but that's about it. We do "something" now, but we don't do it in a
known,
defined way, so while this works for some cases, it does not for others.
And more than that - we are not even really sure where it does and does not
work.
Yes, this was slightly hyperbolic, but I hope you get the point. And I
might be a data-freak, but for some reason, I don't think it is a good idea
to sort "we have too much data" by deleting it haphazardly.
J.
Bah, do you also hate it, when you hit ctrl+enter instead of shift+enter,
and the email gets sent? ... Following up with that last sentence:
I sure agree, that having thousands of Depcheck results is nonsense. And in
this special case, I really agree that pruning the data is a good idea (if
we have a very well defined way of devising the "this is just a duplicate"
thing, that is). I also agree, that we don't need to have the whole history
of all the results in one place, that is fast-read access. But it still
makes sense (to me at least) to just have an archiving policy, rather than
deletion policy, as the first step. And to be honest, I don't think that
the policy for data retention has anything to do with result
de-duplication. Sure it makes the "stupid" undefined approach easier - we
could be going on saying "just download it all, and decide for yourself" a
bit longer, and effectively stall the "NASA-level" decision making. And
once again - I'm not saying that it is necessarily a wrong step to take
now. But we must be very consentient about what we are doing, and why.
Because having less data in the database does not really help with deciding
"what is the latest result in this usecase".
So, to sum it up a bit - I am the last person to go against making our
stuff a bit (a lot) more tailored to the usecases we have. But I need to
have those usecases defined, and ideally have policies in place to support
those usecases, before I feel it is a good idea to do it.
J.
3:35 a.m.
On Thu, May 11, 2017 at 10:13 AM, Josef Skladanka <jskladan(a)redhat.com>
wrote:
Bah, do you also hate it, when you hit ctrl+enter instead of shift+enter,
and the email gets sent?
Nope, I configured my email client (gmail) to be able to undo sending
emails in such case (20-second delay in my case). (I don't even know what
shift+enter is for, it doesn't seem to do anything different than a
standard enter).
... Following up with that last sentence:
I sure agree, that having thousands of Depcheck results is nonsense. And
in this special case, I really agree that pruning the data is a good idea
(if we have a very well defined way of devising the "this is just a
duplicate" thing, that is). I also agree, that we don't need to have the
whole history of all the results in one place, that is fast-read access.
But it still makes sense (to me at least) to just have an archiving policy,
rather than deletion policy, as the first step.
I didn't realize that, but that's conceptually no different from what I
proposed, right? It can still be an external process that prunes the last
day's worth of data during night, just instead of throwing those away
completely, it saves them to a secondary database. So it can be still
implemented outside of resultsdb. Correct?
Also, this could also be OK for RHEL folks, they could have different
policies for archiving, and the old results would still be available, just
in a different database (and with slower access).
And to be honest, I don't think that the policy for data
retention has
anything to do with result de-duplication.
Nope.
Sure it makes the "stupid" undefined approach easier - we
could be going
on saying "just download it all, and decide for yourself" a bit longer, and
effectively stall the "NASA-level" decision making.
Yes, that's how it was meant. If not for depcheck/upgradepath style of
reporting, we might not have had this discussion at all, because all the
other tasks report a very reasonable amount of results (single digits) for
each item.
And once again - I'm not saying that it is necessarily a wrong
step to
take now. But we must be very consentient about what we are doing, and why.
Because having less data in the database does not really help with deciding
"what is the latest result in this usecase".
More brainpower would be welcome on this topic :) If we can't make it
auto-resolving, we can still go back to defining a huge list of all
testcases in the database and their available extra args values, so that
consumers can send very specific queries.
So, to sum it up a bit - I am the last person to go against making our
stuff a bit (a lot) more tailored to the usecases we have. But I need to
have those usecases defined, and ideally have policies in place to support
those usecases, before I feel it is a good idea to do it.
J.
>
6:19 a.m.
On Thu, May 11, 2017 at 10:35 AM, Kamil Paral <kparal(a)redhat.com> wrote:
On Thu, May 11, 2017 at 10:13 AM, Josef Skladanka
<jskladan(a)redhat.com>
wrote:
>
> Bah, do you also hate it, when you hit ctrl+enter instead of shift+enter,
> and the email gets sent?
>
Nope, I configured my email client (gmail) to be able to undo sending
emails in such case (20-second delay in my case). (I don't even know what
shift+enter is for, it doesn't seem to do anything different than a
standard enter).
Mine is configured to enter u"\u2029".
> ... Following up with that last sentence:
>
> I sure agree, that having thousands of Depcheck results is nonsense. And
> in this special case, I really agree that pruning the data is a good idea
> (if we have a very well defined way of devising the "this is just a
> duplicate" thing, that is). I also agree, that we don't need to have the
> whole history of all the results in one place, that is fast-read access.
> But it still makes sense (to me at least) to just have an archiving policy,
> rather than deletion policy, as the first step.
>
I didn't realize that, but that's conceptually no different from what I
proposed, right? It can still be an external process that prunes the last
day's worth of data during night, just instead of throwing those away
completely, it saves them to a secondary database. So it can be still
implemented outside of resultsdb. Correct?
Sure thing. I could have been clearer - I'm not against removing the data
from the "current" dataset (event though I consider it a temp hack for the
issues at hand), just against deleting those completely, since we have no
idea whether we want those or not, I'm leaning towards the safer side.
Also, this could also be OK for RHEL folks, they could have
different
policies for archiving, and the old results would still be available, just
in a different database (and with slower access).
> And to be honest, I don't think that the policy for data retention has
> anything to do with result de-duplication.
>
Nope.
> Sure it makes the "stupid" undefined approach easier - we could be going
> on saying "just download it all, and decide for yourself" a bit longer,
and
> effectively stall the "NASA-level" decision making.
>
Yes, that's how it was meant. If not for depcheck/upgradepath style of
reporting, we might not have had this discussion at all, because all the
other tasks report a very reasonable amount of results (single digits) for
each item.
> And once again - I'm not saying that it is necessarily a wrong step to
> take now. But we must be very consentient about what we are doing, and why.
> Because having less data in the database does not really help with deciding
> "what is the latest result in this usecase".
>
More brainpower would be welcome on this topic :) If we can't make it
auto-resolving, we can still go back to defining a huge list of all
testcases in the database and their available extra args values, so that
consumers can send very specific queries.
I'm not getting the "huge list of all testcases in the database and their
available extra args values", honestly, but consumers can send very
specific queries already... Also, more brainpower is fine, but I'd rather
convince one of you in person, instead of leading a battle on two fronts :)
Also, our views tend to be in the same general direction here, so I only
chime in, disagreeing, if I feel like I really need to. Probably not _the_
best way of doing it, but since we already had those three days of utter
hatred and helpful discussions... :D
> So, to sum it up a bit - I am the last person to go against making our
> stuff a bit (a lot) more tailored to the usecases we have. But I need to
> have those usecases defined, and ideally have policies in place to support
> those usecases, before I feel it is a good idea to do it.
>
> J.
>
>>
>
>
_______________________________________________
Resultsdb-users mailing list -- resultsdb-users(a)lists.fedoraproject.org
To unsubscribe send an email to resultsdb-users-leave@lists.
fedoraproject.org
12:58 a.m.
Excerpts from Kamil Paral's message of 2017-05-11 10:35 +02:00:
On Thu, May 11, 2017 at 10:13 AM, Josef Skladanka
<jskladan(a)redhat.com>
wrote:
> ... Following up with that last sentence:
>
> I sure agree, that having thousands of Depcheck results is nonsense. And
> in this special case, I really agree that pruning the data is a good idea
> (if we have a very well defined way of devising the "this is just a
> duplicate" thing, that is). I also agree, that we don't need to have the
> whole history of all the results in one place, that is fast-read access.
> But it still makes sense (to me at least) to just have an archiving policy,
> rather than deletion policy, as the first step.
>
I didn't realize that, but that's conceptually no different from what I
proposed, right? It can still be an external process that prunes the last
day's worth of data during night, just instead of throwing those away
completely, it saves them to a secondary database. So it can be still
implemented outside of resultsdb. Correct?
Also, this could also be OK for RHEL folks, they could have different
policies for archiving, and the old results would still be available, just
in a different database (and with slower access).
Unfortunately this will be basically a non-starter. Imagine that you
need to go back to an old Bodhi update that shipped three years ago for
some kind of auditing purpose. Bodhi needs to be able to show the same
results, waivers, and decisions today as it did three years ago.
Okay, maybe with Bodhi it is okay to just wing it and say "sorry the
data is gone now" but that doesn't really fly if you imagine Errata Tool
in place of Bodhi.
So I think relying on deleting/moving/archiving data out of the
ResultsDB database just to make it perform well, is not a viable option.
There really has to be a proper solution to the "how do I find the
latest relevant results" problem.
(Doing some one-off pruning to handle pathological cases like old
depcheck results is a different story -- that would be more just about
reducing the size of Fedora's ResultsDB a little, rather than being
crucial to making ResultsDB perform well.)
--
Dan Callaghan <dcallagh(a)redhat.com>
Senior Software Engineer, Products & Technologies Operations
Red Hat
11:32 a.m.
On Thu, 2017-05-11 at 10:02 +0200, Josef Skladanka wrote:
I'm not trying to get personal, or petty, but in almost three
months, we
were not able to agree even on what the "scenario" should be. I came from
"this is nonsense" to "OK, if used systematically, this can be
helpful",
Adam is on the path of switching from "Why do you guys need to make
everything so overcomplicated, this is but a string!" to "Well, some of
what you said actually makes sense, now that I think about it a bit more",
but that's about it. We do "something" now, but we don't do it in a
known,
defined way, so while this works for some cases, it does not for others.
And more than that - we are not even really sure where it does and does not
work.
I'd say that's going a bit far. We *do* do it in a known, defined way
(at least I thought we'd agreed on one). However, it doesn't yet
entirely solve the problem that needs solving. This doesn't seem
surprising, we don't usually get things exactly right with v1.0.
I'm still thinking it through, for now, plus I'm trying to deal with
Beta blockers. The code we have for the present does do the job, even
if it's not pretty, so I'm not super desperate to change stuff right
now. We can think about it for a bit.
One thought I have that is pretty solid (and unchanged) is that I
really *do* think the 'scenario problem' is going to affect the
majority of results. At the very least, a lot of tests are going to be
per-arch. I do believe the idea that the current /latest mechanism will
work for most cases is wrong.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10:50 a.m.
> a) query
`results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
> Rawhide-20170228.n.0.iso` and then go through all the results, make
> them unique by eliminating everything that has the same 'scenario',
> and work with that
This was one of the primary cases I was thinking of, yes. This is the
case that releng would be using to decide whether to release a compose,
for instance.
I'll go slightly off-topic here, but consider this: 'scenario' will help us
identify unique results. But it will not help us decide whether "all" results
have been submitted (maybe some are still running, maybe some have crashed hard and not
sent any results). So there's a good chance that rel-eng tools will need to have deep
knowledge of the testcases anyway. Not just list of all monitored testcases, but also that
this particular test case needs to be performed for bios and uefi (or, that this
particular testcase needs to be present with these two scenarios).
So while having scenarios seems definitely helpful in certain cases, it might not help us
in avoiding having too much knowledge in the gating consumers. Just food for thought.
It's also the case for listing openQA results in Bodhi:
we'll want Bodhi to grab all results for the update, then de-duplicate
by scenario (taking only the most recent result for each scenario).
> b) query `results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
> Rawhide-20170228.n.0.iso&unique_key=scenario` and have the
> duplication filtering implemented inside resultsdb (this way it seems
> it can be implemented in a generic fashion), work with that
Oo, is that possible? If so, neat. But conceptually it's the same as
a), right? Just a case of where the filtering gets done.
Yes, it would be the same, just more resource-friendly. Not sure if Josef will be inclined
to implement this, he became quite good in saying 'no' :)
> c) query `results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
> Rawhide-20170228.n.0.iso&scenario=fedora.Rawhide.Server-dvd-
> iso.x86_64.server_realmd_join_kickstart.64bit` and work with that.
> But in this case the consumer would again have to have a deep
> knowledge of how the scenario is constructed, which would miss the
> whole point, I think.
Well, see the above examples - I was kinda envisioning that you'd
*discover* the scenario you're interested in from some kind of existing
result, or just query for all results from two *different* composes,
then use the 'scenario' key to compare the results for the two composes
(so you can say 'ok, scenario A passed in compose 1 but failed in
compose 2'). This is the sort of thing check-compose does - that's how
it produces a list of 'tests that failed today but passed yesterday',
etc. It doesn't *know* what the scenarios are before it queries the
results, but *once it has the results* it calculates the scenarios and
uses them for comparison.
Fair enough, for this use case it's really useful. But it will not be the primary way
of using scenarios, and that was my point, clarifying how it's going to be used
primarily.
> > Note I don't think we should include the 'item' in the scenario for
all
> > the reasons discussed above; it should just be understood that for
> > different purposes you might want to query on "scenario + item" or
just
> > "scenario".
>
> I don't understand why you would ever want to put testcase name into
> the scenario field. Both the item and the scenario are defined
> externally in a well known fields that are mandatory to fill in.
> There's no guessing like with 'arch' or 'firmware' (whether it
> exists, what the allowed values are). So why wouldn't you specify
> only the extra args that make the execution unique in the scenario
> field? If you ever need to search through multiple testcases with the
> same scenario value (I don't know why you would need that, but let's
> assume you do), you can always do that either for a particular
> testcase like this:
> results?testcases=compose.base_selinux&scenario=fedora.Rawhide.Server
> -dvd-iso.x86_64.64bit
> or across many testcases by using a list or a wildcard:
> results?testcases=compose.*&scenario=fedora.Rawhide.Server-dvd-
> iso.x86_64.64bit
>
> I don't think you even want to do this query:
> results?scenario=foobar
> because you can *bet* that you'll eventually receive some results
> from scratch.tmp.testthis.fake namespace or similar. You should
> always specify at least a wildcarded testcase (compose.openqa.*). I'd
> like our system to allow in-development, proof-of-concept and
> experimental tools to run and submit results. There will be errors,
> mistakes, copycats, everything. But they will be constrained to their
> namespaces. And so should you also limit your queries only to those
> namespaces that are known and maintained with the level of quality
> you need for your purposes. You can imagine the concept to be the
> same as e.g. github - you always work in your namespace, and there's
> no global namespace, because that's just asking for trouble.
Well, the funny thing is, for all the same reasons I come to the other
conclusion :) I entirely agree that you'd never want to use the
scenario *without* the test case name...so in that case, saying "you
should always query for the test case name plus the scenario" seems
like pointless make-work, right? If we know that there's never any
point in just using the scenario values *without* the test case name,
why not just stuff the test case name right into the scenario value so
you only have to query for 'scenario', rather than always having to
query for both 'test case name' and 'scenario'?
To be fair, I can think only of a few minor points:
* Data duplication, makes the field longer and harder to read.
* A possibility for errors. Since tasks are allowed to add arbitrary suffixes to their
testcase names, a task can create tens of "sub-testcases" (e.g. look at
dist.rpmgrill*). The scenario field is not automatically generated but fully under
task's control. There might be copy&paste errors or logical errors in the code,
and the scenario might not correctly reflect the used testcase name.
* Consistency (even though that can be arguable). I'd like to be clear that
testcase+item combination is always the default way to consume results. For more complex
tests, it might also require looking at scenario to distinguish unique results. If I write
instructions that say that results are identified by item+scenario if scenario exists,
otherwise as testcase+item, it seems more complex and less obvious to me. (But this is
maybe just about phrasing).
But it's really a pretty minor point. The system would work fine either
way, it just slightly changes what consumers have to do.
Does anyone else have thoughts on this? I'd like to either land the
'scenario' change in the openQA reporter, or definitely decide we want
to do something else, tomorrow or early next week, so I can move
forward with the 'is this compose releasable?' script and displaying
openQA update test results in Bodhi...
Since you're clearly powered by Duracell batteries [1], please go ahead and implement
this. If we decide to implement this later in resultsdb directly, we can always go and
simplify the consumer code.
Btw, if you end up submitting patches to Bodhi and using 'type=bodhi_update'
queries, please also use the 'since=' argument [2] and set it to the update's
'date_modified' timestamp. That doesn't really implement the higher
reliability of such results (as we talked elsewhere), but it doesn't stress resultsdb
so much, which is always good (we can't do the same for type=koji_build easily, but we
can for type=bodhi_update).
Thanks.
[1] https://www.youtube.com/watch?v=kaF6FxmixJk , am I really that old?
[2] http://docs.resultsdb20.apiary.io/#reference/0/results/browse-the-results...
12:54 p.m.
On Mon, 2017-03-06 at 11:50 -0500, Kamil Paral wrote:
> > a) query
`results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
> > Rawhide-20170228.n.0.iso` and then go through all the results, make
> > them unique by eliminating everything that has the same 'scenario',
> > and work with that
>
> This was one of the primary cases I was thinking of, yes. This is the
> case that releng would be using to decide whether to release a compose,
> for instance.
I'll go slightly off-topic here, but consider this: 'scenario' will
help us identify unique results. But it will not help us decide
whether "all" results have been submitted (maybe some are still
running, maybe some have crashed hard and not sent any results). So
there's a good chance that rel-eng tools will need to have deep
knowledge of the testcases anyway. Not just list of all monitored
testcases, but also that this particular test case needs to be
performed for bios and uefi (or, that this particular testcase needs
to be present with these two scenarios).
So while having scenarios seems definitely helpful in certain cases,
it might not help us in avoiding having too much knowledge in the
gating consumers. Just food for thought.
I have actually been thinking about that problem as well :) And it is
indeed a tricky one. Bodhi already basically ducks the question; it
just shows whatever results are there at the time. We don't have a good
answer for cases like 'can we ship this compose?' or check-compose,
cases where the tool needs to know that testing is complete.
My best idea so far is that we should implement a 'testing complete'
fedmsg with a consistent name for each testing system - so
whatever.test.system.prefix.is.testing.complete or something like that.
Then at least the consumer only has to know which test systems it cares
about, and it can quite trivially trigger whenever it has
'testing.complete' messages from each one for the relevant item. I
really dunno if we can do anything better than this. I suppose if we
also required all systems to send a 'testing.started' message, we could
have some sort of meta-consumer check when it's seen a
'testing.complete' message for each 'testing.started' message and send
out a 'all testing for compose X is complete' message, but even if we
*do* that, it means that any single system can prevent the 'all testing
complete' message going out if it's slow or broken, even if it's one a
given consumer doesn't actually care about. So I don't think consumers
would use such a message even if we managed to build it...
Note that I've already implemented something like this for openQA,
though it doesn't work exactly the way I described at present. Each
openQA 'test complete' message contains a count of currently running or
scheduled tests for the same build, and things like check-compose
trigger when they see a 'job done' message with that count at 0.
> Well, see the above examples - I was kinda envisioning that
you'd
> *discover* the scenario you're interested in from some kind of existing
> result, or just query for all results from two *different* composes,
> then use the 'scenario' key to compare the results for the two composes
> (so you can say 'ok, scenario A passed in compose 1 but failed in
> compose 2'). This is the sort of thing check-compose does - that's how
> it produces a list of 'tests that failed today but passed yesterday',
> etc. It doesn't *know* what the scenarios are before it queries the
> results, but *once it has the results* it calculates the scenarios and
> uses them for comparison.
Fair enough, for this use case it's really useful. But it will not be
the primary way of using scenarios, and that was my point, clarifying
how it's going to be used primarily.
Eh, I dunno if there's much use in denoting one usage as 'primary' and
another as 'not primary'. But sure.
> Well, the funny thing is, for all the same reasons I come to the
other
> conclusion :) I entirely agree that you'd never want to use the
> scenario *without* the test case name...so in that case, saying "you
> should always query for the test case name plus the scenario" seems
> like pointless make-work, right? If we know that there's never any
> point in just using the scenario values *without* the test case name,
> why not just stuff the test case name right into the scenario value so
> you only have to query for 'scenario', rather than always having to
> query for both 'test case name' and 'scenario'?
To be fair, I can think only of a few minor points:
* Data duplication, makes the field longer and harder to read.
This seems pretty minor - honestly there's almost no case for a human
to be manually reading the field ever, except for debugging purposes. I
never actually go and manually inspect what the openQA scenario values
are, there's just no use case for that - it's just a concept that the
tools use to do a job.
* A possibility for errors. Since tasks are allowed to add arbitrary
suffixes to their testcase names, a task can create tens of "sub-
testcases" (e.g. look at dist.rpmgrill*). The scenario field is not
automatically generated but fully under task's control. There might
be copy&paste errors or logical errors in the code, and the scenario
might not correctly reflect the used testcase name.
Hum, I guess, though it doesn't actually *matter* if the scenario still
does its job of being unique within the results for a single item, and
being attached to 'the same' test across multiple items. We're
definitely not going to want consumers to get into the business of
parsing things out of the scenario value, if they actually want one of
the values that makes up the scenario value they should request it
directly. But sure, it's a possibility.
* Consistency (even though that can be arguable). I'd like to be
clear that testcase+item combination is always the default way to
consume results. For more complex tests, it might also require
looking at scenario to distinguish unique results. If I write
instructions that say that results are identified by item+scenario if
scenario exists, otherwise as testcase+item, it seems more complex
and less obvious to me. (But this is maybe just about phrasing).
I...dunno if this is going to be a tenable place to stand, to be
honest. Funnily enough I went through the same process in a somewhat
different context. When thinking about relval-ng in my head, I
initially had the idea that we could kill the Wikitcms 'environment'
columns - which, if you think about it, are basically this 'scenario'
concept - by associating all results with a specific image - which is,
if you think about it, the 'item' concept.
NOTE IF YOU'VE NO IDEA WHAT THIS IS ABOUT: we're talking about the wiki
pages Fedora QA uses to store release validation test results, like
this:
https://fedoraproject.org/wiki/Test_Results:Fedora_26_Branched_20170302.n...
Note each row is for one 'test case', and most rows have multiple
columns for results in different 'environments' (i.e. scenarios) for
that test case. 'relval-ng' is the working name of a system we're
proposing to build to replace the wiki system.
So I was effectively thinking the same as you: we can always just
identify a result as the combination of "a test case" and "a tested
item".
But then I thought, nah, it's still not really that simple. In relval-
ng as in openQA, the easiest example is BIOS vs. UEFI: we have some
tests, e.g. Anaconda_User_Interface_Basic_Video_Driver , where we want
to test on both BIOS and UEFI. This involves the same test case and the
same tested 'item' - an x86_64 installer image - but a different
'scenario'. Unless you start stuffing scenario items into the test case
name (basicvideo.uefi , basicvideo.bios ?) or item name (foobar.iso
BIOS, foobar.iso UEFI ?) - an idea we rejected back at the start of
this thread - I fundamentally don't see a way around this.
There are other examples, though - take the 'Default boot and install'
table, where we consider installing with the same image to a VM and to
bare metal as being different scenarios, and also installing with the
same image written to an optical disc and written to a USB stick.
There's just fundamentally no way around that without invoking the
'scenario' concept, or something very much like it but phrased
differently.
As we've also already noted, the existing package tests have also run
into this problem: "test name + item" is not sufficient to really
define the result, as 'item' is a source package but the same test is
run for all binary package arches and may have different results on
different arches (IIUC).
Basically, my contention is that this 'scenario' concept is going to
just keep on turning up, all over the place, and it's going to be more
realistic to expect to be dealing with a 'scenario' most of the time,
than to expect to be dealing with just 'test case plus item' most of
the time and consider 'scenario' to be a kind of "advanced" thing.
So I think I still slightly prefer the idea of including the test name
in the scenario value, but I really don't have a strong preference
either way. Let's just pick one approach and go with it.
Note if I'm being honest I have a practical reason for this, as openQA
defines the test case name as one of the 'scenario keys', so if we go
with 'test name not in scenario value', in the openQA reporter we'll
have to take the list of 'scenario keys' and then remove the test name
from it. Obviously this is a trivial point, but just to be honest, it's
the real reason why I initially assumed we'd put the test name in the
scenario value: just cos that's how openQA is currently set up to do
it, if you do it the easiest way :)
> But it's really a pretty minor point. The system would work
fine either
> way, it just slightly changes what consumers have to do.
>
> Does anyone else have thoughts on this? I'd like to either land the
> 'scenario' change in the openQA reporter, or definitely decide we want
> to do something else, tomorrow or early next week, so I can move
> forward with the 'is this compose releasable?' script and displaying
> openQA update test results in Bodhi...
Since you're clearly powered by Duracell batteries [1], please go
ahead and implement this. If we decide to implement this later in
resultsdb directly, we can always go and simplify the consumer code.
Right. And I've merged the version where the test name is included in
the scenario value, but again, we can change this later if we want to.
So long as we write the consumer code to query for 'scenario plus test
name', even if it happens to hit an older result where the test name
was included in the scenario, no harm is done, the correct result will
be achieved.
So yeah, openQA results now include a 'scenario' value. For openQA the
scenario value is constructed by joining the values for all keys openQA
considers 'scenario keys' with periods, but we don't actually have to
make this consistent between different systems - the only requirement
is that the value correctly identify the scenario. I can try and send a
patch for Taskotron to do this as well, if you like, or would you
rather do it?
I guess for now I'll adjust autocloudreporter to just provide a hard
coded scenario value, like it provides a hard coded 'test name', in
case we go with the 'test name included in scenario' approach; if we
don't I can just take that back out again.
Btw, if you end up submitting patches to Bodhi and using
'type=bodhi_update' queries, please also use the 'since=' argument
[2] and set it to the update's 'date_modified' timestamp. That
doesn't really implement the higher reliability of such results (as
we talked elsewhere), but it doesn't stress resultsdb so much, which
is always good (we can't do the same for type=koji_build easily, but
we can for type=bodhi_update).
I'm planning to do this, but I did want to talk over with you guys what
would be appropriate for the Taskotron results. Bodhi is definitely
going to have to query for 'type=bodhi_update' results where the item
is the update ID in order to find the openQA results. But I don't know
if we should just make it find the Taskotron results in the same way,
or if it's best to have it query for both koji_build and bodhi_update
results and somehow deduplicate them (so it doesn't show Taskotron
results which were reported against both the Koji build and the Bodhi
update twice). But that's probably a separate thread.
Thanks.
You too!
[1] https://www.youtube.com/watch?v=kaF6FxmixJk , am I really that
old?
They're still using the same ad concept these days, though :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
11:18 a.m.
> I'll go slightly off-topic here, but consider this:
'scenario' will
> help us identify unique results. But it will not help us decide
> whether "all" results have been submitted (maybe some are still
> running, maybe some have crashed hard and not sent any results). So
> there's a good chance that rel-eng tools will need to have deep
> knowledge of the testcases anyway. Not just list of all monitored
> testcases, but also that this particular test case needs to be
> performed for bios and uefi (or, that this particular testcase needs
> to be present with these two scenarios).
>
> So while having scenarios seems definitely helpful in certain cases,
> it might not help us in avoiding having too much knowledge in the
> gating consumers. Just food for thought.
I have actually been thinking about that problem as well :) And it is
indeed a tricky one. Bodhi already basically ducks the question; it
just shows whatever results are there at the time. We don't have a good
answer for cases like 'can we ship this compose?' or check-compose,
cases where the tool needs to know that testing is complete.
My best idea so far is that we should implement a 'testing complete'
fedmsg with a consistent name for each testing system - so
whatever.test.system.prefix.is.testing.complete or something like that.
I'm skeptical about this. The implementation is easier for OpenQA, because it's a
standalone system and you know that the testing is "complete" once all the test
cases passed. But for generic Taskotron tasks (depcheck, rpmgrill, etc), we schedule them
in the trigger based on fedmsg contents and then we don't track them anymore. We have
no idea if and when all of them completed. We could make a complex system to track that,
of course, but it seems to me that it's fundamentally wrong anyway. It moves the test
plan knowledge from the consumer to the producer. So even though rel-eng gating script is
the consumer that should decide whether we're good to go or not, we would move this
logic into OpenQA or Taskotron just to be able to trigger "testing complete"
message. That doesn't seem worth it. And it will break once we have multiple consumers
with different requirements (what happens when one of the consumers needs depcheck and the
other doesn't, when is the testing "complete"?).
I'd rather have a definition file in releng's pagure and send PRs against that,
than emulate the logic in all our testing systems.
So yes, this is a tricky problem indeed. If anyone has a magic solution, I'm
interested.
Then at least the consumer only has to know which test systems it
cares
about, and it can quite trivially trigger whenever it has
'testing.complete' messages from each one for the relevant item. I
really dunno if we can do anything better than this. I suppose if we
also required all systems to send a 'testing.started' message, we could
have some sort of meta-consumer check when it's seen a
'testing.complete' message for each 'testing.started' message and send
out a 'all testing for compose X is complete' message, but even if we
*do* that, it means that any single system can prevent the 'all testing
complete' message going out if it's slow or broken, even if it's one a
given consumer doesn't actually care about. So I don't think consumers
would use such a message even if we managed to build it...
Note that I've already implemented something like this for openQA,
though it doesn't work exactly the way I described at present. Each
openQA 'test complete' message contains a count of currently running or
scheduled tests for the same build, and things like check-compose
trigger when they see a 'job done' message with that count at 0.
So as suggested above, this will work well only until you start adding tests which are
interesting to run, but not necessary for the consumer. Or until we have 2 different
consumers.
> To be fair, I can think only of a few minor points:
> * Data duplication, makes the field longer and harder to read.
This seems pretty minor - honestly there's almost no case for a human
to be manually reading the field ever, except for debugging purposes. I
never actually go and manually inspect what the openQA scenario values
are, there's just no use case for that - it's just a concept that the
tools use to do a job.
I said all those points were minor :)
> * A possibility for errors. Since tasks are allowed to add arbitrary
> suffixes to their testcase names, a task can create tens of "sub-
> testcases" (e.g. look at dist.rpmgrill*). The scenario field is not
> automatically generated but fully under task's control. There might
> be copy&paste errors or logical errors in the code, and the scenario
> might not correctly reflect the used testcase name.
Hum, I guess, though it doesn't actually *matter* if the scenario still
does its job of being unique within the results for a single item, and
being attached to 'the same' test across multiple items. We're
definitely not going to want consumers to get into the business of
parsing things out of the scenario value, if they actually want one of
the values that makes up the scenario value they should request it
directly. But sure, it's a possibility.
I can see this easily going wrong like this:
1. You have dist.rpmlint that includes "dist.rpmlint" in scenario.
2. You decide to create dist.rpmguard check that is very similar, so you copy dist.rpmlint
code and modify it. You forget to adjust the scenario value, so it still contains
"dist.rpmlint" in it.
3. Now when you query for item+scenario combination, as suggested by you, you get invalid
results (you'll get both dist.rpmlint and dist.rpmguard results mixed up, even though
just requested just dist.rpmlint results. Time ordering will decide whether you get a
correct one or incorrect one as the latest result).
Another realization is that namespace would definitely need to be present in scenario key
(otherwise you get all rpmlint results, even from other people, even from scratch, etc).
That again makes it prone to errors when you switch namespaces (going from scratch to
dist, for example).
> * Consistency (even though that can be arguable). I'd like to be
> clear that testcase+item combination is always the default way to
> consume results. For more complex tests, it might also require
> looking at scenario to distinguish unique results. If I write
> instructions that say that results are identified by item+scenario if
> scenario exists, otherwise as testcase+item, it seems more complex
> and less obvious to me. (But this is maybe just about phrasing).
I...dunno if this is going to be a tenable place to stand, to be
honest.
Funnily enough I went through the same process in a somewhat
different context. When thinking about relval-ng in my head, I
initially had the idea that we could kill the Wikitcms 'environment'
columns - which, if you think about it, are basically this 'scenario'
concept -
Yep, I realized scenario is the same as environment, or more precisely a combination of
environments. I even wanted to propose to rename it, but scenario is probably a better
term :)
by associating all results with a specific image - which is,
if you think about it, the 'item' concept.
NOTE IF YOU'VE NO IDEA WHAT THIS IS ABOUT: we're talking about the wiki
pages Fedora QA uses to store release validation test results, like
this:
https://fedoraproject.org/wiki/Test_Results:Fedora_26_Branched_20170302.n...
Note each row is for one 'test case', and most rows have multiple
columns for results in different 'environments' (i.e. scenarios) for
that test case. 'relval-ng' is the working name of a system we're
proposing to build to replace the wiki system.
So I was effectively thinking the same as you: we can always just
identify a result as the combination of "a test case" and "a tested
item".
But then I thought, nah, it's still not really that simple. In relval-
ng as in openQA, the easiest example is BIOS vs. UEFI: we have some
tests, e.g. Anaconda_User_Interface_Basic_Video_Driver , where we want
to test on both BIOS and UEFI. This involves the same test case and the
same tested 'item' - an x86_64 installer image - but a different
'scenario'. Unless you start stuffing scenario items into the test case
name (basicvideo.uefi , basicvideo.bios ?) or item name (foobar.iso
BIOS, foobar.iso UEFI ?) - an idea we rejected back at the start of
this thread - I fundamentally don't see a way around this.
There are other examples, though - take the 'Default boot and install'
table, where we consider installing with the same image to a VM and to
bare metal as being different scenarios, and also installing with the
same image written to an optical disc and written to a USB stick.
There's just fundamentally no way around that without invoking the
'scenario' concept, or something very much like it but phrased
differently.
Agreed.
As we've also already noted, the existing package tests have also run
into this problem: "test name + item" is not sufficient to really
define the result, as 'item' is a source package but the same test is
run for all binary package arches and may have different results on
different arches (IIUC).
At present, we report rpmlint, rpmgrill, abicheck etc as a single result against source
NVR. We did not see a good reason why to separate the results per arch, especially when we
can do the check for all archs in a single pass. You can see the separated results in the
log, of course. The only difference is depcheck, which is executed in several runs, and
it's easier to report that separately.
Basically, my contention is that this 'scenario' concept is going to
just keep on turning up, all over the place, and it's going to be more
realistic to expect to be dealing with a 'scenario' most of the time,
than to expect to be dealing with just 'test case plus item' most of
the time and consider 'scenario' to be a kind of "advanced" thing.
We'll see how often it'll be used, but I agree it will not be rare. For basic
tests like rpmlint it will not be needed, but for anything heavier, it probably will.
So I think I still slightly prefer the idea of including the test name
in the scenario value, but I really don't have a strong preference
either way. Let's just pick one approach and go with it.
I dislike the fact you'll need to maintain the same values in several places
(checkname and namespace in both task formula and your code generating the resultsyaml). I
also don't see good benefits in having it there. So my preference is to avoid putting
namespace+checkname into scenario. I'm not adamant here and if I got anything wrong,
please try to convince me otherwise. Or maybe we can have someone else voice an opinion
here... Josef? :-)
Note if I'm being honest I have a practical reason for this, as openQA
defines the test case name as one of the 'scenario keys', so if we go
with 'test name not in scenario value', in the openQA reporter we'll
have to take the list of 'scenario keys' and then remove the test name
from it. Obviously this is a trivial point, but just to be honest, it's
the real reason why I initially assumed we'd put the test name in the
scenario value: just cos that's how openQA is currently set up to do
it, if you do it the easiest way :)
Ah, now we know! :-) Hmm, maybe we can take a different path here then. We don't
really need to standardize how scenario value looks. We just need to say:
"the purpose of scenario key is to identify unique runs of your test, when
testcase+item combination is not enough. You should include all extradata fields values
that make this test run unique (e.g. 'scenario=x86_64.uefi' for
'arch=x86_64' and 'firmware=uefi' extradata)."
So the recommendation is clear. If you include the test name in openqa results, it does
not negatively affect the uniqueness resolution. We just need to be clear in our
documentation that in order to recognize unique results, you always need to consider
testcase + item + scenario (if present).
Of course if someone takes inspiration from openqa results, he/she might also include
non-important strings (testcase name) in it, but if we're clear in our documentation,
that should not be a problem (i.e. hopefully the consumer authors will read it). Would
that work for you?
> Since you're clearly powered by Duracell batteries [1],
please go
> ahead and implement this. If we decide to implement this later in
> resultsdb directly, we can always go and simplify the consumer code.
Right. And I've merged the version where the test name is included in
the scenario value, but again, we can change this later if we want to.
So long as we write the consumer code to query for 'scenario plus test
name', even if it happens to hit an older result where the test name
was included in the scenario, no harm is done, the correct result will
be achieved.
So yeah, openQA results now include a 'scenario' value.
Great!
For openQA the
scenario value is constructed by joining the values for all keys openQA
considers 'scenario keys' with periods, but we don't actually have to
make this consistent between different systems - the only requirement
is that the value correctly identify the scenario.
Yep.
I can try and send a
patch for Taskotron to do this as well, if you like, or would you
rather do it?
I'll do it this week. Unless you want it in 24 hours, in that case feel free to submit
patches :)
> Btw, if you end up submitting patches to Bodhi and using
> 'type=bodhi_update' queries, please also use the 'since=' argument
> [2] and set it to the update's 'date_modified' timestamp. That
> doesn't really implement the higher reliability of such results (as
> we talked elsewhere), but it doesn't stress resultsdb so much, which
> is always good (we can't do the same for type=koji_build easily, but
> we can for type=bodhi_update).
I'm planning to do this, but I did want to talk over with you guys what
would be appropriate for the Taskotron results. Bodhi is definitely
going to have to query for 'type=bodhi_update' results where the item
is the update ID in order to find the openQA results. But I don't know
if we should just make it find the Taskotron results in the same way,
or if it's best to have it query for both koji_build and bodhi_update
results and somehow deduplicate them (so it doesn't show Taskotron
results which were reported against both the Koji build and the Bodhi
update twice). But that's probably a separate thread.
Yep, let's talk about that separately.
11:27 a.m.
On Tue, 2017-03-07 at 12:18 -0500, Kamil Paral wrote:
> Note if I'm being honest I have a practical reason for this,
as openQA
> defines the test case name as one of the 'scenario keys', so if we go
> with 'test name not in scenario value', in the openQA reporter we'll
> have to take the list of 'scenario keys' and then remove the test name
> from it. Obviously this is a trivial point, but just to be honest, it's
> the real reason why I initially assumed we'd put the test name in the
> scenario value: just cos that's how openQA is currently set up to do
> it, if you do it the easiest way :)
Ah, now we know! :-) Hmm, maybe we can take a different path here
then. We don't really need to standardize how scenario value looks.
We just need to say:
"the purpose of scenario key is to identify unique runs of your
test, when testcase+item combination is not enough. You should
include all extradata fields values that make this test run unique
(e.g. 'scenario=x86_64.uefi' for 'arch=x86_64' and
'firmware=uefi'
extradata)."
So the recommendation is clear. If you include the test name in
openqa results, it does not negatively affect the uniqueness
resolution. We just need to be clear in our documentation that in
order to recognize unique results, you always need to consider
testcase + item + scenario (if present).
Of course if someone takes inspiration from openqa results, he/she
might also include non-important strings (testcase name) in it, but
if we're clear in our documentation, that should not be a problem
(i.e. hopefully the consumer authors will read it). Would that work
for you?
I was kinda thinking the same thing, but at that point we're very close
to "it doesn't include the test case name". I really don't mind a lot
going with that, if you still think it's better. I can take the test
case name out of the openQA scenarios going forward easily enough.
> I can try and send a
> patch for Taskotron to do this as well, if you like, or would you
> rather do it?
I'll do it this week. Unless you want it in 24 hours, in that case
feel free to submit patches :)
This week is fine I think!
> > Btw, if you end up submitting patches to Bodhi and using
> > 'type=bodhi_update' queries, please also use the 'since='
argument
> > [2] and set it to the update's 'date_modified' timestamp. That
> > doesn't really implement the higher reliability of such results (as
> > we talked elsewhere), but it doesn't stress resultsdb so much, which
> > is always good (we can't do the same for type=koji_build easily, but
> > we can for type=bodhi_update).
>
> I'm planning to do this, but I did want to talk over with you guys what
> would be appropriate for the Taskotron results. Bodhi is definitely
> going to have to query for 'type=bodhi_update' results where the item
> is the update ID in order to find the openQA results. But I don't know
> if we should just make it find the Taskotron results in the same way,
> or if it's best to have it query for both koji_build and bodhi_update
> results and somehow deduplicate them (so it doesn't show Taskotron
> results which were reported against both the Koji build and the Bodhi
> update twice). But that's probably a separate thread.
Yep, let's talk about that separately.
Great, I'll ping you about it. There's also the question of whether we
'retrofit' scenarios into existing results, though...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
8:58 p.m.
On Tue, Mar 07, 2017 at 12:18:17PM -0500, Kamil Paral wrote:
> > I'll go slightly off-topic here, but consider this:
'scenario' will
> > help us identify unique results. But it will not help us decide
> > whether "all" results have been submitted (maybe some are still
> > running, maybe some have crashed hard and not sent any results). So
> > there's a good chance that rel-eng tools will need to have deep
> > knowledge of the testcases anyway. Not just list of all monitored
> > testcases, but also that this particular test case needs to be
> > performed for bios and uefi (or, that this particular testcase needs
> > to be present with these two scenarios).
> >
> > So while having scenarios seems definitely helpful in certain cases,
> > it might not help us in avoiding having too much knowledge in the
> > gating consumers. Just food for thought.
>
> I have actually been thinking about that problem as well :) And it is
> indeed a tricky one. Bodhi already basically ducks the question; it
> just shows whatever results are there at the time. We don't have a good
> answer for cases like 'can we ship this compose?' or check-compose,
> cases where the tool needs to know that testing is complete.
>
> My best idea so far is that we should implement a 'testing complete'
> fedmsg with a consistent name for each testing system - so
> whatever.test.system.prefix.is.testing.complete or something like that.
I'm skeptical about this. The implementation is easier for OpenQA,
because it's a standalone system and you know that the testing is
"complete" once all the test cases passed. But for generic Taskotron
tasks (depcheck, rpmgrill, etc), we schedule them in the trigger
based on fedmsg contents and then we don't track them anymore. We
have no idea if and when all of them completed. We could make a
complex system to track that, of course, but it seems to me that
it's fundamentally wrong anyway. It moves the test plan knowledge
from the consumer to the producer. So even though rel-eng gating
script is the consumer that should decide whether we're good to go
or not, we would move this logic into OpenQA or Taskotron just to be
able to trigger "testing complete" message. That doesn't seem worth
it. And it will break once we have multiple consumers with different
requirements (what happens when one of the consumers needs depcheck
and the other doesn't, when is the testing "complete"?).
Yeah, me too. Putting the complexity of "completion" in the test
execution systems seems tough. In other environments, Jenkins jobs
won't know about total completion per item, either.
I'd rather have a definition file in releng's pagure and send
PRs
against that, than emulate the logic in all our testing systems.
Yeah - that's a stopgap that could work.
So yes, this is a tricky problem indeed. If anyone has a magic
solution, I'm interested.
Long-term (in the new few months) I want to start work on the
PolicyEngine service I presented at devconf: a service to store
descriptions of what test results are "good enough" at various gate
points, for different content sets. Something kept in a DB, managed
through a web UI - a place for packagers with ACLs to manage specific
requirements for their package sets, but with the ability for fedora
qe and fedora releng to overlay that with their own global policies.
2528
days inactive
2622
days old
resultsdb-users@lists.fedoraproject.org
49 comments
7 participants
participants (7)
-
Adam Williamson -
Dan Callaghan -
Jan Sedlak -
Josef Skladanka -
Kamil Paral -
Pierre-Yves Chibon -
Ralph Bean