On Wed, 2017-05-03 at 10:09 +1000, Dan Callaghan wrote:
Excerpts from Adam Williamson's message of 2017-05-02 16:15
-07:00:
> So, first the good news: fundamentally, the concept works. I can tell
> Bodhi to query 'all bodhi_update results for the item FEDORA-2017-
> 4e95959f0d' and then display (only) the latest result for each
> 'scenario' that appears in the whole set of results.
I think this might be the fundamental problem here. I think the
ResultsDB "item" should not be an update, but the set of packages in an
update. If someone edits the update to change the set of packages in it,
the previous results are invalidated and new tests are kicked off.
Putting aside the issue of how to actually make the "item" be a set of
packages -- I think this would eliminate all of these issues with OpenQA
scenarios.
Uh, I think you're misunderstanding something, because it just wouldn't
do so at all. Even if openQA tested by package rather than by update,
it would still run the same four tests twice for each package, once
starting from a Server base image, once starting from a Workstation
base image. That change would make no difference whatsoever to this
problem.
OpenQA would simply report results with test cases like this, with
all
your scenarios and flavours and so forth as different permutations in
the test case name:
graphical_installation.fedora.26.server.x86_64.blahblah
graphical_installation.fedora.26.workstation...
This is an entirely different suggestion that has nothing to do with
your other suggestion: you're effectively suggesting that we simply
encode the scenario into the test case name. This was actually one of
the alternatives I floated in my very first mail on this topic, but we
didn't really like it for several reasons.
But unlike your other suggestion this one *would* actually work to help
Bodhi's use case; it would let Bodhi just go ahead and treat the test
case name as the 'scenario' with no need for special casing at all
(assuming we also changed the test case name for Taskotron depcheck
tests to something like 'dist.depcheck.i686' and
'dist.depcheck.x86_64').
and the existing logic for latest-result-by-testcase will work fine
with
no special cases needed in Bodhi. Bodhi would just query for the
ResultsDB "item" consisting of all packages currently in the update.
There is no necessary relationship at all between your two suggestions.
This also eliminates the problem where an update is edited, and Bodhi
queries for item FEDORA-2017-4e95959f0d but it sees older test results
based on the packages that were previously in the update because the new
test runs are not complete yet.
There is no such problem, because we tell Bodhi only to look for
results since the update was last edited.
If you go down that route then it means the *real* problem to solve
is
how to represent an "item" which is a set of packages rather than
a single package.
Sorry, but no, I don't think it does. I think you're misunderstanding
what the 'flavor' is, possibly? It's not about running the same test on
multiple packages within an update, it's about running the same test on
the same set of packages starting from a different base disk image
(which is what the 'flavor' represents, in this particular openQA
configuration).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net