On Thu, 2019-11-21 at 08:40 +0100, Michal Srb wrote:
It looks like resultsdb is acting as some kind of a document-oriented
database where people can store whatever they want. Any reason (except
historical ones of course:)) why not to enforce some kind of schema on
input? I.e. resultsdb would own the schema and others would need to comply,
if they want their results to be used by other services down the road.
Josef Skladanka or Tim Flink could answer that better than me, but AIUI
the answer is basically: history. The current ResultsDB is more or less
ResultsDB 2.0; ResultsDB 1.0 was much more opinionated about how things
interacted with it, and that turned out not to work very well. So 2.0
was intentionally made much simpler to the point where it's essentially
just a database for key pairs with almost no strict rules about what a
'result' consists of.
The advantage of defining and enforcing the schema on resultsdb side
would
be much better clarity (in my opinion) and clear ownership. Do you want to
add a new CI system to the mix? Cool, just store results in resultsdb, and
here's the API/schema. End of the story. No need to fiddle with some
external schemas, trying to understand why/how, or hoping that some
listener will eventually store the results there for me, ...
Well, the listeners aren't just sort of random things that show up
looking for results to submit :) Most of the message consumers that
look for completed tests and forward the results to resultsdb are
*maintained by the same teams that maintain those test systems* (this
is the case for the openQA one - where it's me - and the CI Pipeline
one at least). It just turns out to be a sensible way to do it.
Especially if your test system is not written in Python, which neither
openQA (perl) nor the pipeline (Java, mostly) are; using a message
consumer approach for resultsdb submission means you can write the
resultsdb submission bits in Python and use the Python libraries. The
exception is autocloudreporter, which is simply because I noticed that
autocloud wasn't set up to report results to resultsdb and decided to
just fix it myself instead of waiting for someone else to do it. (I
actually came up with resultsdb_conventions as a logical extension of
sharing code with the openQA reporter when writing
autocloudreporter...)
> Ideally it'd probably be best if this new system could
publish
> resultsdb results in a format that's similar to *either* Taskotron's
> *or* the CI Pipeline's (or even that's a superset of both), and publish
> to fedora-messaging following the 'CI Messages' spec:
>
>
https://pagure.io/fedora-ci/messages
I completely agree that sending standardized CI messages would be super
nice.
Although if Fedora CI systems talk to resultsdb directly, what are the
benefits/incentives for migrating them to CI Messages standard? Or in other
words, are there services in the infrastructure that actually listen on
those raw CI messages? (and cannot just listen on resultsdb notifications).
Well, the main thing is that the CI Messages spec defines messages for
a lot *more* than just "this test completed and generated a result"
(which is all you get from a resultsdb item or the message generated
when one is created). It has messages for "test has been scheduled",
"test is running", "test errored out" and various other things.
The other nice thing about the CI Messages spec is that it's built from
the ground up to make the messages from different systems inter-
compatible; one of the initial ideas of the whole spec was that the
message topics *don't depend on the test system*, so you can find
messages for tests of a given 'thing' regardless of which test system
they come from. The overall idea is that you can build e.g. a web
dashboard which can show you the status of *all* tests for a given,
say, compose or update or pull request, regardless of what test system
ran them. And this is in fact what it's used for inside RH; there's a
thing called the 'CI Dashboard' which does exactly this. It'd be nice
to have a Fedora deployment of the dashboard too, but it will only work
if test systems actually publish messages in the correct format.
In terms of what things we actually have in Fedora doing useful stuff
*right now* do, it's a mishmash, because we haven't got all the systems
actually being inter-compatible yet. For e.g., Greenwave polls results
from resultsdb directly, and has logic to cope with the different
formats produced by the pipeline, Taskotron and openQA (but we could
sure make that nicer if we had stronger conventions about resultsdb
formats). I have a few things which listen for bus messages from openQA
and autocloud; for now these use the 'native' format messages, but if
we actually got everything compliant with CI Messages I'd probably
change them over. There may well be other stuff out there, I don't know
if anyone actually knows for sure "these are all the things that do
stuff with resultsdb results or automatic test system bus messages".
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net