Hi,
I've already tried to ask on #fedora-qa, but I think the mailing list is
the better medium to discuss this. Thanks to jskladen for answering.
We've been looking into Taskotron and Resultsdb for a while and would
like to deploy an instance running RPMGrill[1] and become contributors.
I assume, running rpmgrill in Taskotron could mean, that each rpmgrill
test result translates to an entry in Resultsdb as an outcome (PASSED,
FAILED).
If you are interested, we support more outcome keywords, like INFO or NEEDS_INSPECTION:
https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/library....
Yet the question for the developer as to why it failed is
important.
I have mainly three main questions around Taskatron and specifically
Resultsdb:
1. Ed Santiago is currently running RPMGrill here:
http://rpmgrill-fc20.edsantiago.com:5000/recent
which in terms of presenting results to the user provides a different
experience. The results are grouped by test and provide more insight
as to why a test failed. Test results in Resultsdb currently seem to
have only one outcome and technical details are left on the build
master.
How does a representation like RPMGrill translate into Resultsdb? Are
there plans/ideas on how to provide a more diversified result
representation in Resultsdb (e.g. error reason, hint on how to solve
an error).
At the moment, each result contains a link to the task log, which you can inspect, e.g.:
https://taskotron.fedoraproject.org/resultsdb/results?job_id=45746
points to
https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/36...
That log contains both taskotron execution info and above statements and task output. If
you know an undocumented trick, you can also strip down the path a bit arrive to the
buildbot job info page:
https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/36...
and there you have access to taskotron.log, which contains full debug info, which we use
for debugging libtaskotron issues. This is usually not needed by package maintainers, so
we don't advertise this too much yet.
I agree it would be great if taskotron could execute rpmgrill, and then upload the results
to your special application and nicely display them. Or alternatively generate the html
report locally. Unfortunately we don't have a support for storing and displaying
artifacts (e.g. a html page) at the moment -- but we plan to do it in the future.
If you want to have something working right now, I see several ways to go:
1. You could print out a text representation of the rpmgrill results into the log. You
could also post the results into your web applications, and then include a hyperlink in
the log in order to point people to a better visualization.
2. I guess we could change the "Log →" hyperlink in resultsdb to point to your
web application directly, rather than into our buildbot log. However, I see two concerns
with that, one is that accessing the task log itself would be non-trivial, and second that
displaying any results at all would be relying on your web app availability. If it went
down, nobody could see any results at all.
In the future, generating the html result locally and then showing it from resultsdb
interface is probably the way to go.
2. Are there plans/ideas about implementing a (fulltext-) search over
results?
I'll let others to respond to this, I have no idea.
3. Are there plans/ideas on implementing a waiver mechanism?
We will definitely need this, for most of the checks. But we haven't started yet
designing that feature, so it won't be available any time soon.
BTW, I talked to Miroslav Vadkerti on DevConf about rpmgrill. We should meet up again in
the following weeks and try to cook some actionable plan for rpmgrill in Taskotron.
Kamil