Please take this as stream of conscious feedback from a Fedora
contributor who is a casual maintainer of a few packages. I'm really
excited to see this working and not meaning to be critical; I hope this
is useful.
Okay, so, I submitted an update, and — cool! — there are Automated Test
Results sitting there. And some of them have a subtle little ⓧ , which
I know means "failed" rather than "click to hide this"
Agreed, the icons could be more obvious and visible. That would be an RFE against Bodhi.
because those
are highlighted in pink. Another one has a kind of mellon orange and a
circled !, so I guess it need my attention.
So, click on the first failure. It's from rpmlint, and I know how to
read that output. Okay, this is easy — plain text with FAILED at the
bottom, with "(1 errors, 7 warnings)". I read upwards and see the
errors and warnings. Some of them are a little obscure, but I'm used to
rpmlint.
Ok, next one — the orange warning from dist.rpmgrill. Woah — a whole
bunch of JSON. I search for "warning" or "notice" and get nothing;
there are bunch of "status" : "completed" lines. It's not at all
clear
what I'm supposed to pay attention to. Hmmm.
Yeah, rpmgrill output is hard to read (even I grind my teeth from time to time). Even
though Fedora QA technically maintains the script wrapper (contributed by Ralph Bean),
there's little to do there - rpmgrill doesn't seem to produce any output/artifact
that would be more readable. This would need an RFE against rpmgrill.
We as QA want to ideally execute as many tasks as our machines are able to, as long as
they are at least mildly beneficial. The actual quality of those tasks and their outputs
is going to differ wildly. Currently we directly maintain rpmlint, depcheck, upgradepath
and dockerautotest. Rpmgrill and abicheck are maintained by external parties. We hope more
of the latter will appear as taskotron gets more capable. We can try to devote some time
to polishing external checks, but it's not going to scale. My current idea is that we
will use task namespaces (currently we use mainly "dist.") to distinguish
between "important" and "less important" tasks. We should strive to
provide a good user experience for the important set, and will probably let the less
important set live the life on its own. Bodhi can display just the important set, of both
(with preference for the important one).
I notice that the next
result — the other failed one — is dist.rpmgrill.desktop-lint, so maybe
the warning is just telling me that a sub-test failed.
Yes, it's just a sub-check result (and the log link goes to the full overall log, in
case of rpmgrill).
A few days ago I suggested how to show check results (collapsing sub-checks by default)
here:
https://github.com/fedora-infra/bodhi/issues/951#issuecomment-248843851
There _is_ some
DesktopLint stuff in here, but I'm not quite clear on what I'm seeing
or what it means. There's stuff like "diag" : "Icon file
<var>ufraw</var> not found"... maybe that's it?
Anyway, on to the failed apparent subtest. Oh look, more JSON. Hmmm. Is
this a subset of the previous? Just the relevant section? Looks like
it. Still don't know exactly what to do.... reading the JSON.... ah,
okay, it's not happy about the .desktop files that geeqie installs for
its own menus. That's a false positive; they're actually Fine For What
They Do.
So, the one with a warning could definitely be more clear. For the
other JSON one, it really would be nice to have some explanation of
what's being tested, and have the JSON converted to something human
readable. I hope that's not just me being whiny — it feels barely
useable like this (especially as more new tests are addded).
Yes, so far the effort was more on having them running (at all) than polish. We hope that
some of that polish will come from the task maintainers.
I would definitely not want to block-by-default on a task that is not well readable by a
package maintainer (currently everything is just informational, the worst that can happen
is karma autopush getting disabled - and you're informed about that).
Beyond that... if I'm sure something is a false positive, is there a
way to mark it as suppressed? (I guess not hidden forever; maybe shown
at the bottom of the list as failed-but-okay'd or something.) I imagine
there are gonna be a lot of false positives.
Not at the moment. Some tasks have specific override options (like with upstream rpmlint
you can define you own config file with certain errors whitelisted - but we haven't
implemented support for this in rpmlint task yet). We will need something universal,
mainly for blocking-by-default checks (it could be as simple as allowing the maintainer to
click "force push to stable" in bodhi and provide an explanation why he/she is
ignoring the failed gating checks). However, none of this is implemented yet. Ignoring
arbitrary (recurring) warnings from arbitrary tasks... might be a challenge.
Bonus question! Is there a UI for me to see these for packages which
don't go to bodhi? I usually just build into rawhide unless there is a
security update or major bugfix.
Yes, resultsdb. From
https://taskotron.fedoraproject.org/ your click on "Browse Task
Results" and go to
https://taskotron.fedoraproject.org/resultsdb/jobs . You can then
use the search button to search for a specific NVR (please note that the other fields in
the search button are somewhat broken, sorry), or write a custom query (double sorry),
e.g.
https://taskotron.fedoraproject.org/resultsdb/results?item=sudo-1.8.18-1.... .
You should also receive notifications about failed depcheck and upgradepath from FMN by
default, and you can define your own rules to receive notifications about basically
anything. See Martin's blogpost:
https://mkrizek.wordpress.com/2016/02/24/taskotron-results-notifications/
I'd love to see a task results browser (or at least a link) in Koji UI, similar to
what's currently in Bodhi (maybe it could even share some code?).
Overall, thanks for your observations, and we should definitely do something about it. I
tried to explain why it is the way it is (so much work and so little time, as usual) -
currently it's more of a display of all that is currently being executed inside
Taskotron rather than a well-thought package maintainer experience. Any RFE reported
(against our tools or external tasks) is definitely very appreciated.
Kamil