On Wed, 9 Aug 2017 14:31:49 +0200
Lukas Brabec <lbrabec(a)redhat.com> wrote:
Hey, gang!
As I read through standard interface and tried ansiblized branch of
libtaskotron, I found things that were not exactly clear to me and I
have some questions. My summer afternoon schedule involves feeding
rabbits (true story!) and I keep missing people on IRC, hence this
email.
= Test output and its format =
Standard test interface specifies that [1]:
1) "test system must examine the exit code of the playbook. A zero
exit code is successful test result, non-zero is failure"
and
2) "test suite must treat the file test.log in the artifacts folder as
the main readable output of the test"
ad 1) Examining the exit code is pretty straight forward. The mapping
to outcome would be zero to PASSED and non-zero to FAILED. Currently
we use more than these two outcomes, i.e. INFO and NEEDS_INSPECTION.
Are we still going to use them, if so, what would be the cases? The
playbook can fail by itself (e.g. fail like command not found, or
permission denied), but I presume this failure would be reported to
ExecDB not to ResultsDB. Any thoughts on this?
I think that, for now at least, we won't be using the INFO and
NEEDS_INSPECTION. The standard interface definition is what it is and
they've decided that results are binary. It's possible that could
change in the future but for now, the INFO and NEEDS_INSPECTION states
probably won't be used when reporting to resultsdb.
ad 2) The standard interface does not specify the format of test
output, just that the test.log must be readable. Does this mean that
the output can be in any arbitrary format and the parsing of it would
be left to people who care, i.e. packagers? Wouldn't be this a problem
with if, for example, bodhi wanted to extract/parse this information
from ResultsDB and show it on update page?
As far as I know, this was left vague on purpose so that it can change
and be specified by the the test system. In this case, we'd definitely
need to support the xunit xml format that I suspect most folks will be
using but I'm open to the idea of keeping our results yaml format alive
if it makes sense.
So long as we're explicit about which formats are supported for our
implementation of a test system and don't make silly choices, I think
we're good here.
= Triggering generic tasks =
Standard interface is centered around dist-git style tasks and doesn't
cover generic tasks like rpmlint or rpmdeplint. As these tasks are
Fedora QA specific, are we going to create custom extension to
standard interface, used only by our team, to be able to run generic
tasks?
It'd be nice if it wasn't just us using it but the standard interface
may indeed require some tweaking to get it to cover all of the usecases
that we're interested in.
Do you have any suggestions as to how we could make non-dist-git tasks
work reasonably well without making drastic changes to what the
standard interface currently is?
= Reporting to ResultsDB =
Gating requirements for CI and CD contains [2]:
"It must be possible to represent CI test results in resultsdb."
However standard interface does not speak about resultsdb.
The standard interface arguably shouldn't care about resultsdb. It does
say "it must be possible to represent ..." and not "must report to
resultsdb".
Does this mean, that task playbook won't contain something like
ResultsDB module (in contrast to ResultsDB directive in formulae), as
the task playbook should be agnostic to system in which it is run, and
the reporting will be done by our code in runtask?
That's how I'm understanding things, yes. There may be other systems
that we'll need to interface with but let's cross that bridge if and
when we get there.
= Output of runtask =
Libtaskotron's output is nice and readable, but output of the parts,
handled by ansible now, is not. My knowledge of ansible is still
limited, but as far as my experience goes, debuging ansible playbooks
or even asnible modules is kind of PITA. Are we going to address this
in some way, or just bite the bullet and move along?
Do you have any ideas on how to improve that? One thing that I had in
mind was to look at using ara to make the output a bit easier to digest.
https://github.com/openstack/ara
= Params of runtask =
When I tried ansiblized branch of libtaskotron, I ran into issues such
as unsupported params: ansible told me to run it with "-vvv" param,
which runtask does not understand. Is there a plan on how are we going
to forward such parameters (--ansible-opts= or just forward any params
we don't understand)?
Personally, I think that --ansible-opts= makes sense here but I don't
have terribly strong feelings about it.
Runtask, at the moment, maps our params to ansible-playbook params
and
those defined by standard interface. Are we going to stick with this
or change our params to match the ones of ansible-playbook and
standard interface (e.g. item would become subject, etc)?
I think that it'll likely make sense to adjust some of the params but
to be honest, I haven't looked too closely at the differences in
params. There are at least parts of the standard interface which aren't
going anywhere soon so it would seem wise to start aligning with what
they're doing where it makes sense.
= Future of runtask =
For now, runtask is user-facing part of Taskotron. However, standard
interface is designed in such way, that authors of task playbooks
shouldn't care about Taskotron (or any other system that will run
their code). They can develop the tasks by simply using
ansible-playbook. Does this mean that runtask will become convenient
script for us that parses arguments and spins up a VM? Because
everything else is in wrapping ansbile playbook...
Yeah, they could use ansible-playbook but in my mind, there are more
user friendly ways to run a test than formatting args for
ansible-playbook, downloading the target by myself.
I think that the bigest way we can add value with runtask is making
things easier - spawn declared vms (either in openstack, locally etc.
depending on config), generating an inventory file so that the test
author just has to worry about using what they requested, putting
output into a more easily human understandable format, improving the
local development experience etc.
The role of runtask is certainly going to change as we switch from a
custom yaml-based task language to using Ansible and its runner to
handle most of the heavy lifting. That being said, I do see places
where we can add value and improve the experience of our users.
Tim
Lukas
[1]
https://fedoraproject.org/wiki/Changes/InvokingTests
[2]
https://fedoraproject.org/wiki/Fedora_requirements_for_CI_and_CD
_______________________________________________
qa-devel mailing list -- qa-devel(a)lists.fedoraproject.org
To unsubscribe send an email to qa-devel-leave(a)lists.fedoraproject.org