9:13 a.m.
Hi Dominik,
Thank you for your questions. See my comments inline.
On Thu, 23 Jul 2020 at 14:39, Dominik 'Rathann' Mierzejewski
<dominik(a)greysector.net> wrote:
On Wednesday, 22 July 2020 at 14:49, Petr Šplíchal wrote:
> Hi!
>
> I am writing today to share a proposal for how we can make it
> easier to enable tests in Fedora CI.
>
> We've heard the user story mentioned in the subject many times.
> Fedora developers have frequently mentioned the difficulties which
> they face with enabling tests in Fedora CI. Some people don't
> understand the complex yaml syntax. Other people find that they
> can't easily reproduce issues revealed by Fedora CI on their
> laptops. People have asked us to provide a tool for local testing:
>
> https://pagure.io/fedora-ci/general/issue/4
>
> A few of us have been working on a tool which we hope will improve
> the situation considerably. The proof of concept below shows how
> everyday developer tasks (related to testing) might look in the
> near future:
>
> Enable basic smoke test for my component:
>
> dnf install -y tmt
> tmt init --template mini
> vim plans/example.fmf
> git add . && git commit -m tests && git push
>
> Create a new test based on a shell/beakerlib template:
>
> tmt test create --template shell tests/basic
> tmt test create --template beakerlib tests/advanced
>
> Run all/selected tests safely in a virtual machine:
>
> tmt run
> tmt run test --name smoke
I'd say it's good to have such option, but for quick testing a full VM
is often too heavy and slow to boot. Even a container takes time to set
up and run.
Yes, there are different needs and preferences related to the
environment where the tests should be executed. We want to provide
enough flexibility here so the 'provision' step allows you to
choose your preferred way.
My primary environment would be local.
So for your case that would be 'provision --how local'. If you
have a test box prepared for testing you could also connect to it
using 'provision --how connect'.
> Execute tests in my preferred environment:
>
> tmt run --all provision --how container
> tmt run --all provision --how virtual
> tmt run --all provision --how local
> tmt run --all provision --how ...
I find it difficult to guess what the above commands actually do.
Do they run the tests or do they provision environments to run the
tests? Or both?
The 'tmt run' command is used to run tests. By default it runs all
steps (discover, provision, prepare, execute, report, finish) so
the examples above would run tests in the requested environment.
It is also possible to run only selected steps: For example, one
of the use cases shared with us was 'just tell me what tests would
be run, do not run them', this can be achieved with the following:
tmt run discover
For detailed description of individual steps have a look here:
https://tmt.readthedocs.io/en/latest/spec/steps.html
> I would like to ask for your feedback about this. What do you
> think? Is this something that would be useful for you? What do
> you think of the syntax? We would love to hear back from you
> concerning what essential features you might expect from such a
> tool. It would also be helpful to know if there is anything that
> should be changed early in the tool design.
>
> For more examples and frequently asked questions have a look here:
>
> https://tmt.readthedocs.io/en/latest/examples.html
> https://docs.fedoraproject.org/en-US/ci/tmt/
>
> Looking forward to your feedback and ideas! :)
tmt examples are overcomplicated and not explained in enough detail.
What is "metadata" and why do you need 4 "levels" of it?
Metadata levels allow to store all needed test execution data at
appropriate location, the levels are described here:
https://tmt.readthedocs.io/en/latest/spec.html
We want the config for simple/common use cases (such as running a
shell script as a test) as simple as possible so that users do not
have to study all the details but just create something like this:
execute:
script: ./run-my-test.sh
If one needs to maintain larger set of tests the L1 metadata can
come handy:
https://tmt.readthedocs.io/en/latest/spec/tests.html
What are "tiers" for?
They are used to organize tests into groups by priority.
https://tmt.readthedocs.io/en/latest/spec/tests.html#tier
Which items are mandatory and which are optional?
In order to enable tests you just have to define the 'execute'
step. Use 'tmt init --template mini' to get a quick start.
What's "relevancy"?
It's a test (L1) attribute allowing to filter relevant tests.
https://tmt.readthedocs.io/en/latest/spec/tests.html#relevancy
Currently we're working on open-sourcing the implementation.
Why is there "extra-summary" in addition to
"summary"?
What's "extra-task"?
These are because of backward compatibility, for more see:
https://tmt.readthedocs.io/en/latest/stories/cli.html?highlight=extra-tas...
What are "plans" for?
Plans define which tests should be executed, how the environment
should be provisioned and prepared, how the results should be
reported. See the following page for more details about plans.
https://tmt.readthedocs.io/en/latest/spec/plans.html
What are "stories" for?
Stories are an advanced, optional feature which can help you to track
which features/requirements have been already implemented, tested
and documented.
https://tmt.readthedocs.io/en/latest/spec/stories.html
https://github.com/psss/tmt/tree/master/stories
You're using a lot of terms as if everyone knows what they mean
in your context.
Sorry for that. We're trying to document the concept as best as we
can. Hopefully, the links above have helped. Perhaps we could add
some more references to the examples page.
Thank you for all your feedback. Please, let us know what's still
missing and how we can further improve the docs. Thanks.
psss...
Regards,
Dominik
--
Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
-- from "Collected Sayings of Muad'Dib" by the Princess Irulan
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org