As a developer I want to easily execute/enable tests

[Petr Šplíchal]

Hi!

I am writing today to share a proposal for how we can make it
easier to enable tests in Fedora CI.

We've heard the user story mentioned in the subject many times.
Fedora developers have frequently mentioned the difficulties which
they face with enabling tests in Fedora CI. Some people don't
understand the complex yaml syntax. Other people find that they
can't easily reproduce issues revealed by Fedora CI on their
laptops. People have asked us to provide a tool for local testing:

https://pagure.io/fedora-ci/general/issue/4

A few of us have been working on a tool which we hope will improve
the situation considerably. The proof of concept below shows how
everyday developer tasks (related to testing) might look in the
near future:

Enable basic smoke test for my component:

dnf install -y tmt
    tmt init --template mini
    vim plans/example.fmf
    git add . && git commit -m tests && git push

Create a new test based on a shell/beakerlib template:

tmt test create --template shell tests/basic
    tmt test create --template beakerlib tests/advanced

Run all/selected tests safely in a virtual machine:

tmt run
    tmt run test --name smoke

Execute tests in my preferred environment:

tmt run --all provision --how container
    tmt run --all provision --how virtual
    tmt run --all provision --how local
    tmt run --all provision --how ...

I would like to ask for your feedback about this. What do you
think? Is this something that would be useful for you? What do
you think of the syntax? We would love to hear back from you
concerning what essential features you might expect from such a
tool. It would also be helpful to know if there is anything that
should be changed early in the tool design.

For more examples and frequently asked questions have a look here:

https://tmt.readthedocs.io/en/latest/examples.html
    https://docs.fedoraproject.org/en-US/ci/tmt/

Looking forward to your feedback and ideas! :)

psss...

[Dominik 'Rathann' Mierzejewski]

On Wednesday, 22 July 2020 at 14:49, Petr Šplíchal wrote:
> Hi!
>
> I am writing today to share a proposal for how we can make it
> easier to enable tests in Fedora CI.
>
> We've heard the user story mentioned in the subject many times.
> Fedora developers have frequently mentioned the difficulties which
> they face with enabling tests in Fedora CI. Some people don't
> understand the complex yaml syntax. Other people find that they
> can't easily reproduce issues revealed by Fedora CI on their
> laptops. People have asked us to provide a tool for local testing:
> https://pagure.io/fedora-ci/general/issue/4
>
> A few of us have been working on a tool which we hope will improve
> the situation considerably. The proof of concept below shows how
> everyday developer tasks (related to testing) might look in the
> near future:
>
> Enable basic smoke test for my component:
> dnf install -y tmt
> tmt init --template mini
> vim plans/example.fmf
> git add . && git commit -m tests && git push
>
> Create a new test based on a shell/beakerlib template:
> tmt test create --template shell tests/basic
> tmt test create --template beakerlib tests/advanced
>
> Run all/selected tests safely in a virtual machine:
> tmt run
> tmt run test --name smoke

I'd say it's good to have such option, but for quick testing a full VM
is often too heavy and slow to boot. Even a container takes time to set
up and run.

My primary environment would be local.
> Execute tests in my preferred environment:
> tmt run --all provision --how container
> tmt run --all provision --how virtual
> tmt run --all provision --how local
> tmt run --all provision --how ...

I find it difficult to guess what the above commands actually do.
Do they run the tests or do they provision environments to run the
tests? Or both?
> I would like to ask for your feedback about this. What do you
> think? Is this something that would be useful for you? What do
> you think of the syntax? We would love to hear back from you
> concerning what essential features you might expect from such a
> tool. It would also be helpful to know if there is anything that
> should be changed early in the tool design.
>
> For more examples and frequently asked questions have a look here:
> https://tmt.readthedocs.io/en/latest/examples.html
> https://docs.fedoraproject.org/en-US/ci/tmt/
>
> Looking forward to your feedback and ideas! :)

tmt examples are overcomplicated and not explained in enough detail.
What is "metadata" and why do you need 4 "levels" of it?
What are "tiers" for?
Which items are mandatory and which are optional?
What's "relevancy"?
Why is there "extra-summary" in addition to "summary"?
What's "extra-task"?
What are "plans" for?
What are "stories" for?

You're using a lot of terms as if everyone knows what they mean
in your context.

Regards,
Dominik

[Petr Šplíchal]

Hi Dominik,

Thank you for your questions. See my comments inline.

On Thu, 23 Jul 2020 at 14:39, Dominik 'Rathann' Mierzejewski
 wrote:
> On Wednesday, 22 July 2020 at 14:49, Petr Šplíchal wrote:
> > Hi!
> >
> > I am writing today to share a proposal for how we can make it
> > easier to enable tests in Fedora CI.
> >
> > We've heard the user story mentioned in the subject many times.
> > Fedora developers have frequently mentioned the difficulties which
> > they face with enabling tests in Fedora CI. Some people don't
> > understand the complex yaml syntax. Other people find that they
> > can't easily reproduce issues revealed by Fedora CI on their
> > laptops. People have asked us to provide a tool for local testing:
> > https://pagure.io/fedora-ci/general/issue/4
> >
> > A few of us have been working on a tool which we hope will improve
> > the situation considerably. The proof of concept below shows how
> > everyday developer tasks (related to testing) might look in the
> > near future:
> >
> > Enable basic smoke test for my component:
> > dnf install -y tmt
> > tmt init --template mini
> > vim plans/example.fmf
> > git add . && git commit -m tests && git push
> >
> > Create a new test based on a shell/beakerlib template:
> > tmt test create --template shell tests/basic
> > tmt test create --template beakerlib tests/advanced
> >
> > Run all/selected tests safely in a virtual machine:
> > tmt run
> > tmt run test --name smoke
>
> I'd say it's good to have such option, but for quick testing a full VM
> is often too heavy and slow to boot. Even a container takes time to set
> up and run.

Yes, there are different needs and preferences related to the
environment where the tests should be executed. We want to provide
enough flexibility here so the 'provision' step allows you to
choose your preferred way.
> My primary environment would be local.

So for your case that would be 'provision --how local'. If you
have a test box prepared for testing you could also connect to it
using 'provision --how connect'.
> > Execute tests in my preferred environment:
> > tmt run --all provision --how container
> > tmt run --all provision --how virtual
> > tmt run --all provision --how local
> > tmt run --all provision --how ...
>
> I find it difficult to guess what the above commands actually do.
> Do they run the tests or do they provision environments to run the
> tests? Or both?

The 'tmt run' command is used to run tests. By default it runs all
steps (discover, provision, prepare, execute, report, finish) so
the examples above would run tests in the requested environment.

It is also possible to run only selected steps: For example, one
of the use cases shared with us was 'just tell me what tests would
be run, do not run them', this can be achieved with the following:

tmt run discover

For detailed description of individual steps have a look here:
https://tmt.readthedocs.io/en/latest/spec/steps.html
> > I would like to ask for your feedback about this. What do you
> > think? Is this something that would be useful for you? What do
> > you think of the syntax? We would love to hear back from you
> > concerning what essential features you might expect from such a
> > tool. It would also be helpful to know if there is anything that
> > should be changed early in the tool design.
> >
> > For more examples and frequently asked questions have a look here:
> > https://tmt.readthedocs.io/en/latest/examples.html
> > https://docs.fedoraproject.org/en-US/ci/tmt/
> >
> > Looking forward to your feedback and ideas! :)
>
> tmt examples are overcomplicated and not explained in enough detail.
> What is "metadata" and why do you need 4 "levels" of it?

Metadata levels allow to store all needed test execution data at
appropriate location, the levels are described here:
https://tmt.readthedocs.io/en/latest/spec.html

We want the config for simple/common use cases (such as running a
shell script as a test) as simple as possible so that users do not
have to study all the details but just create something like this:

execute:
        script: ./run-my-test.sh

If one needs to maintain larger set of tests the L1 metadata can
come handy:
https://tmt.readthedocs.io/en/latest/spec/tests.html
> What are "tiers" for?

They are used to organize tests into groups by priority.
https://tmt.readthedocs.io/en/latest/spec/tests.html#tier
> Which items are mandatory and which are optional?

In order to enable tests you just have to define the 'execute'
step. Use 'tmt init --template mini' to get a quick start.
> What's "relevancy"?

It's a test (L1) attribute allowing to filter relevant tests.
https://tmt.readthedocs.io/en/latest/spec/tests.html#relevancy
Currently we're working on open-sourcing the implementation.
> Why is there "extra-summary" in addition to "summary"?
> What's "extra-task"?

These are because of backward compatibility, for more see:
https://tmt.readthedocs.io/en/latest/stories/cli.html?highlight=extra-task#n...
> What are "plans" for?

Plans define which tests should be executed, how the environment
should be provisioned and prepared, how the results should be
reported. See the following page for more details about plans.
https://tmt.readthedocs.io/en/latest/spec/plans.html
> What are "stories" for?

Stories are an advanced, optional feature which can help you to track
which features/requirements have been already implemented, tested
and documented.
https://tmt.readthedocs.io/en/latest/spec/stories.html
https://github.com/psss/tmt/tree/master/stories
> You're using a lot of terms as if everyone knows what they mean
> in your context.

Sorry for that. We're trying to document the concept as best as we
can. Hopefully, the links above have helped. Perhaps we could add
some more references to the examples page.

Thank you for all your feedback. Please, let us know what's still
missing and how we can further improve the docs. Thanks.

psss...
> Regards,
> Dominik
> --
> Fedora   https://getfedora.org  |  RPM Fusion  http://rpmfusion.org
> There should be a science of discontent. People need hard times and
> oppression to develop psychic muscles.
>         -- from "Collected Sayings of Muad'Dib" by the Princess Irulan
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]