On Wednesday, 22 July 2020 at 14:49, Petr Šplíchal wrote:
Hi!
I am writing today to share a proposal for how we can make it easier to enable tests in Fedora CI.
We've heard the user story mentioned in the subject many times. Fedora developers have frequently mentioned the difficulties which they face with enabling tests in Fedora CI. Some people don't understand the complex yaml syntax. Other people find that they can't easily reproduce issues revealed by Fedora CI on their laptops. People have asked us to provide a tool for local testing:
https://pagure.io/fedora-ci/general/issue/4
A few of us have been working on a tool which we hope will improve the situation considerably. The proof of concept below shows how everyday developer tasks (related to testing) might look in the near future:
Enable basic smoke test for my component:
dnf install -y tmt tmt init --template mini vim plans/example.fmf git add . && git commit -m tests && git push
Create a new test based on a shell/beakerlib template:
tmt test create --template shell tests/basic tmt test create --template beakerlib tests/advanced
Run all/selected tests safely in a virtual machine:
tmt run tmt run test --name smoke
I'd say it's good to have such option, but for quick testing a full VM is often too heavy and slow to boot. Even a container takes time to set up and run.
My primary environment would be local.
Execute tests in my preferred environment:
tmt run --all provision --how container tmt run --all provision --how virtual tmt run --all provision --how local tmt run --all provision --how ...
I find it difficult to guess what the above commands actually do. Do they run the tests or do they provision environments to run the tests? Or both?
I would like to ask for your feedback about this. What do you think? Is this something that would be useful for you? What do you think of the syntax? We would love to hear back from you concerning what essential features you might expect from such a tool. It would also be helpful to know if there is anything that should be changed early in the tool design.
For more examples and frequently asked questions have a look here:
https://tmt.readthedocs.io/en/latest/examples.html https://docs.fedoraproject.org/en-US/ci/tmt/
Looking forward to your feedback and ideas! :)
tmt examples are overcomplicated and not explained in enough detail. What is "metadata" and why do you need 4 "levels" of it? What are "tiers" for? Which items are mandatory and which are optional? What's "relevancy"? Why is there "extra-summary" in addition to "summary"? What's "extra-task"? What are "plans" for? What are "stories" for?
You're using a lot of terms as if everyone knows what they mean in your context.
Regards, Dominik