Hi Miro,

Thanks for the reply!

On Wed, Apr 14, 2021 at 02:16:02PM +0200, Miroslav Vadkerti wrote:
>Hi Coiby,
>
>On Tue, Apr 13, 2021 at 12:57 PM Coiby Xu <coxu@redhat.com> wrote:
>
>> On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
>> >On Tue, 6 Apr 2021 at 11:28, Coiby Xu <coxu@redhat.com> wrote:
>> >
>> >> Hi Tomas,
>> >>
>> >> Thank you for the reply!
>> >>
>> >> On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
>> >> >Hi Coiby, thanks for reaching out!
>> >> >
>> >> >Packit service is using the Testing Farm (TF) project to run tests. TF
>> >> >itself uses tmt and fmf projects [tmt]. So if you want to run your
>> >> >tests in the packit workflow, you'd need to make sure they run
>> >> >correctly with tmt tool itself. I am CCing Petr Splichal and Miro
>> >> >Vadkerti, the lead developers of tmt and TF projects. Petr is actually
>> >> >running periodic onboarding sessions where he's helping engineers to
>> >> >onboard their projects. I'm not sure if TF supports testing VM reboots
>> >> >right now, I will defer answering that question to the guys.
>> >> >
>> >> >[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary
>> >> >  https://tmt.readthedocs.io/en/stable/
>> >> >
>> >>
>> >> Last week I played with tmt to run the shell tests. For kdump tests, an
>> >> important question is whether TF supports testing VM reboots. So I wait
>> >> for Petr Splichal and Miro Vadkerti's answer.
>> >>
>> >
>> >Hi!
>> >
>> >As for now, reboot during the test execution itself is not supported in
>> >tmt. Here's the issue where we discuss the feature:
>> >
>> >   - https://github.com/psss/tmt/issues/281
>> >
>> >Reboot is possible using an ansible playbook during the prepare step:
>> >
>> >   - https://tmt.readthedocs.io/en/latest/spec/plans.html#ansible
>> >
>> >However, I'm not sure this approach would be sufficient for the use case
>> >you've described (e.g. several vm instances). I guess Miro will provide
>> >some more details from the TFT point of view.
>> >
>>
>> Hi Petr,
>>
>> Before Miro provides more details, I'd like to run the tests inside a
>> docker
>> and manage multiple VM instances by the tests themselves to evaluate
>> the benefits of Testing Farm.
>
>
>I expect in this reply that "manage multiple VM instances by tests" means,
>that your tests will take care of provisioning of the VMs via available
>/dev/kvm (e.g using libvirt, qemu-kvm, or similar ...) on the test host ...
>
>If that is not what you are expecting, I guess I will need to reply again
>...

Yes, what the kdump tests [2] do is to use qemu-kvm to start multiple VM
instances. This part of work has already be implemented by my collogue.
But this is not I'm expecting. I'm afraid you need you reply again.

Since Packit supports running tests inside a VM, I expect packet to
provision multiple VM instances for a single test and make the console
output of the VMs accessible to test.

If this works, one great benefit
of using Packit is I can run the sames tests for both upstream and
downstream kexec-tools and dracut. Because commits backported from
upstream dracut could also break kexec-tools.

Btw, the kdump tests have already implemented a framework to provision
multiple VM instances and watch the console output. Maybe I can help
implement these features for Packit?

[2] https://src.fedoraproject.org/rpms/kexec-tools/blob/f34/f/tests
>
>....
>
>Currently, for Fedora CI and Packit, we use AWS EC2 instances, which do not
>support
>nested virtualization and we do not yet have bare metal support with AWS :/.
>

I see. So a AWS EC2 instance doesn't support starting a VM via qemu,
right?

>So ... the only way to make this work **now** is to use internal
>infrastructure,
>but that is outside of Packit, and you would need to call
>Testing Farm service yourself (from Github actions for example), scripting
>the communication.
>That way, you could, from upstream pull request, provision internal
>infrastructure and return back results. The only drawback of this solution
>is that the artifacts storage are not accessible from the public, you will
>just get an xunit XML with Pass/Fail results, but the links will lead to
>internal
>Red Hat infrastructure.
>
>Unfortunately, as we do not yet have any users doing exactly this, this
>would
>be a prototype for your team. But should be possible ...
>
>If the internal artifact storage is a no-go for you, we do not have a
>possible
>solution right now. But we will be glad to talk about possibilities, like
>prototyping the use of AWS bare metal instances ....
>
>If the problem is that this goes around Packit, we can have a talk if this
>feature
>(run tests against internal infrastucture) is something we could also
>support via Packit.

This feature is nice to have but I could prefer what has been described
before.

>But this also needs some additional work ...
>
>Unfortunately we are a bit behind with docs, so the best sources we now
>have are:
>
>https://packit.dev/docs/testing-farm/
>https://docs.fedoraproject.org/en-US/ci/tmt/
>

Thanks for recommending these docs!


>But what I wrote here is not included there ...
>
>Best regards,
>/M
>
>
>
>> Tomas said you are running periodic
>> onboarding sessions. When will you launch the next session? Can you share
>> some materials with me since docs.fedoraproject.org/en-US/ci seems to be
>> outdated?
>>
>
>
>
>
>
>--
>Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE / OSCI
>IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95
>TPB-C 2C215 :: Mobile +420 773 944 252
>Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR

--
Best regards,
Coiby