Hi Coiby,

On Tue, Apr 13, 2021 at 12:57 PM Coiby Xu <coxu@redhat.com> wrote:
On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
>On Tue, 6 Apr 2021 at 11:28, Coiby Xu <coxu@redhat.com> wrote:
>
>> Hi Tomas,
>>
>> Thank you for the reply!
>>
>> On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
>> >Hi Coiby, thanks for reaching out!
>> >
>> >Packit service is using the Testing Farm (TF) project to run tests. TF
>> >itself uses tmt and fmf projects [tmt]. So if you want to run your
>> >tests in the packit workflow, you'd need to make sure they run
>> >correctly with tmt tool itself. I am CCing Petr Splichal and Miro
>> >Vadkerti, the lead developers of tmt and TF projects. Petr is actually
>> >running periodic onboarding sessions where he's helping engineers to
>> >onboard their projects. I'm not sure if TF supports testing VM reboots
>> >right now, I will defer answering that question to the guys.
>> >
>> >[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary
>> >  https://tmt.readthedocs.io/en/stable/
>> >
>>
>> Last week I played with tmt to run the shell tests. For kdump tests, an
>> important question is whether TF supports testing VM reboots. So I wait
>> for Petr Splichal and Miro Vadkerti's answer.
>>
>
>Hi!
>
>As for now, reboot during the test execution itself is not supported in
>tmt. Here's the issue where we discuss the feature:
>
>   - https://github.com/psss/tmt/issues/281
>
>Reboot is possible using an ansible playbook during the prepare step:
>
>   - https://tmt.readthedocs.io/en/latest/spec/plans.html#ansible
>
>However, I'm not sure this approach would be sufficient for the use case
>you've described (e.g. several vm instances). I guess Miro will provide
>some more details from the TFT point of view.
>

Hi Petr,

Before Miro provides more details, I'd like to run the tests inside a docker
and manage multiple VM instances by the tests themselves to evaluate
the benefits of Testing Farm.

I expect in this reply that "manage multiple VM instances by tests" means,
that your tests will take care of provisioning of the VMs via available
/dev/kvm (e.g using libvirt, qemu-kvm, or similar ...) on the test host ...

If that is not what you are expecting, I guess I will need to reply again ...

....

Currently, for Fedora CI and Packit, we use AWS EC2 instances, which do not support
nested virtualization and we do not yet have bare metal support with AWS :/.

So ... the only way to make this work **now** is to use internal infrastructure,
but that is outside of Packit, and you would need to call
Testing Farm service yourself (from Github actions for example), scripting the communication.
That way, you could, from upstream pull request, provision internal
infrastructure and return back results. The only drawback of this solution
is that the artifacts storage are not accessible from the public, you will
just get an xunit XML with Pass/Fail results, but the links will lead to internal
Red Hat infrastructure.

Unfortunately, as we do not yet have any users doing exactly this, this would
be a prototype for your team. But should be possible ...

If the internal artifact storage is a no-go for you, we do not have a possible
solution right now. But we will be glad to talk about possibilities, like
prototyping the use of AWS bare metal instances ....

If the problem is that this goes around Packit, we can have a talk if this feature
(run tests against internal infrastucture) is something we could also support via Packit.
But this also needs some additional work ...

Unfortunately we are a bit behind with docs, so the best sources we now have are:


But what I wrote here is not included there ...

Best regards,
/M

 
Tomas said you are running periodic
onboarding sessions. When will you launch the next session? Can you share
some materials with me since docs.fedoraproject.org/en-US/ci seems to be
outdated?


 


--
Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE / OSCI
IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95
TPB-C 2C215 :: Mobile +420 773 944 252
Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR