6:50 a.m.
As we're working more with the test invocation spec [1] and the
ansible implementation [2], there are a few things which aren't 100%
clear to me and I'm hoping to get some clarification on.
Why is "ansible_connection=local" a requirement for the testing system?
Why is obtaining the test subject solely the responsibility of the
testing system? This seems like something that could potentially have
different requirements for different usecases. For example, something
like abicheck needs to grab previous builds in addition to the latest
build of a given package.
I don't understand what we're trying to accomplish with all the
test_<subject>.yml. Is the idea that we'd run test_oci.yml for every
package used in building a container against that container?
What is the point in having test_local.yml and test_rpm.yml? I remember
the idea of in-situ tests being part of the discussion but I thought
that we were pretty limited on what we could do with those since we're
not going to be modifying rpm like debian was able to get changes made
for .deb packages.
Thanks,
Tim
[1] https://fedoraproject.org/wiki/Changes/InvokingTests
[2] https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible
1 p.m.
On 24.05.2017 13:50, Tim Flink wrote:
As we're working more with the test invocation spec [1] and the
ansible implementation [2], there are a few things which aren't 100%
clear to me and I'm hoping to get some clarification on.
Why is "ansible_connection=local" a requirement for the testing system?
The initial invocation of Ansible is meant to act locally. The ansible
scripts are meant to stage their own containers, or virtual machines
(eg: with qemu, or libvirt) and then perform further tasks into them.
Why is obtaining the test subject solely the responsibility of the
testing system? This seems like something that could potentially have
different requirements for different usecases. For example, something
like abicheck needs to grab previous builds in addition to the latest
build of a given package.
It seems like those previous builds are not the test subject. They are
part of the fixtures of the test or test suite. In CI terms, the test
subject is the thing that has changed. The fixtures are things it are
tested against. We didn't include the term 'fixtures' in the terminology
section, but it's usage here is the usual usage within testing.
I don't understand what we're trying to accomplish with all
the
test_<subject>.yml. Is the idea that we'd run test_oci.yml for every
package used in building a container against that container?
When a test is written in such a way that it makes sense to test it as
an OCI image, then we expect that test_oci.yml is present. In some cases
this file will be a boilerplate of a few lines.
What is the point in having test_local.yml and test_rpm.yml? I
remember
the idea of in-situ tests being part of the discussion but I thought
that we were pretty limited on what we could do with those since we're
not going to be modifying rpm like debian was able to get changes made
for .deb packages.
test_rpm.yml installs RPMs and then tests against them. test_local.yml
assumes that everything to be tested has already been installed.
Both test_rpm.yml and test_local.yml operate in-situ, but one performs
an additional action.
Again, as with test_oci.yml if test_rpm.yml is commonly the same
boilerplate, we could find a simple way to reduce that. Do you have
ideas? It does seem to be necessary for packages to be able to deviate
from the default behavior in cases. So such a solution should not be a
blanket thing.
Cheers,
Stef
9:46 a.m.
On Wed, 24 May 2017 20:00:00 +0200
Stef Walter <stefw(a)redhat.com> wrote:
On 24.05.2017 13:50, Tim Flink wrote:
> As we're working more with the test invocation spec [1] and the
> ansible implementation [2], there are a few things which aren't 100%
> clear to me and I'm hoping to get some clarification on.
>
> Why is "ansible_connection=local" a requirement for the testing
> system?
The initial invocation of Ansible is meant to act locally. The ansible
scripts are meant to stage their own containers, or virtual machines
(eg: with qemu, or libvirt) and then perform further tasks into them.
Sure but I'm wondering why this is a requirement. If we were using
shell scripts, this makes a lot of sense. As we're using ansible, there
isn't a whole lot of difference between executing locally vs. remotely
so long as ssh keys are set up properly.
In a way, it feels like we've decided to use the autopkgtest paradigm
but dropped ansible in as "bash++". I'm having a hard time seeing how
the current setup is that much better than bash and something like
beakerlib.
Don't get me wrong, I'm all for doing automation with Ansible - it's
the restrictions placed around that automation that I'm questioning.
> Why is obtaining the test subject solely the responsibility of
the
> testing system? This seems like something that could potentially
> have different requirements for different usecases. For example,
> something like abicheck needs to grab previous builds in addition
> to the latest build of a given package.
It seems like those previous builds are not the test subject. They are
part of the fixtures of the test or test suite. In CI terms, the test
subject is the thing that has changed. The fixtures are things it are
tested against. We didn't include the term 'fixtures' in the
terminology section, but it's usage here is the usual usage within
testing.
> I don't understand what we're trying to accomplish with all the
> test_<subject>.yml. Is the idea that we'd run test_oci.yml for every
> package used in building a container against that container?
When a test is written in such a way that it makes sense to test it as
an OCI image, then we expect that test_oci.yml is present. In some
cases this file will be a boilerplate of a few lines.
When would it make sense to test an RPM as an OCI image? Are we
expecting all the test_<subject>.yml files to be in every repo?
> What is the point in having test_local.yml and test_rpm.yml? I
> remember the idea of in-situ tests being part of the discussion but
> I thought that we were pretty limited on what we could do with
> those since we're not going to be modifying rpm like debian was
> able to get changes made for .deb packages.
test_rpm.yml installs RPMs and then tests against them. test_local.yml
assumes that everything to be tested has already been installed.
I don't understand why we're taking the stance of "we don't plan on
letting you install the RPMs under test but we do expect you to come up
with your own method of booting VMs". It seems very strange but I could
be missing something here.
I understand the need to allow folks to set up and tear down their own
set of VMs for testing but I'm still not understanding why this is the
only method that we'll support. I would have thought that the ability
to specify the type(s) and number of VMs needed would be an easier route
for many tests. Is there something I'm missing?
Another concern I have with installing RPMs for people is what to do
about builds which depend on each other and ordering. Are we assuming
that test case dependencies will declare all the dependencies needed to
install a new build? How will we be handling cases where multiple
builds are needed to test against eachother but are not built at the
same time?
Both test_rpm.yml and test_local.yml operate in-situ, but one
performs
an additional action.
Again, as with test_oci.yml if test_rpm.yml is commonly the same
boilerplate, we could find a simple way to reduce that. Do you have
ideas? It does seem to be necessary for packages to be able to deviate
from the default behavior in cases. So such a solution should not be a
blanket thing.
I agree that there will be and should be some deviation from the
default behavior.
I don't have any good ideas to reduce the boilerplate but I'm still
struggling to understand the details of what we're aiming to do with
all of this. Are we planning to support more than just per
build/container testing with this interface? If I wanted to write a
test for FreeIPA that used multiple hosts to create and poke at an AD
setup (for example), would I be writing that using this interface or is
that out of scope for what we're trying to do?
Tim
11:02 a.m.
On 06.06.2017 16:46, Tim Flink wrote:
On Wed, 24 May 2017 20:00:00 +0200
Stef Walter <stefw(a)redhat.com> wrote:
> On 24.05.2017 13:50, Tim Flink wrote:
>> As we're working more with the test invocation spec [1] and the
>> ansible implementation [2], there are a few things which aren't 100%
>> clear to me and I'm hoping to get some clarification on.
>>
>> Why is "ansible_connection=local" a requirement for the testing
>> system?
>
> The initial invocation of Ansible is meant to act locally. The ansible
> scripts are meant to stage their own containers, or virtual machines
> (eg: with qemu, or libvirt) and then perform further tasks into them.
Sure but I'm wondering why this is a requirement. If we were using
shell scripts, this makes a lot of sense. As we're using ansible, there
isn't a whole lot of difference between executing locally vs. remotely
so long as ssh keys are set up properly.
In a way, it feels like we've decided to use the autopkgtest paradigm
but dropped ansible in as "bash++". I'm having a hard time seeing how
the current setup is that much better than bash and something like
beakerlib.
Don't get me wrong, I'm all for doing automation with Ansible - it's
the restrictions placed around that automation that I'm questioning.
It's easy to describe Ansible as "ssh in a bash for loop", but we're
trying to use it as Ansible.
Requirement: it is the job of the tests (and its roles and/or framework)
to launch any containers and virtual machines ... and run actual tests
inside of them.
The use of ansible_connection=local is a way enshrining that
requirement. But if there's a better way to enshrine that requirement,
then lets do it. Perhaps one will come to mind?
>> Why is obtaining the test subject solely the responsibility
of the
>> testing system? This seems like something that could potentially
>> have different requirements for different usecases. For example,
>> something like abicheck needs to grab previous builds in addition
>> to the latest build of a given package.
>
> It seems like those previous builds are not the test subject. They are
> part of the fixtures of the test or test suite. In CI terms, the test
> subject is the thing that has changed. The fixtures are things it are
> tested against. We didn't include the term 'fixtures' in the
> terminology section, but it's usage here is the usual usage within
> testing.
>
>> I don't understand what we're trying to accomplish with all the
>> test_<subject>.yml. Is the idea that we'd run test_oci.yml for every
>> package used in building a container against that container?
>
> When a test is written in such a way that it makes sense to test it as
> an OCI image, then we expect that test_oci.yml is present. In some
> cases this file will be a boilerplate of a few lines.
When would it make sense to test an RPM as an OCI image? Are we
expecting all the test_<subject>.yml files to be in every repo?
No only those with tests that make sense to be run on that subject. So
for example if mariadb dist-git repo has tests that should be run in an
OCI image, it would have test_oci.yml. In some cases it will be the same
or similar tests, run on a different subject.
One of key things about CI detecting failures is running tests multiple
times, varying certain minor things about it on each run. Often the
thing being varied is the test subject.
>> What is the point in having test_local.yml and test_rpm.yml?
I
>> remember the idea of in-situ tests being part of the discussion but
>> I thought that we were pretty limited on what we could do with
>> those since we're not going to be modifying rpm like debian was
>> able to get changes made for .deb packages.
>
> test_rpm.yml installs RPMs and then tests against them. test_local.yml
> assumes that everything to be tested has already been installed.
I don't understand why we're taking the stance of "we don't plan on
letting you install the RPMs under test but we do expect you to come up
with your own method of booting VMs". It seems very strange but I could
be missing something here.
That's not the case. You can come up with your own method ... but most
tests can just use:
https://admin.fedoraproject.org/pkgdb/package/rpms/standard-test-roles/
One of the great things about using Ansible here is sharing roles for
common cases. Such as starting a VM.
The point is that it's not the job of the CI system to launch the test
subject as a VM or containers, with a specific IP address, architecture,
disks mounts etc... The point is to decouple the tests from that from
the CI system. The test accomplishes that, and usually does so with
standard stuff ^^.
I understand the need to allow folks to set up and tear down their
own
set of VMs for testing but I'm still not understanding why this is the
only method that we'll support. I would have thought that the ability
to specify the type(s) and number of VMs needed would be an easier route
for many tests. Is there something I'm missing?
Because it locks in the tests to a certain CI system, thus creating
exactly the same fiasco we have with beakerlib tests being tied to the
method of invocation.
Another concern I have with installing RPMs for people is what to do
about builds which depend on each other and ordering. Are we assuming
that test case dependencies will declare all the dependencies needed to
install a new build? How will we be handling cases where multiple
builds are needed to test against eachother but are not built at the
same time?
For modules, repo, OCI, QCow2 test subjects obviously this is already
solved. If the compose succeeded, then that works.
For RPMs as test subjects it is up to the CI system to put all these
test subjects together to be tested together in a batch. We could copy
Debian/Ubuntu's mechanism here ... or come up with our own.
See these clauses in the spec:
"The testing system SHOULD stage the tests on Fedora operating system
appropriate for the branch name of the dist-git repository containing
the tests."
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsible
Not everything here has been worked out. If you have possible solutions,
lets talk about them, and if necessary, add them to the spec.
> Both test_rpm.yml and test_local.yml operate in-situ, but one
performs
> an additional action.
>
> Again, as with test_oci.yml if test_rpm.yml is commonly the same
> boilerplate, we could find a simple way to reduce that. Do you have
> ideas? It does seem to be necessary for packages to be able to deviate
> from the default behavior in cases. So such a solution should not be a
> blanket thing.
I agree that there will be and should be some deviation from the
default behavior.
I don't have any good ideas to reduce the boilerplate but I'm still
struggling to understand the details of what we're aiming to do with
all of this. Are we planning to support more than just per
build/container testing with this interface? If I wanted to write a
test for FreeIPA that used multiple hosts to create and poke at an AD
setup (for example), would I be writing that using this interface or is
that out of scope for what we're trying to do?
That's in scope. This is exactly one of the reasons that the tests (more
likely shared roles/framework) launches the VMs that would enable such a
test.
https://github.com/cockpit-project/cockpituous/tree/master/tests
It's something we do all the time with the Cockpit integration tests. We
use local qemu/libvirt VMs for this that start in about 10 seconds. If
we want, once we're underway, we could contribute these tests to Fedora.
Cheers,
Stef
7:33 a.m.
Requirement: it is the job of the tests (and its roles and/or
framework)
to launch any containers and virtual machines ... and run actual tests
inside of them.
The use of ansible_connection=local is a way enshrining that
requirement. But if there's a better way to enshrine that requirement,
then lets do it. Perhaps one will come to mind?
I think Tim is asking from a different perspective.
In Taskotron, we'll have a VM host. On VM host we'll spawn a clean VM for each
executed test suite. In order to execute the test suite, I see two options:
a) run the playbook from VM host and point to the VM as the target system. So ansible
process will be started on the VM host, but the whole execution will happen in the VM.
b) connect to the VM (ssh, paramiko, or ansible) and then run the test suite in the VM as
a local ansible process with local connection. The easiest way of connecting to the VM is
for us most probably again to use ansible, so this will be ansible-in-ansible execution.
The spec currently mandates b) option. But in both options, the test suite is given an
empty machine, with nothing preinstalled/predefined, and it's completely on its own to
set up everything (as required by the spec). So from the test suite POV, both a) and b)
are equivalent. But from the test system POV, running it as a) is easier (no
ansible-in-ansible wrappers).
Please tell me if I'm missing something here. Why wouldn't we allow a) when it
doesn't affect test suite's self-reliance?
Thanks,
Kamil
2509
days inactive
2529
days old
4 comments
3 participants
participants (3)
-
Kamil Páral -
Stef Walter -
Tim Flink