7:55 a.m.
Summary: I'd like to use Ansible "inventory" [0] to describe test
subjects to be tested:
http://docs.ansible.com/ansible/intro_inventory.html
I've done initial exploratory work on this, pull requests below, more to
follow.
In our standard test invocation spec, we refer to "test subjects" as the
thing being tested:
https://fedoraproject.org/wiki/Changes/InvokingTests
Mechanisms related to these test subjects have been poorly thought out
in the spec. This is my fault. Now that we're trying to implement this,
I think we can take what we've learned and update the spec.
Two areas of interest:
1. Transforming test subjects into something testable by Ansible
By using an Ansible inventory directory we can have inventory
scripts launch or prepare a system ready for the Ansible based
tests to run against it. This directory is tests/subjects
The CI testing system will launch the tests like this:
# TEST_SUBJECTS='/path/to/atomic.qcow2 /path/to/sed.rpm' \
ansible-playbook -i tests/subjects tests/tests.yml
The effect of the above command would be to run tests/tests.yml
twice, once against the qcow2 image and once against installed
rpms (in-situ testing).
Ansible supports having a directory as its inventory. All
executable files in that directory are asked to produce
inventory.
Use of an environment variable as a way to pass information
to Ansible inventory scripts is standard practice.
Although each tests *could* have their own inventory scripts,
these will commonly be shared. I've started a pull request here
related to such shared default scripts:
https://pagure.io/standard-test-roles/pull-request/9
You can play with these scripts. One launches a qcow2 image as
a VM and makes that VM available to Ansible via inventory. A second
inventory script installs RPMs and then tells Ansible to run
locally.
Another not yet written inventory script would launch a docker image
and tell Ansible how to connect to it. Another would configure a
module repo ... and so on.
2. A dist-git repo should describe which test subjects are applicable
Following on from the above the tests/subjects directory should
either contain executable inventory scripts that the tests would
likely use to parse $TEST_SUBJECTS into something that Ansible
can execute tests against.
$TEST_SUBJECTS is a space separated list of paths of things
to test. This is a likely bike shed topic, and I'm open to ideas
here.
Each inventory script in tests/subjects directory consumes some
different part of the environment variable(s).
Most tests will *not* want to write their own inventory scripts
and will instead just include symlinks to well known inventory
scripts in standard-test-roles.
https://pagure.io/standard-test-roles/pull-request/9
The symlinks are necessary, as the test *must* be in control
of describing which types of subjects, and thus Ansible inventory
it supports.
Some advanced tests (such as the Cockpit tests or IPA tests) would
launch more complex local inventory and take control of what
inventory is reported to Ansible.
On the spec side, I strongly believe that both of these things should
remain in the firm control of the tests stored in the dist-git repo.
While still having useful shared code to implement the spec.
Related: The ansible_connection=local nonsense in the current roles and
spec would be dropped.
I aim watch for discussion here on this topic for the next days or two,
and create a new wiki page with an updated spec after that:
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo
Cheers,
Stef
1:42 a.m.
On Mon, Jun 19, 2017 at 2:55 PM, Stef Walter <stefw(a)redhat.com> wrote:
Summary: I'd like to use Ansible "inventory" [0] to
describe test
subjects to be tested:
http://docs.ansible.com/ansible/intro_inventory.html
I've done initial exploratory work on this, pull requests below, more to
follow.
We must be careful that in the upcoming ansible 2.4, the inventory system
has been rewritten to be plugin based. It could be interesting to use both
the current version of ansible and the development version of ansible to
validate the inventory approach to avoid surprises.
In our standard test invocation spec, we refer to "test
subjects" as the
thing being tested:
https://fedoraproject.org/wiki/Changes/InvokingTests
Mechanisms related to these test subjects have been poorly thought out
in the spec. This is my fault. Now that we're trying to implement this,
I think we can take what we've learned and update the spec.
Two areas of interest:
1. Transforming test subjects into something testable by Ansible
By using an Ansible inventory directory we can have inventory
scripts launch or prepare a system ready for the Ansible based
tests to run against it. This directory is tests/subjects
The CI testing system will launch the tests like this:
# TEST_SUBJECTS='/path/to/atomic.qcow2 /path/to/sed.rpm' \
ansible-playbook -i tests/subjects tests/tests.yml
The effect of the above command would be to run tests/tests.yml
twice, once against the qcow2 image and once against installed
rpms (in-situ testing).
Ansible supports having a directory as its inventory. All
executable files in that directory are asked to produce
inventory.
Use of an environment variable as a way to pass information
to Ansible inventory scripts is standard practice.
Although each tests *could* have their own inventory scripts,
these will commonly be shared. I've started a pull request here
related to such shared default scripts:
https://pagure.io/standard-test-roles/pull-request/9
You can play with these scripts. One launches a qcow2 image as
a VM and makes that VM available to Ansible via inventory. A second
inventory script installs RPMs and then tells Ansible to run
locally.
Another not yet written inventory script would launch a docker image
and tell Ansible how to connect to it. Another would configure a
module repo ... and so on.
2. A dist-git repo should describe which test subjects are applicable
Following on from the above the tests/subjects directory should
either contain executable inventory scripts that the tests would
likely use to parse $TEST_SUBJECTS into something that Ansible
can execute tests against.
$TEST_SUBJECTS is a space separated list of paths of things
to test. This is a likely bike shed topic, and I'm open to ideas
here.
Each inventory script in tests/subjects directory consumes some
different part of the environment variable(s).
Most tests will *not* want to write their own inventory scripts
and will instead just include symlinks to well known inventory
scripts in standard-test-roles.
https://pagure.io/standard-test-roles/pull-request/9
The symlinks are necessary, as the test *must* be in control
of describing which types of subjects, and thus Ansible inventory
it supports.
Some advanced tests (such as the Cockpit tests or IPA tests) would
launch more complex local inventory and take control of what
inventory is reported to Ansible.
On the spec side, I strongly believe that both of these things should
remain in the firm control of the tests stored in the dist-git repo.
While still having useful shared code to implement the spec.
Related: The ansible_connection=local nonsense in the current roles and
spec would be dropped.
I aim watch for discussion here on this topic for the next days or two,
and create a new wiki page with an updated spec after that:
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo
Could you elaborate on the capabilities that tests could request from the
CI system? And then we'll need to elaborate on conventions to describe the
usage of these capabilities in the inventories and playbooks of the tests.
Another comment that is more general: I strongly feel that we also need to
describe how to run the tests locally without any CI system to help develop
and debug the tests on the packagers' systems.
--
Fred - May the source be with you
6:17 a.m.
On 23.06.2017 08:42, Frederic Lepied wrote:
On Mon, Jun 19, 2017 at 2:55 PM, Stef Walter <stefw(a)redhat.com
<mailto:stefw@redhat.com>> wrote:
Summary: I'd like to use Ansible "inventory" [0] to describe test
subjects to be tested:
http://docs.ansible.com/ansible/intro_inventory.html
<http://docs.ansible.com/ansible/intro_inventory.html>
I've done initial exploratory work on this, pull requests below, more to
follow.
We must be careful that in the upcoming ansible 2.4, the inventory
system has been rewritten to be plugin based. It could be interesting to
use both the current version of ansible and the development version of
ansible to validate the inventory approach to avoid surprises.
Very good point. Are you referring to the inventory_plugins
configuration directive we start to see in ansible.cfg on ansible 2.3
and similar? Or something that goes further than that?
In our standard test invocation spec, we refer to "test subjects" as the
thing being tested:
https://fedoraproject.org/wiki/Changes/InvokingTests
<https://fedoraproject.org/wiki/Changes/InvokingTests>
Mechanisms related to these test subjects have been poorly thought out
in the spec. This is my fault. Now that we're trying to implement this,
I think we can take what we've learned and update the spec.
Two areas of interest:
1. Transforming test subjects into something testable by Ansible
By using an Ansible inventory directory we can have inventory
scripts launch or prepare a system ready for the Ansible based
tests to run against it. This directory is tests/subjects
The CI testing system will launch the tests like this:
# TEST_SUBJECTS='/path/to/atomic.qcow2 /path/to/sed.rpm' \
ansible-playbook -i tests/subjects tests/tests.yml
The effect of the above command would be to run tests/tests.yml
twice, once against the qcow2 image and once against installed
rpms (in-situ testing).
Ansible supports having a directory as its inventory. All
executable files in that directory are asked to produce
inventory.
Use of an environment variable as a way to pass information
to Ansible inventory scripts is standard practice.
Although each tests *could* have their own inventory scripts,
these will commonly be shared. I've started a pull request here
related to such shared default scripts:
https://pagure.io/standard-test-roles/pull-request/9
<https://pagure.io/standard-test-roles/pull-request/9>
You can play with these scripts. One launches a qcow2 image as
a VM and makes that VM available to Ansible via inventory. A second
inventory script installs RPMs and then tells Ansible to run
locally.
Another not yet written inventory script would launch a docker image
and tell Ansible how to connect to it. Another would configure a
module repo ... and so on.
2. A dist-git repo should describe which test subjects are applicable
Following on from the above the tests/subjects directory should
either contain executable inventory scripts that the tests would
likely use to parse $TEST_SUBJECTS into something that Ansible
can execute tests against.
$TEST_SUBJECTS is a space separated list of paths of things
to test. This is a likely bike shed topic, and I'm open to ideas
here.
Each inventory script in tests/subjects directory consumes some
different part of the environment variable(s).
Most tests will *not* want to write their own inventory scripts
and will instead just include symlinks to well known inventory
scripts in standard-test-roles.
https://pagure.io/standard-test-roles/pull-request/9
<https://pagure.io/standard-test-roles/pull-request/9>
The symlinks are necessary, as the test *must* be in control
of describing which types of subjects, and thus Ansible inventory
it supports.
Some advanced tests (such as the Cockpit tests or IPA tests) would
launch more complex local inventory and take control of what
inventory is reported to Ansible.
On the spec side, I strongly believe that both of these things should
remain in the firm control of the tests stored in the dist-git repo.
While still having useful shared code to implement the spec.
Related: The ansible_connection=local nonsense in the current roles and
spec would be dropped.
I aim watch for discussion here on this topic for the next days or two,
and create a new wiki page with an updated spec after that:
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo
<https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo>
Could you elaborate on the capabilities that tests could request from
the CI system? And then we'll need to elaborate on conventions to
describe the usage of these capabilities in the inventories and
playbooks of the tests.
In general the tests do not request things from the CI system.
They use ansible to perform the tasks that they need. The intention is
that the tests are runnable without a CI system, and with various CI
systems, and this standard interface is the only common ground.
Do you have examples where you feel documentation is missing here?
Documentation is next up for me.
Another comment that is more general: I strongly feel that we also
need
to describe how to run the tests locally without any CI system to help
develop and debug the tests on the packagers' systems.
Indeed. This is a documentation problem. After these changes it's next
on my list to fit in ... unless someone beats me to it :)
Fred - May the source be with you
6:47 a.m.
On 23.06.2017 13:17, Stef Walter wrote:
On 23.06.2017 08:42, Frederic Lepied wrote:
>
> On Mon, Jun 19, 2017 at 2:55 PM, Stef Walter <stefw(a)redhat.com
> <mailto:stefw@redhat.com>> wrote:
>
> Summary: I'd like to use Ansible "inventory" [0] to describe test
> subjects to be tested:
>
> http://docs.ansible.com/ansible/intro_inventory.html
> <http://docs.ansible.com/ansible/intro_inventory.html>
<snip>
I have made a proposed change to the spec to enable use of inventory
instead of test subject specific playbooks.
https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsibl...
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo
Your feedback is welcome.
The use of an inventory script is a better way to convert a test subject
into a target that an ansible playbook can be launched against.
Many tests will use default inventory scripts included in
standard-test-roles [0]. While at the same time leaving this firmly
under control of tests that need to be more complex.
I will work on examples next, and split out the documentation from that
page. Sorry for the slow progress here.
Cheers,
Stef
[0] https://pagure.io/standard-test-roles/pull-request/9
6:14 a.m.
I've merged the proposed change below, and worked further on the pull
request implementing the standard inventory scripts:
https://pagure.io/standard-test-roles/pull-request/9#
Next, I'll look into Ansible inventory_plugins as a way to provide these
scripts as a default for tests that do not provide their own
tests/inventory directory.
Cheers,
Stef
On 23.06.2017 13:47, Stef Walter wrote:
On 23.06.2017 13:17, Stef Walter wrote:
> On 23.06.2017 08:42, Frederic Lepied wrote:
>>
>> On Mon, Jun 19, 2017 at 2:55 PM, Stef Walter <stefw(a)redhat.com
>> <mailto:stefw@redhat.com>> wrote:
>>
>> Summary: I'd like to use Ansible "inventory" [0] to describe
test
>> subjects to be tested:
>>
>> http://docs.ansible.com/ansible/intro_inventory.html
>> <http://docs.ansible.com/ansible/intro_inventory.html>
<snip>
I have made a proposed change to the spec to enable use of inventory
instead of test subject specific playbooks.
https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTestsAnsibl...
https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo
Your feedback is welcome.
The use of an inventory script is a better way to convert a test subject
into a target that an ansible playbook can be launched against.
Many tests will use default inventory scripts included in
standard-test-roles [0]. While at the same time leaving this firmly
under control of tests that need to be more complex.
I will work on examples next, and split out the documentation from that
page. Sorry for the slow progress here.
Cheers,
Stef
[0] https://pagure.io/standard-test-roles/pull-request/9
5:04 a.m.
On Fri, Jun 23, 2017 at 1:17 PM, Stef Walter <stefw(a)redhat.com> wrote:
On 23.06.2017 08:42, Frederic Lepied wrote:
>
> On Mon, Jun 19, 2017 at 2:55 PM, Stef Walter <stefw(a)redhat.com
> <mailto:stefw@redhat.com>> wrote:
>
> Summary: I'd like to use Ansible "inventory" [0] to describe test
> subjects to be tested:
>
> http://docs.ansible.com/ansible/intro_inventory.html
> <http://docs.ansible.com/ansible/intro_inventory.html>
>
> I've done initial exploratory work on this, pull requests below,
more to
> follow.
>
>
> We must be careful that in the upcoming ansible 2.4, the inventory
> system has been rewritten to be plugin based. It could be interesting to
> use both the current version of ansible and the development version of
> ansible to validate the inventory approach to avoid surprises.
Very good point. Are you referring to the inventory_plugins
configuration directive we start to see in ansible.cfg on ansible 2.3
and similar? Or something that goes further than that?
I refer to this change:
https://github.com/ansible/ansible/commit/8f97aef1a365cbbbb822d6d09f96af1...
>
>
>
> In our standard test invocation spec, we refer to "test subjects" as
the
> thing being tested:
>
> https://fedoraproject.org/wiki/Changes/InvokingTests
> <https://fedoraproject.org/wiki/Changes/InvokingTests>
>
> Mechanisms related to these test subjects have been poorly thought
out
> in the spec. This is my fault. Now that we're trying to implement
this,
> I think we can take what we've learned and update the spec.
>
>
> Two areas of interest:
>
> 1. Transforming test subjects into something testable by Ansible
>
> By using an Ansible inventory directory we can have inventory
> scripts launch or prepare a system ready for the Ansible based
> tests to run against it. This directory is tests/subjects
>
> The CI testing system will launch the tests like this:
>
> # TEST_SUBJECTS='/path/to/atomic.qcow2 /path/to/sed.rpm' \
> ansible-playbook -i tests/subjects tests/tests.yml
>
> The effect of the above command would be to run tests/tests.yml
> twice, once against the qcow2 image and once against installed
> rpms (in-situ testing).
>
> Ansible supports having a directory as its inventory. All
> executable files in that directory are asked to produce
> inventory.
>
> Use of an environment variable as a way to pass information
> to Ansible inventory scripts is standard practice.
>
> Although each tests *could* have their own inventory scripts,
> these will commonly be shared. I've started a pull request here
> related to such shared default scripts:
>
> https://pagure.io/standard-test-roles/pull-request/9
> <https://pagure.io/standard-test-roles/pull-request/9>
>
> You can play with these scripts. One launches a qcow2 image as
> a VM and makes that VM available to Ansible via inventory. A
second
> inventory script installs RPMs and then tells Ansible to run
> locally.
>
> Another not yet written inventory script would launch a docker
image
> and tell Ansible how to connect to it. Another would configure a
> module repo ... and so on.
>
> 2. A dist-git repo should describe which test subjects are
applicable
>
> Following on from the above the tests/subjects directory should
> either contain executable inventory scripts that the tests would
> likely use to parse $TEST_SUBJECTS into something that Ansible
> can execute tests against.
>
> $TEST_SUBJECTS is a space separated list of paths of things
> to test. This is a likely bike shed topic, and I'm open to ideas
> here.
>
> Each inventory script in tests/subjects directory consumes some
> different part of the environment variable(s).
>
> Most tests will *not* want to write their own inventory scripts
> and will instead just include symlinks to well known inventory
> scripts in standard-test-roles.
>
> https://pagure.io/standard-test-roles/pull-request/9
> <https://pagure.io/standard-test-roles/pull-request/9>
>
> The symlinks are necessary, as the test *must* be in control
> of describing which types of subjects, and thus Ansible inventory
> it supports.
>
> Some advanced tests (such as the Cockpit tests or IPA tests)
would
> launch more complex local inventory and take control of what
> inventory is reported to Ansible.
>
> On the spec side, I strongly believe that both of these things should
> remain in the firm control of the tests stored in the dist-git repo.
> While still having useful shared code to implement the spec.
>
> Related: The ansible_connection=local nonsense in the current roles
and
> spec would be dropped.
>
> I aim watch for discussion here on this topic for the next days or
two,
> and create a new wiki page with an updated spec after that:
>
> https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo
> <https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo>
>
>
>
> Could you elaborate on the capabilities that tests could request from
> the CI system? And then we'll need to elaborate on conventions to
> describe the usage of these capabilities in the inventories and
> playbooks of the tests.
In general the tests do not request things from the CI system.
They use ansible to perform the tasks that they need. The intention is
that the tests are runnable without a CI system, and with various CI
systems, and this standard interface is the only common ground.
Do you have examples where you feel documentation is missing here?
Documentation is next up for me.
Trying to reformulate: if your tests needs resources to be able to run how
do we describe them to the CI system? What we describe up to now is if we
want to test in rpm, ostree or OCI form, here is the playbook to test the
artifact. If to test one of the artifact for example we need to instance 3
VM or a special networking topology or whatever resource, how will it work?
Thanks,
--
Fred - May the source be with you
2481
days inactive
2502
days old
5 comments
2 participants
participants (2)
-
Frederic Lepied -
Stef Walter