Summary: I'd like to use Ansible "inventory" [0] to describe test
subjects to be tested:

http://docs.ansible.com/ansible/intro_inventory.html

I've done initial exploratory work on this, pull requests below, more to
follow.

In our standard test invocation spec, we refer to "test subjects" as the
thing being tested:

https://fedoraproject.org/wiki/Changes/InvokingTests

Mechanisms related to these test subjects have been poorly thought out
in the spec. This is my fault. Now that we're trying to implement this,
I think we can take what we've learned and update the spec.

Two areas of interest:

 1. Transforming test subjects into something testable by Ansible

    By using an Ansible inventory directory we can have inventory
    scripts launch or prepare a system ready for the Ansible based
    tests to run against it. This directory is tests/subjects

    The CI testing system will launch the tests like this:

    # TEST_SUBJECTS='/path/to/atomic.qcow2 /path/to/sed.rpm' \
         ansible-playbook -i tests/subjects tests/tests.yml

    The effect of the above command would be to run tests/tests.yml
    twice, once against the qcow2 image and once against installed
    rpms (in-situ testing).

    Ansible supports having a directory as its inventory. All
    executable files in that directory are asked to produce
    inventory.

    Use of an environment variable as a way to pass information
    to Ansible inventory scripts is standard practice.

    Although each tests *could* have their own inventory scripts,
    these will commonly be shared. I've started a pull request here
    related to such shared default scripts:

    https://pagure.io/standard-test-roles/pull-request/9

    You can play with these scripts. One launches a qcow2 image as
    a VM and makes that VM available to Ansible via inventory. A second
    inventory script installs RPMs and then tells Ansible to run
    locally.

    Another not yet written inventory script would launch a docker image
    and tell Ansible how to connect to it. Another would configure a
    module repo ... and so on.

 2. A dist-git repo should describe which test subjects are applicable

    Following on from the above the tests/subjects directory should
    either contain executable inventory scripts that the tests would
    likely use to parse $TEST_SUBJECTS into something that Ansible
    can execute tests against.

    $TEST_SUBJECTS is a space separated list of paths of things
    to test. This is a likely bike shed topic, and I'm open to ideas
    here.

    Each inventory script in tests/subjects directory consumes some
    different part of the environment variable(s).

    Most tests will *not* want to write their own inventory scripts
    and will instead just include symlinks to well known inventory
    scripts in standard-test-roles.

    https://pagure.io/standard-test-roles/pull-request/9

    The symlinks are necessary, as the test *must* be in control
    of describing which types of subjects, and thus Ansible inventory
    it supports.

    Some advanced tests (such as the Cockpit tests or IPA tests) would
    launch more complex local inventory and take control of what
    inventory is reported to Ansible.

On the spec side, I strongly believe that both of these things should
remain in the firm control of the tests stored in the dist-git repo.
While still having useful shared code to implement the spec.

Related: The ansible_connection=local nonsense in the current roles and
spec would be dropped.

I aim watch for discussion here on this topic for the next days or two,
and create a new wiki page with an updated spec after that:

https://fedoraproject.org/wiki/Changes/InvokingTestsAnsibleTwo