On Wed, 5 Mar 2014 20:58:32 +0900
"Sandro \"red\" Mathys" <red(a)fedoraproject.org> wrote:
Heads-up: I've taken ownership for the external need
Smoketests on Image Build" 
Testing an image takes time and resources, and with several images, it
takes several times that. Since that simply doesn't scale - well,
doesn't even work well without scaling - we want to automate the
smoketests using Taskotron. Unfortunately, Taskotron is still in a
very early development phase and having mentioned only just our (most)
basic needs, Tim estimates it won't be good enough for us in the
target timeframe (i.e. by F21 Alpha). There's a slight chance it will
be ready, but even then we still need time to implement the
So if we want to have Taskotron ready for us in time, we need to find
some developer(s) committing some of their resources. Any takers? If
you're interested, head to  to learn more, there's also a
contribution guide linked from there. I'm sure tflink in #fedora-qa is
happy to talk to people still interested after reading through the
wiki if there's further questions.
To go into a bit more detail, the taskotron bits which would be
required would depend a little bit on what comes out of the cloud image
I'm assuming the following:
- the image creation process will generate a fedmsg upon
completion that's relatively simple to differentiate from other
- this fedmsg contains some unique information which can be used to
obtain a path to the generated image
- the image is not already being loaded into a cloud-ish system
- some cloud-ish system (ie openstack) is available for testing
If my assumptions are correct, the scheduling component would be a
relatively simple change. Taskotron would need one or two new modules
to do the following:
- take the fedmsg information, upload the desired image into
the cloud-ish system and return the image id
- launch an instance with a specified image id, flavor and key id.
return the instance's IP address
There would be a complication here in that Taskotron doesn't have a
concept of shared secrets for the ssh private key or but I'm sure
we could figure something out.
Also, we need to compile our actual needs. What kind of tests do we
want to run? Most basic tests (things that we manually tested in the
past) are probably:
- Was the IP correctly set up?
- Is SSH running and reachable?
- Does the fedora user exist?
- Is the ssh pubkey installed for the fedora user?
- Is the fedora user able to use sudo?
- Is the root account locked?
- Is package installation working?
- Is the firewall disabled?
- Checking cloud-init and systemctl for bootup errors - does that make
sense or is there too much noise?
The easiest way to write a task to check stuff like this would be to
write it in python. Input would be a key id (see the shared secret
comment above) and a user id. The task would check the items that
you're interested in, collate and return the results. Taskotron would
then report results and emit a fedmsg indicating said result (fedmsg
code is yet to be completed but is slated for our first deliverable).
I realize that the documentation for how to do this is non-existent at
the moment. We've been focusing on getting the details figured out and
a staging system up and running before getting howtos written. We have
a trivial example written now (rpmlint) and will have a more
complicated example (depcheck) mostly complete before the end of next
week. While those are indeed package-level checks, they will have
examples of the task definition format (YAML) and the interface needed
for tasks written in python.
I hope this is enough detail to at least outline what would need to be
done. I'd love to see it done but I'm not sure what our (qa-devel)
priorities are going to be once the base Taskotron system is in place
and thus, not sure if it would otherwise be done before F21-alpha.
Please let me know if you have any questions, I'm happy to help where I