Automatic Smoketests for the Cloud Images: What to Test?
tflink at redhat.com
Fri Mar 7 01:44:29 UTC 2014
On Wed, 5 Mar 2014 20:58:32 +0900
"Sandro \"red\" Mathys" <red at fedoraproject.org> wrote:
> Heads-up: I've taken ownership for the external need "Automatic
> Smoketests on Image Build" 
> Testing an image takes time and resources, and with several images, it
> takes several times that. Since that simply doesn't scale - well,
> doesn't even work well without scaling - we want to automate the
> smoketests using Taskotron. Unfortunately, Taskotron is still in a
> very early development phase and having mentioned only just our (most)
> basic needs, Tim estimates it won't be good enough for us in the
> target timeframe (i.e. by F21 Alpha). There's a slight chance it will
> be ready, but even then we still need time to implement the
> So if we want to have Taskotron ready for us in time, we need to find
> some developer(s) committing some of their resources. Any takers? If
> you're interested, head to  to learn more, there's also a
> contribution guide linked from there. I'm sure tflink in #fedora-qa is
> happy to talk to people still interested after reading through the
> wiki if there's further questions.
To go into a bit more detail, the taskotron bits which would be
required would depend a little bit on what comes out of the cloud image
I'm assuming the following:
- the image creation process will generate a fedmsg upon
completion that's relatively simple to differentiate from other
- this fedmsg contains some unique information which can be used to
obtain a path to the generated image
- the image is not already being loaded into a cloud-ish system
- some cloud-ish system (ie openstack) is available for testing
If my assumptions are correct, the scheduling component would be a
relatively simple change. Taskotron would need one or two new modules
to do the following:
- take the fedmsg information, upload the desired image into
the cloud-ish system and return the image id
- launch an instance with a specified image id, flavor and key id.
return the instance's IP address
There would be a complication here in that Taskotron doesn't have a
concept of shared secrets for the ssh private key or but I'm sure
we could figure something out.
> Also, we need to compile our actual needs. What kind of tests do we
> want to run? Most basic tests (things that we manually tested in the
> past) are probably:
> - Was the IP correctly set up?
> - Is SSH running and reachable?
> - Does the fedora user exist?
> - Is the ssh pubkey installed for the fedora user?
> - Is the fedora user able to use sudo?
> - Is the root account locked?
> - Is package installation working?
> - Is the firewall disabled?
> - Checking cloud-init and systemctl for bootup errors - does that make
> sense or is there too much noise?
The easiest way to write a task to check stuff like this would be to
write it in python. Input would be a key id (see the shared secret
comment above) and a user id. The task would check the items that
you're interested in, collate and return the results. Taskotron would
then report results and emit a fedmsg indicating said result (fedmsg
code is yet to be completed but is slated for our first deliverable).
I realize that the documentation for how to do this is non-existent at
the moment. We've been focusing on getting the details figured out and
a staging system up and running before getting howtos written. We have
a trivial example written now (rpmlint) and will have a more
complicated example (depcheck) mostly complete before the end of next
week. While those are indeed package-level checks, they will have
examples of the task definition format (YAML) and the interface needed
for tasks written in python.
I hope this is enough detail to at least outline what would need to be
done. I'd love to see it done but I'm not sure what our (qa-devel)
priorities are going to be once the base Taskotron system is in place
and thus, not sure if it would otherwise be done before F21-alpha.
Please let me know if you have any questions, I'm happy to help where I
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 490 bytes
Desc: not available
More information about the cloud