Automatic Smoketests for the Cloud Images: What to Test?

[Tim Flink]

On Wed, 5 Mar 2014 20:58:32 +0900
"Sandro \"red\" Mathys" <red at fedoraproject.org> wrote:

> Heads-up: I've taken ownership for the external need "Automatic
> Smoketests on Image Build" [0]
> 
> Testing an image takes time and resources, and with several images, it
> takes several times that. Since that simply doesn't scale - well,
> doesn't even work well without scaling - we want to automate the
> smoketests using Taskotron. Unfortunately, Taskotron is still in a
> very early development phase and having mentioned only just our (most)
> basic needs, Tim estimates it won't be good enough for us in the
> target timeframe (i.e. by F21 Alpha). There's a slight chance it will
> be ready, but even then we still need time to implement the
> smoketests.
> 
> So if we want to have Taskotron ready for us in time, we need to find
> some developer(s) committing some of their resources. Any takers? If
> you're interested, head to [1] to learn more, there's also a
> contribution guide linked from there. I'm sure tflink in #fedora-qa is
> happy to talk to people still interested after reading through the
> wiki if there's further questions.

To go into a bit more detail, the taskotron bits which would be
required would depend a little bit on what comes out of the cloud image
generation.

I'm assuming the following:
 - the image creation process will generate a fedmsg upon
   completion that's relatively simple to differentiate from other
   fedmsgs.
 - this fedmsg contains some unique information which can be used to
   obtain a path to the generated image
 - the image is not already being loaded into a cloud-ish system
 - some cloud-ish system (ie openstack) is available for testing

If my assumptions are correct, the scheduling component would be a
relatively simple change. Taskotron would need one or two new modules
to do the following:

 - take the fedmsg information, upload the desired image into
   the cloud-ish system and return the image id

 - launch an instance with a specified image id, flavor and key id.
   return the instance's IP address

There would be a complication here in that Taskotron doesn't have a
concept of shared secrets for the ssh private key or but I'm sure
we could figure something out.

> Also, we need to compile our actual needs. What kind of tests do we
> want to run? Most basic tests (things that we manually tested in the
> past) are probably:
> - Was the IP correctly set up?
> - Is SSH running and reachable?
> - Does the fedora user exist?
> - Is the ssh pubkey installed for the fedora user?
> - Is the fedora user able to use sudo?
> - Is the root account locked?
> - Is package installation working?
> - Is the firewall disabled?
> - Checking cloud-init and systemctl for bootup errors - does that make
> sense or is there too much noise?

The easiest way to write a task to check stuff like this would be to
write it in python. Input would be a key id (see the shared secret
comment above) and a user id. The task would check the items that
you're interested in, collate and return the results. Taskotron would
then report results and emit a fedmsg indicating said result (fedmsg
code is yet to be completed but is slated for our first deliverable).

I realize that the documentation for how to do this is non-existent at
the moment. We've been focusing on getting the details figured out and
a staging system up and running before getting howtos written. We have
a trivial example written now (rpmlint) and will have a more
complicated example (depcheck) mostly complete before the end of next
week. While those are indeed package-level checks, they will have
examples of the task definition format (YAML) and the interface needed
for tasks written in python.

I hope this is enough detail to at least outline what would need to be
done. I'd love to see it done but I'm not sure what our (qa-devel)
priorities are going to be once the base Taskotron system is in place
and thus, not sure if it would otherwise be done before F21-alpha.

Please let me know if you have any questions, I'm happy to help where I
can.

Tim
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/cloud/attachments/20140306/fecf72b8/attachment.sig>