Automatic Smoketests for the Cloud Images: What to Test?

[Tim Flink]

On Fri, 7 Mar 2014 18:46:49 +0900
"Sandro \"red\" Mathys" <red at fedoraproject.org> wrote:

<snip>

> So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone
> of the people involved (at the Red Hat side, I guess) well me why we
> have 3 systems for 1 task? When I took ownership of this "external
> need" (for the Fedora cloud product) I was under the impression we
> only just (are going to) have Taskotron and everyone knows it's THE
> way to go.

Honestly, I think you're comparing apples and oranges here with
Taskotron and valid. On the one hand you have systems like Taskotron
which are (or will/can be) capable of running any number of things from
package-level checks like rpmlint or depcheck, cloud image checks of
the type that we're discussing here, graphical installation media
checks with an anaconda install image, interface with beaker to run
various other checks, run static analysis on SRPMs and just about
anything else that you can think of.

On the other hand, you have valid. As far as I can tell, it was
designed to do one thing (test cloud images) and do that one thing
well. I'm not saying there's anything wrong with that but saying that
the two systems do the same thing isn't really accurate.

As far as CentOS's CI system goes, I don't claim to know much about it.
It sounds similar to what we're planning for Taskotron but I wasn't
able to find many details about how it manages reporting, how jobs are
triggered or what the interface between the tests and the jenkins
runner is.

At its core, Taskotron is a system for triggering somewhat-arbitrary
tasks, executing those tasks and reporting the generated output. Our
goal is to provide a framework with which Fedora developers can
automate tasks to be run in specific situations. Is it capable of
everything right now? No, it isn't. We're in the process of a major
re-write and working off about 2 years of technical debt so as much as
I'd like reality to be different, this is going to take some time
and/or external help to get new features added and working.

I'm not dead set on writing a bunch of custom code for running cloud
image checks. In fact, I'd rather not do that if we don't have to. If it
makes more sense to run RH QE's cloud stuff, then that sounds like a
decent integration point with Taskotron unless there is other HW
available for running them. I'm still assuming that there is no
existing code for getting the generated cloud images from Koji into a
runnable state which we can use (unless someone's got a bunch of usable
EC2 credit that I don't know about), so some work will still be needed
on that front.

I purposely left the part of actually running any individual checks
open-ended because I assume that you all will come up with some method
for checking the things that you're interested in. If you all decide
that you want to use valid for cloud image checks, let's start looking
at what it would take to get that integrated.

Tim
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/cloud/attachments/20140310/8c5c7913/attachment.sig>