Automatic Smoketests for the Cloud Images: What to Test?

Vitaly Kuznetsov vitty at
Fri Mar 7 10:32:18 UTC 2014

"Sandro \"red\" Mathys" <red at> writes:

> On Fri, Mar 7, 2014 at 6:37 PM, Vitaly Kuznetsov <vitty at> wrote:
>> "Sandro \"red\" Mathys" <red at> writes:
>>> Heads-up: I've taken ownership for the external need "Automatic
>>> Smoketests on Image Build" [0]
>>> Testing an image takes time and resources, and with several images, it
>>> takes several times that. Since that simply doesn't scale - well,
>>> doesn't even work well without scaling - we want to automate the
>>> smoketests using Taskotron. Unfortunately, Taskotron is still in a
>>> very early development phase and having mentioned only just our (most)
>>> basic needs, Tim estimates it won't be good enough for us in the
>>> target timeframe (i.e. by F21 Alpha). There's a slight chance it will
>>> be ready, but even then we still need time to implement the
>>> smoketests.
>> I just want to point out the fact that for RHEL/Fedora/etc image testing
>> we have the following tool:
>> We were using it to test Fedora images since F18.
>> To be honest I'm not that familiar with Taskotron but maybe we can
>> collaborate somehow.
>>> So if we want to have Taskotron ready for us in time, we need to find
>>> some developer(s) committing some of their resources. Any takers? If
>>> you're interested, head to [1] to learn more, there's also a
>>> contribution guide linked from there. I'm sure tflink in #fedora-qa is
>>> happy to talk to people still interested after reading through the
>>> wiki if there's further questions.
>>> Also, we need to compile our actual needs. What kind of tests do we
>>> want to run? Most basic tests (things that we manually tested in the
>>> past) are probably:
>>> - Was the IP correctly set up?
>>> - Is SSH running and reachable?
>>> - Does the fedora user exist?
>>> - Is the ssh pubkey installed for the fedora user?
>>> - Is the fedora user able to use sudo?
>>> - Is the root account locked?
>>> - Is package installation working?
>>> - Is the firewall disabled?
>>> - Checking cloud-init and systemctl for bootup errors - does that make
>>> sense or is there too much noise?
>>> What else can / should we test? Obviously the tailored images should
>>> be tested for their specific stuff but lets first focus on the common
>>> ground.
>> In addition to that current image validation has the following:
>> - testing different instace types
>> - empty bash history
>> - selinux contexts test
>> - running services
>> - package set
>> - EBS test
>> - Ephemeral test
>> - available cores/memory test
>> - userdata handling (cloud-init)
>> - several bugs test (regressions)
>> - ... (full test list is here:
>>> Thanks,
>>> Sandro
>>> [0]
>>> [1]
> So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone
> of the people involved (at the Red Hat side, I guess) well me why we
> have 3 systems for 1 task? 

(my personal opinion) I think we rather have plenty of tasks, not
one. Afaict (after 5 min. of reading Taskotron's development plan
Taskotron is designed to replace AutoQA in the first place. 
RHEL's Cloud Image Validation was developed several years ago when the
following task was on the table: we have many AWS regions, many images,
different architectures, we need to try different hardware types and
AWS-specific features (e.g. attach EBS on the fly or test AWS-specific
content delivery) and finally we need to aggregate the result. Existing
test infrastructure was built around Beaker which is not that well
suited for the job and creating a separate tool was considered a
reasonable trade-off.

> When I took ownership of this "external
> need" (for the Fedora cloud product) I was under the impression we
> only just (are going to) have Taskotron and everyone knows it's THE
> way to go.

I personally love collaboration. It would be awesome if we could avoid
spreading resources on '3 systems for 1 task'. I definitely want to know
more about Taskotron and its movement towards cloud image testing.

  Vitaly Kuznetsov

More information about the cloud mailing list