Automatic Smoketests for the Cloud Images: What to Test?
Sandro "red" Mathys
red at fedoraproject.org
Fri Mar 7 09:46:49 UTC 2014
On Fri, Mar 7, 2014 at 6:37 PM, Vitaly Kuznetsov <vitty at redhat.com> wrote:
> "Sandro \"red\" Mathys" <red at fedoraproject.org> writes:
>> Heads-up: I've taken ownership for the external need "Automatic
>> Smoketests on Image Build" 
>> Testing an image takes time and resources, and with several images, it
>> takes several times that. Since that simply doesn't scale - well,
>> doesn't even work well without scaling - we want to automate the
>> smoketests using Taskotron. Unfortunately, Taskotron is still in a
>> very early development phase and having mentioned only just our (most)
>> basic needs, Tim estimates it won't be good enough for us in the
>> target timeframe (i.e. by F21 Alpha). There's a slight chance it will
>> be ready, but even then we still need time to implement the
> I just want to point out the fact that for RHEL/Fedora/etc image testing
> we have the following tool:
> We were using it to test Fedora images since F18.
> To be honest I'm not that familiar with Taskotron but maybe we can
> collaborate somehow.
>> So if we want to have Taskotron ready for us in time, we need to find
>> some developer(s) committing some of their resources. Any takers? If
>> you're interested, head to  to learn more, there's also a
>> contribution guide linked from there. I'm sure tflink in #fedora-qa is
>> happy to talk to people still interested after reading through the
>> wiki if there's further questions.
>> Also, we need to compile our actual needs. What kind of tests do we
>> want to run? Most basic tests (things that we manually tested in the
>> past) are probably:
>> - Was the IP correctly set up?
>> - Is SSH running and reachable?
>> - Does the fedora user exist?
>> - Is the ssh pubkey installed for the fedora user?
>> - Is the fedora user able to use sudo?
>> - Is the root account locked?
>> - Is package installation working?
>> - Is the firewall disabled?
>> - Checking cloud-init and systemctl for bootup errors - does that make
>> sense or is there too much noise?
>> What else can / should we test? Obviously the tailored images should
>> be tested for their specific stuff but lets first focus on the common
> In addition to that current image validation has the following:
> - testing different instace types
> - empty bash history
> - selinux contexts test
> - running services
> - package set
> - EBS test
> - Ephemeral test
> - available cores/memory test
> - userdata handling (cloud-init)
> - several bugs test (regressions)
> - ... (full test list is here: https://github.com/RedHatQE/valid/tree/master/valid/testing_modules)
>>  https://fedoraproject.org/wiki/Cloud_Changelist#External_Need:_Automatic_Smoketests_on_Image_Build
>>  https://fedoraproject.org/wiki/Taskotron
So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone
of the people involved (at the Red Hat side, I guess) well me why we
have 3 systems for 1 task? When I took ownership of this "external
need" (for the Fedora cloud product) I was under the impression we
only just (are going to) have Taskotron and everyone knows it's THE
way to go.
More information about the cloud