Automatic Smoketests for the Cloud Images: What to Test?

[Vitaly Kuznetsov]

"Sandro \"red\" Mathys" <red at fedoraproject.org> writes:

> On Fri, Mar 7, 2014 at 6:37 PM, Vitaly Kuznetsov <vitty at redhat.com> wrote:
>> "Sandro \"red\" Mathys" <red at fedoraproject.org> writes:
>>
>>> Heads-up: I've taken ownership for the external need "Automatic
>>> Smoketests on Image Build" [0]
>>>
>>> Testing an image takes time and resources, and with several images, it
>>> takes several times that. Since that simply doesn't scale - well,
>>> doesn't even work well without scaling - we want to automate the
>>> smoketests using Taskotron. Unfortunately, Taskotron is still in a
>>> very early development phase and having mentioned only just our (most)
>>> basic needs, Tim estimates it won't be good enough for us in the
>>> target timeframe (i.e. by F21 Alpha). There's a slight chance it will
>>> be ready, but even then we still need time to implement the
>>> smoketests.
>>
>> I just want to point out the fact that for RHEL/Fedora/etc image testing
>> we have the following tool:
>>
>> https://github.com/RedHatQE/valid
>>
>> We were using it to test Fedora images since F18.
>>
>> To be honest I'm not that familiar with Taskotron but maybe we can
>> collaborate somehow.
>>
>>>
>>> So if we want to have Taskotron ready for us in time, we need to find
>>> some developer(s) committing some of their resources. Any takers? If
>>> you're interested, head to [1] to learn more, there's also a
>>> contribution guide linked from there. I'm sure tflink in #fedora-qa is
>>> happy to talk to people still interested after reading through the
>>> wiki if there's further questions.
>>>
>>> Also, we need to compile our actual needs. What kind of tests do we
>>> want to run? Most basic tests (things that we manually tested in the
>>> past) are probably:
>>> - Was the IP correctly set up?
>>> - Is SSH running and reachable?
>>> - Does the fedora user exist?
>>> - Is the ssh pubkey installed for the fedora user?
>>> - Is the fedora user able to use sudo?
>>> - Is the root account locked?
>>> - Is package installation working?
>>> - Is the firewall disabled?
>>> - Checking cloud-init and systemctl for bootup errors - does that make
>>> sense or is there too much noise?
>>>
>>> What else can / should we test? Obviously the tailored images should
>>> be tested for their specific stuff but lets first focus on the common
>>> ground.
>>>
>>
>> In addition to that current image validation has the following:
>> - testing different instace types
>> - empty bash history
>> - selinux contexts test
>> - running services
>> - package set
>> - EBS test
>> - Ephemeral test
>> - available cores/memory test
>> - userdata handling (cloud-init)
>> - several bugs test (regressions)
>> - ... (full test list is here: https://github.com/RedHatQE/valid/tree/master/valid/testing_modules)
>>
>>> Thanks,
>>> Sandro
>>>
>>> [0] https://fedoraproject.org/wiki/Cloud_Changelist#External_Need:_Automatic_Smoketests_on_Image_Build
>>> [1] https://fedoraproject.org/wiki/Taskotron
>
> So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone
> of the people involved (at the Red Hat side, I guess) well me why we
> have 3 systems for 1 task? 

(my personal opinion) I think we rather have plenty of tasks, not
one. Afaict (after 5 min. of reading Taskotron's development plan
https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan)
Taskotron is designed to replace AutoQA in the first place. 
RHEL's Cloud Image Validation was developed several years ago when the
following task was on the table: we have many AWS regions, many images,
different architectures, we need to try different hardware types and
AWS-specific features (e.g. attach EBS on the fly or test AWS-specific
content delivery) and finally we need to aggregate the result. Existing
test infrastructure was built around Beaker which is not that well
suited for the job and creating a separate tool was considered a
reasonable trade-off.

> When I took ownership of this "external
> need" (for the Fedora cloud product) I was under the impression we
> only just (are going to) have Taskotron and everyone knows it's THE
> way to go.

I personally love collaboration. It would be awesome if we could avoid
spreading resources on '3 systems for 1 task'. I definitely want to know
more about Taskotron and its movement towards cloud image testing.

-- 
  Vitaly Kuznetsov