Automatic Smoketests for the Cloud Images: What to Test?

[Vitaly Kuznetsov]

"Sandro \"red\" Mathys" <red at fedoraproject.org> writes:

> On Fri, Mar 7, 2014 at 7:32 PM, Vitaly Kuznetsov <vitty at redhat.com> wrote:
>>> So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone
>>> of the people involved (at the Red Hat side, I guess) well me why we
>>> have 3 systems for 1 task?
>>
>> (my personal opinion) I think we rather have plenty of tasks, not
>> one. Afaict (after 5 min. of reading Taskotron's development plan
>> https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan)
>> Taskotron is designed to replace AutoQA in the first place.
>> RHEL's Cloud Image Validation was developed several years ago when the
>> following task was on the table: we have many AWS regions, many images,
>> different architectures, we need to try different hardware types and
>> AWS-specific features (e.g. attach EBS on the fly or test AWS-specific
>> content delivery) and finally we need to aggregate the result. Existing
>> test infrastructure was built around Beaker which is not that well
>> suited for the job and creating a separate tool was considered a
>> reasonable trade-off.
>
> Well, "one" task as in "do cloud image QA".
>
> Thanks, for sharing that insights, really helpful to help my
> understanding. So, do you currently test EC2 only? (Not saying that's
> necessarily bad / too little).

Now it is EC2-only but Google's ComputeEngine was on the horizon.

>
> Now, we do have the RHQE stuff in place and it's already used for
> testing Fedora images...that's good. Is that fully automated? Or to
> what extend?

You run the tool with the data (AMI IDs, region, arch) and get the
result in a meanwhile. It can be fully-automated once we have this data
announced via fedmsg or in any other automated way (now I just read
mailing list and if there are any images announced by Dennis I run the
tool).

>
>>> When I took ownership of this "external
>>> need" (for the Fedora cloud product) I was under the impression we
>>> only just (are going to) have Taskotron and everyone knows it's THE
>>> way to go.
>>
>> I personally love collaboration. It would be awesome if we could avoid
>> spreading resources on '3 systems for 1 task'. I definitely want to know
>> more about Taskotron and its movement towards cloud image testing.
>
> That's why I was a bit confused to find there's actually 3 systems.
> Collaboration is certainly great, but that's not how it's done so
> let's try to improve on this.
>
> So, would you recommend to keep using your tools or rather go with
> Taskotron? Or do we do some things in one and others in the other? Or
> do we try to fully implement your tests in Taskotron and drop doing
> the tests with your tools?
>

Well, it depends on what's our future plan. IMHO once we have images
announced via fedmsg we can have all basic things covered by the existing
tool (and I'm definitely in for integration and support process for the
tool) and it won't take us long to set everything up. With regards to
Taskotron I want to know more on how this 'cloud integration' is planned
as (if I'm not mistaken) there's no code written yet. If merging here
seems reasonable then I'm in. I'll try reaching out to Tim & others on
fedora-qa-devel list.

> Also, Karanbir, what's your (i.e. CentOS's) story? You say you already
> have a CI system running but shared little other information. What CI
> system? Did you already implement image tests? What kind of
> collaboration would you suggest here?
>
> -- Sandro
> _______________________________________________
> cloud mailing list
> cloud at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/cloud
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

-- 
  Vitaly Kuznetsov