Automatic Smoketests for the Cloud Images: What to Test?

[Vitaly Kuznetsov]

"Sandro \"red\" Mathys" <red at fedoraproject.org> writes:

> On Fri, Mar 7, 2014 at 8:20 PM, Vitaly Kuznetsov <vitty at redhat.com> wrote:
>> "Sandro \"red\" Mathys" <red at fedoraproject.org> writes:
>>
>>> On Fri, Mar 7, 2014 at 7:32 PM, Vitaly Kuznetsov <vitty at redhat.com> wrote:
>>>>> So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone
>>>>> of the people involved (at the Red Hat side, I guess) well me why we
>>>>> have 3 systems for 1 task?
>>>>
>>>> (my personal opinion) I think we rather have plenty of tasks, not
>>>> one. Afaict (after 5 min. of reading Taskotron's development plan
>>>> https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan)
>>>> Taskotron is designed to replace AutoQA in the first place.
>>>> RHEL's Cloud Image Validation was developed several years ago when the
>>>> following task was on the table: we have many AWS regions, many images,
>>>> different architectures, we need to try different hardware types and
>>>> AWS-specific features (e.g. attach EBS on the fly or test AWS-specific
>>>> content delivery) and finally we need to aggregate the result. Existing
>>>> test infrastructure was built around Beaker which is not that well
>>>> suited for the job and creating a separate tool was considered a
>>>> reasonable trade-off.
>>>
>>> Well, "one" task as in "do cloud image QA".
>>>
>>> Thanks, for sharing that insights, really helpful to help my
>>> understanding. So, do you currently test EC2 only? (Not saying that's
>>> necessarily bad / too little).
>>
>> Now it is EC2-only but Google's ComputeEngine was on the horizon.
>>
>>>
>>> Now, we do have the RHQE stuff in place and it's already used for
>>> testing Fedora images...that's good. Is that fully automated? Or to
>>> what extend?
>>
>> You run the tool with the data (AMI IDs, region, arch) and get the
>> result in a meanwhile. It can be fully-automated once we have this data
>> announced via fedmsg or in any other automated way (now I just read
>> mailing list and if there are any images announced by Dennis I run the
>> tool).
>>
>>>
>>>>> When I took ownership of this "external
>>>>> need" (for the Fedora cloud product) I was under the impression we
>>>>> only just (are going to) have Taskotron and everyone knows it's THE
>>>>> way to go.
>>>>
>>>> I personally love collaboration. It would be awesome if we could avoid
>>>> spreading resources on '3 systems for 1 task'. I definitely want to know
>>>> more about Taskotron and its movement towards cloud image testing.
>>>
>>> That's why I was a bit confused to find there's actually 3 systems.
>>> Collaboration is certainly great, but that's not how it's done so
>>> let's try to improve on this.
>>>
>>> So, would you recommend to keep using your tools or rather go with
>>> Taskotron? Or do we do some things in one and others in the other? Or
>>> do we try to fully implement your tests in Taskotron and drop doing
>>> the tests with your tools?
>>>
>>
>> Well, it depends on what's our future plan. IMHO once we have images
>> announced via fedmsg we can have all basic things covered by the existing
>> tool (and I'm definitely in for integration and support process for the
>> tool) and it won't take us long to set everything up. With regards to
>> Taskotron I want to know more on how this 'cloud integration' is planned
>> as (if I'm not mistaken) there's no code written yet. If merging here
>> seems reasonable then I'm in. I'll try reaching out to Tim & others on
>> fedora-qa-devel list.
>
> So, what's the status here? Tim's responses to this thread show no
> cloud integration code has been written yet and he's open to have
> valid integrated in Taskotron, particularly if helping hands do most
> of the work so he can keep focusing on other open tasks. Could you
> work on that, Vitaly?
>

I'm still willing to but I have to confess this task ('Take a look at
Taskotron and figure out how me can merge / collaborate / integrate /
whatever') is still in my backlog.  I'll try picking this up next week,
feel free to ping me on irc (vitty) any time.

>>> Also, Karanbir, what's your (i.e. CentOS's) story? You say you already
>>> have a CI system running but shared little other information. What CI
>>> system? Did you already implement image tests? What kind of
>>> collaboration would you suggest here?

-- 
  Vitaly Kuznetsov