Automatic Smoketests for the Cloud Images: What to Test?

[Sandro "red" Mathys]

On Fri, Mar 7, 2014 at 7:32 PM, Vitaly Kuznetsov <vitty at redhat.com> wrote:
>> So we have the RedHatQE tests, Taskotron and CentOS's CI. Can anyone
>> of the people involved (at the Red Hat side, I guess) well me why we
>> have 3 systems for 1 task?
>
> (my personal opinion) I think we rather have plenty of tasks, not
> one. Afaict (after 5 min. of reading Taskotron's development plan
> https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan)
> Taskotron is designed to replace AutoQA in the first place.
> RHEL's Cloud Image Validation was developed several years ago when the
> following task was on the table: we have many AWS regions, many images,
> different architectures, we need to try different hardware types and
> AWS-specific features (e.g. attach EBS on the fly or test AWS-specific
> content delivery) and finally we need to aggregate the result. Existing
> test infrastructure was built around Beaker which is not that well
> suited for the job and creating a separate tool was considered a
> reasonable trade-off.

Well, "one" task as in "do cloud image QA".

Thanks, for sharing that insights, really helpful to help my
understanding. So, do you currently test EC2 only? (Not saying that's
necessarily bad / too little).

Now, we do have the RHQE stuff in place and it's already used for
testing Fedora images...that's good. Is that fully automated? Or to
what extend?

>> When I took ownership of this "external
>> need" (for the Fedora cloud product) I was under the impression we
>> only just (are going to) have Taskotron and everyone knows it's THE
>> way to go.
>
> I personally love collaboration. It would be awesome if we could avoid
> spreading resources on '3 systems for 1 task'. I definitely want to know
> more about Taskotron and its movement towards cloud image testing.

That's why I was a bit confused to find there's actually 3 systems.
Collaboration is certainly great, but that's not how it's done so
let's try to improve on this.

So, would you recommend to keep using your tools or rather go with
Taskotron? Or do we do some things in one and others in the other? Or
do we try to fully implement your tests in Taskotron and drop doing
the tests with your tools?

Also, Karanbir, what's your (i.e. CentOS's) story? You say you already
have a CI system running but shared little other information. What CI
system? Did you already implement image tests? What kind of
collaboration would you suggest here?

-- Sandro