Instalatron: Anaconda testing framework

[Sergio Rubio]

Hey,

Thank you guys for the quick reply. Comments inline.

On Tue, Jul 19, 2011 at 6:48 PM, Tim Flink <tflink at redhat.com> wrote:
>
> Cool, we're always looking for ways to do better testing. As James said,
> honqing and twu are more actively working on the installer automation
> but I have a couple of design questions.
>
> Why virtualbox?
>  - I was under the impression that VB didn't always play nice with the
>   fedora kernels and was a reason to use KVM/qemu instead
>  - Then there's the whole licensing issue ...


The main reason is that I found the "VBoxManage controlvm
keyboardputscancode" stuff in another OSS project and since anaconda
installs can be driven by the keyboard I thought wrapping it would be
fast and easy.

Besides that, we test RHEL5/6 and Fedora VMs in VirtualBox everyday
and never had an issue (at least for testing purposes).

About the licensing issue, we don't need the propietary extensions in
VB 4 so don't care too much.


>
> How well does it adapt to small changes in visual layout?
>  - The scripting looks very rigid to me which is fine for things that
>   don't change very often
>  - Do the scripts have to be completely re-done for small visual
>   changes or is it possible to just change a portion?


You can adjust the "sentitivity". I'm using compare from ImageMagick
that allows this kind of stuff. There's a threshold you can adjust,
but right now it's hardcoded. Things like mouse movements and string
changes are Ok.


>
> How often are you seeing false positives/negatives?
>  - ImageMagick's 'compare' seems awfully sensitive to me, have you seen
>   many problems with the image comparison logic?


We've changed strings, selected and highlighted widgets and the mouse
cursor and the test is not affected. It's farm from ideal but it works
right now :).


>  - If I'm reading the code correctly, it seems to be looking for an
>   exact match

Right, with the exceptions mentioned above.

>
> How are you defining pass/fail?
>  - From my very quick scan of the code, it looks like
>   PASS -> no crash during installation
>   FAIL -> image comparison failures, unexpected events, crash
>  - Is there any verification that the VM boots after install?


I still need to implement this stuff, but I believe we can do it easily.
Right now instalatron-play hangs if the next screen (or crash) is not expected.


>
> How do you handle crashes?
>  - Are you collecting anaconda stack traces on crash?
>  - Otherwise, is the VM in a state that allows for data collection if a
>   crash occurs?

Not handled right now. I'm planing research libguestfs usage to access
vm disk... Not sure yet.

>
> Another thing that you may be interested in is OpenSuSE's openqa project
> [1] and specifically, os-autoinst [2].

Greatly appreciated!

Lovely feedback, thank you.

>
> Thanks,
>
> Tim
>
> [1] http://openqa.opensuse.org/
> [2] http://www.os-autoinst.org/
>
>
> --
> test mailing list
> test at lists.fedoraproject.org
> To unsubscribe:
> https://admin.fedoraproject.org/mailman/listinfo/test
>