Instalatron: Anaconda testing framework

Tim Flink tflink at redhat.com
Tue Jul 19 16:48:31 UTC 2011


On 07/19/2011 09:49 AM, Sergio Rubio wrote:
> Greetings!
> 
> Posted this to the wrong mailing list (David Cantrell suggested using
> this one instead).
> 
> Linking to avoid re-typing :), my apologies:
> 
> https://www.redhat.com/archives/anaconda-devel-list/2011-July/msg00090.html
> 
> In a nutshell, we are looking for better ways to test our Anaconda
> based installer and I thought that speaking to the real guys and
> sharing would be a good way to start.
> 
> Any feedback is greatly appreciated.

Cool, we're always looking for ways to do better testing. As James said,
honqing and twu are more actively working on the installer automation
but I have a couple of design questions.

Why virtualbox?
 - I was under the impression that VB didn't always play nice with the
   fedora kernels and was a reason to use KVM/qemu instead
 - Then there's the whole licensing issue ...

How well does it adapt to small changes in visual layout?
 - The scripting looks very rigid to me which is fine for things that
   don't change very often
 - Do the scripts have to be completely re-done for small visual
   changes or is it possible to just change a portion?

How often are you seeing false positives/negatives?
 - ImageMagick's 'compare' seems awfully sensitive to me, have you seen
   many problems with the image comparison logic?
 - If I'm reading the code correctly, it seems to be looking for an
   exact match

How are you defining pass/fail?
 - From my very quick scan of the code, it looks like
   PASS -> no crash during installation
   FAIL -> image comparison failures, unexpected events, crash
 - Is there any verification that the VM boots after install?

How do you handle crashes?
 - Are you collecting anaconda stack traces on crash?
 - Otherwise, is the VM in a state that allows for data collection if a
   crash occurs?

Another thing that you may be interested in is OpenSuSE's openqa project
[1] and specifically, os-autoinst [2].

Thanks,

Tim

[1] http://openqa.opensuse.org/
[2] http://www.os-autoinst.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 553 bytes
Desc: OpenPGP digital signature
Url : http://lists.fedoraproject.org/pipermail/test/attachments/20110719/29bcaa1c/attachment.bin 


More information about the test mailing list