(I posted the message below on the Fedora LiveCD list and it was suggested that I post the question on this 'test' list as well.)
Hello,
Has anyone set up a continuous build and automated testing system for LiveCD development? I am in the process of creating one and would like to compare notes and discuss approaches. The objective is to perform regression testing on LiveCDs where all configuration is specified in a kickstart file. We must be able to capture the boot messages as part of this testing.
I have the continuous build portion working and am now starting on the automated testing piece. Here is an activity diagram for the build/test phases (note that the testing phase has not yet been implemented): http://jaroker.com/EA4.png (You can ignore the "wake up slave" and "power off slave". We have a noisy server dedicated to the build. Since it is only run 1-2 times per day, the build process turns the server on/off as needed.)
Here is what I am trying to achieve:
- LiveCD is customized via a kickstart file and this kickstart is committed to SVN
- Build server will connect to a workstation (not a virtual server due to loopback restrictions) and check out kickstart
- LiveCD scripts are run on the workstation
- Kernel files are created and custom applications are loaded into the file structure
- ISO image is created
- Qemu is used to boot the ISO image, but where the console is redirected to a TTY
- An Expect script telnets into Qemu to start the boot and the Expect script captures boot messages to a text file (We have this Expect process working and it seems to be stable.)
- Boot messages are analyzed using another script for errors and warnings
- Selenium grid is used to test the boot disk (running under Qemu) by simulating user activity to run regression tests on the LiveCD
- If the original kickstart passes all tests, then the kernel files are committed to SVN as artifacts
Please let me know if anyone has experience doing this, or can recommend simplifications to regression testing. Most importantly, please tell me if I may have overlooked some existing testing framework for LiveCDs. I would hate to do this work only to realize that it has already been done.
Thank you,
Jon