(I posted the message below on the Fedora LiveCD list and it was suggested that I post the question on this 'test' list as well.)
Hello,
Has anyone set up a continuous build and automated testing system for LiveCD development? I am in the process of creating one and would like to compare notes and discuss approaches. The objective is to perform regression testing on LiveCDs where all configuration is specified in a kickstart file. We must be able to capture the boot messages as part of this testing.
I have the continuous build portion working and am now starting on the automated testing piece. Here is an activity diagram for the build/test phases (note that the testing phase has not yet been implemented): http://jaroker.com/EA4.png (You can ignore the "wake up slave" and "power off slave". We have a noisy server dedicated to the build. Since it is only run 1-2 times per day, the build process turns the server on/off as needed.)
Here is what I am trying to achieve:
- LiveCD is customized via a kickstart file and this kickstart is committed to SVN
- Build server will connect to a workstation (not a virtual server due to loopback restrictions) and check out kickstart
- LiveCD scripts are run on the workstation
- Kernel files are created and custom applications are loaded into the file structure
- ISO image is created
- Qemu is used to boot the ISO image, but where the console is redirected to a TTY
- An Expect script telnets into Qemu to start the boot and the Expect script captures boot messages to a text file (We have this Expect process working and it seems to be stable.)
- Boot messages are analyzed using another script for errors and warnings
- Selenium grid is used to test the boot disk (running under Qemu) by simulating user activity to run regression tests on the LiveCD
- If the original kickstart passes all tests, then the kernel files are committed to SVN as artifacts
Please let me know if anyone has experience doing this, or can recommend simplifications to regression testing. Most importantly, please tell me if I may have overlooked some existing testing framework for LiveCDs. I would hate to do this work only to realize that it has already been done.
Thank you,
Jon
On Thu, 2010-02-11 at 16:48 -0500, Jon Jaroker wrote:
(I posted the message below on the Fedora LiveCD list and it was suggested that I post the question on this ‘test’ list as well.)
Hello,
Has anyone set up a continuous build and automated testing system for LiveCD development? I am in the process of creating one and would like to compare notes and discuss approaches. The objective is to perform regression testing on LiveCDs where all configuration is specified in a kickstart file. We must be able to capture the boot messages as part of this testing.
Please let me know if anyone has experience doing this, or can recommend simplifications to regression testing. Most importantly, please tell me if I may have overlooked some existing testing framework for LiveCDs. I would hate to do this work only to realize that it has already been done.
This looks fantastic!
We certainly don't have anything that comprehensive in place at present. What automated testing we have focuses on the traditional installer path, not the live CD. I know Adam Miller is interested in doing some automated testing within the graphical desktop (Xfce in his case) and was interested in automated live image *installation* as a precursor to that, but this is somewhat different from what you're doing.
I think this is really valuable work and what would be most awesome is if it could be integrated into the AutoQA framework somehow. AutoQA - https://fedoraproject.org/wiki/AutoQA is a framework for managing the actual day-to-day running and result generation/storage/management of automated tests, so what you're doing isn't replicating work on the AutoQA framework itself, and it'd be great to have your work within AutoQA's arsenal of tests, I think. Will Woods or Kamil Paral can tell you more about AutoQA, on IRC or via its mailing list - https://fedorahosted.org/mailman/listinfo/autoqa-devel .
-
Adam Williamson -
Jon Jaroker