new project: taskotron-vmbuilder

[Kamil Paral]

Related to our disposable clients initiative in Taskotron, I'm working on ways how to boot and manage those clients. In order to know how to boot them, I need to know how they are created (e.g. cloud images require special magic with cloud-init). That's why I spent some time in the past looking at different ways how to either create custom images or re-use existing ones. The simplest approach turned out to be (in my view) a tool called virt-builder. It's a part of libguestfs project and it's a super easy way to create Fedora disk images - they have them already pre-created and stored on the server, so it just downloads them and customize them a little bit per your requirements:

$ virt-builder fedora-21 --root-password password:fedora --install vim-enhanced --update

I decided to try to use it for creating the base image for our disposable clients. The idea is that virt-builder is run once a day on a server and it generates a fresh new VM disk image. We then use this disk image for all our disposable clients spawned in the future, until a new disk image is generated the next day. Rinse and repeat.

The benefit of this approach (over e.g. official cloud images which might be updated once per month or twice per year) is:
* the clients are always up-to-date
* it's very light on bandwidth - instead of downloading updates for every client on every task run, it's done once per day, centrally
* it's much faster - the clients just boot and execute the task, they don't spend time upgrading each boot
* it's less error-prone - there is a never-ending story of failures when you communicate over network thousands times per day. We know it really well. By updating the client just once, the chance for a network error is much lower, and we can easily repeat it if it fails. Also, it's not a big deal if it completely fails some day, our tasks continue to be executed without a problem, just with a day-old packages installed.
* can be integrated with fedmsg "updates pushed" notification to be almost instantly in sync with the latest updates
* it helps with rescheduling tasks and reproducing issues - if we keep last X-number of VM images, we can easily replay past tasks with exactly the same package set as it had during that time

Subsequently I realized that we will probably also want to have several different base disk images, not just one. For example, certain GNOME checks might require most of GNOME installed, which would mean downloading a lot of packages every time they are run. But with virt-install, we can pre-create different base images for different purposes, and then let the task define which image to use. So for GNOME checks, we would prepare a base image with full GNOME installed.

To make all of this manageable, I created a wrapper around virt-builder and called it taskotron-vmbuilder (ideas for a better name are welcome). The code is available here:

https://bitbucket.org/fedoraqa/taskotron-vmbuilder

The concept is that you can define templates and then run vmbuilder which takes care of the rest - it uses virt-builder to create the image, it makes sure that libtaskotron is installed, it configures stuff like root password and ssh key. We can add any other functionality that is needed. The simplest way to try this is this:

$ ./vmbuilder.py templates/f21-minimal.yaml

There are sample templates for minimal installation and Server and Workstation products. For more details, look into READMEs, it's fairly well documented, I think. My idea is that we will run this daily on the server, store the resulting images somewhere, and update the pointer to the "current one".

I still haven't created project and repo in Phab, but if you approve this as a way forward, I will.

Thanks,
Kamil

[Tim Flink]

On Wed, 29 Apr 2015 11:21:10 -0400 (EDT)
Kamil Paral  wrote:

<snip>
> There are sample templates for minimal installation and Server and
> Workstation products. For more details, look into READMEs, it's
> fairly well documented, I think. My idea is that we will run this
> daily on the server, store the resulting images somewhere, and update
> the pointer to the "current one".

I like the approach and think that it's worth pursuing farther. It
lines up well with the rest of our plans for disposable clients.
> I still haven't created project and repo in Phab, but if you approve
> this as a way forward, I will.

I vote for creating the project and repo in phabricator.

Tim

[Kamil Paral]

> > I still haven't created project and repo in Phab, but if you approve
> > this as a way forward, I will.
>
> I vote for creating the project and repo in phabricator.

Done.

PS: I also noticed that our projects were editable by anyone ("all users"), which didn't seem to me as a great idea, so I changed it to "administrators" for all of them.

[Adam Williamson]

On Wed, 2015-04-29 at 11:21 -0400, Kamil Paral wrote:
> I still haven't created project and repo in Phab, but if you approve 
> this as a way forward, I will.

Just one thought - this could probably be rather useful for openQA
too. We already have one case where we want to begin a test with a
specific existing image (the fedup test that's currently under
review). We could use a system like this to do other tests that don't
involve actually running an installation (I'd quite like to do stuff
like the Base and desktop validation tests and maybe some package
management tests every day, for e.g.)

It might be interesting to look at how openSUSE handles this, as I
believe they do similar stuff.

[Kamil Paral]

> On Wed, 2015-04-29 at 11:21 -0400, Kamil Paral wrote:
> > I still haven't created project and repo in Phab, but if you approve
> > this as a way forward, I will.
>
> Just one thought - this could probably be rather useful for openQA
> too. We already have one case where we want to begin a test with a
> specific existing image (the fedup test that's currently under
> review). We could use a system like this to do other tests that don't
> involve actually running an installation (I'd quite like to do stuff
> like the Base and desktop validation tests and maybe some package
> management tests every day, for e.g.)

If taskotron-vmbuilder works for you as well, that's great. I can add some actions specific for openQA needs, just tell me (or file some tickets).

Of course, you can also use virt-builder directly, if that's easier for you. Currently taskotron-vmbuilder can't do anything extra that virt-builder can't. I just consider using the templates a bit cleaner than creating custom shell scripts for each image type. But it in the future, it might be extended with some extra functionality, like e.g. booting the image once and performing some first boot adjustments that can't be done when chrooted during image creation.