Speeding up the test suite

[Marius Vollmer]

Hi,

as discussed, we want to speed up the test suite.

Here are some VERY round numbers of the current test suite:

VERIFY:             13:00

make-rpms:           2:30
    vm-create:           0:30
    vm-install:          1:30
    check-verify:        9:00

vm boot:             0:10 x 27 = 4:30

If we don't want to touch the tests themselves for tnow, we can do this
to speed this up:

- Make the vms boot faster.

The time to boot includes waiting for the network to be up, which
  needs a DHCP response, wich can be quite slow apparently.

- Make mock faster.

Adding --no-clean to mock let's make-rpms run in about 0:30.  I will
  do this and add a "--clean" option to make-rpms.

- Make vm-install faster.

The idea is to start from a 'prepared' tarball that is already
  up-to-date with the repos and has all the dependencies installed.  The
  'prepared' tarball would be refreshed once per day.

The 'prepared' tarball will be created from the current 'base' tarball
  simply by running the base tarball through vm-create && vm-install
  $DEPS && virt-tar-out.  $DEPS are some reasonable current dependencies
  of Cockpit.

This might bring vm-install down to about 10 seconds.

I'll set this up and keep a freshly prepared tarball online somewhere.

- Make vm-create faster.

We can get rid of this step altogether by using a disk image directly
  instead of a tarball.

With this, we still have the magic base tarball that I have created by
hand.  I tried to automate this with oz-install and virt-install but I
failed.  I didn't try virt-builder since it is not in Fedora 19.

[Marius Vollmer]

Marius Vollmer  writes:
> Here are some VERY round numbers of the current test suite:
> VERIFY:             13:00
>
> make-rpms:           2:30
> vm-create:           0:30
> vm-install:          1:30
> check-verify:        9:00

I have moved all yum stuff out of VERIFY and into the new PREPARE step.
Also, make-rpms now uses --no-clean.  Results:

PREPARE:    17:00
    VERIFY:     10:30

make-rpms:   0:30
    vm-unpack:   0:45
    vm-install:  0:20

The idea is to run PREPARE whenever the OS repos or cockpit-deps changes
enough, and to distribute the result among us.  The actual VERIFY run
then should be isolated from changes in the repos.

Thus, running PREPARE saves time, but it also checks whether any of the
new packages in the repos will break us.  If that happens, we would
continue to use the old tarballs until we figure out the breakage.

As a next step, I think I'll try running the actual check-* scripts in
parallel.  That might be the easiest way to go faster.

[Marius Vollmer]

Marius Vollmer  writes:
> As a next step, I think I'll try running the actual check-* scripts in
> parallel.  That might be the easiest way to go faster.

There are good new and unwelcome news: Good news is that we can get an
easy speedup (from 8 minutes down to 3 on my 4 core laptop), the
unwelcome news is that this is very successful at exposing bugs in
Cockpits and the tests.

We know that the tests are very racy, and Cockpit plus its dependencies
are also not spotless.  If we want to speed up the tests by running them
in parallel, we need to finally fix those.

For example, UDisks2 has a 15 second timeout for a block device wiping
to be reflected in udev.  It seems to be possible to hit this timeout.

[Marius Vollmer]

Marius Vollmer  writes:
> We know that the tests are very racy, and Cockpit plus its dependencies
> are also not spotless.  If we want to speed up the tests by running them
> in parallel, we need to finally fix those.

To be fair, I see the same issues also when running the tests
sequentially on Fedora 20, only less often.  The switch to F20 has
probably also scared up some previously hidden race conditions.