On Sat, 2017-03-18 at 10:42 -0400, Colin Walters wrote:
On Fri, Mar 17, 2017, at 11:00 PM, Dusty Mabe wrote:
> Yes! we would love a UEFI test
Yes; conceptually most of the important scenarios we run through
for Workstation/Server (e.g. dm-crypt versus not) we should also
do for Atomic Host.
Adam, where is the git containing the input test suite for OpenQA?
be aware, though, that the way tests work in openQA is somewhat
idiosyncratic and very...specific. I can give you lots of details if
you like, but the short version is that what we ultimately wind up
actually *doing* so far as Atomic's concerned is: any time a fedmsg
'compose complete!' message appears for a compose that contains an
Atomic Host installer image, openQA will spin up a VM with that ISO
attached as an optical drive and a single empty hard disk, then do
nearly the simplest possible install you can do. It just goes to
INSTALLATION DESTINATION, selects the hard disk as the target and
agrees to let anaconda partition it automatically, starts the install,
sets a root password, and creates a user called 'test'. Once the
install completes, it boots the installed system, logs in as root,
checks whether there are any crash notifications or AVCs, uploads some
resource usage information, and that's it.
> and I think it would actually be good
> to run the atomic-host-tests against these images assuming openQA is
> the right tool for that job.
This part I don't quite agree with - remember the cloud images are
built with the installer ISO, and *those* are tested already with a-h-t.
There's going to be very little that's specific to the ISO path versus
the cloud image - which is part of the whole idea of sharing the
same exact ostree commit across them.
> I thought openQA was more for interactive
> install testing,
It does do more than that - however I see it as mostly valuable for testing
interactive Anaconda runs with the OpenCV approach.
IMO OpenQA's use of OpenCV is less interesting for testing
bits, as is the case for Atomic Host. (We do of course have Cockpit
and the OpenShift web console, but both of those use Selenium already)
openQA is perfectly *capable* of doing non-graphical testing; we do
actually do quite a lot of this just because it's there and relatively
convenient to use. We don't use screen matching to do this, though.
Quite a long time ago openQA grew the ability to do this kind of stuff
through the serial console; you can use a convenience function to run
any command at a console and either assert that it succeeds (which is
implemented by echoing its return code plus a hash of the command -
purely for identification purposes - to the serial console, then
identifying when that text hits the serial console and extracting the
return code), or do some kind of match on whatever the command sends to
openQA doesn't have any huge benefits as a way of doing this compared
to any of the other zillion implementations of 'spin up a VM and run
some commands in it', though, beyond the fact that it can be useful to
have a video recording of what was actually displayed on the console
during the run (which is something very few other test systems
provide). It has a couple of drawbacks, principally the idiosyncracies
of how tests actually work in openQA; it's possible to hide this from
people by writing some kinda middle layer code which would allow you to
just dump a test script in a git repo and the middle layer code would
automatically generate an openQA job that runs the script, I started
writing something along those lines as a side project a while back just
as a proof-of-concept that we could run tests from dist-git in openQA,
but I've never finished it.
If 'the atomic-host-tests' are just scripts that can be run from a
console we could very trivially run them post-install as part of the
openQA testing of the installer image, if desired. I agree with Colin
that this is quite unlikely to produce different results from running
the same tests on the equivalent Atomic host disk images in Taskotron
or Autocloud, but it wouldn't really *hurt* anything (besides eating up
a bit more of our limited openQA worker resources).
We do also do some basic testing of Cockpit in openQA, though nowhere
near as much as its own test suite covers.
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net