Very rough storage validation matrix draft

[Adam Williamson]

Hi, folks. So, our current set of storage validation tests is just a
grab-bag of stuff we've held over from oldui, added and patched
piecemeal; it's not very coherent or consistent and it doesn't come
close to exercising all of the storage stuff in the installer. We wind
up sort of inventing test plans particularly for custom partitioning as
we go along, with the consequence that we're not sure what we're going
to test, what's really important to test when, etc etc.

I've made a few abortive tries at re-doing the storage tests and
basically given up because it's just a hideous thing to try and cover,
but I thought while I'm still on a momentum roll from F20 and remember
some of the issues that came up during F20 validation, I'd take another
cut at it.

Here's what I came up with:

https://fedoraproject.org/wiki/User:Adamwill/Draft_storage_matrix

The proposal is to separate out storage testing from the 'installation
testing' matrix as its own matrix, because I think we can get further
with flexible columns, and it's such a big area the installation matrix
gets pretty unwieldy with all the storage tests stuffed into it.

remember this is *extremely* rough. It may be that we don't think my
organizational approach here is right at all and we should tear it up
and start over. But I thought I'd throw it out there for comments. Most
of the tests, obviously, don't exist yet; they'd be fairly trivial to
write, I think the hard part of the problem here is deciding what we
want to test, and how to organize it.

It's an extremely difficult area; there are so many different variables
to storage configuration that it's utterly impractical to try and cover
every possible permutation even assuming the user uses the interface
perfectly. What I've tried to do is come up with something which would
exercise most of the areas we really seem to want to care about, without
being utterly unwieldy.

It is still very large, that's probably the first thing you'd notice
about it. I doubt we could run this on every build. What I think would
be a nice complement is a somewhat improved version of testcase_stats -
http://testdays.qa.fedoraproject.org/testcase_stats/ - which could track
both dimensions of each matrix, so we could try to strategically ensure
every spot on the table is covered at least once per milestone or once
per release, say.

Here's a quick key to the test names for tests I've made up which may
not  be self-explanatory, yell if you need explanations of any of the
others:

-----

guided_multi_empty - the 'multi' means multiple disks, this is checking
we correctly autopart to multiple disks.

in the custom tests, 'auto' means 'using the autopart mechanism inside
custom partitioning', the blue text that lets you automatically create a
set of volumes.

existing_retain_home - this is the classic 'install over an existing
Fedora install, retain the /home partition' trick.

existing_precreated - this would be a test for running the installer
with a set of empty volumes that you just wanted to assign mount points
to.

add_to_container - add a new volume to free space within an existing
container.

assign_* tests - these are for specifying that a given partition or
container must be on a particular disk.

help_text - just checks the 'help' screen works.

small_disks - this would be a test where you check anaconda correctly
refuses to install to a disk that's too small.

cancel_encryption - this would be to check what happens if you cancel
out of the 'enter a passphrase' dialog.

shrink_maximum - try to shrink a partition to the largest possible size
(right-hand end of the slider)

shrink_minimum - shrink a partition as small as possible (left-hand end
of the slider)

shrink_no_size_change - set action for a partition to 'shrink' but don't
actually change its size

shrink_unusual_sizes - this would be for the bug we discovered in f20,
test the shrinker handles partitions with 'weird' sizes

shrink_change_action - this is just for changing the 'action' in shrink
a bunch of times and making sure it doesn't explode

multiple_trips - run through Installation Destination more than once

custom_resize_no_size - just clear the 'desired size' field for a
partition and hit 'update settings'

custom_resize_invalid_unit - try entering something like '30 ZX' in the
desired size field

custom_resize_return - change the desired size and then change it back
to the original value

custom_invalid_mount_point - try entering a mount point name that is not
allowed (invalid characters, spaces or something)

custom_invalid_filesystem - try putting a system partition on vfat or
bios boot or swap or something

----

Please, absolutely all comments, suggestions, alternative proposals,
flames etc etc welcome. I'm sure we could improve this proposal or do
better, but I think we have to try and do _something_ better than our
current tests. And I think drawing up a table like this, if nothing
else, illustrates what a hard task this is...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net