default file system, was: Comparison to Workstation Technical Specification

Fri Feb 28 21:03:38 UTC 2014

On Fri, 2014-02-28 at 15:46 -0500, Josh Boyer wrote:
> On Fri, Feb 28, 2014 at 3:16 PM, Adam Williamson <awilliam at redhat.com> wrote:
> > On Thu, 2014-02-27 at 11:56 -0500, Bill Nottingham wrote:
> >> Stephen Gallagher (sgallagh at redhat.com) said:
> >> > Directed more broadly at all three products:
> >> >
> >> > Formal proposal (for discussion): All three products agree to use ext4
> >> > for /boot and XFS-on-LVM for all other partitions in the "guided"
> >> > mode. All is fair game in the "custom" mode.
> >> >
> >> > Also, for the sake of everyone's sanity, as we discuss this specific
> >> > proposal, let's hold the conversation to devel at lists.fedoraproject.org
> >> > (making this the last cross-posted message in the thread).
> >>
> >> ... I understand that synergy can help, but given we likely expect usage
> >> of all(*) the local fileystems, is there a reason the three produces need to
> >> share partitioning setup?
> >>
> >> (*) well, not reiserfs
> >
> > We can expect use of them, but if all the products agree, then we at
> > least have one default that we can test to destruction. As discussed in
> > another thread around here somewhere, we (QA) would like to return to
> > the clear distinction between custom- and non-custom partitioning, where
> > non-custom is as 'choice-free' as plausible and correspondingly
> > reliable, and custom is a best-effort thing.
> 
> Can you elaborate on how that's eases the test matrix?

Sure. Just a bit below...

> As I said in a conversation with Stephen yesterday, I don't think it's
> the case that a common layout in partitions/fs is actually reducing
> the test load.  From a component standpoint, sure absolutely it is.
> One filesystem to test is easier than 3.  However, we don't do
> _filesystem_ testing in the context of release testing.

We attempt to test all paths through non-custom installation. Well, at
present we don't cover this very well, but it's always been what QA was
*supposed* to do. Right now, what we have is this mess:

https://fedoraproject.org/wiki/QA:Fedora_20_Install_Results_Template#General_Tests

the tests listed as "Partitioning". They are, really, held over from
oldUI: their instructions have been modified to suit newUI, but the
actual logic of *what tests we have* is based almost entirely on the
oldUI interface. It covers, quite well, all the non-custom choices oldUI
offered. It does not cover all the non-custom choices offered in newUI
as of F20. It comes off as a pretty much random hodgepodge. We have a
test which is just 'go into custom partitioning and...do something'. We
have tests for various filesystems 'as the root fs', but not any of the
other things newUI will do if you pick a variant filesystem. It just
doesn't make an awful lot of sense.

So, at the start of the F21 cycle, I tried to come up with a new
approach to installer partitioning testing that actually covered what
we're supposed to be covering. Chris spent some time thinking about this
too.

What I came up with is this gem:

https://fedoraproject.org/wiki/User:Adamwill/Draft_storage_matrix

There's 28 tests there just to cover the basics of what non-custom
partitioning offers (it still doesn't cover everything). That's a lot of
testing when we tend to iterate a TC/RC every three or four days. And we
have a wonderful set of possible 'variant' factors which just make
everything explode exponentially if you think about it too hard - we
kinda need those tables to be *three (or more!) dimensional*, because
the 'Device tests' don't consider filesystems, and the 'filesystem
tests' don't consider backing devices or architectures ("sure, but does
it work on ARM?")

I hope this goes some way to explaining why we're strongly in favour of
as much simplification on the non-custom path (or 'paths', now we have
multiple products) as is practical. :)

Some of this is susceptible to automation, but some is not, in the sense
that it involves the UI, and automated UI testing is a giant PITA. And
'susceptible to automation' is all very well, but it requires someone to
do the work of automating it.

It's been on my todo list for a while to at least come up with a
collection of kickstarts that exercises the options that are easy to do
with a straight kickstart from an empty VM, but that's only making a
small dent in the problem, really.

If you want to follow our discussion, here's some probably-good entry
points:

https://lists.fedoraproject.org/pipermail/test/2013-December/119447.html
https://lists.fedoraproject.org/pipermail/test/2013-December/119587.html
https://lists.fedoraproject.org/pipermail/test/2013-December/119604.html

>   It's implicit
> in the other tests.

That's kind of one of our major problems right now: "implicit in other
tests" testing sucks, because you don't find things early enough or in
an organized enough way. "Hey, I was testing this other thing and I
found a storage bug!" lends itself to vague bug reports and us
forgetting to keep testing the thing that caused the problem and so not
catching regressions, and stuff...

> So if we have 3 products, which deliver 3 different install media,
> then we still have 3 different things to test regardless of the
> FS/partition scheme chosen by default.  Cloud is delivering cloud
> images.  Workstation is looking at a live image as it's default
> deliverable, and I believe Server is looking at a more traditional
> install approach (with it being the only OS on the machine).  Given
> they all have completely different ways of doing the install, having a
> common fs and layout doesn't particularly change the fact that they
> all still need to be tested.

Of course not, but we are already wired into the three product plan. In
fact, it makes the reduction of choice in terms of storage *more*
urgent, not *less*, because we need to cut our (QA's and/or the
project's as a whole, the point really applies to both) workload
somewhere in order to compensate for the effect of having to test three
products. 28 tests * 2 products (Cloud doesn't use anaconda) = more
pain.

> I do agree on the custom vs. non-custom part, and having the install
> options in each media as choice-free as possible will help overall.

Great.

> Having to do multiple iterations over each product install approach
> makes things even harder.  I think that if the defaults happen to be
> the same it's a bonus, but not a requirement.

I think it's a pretty substantial bonus. The partitioning code is, to
quite a strong extent, a "thing". It doesn't matter *too* much whether
you're running it from a live image or a non-live image, it tends to
break in about the same way (though arch and backing device can affect
it, image type doesn't). If workstation and server are still using the
same set of partitioning code, it *does* make things substantially more
robust if they use the _same path through that partitioning code_ as
their default choice. (I think server and workstation may want to vary
their autopart configurations too, which is more testing pain, but at
least has a more concrete justification, AFAICS, than varying their
filesystems).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net