Managing Fedora Testing

[Keith Lofstrom]

> Keith Lofstrom wrote:
> > I would put myself in the 45% group, and include many of the folks 
> > that are complaining about booting or mirror problems.  The Fedora 
> > test process seems to be geared exclusively for the 5%.  As a 
> > result, Fedora will release with less than 10% of the potentially 
> > available testing effort, with the remaining effort 
> > disproportionally focused.

Jef Spaleta <jspaleta at princeton.edu> writes:
> I would sort of agree with that. I would agree that the aggressive 
> timescale for testing for fedora core means relying on those 5% (I'd put
> it closer to 1%) to be a strong component of the process. I would also
> say that the dropping of the private betas that rhl use to have makes
> the situation somewhat more dramatic looking than it is. As others have
> said, the fedora testing process, is a more open process than what
> happened with rhl. And more open translates directly into sharper edges
> for the informed to impale themselves on while blundering around in the
> dark.
> 
> There is a trade-off here, taking the very long time to do things a
> piece at a time, means the pieces quickly become out of sync with
> upstream development.
...

To reiterate, my concern is that the testing process gets hung up
on the front end, and the distro as a whole is incompletely checked.

I got some feedback, so I will go on to prescription.  A "conservative",
one-piece-at-a-time development process is of course much too slow. 
Instead, I would divide up the work into domains so that people can
choose a domain and test in that.  Where to draw the line is difficult,
but I would probably make the domains "kernel and drivers", "installation
and update", "X and desktops", and "applications".  Obviously, many
problems will cross domains, but to the extent that you can partition
things you reduce the scope of most problems.

20 years ago, I designed whole chips.  Now I design small pieces of
chips with 10 times as many transistors per piece.  Some times my
pieces interact nastily with other pieces, and that can cost a $1M
mask set and 3 months.  But 99% of the problems stay confined in my
box, or in the mid-sized box my piece is in.  This is how the
processors just keep getting cheaper and faster and better and sooner,
because we can partition the hell out of the problem, and this allows
chip designers to generate terabyte-sized designs really, really fast.

Code right now is partitioned into modules and programs and libraries
and packages, then distros.  For those of us interested in testing the
interaction of packages, a single package is too small and a whole
distro is too big.  Hence domains.   If we have domains A, B, and C, then
testers can work with Anew + Bnew + Cnew if they are in the 5% elite,
but the 70% less-skilled testers can work with Aold + Bold + Cnew if
they just want to look at interactions in that domain.  For my four
suggested domains, that suggests 5 test distros rather than one. 
It would be wonderful to do all combinations, or even any combination
with any incremental change in any package, but dependencies make that
impractical.  In any case, all 5 test distros at alpha are designed
to merge into one distro at beta, probably at test 3. 

Schedule would be:
Test 1:    Domains defined, new packages in each domain defined.  
           Find package problems.
Test 2:    Packages upgraded/downgraded/replaced to fix package problems.
           Find between-package problems within domains.
Test 3:    Integrate changed domains into one beta distro.
           Find between-domain problems.
Release0:  This is it!  The complete, perfect distro.
Release1:  Just kidding!   We fixed those killer bugs.
Release2:  We really mean it this time.
Release ...

This will be more work to set up;  I would guess 20% more, initially. 
But it captures 10x more labor, making more available for
desktop/application testing, which in turn reduces the distractions
for the install/kernel testers.  The real question is, do you want to
accomplish 20% testing or 80% testing?   I have real concerns that
Fedora Core 2 release, as a coherent distro, will be 20% tested,
because intense application/interaction beta testing will never happen.
If I wanted untested beta code, I would run windows.

Keith

-- 
Keith Lofstrom           keithl at ieee.org         Voice (503)-520-1993
KLIC --- Keith Lofstrom Integrated Circuits --- "Your Ideas in Silicon"
Design Contracting in Bipolar and CMOS - Analog, Digital, and Scan ICs