Testing test releases: do not update

[Mike A. Harris]

On Tue, 24 Feb 2004, shmuel siegel wrote:

>Test Release - is it merely a convenient snapshot for installation but
>serving no useful purpose after that (other than PR)? This is what I
>would gather from those that advocate that testers stay in sync with
>Rawhide.

Test release of an OS snapshot, ie: "Fedora Core 2 test 1", is
essentially a prebeta snapshot of the OS, which is very likely to
be unstable.

In the past, there were private beta testers and approximately 3
"alpha" releases that were generally not ready for public
consumption, but needed wider testing than could be done
internally.

With the opening up of OS development, and creation of the Fedora 
Project, it was decided to make _all_ of the beta/alpha/whatever 
you want to call it releases public and open.  The upside is that 
there would be more testers this way.  The downside, is that 
people who voluntarily test the first few initial test releases 
without realizing that they are playing with fire, are likely to 
end up with very broken systems, as "test1" is a very first test, 
and is nowhere near "stable".  The warnings in the installer 
should not be taken lightly.


>Rawhide - is it the staging ground for release candidates or is it a
>communication point between a developer and those that are in contact
>with him? If it is the latter, then there needs to be another repository
>that indicates that a package is ready for global testing. If it is the
>former, than I wonder why an intermediate stage exists in the Core 1
>tree.

Rawhide is the current set of packages that were built 
internally.  The rough process, is this:

- Developer updates a package to a newer version, or fixes some 
  bugs, adds patches, whatever.

- Developer builds package locally on his workstation and does 
  whatever testing he deems necessary.

- Developer submits package into the buildsystem, telling it to 
  build the package into the Fedora Core 2 development tree.

- Once the package has been successfully built on all 7 
  architectures, the buildsystem accepts the package into the 
  internal Fedora Core 2 development tree.

The above process is how we update all of our packages during 
development essentially. So what is rawhide then?  Simple.  Once 
every day or so, a script is ran either automated or manually, 
which takes all of the latest src.rpm and binary rpms in the 
current internal Fedora Core development tree, and mirrors them 
to our ftp staging server.  The staging server then pushes the 
rpms to the public ftp servers.  This is called "rawhide".

There is zero QA testing done on any of the rawhide packages, 
because they are not "production ready", they are "work in 
progress, fresh off the press, caveat emptor, beware of large 
dog" or as Jef Spaleta puts it "rawhide might kill babies".

Do not use rawhide if you can not accept the possibility of total 
system meltdown and data destruction.  While it does not occur 
very often, it _CAN_ occur, and it does from time to time.  ;o)


How does rawhide differ from "test1" or other test releases?  
Again, that's a simple question with a simple answer.  A 'test' 
release, or beta, or whatever you want to call it, is a snapshot 
in time of rawhide.  Essentially, rawhide is snapshotted, and 
then ISO's are built, and some testing occurs with them.  Any 
major problems that pop up, we try to fix in hopes that it will 
be as installable as possible for as many people as possible, 
without delaying the test release unnecessarily.  In other words, 
a "test" release is unstable rawhide which might burn your house 
down, only very slightly better.

Packages continue to build into our internal development tree 
after that, which will continue to be mirrored to rawhide daily 
or so, and will go on eventually to become "test2", etc.  
Eventually after all test/beta/release candidates, etc. are done, 
we work up towards the final "gold" OS release which would be 
"Fedora Core 2".

That's it in a nutshell, excluding the finer details.


>I think that Robert Day's initial point is correct. If there is no
>stable baseline then testers are constantly finding superficial bugs;
>deep bugs that take hours of testing will never get reached. Alan Cox is
>also right when he states that a tester should check against the current
>state of rawhide before he reports a bug.

By definition, "in development" *means* "there is no stable 
baseline".


>I think that a lot of the confusion comes form a lack of a
>public test plan and the lack of guidelines for testers. Also,
>for some reason, the difference between internal testing (which
>in the framework of open source I would consider them to be
>dedicated testers) and beta testers (those trying to use the
>features in a real environment) has been totally blurred. Most
>of the arguments against Robert Day were from the perspective of
>internal testers. They are right for their function. But most of
>them didn't need Test 1 except to test Anaconda; they were in
>sync with rawhide anyway. For beta testers, a stable platform is
>needed. If they are not at least pretending to do useful work
>then real life considerations will never be actualized.

If people require a stable platform, they definitely should not 
be using rawhide at all, and should not be using any test 
release.  The entire purpose of the test releases, is that they 
*are not* stable, and we need people willing to test them anyway, 
and to report the instabilities to us, so that those 
instabilities _can_ be fixed.  If nobody ever wants to test 
anything except "stable" packages, then unstable packages wont 
get tested, wont get fixed, and the final OS wont be stable 
either.

>In summary, I also advocate an intermediate repository whose
>sole purpose is to keep the baseline usable.

I disagree, and I oppose the idea of having an additional 
intermediate tree.  It would do nothing really beneficial, would 
make the latest software that now goes into rawhide, remain 
untested for LONGER, and thus remain buggy longer.  It also would 
put an additional burden upon developers to have to track another 
tree, and to manually move packages from the "unstable" to the 
"not quite as unstable" tree from time to time.


>If Redhat is unwilling to do this until development branches to
>Core 3, then I must assume that Redhat regards final release of
>Fedora as the real beta.

Feel free to have that opinion if you wish, just note that your
opinion is not shared by everyone, and is just that - one
opinion.  Others will both agree with you, and disagree with you.

I disagree.


-- 
Mike A. Harris     ftp://people.redhat.com/mharris
OS Systems Engineer - XFree86 maintainer - Red Hat