On Mon, 2009-04-06 at 15:41 +0100, Daniel P. Berrange wrote:
This mail is a braindump of some ideas we've been kicking around
to
try and improve the way we push new releases of upstream virt
packages to Fedora. We want to try and improve the quality of stable
Fedora branches, and also improve testing of rawhide.
Totally agree with the conclusions here.
Taking libvirt as an example, currently new libvirt upstream
releases
get immediately built and pushed to both
- Fedora rawhide
- Fedora stable updates-testing
Depending on the development phase rawhide is in, we get somewhere
between 'zero' and 'lots' of feedback for new builds in rawhide.
Typically before rawhide beta, we get near zero feedback, and from
beta onwards we get alot of feedback.
I don't think "near zero" is quite accurate - there's been a fair bit
of
bugzilla activity on virt bits since the Alpha. One example is that
Fedora rel-eng and QA seem to be doing lots of installs testing using
KVM and filing bugs when it's broken.
Sure, not everyone runs rawhide, but we still need to do what we can to
help make it less broken.
We typically always get significant feedback when the new builds are
pushed to updates-testing.
Personally, I see updates-testing as a "last chance QA" - i.e. it's only
for package updates that we really do believe are ready for stable
updates.
updates-testing is a distro-wide thing, so I expect most people who
enable it aren't specifically interested in testing virt, but they are
interested in being the last line of defence for updates before they hit
stable.
When libvirt-0.6.0 hit my laptop through updates-testing and was
significantly broken, my reaction was along the lines of "wait, I didn't
sign up for this!" - I suspect other people in that situation might
disable updates-testing as a result. That damages not only testing of
virt packages, but testing of the whole distro.
I think it was after that update that I wrote these guidelines and had
it approved by FESCo:
https://fedoraproject.org/wiki/PackageMaintainers/Package_update_guidelines
The obvious problem with what we do for libvirt at the moment, is
that we are introducing major new features into the stable release
stream here. This risks destabalizing the stable Fedora streams,
and also compromises our 'Feature' process because stuff we're
pushing for Fedora 11 appears in Fedora 9 / 10 before F11 even comes
out.
Yeah - that is another aspect. How could we pimp feature X in the F11
release notes if it is pushed to F9 updates even before F11 GA?
And another thing is pushing new features to updates doesn't mean we've
thought about whether the feature will work or not. I saw a bug report
today of PXE booting not working with latest virt-manager in F9
updates-testing because the virtio PXE ROM isn't available on F9. That
just wasn't an issue with the original F9 version.
I think it would be desirable to get the stable Fedora releases onto
a
pretty strong bugfix only policy, but we can only do that if we figure
out a way to get more people testing the stuff we're pushing for rawhide.
There simply isn't enough testing of rawhide prior to beta point. The
bug reports / testing from the stable update-testing repo far exceeds
the feedback we get from rawhide users.
We can keep telling people that rawhide is supposed to be functional
enough for daily use, but frankly history shows otherwise and I don't
see any sign of this ever changing. Things like the mass rebuild,
rpm upgrade, xorg <-> kernel interface are inevitably changing and
cause instability. Personally I do pretty much all of my day-to-day
libvirt work on a stable Fedora 9 / 10 distro, only ever using rawhide
if there is a specific thing I can't get in F9/10 packages.
I agree that it's not wise to use rawhide as your everyday machine,
especially early in the release cycle, but i do believe a significant
majority of the people seriously involved in Fedora have up-to-date
rawhide running on another machine and use this for their Fedora
development.
There are many people who wish to test new virtualization pieces.
They
do not wish to have to debug changed RPM, or changed Xorg or changed
kernel packages, and the rest of rawhide at the same time. While it is
true that problems with rawhide can be resolved when they occur, it is
still a significant waste of time. If I'm interested in testing virt,
I only expect to have to debug virt packages, not the entire OS.
What we need is a way to expose new virtualization features to people
running the current latest stable Fedora release stream. So if they
are interested in testing the new features, they can update to those
packages without having to include & debug the rest of rawhide.
Debian distros have this sort of idea in the 'backports' stream which
gives optional early access to new app releases, separate from the
normal updates and unstable streams. This is distro wide though.
We currently have people doing various adhoc personal repos from scratch
builds, so my suggestion is that formalize this under the umbrella of the
Fedora Virtualization SIG. Specifically, to provide a 'virt-preview' YUM
repo for the most recent stable stream (ie F10, but not F9).
In the designated virtualization package set, anytime we do a build of a
new release for rawhide, we would also do a build for the 'preview' stream.
I think this would be hugely useful to people interested in the latest
virt bits, but without a testing machine for running rawhide.
How about "virt-hide" ? :)
We would explicitly not do these builds for the 'updates'
repos, at least
not without a very significant period of testing & positive feedback.
To allow us to do builds in both the preview stream, and updates stream,
which are not scratch builds, this would unfortunately require another
CVS branch. We'd also need an NEVR that satisfies
- Newer than current stable stream NEVR
- Older than the new build in rawhide
- Allows for newer build to be later made in 'updates' repo
One simple option is to have
- CVS branch of F-10-VIRT_PREVIEW
- Release field for all builds in F-10VIRT-PREVIEW must have leading 0
The leading 0, allows us to later do a build in the main F-10 stable
branch with newer NEVR, without risking that it then become newer
than the rawhide build.
A convenient Makefile rule in the 'devel' branch could automate the
initial work of copying the contents of devel/ to F-10-VIRT-PREVIEW
and setting the N-E-V-R to suitable value. A few manual changes
might be needed, but hopefully not much before the maintainer could
do a build in koji.
Another way would be to create a new CVS branch of 'devel' for each
build - that way you wouldn't have to sync 'devel' into the preview
branch each time. It should be possible to script tweaking the release
field and anything depending on the distro version could be
conditionalised in the spec file in 'devel'.
There would then need to be a way to build a YUM repo from all
builds
in the F-10-VIRT-PREVIEW branch. It should automatically take all the
latest builds from the branch - we want it to be a mini rawhide of
just virt packages, not use the updates infrastructure.
If we could get Fedora infrastructure support for the branching that
would be great, but we could easily do this with a private branch like
(private-F-10-VIRT-PREVIEW) and a cron job to build the YUM repo and
publish in a well-known location, eg
http://virtmaint.fedorapeople.org/
Yep, I think we should do it that way first - if it works out well, we
can lobby rel-eng to generalize the concept for other areas of the
distro too.
So in summary
- All new upstream releases built in rawhide
- New upstream releases also built in stable preview branch if possible
- Only bugfixes built in stable updates/updates-testing branch
- In exceptional circumstances, rebase for preview branch can be
built to updates/updates-testing after alot of positive testing
This would
- Ensure users of stable Fedora release have high confidence in
quality of the updates/updates-testing stream
- Allow users to trivially test new upstream rebases while
staying on the stable distro stream
- Improve testing coverage of major new rawhide features without using
the stable release stream users as guinea pigs
Excellent!
Cheers,
Mark.