Improving release / update process for virt packages

[Daniel P. Berrange]

This mail is a braindump of some ideas we've been kicking around to
try and improve the way we push new releases of upstream virt 
packages to Fedora. We want to try and improve the quality of stable
Fedora branches, and also improve testing of rawhide. 

Taking libvirt as an example, currently new libvirt upstream releases
get immediately built and pushed to both

 - Fedora rawhide
 - Fedora stable updates-testing

Depending on the development phase rawhide is in, we get somewhere 
between 'zero' and 'lots' of feedback for new builds in rawhide. 
Typically before rawhide beta, we get near zero feedback, and from
beta onwards we get alot of feedback.

We typically always get significant feedback when the new builds are
pushed to updates-testing.

The obvious problem with what we do for libvirt at the moment, is
that we are introducing major new features into the stable release
stream here. This risks destabalizing the stable Fedora streams, 
and also compromises our 'Feature' process because stuff we're 
pushing for Fedora 11 appears in Fedora 9 / 10 before F11 even comes
out.

I think it would be desirable to get the stable Fedora releases onto a 
pretty strong bugfix only policy, but we can only do that if we figure
out a way to get more people testing the stuff we're pushing for rawhide.
There simply isn't enough testing of rawhide prior to beta point. The
bug reports / testing from the stable update-testing repo far exceeds
the feedback we get from rawhide users.

We can keep telling people that rawhide is supposed to be functional
enough for daily use, but frankly history shows otherwise and I don't
see any sign of this ever changing. Things like the mass rebuild,
rpm upgrade, xorg <-> kernel interface are inevitably changing and
cause instability. Personally I do pretty much all of my day-to-day
libvirt work on a stable Fedora 9 / 10 distro, only ever using rawhide
if there is a specific thing I can't get in F9/10 packages.

There are many people who wish to test new virtualization pieces. They
do not wish to have to debug changed RPM, or changed Xorg or changed
kernel packages, and the rest of rawhide at the same time. While it is
true that problems with rawhide can be resolved when they occur, it is
still a significant waste of time. If I'm interested in testing virt,
I only expect to have to debug virt packages,  not the entire OS.

What we need is a way to expose new virtualization features to people
running the current latest stable Fedora release stream. So if they 
are interested in testing the new features, they can update to those
packages without having to include & debug the rest of rawhide.

Debian distros have this sort of idea in the 'backports' stream which
gives optional early access to new app releases, separate from the
normal updates and unstable streams. This is distro wide though.

We currently have people doing various adhoc personal repos from scratch
builds, so my suggestion is that formalize this under the umbrella of the
Fedora Virtualization SIG. Specifically, to provide a 'virt-preview' YUM
repo for the most recent stable stream (ie F10, but not F9).

In the designated virtualization package set, anytime we do a build of a
new release for rawhide, we would also do a build for the 'preview' stream.
We would explicitly not do these builds for the 'updates' repos, at least
not without a very significant period of testing & positive feedback.

To allow us to do builds in both the preview stream, and updates stream,
which are not scratch builds, this would unfortunately require another 
CVS branch. We'd also need an NEVR that satisfies

 - Newer than current stable stream NEVR
 - Older than the new build in rawhide
 - Allows for newer build to be later made in 'updates' repo

One simple option is to have

 - CVS branch of F-10-VIRT_PREVIEW
 - Release field for all builds in F-10VIRT-PREVIEW must have leading 0

The leading 0, allows us to later do a build in the main F-10 stable
branch with newer NEVR, without risking that it then become newer
than the rawhide build.

A convenient Makefile rule in the 'devel' branch could automate the
initial work of copying the contents of devel/ to F-10-VIRT-PREVIEW
and setting the N-E-V-R to suitable value. A few manual changes 
might be needed, but hopefully not much before the maintainer could
do a build in koji.

There would then need to be a way to build a YUM repo from all builds
in the F-10-VIRT-PREVIEW branch. It should automatically take all the
latest builds from the branch - we want it to be a mini rawhide of
just virt packages, not use the updates infrastructure.

If we could get Fedora infrastructure support for the branching that
would be great, but we could easily do this with a private branch like
(private-F-10-VIRT-PREVIEW) and a cron job to build the YUM repo and
publish in a well-known location, eg http://virtmaint.fedorapeople.org/

So in summary

 - All new upstream releases built in rawhide
 - New upstream releases also built in stable preview branch if possible
 - Only bugfixes built in stable updates/updates-testing branch
 - In exceptional circumstances, rebase for preview branch can be
   built to updates/updates-testing after alot of positive testing

This would

 - Ensure users of stable Fedora release have high confidence in 
   quality of the updates/updates-testing stream
 - Allow users to trivially test new upstream rebases while 
   staying on the stable distro stream
 - Improve testing coverage of major new rawhide features without using
   the stable release stream users as guinea pigs


Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Mark McLoughlin]

On Mon, 2009-04-06 at 15:41 +0100, Daniel P. Berrange wrote:

>  This mail is a braindump of some ideas we've been kicking around
> to
>  try and improve the way we push new releases of upstream virt 
>  packages to Fedora. We want to try and improve the quality of stable
>  Fedora branches, and also improve testing of rawhide.  
Totally agree with the conclusions here.

>  Taking libvirt as an example, currently new libvirt upstream
> releases
>  get immediately built and pushed to both
>  
>   - Fedora rawhide
>   - Fedora stable updates-testing
>  
>  Depending on the development phase rawhide is in, we get somewhere 
>  between 'zero' and 'lots' of feedback for new builds in rawhide. 
>  Typically before rawhide beta, we get near zero feedback, and from
>  beta onwards we get alot of feedback. 
I don't think "near zero" is quite accurate - there's been a fair bit
of
bugzilla activity on virt bits since the Alpha. One example is that
Fedora rel-eng and QA seem to be doing lots of installs testing using
KVM and filing bugs when it's broken.

Sure, not everyone runs rawhide, but we still need to do what we can to
help make it less broken.

>  We typically always get significant feedback when the new builds are
>  pushed to updates-testing. 
Personally, I see updates-testing as a "last chance QA" - i.e. it's only
for package updates that we really do believe are ready for stable
updates.

updates-testing is a distro-wide thing, so I expect most people who
enable it aren't specifically interested in testing virt, but they are
interested in being the last line of defence for updates before they hit
stable.

When libvirt-0.6.0 hit my laptop through updates-testing and was
significantly broken, my reaction was along the lines of "wait, I didn't
sign up for this!" - I suspect other people in that situation might
disable updates-testing as a result. That damages not only testing of
virt packages, but testing of the whole distro.

I think it was after that update that I wrote these guidelines and had
it approved by FESCo:

https://fedoraproject.org/wiki/PackageMaintainers/Package_update_guidelines

>  The obvious problem with what we do for libvirt at the moment, is
>  that we are introducing major new features into the stable release
>  stream here. This risks destabalizing the stable Fedora streams, 
>  and also compromises our 'Feature' process because stuff we're 
>  pushing for Fedora 11 appears in Fedora 9 / 10 before F11 even comes
>  out. 
Yeah - that is another aspect. How could we pimp feature X in the F11
release notes if it is pushed to F9 updates even before F11 GA?

And another thing is pushing new features to updates doesn't mean we've
thought about whether the feature will work or not. I saw a bug report
today of PXE booting not working with latest virt-manager in F9
updates-testing because the virtio PXE ROM isn't available on F9. That
just wasn't an issue with the original F9 version.

>  I think it would be desirable to get the stable Fedora releases onto
> a 
>  pretty strong bugfix only policy, but we can only do that if we figure
>  out a way to get more people testing the stuff we're pushing for rawhide.
>  There simply isn't enough testing of rawhide prior to beta point. The
>  bug reports / testing from the stable update-testing repo far exceeds
>  the feedback we get from rawhide users.
>  
>  We can keep telling people that rawhide is supposed to be functional
>  enough for daily use, but frankly history shows otherwise and I don't
>  see any sign of this ever changing. Things like the mass rebuild,
>  rpm upgrade, xorg <-> kernel interface are inevitably changing and
>  cause instability. Personally I do pretty much all of my day-to-day
>  libvirt work on a stable Fedora 9 / 10 distro, only ever using rawhide
>  if there is a specific thing I can't get in F9/10 packages. 
I agree that it's not wise to use rawhide as your everyday machine,
especially early in the release cycle, but i do believe a significant
majority of the people seriously involved in Fedora have up-to-date
rawhide running on another machine and use this for their Fedora
development.

>  There are many people who wish to test new virtualization pieces.
> They
>  do not wish to have to debug changed RPM, or changed Xorg or changed
>  kernel packages, and the rest of rawhide at the same time. While it is
>  true that problems with rawhide can be resolved when they occur, it is
>  still a significant waste of time. If I'm interested in testing virt,
>  I only expect to have to debug virt packages,  not the entire OS.
>  
>  What we need is a way to expose new virtualization features to people
>  running the current latest stable Fedora release stream. So if they 
>  are interested in testing the new features, they can update to those
>  packages without having to include & debug the rest of rawhide.
>  
>  Debian distros have this sort of idea in the 'backports' stream which
>  gives optional early access to new app releases, separate from the
>  normal updates and unstable streams. This is distro wide though.
>  
>  We currently have people doing various adhoc personal repos from scratch
>  builds, so my suggestion is that formalize this under the umbrella of the
>  Fedora Virtualization SIG. Specifically, to provide a 'virt-preview' YUM
>  repo for the most recent stable stream (ie F10, but not F9).
>  
>  In the designated virtualization package set, anytime we do a build of a
>  new release for rawhide, we would also do a build for the 'preview' stream.
I think this would be hugely useful to people interested in the latest
virt bits, but without a testing machine for running rawhide.

How about "virt-hide" ? :)

>  We would explicitly not do these builds for the 'updates'
> repos, at least
>  not without a very significant period of testing & positive feedback.
>  
>  To allow us to do builds in both the preview stream, and updates stream,
>  which are not scratch builds, this would unfortunately require another 
>  CVS branch. We'd also need an NEVR that satisfies
>  
>   - Newer than current stable stream NEVR
>   - Older than the new build in rawhide
>   - Allows for newer build to be later made in 'updates' repo
>  
>  One simple option is to have
>  
>   - CVS branch of F-10-VIRT_PREVIEW
>   - Release field for all builds in F-10VIRT-PREVIEW must have leading 0
>  
>  The leading 0, allows us to later do a build in the main F-10 stable
>  branch with newer NEVR, without risking that it then become newer
>  than the rawhide build.
>  
>  A convenient Makefile rule in the 'devel' branch could automate the
>  initial work of copying the contents of devel/ to F-10-VIRT-PREVIEW
>  and setting the N-E-V-R to suitable value. A few manual changes 
>  might be needed, but hopefully not much before the maintainer could
>  do a build in koji. 
Another way would be to create a new CVS branch of 'devel' for each
build - that way you wouldn't have to sync 'devel' into the preview
branch each time. It should be possible to script tweaking the release
field and anything depending on the distro version could be
conditionalised in the spec file in 'devel'.

>  There would then need to be a way to build a YUM repo from all
> builds
>  in the F-10-VIRT-PREVIEW branch. It should automatically take all the
>  latest builds from the branch - we want it to be a mini rawhide of
>  just virt packages, not use the updates infrastructure.
>  
>  If we could get Fedora infrastructure support for the branching that
>  would be great, but we could easily do this with a private branch like
>  (private-F-10-VIRT-PREVIEW) and a cron job to build the YUM repo and
>  publish in a well-known location, eg http://virtmaint.fedorapeople.org/ 
Yep, I think we should do it that way first - if it works out well, we
can lobby rel-eng to generalize the concept for other areas of the
distro too.

>  So in summary
>  
>   - All new upstream releases built in rawhide
>   - New upstream releases also built in stable preview branch if possible
>   - Only bugfixes built in stable updates/updates-testing branch
>   - In exceptional circumstances, rebase for preview branch can be
>     built to updates/updates-testing after alot of positive testing
>  
>  This would
>  
>   - Ensure users of stable Fedora release have high confidence in 
>     quality of the updates/updates-testing stream
>   - Allow users to trivially test new upstream rebases while 
>     staying on the stable distro stream
>   - Improve testing coverage of major new rawhide features without using
>     the stable release stream users as guinea pigs 
Excellent!

Cheers,
Mark.

[Mark McLoughlin]

On Mon, 2009-04-06 at 15:41 +0100, Daniel P. Berrange wrote:

>  So in summary
>  
>   - All new upstream releases built in rawhide
>   - New upstream releases also built in stable preview branch if possible
>   - Only bugfixes built in stable updates/updates-testing branch
>   - In exceptional circumstances, rebase for preview branch can be
>     built to updates/updates-testing after alot of positive testing 
Dan did a great job of laying out the process we intend using for
updates to virt packages going forward. Re-read it if you missed it the
first time:

  http://www.redhat.com/archives/fedora-virt/2009-April/msg00008.html

One thing we hadn't discussed until earlier this week is what to do
about current F9 and F10 updates.

We had pushed libvirt 0.6.0 (late January) and 0.6.1 (early March) to F9
and F10 updates-testing. Neither of these were ever pushed to stable
updates.

Packages aren't intended to live in updates-testing forever - it is
intended that a package gets pushed to stable after some testing, or
gets unpushed if its too unstable. If a package does linger in
updates-testing for too long, it prevents simple bugfix updates for the
previous version from being pushed in the mean time.

Since we weren't confident that pushing 0.6.1 to F9 and F10 stable
updates, we were left with no choice but to unpush it.

So in summary

  - libvirt 0.6.1 has been unpushed from F9 and F10 updates-testing

  - The latest version available in F9 and F10 updates is 0.5.1

  - We do not expect to push new versions to F9 and F10, only bug fix 
    updates for 0.5.1

  - We are still planning on setting up a "preview" repository where 
    the latest versions of virt packages from rawhide will be available 
    to Fedora stable release users

Cheers,
Mark.

[Bill Davidsen]

Mark McLoughlin wrote:
>  On Mon, 2009-04-06 at 15:41 +0100, Daniel P. Berrange wrote:
>
>    
> > So in summary
> >
> >  - All new upstream releases built in rawhide
> >  - New upstream releases also built in stable preview branch if possible
> >  - Only bugfixes built in stable updates/updates-testing branch
> >  - In exceptional circumstances, rebase for preview branch can be
> >    built to updates/updates-testing after alot of positive testing
> >     
>
>  Dan did a great job of laying out the process we intend using for
>  updates to virt packages going forward. Re-read it if you missed it the
>  first time:
>
>    http://www.redhat.com/archives/fedora-virt/2009-April/msg00008.html
>
>  One thing we hadn't discussed until earlier this week is what to do
>  about current F9 and F10 updates.
>
>  We had pushed libvirt 0.6.0 (late January) and 0.6.1 (early March) to F9
>  and F10 updates-testing. Neither of these were ever pushed to stable
>  updates.
>
>  Packages aren't intended to live in updates-testing forever - it is
>  intended that a package gets pushed to stable after some testing, or
>  gets unpushed if its too unstable. If a package does linger in
>  updates-testing for too long, it prevents simple bugfix updates for the
>  previous version from being pushed in the mean time.
>
>  Since we weren't confident that pushing 0.6.1 to F9 and F10 stable
>  updates, we were left with no choice but to unpush it.
>
>  So in summary
>
>    - libvirt 0.6.1 has been unpushed from F9 and F10 updates-testing
>
>    - The latest version available in F9 and F10 updates is 0.5.1
>     
So the updates-testing stuff is just teasers and is never going to be 
generally available without a full update? And if some of our machines 
are on one version and some are on another we live with that, roll our 
own from FC11 source, or force downgrade?

-- 
bill davidsen <davidsen(a)tmr.com&gt;
  CTO TMR Associates, Inc

"You are disgraced professional losers. And by the way, give us our money
back."
    - Representative Earl Pomeroy,  Democrat of North Dakota
on the A.I.G. executives who were paid bonuses  after a federal bailout.

[Mark McLoughlin]

On Fri, 2009-04-17 at 15:46 -0400, Bill Davidsen wrote:
>  Mark McLoughlin wrote:
>  > On Mon, 2009-04-06 at 15:41 +0100, Daniel P. Berrange wrote:
>  >
>  >   
>  >> So in summary
>  >>
>  >>  - All new upstream releases built in rawhide
>  >>  - New upstream releases also built in stable preview branch if possible
>  >>  - Only bugfixes built in stable updates/updates-testing branch
>  >>  - In exceptional circumstances, rebase for preview branch can be
>  >>    built to updates/updates-testing after alot of positive testing
>  >>     
>  >
>  > Dan did a great job of laying out the process we intend using for
>  > updates to virt packages going forward. Re-read it if you missed it the
>  > first time:
>  >
>  >   http://www.redhat.com/archives/fedora-virt/2009-April/msg00008.html
>  >
>  > One thing we hadn't discussed until earlier this week is what to do
>  > about current F9 and F10 updates.
>  >
>  > We had pushed libvirt 0.6.0 (late January) and 0.6.1 (early March) to F9
>  > and F10 updates-testing. Neither of these were ever pushed to stable
>  > updates.
>  >
>  > Packages aren't intended to live in updates-testing forever - it is
>  > intended that a package gets pushed to stable after some testing, or
>  > gets unpushed if its too unstable. If a package does linger in
>  > updates-testing for too long, it prevents simple bugfix updates for the
>  > previous version from being pushed in the mean time.
>  >
>  > Since we weren't confident that pushing 0.6.1 to F9 and F10 stable
>  > updates, we were left with no choice but to unpush it.
>  >
>  > So in summary
>  >
>  >   - libvirt 0.6.1 has been unpushed from F9 and F10 updates-testing
>  >
>  >   - The latest version available in F9 and F10 updates is 0.5.1
>  >   
>  
>  So the updates-testing stuff is just teasers and is never going to be 
>  generally available without a full update? And if some of our machines 
>  are on one version and some are on another we live with that, roll our 
>  own from FC11 source, or force downgrade? 
I'm not sure I fully understand the question, but essentially
updates-testing was being used to give a preview of newer releases of
libvirt which weren't intended to be pushed to stable updates. We've
decided that's a bad idea and removed libvirt from updates-testing.

If you've a machine running 0.6.1 from updates-testing, it would
probably be best to downgrade to 0.6.1. Or, indeed, upgrade to F11.

Cheers,
Mark.

[Mark McLoughlin]

On Mon, 2009-04-06 at 15:41 +0100, Daniel P. Berrange wrote:

>  We currently have people doing various adhoc personal repos from
> scratch
>  builds, so my suggestion is that formalize this under the umbrella of the
>  Fedora Virtualization SIG. Specifically, to provide a 'virt-preview' YUM
>  repo for the most recent stable stream (ie F10, but not F9).
>  
>  In the designated virtualization package set, anytime we do a build of a
>  new release for rawhide, we would also do a build for the 'preview' stream.
>  We would explicitly not do these builds for the 'updates' repos, at least
>  not without a very significant period of testing & positive feedback.
>  
>  To allow us to do builds in both the preview stream, and updates stream,
>  which are not scratch builds, this would unfortunately require another 
>  CVS branch. 
Just thinking about this some more. For now, I'm inclined to go for an
extremely low tech solution which requires no rel-eng support in order
to prove the concept.

Simply:

  1) When you do a build in rawhide, also do:

       make scratch-build TARGET=dist-f11

  2) Note the task number and on fedorapeople.org pull the packages 
     from:

       http://koji.fedoraproject.org/scratch/$(user)/task_$(task)/

     to e.g. ~/public_html/virt-f11-preview

     (We could add a Makefile to automate these two steps)

  3) I'll create a dir which just links to the various known preview 
     repos in peoples homedirs

  4) I'll regularly run createrepo in that dir

Yes, there are disadvantages in using scratch builds - e.g. no new
buildroot, so we can't build against other preview builds. But I'd
prefer to get this kicked off in some form rather and show that it's
useful, before getting into asking rel-eng for a new koji
target/buildroot etc.

>   We'd also need an NEVR that satisfies
>  
>   - Newer than current stable stream NEVR 
This should happen fairly naturally - just don't do a preview build if
you're also going to build it for F-11.

>   - Older than the new build in rawhide 
Satisfied by the dist tag.

>   - Allows for newer build to be later made in 'updates' repo
I'm not really concerned - if e.g. we were doing scratch builds of
libvirt-0.6.3 up to libvirt-0.6.3-10.fc11 and wanted to push to updates,
we could either decide:

  a) Just re-build libvirt-0.6.3-10.fc11 in updates; ignore the fact 
     that some folks have a different build of the same n-v-r

  b) Re-build as libvirt-0.6.3-11.fc12 in rawhide and push 
     libvirt-0.6.3-11.fc11 to updates

It sucks a little, but given the simplicity of just doing scratch
builds, I think we can live with it for now.

Cheers,
Mark.