FYI: The plan for Xen kernels in Fedora 9

[Daniel P. Berrange]

This is a friendly alert of the major plans we have for Xen kernels in 
Fedora 9 timeframe...

Since we first added Xen in Fedora Core 5, our kernels have been based on
a forward-port of XenSource's upstream Xen kernels, to new LKML. For a
long time we ported their 2.6.16 tree to 2.6.18. Now we do ports of their
2.6.18 tree to 2.6.21/22/23, etc.  At the same time, upstream Linux gained
Xen support for i386 DomU, and shortly x86_64  DomU, and is generally
getting ever more virtualization capabilities.

As everyone knows, we have tended to lag behind Fedora's state-of-the-art 
bare metal kernels by several releases due to the effort required to port
Xen to newer LKML releases. Despite our best efforts, this lag has been 
getting worse, not better.

We have taken the decision, that this situation is unacceptable for Fedora 9.
We simply cannot spend more time forward porting Xen kernels. Either Xen has
to be dropped entirely, or we need a different strategy for dealing with the
kernels. Since people seeem to use Xen, we have decided not to drop it :-)

So the plan is to re-focus 100% of all Xen kernel efforts onto paravirt_ops.
LKML already has i386 pv_ops + Xen DomU. We intend to build on this to
add:

- x64_64 pv_ops
  - x86_64 Xen DomU on pv_ops
  - i386 & x86_64  Xen Dom0  on pv_ops
  - memory balloon
  - paravirt framebuffer
  - save/restore

All of this based on same LKML release as Fedora bare metal. If all goes to
plan it may even be in the base kernel RPM, instead of kernel-xen, but thats
a minor concern compared to the actual coding.

Getting all this done for Fedora 9 is seriously ambitious, but it is the only
long term sustainable option, other than dropping Xen entirely.

What this means though, is that Fedora 9 Xen will certainly be going through
periods of instability and will certainly be even buggier than normal. F9
may well end up lacking features compared to Xen in Fedora 8 & earlier (eg no
PCI device passthrough, or CPU hotplug). On the plus side though we will be
100% back in sync with bare metal kernel versions & hopefully even have a
lot of this stuff merged in LKML to make ongoing maintainence sustainable.
Short term pain; Long term gain!

I have not got any ETA on when any of these kernel changes will appear in
rawhide - some time before the F9 feature freeze date is best guesstimate.
We will alert people when the time comes. There is a F9 feature page
with some amount of info about the plan...

http://fedoraproject.org/wiki/Features/XenPvops

In terms of Fedora 6/7/8 maintainence... The kernel-xen in these existing 
releases already lags behind the bare metal kernel version by 2-3 releases.
We do not intend to continue trying to rebase the kernel-xen in existing
Fedora releases. It will be essentially important bug-fix mode only. This
is neccessary to enable maximum resources to be focused on the critical
Fedora 9 Xen work.

Regards,
Dan ...on behalf of some very busy Fedora Xen kernel developers :-)

[Tony Coffman]

All I can say is Bravo!

Thank you and the entire Fedora Xen kernel team for undertaking this 
quite substantial effort.

Regards,
--Tony

Daniel P. Berrange wrote:
> So the plan is to re-focus 100% of all Xen kernel efforts onto paravirt_ops.
> LKML already has i386 pv_ops + Xen DomU. We intend to build on this to
> add:
> x64_64 pv_ops
> x86_64 Xen DomU on pv_ops
> i386 & x86_64  Xen Dom0  on pv_ops
> memory balloon
> paravirt framebuffer
> save/restore
>
> All of this based on same LKML release as Fedora bare metal. If all goes to
> plan it may even be in the base kernel RPM, instead of kernel-xen, but thats
> a minor concern compared to the actual coding.
>
> Getting all this done for Fedora 9 is seriously ambitious, but it is the only
> long term sustainable option, other than dropping Xen entirely.

[Paul Wouters]

On Fri, 30 Nov 2007, Daniel P. Berrange wrote:
> This is a friendly alert of the major plans we have for Xen kernels in
> Fedora 9 timeframe...

Thanks for the warning.
> Since we first added Xen in Fedora Core 5, our kernels have been based on
> a forward-port of XenSource's upstream Xen kernels, to new LKML. For a
> long time we ported their 2.6.16 tree to 2.6.18. Now we do ports of their
> 2.6.18 tree to 2.6.21/22/23, etc.

Welcome to my world of KLIPS and NETKEY/XFRM and trying to merge that into
one IPsec stack so that problem goes away. Unfortunately, upstream moves
rather independantly. Though having the lwm.net site, which at least semi
documents API changes, makes the work a bit easier.
> At the same time, upstream Linux gained
> Xen support for i386 DomU, and shortly x86_64  DomU, and is generally
> getting ever more virtualization capabilities.

I am somewhat confused here? Upstream gained xen support but you're forward
porting xen support?
> As everyone knows, we have tended to lag behind Fedora's state-of-the-art
> bare metal kernels by several releases due to the effort required to port
> Xen to newer LKML releases. Despite our best efforts, this lag has been
> getting worse, not better.

Which is mostly due to XenSource going all "windows centric". Which got worse
when they were bought by Citrix. XenSource (an awful name for a company
relying on closed source code for their enterprise version), has not put in
the sources to keep up to date, probably partially also for not getting
included upstream.
> We have taken the decision, that this situation is unacceptable for Fedora 9.
> We simply cannot spend more time forward porting Xen kernels. Either Xen has
> to be dropped entirely, or we need a different strategy for dealing with the
> kernels. Since people seeem to use Xen, we have decided not to drop it :-)

Being onf of those people, thank you very much. And to add to that, xen has
been almost exclusively usable for serious deployment within Fedora. It has
always been far superiod to other distro's integration or using xensource
source code itself. I might have cursed a bit on the initrd sillyness, and
the PAE incompatibility, but other then that, all our servers are now fully
xenified and run more stable then ever before as real iron machines needing
physical reboots too often.
> So the plan is to re-focus 100% of all Xen kernel efforts onto paravirt_ops.
> LKML already has i386 pv_ops + Xen DomU. We intend to build on this to
> add:

I might be mixing up things, but are you saying you are focussing on adding
paravirt to lguest? And replace xen?
> Getting all this done for Fedora 9 is seriously ambitious, but it is the only
> long term sustainable option, other than dropping Xen entirely.

I understand that too well.
> What this means though, is that Fedora 9 Xen will certainly be going through
> periods of instability and will certainly be even buggier than normal. F9
> may well end up lacking features compared to Xen in Fedora 8 & earlier (eg no
> PCI device passthrough, or CPU hotplug). On the plus side though we will be
> 100% back in sync with bare metal kernel versions & hopefully even have a
> lot of this stuff merged in LKML to make ongoing maintainence sustainable.
> Short term pain; Long term gain!

I think most deployments are simple paravirts with no other hardware then virtual
disks and virtual network cards. So that might not be as bad as it sounds.
> Dan ...on behalf of some very busy Fedora Xen kernel developers :-)

Thanks for all the work on virtualization. Most endusers won't really care
what powers it under the hood, as long as they can simply re-use their
disk images of older fedora releases on newer fedora releases with perhaps
only the addition of the newer kernel modules inside the old disk image.

Regards,

Paul

[Daniel P. Berrange]

On Fri, Nov 30, 2007 at 02:44:13PM -0500, Paul Wouters wrote:
> On Fri, 30 Nov 2007, Daniel P. Berrange wrote:
> > At the same time, upstream Linux gained
> > Xen support for i386 DomU, and shortly x86_64  DomU, and is generally
> > getting ever more virtualization capabilities.
>
> I am somewhat confused here? Upstream gained xen support but you're forward
> porting xen support?

Upstream gained support for i386, DomU Xen support only - that's far from
a complete solution, hence F8 still uses a forward port. This F9 plan is
basically about finishing the upstream Xen to support all the features we
need for Fedora & avoid any more forward porting.
> > So the plan is to re-focus 100% of all Xen kernel efforts onto paravirt_ops.
> > LKML already has i386 pv_ops + Xen DomU. We intend to build on this to
> > add:
>
> I might be mixing up things, but are you saying you are focussing on adding
> paravirt to lguest? And replace xen?

No, lguest is just another user of pv_ops. This is explicitly still a Xen
paravirt solution - we'll still have a Xen 3.x hypervisor underneath, with
same Xen 3.0 hypervisor ABI. So F6/7/8 guests should work on F9 host, and
F9 guest should still work on F6/7/8 host. ABI compatability is key because
that's what makes Xen, Xen :-)

lguest (at this time) is still basically a tool for research & development,
not real world production use.
> > What this means though, is that Fedora 9 Xen will certainly be going through
> > periods of instability and will certainly be even buggier than normal. F9
> > may well end up lacking features compared to Xen in Fedora 8 & earlier (eg no
> > PCI device passthrough, or CPU hotplug). On the plus side though we will be
> > 100% back in sync with bare metal kernel versions & hopefully even have a
> > lot of this stuff merged in LKML to make ongoing maintainence sustainable.
> > Short term pain; Long term gain!
>
> I think most deployments are simple paravirts with no other hardware then virtual
> disks and virtual network cards. So that might not be as bad as it sounds.

Yep, we're basically prioritizing our work to address most common & important
areas first. Eventually we may get to stuff like PCI passthrough & CPU hotplug
but its longer term low priority stuff.

Regards,

[Andy Burns]

On 30/11/2007, Paul Wouters  wrote:
> Though having the lwm.net site, which at least semi
> documents API changes, makes the work a bit easier.

I'm sure Living Word Ministries are glad for the link ;-)

[Dale Bewley]

----- "Daniel P. Berrange"  wrote:
> This is a friendly alert of the major plans we have for Xen kernels in
> Fedora 9 timeframe...

Thanks for the pro-active communication! This sounds like a great idea. Is there a list we can lurk on to monitor progress?

--
Dale Bewley - Unix Administrator - Shields Library - UC Davis
GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD  1753 064D 2583 B098 A0F3

[Daniel P. Berrange]

On Fri, Nov 30, 2007 at 01:24:37PM -0800, Dale Bewley wrote:
> ----- "Daniel P. Berrange"  wrote:
> > This is a friendly alert of the major plans we have for Xen kernels in
> > Fedora 9 timeframe...
>
> Thanks for the pro-active communication! This sounds like a great idea. 
> Is there a list we can lurk on to monitor progress?

This list for Fedora significant updates, or xen-devel for specific tech
details, of the virtualization list at OSDL

Dan.