4:35 p.m.
Below is the the text of the interview, didn't really think about the
lead it, but thats trivial...
*1. Please introduce yourself and what you do and how you got started
working on virtualization.*
I'm one of the lead developers for the libvirt project and am actively
involved in many related areas of open source development (qemu/kvm,
xen, gtk-vnc, virt-manager, to name but a few). I also co-maintain many
of these packages in Fedora and RHEL, along with many others in Red
Hat's virtualization team.
More than three years ago (shortly after transferring into Red Hat's
Engineering team, from consulting services) I was working on the OLPC
project. We needed a way to easily test the OS images we were building
without needing real hardware. As a proof of concept, I hacked up a
simple GTK application to run images them under QEMU. At around the same
time Daniel Veillard had started the libvirt project and there was a
desire for a desktop application to manage Xen using libvirt. So I
switched over to the virtualization team, wrote virt-manager for Fedora
6, and my involvement in all areas of open source virtualization grew
from there.
*2. Many people view the work being done on virtualization as a feature
set of major importance and significance. Can you give us a brief
overview of some of the changes that we can expect to see in F11?
*
The open source virtualization development effort is so large now, that
it is useful to discuss each stream in turn.
At the lowest layer is obviously the Linux kernel & KVM/QEMU. There has
been a major acceleration of development in QEMU and push to merge KVM
into the official QEMU source repository. There's ever continuing work
on performance, stability, scalability & reliability in KVM. PCI device
passthrough is one new feature we're highlighting for Fedora 11. The
return of Xen Dom0 was not to be, as the Dom0 paravirt_ops merge with
the upstream Linux kernel is still an ongoing process.
At the middle layer is libvirt, providing a consistent management API
across different virtualization technologies. New features in libvirt,
since F10, include PCI device passthrough for Xen and KVM, the sVirt
security driver using SELinux to protect KVM guests from each other,
thread safety of all libvirt APIs, improved scalability, reliability and
debugging for the libvirtd daemon and support for SCSI HBAs and
copy-on-write volumes in the storage management APIs.
The top layer covers end user tools such as virt-install & virt-manager.
virt-manager is undergoing a significant (and ongoing) overhaul of its
user interface. The first improvements arriving for Fedora 11 are in the
guest installation process and storage management capabilities. As guest
installation is first task most users try, ensuring this is simple and
reliable is key to making a good first impression. Guest desktop
interaction is another historical pain point which has been a focus for
improvements in Fedora 11.
With every release we also try to make a significant step forward in
security of the virtualization stack. In Fedora 11 the focus has been
on SELinux to protect guests from each other and SASL to authenticate
VNC users.
*3. There have been some large changes in virt-manager and libvirt,
which are at the core of the user experience when it comes to
virtualization. Can you talk to use about those some more?*
The guest installation process and desktop interaction are the most
critical areas for making a good first impression. In the virt-manager
re-design the wizard used for installing new guests has been
streamlined, cutting out three redundant steps. Where possible, it will
automatically detect the type of operating system being installed and
choose the best configuration options to optimize for this OS, no longer
requiring the user to figure this out for themselves. The installation
process now directly utilizes the libvirt storage management APIs to
allow easy creation of files in a variety of formats (raw, qcow2, vmdk,
etc), allocation of new local disk partitions or LVM volumes and access
to LUNs exported by iSCSI targets. This is particularly useful when
remotely managing virtualization hosts, allowing regular administrator
tasks to be performed from the virt-manager UI without resorting to
command line SSH sessions.
The mouse pointer has been a constant source of trouble for
virtualization management applications. Getting the guest mouse pointer
to track the host pointer is essentially impossible with the standard
emulated PS/2 mouse. The solution is to provide a pointer device that
supports absolute motion events, instead of relative events that the
PS/2 mouse provides. For KVM and Xen, this means adding a USB tablet
device, but historically Xorg has not been able to automatically
configure this correctly. This is resolved with Fedora 11 guests,
finally providing a guest pointer that moves in perfect sync with the
host, not requiring the pointer to be confined to the guest window.
Users with non-US layout keyboards have also had a hard time getting
their guests to support input of accented/special characters. The VNC
protocol has now been extended to allow the hardware keycodes to be
passed directly from the VNC client to the guest OS without any
intermediate translation step. This should allow the guest OS complete
control over the keyboard layout mapping, without a need for any special
settings on the host.
The final piece of work was to increase the guest desktop resolution.
The real Cirrus video card that QEMU emulates would never have done more
than 800x600, but there are tricks that can be done in a virtual world.
Thus a simple change to the Xorg cirrus driver allows it to detect that
it is using a Cirrus card emulated by QEMU and increase the guest
desktop resolution to 1024x768. Still not great by today's standards,
but better than before. Longer term plans involve replacing the cirrus
driver in QEMU with something more virtualization friendly.
*4. Also, as people should take note of, there has been a lot of work
done surrounding KVM and getting that well integrated into the whole
virtualization setup in Fedora. How has that work been going and has
anything significant been done in that area in this release?*
Fedora was the first major Linux distribution to integrate KVM back in
the Fedora 7 release. It became the default virtualization technology in
Fedora 9, when it became clear we could no longer maintain the separate
Xen host kernel until it was merged in the upstream Linux kernel. The
great benefit of KVM from an distro integration point of view, is that
it is there by default in all new Linux kernels. All that was required
in Fedora was to turn on the module build and make sure the modules are
always loaded when compatible CPUs are found. libvirt and virt-manager
have also both supported KVM since it was first added to Fedora. Thus
there hasn't been a need for much additional integration work for KVM.
The focus has simply been on improving features available to KVM users
via libvirt and virt-manager.
*
5. Glauber Costa has also done significant work merging KVM and QEMU.
Can you explain to us what QEMU is and why the choice was made to merge
it with QEMU and how that is of benefit to the user base?*
Earlier Fedora releases have have suffered from the divergence of QEMU
and KVM code bases. Upstream QEMU has had releases almost 1 year apart,
while KVM has been releasing at least once a month, if not more, using
snapshots of the QEMU source repository. Thus the features available in
QEMU were far behind those available in KVM even though they both shared
the same fundamental code base and upstream development stream. It also
doubled the work package maintainers had todo for security & bug fixes.
Since Fedora 10 though, the upstream QEMU community has accelerated its
release schedule significantly and many of the KVM features have been
merged back into the main QEMU code base. Thus we judged that the time
was right to attempt to ship a single package containing both QEMU and
KVM built from a single code base. For users this means that parity of
features between QEMU and KVM, while the reduced burden on our Fedora
package maintainers, ensures more timely security and bug fixes. Glauber
also took the opportunity to split out all the virtual BIOS files and
ROMs from QEMU into separate packages and ensure all are fully built
from source using appropriate upstream source releases.
*6. Virtualization and Security are two things that are being discussed
more or less hand in hand these days, as the ability to create and use
virtualized machines expands there are many security risks involved. Can
you speak a bit to the work that was done one improving security both at
the kernel level (sVirt) and also the user level with things like SASL
for VNC Auth?*
In each Fedora release we try to make at least one significant step
forward in the security of our virtualization technology. In Fedora 8,
libvirt gained support for secure remote management using TLS for
encryption and x509 client certificates for authentication, while
GTK-VNC, QEMU, KVM and Xen were also all extended to add a VNC extension
for TLS encryption providing a secure remote desktop. In Fedora 9
libvirt was further extended to support SASL enabling secure remote
managment with Kerberos for authentication and PolicyKit for local
desktop authentication. Fedora 9 and 10 also increased use of SELinux to
protect the host operating system from a compromised or malicious
QEMU/KVM process.
The latter still did not provide any protection between guests, so one
compromised QEMU process would still potentially be able to compromise
another on the same host. Thus James Morris started work on a project
known as sVirt, the first results of which are appearing in Fedora 11.
The focus has been to provide isolation between guests running on a
single host. libvirt directly integrates with SELinux to ensure every
QEMU process it launches runs within a dedicated security context, only
able to access its own assigned disk images. This protection is enabled
by default on all Fedora 11 hosts using libvirt for management. As well
as the security benefits, the end user experiance is improved because
libvirt will automatically manage SELinux labelling for all guest disks,
eliminating a major source of bug reports from previous Fedora releases.
*7. These features have all evolved over time and over the previous
Fedora releases and Fedora, as a distribution, has always been a leader
in the virtualization realm. Can you talk a bit now about the actual
process of developing these features and how many of the improvements
and enhancements to virtualization have come about as a direct result of
the work done previously? Also, what does the future look like?*
Virtualization technology in Fedora is reaping the benefit of very
active upstream projects and the significant developer resources of Red
Hat's Virtualization Engineering team. The combination of these provide
great opportunities for new features to have their debut in Fedora releases.
The ideas for new features come from many sources, some from Fedora
end-user experiances & consequent bug reports, some magically arrive on
cue from upstream projects, while others are things that look to be
important for future RHEL releases. With the PCI device passthrough
feature in F11, the core support was all already done by the upstream
KVM community. This is a important feature for future RHEL, so Red Hat
put resources into a F11 feature to add support to libvirt for PCI
passthrough with KVM and Xen and then expose this in virt-manager.
The feature aiming to improve the guest desktop interaction was a result
of the persistent stream of bug reports from Fedora users. We had been
considering ways to address this over course of several Fedora releases,
but it was not until Fedora 11 that all the pieces of the solution
finally came together from the various upstream projects.
The security improvements in virtualization have a different origin.
Very few end users ever file explicit bug reports asking for the
addition of more authentication / encryption features or to use more
SELinux. If anything users ask for ability to more easily turn off
existing security features. This is a case where the user is not always
right. They do want more security, but they just don't know it yet!
As a developer it is neccessary to be very proactive with security
improvements. This can be particularly challenging work because the
solutions often involve working across multiple upstream communities.
Take the VNC SASL authentication feature in Fedora 11 as an example. The
first step was to write a specification for a new VNC security
extension, have it reviewed & get a code for it allocated by RealVNC.
Work on QEMU was required to implement the server side. Work on GTK-VNC
was needed for client side. For management tools, work on libvirt was
required to get the new security type enabled for guests it launches and
finally virt-manager was extended such that it knows how to login.
That's give different projects involved for one feature. This is only
practical by having a close working relationship with all the upstream
communities and carefully coordinating the work there so it all arrives
in time for the next Fedora release.
For the future we're happy that libvirt gained support for managing
VirtualBox recently and likely to soon have a driver for the Open
Nebular cloud management project. Expect more advancements in sVirt,
allowing for tighter controls on what a virtual machine can do, for
example, ability to restrict network access of guests. libvirt will also
gain the ability to manage host network configuration in Fedora 12,
removing the need to manually configure bridge devices. Container based
virt may make a more formal appearance in Fedora 12 as the native Linux
container (LXC) support improves in the kernel and libvirt. The overhaul
of the virt-manager user interface also continues.
*8. Working on Virtualization must be awfully time consuming and very
involved. Do you enjoy it? What do you do to get away from the
pressures of hacking?*
Working on open source virtualization technology is a great experiance
because it is a really interesting & challenging field, having plenty of
talented developers to work with & learn from. There is plenty of work
still to be done at all levels of the stack from kernel/hypervisor right
through to end user applications & not nearly enough time todo it all.
I'm fortunate to be able to spread my work between upstream projects,
the Fedora community and RHEL releases and maintainence. As for free
time ? What free time :-) I try to find time for a photography, with 4
out of my 5 cameras still using film, rather than digital.