While testing F36, I ran into a problem installing virtualization support on Fedora Server. Until now we installed qemu-kvm, libvirt, virt-install, and additionally cockpit-machines. That combo installs about 338 packages, at least 200 packages graphical desktop related (X11, wayland, gtk3, even poppler pdf renderer, see [1]). This breaks Fedora Servers fundamental headless core concept and installs lots of unusable and thus locally uncontrolled/unmaintained software.
== In this respect, a correction is urgently needed. ==
=== qemu-kvm ===
The cause of the problem is the use of package QEMU-kvm, which is aimed - hitherto unnoticed - at installation on a graphical desktop and therefore completely correctly assumes the appropriate packages. This in turn relies on the @virtualization group, which is the only public group supporting virtualization installation and is described accordingly in the Fedora documentation (without mentioning the graphical bias).
In principle, the solution is quite simple. The qemu-kvm developers were wise enough to create a qemu-kvm-core group at the same time, which omits all graphical references and allows for an excellent installation on Fedora Server.
=== libvirt ===
Unfortunately, the maintainers of libvirt are far from this wisdom and insist that the libvirt package still relies on qemu-kvm, thus installing all the packages that QEMU-kvm-core omits again anyway. An analogous libvirt-core does not exist. Instead, Daniel Berrange refers to the list of individually available single packages, from which one (i.e. server WG) should please compile and test a suitable mix and watch the libvirt development steps to see if changes to the mix are required (see bug report by Martin Pitt [2]).
== So we need to find a way to reliably determine a working set of libvirt packages for Fedora Server (or change the tool set). == Maybe we can find support in Debian/Ubuntu, whose libvirt maintainers have solved the problem, at least according to my first impression.
=== Group virtualization-headless ===
An initial approach to a solution could be the group ==@virtualization-headless==. This is a hidden group (see 'dnf group list -v hidden') that currently does pretty much the same as @virtualization, unfortunately. We could contact the owner and modify the group as required (or create a new group @virtualization-server). We could use that as a means to organize a suitable libvirt mix.
=== Cockpit-machines ===
As a further consequence, the cockpit-machines module is currently effectively unusable. It also has QEMU-kvm defined as a dependency. Even if we fix the libvirt problem with a suitable mix, cockpit-machines would currently undo this effort. Martin Pitt is on the way to fix it, see [3]. But we have to find a solution for F36.
I’ll tag this issue for our next meeting. Any suggestions much appreciated.
[1] https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org...
On Mon, Mar 14, 2022 at 9:34 AM Peter Boy pboy@uni-bremen.de wrote:
While testing F36, I ran into a problem installing virtualization support on Fedora Server. Until now we installed qemu-kvm, libvirt, virt-install, and additionally cockpit-machines. That combo installs about 338 packages, at least 200 packages graphical desktop related (X11, wayland, gtk3, even poppler pdf renderer, see [1]). This breaks Fedora Servers fundamental headless core concept and installs lots of unusable and thus locally uncontrolled/unmaintained software.
== In this respect, a correction is urgently needed. ==
=== qemu-kvm ===
The cause of the problem is the use of package QEMU-kvm, which is aimed - hitherto unnoticed - at installation on a graphical desktop and therefore completely correctly assumes the appropriate packages. This in turn relies on the @virtualization group, which is the only public group supporting virtualization installation and is described accordingly in the Fedora documentation (without mentioning the graphical bias).
It does not install a graphical desktop. It installs graphical environment support packages for guests and hosts. Installing a graphical desktop is quite a different thing altogether.
In principle, the solution is quite simple. The qemu-kvm developers were wise enough to create a qemu-kvm-core group at the same time, which omits all graphical references and allows for an excellent installation on Fedora Server.
It's not quite that simple. The core version also doesn't install support for guest hardware acceleration and other things that would be useful for guests. For example, things like virglrender, spice, qxl, etc. stuff all require host components that you deem "unwanted".
=== libvirt ===
Unfortunately, the maintainers of libvirt are far from this wisdom and insist that the libvirt package still relies on qemu-kvm, thus installing all the packages that QEMU-kvm-core omits again anyway. An analogous libvirt-core does not exist. Instead, Daniel Berrange refers to the list of individually available single packages, from which one (i.e. server WG) should please compile and test a suitable mix and watch the libvirt development steps to see if changes to the mix are required (see bug report by Martin Pitt [2]).
== So we need to find a way to reliably determine a working set of libvirt packages for Fedora Server (or change the tool set). == Maybe we can find support in Debian/Ubuntu, whose libvirt maintainers have solved the problem, at least according to my first impression.
You will not. The maintainers for the Debian packages are the same group of folks that work on the RPM packages in Fedora. The split is synchronized between the Fedora spec and the Debian packaging upstream.
=== Group virtualization-headless ===
An initial approach to a solution could be the group ==@virtualization-headless==. This is a hidden group (see 'dnf group list -v hidden') that currently does pretty much the same as @virtualization, unfortunately. We could contact the owner and modify the group as required (or create a new group @virtualization-server). We could use that as a means to organize a suitable libvirt mix.
=== Cockpit-machines ===
As a further consequence, the cockpit-machines module is currently effectively unusable. It also has QEMU-kvm defined as a dependency. Even if we fix the libvirt problem with a suitable mix, cockpit-machines would currently undo this effort. Martin Pitt is on the way to fix it, see [3]. But we have to find a solution for F36.
Instead of using the virtualization group, just use the cockpit-machines package to pull in the headless components. It's not useful without it anyway.
However, most people will want *all* the virtualization components rather than the "core" version.
Am 14.03.2022 um 15:55 schrieb Neal Gompa ngompa13@gmail.com:
=== qemu-kvm ===
The cause of the problem is the use of package QEMU-kvm, which is aimed - hitherto unnoticed - at installation on a graphical desktop and therefore completely correctly assumes the appropriate packages. This in turn relies on the @virtualization group, which is the only public group supporting virtualization installation and is described accordingly in the Fedora documentation (without mentioning the graphical bias).
It does not install a graphical desktop. It installs graphical environment support packages for guests and hosts. Installing a graphical desktop is quite a different thing altogether.
I hope, my wording is not that misleading. I wrote "installation =on= a graphical desktop“ and not "installation =of= a graphical desktop“. We don't want „graphical environment support packages" for server hosting VMs, that’s the idea of headless. Using virtualization on a desktop is different, there we need "graphical environment support packages“, indeed. And therefore we need different set of packages for installation. Therefore, qwmu-kvm and qemu-kvm-core is a great combination and libvirt and libvirt-core would be if our libvirt maintainers could get around to it.
In principle, the solution is quite simple. The qemu-kvm developers were wise enough to create a qemu-kvm-core group at the same time, which omits all graphical references and allows for an excellent installation on Fedora Server.
It's not quite that simple. The core version also doesn't install support for guest hardware acceleration and other things that would be useful for guests. For example, things like virglrender, spice, qxl, etc. stuff all require host components that you deem "unwanted".
What I "deem unwanted" is unused software installed. Do you really need poppler on a headless Server hosting virtual machines in a datacenter?
And what hardware acceleration? The typical server, even tagged with 10T or more, has a lot of expensive and high-performance equipment, but no graphics hardware with significant acceleration potential. And for those use cases that can make use of graphics hardware acceleration (and equipped with the latest Nvidia gizmo), you may have to install the full set of packages and not just the core. It could be as simple as that.
Maybe we can find support in Debian/Ubuntu, whose libvirt maintainers have solved the problem, at least according to my first impression.
You will not. The maintainers for the Debian packages are the same group of folks that work on the RPM packages in Fedora. The split is synchronized between the Fedora spec and the Debian packaging upstream.
Even better. At least Debian / Ubuntu are able to install libvirt on a headless server without installing about 200+ unused packages. We are (currently) not.
=== Cockpit-machines ===
As a further consequence, the cockpit-machines module is currently effectively unusable. It also has QEMU-kvm defined as a dependency. Even if we fix the libvirt problem with a suitable mix, cockpit-machines would currently undo this effort. Martin Pitt is on the way to fix it, see [3]. But we have to find a solution for F36.
Instead of using the virtualization group, just use the cockpit-machines package to pull in the headless components. It'snot useful without it anyway.
Why (or what) is it „not useful without (Cockpit) anyway“?
Currently it doesn’t help, because cockpit-machines pulls in the same set of software. And in the long term we must be able to provide virtualization support without depending on Cockpit.
However, most people will want *all* the virtualization components rather than the "core" version.
Of the 4 discussants here so far, you are the only one. That is not exactly „most“.
You are right about desktops. But we talk about server here. And most server system administrators don’t want wayland or X11 or GTK installed on their headless servers. Many of them consider even cockpit superfluous. And they don't need a hardware accelerated terminal, from 9600 baud to 14,400 or lightning fast 19200 baud (sorry, just kidding on a serious topic).
What I really miss - unfortunately - is a constructive proposal which of the 200+ software packages we can use on a hosting server and for what purpose or use case. And which we will definitely not.
And it would be great if you would contribute your knowledge and expertise to our server WG, and put such a differentiation into practice.
Best Peter
On Mon, Mar 14, 2022 at 8:31 PM Peter Boy pboy@uni-bremen.de wrote:
Am 14.03.2022 um 15:55 schrieb Neal Gompa ngompa13@gmail.com:
=== qemu-kvm ===
The cause of the problem is the use of package QEMU-kvm, which is aimed - hitherto unnoticed - at installation on a graphical desktop and therefore completely correctly assumes the appropriate packages. This in turn relies on the @virtualization group, which is the only public group supporting virtualization installation and is described accordingly in the Fedora documentation (without mentioning the graphical bias).
It does not install a graphical desktop. It installs graphical environment support packages for guests and hosts. Installing a graphical desktop is quite a different thing altogether.
I hope, my wording is not that misleading. I wrote "installation =on= a graphical desktop“ and not "installation =of= a graphical desktop“. We don't want „graphical environment support packages" for server hosting VMs, that’s the idea of headless. Using virtualization on a desktop is different, there we need "graphical environment support packages“, indeed. And therefore we need different set of packages for installation. Therefore, qwmu-kvm and qemu-kvm-core is a great combination and libvirt and libvirt-core would be if our libvirt maintainers could get around to it.
In principle, the solution is quite simple. The qemu-kvm developers were wise enough to create a qemu-kvm-core group at the same time, which omits all graphical references and allows for an excellent installation on Fedora Server.
It's not quite that simple. The core version also doesn't install support for guest hardware acceleration and other things that would be useful for guests. For example, things like virglrender, spice, qxl, etc. stuff all require host components that you deem "unwanted".
What I "deem unwanted" is unused software installed. Do you really need poppler on a headless Server hosting virtual machines in a datacenter?
And what hardware acceleration? The typical server, even tagged with 10T or more, has a lot of expensive and high-performance equipment, but no graphics hardware with significant acceleration potential. And for those use cases that can make use of graphics hardware acceleration (and equipped with the latest Nvidia gizmo), you may have to install the full set of packages and not just the core. It could be as simple as that.
Paravirtualized hardware eliminates performance drops caused by emulating hardware. That's done by doing "guest hardware acceleration" with hypervisor-oriented hardware (virtio, qxl, etc.) which passes through straight to the host.
You can knock out a virtual host with high IP traffic with an emulated network card because it will peg the CPU, for example. I've done that enough times to know that you don't want that. Same goes for "graphics", "input", etc.
They also make it possible to do things like clipboard redirection, remote USB device attach, etc.
Maybe we can find support in Debian/Ubuntu, whose libvirt maintainers have solved the problem, at least according to my first impression.
You will not. The maintainers for the Debian packages are the same group of folks that work on the RPM packages in Fedora. The split is synchronized between the Fedora spec and the Debian packaging upstream.
Even better. At least Debian / Ubuntu are able to install libvirt on a headless server without installing about 200+ unused packages. We are (currently) not.
It's *always* been possible with Fedora to do that. Both distributions also (sensibly) default to providing a fully featured stack. The difference is that our stack uses metapackages instead of weak dependencies because that makes it easier for project/product dependency mixes (oVirt, OpenStack, KubeVirt, Cockpit, etc.).
=== Cockpit-machines ===
As a further consequence, the cockpit-machines module is currently effectively unusable. It also has QEMU-kvm defined as a dependency. Even if we fix the libvirt problem with a suitable mix, cockpit-machines would currently undo this effort. Martin Pitt is on the way to fix it, see [3]. But we have to find a solution for F36.
Instead of using the virtualization group, just use the cockpit-machines package to pull in the headless components. It'snot useful without it anyway.
Why (or what) is it „not useful without (Cockpit) anyway“?
Currently it doesn’t help, because cockpit-machines pulls in the same set of software. And in the long term we must be able to provide virtualization support without depending on Cockpit.
However, most people will want *all* the virtualization components rather than the "core" version.
Of the 4 discussants here so far, you are the only one. That is not exactly „most“.
You are right about desktops. But we talk about server here. And most server system administrators don’t want wayland or X11 or GTK installed on their headless servers. Many of them consider even cockpit superfluous. And they don't need a hardware accelerated terminal, from 9600 baud to 14,400 or lightning fast 19200 baud (sorry, just kidding on a serious topic).
What I really miss - unfortunately - is a constructive proposal which of the 200+ software packages we can use on a hosting server and for what purpose or use case. And which we will definitely not.
And it would be great if you would contribute your knowledge and expertise to our server WG, and put such a differentiation into practice.
I have extensive knowledge of the virtualization stack, how it's packaged, and why because I have worked with it for many years. However, I am personally stretched so thinly right now that I literally cannot do more than I'm doing now for Fedora.
Am 15.03.2022 um 01:49 schrieb Neal Gompa ngompa13@gmail.com:
On Mon, Mar 14, 2022 at 8:31 PM Peter Boy pboy@uni-bremen.de wrote:
Am 14.03.2022 um 15:55 schrieb Neal Gompa ngompa13@gmail.com:
=== qemu-kvm === ...
Paravirtualized hardware eliminates performance drops caused by emulating hardware. That's done by doing "guest hardware acceleration" with hypervisor-oriented hardware (virtio, qxl, etc.) which passes through straight to the host.
You can knock out a virtual host with high IP traffic with an emulated network card because it will peg the CPU, for example. I've done that enough times to know that you don't want that. Same goes for "graphics", "input", etc.
Agreement on the benefits of paravirtualization. But
They also make it possible to do things like clipboard redirection, remote USB device attach, etc.
How can I do clipboard redirection on a headless hosting server? How will I attach a USB stick in a server rack?
What am I doing with clipboard redirection on a VM providing a Web server, mail hub, and similar services, that are the "bread and butter" applications on VMs. In these cases we don’t need those 200+ graphic related packages, that libvirt pulls in without regard to whether they are needed.
My intention is just to provide lean server installation for the 'bred and butter' use case. And of course, virtualization for the desktop is untouched by this and remains as it is. And we may extend the server virtualization support for use cases, where a remote desktop system (e.g. kind of Citrix) should run on a VM and needs enhanced graphics support. I don’t know if this would be feasible with Fedora resources, but just in case.
Even better. At least Debian / Ubuntu are able to install libvirt on a headless server without installing about 200+ unused packages. We are (currently) not.
It's *always* been possible with Fedora to do that. Both distributions also (sensibly) default to providing a fully featured stack. The difference is that our stack uses metapackages instead of weak dependencies because that makes it easier for project/product dependency mixes (oVirt, OpenStack, KubeVirt, Cockpit, etc.).
Possible yes, but not doable for now. There is no way to install cockpit-machines without accepting 200+ packages not usable. I hope, Martin can change that and can do that in not a too distant future.
And you can’t use libvirt without accepting those 200+ packages, because we have no group or meta package and no valid information which packages are required. Support by developer/maintainer as well as documentation are sparse, a system administrator must try and test back and forth. Not a good state of affairs for a reliable product.
I have extensive knowledge of the virtualization stack, how it's packaged, and why because I have worked with it for many years. However, I am personally stretched so thinly right now that I literally cannot do more than I'm doing now for Fedora.
I understand that. As a very gentle reminder, you applied for Server WG membership and committed to contribute N hours of work per week on it's chores (a short version of my favorite quote from Stephen Smoogen in the revitalization discussion last year).
Perhaps you could use your knowledge and experience to suggest, for example, a set of libvirt individual packages that we may start with to emulate a libvirt-core metapackage. And you could coordinate with Martin that we use the same or compatible set of dependant packages. Not that we end in the same situation as currently.
Just 2 concrete suggestions that I hope are within your time allotment.
Best Peter
Peter Boy [2022-03-15 1:22 +0100]:
As a further consequence, the cockpit-machines module is currently effectively unusable. It also has QEMU-kvm defined as a dependency. Even if we fix the libvirt problem with a suitable mix, cockpit-machines would currently undo this effort. Martin Pitt is on the way to fix it, see [3]. But we have to find a solution for F36.
I sent https://github.com/cockpit-project/cockpit-machines/pull/622 to trim cockpit-machine's RPM dependencies. Today is release day, so this should make it into the F35/36/Rawhide package updates today.
Thanks for pointing out!
Martin
Am 15.03.2022 um 06:25 schrieb Martin Pitt mpitt@redhat.com:
I sent https://github.com/cockpit-project/cockpit-machines/pull/622 to trim cockpit-machine's RPM dependencies. Today is release day, so this should make it into the F35/36/Rawhide package updates today.
This is really great news (and incredibly fast). Thank you very much for your effort.
By the way, I’m writing documentation about using Cockpit for Server administration (see https://docs.stg.fedoraproject.org/en-US/fedora-server/virtualization-vm-ins... or my Fedora Magazin article). I have not been able to find documentation on the machines module. Do you have a link for me? Or can you give me 1-2 sentences what the „session“ option in the „Create VM“ worksheet does (available if not logged in as root)? Thanks a lot!
Peter
Hello Peter,
Peter Boy [2022-03-15 8:29 +0100]:
Am 15.03.2022 um 06:25 schrieb Martin Pitt mpitt@redhat.com:
I sent https://github.com/cockpit-project/cockpit-machines/pull/622 to trim cockpit-machine's RPM dependencies. Today is release day, so this should make it into the F35/36/Rawhide package updates today.
off by one.. release day is *tomorrow*, sorry 😅
This is really great news (and incredibly fast). Thank you very much for your effort.
No worries! I'm glad to clean this up, this has annoyed me for a long time also in our infra (where I reduced the libvirt deps yesterday already).
I have not been able to find documentation on the machines module. Do you have a link for me?
I'm afraid there currently isn't any user-facing docs for cockpit-machines specifically. The generic libvirt documentation explains the concepts, what you can do, etc. -- there is no c-machines specific functionality.
Or can you give me 1-2 sentences what the „session“ option in the „Create VM“ worksheet does (available if not logged in as root)? Thanks a lot!
It's explained here:
https://wiki.libvirt.org/page/FAQ#What_is_the_difference_between_qemu:.2F.2F...
I recommend it as such: Use "system" for production deployments, and "session" for testing/development/experimention as long as you can.
"session" does not support any custom/advanced networking, but works pretty much everwhere (including containers) and without any privileges. "system" requires a *lot* of privileges, but is independent from any user being logged in.
Martin
Hello Martin,
sorry for the late answer.
Am 15.03.2022 um 11:23 schrieb Martin Pitt mpitt@redhat.com:
Or can you give me 1-2 sentences what the „session“ option in the „Create VM“ worksheet does (available if not logged in as root)? Thanks a lot!
It's explained here:
https://wiki.libvirt.org/page/FAQ#What_is_the_difference_between_qemu:.2F.2F...
I recommend it as such: Use "system" for production deployments, and "session" for testing/development/experimention as long as you can.
"session" does not support any custom/advanced networking, but works pretty much everwhere (including containers) and without any privileges. "system" requires a *lot* of privileges, but is independent from any user being logged in.
Thank you very much for the information. I included it in the rsp. article or our server documentation (currently still staging).
Peter
Hello again,
Martin Pitt [2022-03-15 6:25 +0100]:
Peter Boy [2022-03-15 1:22 +0100]:
As a further consequence, the cockpit-machines module is currently effectively unusable. It also has QEMU-kvm defined as a dependency. Even if we fix the libvirt problem with a suitable mix, cockpit-machines would currently undo this effort. Martin Pitt is on the way to fix it, see [3]. But we have to find a solution for F36.
I sent https://github.com/cockpit-project/cockpit-machines/pull/622 to trim cockpit-machine's RPM dependencies. Today is release day, so this should make it into the F35/36/Rawhide package updates today.
FTR, not going to happen today, I'm afraid. There is a lof of impedance mismatches and bugs, packages really don't get tested a lot without the "pull in half a desktop" metapackage, apparently. This needs a lot more yak shaving until it actually works, so I hope I have something in two weeks.
Pitti
server@lists.fedoraproject.org
-
Martin Pitt -
Neal Gompa -
Peter Boy