I am a newcomer to Fedora (linux) virtualization. However, I have been a long time user of VMware products running first on Red Hat Linux and then on Fedora.
When I recently acquired a CPU with hardware virtualization support (AMD Phenom II 940), I decided to "change my problem set" and give Fedora Virtualization a try ... specifically qemu/kvm/etc.
I use virtuals for three purposes: 1) To test software which might destroy a system or testing which requires a lot of re-booting. 2) For development environment to build rpm packages for various systems, i386/x86_64, etc. 3) To run windows.
OK, install the virtualization packages and then install a simple Fedora 11 guest ... naturally it worked fine.
Now lets get down to real business since I would be installing/running a number of virtual systems. The first thing I found was that configuration files, disk images, etc. were scattered but mostly under /etc/libvirt and /var/lib/libvirt. I looked for a runtime parameter which specified where things were to be stored but did not find anything. Furthermore, all of this stuff was in the root ("/") partition and, when I upgraded to the next release, a pain to bring across.
I have three suggestions (I will put these in bugzilla as soon as someone says what package they should be filed against):
1. Put all files (disk images, configuration, etc.) under a single directory (easier to manage).
2. Provide a virtualization configuration parameter for setting the top directory to be used to store virtualization files (make it easier to get things out of root).
3. Do not require DVD/CD ISO images to be in the image directory and do not screw with SELinux settings on ISO files.
For those unfamiliar with it, this is more or less how VMware stuff sets things up.
So much for wishful thinking, how do I make things easier (and get the files out of root)?
OK, what I have come up with is to create a separate partition for all of the files and then "bind mount" it to /var/lib/libvirt and /etc/libvirt.
Does this make sense? Is this going to work? Any other suggestions?
Gene
From: Gene Czarcinski gene@czarc.net
I am a newcomer to Fedora (linux) virtualization. However, I have been a long time user of VMware products running first on Red Hat Linux and thenon
Fedora.
When I recently acquired a CPU with hardware virtualization support (AMD Phenom II 940), I decided to "change my problem set" and give Fedora Virtualization a try ... specifically qemu/kvm/etc.
I use virtuals for three purposes: 1) To test software which might
destroy a
system or testing which requires a lot of re-booting. 2) For development
environment to build rpm packages for various systems, i386/x86_64, etc.
3)
To run windows.
OK, install the virtualization packages and then install a simple Fedora
11
guest ... naturally it worked fine.
Now lets get down to real business since I would be installing/running a number of virtual systems. The first thing I found was that configuration files, disk images, etc. were scattered but mostly under /etc/libvirt and /var/lib/libvirt. I looked for a runtime parameter which specified where
things were to be stored but did not find anything. Furthermore, all of
this
stuff was in the root ("/") partition and, when I upgraded to the next release, a pain to bring across.
I have three suggestions (I will put these in bugzilla as soon as someone says what package they should be filed against):
- Put all files (disk images, configuration, etc.) under a single
directory
(easier to manage).
- Provide a virtualization configuration parameter for setting the top
directory to be used to store virtualization files (make it easier to get
things out of root).
- Do not require DVD/CD ISO images to be in the image directory and do
not
screw with SELinux settings on ISO files.
For those unfamiliar with it, this is more or less how VMware stuff sets things up.
So much for wishful thinking, how do I make things easier (and get the
files
out of root)?
OK, what I have come up with is to create a separate partition for all of
the
files and then "bind mount" it to /var/lib/libvirt and /etc/libvirt.
Does this make sense? Is this going to work? Any other suggestions?
I have been using qemu-kvm pretty much since it was 1st released. All of my images have always defaulted to /var/lib/libvirt/images/ and have never been "scattered" or anywhere else such as /root. As far as I know CD/DVD ISO images could be stored where ever you wanted them (mine never default to /var/lib/libvirt/). Not sure by what you mean by: "and do not screw with SELinux settings on ISO files".
CONFIDENTIALITY: This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and / or privileged information. Any unauthorized review, use, disclosure or distribution of any kind is strictly prohibited. If you are not the intended recipient, please contact the sender via reply e-mail and destroy all copies of the original message. Thank you.
On Thursday 25 June 2009 10:28:50 Gerry Maddock wrote:
From: Gene Czarcinski gene@czarc.net
I am a newcomer to Fedora (linux) virtualization. However, I have been a long time user of VMware products running first on Red Hat Linux and then on
Fedora.
When I recently acquired a CPU with hardware virtualization support (AMD Phenom II 940), I decided to "change my problem set" and give Fedora Virtualization a try ... specifically qemu/kvm/etc.
I use virtuals for three purposes: 1) To test software which might
destroy a
system or testing which requires a lot of re-booting. 2) For development
environment to build rpm packages for various systems, i386/x86_64, etc.
To run windows.
OK, install the virtualization packages and then install a simple Fedora
11
guest ... naturally it worked fine.
Now lets get down to real business since I would be installing/running a number of virtual systems. The first thing I found was that configuration files, disk images, etc. were scattered but mostly under /etc/libvirt and /var/lib/libvirt. I looked for a runtime parameter which specified where
things were to be stored but did not find anything. Furthermore, all of
this
stuff was in the root ("/") partition and, when I upgraded to the next release, a pain to bring across.
I have three suggestions (I will put these in bugzilla as soon as someone says what package they should be filed against):
- Put all files (disk images, configuration, etc.) under a single
directory
(easier to manage).
- Provide a virtualization configuration parameter for setting the top
directory to be used to store virtualization files (make it easier to get
things out of root).
- Do not require DVD/CD ISO images to be in the image directory and do
not
screw with SELinux settings on ISO files.
For those unfamiliar with it, this is more or less how VMware stuff sets things up.
So much for wishful thinking, how do I make things easier (and get the
files
out of root)?
OK, what I have come up with is to create a separate partition for all of
the
files and then "bind mount" it to /var/lib/libvirt and /etc/libvirt.
Does this make sense? Is this going to work? Any other suggestions?
I have been using qemu-kvm pretty much since it was 1st released. All of my images have always defaulted to /var/lib/libvirt/images/ and have never been "scattered" or anywhere else such as /root. As far as I know CD/DVD ISO images could be stored where ever you wanted them (mine never default to /var/lib/libvirt/). Not sure by what you mean by: "and do not screw with SELinux settings on ISO files".
OK, to properly state my "problem", I need to supply some additional info.
1. All of my Fedora systems are multiboot systems with a small (minimal install) Fedora as a "boot selector" and system to define striped LVM logical volumes (which anaconda does not support). Each system has at least two disk drives (they are pretty cheap these days). There are multiple swap and /boot partitions defined. The remainder of the disk space is put into a single LVM volume group.
2. Basic rule: "do not screw with a working system!" ... Only do fresh installs. NEVER upgrade. This was a hard lesson to learn but it keeps me out of trouble,.
3. Each system has four /boot partitions and four "/" logical volumes (named root1-4) ... think of these as old, current, next, and testing. /home and other data are put into separate logical volumes. One of these separate logical volumes has been used for VMware Virtuals. I keep what goes into a "/" (commonly referred to as "root" in Unix/Linux) to just software, etc. for a specific version of a system (e.g., Fedora 10, Fedora 11, Fedora 12, etc). I expect all other stuff to be in separate partitions or LVM logical volumes which I can mount on my new system.
4. After installing a "new" system such as when Fedora 12 is released, I want to do a minimum of work copying over stuff from the old system to the new system.
5. With the current "libvirt" setup I have disk images under /var/lib/libvert/images and the guest's configuration under /etc/libvirt/qemu. Furthermore, I now have an application which can chew up a lot of disk space and all of it coming out of the "/" logical volume. I currently size my "/" logical volumes to handle a lot of software plus a good amount of temporary storage ... BUT I do not have 150GB "/" partitions/logical-volumes!
5. My current VMware virtuals take about 100GB of storage and I expect Fedora Virtualization to need similar resources. It does not make any sense to me to duplicate the storage for each version of the operating system I have installed.
Now, will mount-bind work? Does anyone have a better suggestion? Leaving it as-is is not an option.
As far as SELinux goes, I read (somewhere in documentation) that a) iso images had to be under /var/lib/libvirt/images and that b) the SELinux context values would be set to "virt_image_t". This may not be the way the software actual works but this is what is described in some related documentation.
Gene
On Thu, Jun 25, 2009 at 12:11:24PM -0400, Gene Czarcinski wrote:
On Thursday 25 June 2009 10:28:50 Gerry Maddock wrote:
- All of my Fedora systems are multiboot systems with a small (minimal
install) Fedora as a "boot selector" and system to define striped LVM logical volumes (which anaconda does not support). Each system has at least two disk drives (they are pretty cheap these days). There are multiple swap and /boot partitions defined. The remainder of the disk space is put into a single LVM volume group.
- Basic rule: "do not screw with a working system!" ... Only do fresh
installs. NEVER upgrade. This was a hard lesson to learn but it keeps me out of trouble,.
- Each system has four /boot partitions and four "/" logical volumes (named
root1-4) ... think of these as old, current, next, and testing. /home and other data are put into separate logical volumes. One of these separate logical volumes has been used for VMware Virtuals. I keep what goes into a "/" (commonly referred to as "root" in Unix/Linux) to just software, etc. for a specific version of a system (e.g., Fedora 10, Fedora 11, Fedora 12, etc). I expect all other stuff to be in separate partitions or LVM logical volumes which I can mount on my new system.
- After installing a "new" system such as when Fedora 12 is released, I want
to do a minimum of work copying over stuff from the old system to the new system.
All these points just say you want some kind of config mgmt tool. In keeping config files in /etc/, libvirt is no different to most other apps. You could bind mount /etc/libvirt elsewhere if you really felt you needed to.
- With the current "libvirt" setup I have disk images under
/var/lib/libvert/images and the guest's configuration under /etc/libvirt/qemu. Furthermore, I now have an application which can chew up a lot of disk space and all of it coming out of the "/" logical volume. I currently size my "/" logical volumes to handle a lot of software plus a good amount of temporary storage ... BUT I do not have 150GB "/" partitions/logical-volumes!
That directory isn't compulsory. It is the default location with correct SELinux labelling, but you can ad other directories using 'semanage', and in very latest libvirt, we automatically relabel disks suitably, no matter where they are located.
As far as SELinux goes, I read (somewhere in documentation) that a) iso images had to be under /var/lib/libvirt/images and that b) the SELinux context values would be set to "virt_image_t". This may not be the way the software actual works but this is what is described in some related documentation.
Since Fedora 11 there should be no need to label files yourself
Daniel
Quoting Gene Czarcinski gene@czarc.net:
As far as SELinux goes, I read (somewhere in documentation) that a) iso images had to be under /var/lib/libvirt/images and that b) the SELinux context values would be set to "virt_image_t". This may not be the way the software actual works but this is what is described in some related documentation.
Considering the uniqueness of this setup, maybe it makes more sense to simply disable SELinux and thus avoid labeling issues altogether?
Also, as mentioned previously, strongly recommend using the virsh (or other libvirt dependent commands) if you're going to use libvirt as the management paradigm. Virsh/Libvirt isn't really a requirement for KVM, it just makes life a little easier.
// Thomas
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
On Thursday 25 June 2009 12:17:52 Thomas Sjolshagen wrote:
Quoting Gene Czarcinski gene@czarc.net:
As far as SELinux goes, I read (somewhere in documentation) that a) iso images had to be under /var/lib/libvirt/images and that b) the SELinux context values would be set to "virt_image_t". This may not be the way the software actual works but this is what is described in some related documentation.
Considering the uniqueness of this setup, maybe it makes more sense to simply disable SELinux and thus avoid labeling issues altogether?
I DO NOT consider this an option. If things do not work with respect to SELinux it is either a bug or I do not understand something.
Also, as mentioned previously, strongly recommend using the virsh (or other libvirt dependent commands) if you're going to use libvirt as the management paradigm. Virsh/Libvirt isn't really a requirement for KVM, it just makes life a little easier.
I am just starting and have (as far as I can tell) all of the Virtualization packages installed. But, before I go too far with this stuff, I need to figure out how to handle the libvirt files so that I can esily move to the next version of the system and so it does not fill my "/" logical volume.
Gene
On 06/25/2009 12:36 PM, Gene Czarcinski wrote:
On Thursday 25 June 2009 12:17:52 Thomas Sjolshagen wrote:
Quoting Gene Czarcinskigene@czarc.net:
As far as SELinux goes, I read (somewhere in documentation) that a) iso images had to be under /var/lib/libvirt/images and that b) the SELinux context values would be set to "virt_image_t". This may not be the way the software actual works but this is what is described in some related documentation.
Considering the uniqueness of this setup, maybe it makes more sense to simply disable SELinux and thus avoid labeling issues altogether?
I DO NOT consider this an option. If things do not work with respect to SELinux it is either a bug or I do not understand something.
These days, SELinux works so well that I can pretty much run in enforcing mode and not realize it -- it just works. I did have small problems with passing USB devices through to virtual guests, and some of the newer apps like puppet which are not virtualization related. I filed bugs. Dan Walsh responds to bug reports very quickly and often suggests a workaround fix. I can't really see SELinux as a big issue.
Bob
On Thursday 25 June 2009 13:13:08 Robert L Cochran wrote:
On 06/25/2009 12:36 PM, Gene Czarcinski wrote:
On Thursday 25 June 2009 12:17:52 Thomas Sjolshagen wrote:
Quoting Gene Czarcinskigene@czarc.net:
As far as SELinux goes, I read (somewhere in documentation) that a) iso images had to be under /var/lib/libvirt/images and that b) the SELinux context values would be set to "virt_image_t". This may not be the way the software actual works but this is what is described in some related documentation.
Considering the uniqueness of this setup, maybe it makes more sense to simply disable SELinux and thus avoid labeling issues altogether?
I DO NOT consider this an option. If things do not work with respect to SELinux it is either a bug or I do not understand something.
These days, SELinux works so well that I can pretty much run in enforcing mode and not realize it -- it just works. I did have small problems with passing USB devices through to virtual guests, and some of the newer apps like puppet which are not virtualization related. I filed bugs. Dan Walsh responds to bug reports very quickly and often suggests a workaround fix. I can't really see SELinux as a big issue.
Agreed most emphatically!!! Someone suggesting turning off SELinux gets my blood pressure rising!
Gene
Well I'm a bit confused because it is hard for me to see the connection you are making to ease of virtualization here. I think you must be asking a question equivalent to this one:
"Is it possible to move Fedora virtual machines from one system to another system despite my complicated LVM disk setup?"
Not being an expert, I don't know the answer, but it should (eventually) be possible for you to move virtual machines from disk to disk. I wonder if you can copy a Fedora virtual machine to a Windows system and then run it in VMWare player. I've seen that done for Ubuntu.
Bob
On 06/25/2009 12:11 PM, Gene Czarcinski wrote:
On Thursday 25 June 2009 10:28:50 Gerry Maddock wrote:
From: Gene Czarcinskigene@czarc.net
I am a newcomer to Fedora (linux) virtualization. However, I have been a long time user of VMware products running first on Red Hat Linux and then on
Fedora.
When I recently acquired a CPU with hardware virtualization support (AMD Phenom II 940), I decided to "change my problem set" and give Fedora Virtualization a try ... specifically qemu/kvm/etc.
I use virtuals for three purposes: 1) To test software which might
destroy a
system or testing which requires a lot of re-booting. 2) For development
environment to build rpm packages for various systems, i386/x86_64, etc.
To run windows.
OK, install the virtualization packages and then install a simple Fedora
11
guest ... naturally it worked fine.
Now lets get down to real business since I would be installing/running a number of virtual systems. The first thing I found was that configuration files, disk images, etc. were scattered but mostly under /etc/libvirt and /var/lib/libvirt. I looked for a runtime parameter which specified where
things were to be stored but did not find anything. Furthermore, all of
this
stuff was in the root ("/") partition and, when I upgraded to the next release, a pain to bring across.
I have three suggestions (I will put these in bugzilla as soon as someone says what package they should be filed against):
- Put all files (disk images, configuration, etc.) under a single
directory
(easier to manage).
- Provide a virtualization configuration parameter for setting the top
directory to be used to store virtualization files (make it easier to get
things out of root).
- Do not require DVD/CD ISO images to be in the image directory and do
not
screw with SELinux settings on ISO files.
For those unfamiliar with it, this is more or less how VMware stuff sets things up.
So much for wishful thinking, how do I make things easier (and get the
files
out of root)?
OK, what I have come up with is to create a separate partition for all of
the
files and then "bind mount" it to /var/lib/libvirt and /etc/libvirt.
Does this make sense? Is this going to work? Any other suggestions?
I have been using qemu-kvm pretty much since it was 1st released. All of my images have always defaulted to /var/lib/libvirt/images/ and have never been "scattered" or anywhere else such as /root. As far as I know CD/DVD ISO images could be stored where ever you wanted them (mine never default to /var/lib/libvirt/). Not sure by what you mean by: "and do not screw with SELinux settings on ISO files".
OK, to properly state my "problem", I need to supply some additional info.
- All of my Fedora systems are multiboot systems with a small (minimal
install) Fedora as a "boot selector" and system to define striped LVM logical volumes (which anaconda does not support). Each system has at least two disk drives (they are pretty cheap these days). There are multiple swap and /boot partitions defined. The remainder of the disk space is put into a single LVM volume group.
- Basic rule: "do not screw with a working system!" ... Only do fresh
installs. NEVER upgrade. This was a hard lesson to learn but it keeps me out of trouble,.
- Each system has four /boot partitions and four "/" logical volumes (named
root1-4) ... think of these as old, current, next, and testing. /home and other data are put into separate logical volumes. One of these separate logical volumes has been used for VMware Virtuals. I keep what goes into a "/" (commonly referred to as "root" in Unix/Linux) to just software, etc. for a specific version of a system (e.g., Fedora 10, Fedora 11, Fedora 12, etc). I expect all other stuff to be in separate partitions or LVM logical volumes which I can mount on my new system.
- After installing a "new" system such as when Fedora 12 is released, I want
to do a minimum of work copying over stuff from the old system to the new system.
- With the current "libvirt" setup I have disk images under
/var/lib/libvert/images and the guest's configuration under /etc/libvirt/qemu. Furthermore, I now have an application which can chew up a lot of disk space and all of it coming out of the "/" logical volume. I currently size my "/" logical volumes to handle a lot of software plus a good amount of temporary storage ... BUT I do not have 150GB "/" partitions/logical-volumes!
- My current VMware virtuals take about 100GB of storage and I expect Fedora
Virtualization to need similar resources. It does not make any sense to me to duplicate the storage for each version of the operating system I have installed.
Now, will mount-bind work? Does anyone have a better suggestion? Leaving it as-is is not an option.
As far as SELinux goes, I read (somewhere in documentation) that a) iso images had to be under /var/lib/libvirt/images and that b) the SELinux context values would be set to "virt_image_t". This may not be the way the software actual works but this is what is described in some related documentation.
Gene
Fedora-virt mailing list Fedora-virt@redhat.com https://www.redhat.com/mailman/listinfo/fedora-virt
On Thu, Jun 25, 2009 at 12:11:24PM -0400, Gene Czarcinski wrote:
- With the current "libvirt" setup I have disk images under
/var/lib/libvirt/images and the guest's configuration under /etc/libvirt/qemu. Furthermore, I now have an application which can chew up a lot of disk space and all of it coming out of the "/" logical volume. I currently size my "/" logical volumes to handle a lot of software plus a good amount of temporary storage ... BUT I do not have 150GB "/" partitions/logical-volumes!
I might be missing something here, but isn't it possible to mount an LVM partition on /var/lib/libvirt, and thus keep all the images on a separate place?
Alternatively use a LVM-based storage pool, so all your guests get their own individual LVs. This will also be a lot faster.
Rich.
Gene Czarcinski wrote:
- Put all files (disk images, configuration, etc.) under a single directory
(easier to manage).
With regard to the configuration being in a different area than the disk images --
/etc/libvirt/qemu/** is _not_ intended to be directly edited by hand. Just as /etc/sudoers is supposed to be edited only by visudo, files under that directory should be edited with "virsh edit", "virsh net-edit", or other libvirt-aware mechanisms.
So, as you-the-user aren't supposed to interact with them directly in any way... arguably it doesn't matter much where they happen to be stored.
On Thursday 25 June 2009 10:39:37 Charles Duffy wrote:
Gene Czarcinski wrote:
- Put all files (disk images, configuration, etc.) under a single
directory (easier to manage).
With regard to the configuration being in a different area than the disk images --
/etc/libvirt/qemu/** is _not_ intended to be directly edited by hand. Just as /etc/sudoers is supposed to be edited only by visudo, files under that directory should be edited with "virsh edit", "virsh net-edit", or other libvirt-aware mechanisms.
So, as you-the-user aren't supposed to interact with them directly in any way... arguably it doesn't matter much where they happen to be stored.
See the reply I made to the other message.
Basically it comes does to this:
1. I have multi-boot systems and NEVER screw with a working system ... I only do fresh installs to a separate partition (the "old" system remains intact).
2. Even if copying files from one system to another is not an issue, the stoage space required for guest disk images can be large. I currently have about 100 GB of space used by VMware virtuals but that is in a separate partition for my "/" partition.
Gene
On Thu, Jun 25, 2009 at 10:05:58AM -0400, Gene Czarcinski wrote:
- Put all files (disk images, configuration, etc.) under a single directory
(easier to manage).
Config files live in /etc, and it makes no sense to put disk images there, so this isn't really viable.
- Provide a virtualization configuration parameter for setting the top
directory to be used to store virtualization files (make it easier to get things out of root).
virt-manager allows you configure storage pools in arbitarry locations, not just on local machine, but also use NFS, LVM, parted, iSCSI, etc. The /var/lib/libvirt directory is just a default starting point which you can freely override if desired.
- Do not require DVD/CD ISO images to be in the image directory and do not
screw with SELinux settings on ISO files.
We don't require ISO files to be in a specific directory anymore and will automatically relabel them to ensure they are accessible by VMs.
Dnaiel
-
Charles Duffy -
Daniel P. Berrange -
Gene Czarcinski -
Gerry Maddock -
Richard W.M. Jones -
Robert L Cochran -
Thomas Sjolshagen