I'm considering switching lmc to use qemu directly instead of virt-install+libvirtd.
Currently lmc can use virt-install to create the full range of supported images. But this doesn't work when running inside mock, which is used by Fedora releng's koji system.
It also supports the --no-virt mode which runs anaconda directly in the same environment as lmc. The benefit is less things to setup and it works inside mock. But the drawback is that it uses device-mapper for installing to a partitioned disk image, and device-mapper doesn't work inside mock. So when running in mock you can only create a live iso or a filesystem image, not anything that needs a partitioned disk.
I've been doing some experimenting and it ends up that qemu works inside mock.
Switching to qemu (with optional auto-detection of when it could use qemu-kvm to speed things up) would simplify the code and allow it to create all the image types no matter where it is run from. It is also possible (I haven't tried this yet) that it could be used to create cross-arch images as well, since qemu has a wide range of supported arches.
On Friday, December 18, 2015 05:26:01 PM Brian C. Lane wrote:
I'm considering switching lmc to use qemu directly instead of virt-install+libvirtd.
Currently lmc can use virt-install to create the full range of supported images. But this doesn't work when running inside mock, which is used by Fedora releng's koji system.
It also supports the --no-virt mode which runs anaconda directly in the same environment as lmc. The benefit is less things to setup and it works inside mock. But the drawback is that it uses device-mapper for installing to a partitioned disk image, and device-mapper doesn't work inside mock. So when running in mock you can only create a live iso or a filesystem image, not anything that needs a partitioned disk.
I've been doing some experimenting and it ends up that qemu works inside mock.
Switching to qemu (with optional auto-detection of when it could use qemu-kvm to speed things up) would simplify the code and allow it to create all the image types no matter where it is run from. It is also possible (I haven't tried this yet) that it could be used to create cross-arch images as well, since qemu has a wide range of supported arches.
Would not be opposed to the change, it should work on platforms where we have no hardware virt capability. We (releng) are reaching out to the lvm team to see what can be done to make device-mapper work in a chroot, if it is possible.
Switching to using qemu would require lmc to learn a lot of details about running vms for different arches. But it would enable making images for non native arches as the commands are mostly the same.
To use kvm would we need to bind mount /dev/kvm into the chroot?
Dennis
On Sun, Dec 20, 2015 at 06:45:40PM -0600, Dennis Gilmore wrote:
On Friday, December 18, 2015 05:26:01 PM Brian C. Lane wrote:
I'm considering switching lmc to use qemu directly instead of virt-install+libvirtd.
Currently lmc can use virt-install to create the full range of supported images. But this doesn't work when running inside mock, which is used by Fedora releng's koji system.
It also supports the --no-virt mode which runs anaconda directly in the same environment as lmc. The benefit is less things to setup and it works inside mock. But the drawback is that it uses device-mapper for installing to a partitioned disk image, and device-mapper doesn't work inside mock. So when running in mock you can only create a live iso or a filesystem image, not anything that needs a partitioned disk.
I've been doing some experimenting and it ends up that qemu works inside mock.
Switching to qemu (with optional auto-detection of when it could use qemu-kvm to speed things up) would simplify the code and allow it to create all the image types no matter where it is run from. It is also possible (I haven't tried this yet) that it could be used to create cross-arch images as well, since qemu has a wide range of supported arches.
Would not be opposed to the change, it should work on platforms where we have no hardware virt capability. We (releng) are reaching out to the lvm team to see what can be done to make device-mapper work in a chroot, if it is possible.
Switching to using qemu would require lmc to learn a lot of details about running vms for different arches. But it would enable making images for non native arches as the commands are mostly the same.
True, I'd encapsulate all that in a way that would make it easy to extend or modify. I think the ability to make images for other arches would be a big plus, even if it is a bit slow.
To use kvm would we need to bind mount /dev/kvm into the chroot?
I haven't tried that yet, but at a minimum, yes. I'm not sure what else is needed to make qemu-kvm work.
On Sat, Dec 19, 2015 at 2:26 AM, Brian C. Lane bcl@redhat.com wrote:
I'm considering switching lmc to use qemu directly instead of virt-install+libvirtd.
Currently lmc can use virt-install to create the full range of supported images. But this doesn't work when running inside mock, which is used by Fedora releng's koji system.
Happy silent nights.
We are now also using lmc to build oVirt Node. And we actually do use it in mock (on Jenkins):
http://jenkins.ovirt.org/job/ovirt-node-ng_master_build-artifacts-fc23-x86_6... http://jenkins.ovirt.org/job/ovirt-node-ng_master_build-artifacts-fc23-x86_6...
It also supports the --no-virt mode which runs anaconda directly in the same environment as lmc. The benefit is less things to setup and it works inside mock. But the drawback is that it uses device-mapper for installing to a partitioned disk image, and device-mapper doesn't work inside mock. So when running in mock you can only create a live iso or a filesystem image, not anything that needs a partitioned disk.
Yes. Also a big -1 on the no-virt approach.
The nice isolation we have when using the VM lmc approach is gone, and this can lead to all sorts of issues 8we have seen this over the years with livecd-creator). SELinux is a big candidate for causing trouble in this setup.
I've been doing some experimenting and it ends up that qemu works inside mock.
Switching to qemu (with optional auto-detection of when it could use qemu-kvm to speed things up) would simplify the code and allow it to create all the image types no matter where it is run from. It is also
That's funny. I actually created this set of scripts to fake lmc while lmc was not running in our Jenkins environment: https://github.com/fabiand/image-tools
Those scripts are also using qemu directly, and I made the same observations as you. It's all working quite well. The drawback is that you probably end up doing a bit of the stuff which virt-install is doing, doing yourself. i.e. --inject-initrd. And I suppose there are more features from virt-install which you require.
possible (I haven't tried this yet) that it could be used to create cross-arch images as well, since qemu has a wide range of supported arches.
Doesn't libvirtd also support to run different qemu arches (or however it's called)? And if so, couldn't virt-install support it aswell?
In the end my take would rather be to stick to virt-install and see how it can be run in mock.
My 2ct.
- fabian
Here's my working branch for using qemu with lmc:
https://github.com/bcl/lorax/tree/master-lmc-qemu
and here's a script that can be run from the git repo that will copy things into the repo and run it with mock:
https://github.com/bcl/lorax/tree/master-lmc-qemu
eg. lmc-mock ~/isos/Fedora-Server-netinst-x86_64-23.iso ./fedora-livemini.ks
This branch works fine for me creating live iso's on my dev system directly, and using the above script to run it inside mock. Note the creation of /dev/kvm, without that it takes about 90 minutes to run. With kvm it finishes in about 10 minutes for my minimal (no-gui) test kickstart.
I haven't tried doing any cross-arch installations yet, at the least they are going to be *very* slow compared to using kvm on a matching arch.
I'm leaning towards changing this for F25 to give people time to give it a try and shake out any bugs. So, give it a try and let me know what you think.
anaconda-devel@lists.fedoraproject.org
-
Brian C. Lane -
Dennis Gilmore -
Fabian Deutsch