I have a rather large set of RPMs and such on my host I want to install on my guest using libguestfs. The normal way to do this would be to upload the package(s), install them, then (optionally) remove the RPMs.
However, I don't particularly want to bloat the qcow2 file with the changes made via uploading files which are only going to be deleted when no longer in use. Someday (*sigh*) we'll have 9p-over-virtio support built into qemu; until then, a few ways to get around this present themselves:
- Add a temporary file as a disk to the guestfs session on startup; within the guest, create a new filesystem there, transfer temporary files to it, and delete when done.
Pro: No external dependencies. Con: Not sure of a safe and correct way to find the name of the temporary disk [to ensure that the mkfs operation being scripted refers only to it], particularly when trying to script everything into a single guestfish invocation with a variable number of other disks exposed (ie. using a command line generated by virt-inspect). Upload needs to be file-by-file (or have everything bundled in a tarball beforehand).
- Run a tmpfs mount on the guest; upload RPMs there.
Pro: Fast. No need to worry about device names. Con: Potential for running out of memory. Upload needs to be file-by-file (or have everything bundled in a tarball beforehand).
- Stuff the files to be added into a cramfs filesystem, running mkcramfs on the host.
Pro: Relatively fast. Easy to transfer a full directory structure to the guest at one stroke. Con: Device names an issue. Requires cramfs.ko to be added to the whitelist in appliance/update-initramfs.sh.in.
- Support a FUSE-based mechanism for exporting files, ie. ccgfs-over-vmchannel
Pro: Potential to be the cleanest approach from an end-user perspective. Con: Implementation difficulty; no Fedora or EPEL packages are available for ccgfs.
On Thu, May 28, 2009 at 8:19 PM, Charles Duffy charles@dyfis.net wrote:
I have a rather large set of RPMs and such on my host I want to install on my guest using libguestfs. The normal way to do this would be to upload the package(s), install them, then (optionally) remove the RPMs.
However, I don't particularly want to bloat the qcow2 file with the changes made via uploading files which are only going to be deleted when no longer in use. Someday (*sigh*) we'll have 9p-over-virtio support built into qemu; until then, a few ways to get around this present themselves:
What about setting up a temporary NFS export on the host and mounting that on the guest? I've done installs using yum/rpm directly from the NFS mount, eliminating the need to ever copy the RPMs into anything more than working memory. If network security is an issue, I'm sure you could use a combination of iptables rules and NFS export options to limit access to that single box as well (though, I haven't done that one).
Here's to hoping I didn't miss something blatantly obvious...
Michael
Michael Ansel wrote:
On Thu, May 28, 2009 at 8:19 PM, Charles Duffy charles@dyfis.net wrote:
I have a rather large set of RPMs and such on my host I want to install on my guest using libguestfs. The normal way to do this would be to upload the package(s), install them, then (optionally) remove the RPMs.
However, I don't particularly want to bloat the qcow2 file with the changes made via uploading files which are only going to be deleted when no longer in use. Someday (*sigh*) we'll have 9p-over-virtio support built into qemu; until then, a few ways to get around this present themselves:
What about setting up a temporary NFS export on the host and mounting that on the guest? I've done installs using yum/rpm directly from the NFS mount, eliminating the need to ever copy the RPMs into anything more than working memory. If network security is an issue, I'm sure you could use a combination of iptables rules and NFS export options to limit access to that single box as well (though, I haven't done that one).
As libguestfs (and the software I'm building with it) runs unprivileged and unattended, an added requirement of NFS server support on the host would mean a pretty massive increase in security exposure and administrative load. (Also, the libguestfs appliance doesn't included nfs-utils at present, and I don't know that NFS works over qemu's SLiRP-based "-net user" mode anyhow).
If I were going to go that route, I'd probably lean towards using qemu's existing SMB server support -- but there would still be changes to the guest needed, making it less than pain-free.
[For the moment, I ended up using squashfs; cramfs, mentioned in my previous email, was a Really Bad Idea on account of the size limitations]
On Fri, May 29, 2009 at 12:38:46AM -0400, Michael Ansel wrote:
On Thu, May 28, 2009 at 8:19 PM, Charles Duffy charles@dyfis.net wrote:
I have a rather large set of RPMs and such on my host I want to install on my guest using libguestfs. The normal way to do this would be to upload the package(s), install them, then (optionally) remove the RPMs.
However, I don't particularly want to bloat the qcow2 file with the changes made via uploading files which are only going to be deleted when no longer in use. Someday (*sigh*) we'll have 9p-over-virtio support built into qemu; until then, a few ways to get around this present themselves:
What about setting up a temporary NFS export on the host and mounting that on the guest? I've done installs using yum/rpm directly from the NFS mount, eliminating the need to ever copy the RPMs into anything more than working memory. If network security is an issue, I'm sure you could use a combination of iptables rules and NFS export options to limit access to that single box as well (though, I haven't done that one).
The original libguestfs plan was to use NFS for all file transfers. The reason why we don't intend to do this anymore is twofold: NFS requires root on the host side, and NFS requires either matching UIDs or complex UID-UID mappings. If we were going to implement any protocol, it'd be FTP, because FTP clients are common, can be run non-root, and don't need complicated UID mappings.
Rich.
On Fri, May 29, 2009 at 2:27 AM, Richard W.M. Jones rjones@redhat.com wrote:
The original libguestfs plan was to use NFS for all file transfers. The reason why we don't intend to do this anymore is twofold: NFS requires root on the host side, and NFS requires either matching UIDs or complex UID-UID mappings. If we were going to implement any protocol, it'd be FTP, because FTP clients are common, can be run non-root, and don't need complicated UID mappings.
Okay, I think I'm not completely understanding how libguestfs works, so that is probably why I'm getting confused. *adds that to to-do list* I was looking at it figuring: 1) root (UID=0, everywhere) is the only one installing packages, so the UID mapping lines up perfectly every time 2) you only need root access on the host once when you are building the VM; you don't need NFS after it is already built
I have a feeling this will all become clear to me once I do some research *self-chastisement for speaking before reading*, but for now I need sleep, so that reading shall come tomorrow.
Also, on a side note: Is there a reason all of my replies are automatically directed at individuals vs the list? (I keep having to retype fedora-virt@redhat.com) Or is this another one of those things where I didn't read something I was supposed to...
Thanks!
Michael
Michael Ansel wrote:
- root (UID=0, everywhere) is the only one installing packages, so
the UID mapping lines up perfectly every time
libguestfs in general, and host filesystem access in particular, is good for more than just installing packages, though admittedly that's my immediate use case.
- you only need root access on the host once when you are building
the VM; you don't need NFS after it is already built
Even if there were no use case for access to files off the host other than early installation --
Why is it acceptable for libguestfs to require root access *ever*, when so many alternate transport mechanisms (9p, ccgfs, FTP, qemu's built in smb, etc) are available which wouldn't create this constraint?
If I'm building a piece of software invoked by untrusted users in a lab environment, I don't want that software to need, or have any means to access, root. Ever. For anything, at all.
By the way -- I'm tired too (normal wakeup time in ~3.25 hours), so please excuse my tone if it's a bit harsh.
On Fri, May 29, 2009 at 03:34:21AM -0400, Michael Ansel wrote:
- root (UID=0, everywhere) is the only one installing packages, so
the UID mapping lines up perfectly every time
No - the concept of UIDs in libguestfs is non-intuitive.
Firstly the normal mode of operation is to run libguestfs as non-root. There are plenty of reasons why this is a good idea, I think Charles covered a few of them. So everything we do is oriented around making sure that we don't need to be root to do libguestfs operations.
However, libguestfs runs qemu/kvm (also as non-root) but that boots a mini appliance. *Inside* the appliance the appliance kernel thinks everything is running as root. It's not really from the point of view of the host machine, but inside the appliance that's how it looks.
If we were to put an NFS server inside the appliance and tunnel the connections out to the host, then the host would have to mount the drive as root (or we'd have to fake it somehow - non-trivial with NFS).
If we put an FTP server inside the appliance, things are better. FTP clients are widely available, either standalone or as libraries for many programming languages. They just need to be able to make a TCP connection, so there is no requirement for special privileges. On the server side (inside the appliance) the FTP server is running as "root" so it can make arbitrary changes to the filesystem.
- you only need root access on the host once when you are building
the VM; you don't need NFS after it is already built
With libguestfs currently, you never need root at all, not to build libguestfs, not to run it, not for any operations using it.
Rich.
On May 29, 2009, at 2:44, Richard W.M. Jones wrote:
If we put an FTP server inside the appliance, things are better. FTP clients are widely available, either standalone or as libraries for many programming languages. They just need to be able to make a TCP connection, so there is no requirement for special privileges. On the server side (inside the appliance) the FTP server is running as "root" so it can make arbitrary changes to the filesystem.
Please make FTP die already! :-)
How about HTTP (WebDAV) or - better - just use the ssh file transfer protocol?
- ask
On Fri, May 29, 2009 at 03:13:10AM -0700, Ask Bjørn Hansen wrote:
On May 29, 2009, at 2:44, Richard W.M. Jones wrote:
If we put an FTP server inside the appliance, things are better. FTP clients are widely available, either standalone or as libraries for many programming languages. They just need to be able to make a TCP connection, so there is no requirement for special privileges. On the server side (inside the appliance) the FTP server is running as "root" so it can make arbitrary changes to the filesystem.
Please make FTP die already! :-)
How about HTTP (WebDAV) or - better - just use the ssh file transfer protocol?
ssh isn't too widely supported (from programming language libraries). Does libssh2 even support scp?
WebDAV is possible, but it comes down to how much crap it will depend upon, all of which needs to go into the appliance.
FTP is widely supported and well understood, and the servers are old, small, self-contained code. I really think it's better for this, and not just because I once wrote an FTP server ...
Rich.
Richard W.M. Jones wrote:
FTP is widely supported and well understood, and the servers are old, small, self-contained code. I really think it's better for this, and not just because I once wrote an FTP server ...
Just to weigh in -- my only concern with FTP is that it's limited in what metadata it will transport. ccgfs, for instance, supports full POSIX ACLs and xattrs. Certainly, that's overkill for the common use cases here -- but maybe something to keep in mind.
On Fri, May 29, 2009 at 06:13:05AM -0500, Charles Duffy wrote:
Richard W.M. Jones wrote:
FTP is widely supported and well understood, and the servers are old, small, self-contained code. I really think it's better for this, and not just because I once wrote an FTP server ...
Just to weigh in -- my only concern with FTP is that it's limited in what metadata it will transport. ccgfs, for instance, supports full POSIX ACLs and xattrs. Certainly, that's overkill for the common use cases here -- but maybe something to keep in mind.
I don't think you'd want to transfer ACLs/xattrs from the host into the guest because that implies some fairly close alignment between user / permissioning model between guest & host, cf why NFS isn't a good idea for this. It likeyl be more useful just let the uploaded files get the default ACL/xattr and ue a guestfs command to change them after the fact if needed
Daniel
On Fri, May 29, 2009 at 06:13:05AM -0500, Charles Duffy wrote:
Richard W.M. Jones wrote:
FTP is widely supported and well understood, and the servers are old, small, self-contained code. I really think it's better for this, and not just because I once wrote an FTP server ...
Just to weigh in -- my only concern with FTP is that it's limited in what metadata it will transport. ccgfs, for instance, supports full POSIX ACLs and xattrs. Certainly, that's overkill for the common use cases here -- but maybe something to keep in mind.
In addition to Dan's reply, as long as 'tar' supports it, you should be able to upload it using the 'tar-in' or 'tgz-in' commands:
http://et.redhat.com/~rjones/libguestfs/guestfs.3.html#guestfs_tgz_in
I have also added support for cramfs & squashfs to the appliance, which is yet another way to 'get stuff inside' without needing time- consuming uploads.
Rich.
On Fri, May 29, 2009 at 12:10:05PM +0100, Richard W.M. Jones wrote:
On Fri, May 29, 2009 at 03:13:10AM -0700, Ask Bjørn Hansen wrote:
On May 29, 2009, at 2:44, Richard W.M. Jones wrote:
If we put an FTP server inside the appliance, things are better. FTP clients are widely available, either standalone or as libraries for many programming languages. They just need to be able to make a TCP connection, so there is no requirement for special privileges. On the server side (inside the appliance) the FTP server is running as "root" so it can make arbitrary changes to the filesystem.
Please make FTP die already! :-)
How about HTTP (WebDAV) or - better - just use the ssh file transfer protocol?
ssh isn't too widely supported (from programming language libraries). Does libssh2 even support scp?
WebDAV is possible, but it comes down to how much crap it will depend upon, all of which needs to go into the appliance.
FTP is widely supported and well understood, and the servers are old, small, self-contained code. I really think it's better for this, and not just because I once wrote an FTP server ...
I tend to agree with this - ssh is overkill here - the host <->guest channel is already secure by virtue of using vmchannel, so using ssh is not really required. FTP is a really easy to implement, low overhead protocol that'd fit nicely here. I can't really think of what WebDAV would do for this use case that FTP can't do in a simpler manner.
Daniel
"Daniel P. Berrange" berrange@redhat.com writes:
On Fri, May 29, 2009 at 12:10:05PM +0100, Richard W.M. Jones wrote:
On Fri, May 29, 2009 at 03:13:10AM -0700, Ask Bjørn Hansen wrote:
On May 29, 2009, at 2:44, Richard W.M. Jones wrote:
If we put an FTP server inside the appliance, things are better. FTP clients are widely available, either standalone or as libraries for many programming languages. They just need to be able to make a TCP connection, so there is no requirement for special privileges. On the server side (inside the appliance) the FTP server is running as "root" so it can make arbitrary changes to the filesystem.
Please make FTP die already! :-)
How about HTTP (WebDAV) or - better - just use the ssh file transfer protocol?
ssh isn't too widely supported (from programming language libraries). Does libssh2 even support scp?
WebDAV is possible, but it comes down to how much crap it will depend upon, all of which needs to go into the appliance.
FTP is widely supported and well understood, and the servers are old, small, self-contained code. I really think it's better for this, and not just because I once wrote an FTP server ...
I tend to agree with this - ssh is overkill here - the host <->guest channel is already secure by virtue of using vmchannel, so using ssh is not really required. FTP is a really easy to implement, low overhead protocol that'd fit nicely here. I can't really think of what WebDAV would do for this use case that FTP can't do in a simpler manner.
Daniel
Sounds like even TFTP would do.
On May 29, 2009, at 4:10, Richard W.M. Jones wrote:
just use the ssh file transfer protocol?
ssh isn't too widely supported (from programming language libraries). Does libssh2 even support scp?
libcurl does. On the server side running sshd (and having sftp-server installed) seems pretty manageable.
WebDAV is possible, but it comes down to how much crap it will depend upon, all of which needs to go into the appliance.
nginx has a reasonable WebDAV implementation and few dependencies IIRC. I don't think WebDAV is worth it though; the only real benefit is that it'd make it easy to make a "browse this disk from your web browser" feature.
FTP is widely supported and well understood, and the servers are old, small, self-contained code. I really think it's better for this, and not just because I once wrote an FTP server ...
As long as it's just host->guest communication I agree, it makes no difference. If it's ever extended to remote host -> guest communication FTP will just bring tears and misery.
- ask
On Fri, May 29, 2009 at 5:44 AM, Richard W.M. Jones rjones@redhat.com wrote:
On Fri, May 29, 2009 at 03:34:21AM -0400, Michael Ansel wrote:
- root (UID=0, everywhere) is the only one installing packages, so
the UID mapping lines up perfectly every time
No - the concept of UIDs in libguestfs is non-intuitive.
Firstly the normal mode of operation is to run libguestfs as non-root. There are plenty of reasons why this is a good idea, I think Charles covered a few of them. So everything we do is oriented around making sure that we don't need to be root to do libguestfs operations.
However, libguestfs runs qemu/kvm (also as non-root) but that boots a mini appliance. *Inside* the appliance the appliance kernel thinks everything is running as root. It's not really from the point of view of the host machine, but inside the appliance that's how it looks.
- you only need root access on the host once when you are building
the VM; you don't need NFS after it is already built
With libguestfs currently, you never need root at all, not to build libguestfs, not to run it, not for any operations using it.
First, I just want to say, you guys are amazing: when do you sleep?!
I also want to apologize to the list for my somewhat mis- and un-informed comments last night/early this morning. I do however think I have figured out where my confusion was: I was under the impression that running a virtual machine required root access and libguestfs worked by launching said VM (in order to "run commands in the context of the guest"). It seems I was grossly mistaken on both: virtual machines can run as non-root (though, KVM is currently root only -- possibly changing), and libguestfs does only a miniature launch of the system (? I'm still a little unclear on how the virtual appliance in libguestfs works). I also did not realize that libguestfs was designed with an explicit goal of not requiring root access for anything. In hindsight, I *knew* you didn't have to be root to launch a VM: I've done it countless times to test boot CDs; for some reason (probably because I was up later than normal), I just forgot that fact.
Again, I'm very sorry for my ill-informed comments, and have officially decided I'm no longer allowed to post on mailing lists after midnight!
Michael Ansel
On Thu, May 28, 2009 at 07:19:06PM -0500, Charles Duffy wrote:
- Add a temporary file as a disk to the guestfs session on startup;
within the guest, create a new filesystem there, transfer temporary files to it, and delete when done.
Pro: No external dependencies. Con: Not sure of a safe and correct way to find the name of the temporary disk [to ensure that the mkfs operation being scripted refers only to it], particularly when trying to script everything into a single guestfish invocation with a variable number of other disks exposed (ie. using a command line generated by virt-inspect). Upload needs to be file-by-file (or have everything bundled in a tarball beforehand).
Or you could prepare the disk image in advance using a separate guestfish operation:
http://et.redhat.com/~rjones/libguestfs/recipes.html#tar2vm
When you attach a new disk it has a predictable name, ie. /dev/hda, /dev/hdb and so on. If you know that your guest has only one real disk image, then your prepared image will always be /dev/hdb.
This is what our V2V scripts do in order to upload new RPMs into the host, although at the moment it's not very well tested.
- Run a tmpfs mount on the guest; upload RPMs there.
Pro: Fast. No need to worry about device names. Con: Potential for running out of memory. Upload needs to be file-by-file (or have everything bundled in a tarball beforehand).
The amount of memory available is quite limited.
- Stuff the files to be added into a cramfs filesystem, running mkcramfs
on the host.
Pro: Relatively fast. Easy to transfer a full directory structure to the guest at one stroke. Con: Device names an issue. Requires cramfs.ko to be added to the whitelist in appliance/update-initramfs.sh.in.
We should add this anyway -- please file a bug in bugzilla.
- Support a FUSE-based mechanism for exporting files, ie.
ccgfs-over-vmchannel
Pro: Potential to be the cleanest approach from an end-user perspective. Con: Implementation difficulty; no Fedora or EPEL packages are available for ccgfs.
And the other option would be:
Implement the FTP server!
Rich.
-
Ask Bjørn Hansen -
Charles Duffy -
Daniel P. Berrange -
Markus Armbruster -
Michael Ansel -
Richard W.M. Jones