PATCH: Disk (appliance) creator tool
by Daniel P. Berrange
This patch adds support for another installation variant. The livecd-creator
produces ISO images which boot using syslinux. The image-creator creates a
single file containing a filesystem in which the OS is installated. This patch
adds a new disk-creator, which creates a single file containing a partitioned
disk with potentially many filesystems in which the OS is installed. Furthermore
it installs grub in the boot sector, so this image is immediately bootable in
a virtual machine.
The idea is that this serves as a tool for creating pre-installed appliances
suitable for distribution to end users, with no further installation steps
required, beyond potentially an appliance specific firstboot script.
I've used this tool to create an instance of the oVirt web management appliance
from its kickstart recipe. Currently developers building the appliance have to
boot a VM using the F8 boot.iso and run the kickstart script in the VM and so
on. While this works, involves many steps with potential for failure. This new
tool reduces the problem to simply
disk-creator --name ovirt-wui-demo.dsk ovirt-wui-appliance.ks
A short while later you end up with 'ovirt-wui-demo.dks' which you can boot
straight up in KVM / Xen / etc.
With the background information out of the way, here's some info about the
changes in this patch...
Most of the change here involves re-factoring the imgcreate/fs.py module.
The SparseExtLoopbackMount, SparseLoopbackMount, LoopbackMount classes all
have the built-in limitation that the image being produced corresponds to
a single filesystem / loop mount. With the disk creator, the image being
produced can be partitioned into multiple chunks, with many filesystems.
Furthermore, not all of them require loopback mounts, as the partitions
themselves are already visible via /dev/mapper/
So this patch separates the roles. There are now classes which deal with
accessing / creating disks:
Disk - generic base for disks
RawDisk - a disk backed by a block device
LoopbackDisk - a disk backed by a file
SparseLoopbackDisk - a disk backed by a sparse file
The 'create' method must make the disk visible as a block device - eg
by calling losetup. For RawDisk, this is obviously a no-op. The 'cleanup'
method must undo the 'create' operation.
There are then classes which deal with mounting things:
Mount - generic base for mounts
DiskMount - able to mount a Disk object
ExtDiskMount - able to format/resize ext3 filesystems when mounting
The livecd-creator/image-creator tools are updated to take account of these
API changes.
Next, up we have a new class 'PartitionedMount'. This takes a Disk object
and adds one or more partitions to it. It then creates further Disk and
Mount object instances for each partition. So, mounting a 'PartitionedMount'
object will in fact mount all its partitions (formatting the filesystems
or swap space as needed).
Currently only ext3/swap is supported for partition types. We use parted
to create partitions. This works well enough, but parted will spew lots
of scary looking warnings about being unable to tell the kernel to reload
its partition table. This is because loop devices don't support partition
tables. In this scenario though its not a problem because we then use the
kpartx tool to add entries to /dev/mapper for each partition.
When mounting partitions, PartitionedMount, figures out the correct order
based on the mount points, so it ensures the / is mounted before /boot.
The imgcreator/disk.py class contains the DiskImageCreator class which is
a subclass of ImageCreator. This uses the PartitionedMount object to
do an installation of the kickstart. It is fairly simple at the moment,
only being able to cope with 'part' entries in the kickstart which are
either ext3 or swap, and which have an explicit size given - ie --grow
is not supported. For testing I've been using
part /boot --fstype ext3 --size=100 --ondisk=sda
part swap --fstype swap --size=500 --ondisk=sda
part /var/lib/pgsql --fstype ext3 --size=300 --ondisk=sda
part / --fstype ext3 --size=3000 --ondisk=sda
part /home --fstype ext3 --size=500 --ondisk=sda
part /var --fstype ext3 --size=1000 --ondisk=sda
Which ensures it correctly deals with adding extended and logical partitions.
It doesn't currently care about --ondisk only outputting a single disk
image, but I intend to extend the CLI to allow creation of installs which
spawn multiple output disks.
Aside from partitioning, DiskImageCreator will setup an fstab file containing
entries for all the filesystems and swap space. It will add the grub device
map file, and create a grub.cfg based off the installed kernel. It is able
to cope with a layout where grub config is on /, or a separate /boot partition.
We can't use grub-install since it doesn't understand loopdevices, so to do
the MBR install we invoke grub manually.
If the disk image happens to be the same size or smaller than one of your
USB flash drives, you can also dd the entire image straight to the driver
with no modifications required. It should 'just boot' in any machine able
to boot off USB
Future enhancements for this:
- Autosize partitions when --grow is used
- Support LVM volumes
- Allow splitting across multiple disks (sda, sdb, etc)
- Deal with disk filesytem labels
- Deal with non-ext3 filesystems
- Install to pre-existing block device / LVM vol instead of file
- When staging final image, use qemu-img to generate qcow2, vmdk
or raw files at user's choice. qcow2 would give huge size savings
- Output XML file for use with 'virt-image' tool, so that the entire
build and deploy process consists of nothing more than:
# disk-creator ovirt-wui-appliance.ks
# virt-image ovirt-wui-appliance.xml
The last two options are probably my most immediate TODO items because this
will result in an excellant virtual appliance creation tool for Fedora
derived distros.
Perhaps it should be called appliance-creator instead of disk-creator ?
Final note - the patch contains a tonne of 'print' statements for debugging.
Debugging this stuff can be 'fun' when things go wrong, so I think it may
be useful if we make use of the python logging module throughout the livecd
tools. If folks agree, I'll switch the 'print' to logging.debug(), otherwise
I'll just remove them entirely before reposting this patch
The attached patch changes:
API | 4
Makefile | 3
imgcreate/__init__.py | 3
imgcreate/creator.py | 17 +
imgcreate/disk.py | 20 +-
imgcreate/fs.py | 425 ++++++++++++++++++++++++++++++++++++++-----------
imgcreate/kickstart.py | 3
imgcreate/live.py | 6
livecd-tools.spec | 1
9 files changed, 376 insertions(+), 106 deletions(-)
I'm also attaching the kickstart file I'm using as a demonstration...
Regards,
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
16 years, 2 months
i586 needed, how?
by Warren Togami
anaconda --targetarch=i586 would install i586 packages instead of i686.
I need chroot-creator to do the equivalent so the resulting chroot can
work on both i586 and i686 clients. i586 clients are common with LTSP.
It was suggested that I test "setarch i586" after removing
/etc/platform/rpm. The resulting chroot was still i686.
Any suggestions as to what is the best way the equivalent of anaconda's
--targetarch= can be implemented in ImageCreator?
Warren Togami
wtogami(a)redhat.com
16 years, 2 months
imgcreate/creator.py imgcreate/kickstart.py
by Jeremy Katz
imgcreate/creator.py | 8 +++++++-
imgcreate/kickstart.py | 10 ++++++----
2 files changed, 13 insertions(+), 5 deletions(-)
New commits:
commit e98a30558acc58275ee2c4469e3c4f8bea9b566f
Author: Warren Togami <wtogami(a)redhat.com>
Date: Wed Feb 20 14:39:47 2008 -0500
selinux --disabled fixes
The attached patch seems to fix "selinux --disabled" and other parts
related to selinux. It ensures that if SELinux is disabled, no labels are
set on files as well as not running restorecon or bind-mounting /selinux
diff --git a/imgcreate/creator.py b/imgcreate/creator.py
index c7b1046..11fce2d 100644
--- a/imgcreate/creator.py
+++ b/imgcreate/creator.py
@@ -435,10 +435,14 @@ class ImageCreator(object):
# bind mount system directories into _instroot
for (f, dest) in [("/sys", None), ("/proc", None), ("/dev", None),
- ("/dev/pts", None), ("/selinux", None),
+ ("/dev/pts", None),
(cachesrc, "/var/cache/yum")]:
self.__bindmounts.append(BindChrootMount(f, self._instroot, dest))
+ # /selinux should only be mounted if selinux is enabled (enforcing or permissive)
+ if kickstart.selinux_enabled(self.ks):
+ self.__bindmounts.append(BindChrootMount("/selinux", self._instroot, None))
+
self._do_bindmounts()
os.symlink("../proc/mounts", self._instroot + "/etc/mtab")
@@ -551,6 +555,8 @@ class ImageCreator(object):
if kickstart.exclude_docs(self.ks):
rpm.addMacro("_excludedocs", "1")
+ if not kickstart.selinux_enabled(self.ks):
+ rpm.addMacro("__file_context_path", "%{nil}")
try:
self.__select_packages(ayum)
diff --git a/imgcreate/kickstart.py b/imgcreate/kickstart.py
index a7e0723..30156d8 100644
--- a/imgcreate/kickstart.py
+++ b/imgcreate/kickstart.py
@@ -370,7 +370,7 @@ class SelinuxConfig(KickstartConfig):
f = file(path, "w+")
os.chmod(path, 0644)
- if not ksselinux.selinux:
+ if ksselinux.selinux == ksconstants.SELINUX_DISABLED:
return
if not os.path.exists(self.path("/sbin/restorecon")):
return
@@ -381,9 +381,11 @@ class SelinuxConfig(KickstartConfig):
if os.path.exists(self.path("/usr/sbin/lokkit")):
args = ["/usr/sbin/lokkit", "-f", "--quiet", "--nostart"]
- if ksselinux.selinux:
+ if ksselinux.selinux == ksconstants.SELINUX_ENFORCING:
args.append("--selinux=enforcing")
- else:
+ if ksselinux.selinux == ksconstants.SELINUX_PERMISSIVE:
+ args.append("--selinux=permissive")
+ if ksselinux.selinux == ksconstants.SELINUX_DISABLED:
args.append("--selinux=disabled")
self.call(args)
@@ -483,4 +485,4 @@ def get_post_scripts(ks):
return scripts
def selinux_enabled(ks):
- return ks.handler.selinux.selinux
+ return ks.handler.selinux.selinux == ksconstants.SELINUX_ENFORCING
16 years, 2 months
Why --bind mount /var/cache/yum?
by Warren Togami
imgcreate/creator.py:
# bind mount system directories into _instroot
for (f, dest) in [("/sys", None), ("/proc", None), ("/dev", None),
("/dev/pts", None),
(cachesrc, "/var/cache/yum")]:
self.__bindmounts.append(BindChrootMount(f, self._instroot,
dest))
Anyone know why we are --bind mounting /var/cache/yum from the host into
the chroot during install when we aren't using it?
Warren Togami
wtogami(a)redhat.com
16 years, 2 months
bootloader --append="noapic" does nothing?
by Asheesh Laroia
I noticed when testing my Live CD on a Dell at the Creative Commons office
that it would not boot without the noapic option. (The kernel complained
about this, and told me to send in a debug report for APIC, which I might
do later. We were planning to have this disc out very soon; it'd be great
if the above issue were fixable for this release.)
So, at the advice of
http://www.informit.com/articles/article.aspx?p=1157197&seqNum=4 , I added
this toward the start of my Kickstart file:
bootloader --append="noapic"
Alas, the resulting disc still does not pass "noapic" to the kernel. What
am I supposed to do?
http://buildbot.creativecommons.org:8080/builders/livecontent%20builder/b...
is the long, long log of us building that CD, in case any of that output
is useful.
-- Asheesh.
--
The whole of life is futile unless you consider it as a sporting proposition.
16 years, 2 months
[PATCH] selinux --disabled fixes
by Warren Togami
The attached patch seems to fix "selinux --disabled" and other parts
related to selinux. I have only tested it with chroot-creator with the
"selinux --disabled" case in kickstart. Existing users might want to
verify that it doesn't change their behavior because this makes changes
to the base classes.
Warren Togami
wtogami(a)redhat.com
16 years, 2 months
Simple persistance idea
by don vogt
I've seen messages about complete persistence, which
is, make sure all
changes to the filesystem are persistent.
However, for end user tasks it might be sufficient to
offer persistence
of the /home directory.
Kai Said----------------------------------------
Has the following idea been proposed already?
I figure it would be easy to do.
During boot, scan for a partition with a given label,
say, "fedoralivehome".
If present, that partition gets mounted as /home.
As a consequence, all user settings and documents are
kept across boots.
An usb stick could be partitioned to have the second
partition as /home.
Does this idea make sense?
Thanks,
Kai
Kai said--------------------------------------------
That is essentially the way Centos -5.0 works
(worked?)
although it saves /etc also. I posted this on this
list as a suggestion quite a while ago, but i don't
know what happened. I supposed that people smarter
than me had better ideas. The CentOS implemention is
pretty simple. They put a change in the local startup
init script that looks for a set name. aproximatlty
CENTOSBACKUP and then untars what is in it. On exit
another script in the shutdown sequence asks if you
want to save your data and if so where to save it. If
you answer yes the script tars /home and /etc into
that file. Since it is open software, I guess you
could just pirate the parts of the script you need.
There must be many holes in this procedure. I am
going back in my hole.
16 years, 2 months
ImageCreator "selinux --disable" problem
by Warren Togami
I am trying to reproduce the install behavior of anaconda --noselinux,
where it installs a chroot without labels. [1] I need this for LTSP due
to SELinux chroot limitations, and Dan Walsh confirms that this is my
best option given these current limitations.
First I discovered places in kickstart.py where it is supposed to be
checking that SELinux-from-kickstart file setting was always returning
true. I believe the attached patch fixes this part, although it could
use some review.
To my dismay it continued to install with labels. I then realized that
creator.py's ImageCreator mount() method unconditionally bind mounted
the system's /selinux directory, which is incorrect if "selinux
--disabled" is defined in the kickstart file.
Perhaps my understanding of python is not advanced, but it appears that
there is no good way to check kickstart's selinux setting from the
mount() method due to the way it is abstracted.
Any ideas what could be done here?
Warren Togami
wtogami(a)redhat.com
[1]
OK, it really does have labels, but all the labels are the same.
16 years, 2 months
Storing the image as UDF not IS9660
by Asheesh Laroia
In adding the data I wanted to the Live CD, I quickly ran into the
filename length limitation of ISO9660/Joliet. If I'm willing to break the
Joliet standard, I can get files of length 102 on the disc. Normally the
standard lets me use 64-character filenames.
Meanwhile, as I understand things, UDF is the normal filesystem for DVDs,
is pretty widely-supported, and supports 255-character-long paths, perhaps
longer <http://en.wikipedia.org/wiki/Comparison_of_file_systems>. It
would even work for CDs; it's not as if you'd have to stop supporting
generating CDs. Right now, you generate DVDs as large CDs; using UDF
would let you generate CDs as if they were small DVDs. (Both work on
most OSs, and besides, You would just switch to
Have you guys thought about making livecd-creator output UDF-format
filesystems rather than the limited ISO9660?
-- Asheesh.
--
* Knghtbrd is FAR too tempted to .sig this entire discussion...
16 years, 2 months
