livecd February 2008

livecd@lists.fedoraproject.org

35 participants
63 discussions

PATCH: Disk (appliance) creator tool
by Daniel P. Berrange 21 Feb '08

21 Feb '08

This patch adds support for another installation variant. The livecd-creator produces ISO images which boot using syslinux. The image-creator creates a single file containing a filesystem in which the OS is installated. This patch adds a new disk-creator, which creates a single file containing a partitioned disk with potentially many filesystems in which the OS is installed. Furthermore it installs grub in the boot sector, so this image is immediately bootable in a virtual machine. The idea is that this serves as a tool for creating pre-installed appliances suitable for distribution to end users, with no further installation steps required, beyond potentially an appliance specific firstboot script. I've used this tool to create an instance of the oVirt web management appliance from its kickstart recipe. Currently developers building the appliance have to boot a VM using the F8 boot.iso and run the kickstart script in the VM and so on. While this works, involves many steps with potential for failure. This new tool reduces the problem to simply disk-creator --name ovirt-wui-demo.dsk ovirt-wui-appliance.ks A short while later you end up with 'ovirt-wui-demo.dks' which you can boot straight up in KVM / Xen / etc. With the background information out of the way, here's some info about the changes in this patch... Most of the change here involves re-factoring the imgcreate/fs.py module. The SparseExtLoopbackMount, SparseLoopbackMount, LoopbackMount classes all have the built-in limitation that the image being produced corresponds to a single filesystem / loop mount. With the disk creator, the image being produced can be partitioned into multiple chunks, with many filesystems. Furthermore, not all of them require loopback mounts, as the partitions themselves are already visible via /dev/mapper/ So this patch separates the roles. There are now classes which deal with accessing / creating disks: Disk - generic base for disks RawDisk - a disk backed by a block device LoopbackDisk - a disk backed by a file SparseLoopbackDisk - a disk backed by a sparse file The 'create' method must make the disk visible as a block device - eg by calling losetup. For RawDisk, this is obviously a no-op. The 'cleanup' method must undo the 'create' operation. There are then classes which deal with mounting things: Mount - generic base for mounts DiskMount - able to mount a Disk object ExtDiskMount - able to format/resize ext3 filesystems when mounting The livecd-creator/image-creator tools are updated to take account of these API changes. Next, up we have a new class 'PartitionedMount'. This takes a Disk object and adds one or more partitions to it. It then creates further Disk and Mount object instances for each partition. So, mounting a 'PartitionedMount' object will in fact mount all its partitions (formatting the filesystems or swap space as needed). Currently only ext3/swap is supported for partition types. We use parted to create partitions. This works well enough, but parted will spew lots of scary looking warnings about being unable to tell the kernel to reload its partition table. This is because loop devices don't support partition tables. In this scenario though its not a problem because we then use the kpartx tool to add entries to /dev/mapper for each partition. When mounting partitions, PartitionedMount, figures out the correct order based on the mount points, so it ensures the / is mounted before /boot. The imgcreator/disk.py class contains the DiskImageCreator class which is a subclass of ImageCreator. This uses the PartitionedMount object to do an installation of the kickstart. It is fairly simple at the moment, only being able to cope with 'part' entries in the kickstart which are either ext3 or swap, and which have an explicit size given - ie --grow is not supported. For testing I've been using part /boot --fstype ext3 --size=100 --ondisk=sda part swap --fstype swap --size=500 --ondisk=sda part /var/lib/pgsql --fstype ext3 --size=300 --ondisk=sda part / --fstype ext3 --size=3000 --ondisk=sda part /home --fstype ext3 --size=500 --ondisk=sda part /var --fstype ext3 --size=1000 --ondisk=sda Which ensures it correctly deals with adding extended and logical partitions. It doesn't currently care about --ondisk only outputting a single disk image, but I intend to extend the CLI to allow creation of installs which spawn multiple output disks. Aside from partitioning, DiskImageCreator will setup an fstab file containing entries for all the filesystems and swap space. It will add the grub device map file, and create a grub.cfg based off the installed kernel. It is able to cope with a layout where grub config is on /, or a separate /boot partition. We can't use grub-install since it doesn't understand loopdevices, so to do the MBR install we invoke grub manually. If the disk image happens to be the same size or smaller than one of your USB flash drives, you can also dd the entire image straight to the driver with no modifications required. It should 'just boot' in any machine able to boot off USB Future enhancements for this: - Autosize partitions when --grow is used - Support LVM volumes - Allow splitting across multiple disks (sda, sdb, etc) - Deal with disk filesytem labels - Deal with non-ext3 filesystems - Install to pre-existing block device / LVM vol instead of file - When staging final image, use qemu-img to generate qcow2, vmdk or raw files at user's choice. qcow2 would give huge size savings - Output XML file for use with 'virt-image' tool, so that the entire build and deploy process consists of nothing more than: # disk-creator ovirt-wui-appliance.ks # virt-image ovirt-wui-appliance.xml The last two options are probably my most immediate TODO items because this will result in an excellant virtual appliance creation tool for Fedora derived distros. Perhaps it should be called appliance-creator instead of disk-creator ? Final note - the patch contains a tonne of 'print' statements for debugging. Debugging this stuff can be 'fun' when things go wrong, so I think it may be useful if we make use of the python logging module throughout the livecd tools. If folks agree, I'll switch the 'print' to logging.debug(), otherwise I'll just remove them entirely before reposting this patch The attached patch changes: API | 4 Makefile | 3 imgcreate/__init__.py | 3 imgcreate/creator.py | 17 + imgcreate/disk.py | 20 +- imgcreate/fs.py | 425 ++++++++++++++++++++++++++++++++++++++----------- imgcreate/kickstart.py | 3 imgcreate/live.py | 6 livecd-tools.spec | 1 9 files changed, 376 insertions(+), 106 deletions(-) I'm also attaching the kickstart file I'm using as a demonstration... Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

3 12

i586 needed, how?
by Warren Togami 21 Feb '08

21 Feb '08

anaconda --targetarch=i586 would install i586 packages instead of i686. I need chroot-creator to do the equivalent so the resulting chroot can work on both i586 and i686 clients. i586 clients are common with LTSP. It was suggested that I test "setarch i586" after removing /etc/platform/rpm. The resulting chroot was still i686. Any suggestions as to what is the best way the equivalent of anaconda's --targetarch= can be implemented in ImageCreator? Warren Togami wtogami(a)redhat.com

4 4

imgcreate/creator.py imgcreate/kickstart.py
by Jeremy Katz 20 Feb '08

20 Feb '08

imgcreate/creator.py | 8 +++++++- imgcreate/kickstart.py | 10 ++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) New commits: commit e98a30558acc58275ee2c4469e3c4f8bea9b566f Author: Warren Togami <wtogami(a)redhat.com> Date: Wed Feb 20 14:39:47 2008 -0500 selinux --disabled fixes The attached patch seems to fix "selinux --disabled" and other parts related to selinux. It ensures that if SELinux is disabled, no labels are set on files as well as not running restorecon or bind-mounting /selinux diff --git a/imgcreate/creator.py b/imgcreate/creator.py index c7b1046..11fce2d 100644 --- a/imgcreate/creator.py +++ b/imgcreate/creator.py @@ -435,10 +435,14 @@ class ImageCreator(object): # bind mount system directories into _instroot for (f, dest) in [("/sys", None), ("/proc", None), ("/dev", None), - ("/dev/pts", None), ("/selinux", None), + ("/dev/pts", None), (cachesrc, "/var/cache/yum")]: self.__bindmounts.append(BindChrootMount(f, self._instroot, dest)) + # /selinux should only be mounted if selinux is enabled (enforcing or permissive) + if kickstart.selinux_enabled(self.ks): + self.__bindmounts.append(BindChrootMount("/selinux", self._instroot, None)) + self._do_bindmounts() os.symlink("../proc/mounts", self._instroot + "/etc/mtab") @@ -551,6 +555,8 @@ class ImageCreator(object): if kickstart.exclude_docs(self.ks): rpm.addMacro("_excludedocs", "1") + if not kickstart.selinux_enabled(self.ks): + rpm.addMacro("__file_context_path", "%{nil}") try: self.__select_packages(ayum) diff --git a/imgcreate/kickstart.py b/imgcreate/kickstart.py index a7e0723..30156d8 100644 --- a/imgcreate/kickstart.py +++ b/imgcreate/kickstart.py @@ -370,7 +370,7 @@ class SelinuxConfig(KickstartConfig): f = file(path, "w+") os.chmod(path, 0644) - if not ksselinux.selinux: + if ksselinux.selinux == ksconstants.SELINUX_DISABLED: return if not os.path.exists(self.path("/sbin/restorecon")): return @@ -381,9 +381,11 @@ class SelinuxConfig(KickstartConfig): if os.path.exists(self.path("/usr/sbin/lokkit")): args = ["/usr/sbin/lokkit", "-f", "--quiet", "--nostart"] - if ksselinux.selinux: + if ksselinux.selinux == ksconstants.SELINUX_ENFORCING: args.append("--selinux=enforcing") - else: + if ksselinux.selinux == ksconstants.SELINUX_PERMISSIVE: + args.append("--selinux=permissive") + if ksselinux.selinux == ksconstants.SELINUX_DISABLED: args.append("--selinux=disabled") self.call(args) @@ -483,4 +485,4 @@ def get_post_scripts(ks): return scripts def selinux_enabled(ks): - return ks.handler.selinux.selinux + return ks.handler.selinux.selinux == ksconstants.SELINUX_ENFORCING

1 0

Why --bind mount /var/cache/yum?
by Warren Togami 20 Feb '08

20 Feb '08

imgcreate/creator.py: # bind mount system directories into _instroot for (f, dest) in [("/sys", None), ("/proc", None), ("/dev", None), ("/dev/pts", None), (cachesrc, "/var/cache/yum")]: self.__bindmounts.append(BindChrootMount(f, self._instroot, dest)) Anyone know why we are --bind mounting /var/cache/yum from the host into the chroot during install when we aren't using it? Warren Togami wtogami(a)redhat.com

2 1

bootloader --append="noapic" does nothing?
by Asheesh Laroia 20 Feb '08

20 Feb '08

I noticed when testing my Live CD on a Dell at the Creative Commons office that it would not boot without the noapic option. (The kernel complained about this, and told me to send in a debug report for APIC, which I might do later. We were planning to have this disc out very soon; it'd be great if the above issue were fixable for this release.) So, at the advice of http://www.informit.com/articles/article.aspx?p=1157197&seqNum=4 , I added this toward the start of my Kickstart file: bootloader --append="noapic" Alas, the resulting disc still does not pass "noapic" to the kernel. What am I supposed to do? http://buildbot.creativecommons.org:8080/builders/livecontent%20builder/bui… is the long, long log of us building that CD, in case any of that output is useful. -- Asheesh. -- The whole of life is futile unless you consider it as a sporting proposition.

3 7

[PATCH] selinux --disabled fixes
by Warren Togami 20 Feb '08

20 Feb '08

The attached patch seems to fix "selinux --disabled" and other parts related to selinux. I have only tested it with chroot-creator with the "selinux --disabled" case in kickstart. Existing users might want to verify that it doesn't change their behavior because this makes changes to the base classes. Warren Togami wtogami(a)redhat.com

2 1

Simple persistance idea
by don vogt 20 Feb '08

20 Feb '08

I've seen messages about complete persistence, which is, make sure all changes to the filesystem are persistent. However, for end user tasks it might be sufficient to offer persistence of the /home directory. Kai Said---------------------------------------- Has the following idea been proposed already? I figure it would be easy to do. During boot, scan for a partition with a given label, say, "fedoralivehome". If present, that partition gets mounted as /home. As a consequence, all user settings and documents are kept across boots. An usb stick could be partitioned to have the second partition as /home. Does this idea make sense? Thanks, Kai Kai said-------------------------------------------- That is essentially the way Centos -5.0 works (worked?) although it saves /etc also. I posted this on this list as a suggestion quite a while ago, but i don't know what happened. I supposed that people smarter than me had better ideas. The CentOS implemention is pretty simple. They put a change in the local startup init script that looks for a set name. aproximatlty CENTOSBACKUP and then untars what is in it. On exit another script in the shutdown sequence asks if you want to save your data and if so where to save it. If you answer yes the script tars /home and /etc into that file. Since it is open software, I guess you could just pirate the parts of the script you need. There must be many holes in this procedure. I am going back in my hole.

1 0

cachedir conflict... bug?
by Warren Togami 20 Feb '08

20 Feb '08

Here's an interesting problem... I set my ImageCreator cachedir to /var/yum/cache to share my x86_64 laptop's cache. Maybe this was a bad idea, but an interesting thing happened. [root@newcaprica ltsp]# chroot-creator --target=/opt/ltsp/i386/ /etc/ltsp/ltsp-build-client-ks.cfg Mounting /opt/ltsp/i386/ for chroot installation Retrieving http://homemirror.togami.com/pub/fedora/linux/releases/8/Everything/i386/os… ...OK Retrieving http://homemirror.togami.com/pub/fedora/linux/updates/8/x86_64/repodata/rep… ...OK The resulting chroot successfully completed installation, but chroot to run pretty much anything including /bin/bash failed. x86_64 and i386 packages mixed together in some broken way. [root@newcaprica opt]# yum clean metadata Loading "refresh-updatesd" plugin Loading "basearchonly" plugin 27 metadata files removed Cleaning out the metadata made it work as expected. [root@newcaprica ltsp]# chroot-creator --target=/opt/ltsp/i386/ /etc/ltsp/ltsp-build-client-ks.cfg Mounting /opt/ltsp/i386/ for chroot installation Retrieving http://homemirror.togami.com/pub/fedora/linux/releases/8/Everything/i386/os… ...OK Retrieving http://homemirror.togami.com/pub/fedora/linux/updates/8/i386/repodata/repom… ...OK Retrieving http://homemirror.togami.com/pub/fedora/linux/updates/8/i386/repodata/prima… ...OK So there was some kind of conflict between my x86_64 host laptop and the i386 chroot both trying to use the same yum cache directory. I'm guessing it was a bad idea to try to do so and this is unsupported. However I am very surprised that it didn't error out. Are there any genuine bugs in anything I described above? Shouldn't yum warn or error out when it sees the repodata arch conflict? I'm a little saddened that I can't share the same yum cache directories, because the RPMS themselves could peacefully co-exist if it were not for the metadata. Warren Togami wtogami(a)redhat.com

4 12

ImageCreator "selinux --disable" problem
by Warren Togami 20 Feb '08

20 Feb '08

I am trying to reproduce the install behavior of anaconda --noselinux, where it installs a chroot without labels. [1] I need this for LTSP due to SELinux chroot limitations, and Dan Walsh confirms that this is my best option given these current limitations. First I discovered places in kickstart.py where it is supposed to be checking that SELinux-from-kickstart file setting was always returning true. I believe the attached patch fixes this part, although it could use some review. To my dismay it continued to install with labels. I then realized that creator.py's ImageCreator mount() method unconditionally bind mounted the system's /selinux directory, which is incorrect if "selinux --disabled" is defined in the kickstart file. Perhaps my understanding of python is not advanced, but it appears that there is no good way to check kickstart's selinux setting from the mount() method due to the way it is abstracted. Any ideas what could be done here? Warren Togami wtogami(a)redhat.com [1] OK, it really does have labels, but all the labels are the same.

2 2

Storing the image as UDF not IS9660
by Asheesh Laroia 20 Feb '08

20 Feb '08

In adding the data I wanted to the Live CD, I quickly ran into the filename length limitation of ISO9660/Joliet. If I'm willing to break the Joliet standard, I can get files of length 102 on the disc. Normally the standard lets me use 64-character filenames. Meanwhile, as I understand things, UDF is the normal filesystem for DVDs, is pretty widely-supported, and supports 255-character-long paths, perhaps longer <http://en.wikipedia.org/wiki/Comparison_of_file_systems>. It would even work for CDs; it's not as if you'd have to stop supporting generating CDs. Right now, you generate DVDs as large CDs; using UDF would let you generate CDs as if they were small DVDs. (Both work on most OSs, and besides, You would just switch to Have you guys thought about making livecd-creator output UDF-format filesystems rather than the limited ISO9660? -- Asheesh. -- * Knghtbrd is FAR too tempted to .sig this entire discussion...

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

livecd February 2008