Hi all,
Over the past week or so, I've become familiar with the Kadischi portion of Kadischi (as opposed to anaconda) and put together squashfs support. Unfortunately, I've been depending on cvs diff to generate a patch and the cvs server is down right now. Does anyone have a pristine tree I can diff against? If not, I'll tar up my tree later tonight and stick them somewhere for others to look at.
The drawback of that is I had to make a few unrelated changes because I'm creating a minimal install and certain pieces of kadischi make assumptions of what software is installed in the image.
-Toshio
Toshio Kuratomi wrote:
Hi all,
Over the past week or so, I've become familiar with the Kadischi portion of Kadischi (as opposed to anaconda) and put together squashfs support. Unfortunately, I've been depending on cvs diff to generate a patch and the cvs server is down right now. Does anyone have a pristine tree I can diff against? If not, I'll tar up my tree later tonight and stick them somewhere for others to look at.
The drawback of that is I had to make a few unrelated changes because I'm creating a minimal install and certain pieces of kadischi make assumptions of what software is installed in the image.
It should be back up soon, as someone is working on that last I checked.
If you need a snapshot there is one here: http://autopsy.thetabox.org/SOURCES dated 03272006.
J. Hartline
On Tue, 2006-03-28 at 20:47 -0600, J. Hartline wrote:
It should be back up soon, as someone is working on that last I checked.
If you need a snapshot there is one here: http://autopsy.thetabox.org/SOURCES dated 03272006.
Thanks autopsy!
Here's two patches. The first one enables squashfs. In order to use it you invoke kadischi with the --filesystem option::
kadischi --filesystem=squashfs fc5-repo fedora.iso
The patch makes te following changes: * kadischi.py: - New option --filesystem=[compressed fs type] This allows the ISO creator to select zisofs or squashfs. - Bug fix: Change file variable to filename because file is a python builtin. - Compress the filesystem to a squashfs image if --filesystem was set to squashfs. - Do not pass -z to mkisofs if we aren't compressing a zisofs. * livecd-mkinitrd.sh: - Create a /cdrom directory to mount the cdrom image to directly. - Copy squashfs and loop modules to the initrd. - Create loop[0-9] devices in the initrd. * livecd-linuxrc: - Mount the cd image onto /cdrom instead of /sysroot. - If we're using squashfs, loopback mount it on /sysroot otherwise bind mount the cdrom to /sysroot.
Note that I haven't tested zisofs output recently so it's possible I broke zisofs creation since I started working on this.
The second patch is a collection of bugfixes of varying correctness: * functions.py: import shutil as the cleanup() function uses it. * install-boot.sh: Add selinux=0 to the kernel commandline as neither squashfs nor zisofs support selinux xattrs. * 04auth.sh: Comment out the lokkit call as this may not exist in the install root. The selinux portion of this call is taken care of by adding selinux=0 to the kernel commandlin. The firewall portion isn't. And when we do add support for selinux the kernel commandline won't let us choose between targetted/strict/other. So the correct fix for this needs more thought. * 05fsclean.py: Code "copy this directory recursively" in python to avoid calling rsync in the install root (which may not exist and thus error out.) * movefiles.py: Code a check for file existence before attempting to move it. This should never be reached but my initial kadischi runs were triggering some bug here. I haven't seen the diagnostic message triggered recently so the bug may have gone back into hiding.
Let me know what you think, -Toshio
Wow great :)
nevertheless my post-install script is dying
/usr/local/share/kadischi/post_install_scripts/05fsclean.py Traceback (most recent call last): File "/usr/local/share/kadischi/post_install_scripts/05fsclean.py", line 86, in ? clean_up_root (rootdir) File "/usr/local/share/kadischi/post_install_scripts/05fsclean.py", line 66, in clean_up_root os.makedirs(newentry) UnboundLocalError: local variable 'newentry' referenced before assignment
*** Fatal error: /usr/local/share/kadischi/post_install_scripts/05fsclean.py returned non zero (256) exit code. Aborting execution.
Cleaning up temporary files...
What is wrong with my attached post-install script ?
Hello, Am Mittwoch, den 29.03.2006, 13:27 +0200 schrieb Chitlesh GOORAH:
UnboundLocalError: local variable 'newentry' referenced before assignment
[..]
What is wrong with my attached post-install script ?
variable newentry has no value at the moment the line
os.makedirs(newentry)
is executed.
You should rethink about the logic, maybe the else: before the line the error occurs is just incorrectly aligned and should be the else for the line if os.path.islink(oldentry):
or any other "if"-line there?
Just guessing because of not so much knowledge how it will be invoked.
Regards Ingo PS: I will start testing kadischi soon
On Wed, 2006-03-29 at 13:27 +0200, Chitlesh GOORAH wrote:
Wow great :)
nevertheless my post-install script is dying
/usr/local/share/kadischi/post_install_scripts/05fsclean.py Traceback (most recent call last): File "/usr/local/share/kadischi/post_install_scripts/05fsclean.py", line 86, in ? clean_up_root (rootdir) File "/usr/local/share/kadischi/post_install_scripts/05fsclean.py", line 66, in clean_up_root os.makedirs(newentry) UnboundLocalError: local variable 'newentry' referenced before assignment
*** Fatal error: /usr/local/share/kadischi/post_install_scripts/05fsclean.py returned non zero (256) exit code. Aborting execution.
Cleaning up temporary files...
What is wrong with my attached post-install script ?
Bah -- Tabs vs spaces. I should have enabled tab nanny (python's -tt commandline switch). If you substitute a tab for every four spaces on that line things will work. OTOH, per Autopsy's post, this may not be necessary.
Was there a conscious decision to go with tabs instead of spaces in this project or is it just how the code was presented to us? Just say yes and I'll deal with it; I don't want to start a war :-)
-Toshio
Toshio Kuratomi wrote:
Thanks autopsy!
No problem.
Here's two patches. The first one enables squashfs. In order to use it you invoke kadischi with the --filesystem option::
kadischi --filesystem=squashfs fc5-repo fedora.iso
Excellent. I too think SquashFS as an option is a good idea.. (as opposed to no choice, and just Squash) Everything works like it should upon testing all the SquashFS bits. I had done some work with SquashFS too but using a standard Fedora Core cpio initrd with Nash. Nash however doesn't fallback to RO in it's losetup, so of course, you can't setup a loopback with a Squash image as RW.
I think we can probablu go ahead and at the least get the Squash bits up to CVS, and all of the rest with some exceptions. Let me explain:
+kernel_params='selinux=0'
append initrd=initrd.img $kernel_params INITRD_DBG=x
append xen.gz nosmp noacpi noreboot --- vmlinuz --- initrd.img $kernel_params INITRD_DBG=x
In the future (or even now) we probably don't want to hinder the relevance of the debug option. After all, it is there for debugging. :-P
- # Copy files from the skeldir to root's home
- skeldir = os.path.join(rootdir, 'etc/skel')
- skeldirEnd = len(skeldir) + 1
- # Walk the skeldir copying files
- for root, dirs, files in os.walk(skeldir):
wdir = os.path.join(rootdir, 'root', root[skeldirEnd:])for entry in dirs:newentry = os.path.join(wdir, entry)if (os.path.lexists(newentry)):continueoldentry = os.path.join(root, entry)if os.path.islink(oldentry):os.symlink(os.readlink(oldentry), newentry)else:os.makedirs(newentry)for entry in files:newentry = os.path.join(wdir, entry)if (os.path.lexists(newentry)):continueoldentry = os.path.join(root, entry)if os.path.islink(oldentry):os.symlink(os.readlink(oldentry), newentry)else:shutil.copy(oldentry, newentry)
This seems to be overly glorified to me, /etc/skel will always exist as it belongs to the filesystem package, unless you are using a modified filesystem package, which I do not think is the focus of Kadischi. Kadischi should work with the default subset of stock and or updated packages in Fedora Core. Making it if it doesn't exist is also not neccessary, because it should exist. In any case since rsync is an option configurable by the user and it relieves the dependancy on rsync actually existing.
If there isn't any disagreements with Kadischi having the option of zftrees or Squash images we can at least get the Squash bits into CVS. Thanks Toshio!
J. Hartline
--- "J. Hartline" jasperhartline@adelphia.net wrote:
This seems to be overly glorified to me, /etc/skel will always exist as it belongs to the filesystem package, unless you are using
Isn't the real issue here that kadischi is nuking the contents of /root?
Rather than nuke and then copy /etc/skel, I think the answer is to leave /root alone.
Of course the decision was originally made probably due to /root living in tmpfs, and not wanting things like install.syslog or whatever around.
My suggestion (toshio or someone else will probably beat me to the actual posting of code :) is to overhaul the way kadischi uses filesystems, i.e. tmpfs, and readonly zisofs/squashfs.
Currently root is read-only, and then writable tmpfs directories are bind mounted there. I suggest, and have successfully used in the past, the inverse method, which is to make / a tmpfs, and then bindmount big readonly directories there.
Of course that was a long time ago before the advent (or at least out-of-the-box availability) of unionfs/relayfs. These days we should be able to have a completely generic fully populated read-only / (coming from the full system install), with a full read-write tmpfs / overlayed/unioned/relayfsd on top.
With such a configuration, we can even unwind the special read-only root considerations, and be able to do _anything_. I.e. do a "yum install emacs" right after boot.
Don't get me started with how cool it would be to put a massively prepopulated network filesystem in that unionfs/relayfs stack as well :).
Or user usbstick filesytems, or user configuration filesystems pulled from the network, or... or...
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Wed, 2006-03-29 at 17:50 -0800, Jane Dogalt wrote:
Isn't the real issue here that kadischi is nuking the contents of /root?
Rather than nuke and then copy /etc/skel, I think the answer is to leave /root alone.
Yes. After talking with AutOPSY on irc, I think I'll just let the rootfiles package populate /root and get rid of this code altogether. It's always nice to remove unnecessary complexity rather than add it.
Of course the decision was originally made probably due to /root living in tmpfs, and not wanting things like install.syslog or whatever around.
I'm not certain what the original thought was. If someone knows, be sure to bring it up so I don't kill the functionality incorrectly :-)
Currently root is read-only, and then writable tmpfs directories are bind mounted there. I suggest, and have successfully used in the past, the inverse method, which is to make / a tmpfs, and then bindmount big readonly directories there.
I think with the stateless project working on these issues too we're going to have more commonality with ro being the default and only creating rw areas in specific trouble spots. Of course with unionfs...
Of course that was a long time ago before the advent (or at least out-of-the-box availability) of unionfs/relayfs. These days we should be able to have a completely generic fully populated read-only / (coming from the full system install), with a full read-write tmpfs / overlayed/unioned/relayfsd on top.
Heh -- the infrastructure is falling into place to put unionfs kernel modules into Extras now. So it could be that we can start working on this idea now. This would allow us to feed code back and forth with stateless linux but maintain a (transient) rw filesystem at the same time. I'm going to be concetrating on a few other areas more important to what I'm doing with kadischi, though, so this is an exciting project for someone else to start working on.
-Toshio
--- Toshio Kuratomi toshio@tiki-lounge.com wrote:
Heh -- the infrastructure is falling into place to put unionfs kernel modules into Extras now. So it could be that we can start working on this idea now. This would allow us to feed code back and forth with stateless linux but maintain a (transient) rw filesystem at the same time. I'm going to be concetrating on a few other areas more important to what I'm doing with kadischi, though, so this is an exciting project for someone else to start working on.
I'll be working on this. Actually I was a little confused about the out-of-the-box availability of unionfs. I think there may have been another implementation of unionfs that was called relayfs, but which was a wholly different thing than what fedora ships with as relayfs. Though googling now, I can't find any trace of it, though I can find a stale overlay filesystem (ovlfs), mini_fo, and unionfs. I'll let you folks know if/when I get anything working...
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Toshio Kuratomi wrote:
Here's two patches. The first one enables squashfs. In order to use it you invoke kadischi with the --filesystem option::
Everything was committed to CVS except for the last two file patches. I think we can come up with a better solution than to polish something that seems out of place already. :-P In addition to the SquashFS capability as an option and the rest of the patches I went ahead and situated the manual pages, and changed over to using kadischi.conf rather than build.conf.
You should get notices of these changes if youa re subscribed to fedora-extras-commits mailing list.
J. Hartline
On Thu, 2006-03-30 at 00:12 -0600, J. Hartline wrote:
Toshio Kuratomi wrote:
Here's two patches. The first one enables squashfs. In order to use it you invoke kadischi with the --filesystem option::
Everything was committed to CVS except for the last two file patches. I think we can come up with a better solution than to polish something that seems out of place already. :-P In addition to the SquashFS capability as an option and the rest of the patches I went ahead and situated the manual pages, and changed over to using kadischi.conf rather than build.conf.
Great!
I noticed one issue. Looks like your patches use 8 spaces whereas the original code uses tabs. python will interpret this without complaining (unless tab-nanny, python -tt, is turned on) but it leads to unreadable code when some reads it with tabstops set to another number. So it's just plain poor practice to mix tabs and spaces.
I'll go through and normalize the code and add the -tt option so python stops us from doing this continuously, but we need to decide what style of indentation we like and use it throughout.
Here's my short list of candidates. * tabs * 4 spaces * 8 spaces
I prefer 4 spaces per the python style guidelines:: http://www.python.org/dev/peps/pep-0008/
-Toshio
On Tue, 2006-03-28 at 22:04 -0800, Toshio Kuratomi wrote:
On Tue, 2006-03-28 at 20:47 -0600, J. Hartline wrote:
It should be back up soon, as someone is working on that last I checked.
If you need a snapshot there is one here: http://autopsy.thetabox.org/SOURCES dated 03272006.
Thanks autopsy!
Here's two patches. The first one enables squashfs. In order to use it you invoke kadischi with the --filesystem option::
kadischi --filesystem=squashfs fc5-repo fedora.iso
Realistically, something like a --filesystem is the wrong approach. If squashfs is "right", then we should go down that route. Trying to encode the differences between squashfs and zisofs is the path to madness.
Jeremy
On Mon, 2006-04-10 at 11:20 -0400, Jeremy Katz wrote:
On Tue, 2006-03-28 at 22:04 -0800, Toshio Kuratomi wrote:
On Tue, 2006-03-28 at 20:47 -0600, J. Hartline wrote:
It should be back up soon, as someone is working on that last I checked.
If you need a snapshot there is one here: http://autopsy.thetabox.org/SOURCES dated 03272006.
Thanks autopsy!
Here's two patches. The first one enables squashfs. In order to use it you invoke kadischi with the --filesystem option::
kadischi --filesystem=squashfs fc5-repo fedora.iso
Realistically, something like a --filesystem is the wrong approach. If squashfs is "right", then we should go down that route. Trying to encode the differences between squashfs and zisofs is the path to madness.
Sure -- If someone says "do it", I'll submit a patch to remove the special-casing for zisofs :-) It already makes me uncomfortable that I test the squashfs case but not zisofs.
It could be helpful if people test squashfs vs zisofs and see if there's any time when squashfs doesn't generate faster/smaller/"better" CD images; then it would be a no-brainer to remove the less performant filesystem.
-Toshio
On Mon, 2006-04-10 at 10:28 -0700, Toshio Kuratomi wrote:
On Mon, 2006-04-10 at 11:20 -0400, Jeremy Katz wrote:
On Tue, 2006-03-28 at 22:04 -0800, Toshio Kuratomi wrote:
On Tue, 2006-03-28 at 20:47 -0600, J. Hartline wrote:
It should be back up soon, as someone is working on that last I checked.
If you need a snapshot there is one here: http://autopsy.thetabox.org/SOURCES dated 03272006.
Thanks autopsy!
Here's two patches. The first one enables squashfs. In order to use it you invoke kadischi with the --filesystem option::
kadischi --filesystem=squashfs fc5-repo fedora.iso
Realistically, something like a --filesystem is the wrong approach. If squashfs is "right", then we should go down that route. Trying to encode the differences between squashfs and zisofs is the path to madness.
Sure -- If someone says "do it", I'll submit a patch to remove the special-casing for zisofs :-) It already makes me uncomfortable that I test the squashfs case but not zisofs.
It could be helpful if people test squashfs vs zisofs and see if there's any time when squashfs doesn't generate faster/smaller/"better" CD images; then it would be a no-brainer to remove the less performant filesystem.
The one thing that's "better" about zisofs is that it's transparent and you don't have to do any shenanigans with a loopback mount.
Jeremy
--- Jeremy Katz katzj@redhat.com wrote:
The one thing that's "better" about zisofs is that it's transparent and you don't have to do any shenanigans with a loopback mount.
Good usage of quotes there.
Have you been hacking initrd/ramfs's too much lately? Love the loopback. It is your friend :)
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Mon, 2006-04-10 at 14:28 -0400, Jeremy Katz wrote:
On Mon, 2006-04-10 at 10:28 -0700, Toshio Kuratomi wrote:
Sure -- If someone says "do it", I'll submit a patch to remove the special-casing for zisofs :-) It already makes me uncomfortable that I test the squashfs case but not zisofs.
It could be helpful if people test squashfs vs zisofs and see if there's any time when squashfs doesn't generate faster/smaller/"better" CD images; then it would be a no-brainer to remove the less performant filesystem.
The one thing that's "better" about zisofs is that it's transparent and you don't have to do any shenanigans with a loopback mount.
Patch attached that removes the option to use zisofs. Additionally:
* livecd-linuxrc: - Mount the squashfs with -oro to cleanup a warning message. * kadischi.1: - Document the requirement on mksquashfs instead of mkzftree.
-Toshio
Toshio Kuratomi wrote:
Patch attached that removes the option to use zisofs. Additionally:
- livecd-linuxrc:
- Mount the squashfs with -oro to cleanup a warning message.
- kadischi.1:
- Document the requirement on mksquashfs instead of mkzftree.
Committed.
Thanks. J. Hartline
--- Jeremy Katz katzj@redhat.com wrote:
Realistically, something like a --filesystem is the wrong approach. If squashfs is "right", then we should go down that route. Trying to encode the differences between squashfs and zisofs is the path to madness.
I respectfully disagree. An alternate suggestion would be to standardize on filesystem images living on the iso. I.e. currently the choice is between a 'native' zisofs, and a squashfs image. If say, the choice was between a zisofs image, a squashfs image, and an ext2 image, then the user would have a nice set of options. If in addition you go down the route of unionfs rather than readonly-root and bindmounting, then you have a solution in which even selinux is an option.
I think you might have mentioned a problem with ext2 and cdrom 2k block size. I strongly suspect that if that is a problem, it's not a problem when you put the ext2 filesystem on an image file which lives in an iso.
Having a case/switch statement use alternate tools to mk*fs the image, and then having the initrd mount the right type of filesystem really isn't that ugly at all.
To evangelize unionfs one step further (and competely orthogonal to the rest of this post), one could seperate the system image into two parts.
a) those parts that have been profiled as being used during boot and initial login.
and
b) the rest.
Then if you have seperate fs images for those parts, and store them on the cdrom, you should get improved boot speed, because of less seek thrashing during boot. For more bonus, you get mkisofs to put the boot files on the outer tracks, to possibly get that extra 5%.
I'll go ahead and do a proof of concept on this if no one else is interested.
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Mon, 2006-04-10 at 11:18 -0700, Jane Dogalt wrote:
--- Jeremy Katz katzj@redhat.com wrote:
Realistically, something like a --filesystem is the wrong approach. If squashfs is "right", then we should go down that route. Trying to encode the differences between squashfs and zisofs is the path to madness.
I respectfully disagree. An alternate suggestion would be to standardize on filesystem images living on the iso. I.e. currently the choice is between a 'native' zisofs, and a squashfs image.
What is the advantage to the end user of a zisofs image to the consumer of the liveCD? The loopback complexity is handles by kadischi, so that doesn't count. If there is no benefit, then it is a problem to maintain the extra code path -- No active developers will be using (and therefore testing) the sub-optimal code but users who do will uncover bugs that then have to be fixed.
If say, the choice was between a zisofs image, a squashfs image, and an ext2 image, then the user would have a nice set of options. If in addition you go down the route of unionfs rather than readonly-root and bindmounting, then you have a solution in which even selinux is an option.
If ext2 gives us selinux capability and is generally better than squashfs, then we can replace squashfs with ext2. If ext2 adds selinux and is otherwise inferior to squashfs we can revisit maintaining two filesystems.
I think you might have mentioned a problem with ext2 and cdrom 2k block size. I strongly suspect that if that is a problem, it's not a problem when you put the ext2 filesystem on an image file which lives in an iso.
Having a case/switch statement use alternate tools to mk*fs the image, and then having the initrd mount the right type of filesystem really isn't that ugly at all.
Having to test and maintain a code path that provides no advantages is the real problem. The fact that it is ugly is just icing on the cake.
To evangelize unionfs one step further (and competely orthogonal to the rest of this post), one could seperate the system image into two parts.
a) those parts that have been profiled as being used during boot and initial login.
and
b) the rest.
Then if you have seperate fs images for those parts, and store them on the cdrom, you should get improved boot speed, because of less seek thrashing during boot. For more bonus, you get mkisofs to put the boot files on the outer tracks, to possibly get that extra 5%.
I'll go ahead and do a proof of concept on this if no one else is interested.
Please do. I'm interested in seeing this in action but X autodetection and modifying lokkit come before that on my schedule.
-Toshio
--- Toshio Kuratomi toshio@tiki-lounge.com wrote:
On Mon, 2006-04-10 at 11:18 -0700, Jane Dogalt wrote:
--- Jeremy Katz katzj@redhat.com wrote:
Realistically, something like a --filesystem is the wrong approach. If squashfs is "right", then we should go down that route. Trying to encode the differences between squashfs and zisofs is the path to madness.
I respectfully disagree. An alternate suggestion would be to standardize
on
filesystem images living on the iso. I.e. currently the choice is between
a
'native' zisofs, and a squashfs image.
What is the advantage to the end user of a zisofs image to the consumer of the liveCD? The loopback complexity is handles by kadischi, so that doesn't count. If there is no benefit, then it is a problem to maintain the extra code path -- No active developers will be using (and therefore testing) the sub-optimal code but users who do will uncover bugs that then have to be fixed.
see below
If say, the choice was between a zisofs image, a squashfs image, and an ext2 image, then the user would have a nice
set
of options. If in addition you go down the route of unionfs rather than readonly-root and bindmounting, then you have a solution in which even
selinux
is an option.
If ext2 gives us selinux capability and is generally better than squashfs, then we can replace squashfs with ext2. If ext2 adds selinux and is otherwise inferior to squashfs we can revisit maintaining two filesystems.
I suppose my opinion is that it is a trivial complexity addition, one which kadischi currently possesses (though it could be 'cleaner'), and why revisit later, when we can just clean up and extend what is currently there?
I don't think kadischi is yet in any kind of state where we should be worried too much about "the" optimal solution. I think kadischi is still barely in the experimental state where we need to find out what "the" optimal solution is, or whether or not the optimal solution is to have a trivial additional user option, which facilitates the easy evaluation and comparison of various alternatives.
Obviously ext2 is never going to beat squashfs on size. One would guess uncompressed it would win on speed. Clearly at the moment it's the only one that supports xattrs. Then throw in that you can (I think) run it on top of cloop as yet another option.
I think that kadischi should be a flexible tool that is widely used. I.e. we should be seeing at least dozens of publicized kadischi-generated livecd 'tools' out there, just like one can see dozens of knoppix derivatives. To be that, I think we need to give the user some options for tradeoffs that may make sense in their particular case. Some may need selinux and can live with less compression, others may want to use squashfs for size, and still others may want to use isofs and/or zisofs in order to allow system files to be directly visibly on the cdrom (ok, so I just added more complexity to not throw away the non-loopback mounted case, but you know what- I think end-users will appreciate the choice).
I think you might have mentioned a problem with ext2 and cdrom 2k block
size.
I strongly suspect that if that is a problem, it's not a problem when you
put
the ext2 filesystem on an image file which lives in an iso.
Having a case/switch statement use alternate tools to mk*fs the image, and
then
having the initrd mount the right type of filesystem really isn't that ugly
at
all.
Having to test and maintain a code path that provides no advantages is the real problem. The fact that it is ugly is just icing on the cake.
Ugly is ... I'll hold my forked tongue on that one for now ;)
The advantages I've laid out above. Give the user choice. Make it easy to plug in, test, and compare supertinycramfs-7.0 when it gets invented next year.
To evangelize unionfs one step further (and competely orthogonal to the
rest of
this post), one could seperate the system image into two parts.
a) those parts that have been profiled as being used during boot and
initial
login.
and
b) the rest.
Then if you have seperate fs images for those parts, and store them on the cdrom, you should get improved boot speed, because of less seek thrashing during boot. For more bonus, you get mkisofs to put the boot files on the outer tracks, to possibly get that extra 5%.
I'll go ahead and do a proof of concept on this if no one else is
interested.
Please do. I'm interested in seeing this in action but X autodetection and modifying lokkit come before that on my schedule.
Thanks. I'm just throwing out ideas here. Code beauty lies in the eye of the programmer. I agree that X autodetection and sane firewall config options are a higher priority than this. My vision is for the months ahead here, not just tomorrow and next week.
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Mon, 2006-04-10 at 18:26 -0700, Jane Dogalt wrote:
--- Toshio Kuratomi toshio@tiki-lounge.com wrote:
On Mon, 2006-04-10 at 11:18 -0700, Jane Dogalt wrote:
If say, the choice was between a zisofs image, a squashfs image, and an ext2 image, then the user would have a nice
set
of options. If in addition you go down the route of unionfs rather than readonly-root and bindmounting, then you have a solution in which even
selinux
is an option.
If ext2 gives us selinux capability and is generally better than squashfs, then we can replace squashfs with ext2. If ext2 adds selinux and is otherwise inferior to squashfs we can revisit maintaining two filesystems.
I suppose my opinion is that it is a trivial complexity addition, one which kadischi currently possesses (though it could be 'cleaner'), and why revisit later, when we can just clean up and extend what is currently there?
I don't think kadischi is yet in any kind of state where we should be worried too much about "the" optimal solution. I think kadischi is still barely in the experimental state where we need to find out what "the" optimal solution is, or whether or not the optimal solution is to have a trivial additional user option, which facilitates the easy evaluation and comparison of various alternatives.
Sure, but we do need to worry about it working. If zisofs has significant features that squashfs lacks and vice versa then I can see supporting both at this stage. If not, then keeping a filesystem that provides no real advantages means we have a code path in kadischi that few, if any, developers are going to want to test thoroughly and bugfix. This leads to non-working code later.
Obviously ext2 is never going to beat squashfs on size. One would guess uncompressed it would win on speed.
You'd want to benchmark this -- CDRoms are pretty slow media so it could be that squashfs is faster for our use case as well.
Clearly at the moment it's the only one that supports xattrs.
Yep. And If someone wants to make ext2 images work with kadischi, the code to support multiple filesystems can be written to support ext2 and squashfs. If it's still applicable, it can be revived from the cvs repository. Depending on how long before someone adds it, it may be that all of the code has gone through several rewrites since then.
Then throw in that you can (I think) run it on top of cloop as yet another option.
Hmmm... cloop's source repository is currently down so I cna't look itno it: http://developer.linuxtag.net/knoppix/sources/
Since it's not in the mainline kernel, someone has to package it for Fedora Extras.
I'm not sure that cloop by itself will provide us with something better than squashfs. But if cloop + ext2 is the best way to get compression and SELinux, then it would make sense for now.
OTOH, someone may decide to add xattrs to squashfs in which case I'd ask again, what does ext2 do better than squashfs?
I think that kadischi should be a flexible tool that is widely used. I.e. we should be seeing at least dozens of publicized kadischi-generated livecd 'tools' out there, just like one can see dozens of knoppix derivatives. To be that, I think we need to give the user some options for tradeoffs that may make sense in their particular case. Some may need selinux and can live with less compression, others may want to use squashfs for size, and still others may want to use isofs and/or zisofs in order to allow system files to be directly visibly on the cdrom (ok, so I just added more complexity to not throw away the non-loopback mounted case, but you know what- I think end-users will appreciate the choice).
I agree that having options for tradeoffs make sense. And I think that in the absence of xattrs in squashfs, ext2's SELinux support is a big feature that can be seen in this light. I haven't seen the same kind of compelling reason for zisofs. To the end-user booting the LiveCD the non-loopback case isn't going to be visible. To the creator, kadischi hides the complexity for either case. Most developers wanting to peer inside the ISO from outside the LiveCD environment are going to be able to mount the filesystem and will be willing to do that for the superior performance that squashfs gives.
-Toshio
--- Toshio Kuratomi toshio@tiki-lounge.com wrote:
I agree that having options for tradeoffs make sense. And I think that in the absence of xattrs in squashfs, ext2's SELinux support is a big feature that can be seen in this light. I haven't seen the same kind of compelling reason for zisofs. To the end-user booting the LiveCD the non-loopback case isn't going to be visible. To the creator, kadischi hides the complexity for either case. Most developers wanting to peer inside the ISO from outside the LiveCD environment are going to be able to mount the filesystem and will be willing to do that for the superior performance that squashfs gives.
The motivation I see for wanting to add regular old iso as an option (not even zisofs) is for the user to see files in the iso. And rather, the user (of kadischi)'s users.
I.e. imagine that Chitlesh has his marketing videos (and other media/material) in the generated CD. You might want to include a simple autorun.inf so that when the cd/dvd is inserted in a windows machine, it automatically launches a web browser with links to thumbnails of the video, and a webpage touting what this livecd can do. Along with the livecd's instruction manual.
When actually _booted_, these same files are launched in a full screen webbrowser immediately after gdm autologin.
This way, at a trade show, you can hand the dvd to a non-techie-non-linux suit&tie type, and tell them it will launch an info page in a browser if they put it in their microsoft box.
If you really want to be slick, you even include a run-from-cd installation of firefox-win32 and qemu-win32(or vmware), and then people can even choose to run your appliance-cd purely from within windows. At least just to demonstrate. You reiterate that it will run faster if you actually boot from it.
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Jane Dogalt wrote:
I.e. imagine that Chitlesh has his marketing videos (and other media/material) in the generated CD. You might want to include a simple autorun.inf so that when the cd/dvd is inserted in a windows machine, it automatically launches a web browser with links to thumbnails of the video, and a webpage touting what this livecd can do. Along with the livecd's instruction manual.
When actually _booted_, these same files are launched in a full screen webbrowser immediately after gdm autologin.
This way, at a trade show, you can hand the dvd to a non-techie-non-linux suit&tie type, and tell them it will launch an info page in a browser if they put it in their microsoft box.
If you really want to be slick, you even include a run-from-cd installation of firefox-win32 and qemu-win32(or vmware), and then people can even choose to run your appliance-cd purely from within windows. At least just to demonstrate. You reiterate that it will run faster if you actually boot from it.
This is a likely scenario with an Official forged Fedora Core LiveCD or DVD. Not a Kadischi CD in my opinion. The bootsplash stuff is there to distinguish where and from how the disc came about. Promotional materials and other specific things like that should be distributed with an official disc, not these user-made CDs.
J. Hartline
On Mon, 2006-04-10 at 22:27 -0700, Jane Dogalt wrote:
The motivation I see for wanting to add regular old iso as an option (not even zisofs) is for the user to see files in the iso. And rather, the user (of kadischi)'s users.
As it turns out, you could do this even if the main live CD parts are a loopback mount. They're not mutually exclusive at all
Jeremy
--- Jeremy Katz katzj@redhat.com wrote:
On Mon, 2006-04-10 at 22:27 -0700, Jane Dogalt wrote:
The motivation I see for wanting to add regular old iso as an option (not
even
zisofs) is for the user to see files in the iso. And rather, the user (of kadischi)'s users.
As it turns out, you could do this even if the main live CD parts are a loopback mount. They're not mutually exclusive at all
Yes, that's right.
Honestly I'm not trying to stoke a flamewar here. Maybe for development simplicity it does make sense to just focus on one filesystem type right now, and save features like multiple different filesystem type images being overlaid in the final system with a unionfs to people unafraid of loopback madness like myself (for the timebeing) ;).
-----------
Elsewhere, replying to the isofs shared win/lin files example of chitlesh's marketing videos, jhartline wrote-
This is a likely scenario with an Official forged Fedora Core LiveCD or DVD. Not a Kadischi CD in my opinion. The bootsplash stuff is there to distinguish where and from how the disc came about. Promotional materials and other specific things like that should be distributed with an official disc, not these user-made CDs.
Absolutely, of course chitleshes promotional materials would be used on an official fedora livecd. My point was to illustrate the _type_ of scenario that many end-users of a fedora-livecd generating system would find themselves in (when they are off happily producing their unofficial fedora livecd's).
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Jane Dogalt wrote:
Thanks. I'm just throwing out ideas here. Code beauty lies in the eye of the programmer. I agree that X autodetection and sane firewall config options are a higher priority than this. My vision is for the months ahead here, not just tomorrow and next week.
What does X autodetection mean? X is X.
J. Hartline
What does X autodetection mean? X is X.
Actually A is A, but...
X autodetection is udev 'correctly' supporting hot/coldplug of vga adapters.
In reality it's a few lines of script here and there that accomplish more or less the same thing.
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Wed, 2006-04-12 at 10:12 -0700, Jane Dogalt wrote:
What does X autodetection mean? X is X.
Actually A is A, but...
X autodetection is udev 'correctly' supporting hot/coldplug of vga adapters.
In reality it's a few lines of script here and there that accomplish more or less the same thing.
More realistically, it's the work in X itself to autoconfigure itself based on the monitor and video cards present in the machine. Anything else is going to be less than optimal. The X server already has all kinds of this information (read through your X log sometime), but it basically just throws it out. Instead, we really want it to be using that information to give an "optimal" configuration and then things like resolution can start being a user preference[1]
If someone is interested in helping to work on this, I can put you into contact with one of the X guys who has started looking at it as well
Jeremy
[1] This is already the case, we just have to set up the defaults where we _can't_ easily do probing for monitors
--- Jeremy Katz katzj@redhat.com wrote:
On Wed, 2006-04-12 at 10:12 -0700, Jane Dogalt wrote:
What does X autodetection mean? X is X.
Actually A is A, but...
X autodetection is udev 'correctly' supporting hot/coldplug of vga
adapters.
In reality it's a few lines of script here and there that accomplish more
or
less the same thing.
More realistically, it's the work in X itself to autoconfigure itself based on the monitor and video cards present in the machine. Anything else is going to be less than optimal. The X server already has all kinds of this information (read through your X log sometime), but it basically just throws it out. Instead, we really want it to be using that information to give an "optimal" configuration and then things like resolution can start being a user preference[1]
If someone is interested in helping to work on this, I can put you into contact with one of the X guys who has started looking at it as well
Actually it's called "Xorg -configure". That will generate an xorg.conf file that matches your vga (and sometimes monitor I think) hardware. Then you do some sed or python or perl to tune things like the user preferred resolution.
I was using this feature immediately after it was released in XFree86 several years ago in a mandrake-8.0 livecd. Though my appliance only required 640x480, so I didn't have to worry too much about how successfully the monitor probing (edid stuff) worked.
Then if you want to, you can do some more scripting to replace nv with nvidia etc (assuming you've agreed to all their terms, etc...)
-jdog
Jeremy
[1] This is already the case, we just have to set up the defaults where we _can't_ easily do probing for monitors
-- Fedora-livecd-list mailing list Fedora-livecd-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-livecd-list
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Wed, 2006-04-12 at 13:07 -0700, Jane Dogalt wrote:
Actually it's called "Xorg -configure". That will generate an xorg.conf file that matches your vga (and sometimes monitor I think) hardware. Then you do some sed or python or perl to tune things like the user preferred resolution.
This is actually a really poor way to do things. Frob hardware once to start X, then exit, then start again? It's a bit overkill. Plus, modifying of an xorg.conf is very painful. You also need to take into account whether or not font paths are being used or a font server, etc, etc.
Was harassing ajax some more about this yesterday, though, and we came to the conclusion that if you comment out the modes line from the display section and the horizontal and vertical sync lines from the monitor section, X mostly auto-detects and picks the "best" that can be done. That plus xrandr could get very reasonable automagic X setups.
Jeremy
--- Jeremy Katz katzj@redhat.com wrote:
On Wed, 2006-04-12 at 13:07 -0700, Jane Dogalt wrote:
Actually it's called "Xorg -configure". That will generate an xorg.conf
file
that matches your vga (and sometimes monitor I think) hardware. Then you
do
some sed or python or perl to tune things like the user preferred
resolution.
This is actually a really poor way to do things. Frob hardware once to start X, then exit, then start again? It's a bit overkill.
I'm not exactly sure what 'frob' means, but my guess is that hardware was made to be frobbed. If it was really bad, why is the -configure option there? AFAIC nothing is overkill compared to what kadischi is currently using (i.e. zip).
Plus,
modifying of an xorg.conf is very painful.
Here's some sample code attached... (takes existing config in from pipe, outputs to stdout). I'll agree it was moderately painful, but not at all undoable. (note, code is 5 years old, current xorg.conf syntax might break it). And please don't bother with derisions about how ugly this or perl is. I merely submit this as an example. Look at it if you like, think about it if you like, and use it if you like. If not, post your own code that gets the job done.
You also need to take into
account whether or not font paths are being used or a font server, etc, etc.
I would presume to take whatever default xorg.conf anaconda generates (say forcing it to assume vesa or vga), and then inserting the -configure'd device and monitor section. Then providing a suitably autoconfigured/generic input device section (note my script above puts in ps/2, usb, and lirc mouse, and doesn't really care how many of those the user has).
Was harassing ajax some more about this yesterday, though, and we came to the conclusion that if you comment out the modes line from the display section and the horizontal and vertical sync lines from the monitor section, X mostly auto-detects and picks the "best" that can be done. That plus xrandr could get very reasonable automagic X setups.
If that theoretical solution turns out to work better than what I've already done that I know already works, I'd love to integrate it into my project.
For now though, I'm more of the mindset- get something working asap, and then evolutionarily change it as time, research and development progress. I don't know who this ajax person is, but if they are an X developer, and suggesting that -configure is the wrong thing, then hopefully they will either replace that option with the right thing, or reimplement it as the right thing.
-jdog
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Thu, 2006-04-13 at 12:42 -0700, Jane Dogalt wrote:
--- Jeremy Katz katzj@redhat.com wrote:
On Wed, 2006-04-12 at 13:07 -0700, Jane Dogalt wrote:
Actually it's called "Xorg -configure". That will generate an xorg.conf
file
that matches your vga (and sometimes monitor I think) hardware. Then you
do
some sed or python or perl to tune things like the user preferred
resolution.
This is actually a really poor way to do things. Frob hardware once to start X, then exit, then start again? It's a bit overkill.
I'm not exactly sure what 'frob' means, but my guess is that hardware was made to be frobbed. If it was really bad, why is the -configure option there? AFAIC nothing is overkill compared to what kadischi is currently using (i.e. zip).
Actually, there have been bugs in the past where starting X a second time on the same piece of hardware caused things to break. Additionally, you add a *ton* of time given how long it takes for things to sync and monitors going in and out of sync give a very real impression of "completely busted crap". There's even the occasional bug report about the _single_ sync required going from text mode to graphical mode during the install.
Plus,
modifying of an xorg.conf is very painful.
Here's some sample code attached... (takes existing config in from pipe, outputs to stdout). I'll agree it was moderately painful, but not at all undoable. (note, code is 5 years old, current xorg.conf syntax might break it). And please don't bother with derisions about how ugly this or perl is. I merely submit this as an example. Look at it if you like, think about it if you like, and use it if you like. If not, post your own code that gets the job done.
And when something changes, you are doomed. Been there, done that. As for other code to generate X configs, please see the code used by anaconda. The problem there is that we have no good way of doing probing of "native" resolutions for things like LCDs. Until recently, this was the single most reliable thing to do. It's starting to look like a more minimal X configuration may now be able to start working.
Was harassing ajax some more about this yesterday, though, and we came to the conclusion that if you comment out the modes line from the display section and the horizontal and vertical sync lines from the monitor section, X mostly auto-detects and picks the "best" that can be done. That plus xrandr could get very reasonable automagic X setups.
If that theoretical solution turns out to work better than what I've already done that I know already works, I'd love to integrate it into my project.
For now though, I'm more of the mindset- get something working asap, and then evolutionarily change it as time, research and development progress. I don't know who this ajax person is, but if they are an X developer, and suggesting that -configure is the wrong thing, then hopefully they will either replace that option with the right thing, or reimplement it as the right thing.
So because the option exists, it's the right thing? Seriously, if it worked so great, don't you think we would have just done that long ago instead of writing an entire stack of stuff to deal with X configuration?
The right thing is really basically having xorg.conf die and getting suitable infrastructure into X to handle things like hotplugging of video devices and monitors, actual real input hotplug support (as opposed to the current hack where we just point at /dev/input/mice and it appears to magically work thanks to the kernel actually being helpful), etc. Unfortunately, there's a lot of work to get there. Things are definitely moving in that direction, though -- the ability to now not specify sync ranges and modes are just one step.
Jeremy
On Mon, 2006-04-10 at 18:26 -0700, Jane Dogalt wrote:
--- Toshio Kuratomi toshio@tiki-lounge.com wrote: I suppose my opinion is that it is a trivial complexity addition, one which kadischi currently possesses (though it could be 'cleaner'), and why revisit later, when we can just clean up and extend what is currently there?
It's hardly trivial. Supporting filesystems ends up being quite a bit of work over time to ensure that all new features are supported everywhere.
Obviously ext2 is never going to beat squashfs on size. One would guess uncompressed it would win on speed. Clearly at the moment it's the only one that supports xattrs. Then throw in that you can (I think) run it on top of cloop as yet another option.
Actually, I would expect that it won't really end up winning on speed as your CPU is going to be far faster than your CD drive. Also, cloop isn't an option as long as it's not in the main kernel. xattr support for squashfs is planned and adding it shouldn't be that hard if someone wanted to do the work. The maintainer has been very amenable to such things in previous discussions with him.
Jeremy
On Tue, 2006-03-28 at 22:04 -0800, Toshio Kuratomi wrote:
- livecd-mkinitrd.sh:
- Create a /cdrom directory to mount the cdrom image to directly.
- Copy squashfs and loop modules to the initrd.
- Create loop[0-9] devices in the initrd.
- livecd-linuxrc:
- Mount the cd image onto /cdrom instead of /sysroot.
- If we're using squashfs, loopback mount it on /sysroot otherwise
bind mount the cdrom to /sysroot.
We still need to stop creating our own initrd and be using the initramfs created by the stock mkinitrd and just do the necessary livecd changes with our own secondary initramfs that contains a replacement mkrootdev instead of using the nash built-in. I started on this before leaving on vacation, after I get a chance to sit at my desk later today, I'll send what I have.
- install-boot.sh: Add selinux=0 to the kernel commandline as neither
squashfs nor zisofs support selinux xattrs.
I'd rather be handling this just by setting things in /etc/sysconfig/selinux. Managing kernel options is the road to pain.
- 04auth.sh: Comment out the lokkit call as this may not exist in the
install root. The selinux portion of this call is taken care of by adding selinux=0 to the kernel commandlin. The firewall portion isn't. And when we do add support for selinux the kernel commandline won't let us choose between targetted/strict/other. So the correct fix for this needs more thought.
Lokkit really needs to be installed -- we all but count on it always being present with anaconda.
Jeremy
On Mon, 2006-04-10 at 11:26 -0400, Jeremy Katz wrote:
On Tue, 2006-03-28 at 22:04 -0800, Toshio Kuratomi wrote:
- livecd-mkinitrd.sh:
- Create a /cdrom directory to mount the cdrom image to directly.
- Copy squashfs and loop modules to the initrd.
- Create loop[0-9] devices in the initrd.
- livecd-linuxrc:
- Mount the cd image onto /cdrom instead of /sysroot.
- If we're using squashfs, loopback mount it on /sysroot otherwise
bind mount the cdrom to /sysroot.
We still need to stop creating our own initrd and be using the initramfs created by the stock mkinitrd and just do the necessary livecd changes with our own secondary initramfs that contains a replacement mkrootdev instead of using the nash built-in. I started on this before leaving on vacation, after I get a chance to sit at my desk later today, I'll send what I have.
Send it and I'll get familiar with it :-)
- install-boot.sh: Add selinux=0 to the kernel commandline as neither
squashfs nor zisofs support selinux xattrs.
I'd rather be handling this just by setting things in /etc/sysconfig/selinux. Managing kernel options is the road to pain.
Yes -- I sent a new patch that uses sed to make these changes. Jasper disagreed with that approach however and thinks the kernel commandline is better :-) I'm sure you'll get to that set of patches later in your catch-up-with-email-backlog and can weigh in.
- 04auth.sh: Comment out the lokkit call as this may not exist in the
install root. The selinux portion of this call is taken care of by adding selinux=0 to the kernel commandlin. The firewall portion isn't. And when we do add support for selinux the kernel commandline won't let us choose between targetted/strict/other. So the correct fix for this needs more thought.
Lokkit really needs to be installed -- we all but count on it always being present with anaconda.
Currently anaconda doesn't error out if lokkit is not present -- are the errors discarded but present?
My thinking is that requiring lokkit in the CD is less than ideal. It is being required by the installer but may not be required or wanted for the end-use of the CD. A better approach is to modify lokkit to work with an alternate rootpath so it updates the configuration in the built-image instead of the image it is running on. Then anaconda or kadischi could call "lokkit --rootpath /var/www/kadischi-image --setup-my-firewall-and-selinux" to modify the new image rather than the system it is running on.
-Toshio
livecd@lists.fedoraproject.org
-
Chitlesh GOORAH -
Ingo Schaefer -
J. Hartline -
Jane Dogalt -
Jeremy Katz -
Toshio Kuratomi