[PATCH 6/7] anaconda: liveinst.sh: support turboLiveInst/genMinInstDelta

[Douglas McClendon]

this patch takes care of the anaconda side of taking advantage of 
livecd's created with a livecd-tools that has 
turboLiveInst/genMinInstDelta support.  But the code does check for the 
legacy situation, and handles it gracefully.  Thus this patch is safe 
even in the absence of the actual beneficial final patch to livecd-tools.

[Jeremy Katz]

On Mon, 2007-09-17 at 12:34 -0500, Douglas McClendon wrote:
>  this patch takes care of the anaconda side of taking advantage of 
>  livecd's created with a livecd-tools that has 
>  turboLiveInst/genMinInstDelta support.  But the code does check for the 
>  legacy situation, and handles it gracefully.  Thus this patch is safe 
>  even in the absence of the actual beneficial final patch to livecd-tools. 
So as you alluded in your later mail, this looks like it could be
confused if something else ended up using loop118 instead.  Is there a
good reason not to just set up the minimal snapshot from the initrd and
then just use it if it exists?  That would then make the anaconda side
super-crazy-simple (look for /dev/mapper/live-osimg-min and use it if it
exists).  

The only downside I can see is that the snapshot is always set up which
probably has some resource cost, but I can't see it being that high.

Jeremy

[Douglas McClendon]

Jeremy Katz wrote:
>  On Mon, 2007-09-17 at 12:34 -0500, Douglas McClendon wrote:
> > this patch takes care of the anaconda side of taking advantage of 
> > livecd's created with a livecd-tools that has 
> > turboLiveInst/genMinInstDelta support.  But the code does check for the 
> > legacy situation, and handles it gracefully.  Thus this patch is safe 
> > even in the absence of the actual beneficial final patch to livecd-tools.
>  
>  So as you alluded in your later mail, this looks like it could be
>  confused if something else ended up using loop118 instead.  Is there a
>  good reason not to just set up the minimal snapshot from the initrd and
>  then just use it if it exists?  That would then make the anaconda side
>  super-crazy-simple (look for /dev/mapper/live-osimg-min and use it if it
>  exists).   
The danger would be loop117, as that is the one created at install time. 
  loop118 being locked up at initramfs time.

The reason why I think it should stay as is, instead of setting 
everything up at initramfs time, is because the loop117 gets associated 
with the decompressed-into-devshm contents of loop118.  The 
decompression being 7kb->1.2M in the typical case.

I'd say that the 1.2M ram hit should only be taken during install, and 
that outweighs the benefits of moving the complexity from liveinst.sh to 
initramfs.

And as for the theoretical possibility that loop117 could be in use, 
that should be addressed by a forthcoming patch which removes all the 
hardcoded loop device usage.

-dmc



>  The only downside I can see is that the snapshot is always set up
> which
>  probably has some resource cost, but I can't see it being that high.
>  
>  Jeremy
>  
>  --
>  Fedora-livecd-list mailing list
>  Fedora-livecd-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-livecd-list

[Jeremy Katz]

On Mon, 2007-09-17 at 22:31 -0500, Douglas McClendon wrote: 
>  Jeremy Katz wrote:
>  > On Mon, 2007-09-17 at 12:34 -0500, Douglas McClendon wrote:
>  >> this patch takes care of the anaconda side of taking advantage of 
>  >> livecd's created with a livecd-tools that has 
>  >> turboLiveInst/genMinInstDelta support.  But the code does check for the 
>  >> legacy situation, and handles it gracefully.  Thus this patch is safe 
>  >> even in the absence of the actual beneficial final patch to livecd-tools.
>  > 
>  > So as you alluded in your later mail, this looks like it could be
>  > confused if something else ended up using loop118 instead.  Is there a
>  > good reason not to just set up the minimal snapshot from the initrd and
>  > then just use it if it exists?  That would then make the anaconda side
>  > super-crazy-simple (look for /dev/mapper/live-osimg-min and use it if it
>  > exists).  
>  
>  The danger would be loop117, as that is the one created at install time. 
>    loop118 being locked up at initramfs time. 
Not if new anaconda is used with old livecd-creator -- then loop118
wouldn't have been used and something else could use it while the live
system is running.

>  The reason why I think it should stay as is, instead of setting 
>  everything up at initramfs time, is because the loop117 gets associated 
>  with the decompressed-into-devshm contents of loop118.  The 
>  decompression being 7kb->1.2M in the typical case.
>  
>  I'd say that the 1.2M ram hit should only be taken during install, and 
>  that outweighs the benefits of moving the complexity from liveinst.sh to 
>  initramfs. 
But the time that the memory hit matters the _most_ is during the
install as we have to have swap disabled during parts of it.  Otherwise,
swap is available and can be used.  It also means that there's less need
to worry about the cleanup case in anaconda.  Also, it takes care of my
biggest complaint from the past (anaconda having to know too many
details of how things are setup -- I _really_ like that right now,
anaconda just needs to know to look at block device that's already set
up and nothing more is needed)

Patch ends up being the attached plus the teensy change so that anaconda
will check /dev/mapper/live-osimg-min

Jeremy

[Douglas McClendon]

Jeremy Katz wrote:
>  On Mon, 2007-09-17 at 22:31 -0500, Douglas McClendon wrote: 
> > Jeremy Katz wrote: 
> > The danger would be loop117, as that is the one created at
> install time. 
> >   loop118 being locked up at initramfs time.
>  
>  Not if new anaconda is used with old livecd-creator -- then loop118
>  wouldn't have been used and something else could use it while the live
>  system is running. 
This is true, albeit unlikely as well.  But also fixable by addressing 
the hardcoded loop device issue across the board.

>  
> > The reason why I think it should stay as is, instead of setting 
> > everything up at initramfs time, is because the loop117 gets associated 
> > with the decompressed-into-devshm contents of loop118.  The 
> > decompression being 7kb->1.2M in the typical case.
> >
> > I'd say that the 1.2M ram hit should only be taken during install, and 
> > that outweighs the benefits of moving the complexity from liveinst.sh to 
> > initramfs.
>  
>  But the time that the memory hit matters the _most_ is during the
>  install as we have to have swap disabled during parts of it.  
Obviously this is your call, and from my perspective not a big deal 
either way.  My own personal preference still remains freeing up the 
1.2M of ram during all times other than the install, at the expense of 
the complexity being in anaconda/liveinst.sh, versus being in initramfs.

Perhaps by F9 I'll find a mechanism to achieve this that you like...

  Otherwise,
>  swap is available and can be used.  It also means that there's
> less need
>  to worry about the cleanup case in anaconda.  Also, it takes care of my
>  biggest complaint from the past (anaconda having to know too many
>  details of how things are setup -- I _really_ like that right now,
>  anaconda just needs to know to look at block device that's already set
>  up and nothing more is needed)
>  
>  Patch ends up being the attached plus the teensy change so that anaconda
>  will check /dev/mapper/live-osimg-min 
Looks fine to me.  Thanks for the merge work.

Now I'll move on to seeing what I can do in the next couple days with 
that persistence patch.  And maybe some other interesting things after 
that... ;)

peace...

-dmc

P.S.- I'm still curious on that davej ping regarding unionfs.

[Douglas McClendon]

Douglas McClendon wrote:
>  Jeremy Katz wrote:
> > On Mon, 2007-09-17 at 22:31 -0500, Douglas McClendon wrote:
> >> Jeremy Katz wrote:
>  
> >> The danger would be loop117, as that is the one created at install 
> >> time.   loop118 being locked up at initramfs time.
> >
> > Not if new anaconda is used with old livecd-creator -- then loop118
> > wouldn't have been used and something else could use it while the live
> > system is running.
>  
>  This is true, albeit unlikely as well.  But also fixable by addressing 
>  the hardcoded loop device issue across the board.
>  
> >
> >> The reason why I think it should stay as is, instead of setting 
> >> everything up at initramfs time, is because the loop117 gets 
> >> associated with the decompressed-into-devshm contents of loop118.  
> >> The decompression being 7kb->1.2M in the typical case.
> >>
> >> I'd say that the 1.2M ram hit should only be taken during install, 
> >> and that outweighs the benefits of moving the complexity from 
> >> liveinst.sh to initramfs.
> >
> > But the time that the memory hit matters the _most_ is during the
> > install as we have to have swap disabled during parts of it. 
>  
>  Obviously this is your call, and from my perspective not a big deal 
>  either way.  My own personal preference still remains freeing up the 
>  1.2M of ram during all times other than the install, at the expense of 
>  the complexity being in anaconda/liveinst.sh, versus being in initramfs.
>  
>  Perhaps by F9 I'll find a mechanism to achieve this that you like... 
Or maybe in significantly less time than that...

How's this-

1) In initramfs, instead of lazy unmounting the squashfs, bindmount it 
to /sysroot/mnt/.LiveOS/squashfs

2) store osmin 'uncompressed' on the squashfs

3) in initramfs, set up the osimg-min device, but now with the loop118 
pointing at /sysroot/mnt/.LiveOS/squashfs/osmin

The result- All the same low-impact that the current method has with 
anaconda, but the 1.2M of ram will only get used when anaconda actually 
starts touching /dev/mapper/live-osimg-min

And given that it is a 1.2M file that compresses to 7kb, I'm assuming 
that squashfs will basically read and uncompress it only once.  (and if 
memory is so constrained that during installation it gets swapped out, 
it's probably a good thing that that is possible and squashfs may read 
and uncompress it again).

By jeeves, I think we've got it.  (correct me if I'm wrong or missing 
something)

Thanks for being so stubborn :)

-dmc


>   Otherwise,
> > swap is available and can be used.  It also means that there's less need
> > to worry about the cleanup case in anaconda.  Also, it takes care of my
> > biggest complaint from the past (anaconda having to know too many
> > details of how things are setup -- I _really_ like that right now,
> > anaconda just needs to know to look at block device that's already set
> > up and nothing more is needed)
> >
> > Patch ends up being the attached plus the teensy change so that anaconda
> > will check /dev/mapper/live-osimg-min
>  
>  Looks fine to me.  Thanks for the merge work.
>  
>  Now I'll move on to seeing what I can do in the next couple days with 
>  that persistence patch.  And maybe some other interesting things after 
>  that... ;)
>  
>  peace...
>  
>  -dmc
>  
>  P.S.- I'm still curious on that davej ping regarding unionfs.
>  
>  -- 
>  Fedora-livecd-list mailing list
>  Fedora-livecd-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-livecd-list

[Douglas McClendon]

Douglas McClendon wrote:
>  Douglas McClendon wrote:
> > Jeremy Katz wrote:
> >> On Mon, 2007-09-17 at 22:31 -0500, Douglas McClendon wrote:
> >>> Jeremy Katz wrote:
> >
> >>> The danger would be loop117, as that is the one created at install 
> >>> time.   loop118 being locked up at initramfs time.
> >>
> >> Not if new anaconda is used with old livecd-creator -- then loop118
> >> wouldn't have been used and something else could use it while the live
> >> system is running.
> >
> > This is true, albeit unlikely as well.  But also fixable by addressing 
> > the hardcoded loop device issue across the board.
> >
> >>
> >>> The reason why I think it should stay as is, instead of setting 
> >>> everything up at initramfs time, is because the loop117 gets 
> >>> associated with the decompressed-into-devshm contents of loop118.  
> >>> The decompression being 7kb->1.2M in the typical case.
> >>>
> >>> I'd say that the 1.2M ram hit should only be taken during install, 
> >>> and that outweighs the benefits of moving the complexity from 
> >>> liveinst.sh to initramfs.
> >>
> >> But the time that the memory hit matters the _most_ is during the
> >> install as we have to have swap disabled during parts of it. 
> >
> > Obviously this is your call, and from my perspective not a big deal 
> > either way.  My own personal preference still remains freeing up the 
> > 1.2M of ram during all times other than the install, at the expense of 
> > the complexity being in anaconda/liveinst.sh, versus being in initramfs.
> >
> > Perhaps by F9 I'll find a mechanism to achieve this that you like...
>  
>  Or maybe in significantly less time than that...
>  
>  How's this-
>  
>  1) In initramfs, instead of lazy unmounting the squashfs, bindmount it 
>  to /sysroot/mnt/.LiveOS/squashfs 
bindmount/movemount, whatever...

>  
>  2) store osmin 'uncompressed' on the squashfs
>  
>  3) in initramfs, set up the osimg-min device, but now with the loop118 
>  pointing at /sysroot/mnt/.LiveOS/squashfs/osmin
>  
>  The result- All the same low-impact that the current method has with 
>  anaconda, but the 1.2M of ram will only get used when anaconda actually 
>  starts touching /dev/mapper/live-osimg-min
>  
>  And given that it is a 1.2M file that compresses to 7kb, I'm assuming 
>  that squashfs will basically read and uncompress it only once.  (and if 
>  memory is so constrained that during installation it gets swapped out, 
>  it's probably a good thing that that is possible and squashfs may read 
>  and uncompress it again). 
swapout/dropped-from-buffer/squashfs/page-cache, whatever...

>  By jeeves, I think we've got it.  (correct me if I'm wrong or
> missing 
>  something)
>  
>  Thanks for being so stubborn :)
>  
>  -dmc
>  
>  
> >  Otherwise,
> >> swap is available and can be used.  It also means that there's less need
> >> to worry about the cleanup case in anaconda.  Also, it takes care of my
> >> biggest complaint from the past (anaconda having to know too many
> >> details of how things are setup -- I _really_ like that right now,
> >> anaconda just needs to know to look at block device that's already set
> >> up and nothing more is needed)
> >>
> >> Patch ends up being the attached plus the teensy change so that anaconda
> >> will check /dev/mapper/live-osimg-min
> >
> > Looks fine to me.  Thanks for the merge work.
> >
> > Now I'll move on to seeing what I can do in the next couple days with 
> > that persistence patch.  And maybe some other interesting things after 
> > that... ;)
> >
> > peace...
> >
> > -dmc
> >
> > P.S.- I'm still curious on that davej ping regarding unionfs.
> >
> > -- 
> > Fedora-livecd-list mailing list
> > Fedora-livecd-list(a)redhat.com
> > https://www.redhat.com/mailman/listinfo/fedora-livecd-list
>  
>  -- 
>  Fedora-livecd-list mailing list
>  Fedora-livecd-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-livecd-list

[Jeremy Katz]

On Tue, 2007-09-18 at 15:09 -0500, Douglas McClendon wrote:
>  Douglas McClendon wrote:
>  > Jeremy Katz wrote:
>  >> On Mon, 2007-09-17 at 22:31 -0500, Douglas McClendon wrote:
>  >>> Jeremy Katz wrote:
>  > 
>  >>> The danger would be loop117, as that is the one created at install 
>  >>> time.   loop118 being locked up at initramfs time.
>  >>
>  >> Not if new anaconda is used with old livecd-creator -- then loop118
>  >> wouldn't have been used and something else could use it while the live
>  >> system is running.
>  > 
>  > This is true, albeit unlikely as well.  But also fixable by addressing 
>  > the hardcoded loop device issue across the board.
>  > 
>  >>
>  >>> The reason why I think it should stay as is, instead of setting 
>  >>> everything up at initramfs time, is because the loop117 gets 
>  >>> associated with the decompressed-into-devshm contents of loop118.  
>  >>> The decompression being 7kb->1.2M in the typical case.
>  >>>
>  >>> I'd say that the 1.2M ram hit should only be taken during install, 
>  >>> and that outweighs the benefits of moving the complexity from 
>  >>> liveinst.sh to initramfs.
>  >>
>  >> But the time that the memory hit matters the _most_ is during the
>  >> install as we have to have swap disabled during parts of it. 
>  > 
>  > Obviously this is your call, and from my perspective not a big deal 
>  > either way.  My own personal preference still remains freeing up the 
>  > 1.2M of ram during all times other than the install, at the expense of 
>  > the complexity being in anaconda/liveinst.sh, versus being in initramfs.
>  > 
>  > Perhaps by F9 I'll find a mechanism to achieve this that you like...
>  
>  Or maybe in significantly less time than that...
>  
>  How's this-
>  
>  1) In initramfs, instead of lazy unmounting the squashfs, bindmount it 
>  to /sysroot/mnt/.LiveOS/squashfs
>  
>  2) store osmin 'uncompressed' on the squashfs
>  
>  3) in initramfs, set up the osimg-min device, but now with the loop118 
>  pointing at /sysroot/mnt/.LiveOS/squashfs/osmin
>  
>  The result- All the same low-impact that the current method has with 
>  anaconda, but the 1.2M of ram will only get used when anaconda actually 
>  starts touching /dev/mapper/live-osimg-min 
Okay, one better -- what if we just delay the lazy unmount of /squashfs
until after we've done the osimg-min setup.  There's no reason it has to
be mounted under the sysroot, we just have to wait to unmount the
squashed copy.

Although then the other question is what to do when we're not squashed.
Do we just then want to put it on the ext3fs?

Jeremy

[Douglas McClendon]

Jeremy Katz wrote:
>  On Tue, 2007-09-18 at 15:09 -0500, Douglas McClendon wrote:
> > Douglas McClendon wrote:
> >> Jeremy Katz wrote:
> >>> On Mon, 2007-09-17 at 22:31 -0500, Douglas McClendon wrote:
> >>>> Jeremy Katz wrote:
> >>>> The danger would be loop117, as that is the one created at install 
> >>>> time.   loop118 being locked up at initramfs time.
> >>> Not if new anaconda is used with old livecd-creator -- then loop118
> >>> wouldn't have been used and something else could use it while the live
> >>> system is running.
> >> This is true, albeit unlikely as well.  But also fixable by addressing 
> >> the hardcoded loop device issue across the board.
> >>
> >>>> The reason why I think it should stay as is, instead of setting 
> >>>> everything up at initramfs time, is because the loop117 gets 
> >>>> associated with the decompressed-into-devshm contents of loop118.  
> >>>> The decompression being 7kb->1.2M in the typical case.
> >>>>
> >>>> I'd say that the 1.2M ram hit should only be taken during install, 
> >>>> and that outweighs the benefits of moving the complexity from 
> >>>> liveinst.sh to initramfs.
> >>> But the time that the memory hit matters the _most_ is during the
> >>> install as we have to have swap disabled during parts of it. 
> >> Obviously this is your call, and from my perspective not a big deal 
> >> either way.  My own personal preference still remains freeing up the 
> >> 1.2M of ram during all times other than the install, at the expense of 
> >> the complexity being in anaconda/liveinst.sh, versus being in initramfs.
> >>
> >> Perhaps by F9 I'll find a mechanism to achieve this that you like...
> > Or maybe in significantly less time than that...
> >
> > How's this-
> >
> > 1) In initramfs, instead of lazy unmounting the squashfs, bindmount it 
> > to /sysroot/mnt/.LiveOS/squashfs
> >
> > 2) store osmin 'uncompressed' on the squashfs
> >
> > 3) in initramfs, set up the osimg-min device, but now with the loop118 
> > pointing at /sysroot/mnt/.LiveOS/squashfs/osmin
> >
> > The result- All the same low-impact that the current method has with 
> > anaconda, but the 1.2M of ram will only get used when anaconda actually 
> > starts touching /dev/mapper/live-osimg-min
>  
>  Okay, one better -- what if we just delay the lazy unmount of /squashfs
>  until after we've done the osimg-min setup.  There's no reason it has to
>  be mounted under the sysroot, we just have to wait to unmount the
>  squashed copy. 
Yeah.  The thought occurred to me because I was doing the movemounting 
myself for other reasons.  You are right, movemounting has nothing to do 
with the issue.

>  
>  Although then the other question is what to do when we're not squashed.
>  Do we just then want to put it on the ext3fs? 
Here is the heart of something I think I mentioned that was wrong a long 
time ago.  I mentioned something like "osmin can't live on the squashfs 
for obvious reasons".  Actually, the truth is "osmin can't live on the 
ext3 for obvious reasons".  And the subtle error there kept the idea out 
of my mind.

In case it's not obvious- osmin must be created after os.img is final.

But...

It occurs to me that because of what anaconda is doing, I.e. a big 2G 
dd, it may actually be problematic that squashfs pages out(?) the osmin 
data, and needs to reread it from cdrom often, thus screwing up a nice 
big linear seek-free read.

My first thought is to do a performance test, and see what happens and 
if it matters.

For now, as is, is good enough.  We'll see if it can be made better.

-dmc

[Jeremy Katz]

On Tue, 2007-09-18 at 15:48 -0500, Douglas McClendon wrote:
>  Jeremy Katz wrote:
>  > Although then the other question is what to do when we're not squashed.
>  > Do we just then want to put it on the ext3fs?
>  
>  Here is the heart of something I think I mentioned that was wrong a long 
>  time ago.  I mentioned something like "osmin can't live on the squashfs 
>  for obvious reasons".  Actually, the truth is "osmin can't live on the 
>  ext3 for obvious reasons".  And the subtle error there kept the idea out 
>  of my mind. 
The simple answer is just don't do the osmin bits if you're not using
squashfs.  Given that it's a debug option only, that's probably not
crazy

>  But...
>  
>  It occurs to me that because of what anaconda is doing, I.e. a big 2G 
>  dd, it may actually be problematic that squashfs pages out(?) the osmin 
>  data, and needs to reread it from cdrom often, thus screwing up a nice 
>  big linear seek-free read. 
Famous last words are "it shouldn't really make a difference" ;)

Jeremy

[Douglas McClendon]

Jeremy Katz wrote:
>  On Tue, 2007-09-18 at 15:48 -0500, Douglas McClendon wrote:
> > Jeremy Katz wrote:
> >> Although then the other question is what to do when we're not squashed.
> >> Do we just then want to put it on the ext3fs?
> > Here is the heart of something I think I mentioned that was wrong a long 
> > time ago.  I mentioned something like "osmin can't live on the squashfs 
> > for obvious reasons".  Actually, the truth is "osmin can't live on the
>
> > ext3 for obvious reasons".  And the subtle error there kept the idea out 
> > of my mind.
>  
>  The simple answer is just don't do the osmin bits if you're not using
>  squashfs.  Given that it's a debug option only, that's probably not
>  crazy 
Thats the answer to...?  to not having the --ignore-deleted and 
--turbo-liveinst options?  I agree.

One big problem with skip-compression as is, is that when the ext3 image 
lives natively on the iso9660

1) iso9660 does not(?) support sparse files, so currently with your 4.0G 
ext3 image containing 2.2G of data, you have to store 4.0G data.

2) iso9660 sortof has 2G filesize limits (apparently 2G->4.2G might 
work, and 4.2G->8TB could theoretically work...)

One possible solution for 1), might be to house the sparse file in a 
container filesystem (either squashfs, or a properly minimized ext2).

One possible solution for 2) would be to split the container filesystem 
(1.75G chunks?), and then associate a loop device with each, and use a 
devicemapper linear device to stitch them all together into one big image.

>  
> > But...
> >
> > It occurs to me that because of what anaconda is doing, I.e. a big 2G 
> > dd, it may actually be problematic that squashfs pages out(?) the osmin 
> > data, and needs to reread it from cdrom often, thus screwing up a nice 
> > big linear seek-free read.
>  
>  Famous last words are "it shouldn't really make a difference" ;) 
Ok, now I think I've _really_ got it figured out-

1) at livecd creator time, put the osmin in it's own squashfs.  I just 
tried this, and it produces a 12kb squashfs image.

2) now, at initramfs time, copy the squashfs image to tmpfs, loop mount 
it, associate the loop118 device with the file inside that squashfs, and 
then build the devicemapper minimized device

*NOW*, osmin once again only lives in tmpfs, *AND* ram is only used for 
its 1.2M of data during anaconda install time.

-dmc

[Jerry Vonau]

Douglas McClendon wrote:
>  Jeremy Katz wrote:
> > On Tue, 2007-09-18 at 15:48 -0500, Douglas McClendon wrote:
> >> Jeremy Katz wrote:
> >>> Although then the other question is what to do when we're not squashed.
> >>> Do we just then want to put it on the ext3fs?
> >> Here is the heart of something I think I mentioned that was wrong a
> >> long time ago.  I mentioned something like "osmin can't live on the
> >> squashfs for obvious reasons".  Actually, the truth is "osmin
> can't
> >> live on the ext3 for obvious reasons".  And the subtle error there
> >> kept the idea out of my mind.
> >
> > The simple answer is just don't do the osmin bits if you're not using
> > squashfs.  Given that it's a debug option only, that's probably not
> > crazy
>  
>  Thats the answer to...?  to not having the --ignore-deleted and
>  --turbo-liveinst options?  I agree.
>  
>  One big problem with skip-compression as is, is that when the ext3 image
>  lives natively on the iso9660
>  
>  1) iso9660 does not(?) support sparse files, so currently with your 4.0G
>  ext3 image containing 2.2G of data, you have to store 4.0G data.
>  
>  2) iso9660 sortof has 2G filesize limits (apparently 2G->4.2G might
>  work, and 4.2G->8TB could theoretically work...)
>  
>  One possible solution for 1), might be to house the sparse file in a
>  container filesystem (either squashfs, or a properly minimized ext2).
>  
>  One possible solution for 2) would be to split the container filesystem
>  (1.75G chunks?), and then associate a loop device with each, and use a
>  devicemapper linear device to stitch them all together into one big image. 
This sounds interesting, would is be possible to do something like have
a "base" image and add let's say a snapshot between gnome or kde and
"base" in a second image file? If that is possible, then using "base"
in
place of stage2.img looks promising for a combined live/install/rescue
cdrom.

Thoughts anyone?

Jerry

[Douglas McClendon]

Jerry Vonau wrote:
>  Douglas McClendon wrote:
> > Jeremy Katz wrote:
> >> On Tue, 2007-09-18 at 15:48 -0500, Douglas McClendon wrote:
> >>> Jeremy Katz wrote:
> >>>> Although then the other question is what to do when we're not
> squashed.
> >>>> Do we just then want to put it on the ext3fs?
> >>> Here is the heart of something I think I mentioned that was wrong a
> >>> long time ago.  I mentioned something like "osmin can't live on the
> >>> squashfs for obvious reasons".  Actually, the truth is "osmin
> can't
> >>> live on the ext3 for obvious reasons".  And the subtle error there
> >>> kept the idea out of my mind.
> >> The simple answer is just don't do the osmin bits if you're not using
> >> squashfs.  Given that it's a debug option only, that's probably not
> >> crazy
> > Thats the answer to...?  to not having the --ignore-deleted and
> > --turbo-liveinst options?  I agree.
> >
> > One big problem with skip-compression as is, is that when the ext3 image
> > lives natively on the iso9660
> >
> > 1) iso9660 does not(?) support sparse files, so currently with your 4.0G
> > ext3 image containing 2.2G of data, you have to store 4.0G data.
> >
> > 2) iso9660 sortof has 2G filesize limits (apparently 2G->4.2G might
> > work, and 4.2G->8TB could theoretically work...)
> >
> > One possible solution for 1), might be to house the sparse file in a
> > container filesystem (either squashfs, or a properly minimized ext2).
> >
> > One possible solution for 2) would be to split the container filesystem
> > (1.75G chunks?), and then associate a loop device with each, and use a
> > devicemapper linear device to stitch them all together into one big image.
>  
>  This sounds interesting, would is be possible to do something like have
>  a "base" image and add let's say a snapshot between gnome or kde and
>  "base" in a second image file? If that is possible, then using "base"
> in
>  place of stage2.img looks promising for a combined live/install/rescue
>  cdrom.
>  
>  Thoughts anyone? 
I think you are thinking more here about devicemapper snapshot devices. 
  I was talking about something that just split up one logical 
filesystem into parts that needed to all be put back together before 
having something useful.

But yes, one could do something like like that with snapshot devices. 
But I think there is an easier way to get what you described.

If you just took the existing livecd image, and copied the contents of 
the traditional installer DVD into it (either the whole iso, or just the 
/Fedora (or /Packages if its called that now) directory).  Then you 
could have a boot argument, which would be used in place of the existing 
'live' boot argument, say 'installer'.  Then where the current
'live' 
argument is detected, it would detect 'installer', and run an 
environment like the stage2 that you are thinking of.

But, I have pondered other similar uses of snapshots which are very much 
like what you suggested.  One thing you could do, is have the base, and 
then a snapshot for the gnome livecd, a different snapshot for the kde 
livecd, another for FEL and development, etc...  and then at the boot 
menu, let the user choose which spin to boot up, and at install time, 
let the user choose which spin to install.  Of course this mostly makes 
sense for a LiveDVD where you have that kind of room to spare.  But 
there may be better ways to get the same functionality without using 
dm-snapshots.

$0.02...

-dmc

[Jeremy Katz]

On Tue, 2007-09-18 at 19:12 -0500, Douglas McClendon wrote:
>  Jeremy Katz wrote:
>  > On Tue, 2007-09-18 at 15:48 -0500, Douglas McClendon wrote:
>  >> Jeremy Katz wrote:
>  >>> Although then the other question is what to do when we're not squashed.
>  >>> Do we just then want to put it on the ext3fs?
>  >> Here is the heart of something I think I mentioned that was wrong a long 
>  >> time ago.  I mentioned something like "osmin can't live on the squashfs
>
>  >> for obvious reasons".  Actually, the truth is "osmin can't live on
> the 
>  >> ext3 for obvious reasons".  And the subtle error there kept the idea out 
>  >> of my mind.
>  > 
>  > The simple answer is just don't do the osmin bits if you're not using
>  > squashfs.  Given that it's a debug option only, that's probably not
>  > crazy
>  
>  Thats the answer to...?  to not having the --ignore-deleted and 
>  --turbo-liveinst options?  I agree. 
To where to put the osmin image in the case of not having a squashfs.  

>  One big problem with skip-compression as is, is that when the ext3
> image 
>  lives natively on the iso9660 
Yeah, I don't see --skip-compression as something that's ever that
interesting for "real" uses.

>  >> But...
>  >>
>  >> It occurs to me that because of what anaconda is doing, I.e. a big 2G 
>  >> dd, it may actually be problematic that squashfs pages out(?) the osmin 
>  >> data, and needs to reread it from cdrom often, thus screwing up a nice 
>  >> big linear seek-free read.
>  > 
>  > Famous last words are "it shouldn't really make a difference" ;)
>  
>  Ok, now I think I've _really_ got it figured out-
>  
>  1) at livecd creator time, put the osmin in it's own squashfs.  I just 
>  tried this, and it produces a 12kb squashfs image.
>  
>  2) now, at initramfs time, copy the squashfs image to tmpfs, loop mount 
>  it, associate the loop118 device with the file inside that squashfs, and 
>  then build the devicemapper minimized device
>  
>  *NOW*, osmin once again only lives in tmpfs, *AND* ram is only used for 
>  its 1.2M of data during anaconda install time. 
Could work too -- I'm not sure it's better or worse than just sticking
it in the squashfs, though.

Jeremy