Progress Report: Upping the 4gb filesystem limit

[David Laban]

Just a heads up that I'm working on a couple of patches to
livecd-iso-to-disk that let you up the filesystem limit.

The current setup is:

filesystem
|_ livecd.iso
   |_ squashfs.img --read-only?
      |_ ext3fs.img

I have prototyped it, and I had to:

* cp --sparse=always ext3fs.img <filesystem with sparse support and
2.5GB free> // If I could mount the squashfs image read-write or do
some kind of snapshot/overlay, this would be unneccesary

* dd seek=8GB of=ext3fs.img
* losetup ext3fs.img
* resize2fs /dev/loopX // automatically expands to fill the whole
'partition' == 8GB
* mkdir -p tmp/LiveOS/
* mv ext3fs.img tmp/LiveOS/
* mksquashfs tmp squashfs.img // takes ages: has to re-compress 2.5GB

Can anyone give any hints on making this more efficient, or should I
submit a patch to
http://git.fedorahosted.org/git/?p=hosted/livecd;a=blob;f=tools/livecd-iso-t...
as it is?

Also, I need to do more research into snapshots (had a wild goose
chase grepping the kernel source for "overlay") but I think that I
might be able to re-compact the overlay file into the squashfs using
the above process. Would anyone be interested in seeing a patch of
that form?

Also, I think that pre-allocating 2GB from /dev/zero on FAT
filesystems is pointless and needlessly slow. Doesn't the overlay grow
itself anyway? Also, isn't the limit 4GB for FAT32?

David.

[Frederick Grose]

On Sun, Jun 17, 2012 at 6:25 PM, David Laban  wrote:
> Just a heads up that I'm working on a couple of patches to
> livecd-iso-to-disk that let you up the filesystem limit.
>
> The current setup is:
>
> filesystem
> |_ livecd.iso
>   |_ squashfs.img --read-only?
>      |_ ext3fs.img
>
> I have prototyped it, and I had to:
> cp --sparse=always ext3fs.img <filesystem with sparse support and
>
> 2.5GB free> // If I could mount the squashfs image read-write or do
> some kind of snapshot/overlay, this would be unneccesary
> dd seek=8GB of=ext3fs.img
> losetup ext3fs.img
> resize2fs /dev/loopX // automatically expands to fill the whole
>
> 'partition' == 8GB
> mkdir -p tmp/LiveOS/
> mv ext3fs.img tmp/LiveOS/
> mksquashfs tmp squashfs.img // takes ages: has to re-compress 2.5GB
>
> Can anyone give any hints on making this more efficient, or should I
> submit a patch to
>
> http://git.fedorahosted.org/git/?p=hosted/livecd;a=blob;f=tools/livecd-iso-t...
> as it is?
>
> Also, I need to do more research into snapshots (had a wild goose
> chase grepping the kernel source for "overlay") but I think that I
> might be able to re-compact the overlay file into the squashfs using
> the above process. Would anyone be interested in seeing a patch of
> that form?

Yes, such a patch is feasible.
See
http://wiki.sugarlabs.org/go/LiveOS_image

for more information on the LiveOS image.

I have a version of edit-livecd,
http://git.fedorahosted.org/git/?p=hosted/livecd;a=blob;f=tools/edit-livecd;...
, which I call editliveos.py, that uses a virtual filesystem mirror to
merge the ext3fs.img with the overlay to create an updated, single
filesystem image file (instead of using rsync, as the current version does
with the --clone option).

(I haven't tested the rsync version in a long time, since it failed,
http://www.mail-archive.com/[e-mail]/msg01052.html
, and I haven't seen other reports of its use.)

My current version of editliveos.py has a perplexing bug where an image
built from a live running source will reboot, but one built from an
attached LiveOS source fails to reboot.  This was not previously a problem,
and I suspect some side effect of the systemd startup changes on the way
new sessions or seats are authenticated.

The new version involves updates to fs.py, util.py, live.py, and creator.py
and I have not submitted patches yet for review, but I have just posted
whole files at
http://git.sugarlabs.org/soas/sugar-clone-extensions/commit/8c4e03323fc9a7e6...

--Fred
> Also, I think that pre-allocating 2GB from /dev/zero on FAT
> filesystems is pointless and needlessly slow. Doesn't the overlay grow
> itself anyway?

The overlay file is fixed in size.

Also, isn't the limit 4GB for FAT32?
> David.
>
> livecd mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/livecd

[Frederick Grose]

On Sun, Jun 17, 2012 at 8:19 PM, Frederick Grose  wrote:
> On Sun, Jun 17, 2012 at 6:25 PM, David Laban  wrote:
> > Just a heads up that I'm working on a couple of patches to
> > livecd-iso-to-disk that let you up the filesystem limit.
> >
> > The current setup is:
> >
> > filesystem
> > |_ livecd.iso
> >   |_ squashfs.img --read-only?
> >      |_ ext3fs.img
> >
> > I have prototyped it, and I had to:
> > cp --sparse=always ext3fs.img <filesystem with sparse support and
> >
> > 2.5GB free> // If I could mount the squashfs image read-write or do
> > some kind of snapshot/overlay, this would be unneccesary
> > dd seek=8GB of=ext3fs.img
> > losetup ext3fs.img
> > resize2fs /dev/loopX // automatically expands to fill the whole
> >
> > 'partition' == 8GB
> > mkdir -p tmp/LiveOS/
> > mv ext3fs.img tmp/LiveOS/
> > mksquashfs tmp squashfs.img // takes ages: has to re-compress 2.5GB
> >
> > Can anyone give any hints on making this more efficient, or should I
> > submit a patch to
> >
> > http://git.fedorahosted.org/git/?p=hosted/livecd;a=blob;f=tools/livecd-iso-t...
> > as it is?
> >
> > Also, I need to do more research into snapshots (had a wild goose
> > chase grepping the kernel source for "overlay") but I think that I
> > might be able to re-compact the overlay file into the squashfs using
> > the above process. Would anyone be interested in seeing a patch of
> > that form?
>
> Yes, such a patch is feasible.
> See
> http://wiki.sugarlabs.org/go/LiveOS_image
>
> for more information on the LiveOS image.
>
> I have a version of edit-livecd,
>
> http://git.fedorahosted.org/git/?p=hosted/livecd;a=blob;f=tools/edit-livecd;...
> , which I call editliveos.py, that uses a virtual filesystem mirror to
> merge the ext3fs.img with the overlay to create an updated, single
> filesystem image file (instead of using rsync, as the current version does
> with the --clone option).
>
> (I haven't tested the rsync version in a long time, since it failed,
> http://www.mail-archive.com/[e-mail]/msg01052.html
> , and I haven't seen other reports of its use.)
>
> My current version of editliveos.py has a perplexing bug where an image
> built from a live running source will reboot, but one built from an
> attached LiveOS source fails to reboot.  This was not previously a problem,
> and I suspect some side effect of the systemd startup changes on the way
> new sessions or seats are authenticated.
>
> The new version involves updates to fs.py, util.py, live.py, and
> creator.py and I have not submitted patches yet for review, but I have just
> posted whole files at
>
> http://git.sugarlabs.org/soas/sugar-clone-extensions/commit/8c4e03323fc9a7e6...

editliveos.py includes several new options for adjusting the image
filesystem size as well as home.img and overlay file sizes. I've posted the
output of editliveos.py --help at this wiki section,
http://wiki.sugarlabs.org/go/Sugar_on_a_Stick/Sugar_Clone#editliveos.help
(Expand the [show] frame.)

(Most of the other content on that page is from earlier work.)
>        --Fred
>
>
> > Also, I think that pre-allocating 2GB from /dev/zero on FAT
> > filesystems is pointless and needlessly slow. Doesn't the overlay grow
> > itself anyway?
>
> The overlay file is fixed in size.
>
> Also, isn't the limit 4GB for FAT32?
> > David.
> >
> > livecd mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/livecd

[Bruno Wolff III]

On Sun, Jun 17, 2012 at 23:25:58 +0100,
   David Laban  wrote:
> Just a heads up that I'm working on a couple of patches to
> livecd-iso-to-disk that let you up the filesystem limit.

One thing to note is that there is a move to replace livecd-creator with 
livemedia-creator. Depending on why you are doing this, it might be better 
to see if the same limit exists in livemedia-creator and if so, consider 
on fixing the limit there.