Allowing livecd images large than 4GiB

[Bruno Wolff III]

I have submitted a feature request (allow with a patch to implement the
feature) for livecd-creator to use udf instead of iso9660 so that the
images can be larger than 4GiB. This would allow full use of DVDs
(the lowest maximum size for standard sized disks is 4.7GB) adding about
400MB of compressed space. This could also be used to create much larger
spins for use on live usb's.
I have successfully tested my patch on a spin that was over 4GiB, but still
fit on a DVD. The testing wasn't that extensive, but it booted and I was
able to run a couple of games without seeing a problem.
I am not an expert in DVD/CD formats so there may be things I have missed.

The feature request is bug 476696.

[Bryan J Smith]

On Tue, 2008-12-16 at 10:09 -0600, Bruno Wolff III wrote:
>  I have submitted a feature request (allow with a patch to implement
> the
>  feature) for livecd-creator to use udf instead of iso9660 so that the
>  images can be larger than 4GiB. This would allow full use of DVDs
>  (the lowest maximum size for standard sized disks is 4.7GB) adding about
>  400MB of compressed space. This could also be used to create much larger
>  spins for use on live usb's.
>  I have successfully tested my patch on a spin that was over 4GiB, but still
>  fit on a DVD. The testing wasn't that extensive, but it booted and I was
>  able to run a couple of games without seeing a problem.
>  I am not an expert in DVD/CD formats so there may be things I have missed.
>  The feature request is bug 476696. 
1)  I've mastered ISO 9660 Yellow Book tracks ("data" aka ".iso
files")
bigger than 4GiB before with mkisofs.  I have used 3rd Gen Matsushita
DVD-RAM drives (the first with DVD-R) for over 6 years to record these
images to DVD-R, and the LG GSA SuperMulti DVD Consortium firmware for
the last 4 years.

I noted this from Wikipedia ...

  "It is, however, possible to circumvent this limitation by using
   the multi-extent (fragmentation) feature of ISO 9660 Level 3.
   With this, files larger than 4 GiB can be split up into multiple
   extents (sequential series of sectors), each not exceeding the 4
   GiB limit. For example, the free software such as infrarecorder
   and mkisofs as well as Roxio Toast are able to create ISO 9660
   filesystems that use multi-extent files to store files larger
   than 4 GiB on appropriate media such as recordable DVDs.
   Empirical tests with a 4.2 GiB fragmented file on a DVD media have
   shown that Microsoft Windows XP supports this, while Mac OS X (as of
   10.4.8) does not handle this case properly. In the case of Mac OS X,
   the driver appears not to support file fragmentation at all (i.e. it
   only supports ISO 9660 Level 2 but not Level 3). Linux supports
   multiple extents.[4] FreeBSD only shows and reads the last extent
   of a multi-extent file."

2)  Does El'Torito support booting from UDF media, instead of ISO 9660
Yellow Book tracks?  Or ISOLinux for that matter?

If anything, it seems there is no reason not to continue utilizing ISO
9660.  I see no reason that you wouldn't have ISOLinux and all of its
files inside of the first 4GiB, before the Linux kernel loads with ISO
9660 Level 3 support.  Heck, does ISOLinux support Level 3 as well?

-- Bryan

P.S.  It's best to clarify DVD-R(G) as 4.35GiB and not just 4.7GB (which
some people may assume is 4.7GiB, when it's only 4.35GiB).


-- 
Bryan J Smith - Senior Consultant - Red Hat GPS SE US
mailto:[e-mail]      [phone] (Mobile) 
mailto:[e-mail]  (non-RH/ext to Blackberry) 
----------------------------------------------------- 
For every dollar you spend  on Red Hat solutions, you
not only fund  the leading community  development re-
source, but you receive the  #1 IT industry leader in
corporate value.  http://www.redhat.com/promo/vendor/

[Bryan J Smith]

On Tue, 2008-12-16 at 12:10 -0500, Bryan J Smith wrote:
>  2)  Does El'Torito support booting from UDF media, instead of ISO
> 9660
>  Yellow Book tracks?  Or ISOLinux for that matter? 
BTW, what did you test on?
I could see this working on EFI systems.
There are always non PC BIOS considerations.
If you tested on PC BIOS, which one?
Just curious.

>  P.S.  It's best to clarify DVD-R(G) as 4.35GiB and not just 4.7GB
> (which
>  some people may assume is 4.7GiB, when it's only 4.35GiB). 


-- 
Bryan J Smith - Senior Consultant - Red Hat GPS SE US
mailto:[e-mail]      [phone] (Mobile) 
mailto:[e-mail]  (non-RH/ext to Blackberry) 
----------------------------------------------------- 
For every dollar you spend  on Red Hat solutions, you
not only fund  the leading community  development re-
source, but you receive the  #1 IT industry leader in
corporate value.  http://www.redhat.com/promo/vendor/

[Bruno Wolff III]

On Tue, Dec 16, 2008 at 12:19:15 -0500,
  Bryan J Smith <bjs(a)redhat.com&gt; wrote:
>  On Tue, 2008-12-16 at 12:10 -0500, Bryan J Smith wrote:
>  > 2)  Does El'Torito support booting from UDF media, instead of ISO 9660
>  > Yellow Book tracks?  Or ISOLinux for that matter?
>  
>  BTW, what did you test on?
>  I could see this working on EFI systems.
>  There are always non PC BIOS considerations.
>  If you tested on PC BIOS, which one?
>  Just curious. 
I tested it on a dual athlon system. The last motherboard update was something
on the order of 5 years ago. So I would expect things to work on a wide
range of systems.

My reading of the documentation was that the iso image looks like iso9660
except the file size for the isoimage is reduced and hence is inconsistent
with the udf information. The documentation said that to access the file
you would need to mount it with the udf or ufs drivers. My expectation
is the bios never gets that far as it only needs to see the boot information.
Once things are booting and linux is running I wouldn't expect problems.

[Bryan J Smith]

On Tue, 2008-12-16 at 11:33 -0600, Bruno Wolff III wrote:
>  I tested it on a dual athlon system. The last motherboard update was
> something
>  on the order of 5 years ago. So I would expect things to work on a wide
>  range of systems. 
Well, then I'd say you tested it.  You're sure the filesystem is UDF
Plain, correct?  It wasn't ISO 9660 Yellow Book (possibly as a result of
UDF not being accepted as an option for some reason)?

>  My reading of the documentation was that the iso image looks like
> iso9660
>  except the file size for the isoimage is reduced and hence is inconsistent
>  with the udf information. 
I've actually never dissected the ISO files I regularly create with
mkisofs.  But I know I've gone beyond 4.0GiB (~4.3GB) before, so they
must be Level 3.

We're kinda spoiled on Linux, because an .iso file != as written, but
the kernel hides a lot of that in the block access.  An .iso file, last
time I checked, is ISO 9660 "Yellow Book" (data) track.  So I'm curious
if an .iso file, as well as written to optical media, is still a single
track for > 4.0GiB (> ~4.3GB), or multiple Yellow Book tracks with the
extent in the first. 

>  The documentation said that to access the file
>  you would need to mount it with the udf or ufs drivers. My expectation
>  is the bios never gets that far as it only needs to see the boot information. 
Depends.  I was under the impression that when not using PC BIOS floppy
emulation, but native mode with El Torito, it uses still requires and
accesses the ISO 9660 Yellow Track, which needs to have filenames
limited to 31 characters.

But I might be confusing ISOLinux specifics.

>  Once things are booting and linux is running I wouldn't expect
> problems. 
Agreed there.


-- 
Bryan J Smith - Senior Consultant - Red Hat GPS SE US
mailto:[e-mail]      [phone] (Mobile) 
mailto:[e-mail]  (non-RH/ext to Blackberry) 
----------------------------------------------------- 
For every dollar you spend  on Red Hat solutions, you
not only fund  the leading community  development re-
source, but you receive the  #1 IT industry leader in
corporate value.  http://www.redhat.com/promo/vendor/

[Bruno Wolff III]

On Tue, Dec 16, 2008 at 12:57:22 -0500,
  Bryan J Smith <bjs(a)redhat.com&gt; wrote:
>  
>  Well, then I'd say you tested it.  You're sure the filesystem is UDF
>  Plain, correct?  It wasn't ISO 9660 Yellow Book (possibly as a result of
>  UDF not being accepted as an option for some reason)? 
I am not sure how to tell. I changed things so the initscript was going to
mount it using a fs type of udf. On the test before I changed the drivers
and the fs type the mount failed.

>  I've actually never dissected the ISO files I regularly create
> with
>  mkisofs.  But I know I've gone beyond 4.0GiB (~4.3GB) before, so they
>  must be Level 3. 
The livecd-creator process barfs in the squashfs file system is over
4 GIB compressed. The mkisofs documentation says this should happen
at 2 GiB. So I am not entirely sure what gives there.

>  We're kinda spoiled on Linux, because an .iso file != as written,
> but
>  the kernel hides a lot of that in the block access.  An .iso file, last
>  time I checked, is ISO 9660 "Yellow Book" (data) track.  So I'm curious
>  if an .iso file, as well as written to optical media, is still a single
>  track for > 4.0GiB (> ~4.3GB), or multiple Yellow Book tracks with the
>  extent in the first.  
I suspect it was one track, but I don't know for sure. I used k3b and the
burn iso dvd tool.

>  Depends.  I was under the impression that when not using PC BIOS
> floppy
>  emulation, but native mode with El Torito, it uses still requires and
>  accesses the ISO 9660 Yellow Track, which needs to have filenames
>  limited to 31 characters. 
But there aren't many files visible at that level. A few things in the
equivalent of /boot and the rest is the file containing the squashfs
file system.

[Bryan J Smith]

On Tue, 2008-12-16 at 14:25 -0600, Bruno Wolff III wrote:
>  The livecd-creator process barfs in the squashfs file system is over
>  4 GIB compressed. 
Are you sure that's squashfs, or the kickstart image size?
I ran into that too, but it was the image size.

>  The mkisofs documentation says this should happen
>  at 2 GiB. So I am not entirely sure what gives there. 
You may be correct.

>  I suspect it was one track, but I don't know for sure. I used k3b
> and the
>  burn iso dvd tool. 
k3b actually uses several tools.  It can use dvd+rw-tools as well.  

E.g., DVD+R media is actually DVD+RW, and nothing like CD-R or DVD-R.

DVD+RW Consortium (Sony/Philips) firmware also has a different focus
than DVD Consortium (Matsushita/Pioneer) drives.

>  But there aren't many files visible at that level. A few things
> in the
>  equivalent of /boot and the rest is the file containing the squashfs
>  file system. 
But doesn't the PC BIOS' El Torito support need to read the Yellow Book
track to get to them?  That's what I assumed, but I could be wrong.  I
mean, it still has to load ISOLinux.


-- 
Bryan J Smith - Senior Consultant - Red Hat GPS SE US
mailto:[e-mail]      [phone] (Mobile) 
mailto:[e-mail]  (non-RH/ext to Blackberry) 
----------------------------------------------------- 
For every dollar you spend  on Red Hat solutions, you
not only fund  the leading community  development re-
source, but you receive the  #1 IT industry leader in
corporate value.  http://www.redhat.com/promo/vendor/

[Bruno Wolff III]

On Tue, Dec 16, 2008 at 12:10:30 -0500,
  Bryan J Smith <bjs(a)redhat.com&gt; wrote:
>  
>  If anything, it seems there is no reason not to continue utilizing ISO
>  9660.  I see no reason that you wouldn't have ISOLinux and all of its
>  files inside of the first 4GiB, before the Linux kernel loads with ISO
>  9660 Level 3 support.  Heck, does ISOLinux support Level 3 as well? 
Is that better supported than udf? Is it easy to use with mkisofs?
Is there a way to guaranty proper placement of /boot where the bios
can see it?

[Jeremy Katz]

On Tue, 2008-12-16 at 10:09 -0600, Bruno Wolff III wrote:
>  I have submitted a feature request (allow with a patch to implement
> the
>  feature) for livecd-creator to use udf instead of iso9660 so that the
>  images can be larger than 4GiB. This would allow full use of DVDs
>  (the lowest maximum size for standard sized disks is 4.7GB) adding about
>  400MB of compressed space. This could also be used to create much larger
>  spins for use on live usb's. 
I'm not against allowing udf, but we should probably use iso9660 unless
the size forces us to need udf just to avoid problems on as many systems
as we can.  

Jeremy

[Bruno Wolff III]

On Tue, Dec 16, 2008 at 17:49:01 -0500,
  Jeremy Katz <katzj(a)redhat.com&gt; wrote:
>  On Tue, 2008-12-16 at 10:09 -0600, Bruno Wolff III wrote:
>  > I have submitted a feature request (allow with a patch to implement the
>  > feature) for livecd-creator to use udf instead of iso9660 so that the
>  > images can be larger than 4GiB. This would allow full use of DVDs
>  > (the lowest maximum size for standard sized disks is 4.7GB) adding about
>  > 400MB of compressed space. This could also be used to create much larger
>  > spins for use on live usb's.
>  
>  I'm not against allowing udf, but we should probably use iso9660 unless
>  the size forces us to need udf just to avoid problems on as many systems
>  as we can.   
So something that checked the size of the mksquashfs and switched automatically
between the two and emitting a message about this could be acceptable?
I don't really know python yet, so I may not be able to easily change
my previous patch to do that, but want to check before going down that road.
I think having access to teh extra space will be nice for the Games Spin as
things are pretty tight right now and a few hundred MB would be nice to have.

[Jeremy Katz]

On Dec 16, 2008, at 6:09 PM, Bruno Wolff III <bruno(a)wolff.to&gt; wrote:

>  On Tue, Dec 16, 2008 at 17:49:01 -0500,
>   Jeremy Katz <katzj(a)redhat.com&gt; wrote:
> > On Tue, 2008-12-16 at 10:09 -0600, Bruno Wolff III wrote:
> >> I have submitted a feature request (allow with a patch to  
> >> implement the
> >> feature) for livecd-creator to use udf instead of iso9660 so that  
> >> the
> >> images can be larger than 4GiB. This would allow full use of DVDs
> >> (the lowest maximum size for standard sized disks is 4.7GB) adding  
> >> about
> >> 400MB of compressed space. This could also be used to create much  
> >> larger
> >> spins for use on live usb's.
> >
> > I'm not against allowing udf, but we should probably use iso9660  
> > unless
> > the size forces us to need udf just to avoid problems on as many  
> > systems
> > as we can.
>
>  So something that checked the size of the mksquashfs and switched  
>  automatically
>  between the two and emitting a message about this could be acceptable?
>  I don't really know python yet, so I may not be able to easily change
>  my previous patch to do that, but want to check before going down  
>  that road. 
Yep, exactly what I'm thinking. And it should be a fairly straight  
forward matter of doing os.stat on the squashfs and then looking at  
the size from that

Jeremy

[Bryan J Smith]

On Tue, 2008-12-16 at 17:49 -0500, Jeremy Katz wrote:
>  I'm not against allowing udf, but we should probably use iso9660
> unless
>  the size forces us to need udf just to avoid problems on as many systems
>  as we can.   
I just wanted to note that I was analyzing from a standpoint of
ignorance and assumption, and not hands-on testing with the patch.  I
don't anyone basing decisions on my commentary other than as a "review,"
but not actual testing.


-- 
Bryan J Smith - Senior Consultant - Red Hat GPS SE US
mailto:[e-mail]      [phone] (Mobile) 
mailto:[e-mail]  (non-RH/ext to Blackberry) 
----------------------------------------------------- 
For every dollar you spend  on Red Hat solutions, you
not only fund  the leading community  development re-
source, but you receive the  #1 IT industry leader in
corporate value.  http://www.redhat.com/promo/vendor/