F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/CompressKernelFirmware

== Summary ==
Compress Kernel Firmwares to reduce on disk size

== Owner ==
* Name: [[User:pbrobinson| Peter Robinson]]
* Email: [mailto:[e-mail]| pbrobinson(a)fedoraproject.org]

== Detailed Description ==

Since the linux 5.3 kernel there has been support for loading firmware
from xz compressed firmware. The upstream linux-firmware respository
is now over 900Mb, not including other kernel firmware that are in
Fedora but come from other sources. By compessing the firmware with
"xz -C crc32", the only option currently supported in the kernel, we
can reduce the ondisk size of the firmware by almost half.

== Benefit to Fedora ==

Reduced on disk size of the firmware used by the kernel.

== Scope ==
* Proposal owners:
** Add support upstream to the linux-firmware copy-firmware script to
compess the firmware and create the symlinks to the compressed
firmware
** Enable the upstream support in the Fedora linux-firmware package to
compress the firmware at build time
** Enable compressing firmware in packages that contain firmware used
by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware
** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete)
* Other developers:
** No impact
* Policies and guidelines: N/A (not a System Wide Change)
* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==

There should be no upgrade impact, the support was added to the
generic firmware loader interface in the kernel.

== How To Test ==

Check devices that need firmware still work. Some devices that need
firmware include:
* GPU drivers such as nvidia, AMD and i915
* Wireless drivers such as those from Intel, Broadcom/Cyprus, TI,
Qualcomm and Marvel
* Wired network interfaces
* Storage drivers
* USB controllers and drivers

== User Experience ==

Generally users should notice little to no difference.

== Dependencies ==
N/A (not a System Wide Change)

== Contingency Plan ==

* Contingency mechanism: Don't compress kernel firmware
* Contingency deadline: GA
* Blocks release? No.
* Blocks product? No.

== Documentation ==
N/A (not a System Wide Change)

== Release Notes ==
N/A


-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis

[Neal Gompa]

On Tue, Sep 29, 2020 at 4:30 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  https://fedoraproject.org/wiki/Changes/CompressKernelFirmware
>
>  == Summary ==
>  Compress Kernel Firmwares to reduce on disk size
>
>  == Owner ==
>  * Name: [[User:pbrobinson| Peter Robinson]]
>  * Email: [mailto:[e-mail]| pbrobinson(a)fedoraproject.org]
>
>  == Detailed Description ==
>
>  Since the linux 5.3 kernel there has been support for loading firmware
>  from xz compressed firmware. The upstream linux-firmware respository
>  is now over 900Mb, not including other kernel firmware that are in
>  Fedora but come from other sources. By compessing the firmware with
>  "xz -C crc32", the only option currently supported in the kernel, we
>  can reduce the ondisk size of the firmware by almost half.
>  
I vaguely recall that there was some effort to add zstd support to
compress kernel stuff. Could we consider using that for this instead
of xz? Or is that still only for kernel modules?


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Peter Robinson]

On Tue, Sep 29, 2020 at 9:41 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Tue, Sep 29, 2020 at 4:30 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  >
>  > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware
>  >
>  > == Summary ==
>  > Compress Kernel Firmwares to reduce on disk size
>  >
>  > == Owner ==
>  > * Name: [[User:pbrobinson| Peter Robinson]]
>  > * Email: [mailto:[e-mail]| pbrobinson(a)fedoraproject.org]
>  >
>  > == Detailed Description ==
>  >
>  > Since the linux 5.3 kernel there has been support for loading firmware
>  > from xz compressed firmware. The upstream linux-firmware respository
>  > is now over 900Mb, not including other kernel firmware that are in
>  > Fedora but come from other sources. By compessing the firmware with
>  > "xz -C crc32", the only option currently supported in the kernel, we
>  > can reduce the ondisk size of the firmware by almost half.
>  >
>
>  I vaguely recall that there was some effort to add zstd support to
>  compress kernel stuff. Could we consider using that for this instead
>  of xz? Or is that still only for kernel modules? 
ATM the only thing that's upstream is xz hence why that's the only
thing mentioned.

[Zbigniew Jędrzejewski-Szmek]

On Tue, Sep 29, 2020 at 09:46:14PM +0100, Peter Robinson wrote:
>  On Tue, Sep 29, 2020 at 9:41 PM Neal Gompa <ngompa13(a)gmail.com&gt;
> wrote:
>  >
>  > On Tue, Sep 29, 2020 at 4:30 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  > >
>  > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware
>  > >
>  > > == Summary ==
>  > > Compress Kernel Firmwares to reduce on disk size
>  > >
>  > > == Owner ==
>  > > * Name: [[User:pbrobinson| Peter Robinson]]
>  > > * Email: [mailto:[e-mail]| pbrobinson(a)fedoraproject.org]
>  > >
>  > > == Detailed Description ==
>  > >
>  > > Since the linux 5.3 kernel there has been support for loading firmware
>  > > from xz compressed firmware. The upstream linux-firmware respository
>  > > is now over 900Mb, not including other kernel firmware that are in
>  > > Fedora but come from other sources. By compessing the firmware with
>  > > "xz -C crc32", the only option currently supported in the kernel, we
>  > > can reduce the ondisk size of the firmware by almost half.
>  > >
>  >
>  > I vaguely recall that there was some effort to add zstd support to
>  > compress kernel stuff. Could we consider using that for this instead
>  > of xz? Or is that still only for kernel modules?
>  
>  ATM the only thing that's upstream is xz hence why that's the only
>  thing mentioned. 
zstd support has been merged for 5.9:
https://kernelnewbies.org/Linux_5.9#Support_for_ZSTD_compressed_kernel.2C....
Does this support also extend to firmware?

Zbyszek

[Peter Robinson]

On Thu, Oct 22, 2020 at 7:53 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Tue, Sep 29, 2020 at 09:46:14PM +0100, Peter Robinson wrote:
>  > On Tue, Sep 29, 2020 at 9:41 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > >
>  > > On Tue, Sep 29, 2020 at 4:30 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  > > >
>  > > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware
>  > > >
>  > > > == Summary ==
>  > > > Compress Kernel Firmwares to reduce on disk size
>  > > >
>  > > > == Owner ==
>  > > > * Name: [[User:pbrobinson| Peter Robinson]]
>  > > > * Email: [mailto:[e-mail]|
> pbrobinson(a)fedoraproject.org]
>  > > >
>  > > > == Detailed Description ==
>  > > >
>  > > > Since the linux 5.3 kernel there has been support for loading firmware
>  > > > from xz compressed firmware. The upstream linux-firmware respository
>  > > > is now over 900Mb, not including other kernel firmware that are in
>  > > > Fedora but come from other sources. By compessing the firmware with
>  > > > "xz -C crc32", the only option currently supported in the
> kernel, we
>  > > > can reduce the ondisk size of the firmware by almost half.
>  > > >
>  > >
>  > > I vaguely recall that there was some effort to add zstd support to
>  > > compress kernel stuff. Could we consider using that for this instead
>  > > of xz? Or is that still only for kernel modules?
>  >
>  > ATM the only thing that's upstream is xz hence why that's the only
>  > thing mentioned.
>
>  zstd support has been merged for 5.9:
>
> https://kernelnewbies.org/Linux_5.9#Support_for_ZSTD_compressed_kernel.2C....
>  Does this support also extend to firmware? 
Nope, it's unrelated.

[Martin Kolman]

On Tue, 2020-09-29 at 16:29 -0400, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/CompressKernelFirmware
>  
>  == Summary ==
>  Compress Kernel Firmwares to reduce on disk size
>  
>  == Owner ==
>  * Name: [[User:pbrobinson| Peter Robinson]]
>  * Email: [mailto:[e-mail]| pbrobinson(a)fedoraproject.org]
>  
>  == Detailed Description ==
>  
>  Since the linux 5.3 kernel there has been support for loading firmware
>  from xz compressed firmware. The upstream linux-firmware respository
>  is now over 900Mb, not including other kernel firmware that are in
>  Fedora but come from other sources. By compessing the firmware with
>  "xz -C crc32", the only option currently supported in the kernel, we
>  can reduce the ondisk size of the firmware by almost half. 
I wonder if this can
also help make the boot isos and installation images smaller ? I'm sure they haver
some firmware on
them. Or alternatively we could reduce the potentially quite meintenance intensive
firmware culling we do at the moment.

>  
>  == Benefit to Fedora ==
>  
>  Reduced on disk size of the firmware used by the kernel.
>  
>  == Scope ==
>  * Proposal owners:
>  ** Add support upstream to the linux-firmware copy-firmware script to
>  compess the firmware and create the symlinks to the compressed
>  firmware
>  ** Enable the upstream support in the Fedora linux-firmware package to
>  compress the firmware at build time
>  ** Enable compressing firmware in packages that contain firmware used
>  by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware
>  ** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete)
>  * Other developers:
>  ** No impact
>  * Policies and guidelines: N/A (not a System Wide Change)
>  * Trademark approval: N/A (not needed for this Change)
>  
>  == Upgrade/compatibility impact ==
>  
>  There should be no upgrade impact, the support was added to the
>  generic firmware loader interface in the kernel.
>  
>  == How To Test ==
>  
>  Check devices that need firmware still work. Some devices that need
>  firmware include:
>  * GPU drivers such as nvidia, AMD and i915
>  * Wireless drivers such as those from Intel, Broadcom/Cyprus, TI,
>  Qualcomm and Marvel
>  * Wired network interfaces
>  * Storage drivers
>  * USB controllers and drivers
>  
>  == User Experience ==
>  
>  Generally users should notice little to no difference.
>  
>  == Dependencies ==
>  N/A (not a System Wide Change)
>  
>  == Contingency Plan ==
>  
>  * Contingency mechanism: Don't compress kernel firmware
>  * Contingency deadline: GA
>  * Blocks release? No.
>  * Blocks product? No.
>  
>  == Documentation ==
>  N/A (not a System Wide Change)
>  
>  == Release Notes ==
>  N/A
>  
>  
>  -- 
>  Ben Cotton
>  He / Him / His
>  Senior Program Manager, Fedora & CentOS Stream
>  Red Hat
>  TZ=America/Indiana/Indianapolis
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Peter Robinson]

On Thu, Oct 22, 2020 at 1:47 PM Martin Kolman <mkolman(a)redhat.com&gt; wrote:
>  On Tue, 2020-09-29 at 16:29 -0400, Ben Cotton wrote:
>  > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware
>  >
>  > == Summary ==
>  > Compress Kernel Firmwares to reduce on disk size
>  >
>  > == Owner ==
>  > * Name: [[User:pbrobinson| Peter Robinson]]
>  > * Email: [mailto:[e-mail]| pbrobinson(a)fedoraproject.org]
>  >
>  > == Detailed Description ==
>  >
>  > Since the linux 5.3 kernel there has been support for loading firmware
>  > from xz compressed firmware. The upstream linux-firmware respository
>  > is now over 900Mb, not including other kernel firmware that are in
>  > Fedora but come from other sources. By compessing the firmware with
>  > "xz -C crc32", the only option currently supported in the kernel, we
>  > can reduce the ondisk size of the firmware by almost half.
>  I wonder if this can also help make the boot isos and installation images smaller ?
> I'm sure they haver some firmware on
>  them. Or alternatively we could reduce the potentially quite meintenance intensive
> firmware culling we do at the moment. 
I wouldn't expect it to have too much of an image on the size of rpms
in the installer as the rpms are already compressed, this is more
about on disk install size.

> >
> > == Benefit to Fedora ==
> >
> > Reduced on disk size of the firmware used by the kernel.
> >
> > == Scope ==
> > * Proposal owners:
> > ** Add support upstream to the linux-firmware copy-firmware script to
> > compess the firmware and create the symlinks to the compressed
> > firmware
> > ** Enable the upstream support in the Fedora linux-firmware package to
> > compress the firmware at build time
> > ** Enable compressing firmware in packages that contain firmware used
> > by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware
> > ** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete)
> > * Other developers:
> > ** No impact
> > * Policies and guidelines: N/A (not a System Wide Change)
> > * Trademark approval: N/A (not needed for this Change)
> >
> > == Upgrade/compatibility impact ==
> >
> > There should be no upgrade impact, the support was added to the
> > generic firmware loader interface in the kernel.
> >
> > == How To Test ==
> >
> > Check devices that need firmware still work. Some devices that need
> > firmware include:
> > * GPU drivers such as nvidia, AMD and i915
> > * Wireless drivers such as those from Intel, Broadcom/Cyprus, TI,
> > Qualcomm and Marvel
> > * Wired network interfaces
> > * Storage drivers
> > * USB controllers and drivers
> >
> > == User Experience ==
> >
> > Generally users should notice little to no difference.
> >
> > == Dependencies ==
> > N/A (not a System Wide Change)
> >
> > == Contingency Plan ==
> >
> > * Contingency mechanism: Don't compress kernel firmware
> > * Contingency deadline: GA
> > * Blocks release? No.
> > * Blocks product? No.
> >
> > == Documentation ==
> > N/A (not a System Wide Change)
> >
> > == Release Notes ==
> > N/A
> >
> >
> > --
> > Ben Cotton
> > He / Him / His
> > Senior Program Manager, Fedora & CentOS Stream
> > Red Hat
> > TZ=America/Indiana/Indianapolis
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Martin Kolman]

On Thu, 2020-10-22 at 14:57 +0100, Peter Robinson wrote:
>  On Thu, Oct 22, 2020 at 1:47 PM Martin Kolman
> <mkolman(a)redhat.com&gt; wrote:
>  > On Tue, 2020-09-29 at 16:29 -0400, Ben Cotton wrote:
>  > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware
>  > > 
>  > > == Summary ==
>  > > Compress Kernel Firmwares to reduce on disk size
>  > > 
>  > > == Owner ==
>  > > * Name: [[User:pbrobinson| Peter Robinson]]
>  > > * Email: [mailto:[e-mail]| pbrobinson(a)fedoraproject.org]
>  > > 
>  > > == Detailed Description ==
>  > > 
>  > > Since the linux 5.3 kernel there has been support for loading firmware
>  > > from xz compressed firmware. The upstream linux-firmware respository
>  > > is now over 900Mb, not including other kernel firmware that are in
>  > > Fedora but come from other sources. By compessing the firmware with
>  > > "xz -C crc32", the only option currently supported in the kernel, we
>  > > can reduce the ondisk size of the firmware by almost half.
>  > I wonder if this can also help make the boot isos and installation images smaller ?
> I'm sure they haver some
>  > firmware on
>  > them. Or alternatively we could reduce the potentially quite meintenance intensive
> firmware culling we do at the
>  > moment.
>  
>  I wouldn't expect it to have too much of an image on the size of rpms
>  in the installer as the rpms are already compressed, this is more
>  about on disk install size. 
Hmm, right - the installation environment squashfs is
already compressed so indeed, no gains likely expected.

>  
>  > > == Benefit to Fedora ==
>  > > 
>  > > Reduced on disk size of the firmware used by the kernel.
>  > > 
>  > > == Scope ==
>  > > * Proposal owners:
>  > > ** Add support upstream to the linux-firmware copy-firmware script to
>  > > compess the firmware and create the symlinks to the compressed
>  > > firmware
>  > > ** Enable the upstream support in the Fedora linux-firmware package to
>  > > compress the firmware at build time
>  > > ** Enable compressing firmware in packages that contain firmware used
>  > > by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware
>  > > ** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete)
>  > > * Other developers:
>  > > ** No impact
>  > > * Policies and guidelines: N/A (not a System Wide Change)
>  > > * Trademark approval: N/A (not needed for this Change)
>  > > 
>  > > == Upgrade/compatibility impact ==
>  > > 
>  > > There should be no upgrade impact, the support was added to the
>  > > generic firmware loader interface in the kernel.
>  > > 
>  > > == How To Test ==
>  > > 
>  > > Check devices that need firmware still work. Some devices that need
>  > > firmware include:
>  > > * GPU drivers such as nvidia, AMD and i915
>  > > * Wireless drivers such as those from Intel, Broadcom/Cyprus, TI,
>  > > Qualcomm and Marvel
>  > > * Wired network interfaces
>  > > * Storage drivers
>  > > * USB controllers and drivers
>  > > 
>  > > == User Experience ==
>  > > 
>  > > Generally users should notice little to no difference.
>  > > 
>  > > == Dependencies ==
>  > > N/A (not a System Wide Change)
>  > > 
>  > > == Contingency Plan ==
>  > > 
>  > > * Contingency mechanism: Don't compress kernel firmware
>  > > * Contingency deadline: GA
>  > > * Blocks release? No.
>  > > * Blocks product? No.
>  > > 
>  > > == Documentation ==
>  > > N/A (not a System Wide Change)
>  > > 
>  > > == Release Notes ==
>  > > N/A
>  > > 
>  > > 
>  > > --
>  > > Ben Cotton
>  > > He / Him / His
>  > > Senior Program Manager, Fedora & CentOS Stream
>  > > Red Hat
>  > > TZ=America/Indiana/Indianapolis
>  > > _______________________________________________
>  > > devel mailing list -- devel(a)lists.fedoraproject.org
>  > > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Brian C. Lane]

On Thu, Oct 22, 2020 at 02:46:47PM +0200, Martin Kolman wrote:
>  I wonder if this can also help make the boot isos and installation
> images smaller ? I'm sure they haver some firmware on
>  them. Or alternatively we could reduce the potentially quite meintenance intensive
> firmware culling we do at the moment. 
By itself, probably not since things are already compressed.

But when I was doing some experiments a few months back I discovered
that using the Bcj armthumb compression reduced the size of the squashfs
image by about 400k -- I assumed it was because a bunch of the firmware
is actually arm assembly but didn't look any deeper.

So if it's possible to apply arch-specific compression to them it may
reduce things slightly.

Brian

-- 
Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart

[Chris Adams]

Once upon a time, Martin Kolman <mkolman(a)redhat.com&gt; said:
>  I wonder if this can also help make the boot isos and installation
> images smaller ? I'm sure they haver some firmware on
>  them. Or alternatively we could reduce the potentially quite meintenance intensive
> firmware culling we do at the moment. 
One thing I hadn't realized until recently is that the kernel, core
modules, and some firmware are duplicated on the install ISOs.

First, on the ISO, vmlinuz and initrd.img are duplicated in the isolinux
and images/pxeboot directories.  Or are these maybe hardlinked?  I can't
remember if you can even do that on ISO.

Beyond that, the kernel is on the ISO in isolinux/vmlinuz, and inside
the rootfs image in both /boot and /usr/lib/modules/<version>.  The core
modules and some firmware are in the ISO in initrd.img as well as in the
rootfs image.

It'd be nice to remove some of that duplication - is there any reason to
have any vmlinuz inside the rootfs image?  Could modules and firmware
copied to the initrd.img be removed as well (and then referenced from
the different mount point as needed in the installer, or maybe bind
mounted back where they came from if needed)?  While this would make the
installer setup a little more complicated, it would save some space.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Brian C. Lane]

On Thu, Oct 22, 2020 at 12:56:26PM -0500, Chris Adams wrote:
>  Once upon a time, Martin Kolman <mkolman(a)redhat.com&gt; said:
>  > I wonder if this can also help make the boot isos and installation images smaller ?
> I'm sure they haver some firmware on
>  > them. Or alternatively we could reduce the potentially quite meintenance intensive
> firmware culling we do at the moment.
>  
>  One thing I hadn't realized until recently is that the kernel, core
>  modules, and some firmware are duplicated on the install ISOs.
>  
>  First, on the ISO, vmlinuz and initrd.img are duplicated in the isolinux
>  and images/pxeboot directories.  Or are these maybe hardlinked?  I can't
>  remember if you can even do that on ISO.
>  
>  Beyond that, the kernel is on the ISO in isolinux/vmlinuz, and inside
>  the rootfs image in both /boot and /usr/lib/modules/<version>.  The core
>  modules and some firmware are in the ISO in initrd.img as well as in the
>  rootfs image.
>  
>  It'd be nice to remove some of that duplication - is there any reason to
>  have any vmlinuz inside the rootfs image?  Could modules and firmware
>  copied to the initrd.img be removed as well (and then referenced from
>  the different mount point as needed in the installer, or maybe bind
>  mounted back where they came from if needed)?  While this would make the
>  installer setup a little more complicated, it would save some space. 
I tried :) It ends up that there are users who boot the install.img with
fips=1 which means it needs to be present. It was removed for a bit, and
then I reverted the removal:

https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c2...

Brian

-- 
Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart

[Chris Adams]

Once upon a time, Brian C. Lane <bcl(a)redhat.com&gt; said:
>  I tried :) It ends up that there are users who boot the install.img
> with
>  fips=1 which means it needs to be present. It was removed for a bit, and
>  then I reverted the removal:
>  
>  https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c2... 
Ah.  Well, it was worth a try.  Since /boot and /usr/lib/modules are on
the same filesystem (in the particular case of the installer rootfs),
could they at least be hardlinked?  Although theoretically compressing
it should save that space I guess.

I still might play around with some overlay or bind-mount trickeration
to remove duplication from the boot ISO and rootfs.  Seems like that
should still pass muster... but I know zilch about FIPS mode.

And there's still the ISO duplication between images/pxelinux and
isolinux vmlinuz+initrd.img.  I think the initrd.img might have been
different at one point, but it seems like that was a very long time ago.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Chris Murphy]

On Thu, Oct 22, 2020 at 5:35 PM Chris Adams <linux(a)cmadams.net&gt; wrote:
>  Once upon a time, Brian C. Lane <bcl(a)redhat.com&gt; said:
>  > I tried :) It ends up that there are users who boot the install.img with
>  > fips=1 which means it needs to be present. It was removed for a bit, and
>  > then I reverted the removal:
>  >
>  > https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c2...
>
>  Ah.  Well, it was worth a try.  Since /boot and /usr/lib/modules are on
>  the same filesystem (in the particular case of the installer rootfs),
>  could they at least be hardlinked?  Although theoretically compressing
>  it should save that space I guess. 
Pretty sure mksquashfs automatically dedups files by hardinking them
but only if we remove the nested ext4 file system. A so-called "plain
squashfs" image. That's a feature proposal for Fedora 34.

>  And there's still the ISO duplication between images/pxelinux
> and
>  isolinux vmlinuz+initrd.img.  I think the initrd.img might have been
>  different at one point, but it seems like that was a very long time ago. 
GRUB and U-BOOT can directly read squashfs. If only syslinux/isolinux
could do that, everything could be on the squashfs file and deduped.
I'm not sure it's possible (still) for UEFI and BIOS GRUB to co-exist
on one media, hence using isolinux to boot BIOS.

ISO 9660 doesn't support hardlinks or symlinks. UDF supports both. But
I think xorriso doesn't support UDF and also I'm not sure isolinux can
boot UDF either. GRUB can.


-- 
Chris Murphy

[Chris Adams]

Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
>  On Thu, Oct 22, 2020 at 5:35 PM Chris Adams <linux(a)cmadams.net&gt;
> wrote:
>  > Once upon a time, Brian C. Lane <bcl(a)redhat.com&gt; said:
>  > > I tried :) It ends up that there are users who boot the install.img with
>  > > fips=1 which means it needs to be present. It was removed for a bit, and
>  > > then I reverted the removal:
>  > >
>  > > https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c2...
>  >
>  > Ah.  Well, it was worth a try.  Since /boot and /usr/lib/modules are on
>  > the same filesystem (in the particular case of the installer rootfs),
>  > could they at least be hardlinked?  Although theoretically compressing
>  > it should save that space I guess.
>  
>  Pretty sure mksquashfs automatically dedups files by hardinking them
>  but only if we remove the nested ext4 file system. A so-called "plain
>  squashfs" image. That's a feature proposal for Fedora 34. 
Nice.

>  > And there's still the ISO duplication between
> images/pxelinux and
>  > isolinux vmlinuz+initrd.img.  I think the initrd.img might have been
>  > different at one point, but it seems like that was a very long time ago.
>  
>  GRUB and U-BOOT can directly read squashfs. If only syslinux/isolinux
>  could do that, everything could be on the squashfs file and deduped.
>  I'm not sure it's possible (still) for UEFI and BIOS GRUB to co-exist
>  on one media, hence using isolinux to boot BIOS.
>  
>  ISO 9660 doesn't support hardlinks or symlinks. UDF supports both. But
>  I think xorriso doesn't support UDF and also I'm not sure isolinux can
>  boot UDF either. GRUB can. 
Well, there is value in having the vmlinuz/initrd.img directly available
on the ISO (for people setting up PXE boot for example).  I don't see
any reason for there to be an images/pxelinux directory though - you can
just get the same two files from the isolinux directory.

Although: now that I look at the ISO structure with isoinfo, it does
appear that the files are not actually duplicated - they are pointing to
the same sectors.  Reading the genisoimage man page, it does recognize
hard links and make only one copy on the resulting ISO image.  Should
have done more research on that part first - d'oh!
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Daniel P. Berrangé]

On Fri, Oct 23, 2020 at 09:17:52AM -0500, Chris Adams wrote:
>  Well, there is value in having the vmlinuz/initrd.img directly
> available
>  on the ISO (for people setting up PXE boot for example).  I don't see
>  any reason for there to be an images/pxelinux directory though - you can
>  just get the same two files from the isolinux directory. 
Note the .treeinfo files point to $ROOT/images/pxeboot/{vmlinuz,initrd.img}.
Tthese paths are also referenced by the libosinfo database, and used
for automating VM installs from virt-install/virt-manager/GNOME Boxes,
and probably more.  IOW please be very wary of eliminating the
images/pxeboot directory as it could trigger a cascade of brokeness.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Chris Murphy]

On Fri, Oct 23, 2020 at 8:18 AM Chris Adams <linux(a)cmadams.net&gt; wrote:
>  Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said: 
>  Although: now that I look at the ISO structure with isoinfo, it does
>  appear that the files are not actually duplicated - they are pointing to
>  the same sectors.  Reading the genisoimage man page, it does recognize
>  hard links and make only one copy on the resulting ISO image.  Should
>  have done more research on that part first - d'oh! 
That's curious, when I mount the server dvd ISO, those files have
different inode numbers.

I can't tell from the isoinfo man page what the numbers in brackets mean.

 Directory listing of /images/pxeboot/
d---------   0    0    0            2048 Oct 19 2020 [     34 02]  .
d---------   0    0    0            2048 Oct 19 2020 [     33 02]  ..
----------   0    0    0             441 Oct 19 2020 [ 306435 00]  TRANS.TBL
----------   0    0    0        78845696 Oct 19 2020 [ 306436 00]  initrd.img
----------   0    0    0        11698160 Oct 15 2020 [ 344935 00]  vmlinuz


And it looks like FIEMAP doesn't work on ISO 9660. I get nothing
useful from filefrag -v.

-- 
Chris Murphy

[Roberto Ragusa]

On 2020-10-23 17:56, Chris Murphy wrote:

>  That's curious, when I mount the server dvd ISO, those files
> have
>  different inode numbers.
>  
>  I can't tell from the isoinfo man page what the numbers in brackets mean. 
Even if inodes are different, data blocks can be reused.
The filesystem is readonly so there is no COW complication.

I've tried to have a dir with 9 hardinked copies of a 9MB file.
A simple mkisofs -o ../test.iso gave me a 10MB iso file (instead of >80MB).

Regards.

-- 
    Roberto Ragusa    mail at robertoragusa.it

[Brian C. Lane]

On Fri, Oct 23, 2020 at 09:56:11AM -0600, Chris Murphy wrote:
>  On Fri, Oct 23, 2020 at 8:18 AM Chris Adams <linux(a)cmadams.net&gt;
> wrote:
>  >
>  > Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
>  
>  > Although: now that I look at the ISO structure with isoinfo, it does
>  > appear that the files are not actually duplicated - they are pointing to
>  > the same sectors.  Reading the genisoimage man page, it does recognize
>  > hard links and make only one copy on the resulting ISO image.  Should
>  > have done more research on that part first - d'oh!
>  
>  That's curious, when I mount the server dvd ISO, those files have
>  different inode numbers. 
Sorry for the late reply on this, yet, they are hardlinked. I don't know
if the mounted representation makes that clear, but lorax specifically
hardlinks them in x86.tmpl

Brian

-- 
Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart

[Gerd Hoffmann]

  Hi,

>  > That's curious, when I mount the server dvd ISO, those files
> have
>  > different inode numbers.
>  
>  Sorry for the late reply on this, yet, they are hardlinked. 
Well, iso9660 is pretty simple.  File data is basically referenced as
"offset,length".  So hardlinked files can be trivially be made
referencing the same data blocks.  There isn't something like a link
count field in the directory listing, so this is *not* visible with
"ls".  The only way to figure the link count is scanning the whole file
system and see whenever there are other files with the same offset, and
as far I know nobody goes that extra mile just to figure the link count
for "ls".

take care,
  Gerd

[Thomas Schmitt]

Hi,

Gerd Hoffmann wrote:
>  There isn't something like a link
>  count field in the directory listing, so this is *not* visible with
>  "ls". 
There is something like a link count field. But a link count > 1 makes
full sense only if the link siblings have the same inode number. Those
numbers can be recorded in Rock Ridge entry PX if Rock Ridge 1.12 is
enabled:

  xorriso -compliance new_rr -as mkisofs ...usual.options...

But the production of Fedora ISOs does not generate them this way and the
interpretation in the Linux kernel would ignore them.


>  The only way to figure the link count is scanning the whole file
>  system 
Not with Linux.

  # mount Fedora-Workstation-Live-x86_64-31-1.9.iso
  $ ls -l /mnt/iso/images/pxeboot
  total 65293
  -rw-r--r-- 2 root root 57536044 Oct 24  2019 initrd.img
  -rwxr-xr-x 2 root root  9323208 Oct 21  2019 vmlinuz

The PX entry has a field "POSIX File Links" which "shall be used for the
st_nlink field of POSIX:5.6.1.".
This is present even in RRIP 1.10 but makes few sense there, because no
inode numbers can be recorded in 1.10.

Nevertheless Linux interprets this value in fs/isofs/rock.c :

                case SIG('P', 'X'):
                        inode->i_mode = isonum_733(rr->u.PX.mode);
                        set_nlink(inode, isonum_733(rr->u.PX.n_links));

Simple example:

  echo hello >/tmp/x

  xorriso -as mkisofs -o test.iso -graft-points /x=/tmp/x /y=/tmp/x

  mount test.iso /mnt/iso

  ls -li /mnt/iso

yields

  total 1
  1159 -rw-r--r-- 2 thomas thomas 6 Oct 26 09:58 x
  1162 -rw-r--r-- 2 thomas thomas 6 Oct 26 09:58 y

But, as said, Linux does not care if a PX entry has length >= 44 and thus
a field "File Serial Number". The struct RR_PX_s in fs/isofs/rock.h has
no field member for byte offset 36 to 43.
So we have no consistent display of effectively hardlinked files.

The shown inode numbers roughly tell where the directory entries of the
files are:

  $ expr 1159 '*' 32
  37088
  $ dd if=test.iso bs=1 skip=37088 count=110 | od -c
  0000000  \b   :   (  \0   j  \0   !  \0  \0  \0  \0  \0  \0   ! 006  \0
  0000020  \0  \0  \0  \0  \0 006   x  \n 032  \b   : 026  \0  \0  \0  \0
  0000040 001  \0  \0 001 004   X   .   ;   1  \0   P   X   $ 001 244 201
  0000060  \0  \0  \0  \0 201 244 002  \0  \0  \0  \0  \0  \0 002 350 003
  0000100  \0  \0  \0  \0 003 350 350 003  \0  \0  \0  \0 003 350   T   F
  0000120 032 001 016   x  \n 032  \b   : 026  \0   x  \a 020  \v 030   /
  0000140  \0   x  \n 032  \b   : 026  \0   N   M 006 001  \0   x
  0000156

The ISO 9660 name start byte "X" marks byte offset 33 in the directory
record.  So the "j" = 106 decimal is the size of the record. The record
ends by the Rock Ridge name entry NM, telling the Rock Ridge file name "x".
PX has size 36 and thus contains no File Serial number.

With RRIP 1.12 it looks like:

  $ dd if=test.iso bs=1 skip=37088 count=144 | od -c
  0000000 032 001 016   x  \n 032  \t 020   *  \0   x  \n 032  \t 020   *
  0000020  \0   x  \n 032  \t 020   *  \0   N   M 005 001 004  \0   r  \0
  0000040   !  \0  \0  \0  \0  \0  \0   ! 006  \0  \0  \0  \0  \0  \0 006
  0000060   x  \n 032  \b   : 026  \0  \0  \0  \0 001  \0  \0 001 004   X
  0000100   .   ;   1  \0   P   X   , 001 244 201  \0  \0  \0  \0 201 244
  0000120 002  \0  \0  \0  \0  \0  \0 002 350 003  \0  \0  \0  \0 003 350
  0000140 350 003  \0  \0  \0  \0 003 350 002  \0  \0  \0  \0  \0  \0 002
  0000160   T   F 032 001 016   x  \n 032  \b   : 026  \0   x  \n 032  \b
  0000200   :   (  \0   x  \n 032  \b   : 026  \0   N   M 006 001  \0   x

The directory record starts at the first "r", i.e. length is 114.
Here we see PX of size 44 (",") with last field value
  002  \0  \0  \0  \0  \0  \0 002
which is how ISO 9660 expresses the 32 bit value 2 (before "T   F").

In the next directory record we see the same File Serial Number 2 (again
before the field name "T   F"):

  $ dd if=test.iso bs=1 skip=37232 count=114 | od -c
  0000000   r  \0   !  \0  \0  \0  \0  \0  \0   ! 006  \0  \0  \0  \0  \0
  0000020  \0 006   x  \n 032  \b   : 026  \0  \0  \0  \0 001  \0  \0 001
  0000040 004   Y   .   ;   1  \0   P   X   , 001 244 201  \0  \0  \0  \0
  0000060 201 244 002  \0  \0  \0  \0  \0  \0 002 350 003  \0  \0  \0  \0
  0000100 003 350 350 003  \0  \0  \0  \0 003 350 002  \0  \0  \0  \0  \0
  0000120  \0 002   T   F 032 001 016   x  \n 032  \b   : 026  \0   x  \n
  0000140 032  \b   :   (  \0   x  \n 032  \b   : 026  \0   N   M 006 001
  0000160  \0   y
  0000162

I am not aware how complicated it would be to put that number 2 into
struct inode.i_ino without confusing the overall isofs driver.
The act itself would be quite trivial in above code snippet after having
extended struct RR_PX_s by the RRIP 1.12 field and interpreting the
member .len of struct rock_ridge to evaluate whether the new field is
valid.

Without reading the inode number of a file from the ISO it makes few sense
to read its link count instead of setting it flatly to 1. So an easy
change towards consistency would be to ignore rr->u.PX.n_links.


(If anybody here has a standing in kernel modules cdrom, scsi, or isofs:
I have bug fixes and helpful enhancements, but linux-scsi ignores my
first patch submission attempt since > 6 weeks.)


Have a nice day :)

Thomas

[Thomas Schmitt]

Hi,

Chris Adams wrote:
>  Although: now that I look at the ISO structure with isoinfo, it does
>  appear that the files are not actually duplicated 
ECMA-119 (ISO 9660 for free) allows the equivalent to hardlinking by

  6.5.1 Relation to File Sections
  ...
  A File Section may be part of more than one file and may occur more
  than once in the same file. A File Section may be identified by more
  than one record in the same or a different directory.

As you found out, genisoimage makes use of this prescription by putting
together hardlink siblings from the input filesystem.
libisofs underneath xorriso does the same.

-----------------------------------------------------------------------

Rock Ridge 1.12 as add-on to ISO9660 records the equivalent to inode
numbers as "File Serial Number" in PX entries.

But the Fedora ISOs i have (from genisoimage and xorriso) are equipped
with PX entries of version 1.10 which do not bear this field.
Whatever, the Linux kernel ignores the recorded numbers of version 1.12
and rather computes the inode numbers from the byte address of the
directory record divided by 32.
So two different file paths get two different artificial inode numbers.
(See isofs_get_ino() in fs/isofs/isofs.h.)

The consequence is that the hard-link-like relations do not show up in
mounted ISOs and that ISOs made from mounted ISOs don't bear this relation
either.


Have a nice day :)

Thomas

[Neal Gompa]

On Fri, Oct 23, 2020 at 1:39 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Thu, Oct 22, 2020 at 5:35 PM Chris Adams <linux(a)cmadams.net&gt; wrote:
>  >
>  > Once upon a time, Brian C. Lane <bcl(a)redhat.com&gt; said:
>  > > I tried :) It ends up that there are users who boot the install.img with
>  > > fips=1 which means it needs to be present. It was removed for a bit, and
>  > > then I reverted the removal:
>  > >
>  > > https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c2...
>  >
>  > Ah.  Well, it was worth a try.  Since /boot and /usr/lib/modules are on
>  > the same filesystem (in the particular case of the installer rootfs),
>  > could they at least be hardlinked?  Although theoretically compressing
>  > it should save that space I guess.
>
>  Pretty sure mksquashfs automatically dedups files by hardinking them
>  but only if we remove the nested ext4 file system. A so-called "plain
>  squashfs" image. That's a feature proposal for Fedora 34.
>
>  > And there's still the ISO duplication between images/pxelinux and
>  > isolinux vmlinuz+initrd.img.  I think the initrd.img might have been
>  > different at one point, but it seems like that was a very long time ago.
>
>  GRUB and U-BOOT can directly read squashfs. If only syslinux/isolinux
>  could do that, everything could be on the squashfs file and deduped.
>  I'm not sure it's possible (still) for UEFI and BIOS GRUB to co-exist
>  on one media, hence using isolinux to boot BIOS.
>
>  ISO 9660 doesn't support hardlinks or symlinks. UDF supports both. But
>  I think xorriso doesn't support UDF and also I'm not sure isolinux can
>  boot UDF either. GRUB can.
>  
Could we just drop syslinux/isolinux and use GRUB there? Other
distributions have done that because of how much of a pain syslinux
is. And it would make the ISO setup a lot less special...



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Chris Murphy]

On Fri, Oct 23, 2020 at 8:33 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Fri, Oct 23, 2020 at 1:39 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  > I'm not sure it's possible (still) for UEFI and BIOS
> GRUB to co-exist
>  > on one media, hence using isolinux to boot BIOS. 
>  Could we just drop syslinux/isolinux and use GRUB there? Other
>  distributions have done that because of how much of a pain syslinux
>  is. And it would make the ISO setup a lot less special... 
See above. Not sure. :P

I'm guessing it must be possible for the two GRUBs to co-exist if
other distros produce a single image that can boot UEFI and BIOS. I
don't know the history of why we left it as isolinux for BIOS though.

-- 
Chris Murphy

[Thomas Schmitt]

Hi,

Neal Gompa wrote:
>  Could we just drop syslinux/isolinux and use GRUB there? Other
>  distributions have done that 
I understand Ubuntu did it just yesterday.
  https://releases.ubuntu.com/20.10/ubuntu-20.10-desktop-amd64.iso

The lesson from
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148
  https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1895131
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308
is is to append the EFI partition image to the end of the ISO, use
xorriso partition offset 16 to enclose it into the overall ISO's
filesystem size, mark the partitions by valid GPT (because of recent
Lenovo laptops), and add a MBR partition hack which bears the "bootable"
flag (for old HP laptops).

I.e.

  mbr=...disk.path.to.grub.mbr.template.with.bios.x86.code...
  efi=...disk.path.to.FAT.filesystem.image.for.EFI...

  --grub2-mbr "$mbr"
  -append_partition 2 0xEF "$efi"
  -partition_offset 16
  -appended_part_as_gpt
  --mbr-force-bootable
  ...
  -b 'boot/grub/i386-pc/eltorito.img'
  ...
  -eltorito-alt-boot
  -e '--interval:appended_partition_2'
  -no-emul-boot

instead of

  -isohybrid-mbr ...disk.path.to.../isohdpfx.bin
  ...
  -b 'isolinux/isolinux.bin'
  ...
  -eltorito-alt-boot
  -e 'images/efiboot.img'
  -no-emul-boot
  -isohybrid-gpt-basdat

The equipment as used by grub-mkrescue should work. For getting an
impression, install the packages for BIOS, 64-bit x86 EFI, 32-bit x86 EFI,
and then run grub-mkrescue with some dummy directory as input.
The emerging GPT is ugly because grub-mkrescue puts the EFI partition
image as file into the ISO and marks it there. But the MBR and
/boot/grub/i386-pc/eltorito.img will probably be what you want for legacy
BIOS.
grub-mkrescue for 64-bit EFI creates a HFS+ filesystem tree and marks it
by an Apple Partition Map. One should consider whether there is still an
audience of old x86 Macs which doe not do EFI.

The new GRUB-only Ubuntu ISO seems to boot on as many systems as did the
old layout with ISOLINUX and GRUB, invented by Matthew J. Garrett for
Fedora 8 years ago.
(Fedora adds a HFS filesystem image and a Apple Partition Map, which Ubuntu
did not do in the last years. It could be replaced by a HFS+ tree like
for grub-mkrescue.)


Further an unpacked copy of the EFI partition filesystem's \BOOT\EFI
should be present in the ISO as /BOOT/EFI. MS-Windows users seem to be
used to copy the whole ISO file tree into a FAT partition and to expect
that it then boots by EFI. (The scandal is that this really does work.)


Chris Murphy wrote:
>  I don't know the history of why we left it as isolinux for BIOS
> though. 
Given above woes of Ubuntu, i'd say it was prudent conservatism.
The switch needed nerves and some help from bystanders.


Have a nice day :)

Thomas