F37 proposal: Linux Firmware Minimization (late System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==
Split linux-firmware into more subpackages, and add the ability to
automatically install firmware based on the hardware present

== Owner ==
* Name: [[User:Salimma|Michel Alexandre Salim]],
[[User:Dcavalca|Davide Cavalca]]
* Email: michel(a)fb.com, dcavalca(a)fb.com


== Detailed Description ==
The `linux-firmware` RPM is very large (175M src.rpm, 287M
*.noarch.rpm per 20211027-126) that bundles most of the system
firmware loaded by the kernel, regardless of whether it’s actually
needed. Some additional firmwares are already split up into individual
subpackages. This change would extend that, splitting out most firmare
into appropriate subpackages. The Change would also make the
subpackages `Supplements` the appropriate `modalias(...)` for the
hardware they support and `Provides`
`firmware(kmodname/firmwarefile.bin)`

Candidates for splitting out:
* CPU firmwares
* GPU firmwares
* Non-Intel wifi firmwares
* Bluetooth firmwares

Conversely some firmware could probably be grouped together, e.g. have
iwlwifi-firmware that pulls in iwl*-firmware (corresponding to
openSUSE's kernel-firmware-iwlwifi).

We will also introduce `linux-firmware-all` that will pull in all
subpackages; this will make it easier to manage the firmware packages
in different Fedora variants.

A DNF plugin will be written to make use of the `Supplements` metadata
to automatically install the appropriate firmware packages based on
the hardware present on the system (see openSUSE's `libzypp`:
https://github.com/openSUSE/libzypp/blob/a34d857dbe3b16d4a7e0219cd213cc5a...
and
https://github.com/openSUSE/libzypp/blob/7f345ea4892fd02345e8de47c2a08ab5...)

== Feedback ==


== Benefit to Fedora ==
This would save a lot of disk space on Fedora installations, and make
it easier to test individual firmware updates by replacing a single
subpackage. It will also bring us into alignment with openSUSE (see
their
[https://code.opensuse.org/package/kernel-firmware/blob/master/f/kernel-fi...
kernel-firmware.spec]).

== Scope ==
* Proposal owners:
** Create test COPR with proposed changes
** Add some additional firmware subpackages
** Add the required metadata (supplements, provides)
** Write DNF plugin to make use of supplements
** Write libdnf plugin for PackageKit (and in the future, microdnf)
** refactor [https://pagure.io/fedora-comps/blob/main/f/comps-f37.xml.in#_2887-2903
comps for F37]

* Other developers:
** Validate that installation works with iwl*-firmware replaced by
iwlwifi-firmware
** Test the DNF plugin
** If it works well, preinstall the DNF plugin and stop installing
iwlwifi-firmware or any of the iwl*-firmware

* Release engineering: [https://pagure.io/releng/issues #Releng issue number]
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:


== Upgrade/compatibility impact ==
No compatibility impact.


== How To Test ==
* Enable the test COPR
* Verify that after upgrading, no firmware subpackages are removed
* Install the DNF plugin
* Verify that the DNF plugin works: `dnf mark remove` a firmware
package that is needed, then verify that `dnf autoremove` does not
remove it
* `dnf mark remove` all firmware packages, and verify that `dnf
autoremove` only remove the unneeded packages


== User Experience ==
More disk space available, no functional changes.

== Dependencies ==
No dependency, `linux-firmware` is not required by any package

== Contingency Plan ==
* Contingency mechanism: If the plugin is not ready in time, the other
changes would preserve the status quo. So we can postpone shipping the
DNF plugin to F38.
* Contingency deadline:<br>Decide on whether to postpone the DNF
plugin by beta freeze
* Blocks release? No


== Documentation ==
Documentation will be added in
[https://hackmd.io/KnHbJ40pQ8eJjc1H80R10w this HackMD]

== Release Notes ==



-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Michael Catanzaro]

On Fri, Jul 1 2022 at 01:54:58 PM -0400, Ben Cotton 
<bcotton(a)redhat.com&gt; wrote:
>  A DNF plugin will be written to make use of the `Supplements`
> metadata
>  to automatically install the appropriate firmware packages based on
>  the hardware present on the system (see openSUSE's `libzypp`:
>
> https://github.com/openSUSE/libzypp/blob/a34d857dbe3b16d4a7e0219cd213cc5a...
>  and 
>
> https://github.com/openSUSE/libzypp/blob/7f345ea4892fd02345e8de47c2a08ab5...)
What about Fedora editions that don't have dnf (Silverblue, Kinoite, 
CoreOS)?

[Neal Gompa]

On Fri, Jul 1, 2022 at 2:10 PM Michael Catanzaro <mcatanzaro(a)gnome.org&gt; wrote:
>  On Fri, Jul 1 2022 at 01:54:58 PM -0400, Ben Cotton
>  <bcotton(a)redhat.com&gt; wrote:
>  > A DNF plugin will be written to make use of the `Supplements` metadata
>  > to automatically install the appropriate firmware packages based on
>  > the hardware present on the system (see openSUSE's `libzypp`:
>  >
> https://github.com/openSUSE/libzypp/blob/a34d857dbe3b16d4a7e0219cd213cc5a...
>  > and
>  >
> https://github.com/openSUSE/libzypp/blob/7f345ea4892fd02345e8de47c2a08ab5...)
>
>  What about Fedora editions that don't have dnf (Silverblue, Kinoite,
>  CoreOS)? 
They load libdnf through rpm-ostree, so a libdnf plugin will still work.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Dusty Mabe]

On 7/1/22 18:14, Neal Gompa wrote:
>  On Fri, Jul 1, 2022 at 2:10 PM Michael Catanzaro
> <mcatanzaro(a)gnome.org&gt; wrote:
> >
> > On Fri, Jul 1 2022 at 01:54:58 PM -0400, Ben Cotton
> > <bcotton(a)redhat.com&gt; wrote:
> >> A DNF plugin will be written to make use of the `Supplements` metadata
> >> to automatically install the appropriate firmware packages based on
> >> the hardware present on the system (see openSUSE's `libzypp`:
> >>
> https://github.com/openSUSE/libzypp/blob/a34d857dbe3b16d4a7e0219cd213cc5a...
> >> and
> >>
> https://github.com/openSUSE/libzypp/blob/7f345ea4892fd02345e8de47c2a08ab5...)
> >
> > What about Fedora editions that don't have dnf (Silverblue, Kinoite,
> > CoreOS)?
>  
>  They load libdnf through rpm-ostree, so a libdnf plugin will still work.
>   
I'm not the most qualified to speak to this point, but I'm not sure that just
because rpm-ostree uses libdnf
under the hood this will automatically just work. I suspect that it won't work.
Probably for those variants
we'd just need to include the `linux-firmware-all` package (or whatever equivalent to
get what we currently have).

Dusty

[Timothée Ravier]

Agree with Dusty here: it's likely that the DNF plugin won't be used at all with
rpm-ostree.

This change would however enable some rpm-ostree variants to more finely select the
firmwares that they want: for example, Fedora CoreOS could skip all WiFi related firmwares
and Fedora Silverblue could skip weird server hardware firmwares.

[Chris Murphy]

On Fri, Jul 1, 2022 at 9:47 PM Dusty Mabe <dusty(a)dustymabe.com&gt; wrote:
>  On 7/1/22 18:14, Neal Gompa wrote:
>  > On Fri, Jul 1, 2022 at 2:10 PM Michael Catanzaro <mcatanzaro(a)gnome.org&gt;
> wrote:
>  >>
>  >> On Fri, Jul 1 2022 at 01:54:58 PM -0400, Ben Cotton
>  >> <bcotton(a)redhat.com&gt; wrote:
>  >>> A DNF plugin will be written to make use of the `Supplements` metadata
>  >>> to automatically install the appropriate firmware packages based on
>  >>> the hardware present on the system (see openSUSE's `libzypp`:
>  >>>
> https://github.com/openSUSE/libzypp/blob/a34d857dbe3b16d4a7e0219cd213cc5a...
>  >>> and
>  >>>
> https://github.com/openSUSE/libzypp/blob/7f345ea4892fd02345e8de47c2a08ab5...)
>  >>
>  >> What about Fedora editions that don't have dnf (Silverblue, Kinoite,
>  >> CoreOS)?
>  >
>  > They load libdnf through rpm-ostree, so a libdnf plugin will still work.
>  >
>
>  I'm not the most qualified to speak to this point, but I'm not sure that just
> because rpm-ostree uses libdnf
>  under the hood this will automatically just work. I suspect that it won't work.
> Probably for those variants
>  we'd just need to include the `linux-firmware-all` package (or whatever equivalent to
> get what we currently have). 

Or could the plugin be used to create rpm-ostree layers with the
necessary firmware packages?

-- 
Chris Murphy

[Neal Gompa]

On Fri, Jul 1, 2022 at 1:57 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization
>
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>
>  == Summary ==
>  Split linux-firmware into more subpackages, and add the ability to
>  automatically install firmware based on the hardware present
>
>  == Owner ==
>  * Name: [[User:Salimma|Michel Alexandre Salim]],
>  [[User:Dcavalca|Davide Cavalca]]
>  * Email: michel(a)fb.com, dcavalca(a)fb.com
>
>
>  == Detailed Description ==
>  The `linux-firmware` RPM is very large (175M src.rpm, 287M
>  *.noarch.rpm per 20211027-126) that bundles most of the system
>  firmware loaded by the kernel, regardless of whether it’s actually
>  needed. Some additional firmwares are already split up into individual
>  subpackages. This change would extend that, splitting out most firmare
>  into appropriate subpackages. The Change would also make the
>  subpackages `Supplements` the appropriate `modalias(...)` for the
>  hardware they support and `Provides`
>  `firmware(kmodname/firmwarefile.bin)`
>
>  Candidates for splitting out:
>  * CPU firmwares
>  * GPU firmwares
>  * Non-Intel wifi firmwares
>  * Bluetooth firmwares
>
>  Conversely some firmware could probably be grouped together, e.g. have
>  iwlwifi-firmware that pulls in iwl*-firmware (corresponding to
>  openSUSE's kernel-firmware-iwlwifi).
>
>  We will also introduce `linux-firmware-all` that will pull in all
>  subpackages; this will make it easier to manage the firmware packages
>  in different Fedora variants.
>
>  A DNF plugin will be written to make use of the `Supplements` metadata
>  to automatically install the appropriate firmware packages based on
>  the hardware present on the system (see openSUSE's `libzypp`:
>
> https://github.com/openSUSE/libzypp/blob/a34d857dbe3b16d4a7e0219cd213cc5a...
>  and
> https://github.com/openSUSE/libzypp/blob/7f345ea4892fd02345e8de47c2a08ab5...)
>
>  == Feedback ==
>
>
>  == Benefit to Fedora ==
>  This would save a lot of disk space on Fedora installations, and make
>  it easier to test individual firmware updates by replacing a single
>  subpackage. It will also bring us into alignment with openSUSE (see
>  their
> [https://code.opensuse.org/package/kernel-firmware/blob/master/f/kernel-fi...
>  kernel-firmware.spec]).
>  
This is sorely needed, and I'm really looking forward to this being implemented!



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Miroslav Suchý]

Dne 01. 07. 22 v 19:54 Ben Cotton napsal(a):
>  == Detailed Description ==
>  The `linux-firmware` RPM is very large (175M src.rpm, 287M
>  *.noarch.rpm per 20211027-126) that bundles most of the system
>  firmware loaded by the kernel, regardless of whether it’s actually
>  needed. Some additional firmwares are already split up into individual
>  subpackages. This change would extend that, splitting out most firmare
>  into appropriate subpackages. The Change would also make the
>  subpackages `Supplements` the appropriate `modalias(...)` for the
>  hardware they support and `Provides`
>  `firmware(kmodname/firmwarefile.bin)` 
Wow, I wanted this for loong time. Thank you for doing this.

>  A DNF plugin will be written to make use of the `Supplements`
> metadata
>  to automatically install the appropriate firmware packages based on
>  the hardware present on the system (see openSUSE's `libzypp`:
>
> https://github.com/openSUSE/libzypp/blob/a34d857dbe3b16d4a7e0219cd213cc5a...
>  and
> https://github.com/openSUSE/libzypp/blob/7f345ea4892fd02345e8de47c2a08ab5...)
Can this be paired with simple command line tool which will "just" print the
appropriate packages? It would  be usefull 
in e.g., creating Ansible playbooks for the machine.

Miroslav

[Vitaly Zaitsev]

On 01/07/2022 19:54, Ben Cotton wrote:
>  Split linux-firmware into more subpackages, and add the ability to
>  automatically install firmware based on the hardware present 
+1. Finally. Thanks for implementing this.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Ben Cotton]

On Fri, Jul 1, 2022, 1:54 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:

>  https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization

This proposal has been withdrawn by the owners.

[Neal Gompa]

On Sat, Jul 2, 2022 at 12:29 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  On Fri, Jul 1, 2022, 1:54 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
> >
> > https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization
>
>
>  This proposal has been withdrawn by the owners. 
All my hopes and dreams dashed with this statement. :(


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Ralf Corsépius]

Am 02.07.22 um 18:27 schrieb Ben Cotton:
>  
>  
>  On Fri, Jul 1, 2022, 1:54 PM Ben Cotton <bcotton(a)redhat.com 
>  <mailto:[e-mail]>> wrote:
>  
>      https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization
>      <https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization>
>  
>  
>  This proposal has been withdrawn by the owners. 
Any explanation why?

Ralf

[Vitaly Zaitsev]

On 02/07/2022 18:27, Ben Cotton wrote:
>  This proposal has been withdrawn by the owners. 
Why? Very useful feature.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Peter Robinson]

On Sat, Jul 2, 2022 at 8:36 PM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  On 02/07/2022 18:27, Ben Cotton wrote:
>  > This proposal has been withdrawn by the owners.
>
>  Why? Very useful feature. 
Because they didn't coordinate with the maintainers, it wasn't ready
and overall has flaws that need to be addressed. Something to a
similar effect will be coordinated and submitted when it's actually
viable.