F39 Change Proposal: Clean Systemd-boot installs (Self-Contained_

[Aoife Moloney]

https://fedoraproject.org/wiki/Changes/cleanup_systemd_install

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==
Fedora default installs with a shim + grub bootloader on EFI
platforms, yet has been shipping systemd-boot in various forms for a
number of releases. There are a few howto's which describe how to
replace grub with systemd-boot with varying levels of functionality.
This should be easier with a formalized default method that can be
built upon. This proposal aims to complete the work started with
anaconda (inst.sdboot), kickstart (bootloader --sdboot) such that the
"everything" media can install a grub free machine.

== Owner ==
* Name: [[User:jlinton| Jeremy Linton]]

* Email: <jeremy.linton(a)arm.com&gt;

* Name: Possibly others since it may touch -comps, systemd-boot, etc


== Detailed Description ==
As a first pass, the 'inst.sdboot' option already in anaconda should
work. As it stands, that replaces grub+shim with the systemd-boot
loader, and moves the kernel + initrd to the EFI system partition
(ESP). It doesn't attempt to create unified kernel images, so the
existing `dnf update`, `kdumpctl`, and `make install` in a kernel
source directory should all work. The vast majority of this work has
been done, leaving only two action items, removing grubby from core,
and merging a shimming package (sdubby) into the fedora repos.

Beyond that there are various enhancements which can be made to remove
the /boot partition (leaving the EFI at /boot/efi), enrolling fedora
keys if the secure boot mode is "Setup", adding options to enable
shim+systemd-boot, assuring that there is a systemd-boot-signed
package, etc.

The advantages of just enabling the systemd-boot loader without UKIs
or restructuring the /boot and /boot/efi mount points result in a
wider range of supported machines and a more familiar environment for
users and applications. AKA, by not changing the HostOnly/initrd build
process the vast majority of UEFI machines are supported.

To be clear the intention isn't to replace grub, but to co-exist
alongside as an alternative bootloader.

== Feedback ==


== Benefit to Fedora ==

Fedora is considered a forward looking distro. As systemd-boot and
UKIs gain traction it should be straightforward for users/testers to
try out this option in their own environments with a well defined
configuration.

Potentially in the future, once secure boot/etc is straightened out
the simpler/cleaner code base may prove to be more secure, or a
consistent set of measured boot PCRs may enable a simpler (for the end
user) encrypted storage environment.

== Scope ==
* Proposal owners:

At the moment two things remain open:

https://pagure.io/fedora-comps/pull-request/838

and:

https://bugzilla.redhat.com/show_bug.cgi?id=2134972

Both of which are largely in the "needs more discussion" state, but
otherwise are complete as they stand.

There is also an open kexec-tools + aarch64 zboot set that needs to be
merged in order to support kdump properly on aarch64 platforms,
although that problem is caused by zboot and affects grub as well.
Zboot is required for systemd-boot at the moment.

* Other developers:


Depending on the results of the discussion above: Its possible the
systemd maintainers, kdumpctl, etc may need changes.

* Release engineering: [https://pagure.io/releng/issues #Releng issue number]

* Policies and guidelines: N/A (not needed for this Change)

* Trademark approval: N/A (not needed for this Change)

* Alignment with Community Initiatives:


== Upgrade/compatibility impact ==

Ideally nothing as we aren't deprecating or changing the shim + grub boot paths.



== How To Test ==

# Have a VM or non critical test machine that can be reinstalled at will.
# Assure secure boot is disabled or in setup mode.
# Pass `inst.sdboot` on the kernel/grub command line presented on the
install media and install as normal.
## possibly adding additional space to the EFI system partition during
partitioning to guarantee there is sufficient space for the number of
bootable kernels active on the machine (~100MB each should be more
than sufficient)
## Alternatively `--sdboot` can be added to the bootloader command in
kickstarts, and the partitions/etc adjusted there
# Use the machine as normal.
# Report issues during upgrades, or with any packages that can't find
kernel images. Everything besides the loader entries, kernel image,
and generated initrds should remain in /boot.


== User Experience ==

Ideally, after the initial install the fedora experience should
generally remain the same. There may be slight differences in boot
timings (at least on aarch64 possibly slightly faster) and the bootctl
utility may have more information and work properly.


== Dependencies ==

Systemd-boot, described in the comps and sdubby review.




== Contingency Plan ==




== Documentation ==


*https://anaconda-installer.readthedocs.io/en/latest/boot-options.html#inst-sdboot

or

*https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#bootloader

== Release Notes ==



-- 
Aoife Moloney

Product Owner

Community Platform Engineering Team

Red Hat EMEA

Communications House

Cork Road

Waterford

[Daniel P. Berrangé]

On Thu, Jun 22, 2023 at 04:59:38PM +0100, Aoife Moloney wrote:
>  https://fedoraproject.org/wiki/Changes/cleanup_systemd_install
>  
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  == Summary ==
>  Fedora default installs with a shim + grub bootloader on EFI
>  platforms, yet has been shipping systemd-boot in various forms for a
>  number of releases. There are a few howto's which describe how to
>  replace grub with systemd-boot with varying levels of functionality.
>  This should be easier with a formalized default method that can be
>  built upon. This proposal aims to complete the work started with
>  anaconda (inst.sdboot), kickstart (bootloader --sdboot) such that the
>  "everything" media can install a grub free machine. 
snip

>  Beyond that there are various enhancements which can be made to
> remove
>  the /boot partition (leaving the EFI at /boot/efi), enrolling fedora
>  keys if the secure boot mode is "Setup", adding options to enable
>  shim+systemd-boot, assuring that there is a systemd-boot-signed
>  package, etc. 
This is the $million question to - is there any proposal and/or agreement
by relevant maintainers, to start signing systemd-boot with Fedora SecureBoot
certs yet ? Without that, sdboot looks destined to remain a niche use case.

If that's part of this proposal, then it feels more like a system-wide
change, than self-contained.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Jeremy Linton]

Hi,

On 6/22/23 11:21, Daniel P. Berrangé wrote:
>  On Thu, Jun 22, 2023 at 04:59:38PM +0100, Aoife Moloney wrote:
> > https://fedoraproject.org/wiki/Changes/cleanup_systemd_install
> >
> > This document represents a proposed Change. As part of the Changes
> > process, proposals are publicly announced in order to receive
> > community feedback. This proposal will only be implemented if approved
> > by the Fedora Engineering Steering Committee.
> >
> > == Summary ==
> > Fedora default installs with a shim + grub bootloader on EFI
> > platforms, yet has been shipping systemd-boot in various forms for a
> > number of releases. There are a few howto's which describe how to
> > replace grub with systemd-boot with varying levels of functionality.
> > This should be easier with a formalized default method that can be
> > built upon. This proposal aims to complete the work started with
> > anaconda (inst.sdboot), kickstart (bootloader --sdboot) such that the
> > "everything" media can install a grub free machine.
>  
>  snip
>  
> > Beyond that there are various enhancements which can be made to remove
> > the /boot partition (leaving the EFI at /boot/efi), enrolling fedora
> > keys if the secure boot mode is "Setup", adding options to enable
> > shim+systemd-boot, assuring that there is a systemd-boot-signed
> > package, etc.
>  
>  This is the $million question to - is there any proposal and/or agreement
>  by relevant maintainers, to start signing systemd-boot with Fedora SecureBoot
>  certs yet ? Without that, sdboot looks destined to remain a niche use case. 
I think there remain a number of open questions, WRT how systemd-boot 
will settle down. In this case, AFAIK there is an open fedora infra work 
item for signing it, per zbyszek, who I've put on CC in case he misses 
this discussion. Of course due to the fact that not every platform has 
UEFI, and there are features in grub which aren't/shouldn't be 
duplicated in systemd-boot means that it will never be capable of 100% 
replacing grub on every platform.


>  
>  If that's part of this proposal, then it feels more like a system-wide
>  change, than self-contained. 
Well given it only affects a couple packages, and is optional, I flipped 
a coin and said its not really system wide, but given its in the boot 
path I do see the argument the other way as well.

But, IMHO the largest change is moving the boot kernel/initrd to the 
ESP, rather than the use of systemd-boot itself. Given the filesystem 
layout remains the same outside of those two items and the BLS entries, 
all the common tools (dracut/etc) seem to be able to deal with it, and 
random other packages not manipulating kernel/initrd images are behavior 
should not be affected anymore than putting reFind/whatever in the boot 
path.


Thanks,

[Lennart Poettering]

On Do, 22.06.23 12:24, Jeremy Linton (jeremy.linton(a)arm.com) wrote:

>  But, IMHO the largest change is moving the boot kernel/initrd to the
> ESP,
>  rather than the use of systemd-boot itself. Given the filesystem layout
>  remains the same outside of those two items and the BLS entries, all the
>  common tools (dracut/etc) seem to be able to deal with it, and random other
>  packages not manipulating kernel/initrd images are behavior should not be
>  affected anymore than putting reFind/whatever in the boot path. 
While putting kernle/initrd (UKIs) into the ESP is a good thing
systemd-boot actually is fine with loading them from a separate boot
partition to, i.e. the XBOOTLDR partition, which in many ways is
similar to traditional /boot/, but must be marked via recognized
XBOOTLDR partition type uuid. (also it must be a firmware recognizable
fs, i.e. vfat)

Such an XBOOTLDR partition is supposed to be the escape hatch if the
ESP is pre-existing and too small to contain multiple full sized
UKIs. But you could also make it an escape hatch for simplifying
transitions from the status quo ante.

Lennart

--
Lennart Poettering, Berlin

[Daniel P. Berrangé]

On Mon, Jun 26, 2023 at 12:19:46PM +0200, Lennart Poettering wrote:
>  On Do, 22.06.23 12:24, Jeremy Linton (jeremy.linton(a)arm.com) wrote:
>  
>  > But, IMHO the largest change is moving the boot kernel/initrd to the ESP,
>  > rather than the use of systemd-boot itself. Given the filesystem layout
>  > remains the same outside of those two items and the BLS entries, all the
>  > common tools (dracut/etc) seem to be able to deal with it, and random other
>  > packages not manipulating kernel/initrd images are behavior should not be
>  > affected anymore than putting reFind/whatever in the boot path.
>  
>  While putting kernle/initrd (UKIs) into the ESP is a good thing
>  systemd-boot actually is fine with loading them from a separate boot
>  partition to, i.e. the XBOOTLDR partition, which in many ways is
>  similar to traditional /boot/, but must be marked via recognized
>  XBOOTLDR partition type uuid. (also it must be a firmware recognizable
>  fs, i.e. vfat) 
NB, Fedora doesn't currently have a supported way to making /boot
be compliant with XBOOTLDR requirements, as anaconda blocks use of
vfat on /boot:

   https://bugzilla.redhat.com/show_bug.cgi?id=2106706

>  Such an XBOOTLDR partition is supposed to be the escape hatch if the
>  ESP is pre-existing and too small to contain multiple full sized
>  UKIs. But you could also make it an escape hatch for simplifying
>  transitions from the status quo ante. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Richard W.M. Jones]

On Thu, Jun 22, 2023 at 12:24:04PM -0500, Jeremy Linton wrote:
>  But, IMHO the largest change is moving the boot kernel/initrd to the
>  ESP, rather than the use of systemd-boot itself. Given the
>  filesystem layout remains the same outside of those two items and
>  the BLS entries, all the common tools (dracut/etc) seem to be able
>  to deal with it, and random other packages not manipulating
>  kernel/initrd images are behavior should not be affected anymore
>  than putting reFind/whatever in the boot path. 
This change is likely to affect supermin, libguestfs, guestfs-tools &
virt-v2v, some BZs would be appreciated.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Jeremy Linton]

Hi,

Thanks for looking at this, and sorry about the delay I was on PTO for a 
few days.

On 6/28/23 09:15, Richard W.M. Jones wrote:
>  On Thu, Jun 22, 2023 at 12:24:04PM -0500, Jeremy Linton wrote:
> > But, IMHO the largest change is moving the boot kernel/initrd to the
> > ESP, rather than the use of systemd-boot itself. Given the
> > filesystem layout remains the same outside of those two items and
> > the BLS entries, all the common tools (dracut/etc) seem to be able
> > to deal with it, and random other packages not manipulating
> > kernel/initrd images are behavior should not be affected anymore
> > than putting reFind/whatever in the boot path.
>  
>  This change is likely to affect supermin, libguestfs, guestfs-tools &
>  virt-v2v, some BZs would be appreciated. 
I have tested some of these, and opened:

supermin: Appears to have problem with the PE + compressed kernel image 
(although likely more a qemu issue) otherwise basic functionality works?

https://bugzilla.redhat.com/show_bug.cgi?id=2222137

libguestfs, largely a  placeholder bug:

https://bugzilla.redhat.com/show_bug.cgi?id=2222139


gestfs-tools, again a placeholder:

https://bugzilla.redhat.com/show_bug.cgi?id=2222140


and finally virt-v2v is confused by systemd-boot based images and cannot 
detect the bootloader.

https://bugzilla.redhat.com/show_bug.cgi?id=2222485


Thanks again,

jlinton

[Zbigniew Jędrzejewski-Szmek]

>  https://fedoraproject.org/wiki/Changes/cleanup_systemd_install
>  == Detailed Description ==
>  As a first pass, the 'inst.sdboot' option already in anaconda should
>  work. As it stands, that replaces grub+shim with the systemd-boot
>  loader, and moves the kernel + initrd to the EFI system partition
>  (ESP). 
In general, I think it's great to see this happening. Thank you!

>  It doesn't attempt to create unified kernel images, so the
>  existing `dnf update`, `kdumpctl`, and `make install` in a kernel
>  source directory should all work. 
Nitpick: 'make install' invokes /usr/bin/installkernel which (sometimes?)
invokes kernel-install which will create a UKI, if the configuration
specifies that. This sentence implies that 'make install' has some
issue with UKIs, but that is not true.

/usr/bin/installkernel does a lot of stuff, most of it probably wrong.
We could 'ln -sf --relative /usr/bin/kernel-install /usr/sbin/installkernel'
and then 'make install' would work without grubby.

>  The vast majority of this work has
>  been done, leaving only two action items, removing grubby from core,
>  and merging a shimming package (sdubby) into the fedora repos. 
As I wrote before in the bug, what is really grubby and sdubby needed for?
Most of what grubby does is arcane archaic stuff that we don't need.
With grubby/sdubby we have an additional layer that adds complexity
and is very inflexible. Why can't we have Anaconda write the Boot Loader
Specification files directly or invoke kernel-install?

Zbyszek

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jun 22, 2023 at 05:28:41PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > It doesn't attempt to create unified kernel images, so the
>  > existing `dnf update`, `kdumpctl`, and `make install` in a kernel
>  > source directory should all work.
>  
>  Nitpick: 'make install' invokes /usr/bin/installkernel which (sometimes?)
>  invokes kernel-install which will create a UKI, if the configuration
>  specifies that. This sentence implies that 'make install' has some
>  issue with UKIs, but that is not true.
>  
>  /usr/bin/installkernel does a lot of stuff, most of it probably wrong.
>  We could 'ln -sf --relative /usr/bin/kernel-install /usr/sbin/installkernel'
>  and then 'make install' would work without grubby. 
https://src.fedoraproject.org/rpms/grubby/pull-request/8

Zbyszek

[Jeremy Linton]

Hi,

On 6/22/23 12:28, Zbigniew Jędrzejewski-Szmek wrote:
> > https://fedoraproject.org/wiki/Changes/cleanup_systemd_install
>  
> > == Detailed Description ==
> > As a first pass, the 'inst.sdboot' option already in anaconda should
> > work. As it stands, that replaces grub+shim with the systemd-boot
> > loader, and moves the kernel + initrd to the EFI system partition
> > (ESP).
>  
>  In general, I think it's great to see this happening. Thank you! 
Well thanks for looking at it :)

>  
> > It doesn't attempt to create unified kernel images, so the
> > existing `dnf update`, `kdumpctl`, and `make install` in a kernel
> > source directory should all work.
>  
>  Nitpick: 'make install' invokes /usr/bin/installkernel which (sometimes?)
>  invokes kernel-install which will create a UKI, if the configuration
>  specifies that. This sentence implies that 'make install' has some
>  issue with UKIs, but that is not true.
>  
>  /usr/bin/installkernel does a lot of stuff, most of it probably wrong.
>  We could 'ln -sf --relative /usr/bin/kernel-install /usr/sbin/installkernel'
>  and then 'make install' would work without grubby. 
Right, which is one of the 1/2 dozen things in the sdubby package, and 
with my limited testing appears to work fine.

>  
> > The vast majority of this work has
> > been done, leaving only two action items, removing grubby from core,
> > and merging a shimming package (sdubby) into the fedora repos.
>  
>  As I wrote before in the bug, what is really grubby and sdubby needed for?
>  Most of what grubby does is arcane archaic stuff that we don't need.
>  With grubby/sdubby we have an additional layer that adds complexity
>  and is very inflexible. Why can't we have Anaconda write the Boot Loader
>  Specification files directly or invoke kernel-install? 
The critical missing piece of grubby has been the BLS entry 
manipulation, usually adding or stripping kernel boot parameters. Which 
at least in fedora, grubby's shell "API" is the defacto way for other 
programs (ex: kdumpctl) to perform this. That is what is being provided 
in the sdubby package, outside of symlinks and packaging stuff. So, your 
right there is a _LOT_ of seemingly redundant stuff in grubby if the 
system fits into the somewhat narrow set of criteria already required 
for systemd-boot. AKA basic UEFI secure boot path without 3rd party 
kernel modules/etc.

Maybe a better name would have been bls-entry-tools, since its largely 
just shimming between bootctl and everything else with sed. It might 
make sense from a fedora perspective if something like bootctl did this 
by default, but at least when it comes to output formats/etc its a bit 
ugly, and I don't really think the grubby "API" as such is being used by 
anyone outside of the fedora derived distros.

So, it makes sense if that functionality exists, that anaconda would use 
it, rather than attempting an install only version. Hence why this isn't 
in anaconda.

Thanks again,
jlinton

[Jeremy Linton]

Hi,

On 6/22/23 12:28, Zbigniew Jędrzejewski-Szmek wrote:
> > https://fedoraproject.org/wiki/Changes/cleanup_systemd_install
>  
> > == Detailed Description ==
> > As a first pass, the 'inst.sdboot' option already in anaconda should
> > work. As it stands, that replaces grub+shim with the systemd-boot
> > loader, and moves the kernel + initrd to the EFI system partition
> > (ESP).
>  
>  In general, I think it's great to see this happening. Thank you!
>  
> > It doesn't attempt to create unified kernel images, so the
> > existing `dnf update`, `kdumpctl`, and `make install` in a kernel
> > source directory should all work.
>  
>  Nitpick: 'make install' invokes /usr/bin/installkernel which (sometimes?)
>  invokes kernel-install which will create a UKI, if the configuration
>  specifies that. This sentence implies that 'make install' has some
>  issue with UKIs, but that is not true. 
As clarification, I added a "." on the wiki version of the proposal so 
that its hopefully clearer that we aren't excluding UKIs from working 
with the usual tools.

Thanks,

>  
>  /usr/bin/installkernel does a lot of stuff, most of it probably wrong.
>  We could 'ln -sf --relative /usr/bin/kernel-install /usr/sbin/installkernel'
>  and then 'make install' would work without grubby.
>  
> > The vast majority of this work has
> > been done, leaving only two action items, removing grubby from core,
> > and merging a shimming package (sdubby) into the fedora repos.
>  
>  As I wrote before in the bug, what is really grubby and sdubby needed for?
>  Most of what grubby does is arcane archaic stuff that we don't need.
>  With grubby/sdubby we have an additional layer that adds complexity
>  and is very inflexible. Why can't we have Anaconda write the Boot Loader
>  Specification files directly or invoke kernel-install?
>  
>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Gerd Hoffmann]

>  == Detailed Description ==
>  As a first pass, the 'inst.sdboot' option already in anaconda should
>  work. As it stands, that replaces grub+shim with the systemd-boot
>  loader, and moves the kernel + initrd to the EFI system partition
>  (ESP). It doesn't attempt to create unified kernel images, so the
>  existing `dnf update`, `kdumpctl`, and `make install` in a kernel
>  source directory should all work. 
What is the plan for existing UKIs, for example kernel-uki-virt.rpm?

I'd expect they should work fine without extra work (i.e. try a
kickstart file with '-kernel' + '-kernel-core' + 'kernel-uki-virt'
in
the file list).

[ Side node: Right now the kernel-uki-virt %postinstall just does a copy
  to /boot/efi/EFI/Linux where systemd-boot should find them just fine.
  After systemd v254 release we which brings some kernel-install
  improvements for UKIs we should be able to switch over to use
  kernel-install instead ].

take care,
  Gerd

[Jeremy Linton]

Hi,

On 6/23/23 05:27, Gerd Hoffmann wrote:
> > == Detailed Description ==
> > As a first pass, the 'inst.sdboot' option already in anaconda should
> > work. As it stands, that replaces grub+shim with the systemd-boot
> > loader, and moves the kernel + initrd to the EFI system partition
> > (ESP). It doesn't attempt to create unified kernel images, so the
> > existing `dnf update`, `kdumpctl`, and `make install` in a kernel
> > source directory should all work.
>  
>  What is the plan for existing UKIs, for example kernel-uki-virt.rpm?
>  
>  I'd expect they should work fine without extra work (i.e. try a
>  kickstart file with '-kernel' + '-kernel-core' +
> 'kernel-uki-virt' in
>  the file list). 
Since no one else said anything, I will comment that I've not tested the 
uki package on top of systemd-boot + normal kernel flow. But the 
knowledge of where to put the kernels (/boot or ESP) is autodetected by 
the kernel-install script sorta automatically, and when grub isn't in 
the picture its happy to put them on the ESP alongside the BLS entries.

Presumably as you note below kernel-install should be able to do this 
for UKI kernels as well, removing the need to move them. Although the 
BLS type is set to "type1" and puts the kernel + initrd in 
/boot/efi/`cat /etc/machine-id`/`uname -r`/. So, setting the 
KERNEL_INSTALL_LAYOUT=uki should override the entries.srel and get them 
in the right place.

So, in theory it should work, but probably needs tweaking here/there.

Something else to test is what happens if both UKIs and traditional 
kernel + initrd exist on the ESP as the docs seem to want one or the 
other tied to "type1"/"type2"


>  
>  [ Side node: Right now the kernel-uki-virt %postinstall just does a copy
>     to /boot/efi/EFI/Linux where systemd-boot should find them just fine.
>     After systemd v254 release we which brings some kernel-install
>     improvements for UKIs we should be able to switch over to use
>     kernel-install instead ].
>  
>  take care,
>     Gerd
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue