pre-change: lower printk setting after switching to real root

[Colin Walters]

We recently did https://github.com/coreos/fedora-coreos-config/pull/1840 for Fedora CoreOS
(more background: https://github.com/coreos/fedora-coreos-tracker/issues/1244 ) and
I'd like to consider applying this to all Fedora editions.

There'd be no impact on desktop systems (commonly installed via Anaconda and hence
using `quiet`).  

The benefit is for server systems where we *do* want some kernel output at boot, but once
we've successfully booted we don't want to emit a message every time podman/docker
creates a bridge device for example.

Concretely today, I noticed that the RHEL 8.6 Cloud Guest image also does not include
`quiet` and so the kernel console log is full of the same spam at runtime, and I think it
makes sense to do this change across all Fedora derivatives.

[Kevin Fenzi]

On Fri, Jul 15, 2022 at 10:03:48AM -0400, Colin Walters wrote:
>  We recently did
> https://github.com/coreos/fedora-coreos-config/pull/1840 for Fedora CoreOS (more
> background: https://github.com/coreos/fedora-coreos-tracker/issues/1244 ) and I'd like
> to consider applying this to all Fedora editions.
>  
>  There'd be no impact on desktop systems (commonly installed via Anaconda and hence
> using `quiet`).  
>  
>  The benefit is for server systems where we *do* want some kernel output at boot, but once
> we've successfully booted we don't want to emit a message every time podman/docker
> creates a bridge device for example.
>  
>  Concretely today, I noticed that the RHEL 8.6 Cloud Guest image also does not include
> `quiet` and so the kernel console log is full of the same spam at runtime, and I think it
> makes sense to do this change across all Fedora derivatives. 
It sounds like a reasonable idea. I assume someone could override it
when/if they needed/wanted the more verbose messages?

I'd say write up a change on it. 

kevin

[Lennart Poettering]

On Fr, 15.07.22 10:03, Colin Walters (walters(a)verbum.org) wrote:

>  We recently did
>  https://github.com/coreos/fedora-coreos-config/pull/1840 for Fedora
>  CoreOS (more background:
>  https://github.com/coreos/fedora-coreos-tracker/issues/1244 ) and
>  I'd like to consider applying this to all Fedora editions.
>
>  There'd be no impact on desktop systems (commonly installed via
>  Anaconda and hence using `quiet`).
>
>  The benefit is for server systems where we *do* want some kernel
>  output at boot, but once we've successfully booted we don't want to
>  emit a message every time podman/docker creates a bridge device for
>  example.
>
>  Concretely today, I noticed that the RHEL 8.6 Cloud Guest image also
>  does not include `quiet` and so the kernel console log is full of
>  the same spam at runtime, and I think it makes sense to do this
>  change across all Fedora derivatives. 
I am note entirely sure if this feature has merit or not, but I don't
want to comment on that. However, I'd like to suggest that if you do
this, please consider replacing this line:

<snip>
ExecStart=/bin/bash -euo pipefail -c 'mkdir -p /run/sysctl.d && chcon
--reference=/etc/sysctl.d /run/sysctl.d && echo "kernel.printk = 4" >
/run/sysctl.d/01-coreos-printk.conf'
</snip>

by something like this:

<snip>
ExecStart=/usr/bin/systemd-tmpfiles --create -
StandardInputText=f /run/sysctl.d/01-coreos-printk.conf - - - - kernel.printk 4
</snip>

Benefits: no shell, single process forked, no explicit selinux stuff,
or explicit mkdir, and other MACs will be honoured too if they exist.

Lennart

--
Lennart Poettering, Berlin

[Colin Walters]

On Tue, Jul 19, 2022, at 12:24 PM, Lennart Poettering wrote:
>  by something like this:
>
>  <snip>
>  ExecStart=/usr/bin/systemd-tmpfiles --create -
>  StandardInputText=f /run/sysctl.d/01-coreos-printk.conf - - - - kernel.printk 4
>  </snip>
>
>  Benefits: no shell, single process forked, no explicit selinux stuff,
>  or explicit mkdir, and other MACs will be honoured too if they exist. 
Unfortunately doesn't work today since:
[  243.300955] audit: type=1400 audit(1658251774.506:317): avc:  denied  { getattr } for 
pid=1801 comm="systemd-sysctl"
path="/run/sysctl.d/01-coreos-printk.conf" dev="tmpfs" ino=934
scontext=system_u:system_r:systemd_sysctl_t:s0 tcontext=system_u:object_r:var_run_t:s0
tclass=file permissive=1

But yes, I will look at getting that added to policy.

(FTR there was also a missing `=` in the sysctl text)

[Colin Walters]

On Tue, Jul 19, 2022, at 12:24 PM, Lennart Poettering wrote:
>  On Fr, 15.07.22 10:03, Colin Walters (walters(a)verbum.org) wrote:
>
> > We recently did
> > https://github.com/coreos/fedora-coreos-config/pull/1840 for Fedora
> > CoreOS (more background:
> > https://github.com/coreos/fedora-coreos-tracker/issues/1244 ) and
> > I'd like to consider applying this to all Fedora editions.
> >
> > There'd be no impact on desktop systems (commonly installed via
> > Anaconda and hence using `quiet`).
> >
> > The benefit is for server systems where we *do* want some kernel
> > output at boot, but once we've successfully booted we don't want to
> > emit a message every time podman/docker creates a bridge device for
> > example.
> >
> > Concretely today, I noticed that the RHEL 8.6 Cloud Guest image also
> > does not include `quiet` and so the kernel console log is full of
> > the same spam at runtime, and I think it makes sense to do this
> > change across all Fedora derivatives.
>
>  I am note entirely sure if this feature has merit or not, 
Definitely interested in your opinion, because...a bikeshed here is where to put this unit
if it's not systemd.

For CoreOS, we have this nice "overlay" git repository for stuff like this that
isn't an RPM, it has a lot of our miscellaneous systemd units and config files and
such; so we can just do a pull request, have that pull request go through CI and merge and
then it gets baked into the image and ship...no "manual package builds".  The
closest analogue in the yum world (i.e. only understands RPMs) is probably the
generic-release type packages.  So in theory it could go there.

But this all said...it perhaps is worth considering the alternative, which is just this
one-liner diff to the kernel config (AFAIK):

diff --git a/kernel-x86_64-fedora.config b/kernel-x86_64-fedora.config
index 517763fc9..b4b5708a3 100644
--- a/kernel-x86_64-fedora.config
+++ b/kernel-x86_64-fedora.config
@@ -981,7 +981,7 @@ CONFIG_COMPAT_32BIT_TIME=y
 # CONFIG_COMPILE_TEST is not set
 CONFIG_CONFIGFS_FS=y
 CONFIG_CONNECTOR=y
-CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
+CONFIG_CONSOLE_LOGLEVEL_DEFAULT=4
 CONFIG_CONSOLE_LOGLEVEL_QUIET=3
 CONFIG_CONTEXT_SWITCH_TRACER=y
 # CONFIG_CONTEXT_TRACKING_FORCE is not set


The implications to that are obviously much larger, which is why I hesitated to propose
it.  While the stream of debug-level spew for servers has caused serious problems, it
feels odd to me to switch servers to be entirely quiet by default.  I am *certain* there
are people who are doing CI/testing systems and are gathering the kernel console today and
expect non-quiet output by default.

But, this is also a much simpler change to understand, and anyone who wants it can start
specifying e.g. `debug` on the kernel command line.

I'm certainly curious about the opinions of the kernel maintainers in particular
here.

[Lennart Poettering]

On Do, 21.07.22 05:56, Colin Walters (walters(a)verbum.org) wrote:

>  >> We recently did
>  >> https://github.com/coreos/fedora-coreos-config/pull/1840 for Fedora
>  >> CoreOS (more background:
>  >> https://github.com/coreos/fedora-coreos-tracker/issues/1244 ) and
>  >> I'd like to consider applying this to all Fedora editions.
>  >>
>  >> There'd be no impact on desktop systems (commonly installed via
>  >> Anaconda and hence using `quiet`).
>  >>
>  >> The benefit is for server systems where we *do* want some kernel
>  >> output at boot, but once we've successfully booted we don't want to
>  >> emit a message every time podman/docker creates a bridge device for
>  >> example.
>  >>
>  >> Concretely today, I noticed that the RHEL 8.6 Cloud Guest image also
>  >> does not include `quiet` and so the kernel console log is full of
>  >> the same spam at runtime, and I think it makes sense to do this
>  >> change across all Fedora derivatives.
>  >
>  > I am note entirely sure if this feature has merit or not,
>
>  Definitely interested in your opinion, because...a bikeshed here is
>  where to put this unit if it's not systemd. 
The thing is that serial console output from the kernel is really slow
and slows down the kernel (because it's synchronous, in some way). And
this slowness is not just minor issue you can ignore, it's actually
massive, if you do that across your fleet.

OTOH I am also not sure that this kinda of cosmetics really matter on
servers. They are usually not interacted with on their console, except
for debugging. But in that case, wouldn't it be better to just allow
the whole logs to go through?

So there are reasons not to log to the console needlessly, and there
are reasons to always log to the console to improve debugging.

Hence, it doesn't appear to me that it is clear that doing what you
are proposing is universially a good idea. It kinda feels as if in 30%
of uses it might be beneficial, in 30% people would hate it, and in
30% people might not care...

That's why I don't really want to agree or disagree with the proposal,
it's not clear at all to me that what you are doing there is a good
idea across the board. It might be in the coreos case, if you say so,
but universally, i am not sure.

Lennart

--
Lennart Poettering, Berlin