Request For Insights On Kernel Security Hardening Practices In Fedora

[Yinhao Hu]

Dear All,

We are academic researchers from Huazhong University of Science and
Technology, China. To foster a healthier Linux kernel community and
enhance the overall security of Linux distributions, we are conducting a
study on kernel security hardening deployments across various Linux
distributions.

In our research, we analyzed kernel config files and the /proc
filesystem by installing and running multiple distribution ISO images.
This allowed us to enumerate the default deployment of kernel defense
mechanisms at runtime.

So far, we have cataloged over 50 kernel security hardening features and
documented inconsistencies in their deployment across different
distributions. The results of our analysis are accessible via the
following link:
https://docs.google.com/spreadsheets/d/17QRr04pqK1K4-VoHXW2-9KgPd4uV8Q4I-NNk....

Given Fedora’s reputation for exceptional performance and rich features,
we conducted a detailed investigation into its kernel security hardening
strategies. To further deepen our understanding, we would greatly
appreciate your input on the following questions:

1. Effectiveness of Kernel Security Hardening

1.1 Do you consider deploying kernel security hardening features to
be an effective strategy for ensuring operating system security?

2. Configuration Strategy for Default Kernel Security Hardening Options

2.1 What are the primary criteria for selecting kernel security
hardening options in your distribution?

2.2 How are configurable security hardening features (e.g.,
unprivileged_bpf_disabled) typically set (e.g., 0, 1, or 2), and what
are the main considerations involved?

2.3 How do you balance the trade-off between side-effects (e.g.,
performance overhead) and the enhanced security introduced by kernel
security hardening?

2.4 Does the tolerance for performance overhead vary across
different application scenarios?

2.5 Are there other negative factors, such as compatibility issues,
that are considered when enabling security hardening features?

3. Customized Configurations

3.1 Do you provide different kernel security hardening
configurations tailored to specific user groups?

4. Best Practices and Recommendations

4.1 Are there any best practices or recommendations you can share
regarding kernel security hardening?

4.2 Are there relevant documents or materials available for reference?

The purpose of these questions is to gain a deeper understanding of your
security protection strategies. Your insights would be immensely
valuable to our study.

Thank you for taking the time to review our questions. We look forward
to your response.

Best regards,
Yinhao Hu, PhD candidate
[e-mail]
Huazhong University of Science and Technology

[Jeremy Linton]

Hi,

I think its worthwhile to note that the spreadsheet is AFAIK out of date because I checked for example: CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_RANDOM_KMALLOC_CACHES running an updated F40 and both are enabled.

So, I'm guessing no one ran 'dnf update' before recording the results, or they are 6+ months old and should probably be updated. That can be done either with 'dnf update' or by installing/upgrading to F41.

And speaking as a community member I think 1.1 is pretty easy to answer "yes".

Along the same vein I think reading the discussions around frame pointers over the past year can provide an idea about how the fedora community tries to find balance when determining performance/features/security/etc tradeoffs.

________________________________
From: Yinhao Hu 
Sent: Friday, December 13, 2024 12:11 AM
To: [e-mail] 
Cc: [e-mail] ; [e-mail] 
Subject: Request For Insights On Kernel Security Hardening Practices In Fedora

Dear All,

We are academic researchers from Huazhong University of Science and
Technology, China. To foster a healthier Linux kernel community and
enhance the overall security of Linux distributions, we are conducting a
study on kernel security hardening deployments across various Linux
distributions.

In our research, we analyzed kernel config files and the /proc
filesystem by installing and running multiple distribution ISO images.
This allowed us to enumerate the default deployment of kernel defense
mechanisms at runtime.

So far, we have cataloged over 50 kernel security hardening features and
documented inconsistencies in their deployment across different
distributions. The results of our analysis are accessible via the
following link:
https://docs.google.com/spreadsheets/d/17QRr04pqK1K4-VoHXW2-9KgPd4uV8Q4I-NNk....

Given Fedora’s reputation for exceptional performance and rich features,
we conducted a detailed investigation into its kernel security hardening
strategies. To further deepen our understanding, we would greatly
appreciate your input on the following questions:

1. Effectiveness of Kernel Security Hardening

1.1 Do you consider deploying kernel security hardening features to
be an effective strategy for ensuring operating system security?

2. Configuration Strategy for Default Kernel Security Hardening Options

2.1 What are the primary criteria for selecting kernel security
hardening options in your distribution?

2.2 How are configurable security hardening features (e.g.,
unprivileged_bpf_disabled) typically set (e.g., 0, 1, or 2), and what
are the main considerations involved?

2.3 How do you balance the trade-off between side-effects (e.g.,
performance overhead) and the enhanced security introduced by kernel
security hardening?

2.4 Does the tolerance for performance overhead vary across
different application scenarios?

2.5 Are there other negative factors, such as compatibility issues,
that are considered when enabling security hardening features?

3. Customized Configurations

3.1 Do you provide different kernel security hardening
configurations tailored to specific user groups?

4. Best Practices and Recommendations

4.1 Are there any best practices or recommendations you can share
regarding kernel security hardening?

4.2 Are there relevant documents or materials available for reference?

The purpose of these questions is to gain a deeper understanding of your
security protection strategies. Your insights would be immensely
valuable to our study.

Thank you for taking the time to review our questions. We look forward
to your response.

Best regards,
Yinhao Hu, PhD candidate
[e-mail]
Huazhong University of Science and Technology
--
_______________________________________________
kernel mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[Yinhao Hu]

Hi,

Thank you very much for your detailed response. We sincerely apologize
for the confusion; the data we presented was based on the results from
June 2024, with the specific kernel version indicated at the time. There
is indeed a discrepancy with the latest data. Our intention is not to
highlight or downplay any particular distribution, but rather to
illustrate, to some extent, the impact of kernel security hardening in
real-world scenarios.

The CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_RANDOM_KMALLOC_CACHES
were merged upstream through commit IDs
6471384af2a6530696fc0203bafe4de41a23c9ef and
3c6152940584290668b35fa0800026f6a1ae05fe, respectively. These changes
were introduced at least 11 months prior to June of this year.

Interestingly, the Fedora 40 desktop kernel version
(6.8.5-301.fc40.x86_64) has not enabled these security hardening, even
though they are included in the latest Fedora version. Could you help
clarify the reasoning behind the decision to enable these two security
hardenings?

Additionally, there seems to be a delay between the upstream
introduction of these security hardenings and their actual
deployment—what factors might be contributing to this delay?

In addition, we also investigated historical versions of Fedora, with
relevant data available at the following link:
https://docs.google.com/spreadsheets/d/1Q2im5w6wwJmzF6TD1OrXMKA3erRpAN-BWVeX...

From the results, it appears that the security hardening deployment for
the desktop and server editions are almost identical, except for
unprivileged_userfaultfd. Can we conclude that Fedora does not take
different application scenarios into account when enabling its security
hardenings?

On 2024/12/14 3:13, Jeremy Linton wrote:
> Hi,
>
> I think its worthwhile to note that the spreadsheet is AFAIK out of date
> because I checked for example: CONFIG_INIT_ON_ALLOC_DEFAULT_ON and
> CONFIG_RANDOM_KMALLOC_CACHES running an updated F40 and both are enabled.
>
> So, I'm guessing no one ran 'dnf update' before recording the results,
> or they are 6+ months old and should probably be updated. That can be
> done either with 'dnf update' or by installing/upgrading to F41.
>
> And speaking as a community member I think 1.1 is pretty easy to answer
> "yes".
>
> Along the same vein I think reading the discussions around frame
> pointers over the past year can provide an idea about how the fedora
> community tries to find balance when determining performance/features/
> security/etc tradeoffs.

[Jeremy Linton]

Hi,

On 12/13/24 10:40 PM, Yinhao Hu wrote:
> Hi,
>
> Thank you very much for your detailed response. We sincerely apologize
> for the confusion; the data we presented was based on the results from
> June 2024, with the specific kernel version indicated at the time. There
> is indeed a discrepancy with the latest data. Our intention is not to
> highlight or downplay any particular distribution, but rather to
> illustrate, to some extent, the impact of kernel security hardening in
> real-world scenarios.
>
> The CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_RANDOM_KMALLOC_CACHES
> were merged upstream through commit IDs
> 6471384af2a6530696fc0203bafe4de41a23c9ef and
> 3c6152940584290668b35fa0800026f6a1ae05fe, respectively. These changes
> were introduced at least 11 months prior to June of this year.
>
> Interestingly, the Fedora 40 desktop kernel version
> (6.8.5-301.fc40.x86_64) has not enabled these security hardening, even
> though they are included in the latest Fedora version. Could you help
> clarify the reasoning behind the decision to enable these two security
> hardenings?

That is the part I'm confused about: You say this is not enabled in F40 
despite them being enabled in F41/latest. Which is not true, they _ARE_ 
currently enabled in F40 as well.
> cat /etc/os-release |grep PRETTY

PRETTY_NAME="Fedora Linux 40 (Workstation Edition)"
> grep ALLOC_DEFAULT_ON /boot/config-`uname -r`

CONFIG_INIT_ON_ALLOC_DEFAULT_ON=y
> grep KMALLOC /boot/config-`uname -r`

CONFIG_RANDOM_KMALLOC_CACHES=y

And have been since April and May.

The changes to enable those two features were done in the fedora kernel 
6.9 timeframe. So, by June when the supported kernel was 6.9 you should 
have seen them enabled. But as I mentioned in the original comment I 
think someone forgot to apply security/etc updates before running your 
tests. A release image with the stale kernel was installed, and the 
update query was ignored.
> Additionally, there seems to be a delay between the upstream
> introduction of these security hardenings and their actual
> deployment—what factors might be contributing to this delay?

Well in this case it looks like the configuration needed to be manually 
enabled.

Fedora tends to follow upstream kernel defaults unless someone manually 
reviews/suggests a change. Which to this day upstream defconfigs don't 
appear to be enabling some of this stuff without also explicitly 
manually enabling the hardening configs. So, it took a fedora maintainer 
reviewing and testing the upstream config changes and overriding them 
because when the feature first appeared it was default disabled. 
Similarly with the generic harding configs.

So if your looking for a 'reason' for a delay in this case, its because 
when upstream introduced the feature(s) they didn't think it was 
production ready otherwise they would have defaulted it on. For large 
features sometimes that is due to perf/ABI/etc tradeoffs and making 
large changes to enable features (ex shadow stacks, or BTI) requires 
distro maintainers to consider whether the feature works with all 
10-100k packages the distro builds/supports. Hence the need for a manual 
override.

Particularly cross distro, so many of these things are dictated by when 
something is released. Comparing a 5 year old debian vs a bleeding edge 
fedora is obviously going to show differences, and that is still there 
comparing the latest debian/suse/etc with the latest fedora/leap/etc 
simply because of the release date. A couple weeks could be a new 
compiler, kernel, glibc, gnome or any number of other things. Distro 
maintainers tend to cooperate cross distro, so when a feature appears 
simply sorting by distro release date will frequently show nothing more 
than a delta between when the feature was created and when it worked 
itself into the latest release.
> In addition, we also investigated historical versions of Fedora, with
> relevant data available at the following link:
> https://docs.google.com/spreadsheets/d/1Q2im5w6wwJmzF6TD1OrXMKA3erRpAN-BWVeX...
>
> From the results, it appears that the security hardening deployment for
> the desktop and server editions are almost identical, except for
> unprivileged_userfaultfd. Can we conclude that Fedora does not take
> different application scenarios into account when enabling its security
> hardenings?

Fedora ships a single kernel image across all of the released revisions, 
that image changes during the lifecycle of a release.

Again, that image is updated during the product lifecycle, which means 
configurations may also change as they did between the original F40 
release image and later kernel updates. There are potentially 
differences in rawhide, and of course the debug kernels have differing 
options. But, the expectation with fedora is that in order to be in a 
supported configuration all the latest updates have been applied.

So, you should _not_ be seeing kernel differences between "Workstation" 
and "Server" installs of the same vintage unless their updates aren't 
applied when a given test is run, and your inadvertently also comparing 
different kernels.
> On 2024/12/14 3:13, Jeremy Linton wrote:
> > Hi,
> >
> > I think its worthwhile to note that the spreadsheet is AFAIK out of date
> > because I checked for example: CONFIG_INIT_ON_ALLOC_DEFAULT_ON and
> > CONFIG_RANDOM_KMALLOC_CACHES running an updated F40 and both are enabled.
> >
> > So, I'm guessing no one ran 'dnf update' before recording the results,
> > or they are 6+ months old and should probably be updated. That can be
> > done either with 'dnf update' or by installing/upgrading to F41.
> >
> > And speaking as a community member I think 1.1 is pretty easy to answer
> > "yes".
> >
> > Along the same vein I think reading the discussions around frame
> > pointers over the past year can provide an idea about how the fedora
> > community tries to find balance when determining performance/features/
> > security/etc tradeoffs.

[Yinhao Hu]

Hi,

It is true that after installing the officially released ISO image, we
directly collected the kernel configurations without performing a system
update. In the case of Fedora, where system components are frequently
updated within a couple of weeks while the official ISO images remain
unchanged, this was indeed an oversight in our experiment. In the future
investigation, we will make sure to account for system updates. We
really appreciate your patient response!

On 2024/12/18 07:08, Jeremy Linton wrote:
> > Thank you very much for your detailed response. We sincerely apologize
> > for the confusion; the data we presented was based on the results from
> > June 2024, with the specific kernel version indicated at the time. There
> > is indeed a discrepancy with the latest data. Our intention is not to
> > highlight or downplay any particular distribution, but rather to
> > illustrate, to some extent, the impact of kernel security hardening in
> > real-world scenarios.
> >
> > The CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_RANDOM_KMALLOC_CACHES
> > were merged upstream through commit IDs
> > 6471384af2a6530696fc0203bafe4de41a23c9ef and
> > 3c6152940584290668b35fa0800026f6a1ae05fe, respectively. These changes
> > were introduced at least 11 months prior to June of this year.
> >
> > Interestingly, the Fedora 40 desktop kernel version
> > (6.8.5-301.fc40.x86_64) has not enabled these security hardening, even
> > though they are included in the latest Fedora version. Could you help
> > clarify the reasoning behind the decision to enable these two security
> > hardenings?
>
> That is the part I'm confused about: You say this is not enabled in
> F40 despite them being enabled in F41/latest. Which is not true, they
> _ARE_ currently enabled in F40 as well.
> > cat /etc/os-release |grep PRETTY
>
> PRETTY_NAME="Fedora Linux 40 (Workstation Edition)"
> > grep ALLOC_DEFAULT_ON /boot/config-`uname -r`
>
> CONFIG_INIT_ON_ALLOC_DEFAULT_ON=y
> > grep KMALLOC /boot/config-`uname -r`
>
> CONFIG_RANDOM_KMALLOC_CACHES=y
>
> And have been since April and May.
>
> The changes to enable those two features were done in the fedora
> kernel 6.9 timeframe. So, by June when the supported kernel was 6.9
> you should have seen them enabled. But as I mentioned in the original
> comment I think someone forgot to apply security/etc updates before
> running your tests. A release image with the stale kernel was
> installed, and the update query was ignored.

Thank you for the explanation. I understand the rationale behind
Fedora's decision to enable security hardening. It makes sense to follow
the upstream kernel's default configuration. When a security hardening
is not enabled by default in the upstream kernel, I appreciate that
Fedora maintainers carefully consider the product's features,
trade-offs, package compatibility, and so on, rather than making
arbitrary decisions about whether to enable it. This approach is highly
commendable.
> Fedora tends to follow upstream kernel defaults unless someone
> manually reviews/suggests a change. Which to this day upstream
> defconfigs don't appear to be enabling some of this stuff without also
> explicitly manually enabling the hardening configs. So, it took a
> fedora maintainer reviewing and testing the upstream config changes
> and overriding them because when the feature first appeared it was
> default disabled. Similarly with the generic harding configs.
>
> So if your looking for a 'reason' for a delay in this case, its
> because when upstream introduced the feature(s) they didn't think it
> was production ready otherwise they would have defaulted it on. For
> large features sometimes that is due to perf/ABI/etc tradeoffs and
> making large changes to enable features (ex shadow stacks, or BTI)
> requires distro maintainers to consider whether the feature works with
> all 10-100k packages the distro builds/supports. Hence the need for a
> manual override.

It's a great point. We will definitely take this into consideration when
comparing different distributions (debian/fedora/suse etc) in the
future. Thanks again!
> Particularly cross distro, so many of these things are dictated by
> when something is released. Comparing a 5 year old debian vs a
> bleeding edge fedora is obviously going to show differences, and that
> is still there comparing the latest debian/suse/etc with the latest
> fedora/leap/etc simply because of the release date. A couple weeks
> could be a new compiler, kernel, glibc, gnome or any number of other
> things. Distro maintainers tend to cooperate cross distro, so when a
> feature appears simply sorting by distro release date will frequently
> show nothing more than a delta between when the feature was created
> and when it worked itself into the latest release.

I understand Fedora's strategy of using a shared kernel image across all
of the released revisions and fully agree with it, as it aligns with the
research findings in the link we provided.

If I were to nitpick, the behavior of |`unprivileged_userfaultfd|` left
me a bit puzzled. After applying an _update_ and rebooting both Fedora
36 "Workstation" and "Server", |`unprivileged_userfaultfd|` exhibited
inconsistent behavior. Below are my test results:
> cat /etc/os-release | grep PRETTY PRETTY_NAME="FedoraLinux 36

(WorkstationEdition)" > uname -r 6.2.15-100.fc36.x86_64 > sudocat
/proc/sys/vm/unprivileged_userfaultfd 1
> cat /etc/os-release | grep PRETTY PRETTY_NAME="Fedora Linux 36 (Server Edition)" > uname -r

6.2.15-100.fc36.x86_64 > sudo cat /proc/sys/vm/unprivileged_userfaultfd 0

Although I understand that Fedora 36 is no longer actively maintained,
there should be no kernel differences. Therefore, I’m uncertain whether
the result above is due to an unintentional misconfiguration or an issue
on my end. Apologies for bothering you again.
> > In addition, we also investigated historical versions of Fedora, with
> > relevant data available at the following link:
> > https://docs.google.com/spreadsheets/d/1Q2im5w6wwJmzF6TD1OrXMKA3erRpAN-BWVeX...
> >
> > From the results, it appears that the security hardening deployment for
> > the desktop and server editions are almost identical, except for
> > unprivileged_userfaultfd. Can we conclude that Fedora does not take
> > different application scenarios into account when enabling its security
> > hardenings?
>
> Fedora ships a single kernel image across all of the released
> revisions, that image changes during the lifecycle of a release.
>
> Again, that image is updated during the product lifecycle, which means
> configurations may also change as they did between the original F40
> release image and later kernel updates. There are potentially
> differences in rawhide, and of course the debug kernels have differing
> options. But, the expectation with fedora is that in order to be in a
> supported configuration all the latest updates have been applied.
>
> So, you should _not_ be seeing kernel differences between
> "Workstation" and "Server" installs of the same vintage unless their
> updates aren't applied when a given test is run, and your
> inadvertently also comparing different kernels.