3:43 p.m.
I noticed that by default, Qubes OS has voluntary kernel preemption
as opposed to full preemption. I found that enabling full preemption
(preempt=full on kernel command line) makes the system significantly
more responsive under heavy I/O load. In particular, if I build a
kernel in a Qubes OS VM, it significantly degrades responsiveness
without preempt=full. With preempt=full, the system remains
responsive. The storage stack used is LVM thin provisioning on LUKS,
and I have observed significant CPU usage in dom0 kernel threads with
names that indicate they are related to dm-thin and dm-crypt.
The kernel config used by the Qubes kernel package I use (6.1.28) is
based on Fedora 37’s config, and Marek Marczykowski-Górecki (CCd)
indicated that the same arguments apply to Fedora. Therefore, I am
asking if Fedora should use full kernel preemption by default.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
6:14 p.m.
On Sat, May 20, 2023 at 3:44 PM Demi Marie Obenour
<demiobenour(a)gmail.com> wrote:
I noticed that by default, Qubes OS has voluntary kernel preemption
as opposed to full preemption. I found that enabling full preemption
(preempt=full on kernel command line) makes the system significantly
more responsive under heavy I/O load. In particular, if I build a
kernel in a Qubes OS VM, it significantly degrades responsiveness
without preempt=full. With preempt=full, the system remains
responsive. The storage stack used is LVM thin provisioning on LUKS,
and I have observed significant CPU usage in dom0 kernel threads with
names that indicate they are related to dm-thin and dm-crypt.
For workstation, preempt=full does likely make sense, and I have been
running it for a while. For server, maybe not so much. That is the
joy of dynamic preempt. You can boot with preempt=full and run that
way if you like. There is an open issue for the workstation WG to
look at making preempt=full the default there at some point.
The kernel config used by the Qubes kernel package I use (6.1.28) is
based on Fedora 37’s config, and Marek Marczykowski-Górecki (CCd)
indicated that the same arguments apply to Fedora. Therefore, I am
asking if Fedora should use full kernel preemption by default.
I would still say no, as we do have dynamic, and you can set it to
whatever you like.
Justin
> Sincerely,
> Demi Marie Obenour (she/her/hers)_______________________________________________
8:53 a.m.
New subject: Should Fedora switch to full kernel preemption
(CONFIG_PREEMPT=y)?
On Sat, May 20, 2023 at 06:14:17PM -0500, Justin Forbes wrote:
On Sat, May 20, 2023 at 3:44 PM Demi Marie Obenour
<demiobenour(a)gmail.com> wrote:
>
> I noticed that by default, Qubes OS has voluntary kernel preemption
> as opposed to full preemption. I found that enabling full preemption
> (preempt=full on kernel command line) makes the system significantly
> more responsive under heavy I/O load. In particular, if I build a
> kernel in a Qubes OS VM, it significantly degrades responsiveness
> without preempt=full. With preempt=full, the system remains
> responsive. The storage stack used is LVM thin provisioning on LUKS,
> and I have observed significant CPU usage in dom0 kernel threads with
> names that indicate they are related to dm-thin and dm-crypt.
>
For workstation, preempt=full does likely make sense, and I have been
running it for a while. For server, maybe not so much. That is the
joy of dynamic preempt. You can boot with preempt=full and run that
way if you like. There is an open issue for the workstation WG to
look at making preempt=full the default there at some point.
What's the downside from full pre-empt that makes it inappropriate
as the defualt for Fedora server spins too ? Is it that it is
trading off overall peak performance in favour of reduced latency,
and we think servers would prefer the peak performance in general ?
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:03 a.m.
On Thu, Jul 20, 2023, at 9:53 AM, Daniel P. Berrangé wrote:
What's the downside from full pre-empt that makes it
inappropriate
as the defualt for Fedora server spins too ? Is it that it is
trading off overall peak performance in favour of reduced latency,
and we think servers would prefer the peak performance in general ?
Maybe. But I'm not aware of any recent data one way or the other, certainly nothing
Fedora specific.
I don't expect full preemption will impact servers in any meaningful way. I just
can't prove that.
--
Chris Murphy
9:17 a.m.
On Thu, Jul 20, 2023 at 8:53 AM Daniel P. Berrangé <berrange(a)redhat.com> wrote:
On Sat, May 20, 2023 at 06:14:17PM -0500, Justin Forbes wrote:
> On Sat, May 20, 2023 at 3:44 PM Demi Marie Obenour
> <demiobenour(a)gmail.com> wrote:
> >
> > I noticed that by default, Qubes OS has voluntary kernel preemption
> > as opposed to full preemption. I found that enabling full preemption
> > (preempt=full on kernel command line) makes the system significantly
> > more responsive under heavy I/O load. In particular, if I build a
> > kernel in a Qubes OS VM, it significantly degrades responsiveness
> > without preempt=full. With preempt=full, the system remains
> > responsive. The storage stack used is LVM thin provisioning on LUKS,
> > and I have observed significant CPU usage in dom0 kernel threads with
> > names that indicate they are related to dm-thin and dm-crypt.
> >
> For workstation, preempt=full does likely make sense, and I have been
> running it for a while. For server, maybe not so much. That is the
> joy of dynamic preempt. You can boot with preempt=full and run that
> way if you like. There is an open issue for the workstation WG to
> look at making preempt=full the default there at some point.
What's the downside from full pre-empt that makes it inappropriate
as the defualt for Fedora server spins too ? Is it that it is
trading off overall peak performance in favour of reduced latency,
and we think servers would prefer the peak performance in general ?
That is the downside. It might be interesting to ask the performance
group for RHEL to run a battery against preempt=full and see what the
results are. The ELN kernel does enable PREEMPT_DYNAMIC and there
should already be a baseline to compare against there. I just do not
know if they have space cycles to run those tests. No matter what
default we choose, there are no plans to turn off PREEMPT_DYNAMIC
meaning any user is free to choose the preempt mode that works best
for them. I have been running preempt=full for quite some time on my
desktops systems here.
Justin
5:51 p.m.
New subject: Should Fedora switch to full kernel preemption
(CONFIG_PREEMPT=y)?
On Thu, 20 Jul 2023 09:17:51 -0500
Justin Forbes <jmforbes(a)linuxtx.org> wrote:
for them. I have been running preempt=full for quite some time on my
desktops systems here.
I've been running custom compiled kernels with preempt_full=Y on my
desktop for several years (since at least kernel 5.1). Currently
running 6.5 with it enabled.
7:44 a.m.
New subject: Should Fedora switch to full kernel preemption
(CONFIG_PREEMPT=y)?
Dne 20. 05. 23 v 22:43 Demi Marie Obenour napsal(a):
I noticed that by default, Qubes OS has voluntary kernel preemption
as opposed to full preemption. I found that enabling full preemption
(preempt=full on kernel command line) makes the system significantly
more responsive under heavy I/O load. In particular, if I build a
kernel in a Qubes OS VM, it significantly degrades responsiveness
without preempt=full. With preempt=full, the system remains
responsive. The storage stack used is LVM thin provisioning on LUKS,
and I have observed significant CPU usage in dom0 kernel threads with
names that indicate they are related to dm-thin and dm-crypt.
The kernel config used by the Qubes kernel package I use (6.1.28) is
based on Fedora 37’s config, and Marek Marczykowski-Górecki (CCd)
indicated that the same arguments apply to Fedora. Therefore, I am
asking if Fedora should use full kernel preemption by default.
Hi Demi
Could you please provide 'dmsetup table' - so we could see how doe your
device stack looks like ?
Aren't you disabling work-queues on the table line for crypt targets ?
Regards
Zdenek
6:47 p.m.
New subject: Should Fedora switch to full kernel preemption
(CONFIG_PREEMPT=y)?
On 5/24/23 08:44, Zdenek Kabelac wrote:
Dne 20. 05. 23 v 22:43 Demi Marie Obenour napsal(a):
> I noticed that by default, Qubes OS has voluntary kernel preemption
> as opposed to full preemption. I found that enabling full preemption
> (preempt=full on kernel command line) makes the system significantly
> more responsive under heavy I/O load. In particular, if I build a
> kernel in a Qubes OS VM, it significantly degrades responsiveness
> without preempt=full. With preempt=full, the system remains
> responsive. The storage stack used is LVM thin provisioning on LUKS,
> and I have observed significant CPU usage in dom0 kernel threads with
> names that indicate they are related to dm-thin and dm-crypt.
>
> The kernel config used by the Qubes kernel package I use (6.1.28) is
> based on Fedora 37’s config, and Marek Marczykowski-Górecki (CCd)
> indicated that the same arguments apply to Fedora. Therefore, I am
> asking if Fedora should use full kernel preemption by default.
Hi Demi
Could you please provide 'dmsetup table' - so we could see how doe your
device stack looks like ?
The output of 'dmsetup table' contains all qube names so I would prefer to not
post it publicly. The device stack is NVMe -> crypt -> linear -> thin pool ->
thin.
Aren't you disabling work-queues on the table line for crypt
targets ?
The only optional parameter passed to dm_crypt is allow_discards,
so presumably no.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
8:38 a.m.
New subject: Should Fedora switch to full kernel preemption
(CONFIG_PREEMPT=y)?
Dne 25. 05. 23 v 1:47 Demi Marie Obenour napsal(a):
On 5/24/23 08:44, Zdenek Kabelac wrote:
> Dne 20. 05. 23 v 22:43 Demi Marie Obenour napsal(a):
>> I noticed that by default, Qubes OS has voluntary kernel preemption
>> as opposed to full preemption. I found that enabling full preemption
>> (preempt=full on kernel command line) makes the system significantly
>> more responsive under heavy I/O load. In particular, if I build a
>> kernel in a Qubes OS VM, it significantly degrades responsiveness
>> without preempt=full. With preempt=full, the system remains
>> responsive. The storage stack used is LVM thin provisioning on LUKS,
>> and I have observed significant CPU usage in dom0 kernel threads with
>> names that indicate they are related to dm-thin and dm-crypt.
>>
>> The kernel config used by the Qubes kernel package I use (6.1.28) is
>> based on Fedora 37’s config, and Marek Marczykowski-Górecki (CCd)
>> indicated that the same arguments apply to Fedora. Therefore, I am
>> asking if Fedora should use full kernel preemption by default.
>
> Hi Demi
>
>
> Could you please provide 'dmsetup table' - so we could see how doe your
> device stack looks like ?
The output of 'dmsetup table' contains all qube names so I would prefer to not
post it publicly. The device stack is NVMe -> crypt -> linear -> thin pool
-> thin.
Hi
You could always 'anonymize' table names with awk/python/perl/sed - aka just
leave there first letter - names are not important - rather the content of
table line itself and the complexity of device stack. There are some limits
over which the active usage of thins may degrade the performance of the system.
But it's always easier to see & reproduce on some user's data.
So just unify/mange/anonymize the names - but tables would be interesting to see.
You may also add 'dmsetup info -c' & lsblk
Regards
Zdenek
1:12 p.m.
On Sat, May 20, 2023, at 4:43 PM, Demi Marie Obenour wrote:
Therefore, I am
asking if Fedora should use full kernel preemption by default.
https://pagure.io/fedora-workstation/issue/228
The outstanding questions:
a. Do we need some tests that help decide this with metrics? If so what should those be?
b. Should it be restricted to the desktop edition/spins? Or Fedora wide?
c. How do we make the change?
I have no ideas for a.) and I don't know what would be a sufficient sample of
workloads across all of Fedora; or whether to separately test the different editions.
I think if the answer to b.) is it should be Fedora-wide, means there's more pressure
to answer yes to a.)
Whereas if it's focused on desktops, perhaps less testing is needed?
Still another strategy might be to just make preempt full the default Fedora wide in
Rawhide with enough advance notice (like, nowish or soon after Fedora 39 branches). And
once it starts causing issues, revert. That's a bit spaghetti on the wall test
strategy but it's also cheap. (I am aware this is not a good strategy for testing
doneness of spaghetti.)
Regarding c.) gets tricky if it's not Fedora wide. The simplest case is, the change
applies only to desktops, everything else remains on voluntary. The only way I'm aware
of we can change it on the desktops is to set a kernel parameter. This means we need to
have Anaconda set it as a boot parameter, but only for desktops. There is a debugfs
facility for changing it during runtime, but this is not reliable due to debugfs being
disabled when kernel lockdown is in place, which is tied to Secure Boot being enabled.
Hence, the lkml thread referenced in the issue. But the lkml thread went no where (no
responses). Possibly it wasn't provocative enough. Or simply there are no plans to
expose this switch anywhere else. https://lkml.org/lkml/2023/4/11/1291
Conversely if even two other Fedora editions/spins/variants want some kind of opt out or
opt in, it quickly gets tricky to do all that unique work setting a boot parameter, rather
than just flipping the default across the board and not having a boot parameter.
Anyway, it seems like it would be semi-straightforward to do it in Anaconda just for
desktops using kickstart `bootloader --append=` command.
https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#bootloader
--
Chris Murphy
5:50 p.m.
On Wed, May 24, 2023, at 2:12 PM, Chris Murphy wrote:
On Sat, May 20, 2023, at 4:43 PM, Demi Marie Obenour wrote:
Therefore, I am
> asking if Fedora should use full kernel preemption by default.
https://pagure.io/fedora-workstation/issue/228
The outstanding questions:
a. Do we need some tests that help decide this with metrics? If so what
should those be?
b. Should it be restricted to the desktop edition/spins? Or Fedora wide?
c. How do we make the change?
I have no ideas for a.) and I don't know what would be a sufficient
sample of workloads across all of Fedora; or whether to separately test
the different editions.
I think if the answer to b.) is it should be Fedora-wide, means there's
more pressure to answer yes to a.)
Whereas if it's focused on desktops, perhaps less testing is needed?
I lost track of this so unfortunately there's no Fedora 39 proposal.
If the only comparison is preempt full vs voluntary, that's a simple A vs B test to
run a pile of benchmarks against. The question then is what set of benchmarks do we want
to use for this? We could use the Phoronix test suite. At least we'd get an idea if
the change would result in bad press (everyone laugh at the silly bad joke).
https://github.com/phoronix-test-suite/phoronix-test-suite
Otherwise I don't have any ideas what would be a suitable test suit. And also still
unanswered is if the change should happen Fedora wide or restricted to desktops.
If restricted to desktops, then we can only do it with kernel parameters. That probably
means doing it in Anaconda kickstart, with a per edition/spin option for doing so.
For this issue to progress, it needs one or more co-owners to help get it turned into a
feature for Fedora 40.
--
Chris Murphy
8:51 a.m.
New subject: Should Fedora switch to full kernel preemption
(CONFIG_PREEMPT=y)?
On Wed, Jul 19 2023 at 06:50:24 PM -0400, Chris Murphy
<lists(a)colorremedies.com> wrote:
If restricted to desktops, then we can only do it with kernel
parameters. That probably means doing it in Anaconda kickstart, with
a per edition/spin option for doing so.
I'm not fond of this solution. In practice, this would likely mean the
new preemption mode would apply only to new installs and not also to
upgraded systems, right?
9:01 a.m.
On Thu, Jul 20, 2023, at 9:51 AM, Michael Catanzaro wrote:
On Wed, Jul 19 2023 at 06:50:24 PM -0400, Chris Murphy
<lists(a)colorremedies.com> wrote:
> If restricted to desktops, then we can only do it with kernel
> parameters. That probably means doing it in Anaconda kickstart, with
> a per edition/spin option for doing so.
I'm not fond of this solution. In practice, this would likely mean the
new preemption mode would apply only to new installs and not also to
upgraded systems, right?
Probably.
A grubby command could add the proper kernel parameter to the bootloader configuration,
but grubby isn't being used anywhere in Fedora (not anaconda for initial installs, not
when installing kernel upgrades) so it isn't being exercised.
The more users customize, the less likely grubby is going to detect their use case and
fail gracefully. grubby would need to have a strict "only add new parameter"
option. Right now it also replaces the kernel command line with the contents of
/etc/kernel/cmdline to all bootloader snippets in /boot/loader/entries - including
snippets for other OS's, which would be bad.
--
Chris Murphy
319
days inactive
380
days old
12 comments
7 participants
participants (7)
-
Chris Murphy -
Daniel P. Berrangé -
Demi Marie Obenour -
Justin Forbes -
Michael Catanzaro -
stan -
Zdenek Kabelac