4:30 a.m.
On Mon, Mar 14, 2022 at 2:09 AM Huang, Ying <ying.huang(a)intel.com> wrote:
Hi, Yu,
Yu Zhao <yuzhao(a)google.com> writes:
> diff --git a/mm/Kconfig b/mm/Kconfig
> index 3326ee3903f3..747ab1690bcf 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -892,6 +892,16 @@ config ANON_VMA_NAME
> area from being merged with adjacent virtual memory areas due to the
> difference in their name.
>
> +# the multi-gen LRU {
> +config LRU_GEN
> + bool "Multi-Gen LRU"
> + depends on MMU
> + # the following options can use up the spare bits in page flags
> + depends on !MAXSMP && (64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP)
LRU_GEN depends on !MAXSMP. So, What is the maximum NR_CPUS supported
by LRU_GEN?
LRU_GEN doesn't really care about NR_CPUS. IOW, it doesn't impose a
max number. The dependency is with NODES_SHIFT selected by MAXSMP:
default "10" if MAXSMP
This combined with LAST_CPUPID_SHIFT can exhaust the spare bits in page flags.
MAXSMP is meant for kernel developers to test their code, and it
should not be used in production [1]. But some distros unfortunately
ship kernels built with this option, e.g., Fedora and Ubuntu. And
their users reported build errors to me after they applied MGLRU on
those kernels ("Not enough bits in page flags"). Let me add Fedora and
Ubuntu to this thread.
Fedora and Ubuntu,
Could you please clarify if there is a reason to ship kernels built
with MAXSMP? Otherwise, please consider disabling this option. Thanks.
As per above, MAXSMP enables ridiculously large numbers of CPUs and
NUMA nodes for testing purposes. It is detrimental to performance,
e.g., CPUMASK_OFFSTACK.
[1] https://lore.kernel.org/lkml/20131106055634.GA24044@gmail.com/
7:34 p.m.
New subject: [PATCH v9 05/14] mm: multi-gen LRU: groundwork
Yu Zhao <yuzhao(a)google.com> writes:
On Mon, Mar 14, 2022 at 2:09 AM Huang, Ying
<ying.huang(a)intel.com> wrote:
>
> Hi, Yu,
>
> Yu Zhao <yuzhao(a)google.com> writes:
> > diff --git a/mm/Kconfig b/mm/Kconfig
> > index 3326ee3903f3..747ab1690bcf 100644
> > --- a/mm/Kconfig
> > +++ b/mm/Kconfig
> > @@ -892,6 +892,16 @@ config ANON_VMA_NAME
> > area from being merged with adjacent virtual memory areas due to the
> > difference in their name.
> >
> > +# the multi-gen LRU {
> > +config LRU_GEN
> > + bool "Multi-Gen LRU"
> > + depends on MMU
> > + # the following options can use up the spare bits in page flags
> > + depends on !MAXSMP && (64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP)
>
> LRU_GEN depends on !MAXSMP. So, What is the maximum NR_CPUS supported
> by LRU_GEN?
LRU_GEN doesn't really care about NR_CPUS. IOW, it doesn't impose a
max number. The dependency is with NODES_SHIFT selected by MAXSMP:
default "10" if MAXSMP
This combined with LAST_CPUPID_SHIFT can exhaust the spare bits in page flags.
From the following code snippets from page-flags-layout.h,
LAST_CPUPID_SHIFT is related to NR_CPUS instead of NODES_SHIFT.
#define LAST__PID_SHIFT 8
#define LAST__PID_MASK ((1 << LAST__PID_SHIFT)-1)
#define LAST__CPU_SHIFT NR_CPUS_BITS
#define LAST__CPU_MASK ((1 << LAST__CPU_SHIFT)-1)
#define LAST_CPUPID_SHIFT (LAST__PID_SHIFT+LAST__CPU_SHIFT)
Best Regards,
Huang, Ying
[snip]
7:50 p.m.
New subject: [PATCH v9 05/14] mm: multi-gen LRU: groundwork
On Mon, Mar 14, 2022 at 6:34 PM Huang, Ying <ying.huang(a)intel.com> wrote:
Yu Zhao <yuzhao(a)google.com> writes:
> On Mon, Mar 14, 2022 at 2:09 AM Huang, Ying <ying.huang(a)intel.com> wrote:
>>
>> Hi, Yu,
>>
>> Yu Zhao <yuzhao(a)google.com> writes:
>> > diff --git a/mm/Kconfig b/mm/Kconfig
>> > index 3326ee3903f3..747ab1690bcf 100644
>> > --- a/mm/Kconfig
>> > +++ b/mm/Kconfig
>> > @@ -892,6 +892,16 @@ config ANON_VMA_NAME
>> > area from being merged with adjacent virtual memory areas due to
the
>> > difference in their name.
>> >
>> > +# the multi-gen LRU {
>> > +config LRU_GEN
>> > + bool "Multi-Gen LRU"
>> > + depends on MMU
>> > + # the following options can use up the spare bits in page flags
>> > + depends on !MAXSMP && (64BIT || !SPARSEMEM ||
SPARSEMEM_VMEMMAP)
>>
>> LRU_GEN depends on !MAXSMP. So, What is the maximum NR_CPUS supported
>> by LRU_GEN?
>
> LRU_GEN doesn't really care about NR_CPUS. IOW, it doesn't impose a
> max number. The dependency is with NODES_SHIFT selected by MAXSMP:
> default "10" if MAXSMP
> This combined with LAST_CPUPID_SHIFT can exhaust the spare bits in page flags.
From the following code snippets from page-flags-layout.h,
LAST_CPUPID_SHIFT is related to NR_CPUS instead of NODES_SHIFT.
It is. But LAST_CPUPID_NOT_IN_PAGE_FLAGS should always work but
NODE_NOT_IN_PAGE_FLAGS doesn't.
1:58 p.m.
New subject: [PATCH v9 05/14] mm: multi-gen LRU: groundwork
On Mon, Mar 14, 2022 at 4:30 AM Yu Zhao <yuzhao(a)google.com> wrote:
On Mon, Mar 14, 2022 at 2:09 AM Huang, Ying <ying.huang(a)intel.com> wrote:
>
> Hi, Yu,
>
> Yu Zhao <yuzhao(a)google.com> writes:
> > diff --git a/mm/Kconfig b/mm/Kconfig
> > index 3326ee3903f3..747ab1690bcf 100644
> > --- a/mm/Kconfig
> > +++ b/mm/Kconfig
> > @@ -892,6 +892,16 @@ config ANON_VMA_NAME
> > area from being merged with adjacent virtual memory areas due to the
> > difference in their name.
> >
> > +# the multi-gen LRU {
> > +config LRU_GEN
> > + bool "Multi-Gen LRU"
> > + depends on MMU
> > + # the following options can use up the spare bits in page flags
> > + depends on !MAXSMP && (64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP)
>
> LRU_GEN depends on !MAXSMP. So, What is the maximum NR_CPUS supported
> by LRU_GEN?
LRU_GEN doesn't really care about NR_CPUS. IOW, it doesn't impose a
max number. The dependency is with NODES_SHIFT selected by MAXSMP:
default "10" if MAXSMP
This combined with LAST_CPUPID_SHIFT can exhaust the spare bits in page flags.
MAXSMP is meant for kernel developers to test their code, and it
should not be used in production [1]. But some distros unfortunately
ship kernels built with this option, e.g., Fedora and Ubuntu. And
their users reported build errors to me after they applied MGLRU on
those kernels ("Not enough bits in page flags"). Let me add Fedora and
Ubuntu to this thread.
Fedora and Ubuntu,
Could you please clarify if there is a reason to ship kernels built
with MAXSMP? Otherwise, please consider disabling this option. Thanks.
As per above, MAXSMP enables ridiculously large numbers of CPUs and
NUMA nodes for testing purposes. It is detrimental to performance,
e.g., CPUMASK_OFFSTACK.
It was enabled for Fedora, and RHEL because we did need more than 512
CPUs, originally only in RHEL until SGI (years ago) complained that
they were testing very large machines with Fedora. The testing done
on RHEL showed that the performance impact was minimal. For a very
long time we had MAXSMP off and carried a patch which allowed us to
turn on CPUMASK_OFFSTACK without debugging because there was supposed
to be "something else" coming. In 2019 we gave up, dropped that patch
and just turned on MAXSMP.
I do not have any metrics for how often someone runs Fedora on a
ridiculously large machine these days, but I would guess that number
is not 0.
Justin
> [1] https://lore.kernel.org/lkml/20131106055634.GA24044@gmail.com/
>
2:17 p.m.
New subject: [PATCH v9 05/14] mm: multi-gen LRU: groundwork
On 3/21/22 14:58, Justin Forbes wrote:
On Mon, Mar 14, 2022 at 4:30 AM Yu Zhao <yuzhao(a)google.com>
wrote:
>
> On Mon, Mar 14, 2022 at 2:09 AM Huang, Ying <ying.huang(a)intel.com> wrote:
>>
>> Hi, Yu,
>>
>> Yu Zhao <yuzhao(a)google.com> writes:
>>> diff --git a/mm/Kconfig b/mm/Kconfig
>>> index 3326ee3903f3..747ab1690bcf 100644
>>> --- a/mm/Kconfig
>>> +++ b/mm/Kconfig
>>> @@ -892,6 +892,16 @@ config ANON_VMA_NAME
>>> area from being merged with adjacent virtual memory areas due to
the
>>> difference in their name.
>>>
>>> +# the multi-gen LRU {
>>> +config LRU_GEN
>>> + bool "Multi-Gen LRU"
>>> + depends on MMU
>>> + # the following options can use up the spare bits in page flags
>>> + depends on !MAXSMP && (64BIT || !SPARSEMEM ||
SPARSEMEM_VMEMMAP)
>>
>> LRU_GEN depends on !MAXSMP. So, What is the maximum NR_CPUS supported
>> by LRU_GEN?
>
> LRU_GEN doesn't really care about NR_CPUS. IOW, it doesn't impose a
> max number. The dependency is with NODES_SHIFT selected by MAXSMP:
> default "10" if MAXSMP
> This combined with LAST_CPUPID_SHIFT can exhaust the spare bits in page flags.
>
> MAXSMP is meant for kernel developers to test their code, and it
> should not be used in production [1]. But some distros unfortunately
> ship kernels built with this option, e.g., Fedora and Ubuntu. And
> their users reported build errors to me after they applied MGLRU on
> those kernels ("Not enough bits in page flags"). Let me add Fedora and
> Ubuntu to this thread.
>
> Fedora and Ubuntu,
>
> Could you please clarify if there is a reason to ship kernels built
> with MAXSMP? Otherwise, please consider disabling this option. Thanks.
>
> As per above, MAXSMP enables ridiculously large numbers of CPUs and
> NUMA nodes for testing purposes. It is detrimental to performance,
> e.g., CPUMASK_OFFSTACK.
It was enabled for Fedora, and RHEL because we did need more than 512
CPUs, originally only in RHEL until SGI (years ago) complained that
they were testing very large machines with Fedora. The testing done
on RHEL showed that the performance impact was minimal. For a very
long time we had MAXSMP off and carried a patch which allowed us to
turn on CPUMASK_OFFSTACK without debugging because there was supposed
to be "something else" coming. In 2019 we gave up, dropped that patch
and just turned on MAXSMP.
I do not have any metrics for how often someone runs Fedora on a
ridiculously large machine these days, but I would guess that number
is not 0.
It is not 0. I've seen data from large systems (1000+ logical threads)
that are running Fedora albeit with a modified Fedora kernel.
Additionally the max limit for CPUS in RHEL is 1792, however, we have
recently had a request to *double* that to 3584. You should just assume
that number will continue to increase.
P.
Justin
> [1] https://lore.kernel.org/lkml/20131106055634.GA24044@gmail.com/
>
_______________________________________________
kernel mailing list -- kernel(a)lists.fedoraproject.org
To unsubscribe send an email to kernel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
11:52 p.m.
New subject: [PATCH v9 05/14] mm: multi-gen LRU: groundwork
On Mon, Mar 21, 2022 at 1:18 PM Prarit Bhargava <prarit(a)redhat.com> wrote:
On 3/21/22 14:58, Justin Forbes wrote:
> On Mon, Mar 14, 2022 at 4:30 AM Yu Zhao <yuzhao(a)google.com> wrote:
>>
>> On Mon, Mar 14, 2022 at 2:09 AM Huang, Ying <ying.huang(a)intel.com> wrote:
>>>
>>> Hi, Yu,
>>>
>>> Yu Zhao <yuzhao(a)google.com> writes:
>>>> diff --git a/mm/Kconfig b/mm/Kconfig
>>>> index 3326ee3903f3..747ab1690bcf 100644
>>>> --- a/mm/Kconfig
>>>> +++ b/mm/Kconfig
>>>> @@ -892,6 +892,16 @@ config ANON_VMA_NAME
>>>> area from being merged with adjacent virtual memory areas due
to the
>>>> difference in their name.
>>>>
>>>> +# the multi-gen LRU {
>>>> +config LRU_GEN
>>>> + bool "Multi-Gen LRU"
>>>> + depends on MMU
>>>> + # the following options can use up the spare bits in page flags
>>>> + depends on !MAXSMP && (64BIT || !SPARSEMEM ||
SPARSEMEM_VMEMMAP)
>>>
>>> LRU_GEN depends on !MAXSMP. So, What is the maximum NR_CPUS supported
>>> by LRU_GEN?
>>
>> LRU_GEN doesn't really care about NR_CPUS. IOW, it doesn't impose a
>> max number. The dependency is with NODES_SHIFT selected by MAXSMP:
>> default "10" if MAXSMP
>> This combined with LAST_CPUPID_SHIFT can exhaust the spare bits in page flags.
>>
>> MAXSMP is meant for kernel developers to test their code, and it
>> should not be used in production [1]. But some distros unfortunately
>> ship kernels built with this option, e.g., Fedora and Ubuntu. And
>> their users reported build errors to me after they applied MGLRU on
>> those kernels ("Not enough bits in page flags"). Let me add Fedora
and
>> Ubuntu to this thread.
>>
>> Fedora and Ubuntu,
>>
>> Could you please clarify if there is a reason to ship kernels built
>> with MAXSMP? Otherwise, please consider disabling this option. Thanks.
>>
>> As per above, MAXSMP enables ridiculously large numbers of CPUs and
>> NUMA nodes for testing purposes. It is detrimental to performance,
>> e.g., CPUMASK_OFFSTACK.
>
> It was enabled for Fedora, and RHEL because we did need more than 512
> CPUs, originally only in RHEL until SGI (years ago) complained that
> they were testing very large machines with Fedora. The testing done
> on RHEL showed that the performance impact was minimal. For a very
> long time we had MAXSMP off and carried a patch which allowed us to
> turn on CPUMASK_OFFSTACK without debugging because there was supposed
> to be "something else" coming. In 2019 we gave up, dropped that patch
> and just turned on MAXSMP.
>
> I do not have any metrics for how often someone runs Fedora on a
> ridiculously large machine these days, but I would guess that number
> is not 0.
It is not 0. I've seen data from large systems (1000+ logical threads)
that are running Fedora albeit with a modified Fedora kernel.
Additionally the max limit for CPUS in RHEL is 1792, however, we have
recently had a request to *double* that to 3584. You should just assume
that number will continue to increase.
Good to know. Thanks.
From the standpoint of overhead, I'd consider NR_CPUS=4096 and
NODES_SHIFT=7 as the next step, before going with MAXSMP.
788
days inactive
796
days old
kernel@lists.fedoraproject.org
5 comments
4 participants
participants (4)
-
Huang, Ying -
Justin Forbes -
Prarit Bhargava -
Yu Zhao