CONFIG_NR_CPUS

[Xose Vazquez Perez]

>  * Tue Jan 17 2012 Dave Jones <davej(a)redhat.com&gt;
>  - Rawhide builds now use MAXSMP on x86.
>  - For release builds, set x86-64 to support 64 CPUs.
>    If larger systems become widespread, we can increase in an update. 
_today_

amd:   4sockets * 16cores = 64
intel: 4sockets * 10cores * 2threads = 80

[Josh Boyer]

On Sat, May 19, 2012 at 02:54:33PM +0200, Xose Vazquez Perez wrote:
>  
>  >* Tue Jan 17 2012 Dave Jones <davej(a)redhat.com&gt;
>  >- Rawhide builds now use MAXSMP on x86.
>  >- For release builds, set x86-64 to support 64 CPUs.
>  >  If larger systems become widespread, we can increase in an update.
>  
>  _today_
>  
>  amd:   4sockets * 16cores = 64 
Awesome.  Got that covered still.

>  intel: 4sockets * 10cores * 2threads = 80 
Which particular CPU/Motherboard combo is that, and how often do we see
it in Fedora?

I'm not opposed to bumping it up to 128 or something, but I'm curious
how many people are actually going to see benefits.

josh

[Gilboa Davara]

On Sun, May 20, 2012 at 4:07 AM, Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  On Sat, May 19, 2012 at 02:54:33PM +0200, Xose Vazquez Perez wrote:
>  >
>  > >* Tue Jan 17 2012 Dave Jones <davej(a)redhat.com&gt;
>  > >- Rawhide builds now use MAXSMP on x86.
>  > >- For release builds, set x86-64 to support 64 CPUs.
>  > >  If larger systems become widespread, we can increase in an update.
>  >
>  > _today_
>  >
>  > amd:   4sockets * 16cores = 64
>
>  Awesome.  Got that covered still.
>
>  > intel: 4sockets * 10cores * 2threads = 80
>
>  Which particular CPU/Motherboard combo is that, and how often do we see
>  it in Fedora?
>
>  I'm not opposed to bumping it up to 128 or something, but I'm curious
>  how many people are actually going to see benefits.
>
>  josh 
At least in my case I did run Fedora 12-16 on 4S and 8S machines to
test software scalability on (extreme) high-end hardware.
Though, IMHO anyone that's crazy enough to run Fedora on a high-end
4S/8S machine is more than capable of rebuilding the kernel with
CONFIG_NR_CPUS 256...

However, given the fact that x86_64 machines tend to be far less
memory constrained than i686 machines, I doubt that raising the limit
to 128 will cause too many issues. (Isn't NR_CPUS == 512 in el6?)

- Gilboa

[Ruben Kerkhof]

On Sun, May 20, 2012 at 2:32 PM, Gilboa Davara <gilboad(a)gmail.com&gt; wrote:
>  On Sun, May 20, 2012 at 4:07 AM, Josh Boyer
> <jwboyer(a)redhat.com&gt; wrote:
> >
> > On Sat, May 19, 2012 at 02:54:33PM +0200, Xose Vazquez Perez wrote:
> > >
> > > >* Tue Jan 17 2012 Dave Jones <davej(a)redhat.com&gt;
> > > >- Rawhide builds now use MAXSMP on x86.
> > > >- For release builds, set x86-64 to support 64 CPUs.
> > > >  If larger systems become widespread, we can increase in an update.
> > >
> > > _today_
> > >
> > > amd:   4sockets * 16cores = 64
> >
> > Awesome.  Got that covered still.
> >
> > > intel: 4sockets * 10cores * 2threads = 80
> >
> > Which particular CPU/Motherboard combo is that, and how often do we see
> > it in Fedora?
> >
> > I'm not opposed to bumping it up to 128 or something, but I'm curious
> > how many people are actually going to see benefits.
> >
> > josh
>
>  At least in my case I did run Fedora 12-16 on 4S and 8S machines to
>  test software scalability on (extreme) high-end hardware.
>  Though, IMHO anyone that's crazy enough to run Fedora on a high-end
>  4S/8S machine is more than capable of rebuilding the kernel with
>  CONFIG_NR_CPUS 256...
>
>  However, given the fact that x86_64 machines tend to be far less
>  memory constrained than i686 machines, I doubt that raising the limit
>  to 128 will cause too many issues. (Isn't NR_CPUS == 512 in el6?) 
It's even higher these days:

grep NR_CPUS /boot/config-2.6.32-220.13.1.el6.x86_64
CONFIG_NR_CPUS=4096

I'm curious, has anyone measured what the memory overhead is of
keeping NR_CPUS at 512?
arch/x86/Kconfig says "This is purely to save memory - each supported
CPU adds approximately eight kilobytes to the kernel image." If that's
true, 512 cpus use 4MB, something I'm willing to live with on my 64bit
 servers.

Thanks,

Ruben

[Prarit Bhargava]

On 05/20/2012 11:23 AM, Ruben Kerkhof wrote:
>  On Sun, May 20, 2012 at 2:32 PM, Gilboa Davara
> <gilboad(a)gmail.com&gt; wrote:
> > On Sun, May 20, 2012 at 4:07 AM, Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
> >>
> >> On Sat, May 19, 2012 at 02:54:33PM +0200, Xose Vazquez Perez wrote:
> >>>
> >>>> * Tue Jan 17 2012 Dave Jones <davej(a)redhat.com&gt;
> >>>> - Rawhide builds now use MAXSMP on x86.
> >>>> - For release builds, set x86-64 to support 64 CPUs.
> >>>>  If larger systems become widespread, we can increase in an update.
> >>>
> >>> _today_
> >>>
> >>> amd:   4sockets * 16cores = 64
> >>
> >> Awesome.  Got that covered still.
> >>
> >>> intel: 4sockets * 10cores * 2threads = 80
> >>
> >> Which particular CPU/Motherboard combo is that, and how often do we see
> >> it in Fedora?
> >>
> >> I'm not opposed to bumping it up to 128 or something, but I'm curious
> >> how many people are actually going to see benefits.
> >>
> >> josh
> >
> > At least in my case I did run Fedora 12-16 on 4S and 8S machines to
> > test software scalability on (extreme) high-end hardware.
> > Though, IMHO anyone that's crazy enough to run Fedora on a high-end
> > 4S/8S machine is more than capable of rebuilding the kernel with
> > CONFIG_NR_CPUS 256...
> >
> > However, given the fact that x86_64 machines tend to be far less
> > memory constrained than i686 machines, I doubt that raising the limit
> > to 128 will cause too many issues. (Isn't NR_CPUS == 512 in el6?)
>  
>  It's even higher these days:
>  
>  grep NR_CPUS /boot/config-2.6.32-220.13.1.el6.x86_64
>  CONFIG_NR_CPUS=4096 
This is from RHEL which has a markedly different user base than Fedora does.  I
remember setting this in RHEL based on a partner request who ships LARGE systems
(4096 CPUs, many many many TB of memory).

This is not the number you want for Fedora.

>  
>  I'm curious, has anyone measured what the memory overhead is of
>  keeping NR_CPUS at 512? 
I think we did that in RHEL.  It is insignificant IIRC.  I'll try and dig out
some numbers...

P.

[Xose Vazquez Perez]

On 05/20/2012 03:07 AM, Josh Boyer wrote:

> > intel: 4sockets * 10cores * 2threads = 80
>
>  Which particular CPU/Motherboard combo is that, and how often do we see
>  it in Fedora? 
- Xeon E7:
http://en.wikipedia.org/wiki/Nehalem_%28microarchitecture%29#Server_.2F_D...

http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html
http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-...
http://www.dell.com/us/business/p/poweredge-r910/pd

- Opteron 6200:
http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Opteron_...

http://www-03.ibm.com/systems/x/hardware/rack/x3755m3/index.html
http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-...
http://www.dell.com/us/business/p/poweredge-r815/pd
http://www.supermicro.nl/Aplus/motherboard/Opteron6000/SR56x0/H8QG6-F.cfm

[.....]

secondary archs:
z196: 24sockets * 4cores = 96, but only 80IFL.
power7: 32sockets * 8cores * 4 threads = 1024, 795 and 775 servers.
sparc: we will see, when oracle release Oracle Enterprise Linux for SPARC.

[Prarit Bhargava]

On 05/20/2012 06:20 PM, Xose Vazquez Perez wrote:
>  On 05/20/2012 03:07 AM, Josh Boyer wrote:
>  
> >> intel: 4sockets * 10cores * 2threads = 80
> >
> > Which particular CPU/Motherboard combo is that, and how often do we see
> > it in Fedora?
>  
>  - Xeon E7:
>
> http://en.wikipedia.org/wiki/Nehalem_%28microarchitecture%29#Server_.2F_D...
>  
>  
>  http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html
>
> http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-...
>  
>  http://www.dell.com/us/business/p/poweredge-r910/pd
>  
>  - Opteron 6200:
>
> http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Opteron_...
>  
>  
>  http://www-03.ibm.com/systems/x/hardware/rack/x3755m3/index.html
>
> http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-...
>  
>  http://www.dell.com/us/business/p/poweredge-r815/pd
>  http://www.supermicro.nl/Aplus/motherboard/Opteron6000/SR56x0/H8QG6-F.cfm
>  
>  [.....]
>   
Xose,

We should start supporting these in Fedora.  These are not unusual to find in
systems.

Josh, the reality is that 64 isn't "enough" anymore.  Even deskside systems
are
starting to ship with 32 cores, so increasing the number of CPUs to 128 is a
good idea.  I've had a lot of experience with larger systems.  The increase in
memory footprint is minimal (a few Kb, not Mb).

I do know that some vendors who do upstream development, do install Fedora on
systems with greater than 64 logical cpus.  It would be good if we could support
them fully.

Jumping to 128 is a very good idea.  We know it is stable given RHEL's support
of 4096 (again, I don't think Fedora should entertain jumping to 4096 for a
variety of reasons).

In short, ACK ;)

P.

[Josh Boyer]

On Mon, May 21, 2012 at 08:42:12AM -0400, Prarit Bhargava wrote:
>  Josh, the reality is that 64 isn't "enough" anymore. 
> Even deskside systems are
>  starting to ship with 32 cores, so increasing the number of CPUs to 128 is a
>  good idea.  I've had a lot of experience with larger systems.  The increase in
>  memory footprint is minimal (a few Kb, not Mb). 
I'll get this into rawhide shortly unless someone else brings up
counter-points (doubtful).

>  Jumping to 128 is a very good idea.  We know it is stable given
> RHEL's support
>  of 4096 (again, I don't think Fedora should entertain jumping to 4096 for a
>  variety of reasons). 
I'm not concerned with the stability aspect of it.  Before the
recent-ish change, we had it set to 256 on released kernels and 512 for
rawhide/debug kernels.  We know it works.  For rawhide/debug kernels we
already set it to MAXSMP.

For the record, I'm not going to change 32-bit kernels.  I don't see
much of a need to increase that beyond what it is today and the same
arguments about more cores being added don't really apply to 32-bit
chips.

(Before someone points out that you can run 32-bit kernels on 64-bit
hardware, I just wanted to say I don't care.  If you have a CPU that has
128 cores and want to use them all, use a 64-bit kernel ;) .)

josh

[Prarit Bhargava]

>  
>  For the record, I'm not going to change 32-bit kernels.  I don't see
>  much of a need to increase that beyond what it is today and the same
>  arguments about more cores being added don't really apply to 32-bit
>  chips.
>   
I'd like to see 32-bit go away :)  Please? :) :) :)

Seriously though, I think giving people yet another excuse to migrate away from
32-bit is a good idea and I cannot see any merit in increasing the 32-bit NR_CPUS.

P.

[Xose Vazquez Perez]

On 05/21/2012 12:20 AM, Xose Vazquez Perez wrote:

>  On 05/20/2012 03:07 AM, Josh Boyer wrote:
>
> >> intel: 4sockets * 10cores * 2threads = 80
> >
> > Which particular CPU/Motherboard combo is that, and how often do we see
> > it in Fedora?
>
>  - Xeon E7:
>
> http://en.wikipedia.org/wiki/Nehalem_%28microarchitecture%29#Server_.2F_D...
>
>
>  http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html
>
> http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-...
>  http://www.dell.com/us/business/p/poweredge-r910/pd 
bigger *Xeon* server, 8_sockets * 10_cores * 2_threads = 160

Sun Fire X4800 M2 Server:
http://www.oracle.com/us/products/servers-storage/servers/x86/sun-fire-x4...

Super Micro X8OBN-F:
http://www.supermicro.com/products/motherboard/Xeon7000/7500/x8obn-f.cfm

2x HP ProLiant DL980 G7 Server (HP PREMA):
http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-...

2x IBM System x3950 X5 (2-node):
http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html

SGI servers need MAXSMP.

[Xose Vazquez Perez]

On 05/24/2012 01:21 AM, Xose Vazquez Perez wrote:

>  bigger *Xeon* server, 8_sockets * 10_cores * 2_threads = 160
>
>  Sun Fire X4800 M2 Server:
>
> http://www.oracle.com/us/products/servers-storage/servers/x86/sun-fire-x4...
>
>
>  Super Micro X8OBN-F:
>  http://www.supermicro.com/products/motherboard/Xeon7000/7500/x8obn-f.cfm
>
>  2x HP ProLiant DL980 G7 Server (HP PREMA):
>
> http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-...
>
>
>  2x IBM System x3950 X5 (2-node):
>  http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html
>
>  SGI servers need MAXSMP. 
Also 
http://www.fujitsu.com/fts/products/computing/servers/primergy/rack/rx900/

... and more are coming.

Good bye Itanium, SPARC and POWER.

[Josh Boyer]

On Thu, May 24, 2012 at 01:35:59AM +0200, Xose Vazquez Perez wrote:
>  On 05/24/2012 01:21 AM, Xose Vazquez Perez wrote:
>  
>  >bigger *Xeon* server, 8_sockets * 10_cores * 2_threads = 160
>  >
>  >Sun Fire X4800 M2 Server:
>
> >http://www.oracle.com/us/products/servers-storage/servers/x86/sun-fire-x4800-m2/overview/index.html
>  >
>  >
>  >Super Micro X8OBN-F:
>  >http://www.supermicro.com/products/motherboard/Xeon7000/7500/x8obn-f.cfm
>  >
>  >2x HP ProLiant DL980 G7 Server (HP PREMA):
>
> >http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-4222584.html?dnr=1
>  >
>  >
>  >2x IBM System x3950 X5 (2-node):
>  >http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html
>  >
>  >SGI servers need MAXSMP.
>  
>  Also http://www.fujitsu.com/fts/products/computing/servers/primergy/rack/rx900/
>  
>  ... and more are coming. 
We made the change 2 days ago.  You're belaboring the point just a bit
now.

Also, the cheapest of these is $13k.  That's roughly 10x the cost of a
mid-high end home desktop and well outside Fedora's normal target user.
If someone wants to spend that kind of money on an enterprise server and
run Fedora on it, I'll gladly thank them for being a user but most of
those kinds of purchases are done by businesses that want something a
bit more stable and supported.

>  Good bye Itanium, SPARC and POWER. 
Hello ARM?

josh