computation-friendly kernels

[Bevan C. Bennett]

I suspect a full answer to this may be too involved for this list, but 
maybe someone a little more into kernel hacking than I've been lately 
could point me in the right direction.

I have two identical servers, one running FC1 and one running severn 
(RHEL3beta). The installed kernels are:

kernel-smp-2.4.22-1.2138.nptl (FC1)
kernel-smp-2.4.21-1.1931.2.399.ent (severn)

I have a very highly multithreaded application that runs generally 4x 
slower on FC1, but occassionally takes 20x as long. Interestingly, the 
"user" time in all cases is very consistant, and on the non-FC1 system 
real time is only fractionally higher than user time.  So... I suspect 
that there's some kernel parameter, memory management scheme or 
scheduling configuration difference causing the performance disparity.

My question is whether there's any hope of discovering which difference 
is involved and 'fixing' the FC1 system (or even of easily enumerating 
what the differences actually are)?

I know that FC is aimed more at a desktop/general user release (and is 
the best Linux I've used so far for that purpose), and that ideally I'd 
help support RedHat by purchasing 100 copies of RHEL for my heavy 
computing, but there are unfortunate budgetary forces involved that 
prohibit that at the moment. If the company situation improves, that's 
the way I'd like to go eventually.

I'm willing to compile custom kernels for a 4x speedup and better 
consistancy, but I'd like to keep the number of different distributions 
in active use to a minimum (every different distribution adds more work 
in creating and maintaining compatible configuration info).

Any informed suggestions from the community?

[Alexander Dalloz]

Am Do, den 08.01.2004 schrieb Bevan C. Bennett um 19:38:
>  I suspect a full answer to this may be too involved for this list,
> but 
>  maybe someone a little more into kernel hacking than I've been lately 
>  could point me in the right direction.
>  
>  I have two identical servers, one running FC1 and one running severn 
>  (RHEL3beta). The installed kernels are:
>  
>  kernel-smp-2.4.22-1.2138.nptl (FC1)
>  kernel-smp-2.4.21-1.1931.2.399.ent (severn) 
You mean not Severn, which was a Fedora beta, but you mean Taroon. I
just checked that over the kernel version you give and that matches.

>  I have a very highly multithreaded application that runs generally 4x
>
>  slower on FC1, but occassionally takes 20x as long. Interestingly, the 
>  "user" time in all cases is very consistant, and on the non-FC1 system 
>  real time is only fractionally higher than user time.  So... I suspect 
>  that there's some kernel parameter, memory management scheme or 
>  scheduling configuration difference causing the performance disparity. 
4 times up to 20 times faster on Taroon compared to Fedora Core 1 is
amazingly. Are you running one specific multithreaded application or a
set of those? How do you measure the time it takes - by "time program"?

>  My question is whether there's any hope of discovering which
> difference 
>  is involved and 'fixing' the FC1 system (or even of easily enumerating 
>  what the differences actually are)? 
The RHEL kernel is very optimized for big machines with much RAM,
multiprozessors and so on. Can you describe your machine park a little
bit?

>  I know that FC is aimed more at a desktop/general user release (and
> is 
>  the best Linux I've used so far for that purpose), and that ideally I'd 
>  help support RedHat by purchasing 100 copies of RHEL for my heavy 
>  computing, but there are unfortunate budgetary forces involved that 
>  prohibit that at the moment. If the company situation improves, that's 
>  the way I'd like to go eventually.
>  
>  I'm willing to compile custom kernels for a 4x speedup and better 
>  consistancy, but I'd like to keep the number of different distributions 
>  in active use to a minimum (every different distribution adds more work 
>  in creating and maintaining compatible configuration info).
>  
>  Any informed suggestions from the community? 
Maybe you should post you question on the fedora-devel list. It seems
the Redhat folks are paying there more attention to such questions than
here.

Regards

Alexander


-- 
Alexander Dalloz | Enger, Germany
PGP key valid: made 13.07.1999
PGP fingerprint: 2307 88FD 2D41 038E 7416  14CD E197 6E88 ED69 5653

[Bevan C. Bennett]

Alexander Dalloz wrote:

>  You mean not Severn, which was a Fedora beta, but you mean Taroon. I
>  just checked that over the kernel version you give and that matches. 
Oops. Indeed I meant Taroon. Too many nonsensical (to me) names...

>  4 times up to 20 times faster on Taroon compared to Fedora Core 1 is
>  amazingly. Are you running one specific multithreaded application or a
>  set of those? How do you measure the time it takes - by "time program"? 
Yes, I've just been doing some basic profiling with 'time program' using 
one specific multithreaded (Java-based) application with various input 
files. To be more exact, I only see the differences on the largest of 
these sample cases, less intensive runs show a smaller difference.

FC1.3 runs...
realtime: 2128s 2271s 7418s
usertime:  693s  697s  748s

Taroon runs...
realtime:  735s  765s
usertime:  693s  728s

>  The RHEL kernel is very optimized for big machines with much RAM,
>  multiprozessors and so on. Can you describe your machine park a little
>  bit? 
8 x dual 2.8 GHz Dell PowerEdge 1750
8 x dual 2.4 GHz Dell PowerEdge 2650
8 x dual 2.2 GHz Dell PowerEdge 2650
8 x dual 1.2GHz IBM eServers
12 x dual 1.2GHz Dell PowerEdge 2550

all systems have 2GB of RAM
The testing was done primarily using the 1750s, but I get comparable 
results from the 2650s. Most of the pool is still running RH7.3 and I'm 
trying to figure out the best upgrade path for them (these tests don't 
even run without NPTL).

> >Any informed suggestions from the community?
>  Maybe you should post you question on the fedora-devel list. It seems
>  the Redhat folks are paying there more attention to such questions than
>  here. 
Thanks. I'll probably want to write up a slightly more technically 
complete version for that venue. Here I'm mostly hoping for general 
kernel-analysis advice.

[Bob Arendt]

Bevan C. Bennett wrote:
>  Yes, I've just been doing some basic profiling with 'time
> program' using 
>  one specific multithreaded (Java-based) application with various input 
>  files. To be more exact, I only see the differences on the largest of 
>  these sample cases, less intensive runs show a smaller difference. 
Might it be possible that you're seeing I/O subsystem differences, rather than
computational/scheduling effects?  What kind of split do you see if
something is largely compute-only vs. I/O-only (minimal computation).
-Bob Arendt

[Bill Rugolsky Jr.]

On Thu, Jan 08, 2004 at 12:16:17PM -0800, Bevan C. Bennett wrote:
>  Yes, I've just been doing some basic profiling with 'time
> program' using 
>  one specific multithreaded (Java-based) application with various input 
>  files. To be more exact, I only see the differences on the largest of 
>  these sample cases, less intensive runs show a smaller difference. 
When you say "largest", which parameter are you scaling?  Are you
using a significant fraction of RAM?  RHEL and Fedora have
a completely different VM.  Also, highmem support in FC1 may be broken.

I'd suggest using Oprofile or readprofile to find where the kernel is spending
its time.

Regards,

	Bill Rugolsky

[Bevan C. Bennett]

Bill Rugolsky Jr. wrote:

>  When you say "largest", which parameter are you scaling?
Input cell size, which increases the number of threads and therefore RAM 
utilization.

>  Are you using a significant fraction of RAM? 
Very much so. We're pushing our RAM very hard, and considering 
installing even more.

>  RHEL and Fedora have a completely different VM. 
Ah, then the kernel differences are significant enough not to be easily 
enumerated or replaced. How do their VMs compare to 2.6?

>  Also, highmem support in FC1 may be broken. 
That's not good, if it's true.

>  I'd suggest using Oprofile or readprofile to find where the
> kernel is spending
>  its time. 
Thanks! I'll try that.

[Dave Jones]

On Thu, 2004-01-08 at 22:05, Bevan C. Bennett wrote:

>   > Are you using a significant fraction of RAM?
>  Very much so. We're pushing our RAM very hard, and considering 
>  installing even more. 
How much RAM exactly ? It's possible the VM changes that went into
2.4.23 may help on big boxes. It is quite a radical amount of change
however (which is why it never made it into FC1 at all)

>   > RHEL and Fedora have a completely different VM.
>  Ah, then the kernel differences are significant enough not to be easily 
>  enumerated or replaced. How do their VMs compare to 2.6? 
FC1 - Stock 2.4 VM from 2.4.22 (Before the recent large -aa merge)
Taroon - Rik van Riel's rmap vm.
2.6 - Completely different again, but with some elements of rmap vm
(Ie the actual reverse mappings part).

>  > Also, highmem support in FC1 may be broken.
>  That's not good, if it's true. 
I would use 'suboptimal' rather than broken if its the issues I'm
thinking of.

	Dave

[Bevan C. Bennett]

Dave Jones wrote:
>  How much RAM exactly ? It's possible the VM changes that went
> into
>  2.4.23 may help on big boxes. It is quite a radical amount of change
>  however (which is why it never made it into FC1 at all) 
It's "only" 2GB of RAM, so it shouldn't involve the newer 
'big-big-bigmem' support code.

> > > RHEL and Fedora have a completely different VM.
> >Ah, then the kernel differences are significant enough not to be easily 
> >enumerated or replaced. How do their VMs compare to 2.6?
>  
>  FC1 - Stock 2.4 VM from 2.4.22 (Before the recent large -aa merge)
>  Taroon - Rik van Riel's rmap vm.
>  2.6 - Completely different again, but with some elements of rmap vm
>  (Ie the actual reverse mappings part). 
Thanks! That's extremely useful. Since 2.6 will eventually be making it 
into my favorite distributions, I'll see if I can get one installed for 
comparative testing.

Thanks again for the time and answers.
I'll try to continue to help others in return.

[Bill Rugolsky Jr.]

On Thu, Jan 08, 2004 at 11:33:47PM +0000, Dave Jones wrote:
>  > > Also, highmem support in FC1 may be broken.
>  > That's not good, if it's true.
>  
>  I would use 'suboptimal' rather than broken if its the issues I'm
>  thinking of. 
Yes, I should have said "sub-optimal under some circumstances."
Sorry Dave, no insult intended.

The great irony of the 2.4 series is that 2.4.early sucked in large
part because Linus didn't take Rik's patches, then he started a revolt by
integrating Andrea's VM.  And 2.4.late has sucked because until recently,
Marcelo hasn't integrated Andrea's patches.

I'm not convinced that removing rmap from the FC1 kernel was the right
decision. I suppose I'll have to do some patching and testing. :-/

	-Bill

[Dave Jones]

On Fri, 2004-01-09 at 00:02, Bill Rugolsky Jr. wrote:

>  I'm not convinced that removing rmap from the FC1 kernel was the
> right
>  decision. I suppose I'll have to do some patching and testing. :-/ 
the 'stay close to mainline' mantra of fc2 was originally going to be
the way to go for fc1 too, but things didnt work out that way.

The RHL9 kernel has a mismash of rmap patches + various bits that
folks did some of which went into later upstream rmap patches, some
didn't.  For a recent RHL7/8/9 update, I looked into updating the rmap
patch to something more recent from upstream. It was too painful for
words, and would've required a) lots of time to get right and
b) proper understanding of the issues in vm land.

The black art of vm hacking is something I try to avoid where possible,
so creating another forked vm for FC1 was something I really wanted
to avoid.  That said, the large number of VM updates in mainline 2.4
since 2.4.22 means that effectively FC1 has a forked VM right now.

I'm going to look into merging the vm changes for a future fc1 update.
Right now there still seem to be some niggles that need to be worked
out, but it's certainly no worse than whats in FC1 by all accounts.

I can't win..

	Dave

[Bill Rugolsky Jr.]

On Fri, Jan 09, 2004 at 02:14:29AM +0000, Dave Jones wrote:
>  the 'stay close to mainline' mantra of fc2 was originally
> going to be
>  the way to go for fc1 too, but things didnt work out that way. 
 
That would presuppose that 2.4 mainline was deployable for production use.
That hasn't been true for all of 2.4. The choice has largely been -ac or -aa.

Fortunately, deploying a (near) vanilla 2.6 kernel on common hardware
is beginning to look like a real possibility.

>  The RHL9 kernel has a mismash of rmap patches + various bits that
>  folks did some of which went into later upstream rmap patches, some
>  didn't.  For a recent RHL7/8/9 update, I looked into updating the rmap
>  patch to something more recent from upstream. It was too painful for
>  words, and would've required a) lots of time to get right and
>  b) proper understanding of the issues in vm land. 
 
Having looked at merging pieces of three or four trees  myself,
I feel your pain. :-p

>  I'm going to look into merging the vm changes for a future fc1
> update.
>  Right now there still seem to be some niggles that need to be worked
>  out, but it's certainly no worse than whats in FC1 by all accounts.
>  
>  I can't win.. 
 
Just one user's perspective, but at this point I'd suggest responding
to serious bug reports in the FC1 kernel, and concentrating effort on getting
2.6 into shape (forward porting fixes from 2.4, getting the less common
block drivers working, etc.). FC2 will be upon us soon. :-)

And thanks for all your hard work -- it is greatly appreciated.

	Bill Rugolsky

[Dave Jones]

On Fri, 2004-01-09 at 03:15, Bill Rugolsky Jr. wrote:

>  > I'm going to look into merging the vm changes for a future
> fc1 update.
>  > Right now there still seem to be some niggles that need to be worked
>  > out, but it's certainly no worse than whats in FC1 by all accounts.
>  > 
>  > I can't win..
>   
>  Just one user's perspective, but at this point I'd suggest responding
>  to serious bug reports in the FC1 kernel, and concentrating effort on getting
>  2.6 into shape (forward porting fixes from 2.4, getting the less common
>  block drivers working, etc.). FC2 will be upon us soon. :-) 
Thanks to the benefits of DSL disappearing for 5 hours last night, I
thought about this some more. The next thing in the pipeline after the
current stuff in -testing for FC1 is getting Tux enabled again, and
getting that well tested.  (There are a number of reasons for merging
this, including a migration point for RHL9 users).

Given that Tux might take 2-3 weeks of testing/devel, we'd be pushing
into February/March before I could get any of the vm updates into FC1,
at which point there's going to be a lot of time taken up getting FC2
into shape. Given the short life of FC1 from that point -> FC2 release
(and shortly afterwards, fc2 eol), the relative payoffs may not be worth
the time.  Some of the smaller changes to the mainline vm may be
candidates, but they'll likely all need rediffing thanks to various
patches we have which touch vm internals already (nptl, tux, execshield,
etc)

We'll see how things go.

	Dave

[Bevan C. Bennett]

Dave Jones wrote:

>  
>  FC1 - Stock 2.4 VM from 2.4.22 (Before the recent large -aa merge)
>  Taroon - Rik van Riel's rmap vm.
>  2.6 - Completely different again, but with some elements of rmap vm
>  (Ie the actual reverse mappings part). 
FWIW, I installed kernel-smp-2.6.0-1.30 and the test program ran quite 
well (not exhibiting the FC1 thrashing), so it's quite possibly rmap (or 
at least vm) related. Thanks again.

I'm trying to figure out a little bit of 2.6/NIC/kudzu weirdness that's 
preventing the new system from bringing up it's network without 
intervention, but that's for a different topic.

[Dave Jones]

On Thu, 2004-01-08 at 21:17, Bill Rugolsky Jr. wrote:
>  Also, highmem support in FC1 may be broken. 
That would be news to me. My 1.5GB desktop seems to behave fine.
What makes you think this is broken?

> 4GB has had very little (if any) testing however, (but should 
be
fine).

	Dave