Re: locking performance and scalability (eye candy gnuplots inside!)

[Howard Chu]

>  The spinlocks of course just do their thing, and then we're much
> more
>  reliant upon the hardware to provide fairness, and as we see, this may
>  not be a safe bet.
>
>  We also see in the third graph and fifth graphs that the first core
>  performed badly compared to the others.  I do not know why, but it needs
>  explanation.
>
>  Relating to this though, although I don't have other plots, but I did do
>  quite a lot of benchmarking with the prototype benchmark app, and I
>  seemed to find on some machines that some cores *always* did badly, and
>  for *all* lock types.  I have a weak suspicion it reflects internal
>  memory access arrangements in the core itself.  However, this was on VMs
>  - although they were all dedicated hardware, so I had no neighbours on
>  the box - so it could just be Xen doing strange things. 
Even though it's a VM, numactl -H may still show something relevant.

BerkeleyDB did adaptive locking, using a spinlock before falling back to a 
heavier weight system mutex. In practice we always found that spinlocks are 
only a win within a single CPU socket; as soon as you have cross-socket 
contention they're horrible.

The other obvious contributor to system balance is interrupt steering. Most 
kernels seem to have irqbalance working by default these days, but you may 
still wind up with a particular core getting more than a fair share of 
interrupts sent to it. Again, not something you can directly tweak inside a VM 
with any meaningful effect. Aside from that, it's not always beneficial to do 
what irqbalance does. Sometimes you get much better system throughput by 
sacrificing a core, sending all interrupts to it, leaving all the remaining 
cores for the application. Fewer context switches on the remaining cores, 
which leads to much higher efficiency.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

locking performance and scalability (eye candy gnuplots inside!)

[liblfds admin]

On 08/03/16 22:35, Howard Chu wrote:
>  Even though it's a VM, numactl -H may still show something
> relevant. 
I'll try it next time I have one running.

>  BerkeleyDB did adaptive locking, using a spinlock before falling back
> to
>  a heavier weight system mutex. 
I believe the Windows Critical Section lock works in this way.

>  In practice we always found that
>  spinlocks are only a win within a single CPU socket; as soon as you have
>  cross-socket contention they're horrible. 
The benchmarks I've performed so far finds the same result.

>  The other obvious contributor to system balance is interrupt
> steering.
>  Most kernels seem to have irqbalance working by default these days, but
>  you may still wind up with a particular core getting more than a fair
>  share of interrupts sent to it. 
Would there be so many on a system which is doing nothing except running 
the benchmark, that it would significantly skew the results?  Each 
benchmark (i.e. each individual chart) ran here for five seconds.

One other matter I omitted from the original post - NUMA.

The systems tested are both single NUMA node.  The benchmark is going to 
be NUMA aware, but NUMA doesn't quite make sense - typically if I am 
writing an application or server, I try to treat each NUMA node as if it 
were a separate PC, i.e. one process per NUMA node.  As such, 
benchmarking needs to run at least twice, one on a single NUMA node, the 
second time on all nodes, with striped allocations.

locking performance and scalability (eye candy gnuplots inside!)

[Rich Megginson]

On 03/08/2016 03:36 PM, liblfds admin wrote:
>  On 08/03/16 22:35, Howard Chu wrote:
> > Even though it's a VM, numactl -H may still show something relevant.
>
>  I'll try it next time I have one running.
>
> > BerkeleyDB did adaptive locking, using a spinlock before falling back to
> > a heavier weight system mutex.
>
>  I believe the Windows Critical Section lock works in this way.
>
> > In practice we always found that
> > spinlocks are only a win within a single CPU socket; as soon as you have
> > cross-socket contention they're horrible.
>
>  The benchmarks I've performed so far finds the same result.
>
> > The other obvious contributor to system balance is interrupt steering.
> > Most kernels seem to have irqbalance working by default these days, but
> > you may still wind up with a particular core getting more than a fair
> > share of interrupts sent to it.
>
>  Would there be so many on a system which is doing nothing except 
>  running the benchmark, that it would significantly skew the results?  
>  Each benchmark (i.e. each individual chart) ran here for five seconds.
>
>  One other matter I omitted from the original post - NUMA.
>
>  The systems tested are both single NUMA node.  The benchmark is going 
>  to be NUMA aware, but NUMA doesn't quite make sense - typically if I 
>  am writing an application or server, I try to treat each NUMA node as 
>  if it were a separate PC, i.e. one process per NUMA node.  As such, 
>  benchmarking needs to run at least twice, one on a single NUMA node, 
>  the second time on all nodes, with striped allocations. 
I created a ticket for this to try to capture this discussion: 
https://fedorahosted.org/389/ticket/48773

>  -- 
>  389 users mailing list
>  389-users@%(host_name)s
>  http://lists.fedoraproject.org/admin/lists/[e-mail]

locking performance and scalability (eye candy gnuplots inside!)

[liblfds admin]

On 22/03/16 15:58, Rich Megginson wrote:
>  I created a ticket for this to try to capture this discussion:
>  https://fedorahosted.org/389/ticket/48773 
Timing! - I've literally 30 mins ago finally got to the point where the 
proper benchmark app is complete enough to be working, and it includes 
rwlocks.  I've just run a first test - debug code, single thread, 
rwlocks used write-only run at about half the speed of a plain mutex.

locking performance and scalability (eye candy gnuplots inside!)

[liblfds admin]

BTree benchmark gnuplot.

There is a balanced btree in the lock-free literature - but this btree 
is something I threw together off the top of my head a little while ago 
just to get something tree-like out there; it's add-only, and unbalanced.

http://www.liblfds.org/liblfds710_BTree.png

This is a *preliminary* result.  I've literally just got the gnuplot 
output working, and the benchmark is only a few days old - my concern is 
that the locking benchmarks may be underplaying their performance - I 
may have them performing four locks per iteration when they should only 
be performing two - but I need to think more about how the benchmark 
works, whether or not what they're doing now is fair; it's not obviously 
clear.

Either way I think it won't matter - you can see in the final chart just 
how poorly locking scales compared to non-locking - even if locking 
doubled in performance, they'd all still be half the speed of lock-free.

This of course is not THE btree, the balanced lock-free normal 
add-and-delete btree, but it's hopefully gives a good indication of what 
general performance can be expected.

Having said all that, a lock-free hash, a real one, ought to scale 
pretty much linearly, so it'll put the btree in the shade =-)