Hi,
folks
We’ve
been really pleased with our 389 servers, which have been
successfully running as a multi-master pair in production
for 7 weeks, following (elapsed) months of development.
Unfortunately,
in the last few days their performance has radically
degraded to the point where they are becoming unusable due
to excessive memory consumption. At first, we suspected a
recent update to the package – but are no longer convinced
that’s the problem.
I’d
really appreciate any suggestions on how to troubleshoot
this further.
SYMPTOMS
389
ignores its 4GB userRoot
nsslapd-cachememsize and its overall memory usage
expands to encompass all of the server’s 8GB RAM – and a
large proportion of its 10GB swap. The service eventually
fails, and hangs the server.
DETAILS
As
per the previous guidance for RHEL 6 installations, we had
been using the packages from the COPRS repository (389-ds-base-1.2.11.32-1
aka
1.2.11.32 B2014.247.2316).
(As
several of our applications require CoS/dynamic attributes,
earlier versions of the RHEL packages were unusable for us
until this fix was ported:
https://fedorahosted.org/389/ticket/47762#comment:8
– which was only recently applied to the RHEL packages).
Due
to an unfortunate misconfiguration, the production servers
received automated updates – and were upgraded to RHEL 6.6,
the latest COPRS package (1.2.11.32-1), and a new kernel :-/
The
directory contains 236,340 entries, and
id2entry.db4 is 3.6GB.
STEPS
TAKEN
We’ve
tried to balance isolating the problem with maintaining a
critical service for our users:
·
An
additional 8GB RAM (and additional CPU) was added to the VM
to mitigate the immediate problem – but this was soon
swallowed up too.
·
Attempting
to reproduce this on our equivalent development servers has
not been successful, even when subjecting them to load.
·
Downgrading
to the RHEL packages (1.2.11.15-48
aka 1.2.11.15 B2014.300.2010) as per Noriko’s
very helpful procedure below (though I stopped the services
prior to the downgrade). No obvious difference.
·
Rebooting
into the previous kernel (2.6.32-431.29.2). This slowed the
problem, but only to an extent.
Not
attempted yet:
·
Turning
on debugging on the production servers, as they’re in a
fragile and sluggish state.
·
Building
the latest 1.3.x build on RHEL 6. Will be attempting this
soon…
·
Installing
RHEL 7.0 on a fresh VM, as we don’t have experience with 7
yet.
·
Downgrading
to the previous COPRS package, as no longer publicly
available(?)
Please
let me know if more information would help…