389 DS memory growth

[Nazarenko, Alexander]

Hello colleagues,
On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9.  We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update.
Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.

Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas.  Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.

Currently we use system controls to restrict the memory usage.

My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation?
Thanks,
- Alex

[Harald Strack]

Hi,

we experience still the same Problems. The OOM Killer kills the ns-slapd 
process regulary.

Versions (CentOS 7)

#  rpm -qa | grep 389
389-adminutil-1.1.22-2.el7.x86_64
389-ds-base-libs-1.3.11.1-3.el7_9.x86_64
389-ds-base-devel-1.3.11.1-3.el7_9.x86_64
389-ds-base-1.3.11.1-3.el7_9.x86_64
389-ds-base-snmp-1.3.11.1-3.el7_9.x86_64
389-ds-base-1.3.10.2-8.el7_9.x86_64
389-ds-base-devel-1.3.10.2-8.el7_9.x86_64
389-admin-1.1.46-4.el7.x86_64
389-adminutil-devel-1.1.22-2.el7.x86_64
389-ds-base-libs-1.3.10.2-8.el7_9.x86_64

# uname -r
3.10.0-1160.11.1.el7.x86_64

The behaviour may be related to unindexed searches, since we have 
another node that has nearly no unindexed searches and behaves more or 
less as usual.

br

Harald

On 17.04.23 03:07, Nazarenko, Alexander wrote:
> Hello colleagues,
>
> On March 22nd we updated the 389-ds-base.x86_64 and 
> 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production 
> servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9.  
> We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to 
> kernel-3.10.0-1160.88.1.el7.x86_64 during the same update.
>
> Approximately 12 days later, on April 3rd, all the hosts started 
> exhibiting memory growth issues whereby the “slapd” process was using 
> over 90% of the available system memory of 32GB, which was NOT 
> happening for a couple of years prior to applying any of the available 
> package updates on the systems.
>
> Two of the eight hosts act as Primaries (formerly referred to as 
> masters), while 6 of the hosts act as read-only replicas. Three of the 
> read-only replicas are used by our authorization system while the 
> other three read-only replicas are used by customer-based applications.
>
> Currently we use system controls to restrict the memory usage.
>
> My question is whether this is something that other users also 
> experience, and what is the recommended way to stabilize the DS 
> servers in this type of situation?
>
> Thanks,
> Alex
>
> 389-devel mailing list [e-mail]
> To unsubscribe send an email [e-mail]
> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue

[Thierry Bordaz]

Hi,

389 1.3.11-1-3 is the latest version on that branch and is quite widely 
installed. So far we are not informed of memory leaks on that release. 
If cache tuning are reasonable vs available memory and you suspect a 
leak, I would suggest to use valgrind and the suspected unindexed search.

best regards
thierry

On 11/13/23 12:51, Harald Strack wrote:
> Hi,
>
> we experience still the same Problems. The OOM Killer kills the 
> ns-slapd process regulary.
>
> Versions (CentOS 7)
>
> #  rpm -qa | grep 389
> 389-adminutil-1.1.22-2.el7.x86_64
> 389-ds-base-libs-1.3.11.1-3.el7_9.x86_64
> 389-ds-base-devel-1.3.11.1-3.el7_9.x86_64
> 389-ds-base-1.3.11.1-3.el7_9.x86_64
> 389-ds-base-snmp-1.3.11.1-3.el7_9.x86_64
> 389-ds-base-1.3.10.2-8.el7_9.x86_64
> 389-ds-base-devel-1.3.10.2-8.el7_9.x86_64
> 389-admin-1.1.46-4.el7.x86_64
> 389-adminutil-devel-1.1.22-2.el7.x86_64
> 389-ds-base-libs-1.3.10.2-8.el7_9.x86_64
>
> # uname -r
> 3.10.0-1160.11.1.el7.x86_64
>
> The behaviour may be related to unindexed searches, since we have 
> another node that has nearly no unindexed searches and behaves more or 
> less as usual.
>
> br
>
> Harald
>
> On 17.04.23 03:07, Nazarenko, Alexander wrote:
> > Hello colleagues,
> >
> > On March 22nd we updated the 389-ds-base.x86_64 and 
> > 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production 
> > servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9.  
> > We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to 
> > kernel-3.10.0-1160.88.1.el7.x86_64 during the same update.
> >
> > Approximately 12 days later, on April 3rd, all the hosts started 
> > exhibiting memory growth issues whereby the “slapd” process was using 
> > over 90% of the available system memory of 32GB, which was NOT 
> > happening for a couple of years prior to applying any of the 
> > available package updates on the systems.
> >
> > Two of the eight hosts act as Primaries (formerly referred to as 
> > masters), while 6 of the hosts act as read-only replicas.  Three of 
> > the read-only replicas are used by our authorization system while the 
> > other three read-only replicas are used by customer-based applications.
> >
> > Currently we use system controls to restrict the memory usage.
> >
> > My question is whether this is something that other users also 
> > experience, and what is the recommended way to stabilize the DS 
> > servers in this type of situation?
> >
> > Thanks,
> > Alex
> >
> > 389-devel mailing list [e-mail]
> > To unsubscribe send an email [e-mail]
> > Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> > Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
>
> -- 
> Harald Strack
> Geschäftsführer
>
> ssystems GmbH
> Kastanienallee 74
> 10435 Berlin
>
> 389-devel mailing list [e-mail]
> To unsubscribe send an email [e-mail]
> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue

[Harald Strack]

Hi,

caches where in autosizing mode, this seems not to work in our case (?)

grep nsslapd-cache-autosize /etc/dirsrv/slapd-ldap1/dse.ldif
nsslapd-cache-autosize: 10
nsslapd-cache-autosize-split: 40

Only when we set

nsslapd-cache-autosize: 0

and reduce all cashes by a factor of about 10 and 100 (see below):

@@ -1521,11 +1521,11 @@
  numSubordinates: 4
  nsslapd-suffix: cn=changelog
  nsslapd-cachesize: -1
-nsslapd-cachememsize: 671088640
+nsslapd-cachememsize: 67108864
  nsslapd-readonly: off
  nsslapd-require-index: off
  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/changelog
-nsslapd-dncachememsize: 134217728
+nsslapd-dncachememsize: 13421772

dn: cn=config,cn=ldbm database,cn=plugins,cn=config
  objectClass: top
@@ -1539,7 +1539,7 @@
  nsslapd-mode: 600
  nsslapd-idlistscanlimit: 4000
  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db
-nsslapd-dbcachesize: 1342657822
+nsslapd-dbcachesize: 13426578
  nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-ldap1/db
  nsslapd-db-durable-transaction: on
  nsslapd-db-transaction-wait: off
@@ -1586,7 +1586,7 @@
  numSubordinates: 4
  nsslapd-suffix: o=netscaperoot
  nsslapd-cachesize: -1
-nsslapd-cachememsize: 671088640
+nsslapd-cachememsize: 67108864
  nsslapd-readonly: off
  nsslapd-require-index: off
  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/NetscapeRoot
@@ -1600,11 +1600,11 @@
  numSubordinates: 10
  nsslapd-suffix: dc=fhm,dc=edu
  nsslapd-cachesize: -1
-nsslapd-cachememsize: 671088640
+nsslapd-cachememsize: 67108864
  nsslapd-readonly: off
  nsslapd-require-index: off
  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/userRoot
-nsslapd-dncachememsize: 134217728
+nsslapd-dncachememsize: 13421772

the memory consumption keeps stable. The machine has 32 GB memory, so 
the values prefixed with a minus above should easyly fit.

On 13.11.23 13:22, Thierry Bordaz wrote:
> Hi,
>
> 389 1.3.11-1-3 is the latest version on that branch and is quite 
> widely installed. So far we are not informed of memory leaks on that 
> release. If cache tuning are reasonable vs available memory and you 
> suspect a leak, I would suggest to use valgrind and the suspected 
> unindexed search.
>
> best regards
> thierry

[Thierry Bordaz]

So previous tuning was

* dbcache 1.3Gb
  * 3 backends with 130Mb

How large was it ? How many nsslapd-threadnumber ?

if with caches reduced by 10 times (130Mb, 3* 13Mb) it remains stable 
that means there are no leak. But caches are very small. What is the 
available memory  on the box ?

best regards
Theirry

On 11/16/23 20:19, Harald Strack wrote:
> Hi,
>
> caches where in autosizing mode, this seems not to work in our case (?)
>
> grep nsslapd-cache-autosize /etc/dirsrv/slapd-ldap1/dse.ldif
> nsslapd-cache-autosize: 10
> nsslapd-cache-autosize-split: 40
>
> Only when we set
>
> nsslapd-cache-autosize: 0
>
> and reduce all cashes by a factor of about 10 and 100 (see below):
>
> @@ -1521,11 +1521,11 @@
>  numSubordinates: 4
>  nsslapd-suffix: cn=changelog
>  nsslapd-cachesize: -1
> -nsslapd-cachememsize: 671088640
> +nsslapd-cachememsize: 67108864
>  nsslapd-readonly: off
>  nsslapd-require-index: off
>  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/changelog
> -nsslapd-dncachememsize: 134217728
> +nsslapd-dncachememsize: 13421772
>
> dn: cn=config,cn=ldbm database,cn=plugins,cn=config
>  objectClass: top
> @@ -1539,7 +1539,7 @@
>  nsslapd-mode: 600
>  nsslapd-idlistscanlimit: 4000
>  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db
> -nsslapd-dbcachesize: 1342657822
> +nsslapd-dbcachesize: 13426578
>  nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-ldap1/db
>  nsslapd-db-durable-transaction: on
>  nsslapd-db-transaction-wait: off
> @@ -1586,7 +1586,7 @@
>  numSubordinates: 4
>  nsslapd-suffix: o=netscaperoot
>  nsslapd-cachesize: -1
> -nsslapd-cachememsize: 671088640
> +nsslapd-cachememsize: 67108864
>  nsslapd-readonly: off
>  nsslapd-require-index: off
>  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/NetscapeRoot
> @@ -1600,11 +1600,11 @@
>  numSubordinates: 10
>  nsslapd-suffix: dc=fhm,dc=edu
>  nsslapd-cachesize: -1
> -nsslapd-cachememsize: 671088640
> +nsslapd-cachememsize: 67108864
>  nsslapd-readonly: off
>  nsslapd-require-index: off
>  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/userRoot
> -nsslapd-dncachememsize: 134217728
> +nsslapd-dncachememsize: 13421772
>
> the memory consumption keeps stable. The machine has 32 GB memory, so 
> the values prefixed with a minus above should easyly fit.
>
> On 13.11.23 13:22, Thierry Bordaz wrote:
> > Hi,
> >
> > 389 1.3.11-1-3 is the latest version on that branch and is quite 
> > widely installed. So far we are not informed of memory leaks on that 
> > release. If cache tuning are reasonable vs available memory and you 
> > suspect a leak, I would suggest to use valgrind and the suspected 
> > unindexed search.
> >
> > best regards
> > thierry
>
> -- 
> Harald Strack
> Geschäftsführer
>
> ssystems GmbH
> Kastanienallee 74
> 10435 Berlin
>
> --
> _______________________________________________
> 389-devel mailing list [e-mail]
> To unsubscribe send an email [e-mail]
> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue

[Harald Strack]

Maybe it is related to our other Problem "[389-devel] Re: Locking 
Problem when deleting in parallel" in a way that IO Bottlenecks causing 
the caching to malloc and retry over and over? Just a theory...

On 17.11.23 11:56, Thierry Bordaz wrote:
> So previous tuning was
> dbcache 1.3Gb
> 3 backends with 130Mb
>
> How large was it ? How many nsslapd-threadnumber ?

nsslapd-threadnumber: 128
> if with caches reduced by 10 times (130Mb, 3* 13Mb) it remains stable 
> that means there are no leak. But caches are very small. What is the 
> available memory  on the box ?

The machine has 32 GB memory, so the values prefixed with a minus above 
should easyly fit.(see below)
> best regards
> Theirry
>
> On 11/16/23 20:19, Harald Strack wrote:
> > Hi,
> >
> > caches where in autosizing mode, this seems not to work in our case (?)
> >
> > grep nsslapd-cache-autosize /etc/dirsrv/slapd-ldap1/dse.ldif
> > nsslapd-cache-autosize: 10
> > nsslapd-cache-autosize-split: 40
> >
> > Only when we set
> >
> > nsslapd-cache-autosize: 0
> >
> > and reduce all cashes by a factor of about 10 and 100 (see below):
> >
> > @@ -1521,11 +1521,11 @@
> >  numSubordinates: 4
> >  nsslapd-suffix: cn=changelog
> >  nsslapd-cachesize: -1
> > -nsslapd-cachememsize: 671088640
> > +nsslapd-cachememsize: 67108864
> >  nsslapd-readonly: off
> >  nsslapd-require-index: off
> >  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/changelog
> > -nsslapd-dncachememsize: 134217728
> > +nsslapd-dncachememsize: 13421772
> >
> > dn: cn=config,cn=ldbm database,cn=plugins,cn=config
> >  objectClass: top
> > @@ -1539,7 +1539,7 @@
> >  nsslapd-mode: 600
> >  nsslapd-idlistscanlimit: 4000
> >  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db
> > -nsslapd-dbcachesize: 1342657822
> > +nsslapd-dbcachesize: 13426578
> >  nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-ldap1/db
> >  nsslapd-db-durable-transaction: on
> >  nsslapd-db-transaction-wait: off
> > @@ -1586,7 +1586,7 @@
> >  numSubordinates: 4
> >  nsslapd-suffix: o=netscaperoot
> >  nsslapd-cachesize: -1
> > -nsslapd-cachememsize: 671088640
> > +nsslapd-cachememsize: 67108864
> >  nsslapd-readonly: off
> >  nsslapd-require-index: off
> >  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/NetscapeRoot
> > @@ -1600,11 +1600,11 @@
> >  numSubordinates: 10
> >  nsslapd-suffix: dc=fhm,dc=edu
> >  nsslapd-cachesize: -1
> > -nsslapd-cachememsize: 671088640
> > +nsslapd-cachememsize: 67108864
> >  nsslapd-readonly: off
> >  nsslapd-require-index: off
> >  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/userRoot
> > -nsslapd-dncachememsize: 134217728
> > +nsslapd-dncachememsize: 13421772
> >
> > the memory consumption keeps stable. The machine has 32 GB memory, so 
> > the values prefixed with a minus above should easyly fit.
> >
> > On 13.11.23 13:22, Thierry Bordaz wrote:
> > > Hi,
> > >
> > > 389 1.3.11-1-3 is the latest version on that branch and is quite 
> > > widely installed. So far we are not informed of memory leaks on that 
> > > release. If cache tuning are reasonable vs available memory and you 
> > > suspect a leak, I would suggest to use valgrind and the suspected 
> > > unindexed search.
> > >
> > > best regards
> > > thierry
> >
> > -- 
> > Harald Strack
> > Geschäftsführer
> >
> > ssystems GmbH
> > Kastanienallee 74
> > 10435 Berlin
> >
> > --
> > _______________________________________________
> > 389-devel mailing list [e-mail]
> > To unsubscribe send an email [e-mail]
> > Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> > Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
>
> --
> _______________________________________________
> 389-devel mailing list [e-mail]
> To unsubscribe send an email [e-mail]
> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue

[Thierry Bordaz]

On 11/17/23 15:46, Harald Strack wrote:
> Maybe it is related to our other Problem "[389-devel] Re: Locking 
> Problem when deleting in parallel" in a way that IO Bottlenecks 
> causing the caching to malloc and retry over and over? Just a theory...

Yes possibly
> On 17.11.23 11:56, Thierry Bordaz wrote:
> > So previous tuning was
> > dbcache 1.3Gb
> > 3 backends with 130Mb
> >
> > How large was it ? How many nsslapd-threadnumber ?
>
> nsslapd-threadnumber: 128

It looks a bit high to me. Also each thread will consume a minimal 
amount for memory, so the more threads/workers the bigger memory 
footprint. A good fit would be close to the number of core-thread (e.g. 
16). But it depends on the workload, if you have many long request and 
you are fear to exhaust the number of workers the 128 looks good.
> > if with caches reduced by 10 times (130Mb, 3* 13Mb) it remains stable 
> > that means there are no leak. But caches are very small. What is the 
> > available memory  on the box ?
>
> The machine has 32 GB memory, so the values prefixed with a minus 
> above should easyly fit.(see below)
> > best regards
> > Theirry
> >
> > On 11/16/23 20:19, Harald Strack wrote:
> > > Hi,
> > >
> > > caches where in autosizing mode, this seems not to work in our case (?)
> > >
> > > grep nsslapd-cache-autosize /etc/dirsrv/slapd-ldap1/dse.ldif
> > > nsslapd-cache-autosize: 10
> > > nsslapd-cache-autosize-split: 40
> > >
> > > Only when we set
> > >
> > > nsslapd-cache-autosize: 0
> > >
> > > and reduce all cashes by a factor of about 10 and 100 (see below):
> > >
> > > @@ -1521,11 +1521,11 @@
> > >  numSubordinates: 4
> > >  nsslapd-suffix: cn=changelog
> > >  nsslapd-cachesize: -1
> > > -nsslapd-cachememsize: 671088640
> > > +nsslapd-cachememsize: 67108864
> > >  nsslapd-readonly: off
> > >  nsslapd-require-index: off
> > >  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/changelog
> > > -nsslapd-dncachememsize: 134217728
> > > +nsslapd-dncachememsize: 13421772
> > >
> > > dn: cn=config,cn=ldbm database,cn=plugins,cn=config
> > >  objectClass: top
> > > @@ -1539,7 +1539,7 @@
> > >  nsslapd-mode: 600
> > >  nsslapd-idlistscanlimit: 4000
> > >  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db
> > > -nsslapd-dbcachesize: 1342657822
> > > +nsslapd-dbcachesize: 13426578
> > >  nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-ldap1/db
> > >  nsslapd-db-durable-transaction: on
> > >  nsslapd-db-transaction-wait: off
> > > @@ -1586,7 +1586,7 @@
> > >  numSubordinates: 4
> > >  nsslapd-suffix: o=netscaperoot
> > >  nsslapd-cachesize: -1
> > > -nsslapd-cachememsize: 671088640
> > > +nsslapd-cachememsize: 67108864
> > >  nsslapd-readonly: off
> > >  nsslapd-require-index: off
> > >  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/NetscapeRoot
> > > @@ -1600,11 +1600,11 @@
> > >  numSubordinates: 10
> > >  nsslapd-suffix: dc=fhm,dc=edu
> > >  nsslapd-cachesize: -1
> > > -nsslapd-cachememsize: 671088640
> > > +nsslapd-cachememsize: 67108864
> > >  nsslapd-readonly: off
> > >  nsslapd-require-index: off
> > >  nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/userRoot
> > > -nsslapd-dncachememsize: 134217728
> > > +nsslapd-dncachememsize: 13421772
> > >
> > > the memory consumption keeps stable. The machine has 32 GB memory, 
> > > so the values prefixed with a minus above should easyly fit.
> > >
> > > On 13.11.23 13:22, Thierry Bordaz wrote:
> > > > Hi,
> > > >
> > > > 389 1.3.11-1-3 is the latest version on that branch and is quite 
> > > > widely installed. So far we are not informed of memory leaks on 
> > > > that release. If cache tuning are reasonable vs available memory 
> > > > and you suspect a leak, I would suggest to use valgrind and the 
> > > > suspected unindexed search.
> > > >
> > > > best regards
> > > > thierry
> > >
> > > -- 
> > > Harald Strack
> > > Geschäftsführer
> > >
> > > ssystems GmbH
> > > Kastanienallee 74
> > > 10435 Berlin
> > >
> > > --
> > > _______________________________________________
> > > 389-devel mailing list [e-mail]
> > > To unsubscribe send an email [e-mail]
> > > Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> > > Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
> >
> > --
> > _______________________________________________
> > 389-devel mailing list [e-mail]
> > To unsubscribe send an email [e-mail]
> > Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> > Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
>
> -- 
> Harald Strack
> Geschäftsführer
>
> ssystems GmbH
> Kastanienallee 74
> 10435 Berlin
>
> --
> _______________________________________________
> 389-devel mailing list [e-mail]
> To unsubscribe send an email [e-mail]
> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]....
> Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue