Fedora 31 Self-Contained Change proposal: Ship BerkleyDB backend as a module

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/OpenLDAPwithBerkleyDBasModule

== Summary ==
Change the ''openldap-servers'' package so that BDB and HDB backends
are required to be dynamically loaded.

== Owner ==
* Name: [[User:mhonek| Matus Honek]]
* Email: mhonek (at) redhat (dot) com

== Detailed Description ==

So far the BDB and HDB were statically built with the ''slapd'' binary
and merely declaring `database bdb` or `database hdb` would just work.
Change introduces an additional requirement of explicitly declaring to
load the backend's SO file according to the documentation of dynamic
modules. The respective new modules will be shipped similarly to the
rest of the already shipped modules.

This change is directed to conduct a smoother experience before the
backends are removed in a next Fedora release.

== Benefit to Fedora ==

A step on a way to remove unsupported (both by OpenLDAP and BerkleyDB
upstream) piece of software.

== Scope ==
* Proposal owners:
Change the SPEC file accordingly.
* Other developers: N/A
* Policies and guidelines: N/A
* Trademark approval: N/A (not needed for this Change)


== Upgrade/compatibility impact ==
Server using BDB or HDB backends without modified configuration would
fail to start. See ''User Experience'' section for more information.

== How To Test ==
N/A (not a System Wide Change)

== User Experience ==
If a user is using either BDB or HDB they have two options:
*  migrate to the fully supported MDB backend (preferred)
*  add a `moduleload` configuration declaration (discouraged)

=== Migrating to MDB ===
The steps required to migrate a database are following:
*  Stop the slapd server.
*  Export data to an LDIF file using ''slapcat''.
*  Change the server's configuration replacing the BDB/HDB sections
with its MDB counterparts.
*  Import data to a new database from the LDIF file using ''slapadd''.
*  Start the slapd server.

=== ModuleLoad the BDB/HDB backend ===
Depending on the configuration style and backend type, user should add
a declaration in order to load the backend library: add option
`moduleload` (slapd.conf(5), section ''GLOBAL CONFIGURATION OPTIONS'')
or attribute `olcModuleLoad` (slapd-config(5), section ''DYNAMIC
MODULE OPTIONS'') with value `back_bdb` and/or `back_hdb`.

== Dependencies ==
None.

== Contingency Plan ==
* Contingency mechanism: Revert the change.
* Contingency deadline: Anytime.
* Blocks release? No.
* Blocks product? None.

== Documentation ==
N/A (not a System Wide Change)

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Stephen Gallagher]

On Tue, Jul 23, 2019 at 8:45 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  https://fedoraproject.org/wiki/Changes/OpenLDAPwithBerkleyDBasModule
>
>  == Summary ==
>  Change the ''openldap-servers'' package so that BDB and HDB backends
>  are required to be dynamically loaded.
>
>  == Owner ==
>  * Name: [[User:mhonek| Matus Honek]]
>  * Email: mhonek (at) redhat (dot) com
>  
...

>  == Upgrade/compatibility impact ==
>  Server using BDB or HDB backends without modified configuration would
>  fail to start. See ''User Experience'' section for more information.
>  
Is this avoidable or do you consider it desirable? Is there any harm
in shipping a default configuration that does the loadmodule for both
deprecated backends?

My guess here is that you want this to be an explicit breakage to help
users learn that the backend is no longer supported. If that's the
case, I'd like to see that spelled out in the Change.

Do any tools exist to simplify the conversion to MDB? Can this be automated?

[Jason L Tibbitts III]

> >>>> "SG" == Stephen Gallagher
> <sgallagh(a)redhat.com&gt; writes: 
SG> Do any tools exist to simplify the conversion to MDB? Can this be
SG> automated?

I'd like to know this as well.  It's always better to provide tools or
extremely clear and detailed instructions, because it's not safe to
assume that people know how to do this kind of thing even if they are
running an LDAP server.

I don't particularly have a problem with being forced to enable some
compatibility option after an OS upgrade in advance of functionality
actually being deprecated and not able to be used at all.  The latter
gets you into a pretty bad "no way out" situation.  ("You should have
read the release notes!  Hope you have backups.")  It's still better to
get this kind of thing documented as well as possible.
 
 - J<

[Matus Honek]

On Wed, Jul 31, 2019 at 4:57 PM Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:
>  On Tue, Jul 23, 2019 at 8:45 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  >
>  > https://fedoraproject.org/wiki/Changes/OpenLDAPwithBerkleyDBasModule
>  >
>  > == Summary ==
>  > Change the ''openldap-servers'' package so that BDB and HDB
> backends
>  > are required to be dynamically loaded.
>  >
>  > == Owner ==
>  > * Name: [[User:mhonek| Matus Honek]]
>  > * Email: mhonek (at) redhat (dot) com
>  >
>
>  ...
>
>  > == Upgrade/compatibility impact ==
>  > Server using BDB or HDB backends without modified configuration would
>  > fail to start. See ''User Experience'' section for more
> information.
>  >
>
>  Is this avoidable or do you consider it desirable? Is there any harm
>  in shipping a default configuration that does the loadmodule for both
>  deprecated backends? 
The default configuration is shipped only when the package is newly
installed. Given the BDB/HDB should not be used there is no point in
pre-loading these in new instalments.

>  My guess here is that you want this to be an explicit breakage to help
>  users learn that the backend is no longer supported. If that's the
>  case, I'd like to see that spelled out in the Change. 
Correct. I thought this was put clearly but I sure can improve this.
Would be spelling this in Detailed Description section acceptable?

>  Do any tools exist to simplify the conversion to MDB? Can this be automated? 
I am not aware of any such tools. Numerous demands on upstream were
always responded with more-or-less what I put in the section
describing the conversion. If any errors are encountered during the
process they need to be understood and then fixed manually.

Automatic conversion would be far from trivial (and possibly
load-intensive depending on sizes). There are two types of
configuration (plaintext and "the more complicated one"). A server may
have several instances of backends. Each backend type has its own
configuration specifics (MDB being easier but still a "human guess"
should be made to configure the expected size of it). Not rarely users
put configuration into various locations which are not guessable
automatically at all. Additionally, users should ideally see the
possible errors during the process of conversion and verify the data
has not broken.

One relatively simple thing to provide would be a pre-run script that
would disallow running the slapd's systemd service if the
configuration contains BDB/HDB instances and the moduleload is not
present, writing into journal clearly what happened and what needs to
be done.

This is not to give reasons not to automate the process, rather
explain that understanding and thought needs to go into each
instalment's conversion(s).
-- 
Matúš Honěk
Software Engineer
Red Hat Czech

[Stephen Gallagher]

On Thu, Aug 1, 2019 at 6:57 AM Matus Honek <mhonek(a)redhat.com&gt; wrote:
>  On Wed, Jul 31, 2019 at 4:57 PM Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:
>  >
>  > On Tue, Jul 23, 2019 at 8:45 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  > >
>  > > https://fedoraproject.org/wiki/Changes/OpenLDAPwithBerkleyDBasModule
>  > >
>  > > == Summary ==
>  > > Change the ''openldap-servers'' package so that BDB and HDB
> backends
>  > > are required to be dynamically loaded.
>  > >
>  > > == Owner ==
>  > > * Name: [[User:mhonek| Matus Honek]]
>  > > * Email: mhonek (at) redhat (dot) com
>  > >
>  >
>  > ...
>  >
>  > > == Upgrade/compatibility impact ==
>  > > Server using BDB or HDB backends without modified configuration would
>  > > fail to start. See ''User Experience'' section for more
> information.
>  > >
>  >
>  > Is this avoidable or do you consider it desirable? Is there any harm
>  > in shipping a default configuration that does the loadmodule for both
>  > deprecated backends?
>
>  The default configuration is shipped only when the package is newly
>  installed. Given the BDB/HDB should not be used there is no point in
>  pre-loading these in new instalments.
>  
Understood.

>  >
>  > My guess here is that you want this to be an explicit breakage to help
>  > users learn that the backend is no longer supported. If that's the
>  > case, I'd like to see that spelled out in the Change.
>
>  Correct. I thought this was put clearly but I sure can improve this.
>  Would be spelling this in Detailed Description section acceptable? 
Yes, please.

>  >
>  > Do any tools exist to simplify the conversion to MDB? Can this be automated?
>
>  I am not aware of any such tools. Numerous demands on upstream were
>  always responded with more-or-less what I put in the section
>  describing the conversion. If any errors are encountered during the
>  process they need to be understood and then fixed manually.
>
>  Automatic conversion would be far from trivial (and possibly
>  load-intensive depending on sizes). There are two types of
>  configuration (plaintext and "the more complicated one"). A server may
>  have several instances of backends. Each backend type has its own
>  configuration specifics (MDB being easier but still a "human guess"
>  should be made to configure the expected size of it). Not rarely users
>  put configuration into various locations which are not guessable
>  automatically at all. Additionally, users should ideally see the
>  possible errors during the process of conversion and verify the data
>  has not broken.
>
>  One relatively simple thing to provide would be a pre-run script that
>  would disallow running the slapd's systemd service if the
>  configuration contains BDB/HDB instances and the moduleload is not
>  present, writing into journal clearly what happened and what needs to
>  be done.
>  
This would be highly desirable. It would go a long way towards helping
users understand the transition. Including a web link in the journal
message to a detailed conversion HOWTO would be ideal.

>  This is not to give reasons not to automate the process, rather
>  explain that understanding and thought needs to go into each
>  instalment's conversion(s). 
Sure, makes sense. Thank you for clarifying.

[Matus Honek]

Hello Stephen,

I've made changes according to your notes [1], please have a look. And
thanks for the feedback!

[1]
https://fedoraproject.org/w/index.php?title=Changes%2FOpenLDAPwithBerkley...

On Thu, Aug 1, 2019 at 1:49 PM Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:
>  On Thu, Aug 1, 2019 at 6:57 AM Matus Honek <mhonek(a)redhat.com&gt; wrote:
>  >
>  > On Wed, Jul 31, 2019 at 4:57 PM Stephen Gallagher <sgallagh(a)redhat.com&gt;
> wrote:
>  > >
>  > > On Tue, Jul 23, 2019 at 8:45 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  > > >
>  > > > https://fedoraproject.org/wiki/Changes/OpenLDAPwithBerkleyDBasModule
>  > > >
>  > > > == Summary ==
>  > > > Change the ''openldap-servers'' package so that BDB and
> HDB backends
>  > > > are required to be dynamically loaded.
>  > > >
>  > > > == Owner ==
>  > > > * Name: [[User:mhonek| Matus Honek]]
>  > > > * Email: mhonek (at) redhat (dot) com
>  > > >
>  > >
>  > > ...
>  > >
>  > > > == Upgrade/compatibility impact ==
>  > > > Server using BDB or HDB backends without modified configuration would
>  > > > fail to start. See ''User Experience'' section for more
> information.
>  > > >
>  > >
>  > > Is this avoidable or do you consider it desirable? Is there any harm
>  > > in shipping a default configuration that does the loadmodule for both
>  > > deprecated backends?
>  >
>  > The default configuration is shipped only when the package is newly
>  > installed. Given the BDB/HDB should not be used there is no point in
>  > pre-loading these in new instalments.
>  >
>
>  Understood.
>
>  > >
>  > > My guess here is that you want this to be an explicit breakage to help
>  > > users learn that the backend is no longer supported. If that's the
>  > > case, I'd like to see that spelled out in the Change.
>  >
>  > Correct. I thought this was put clearly but I sure can improve this.
>  > Would be spelling this in Detailed Description section acceptable?
>
>  Yes, please.
>
>  >
>  > >
>  > > Do any tools exist to simplify the conversion to MDB? Can this be automated?
>  >
>  > I am not aware of any such tools. Numerous demands on upstream were
>  > always responded with more-or-less what I put in the section
>  > describing the conversion. If any errors are encountered during the
>  > process they need to be understood and then fixed manually.
>  >
>  > Automatic conversion would be far from trivial (and possibly
>  > load-intensive depending on sizes). There are two types of
>  > configuration (plaintext and "the more complicated one"). A server may
>  > have several instances of backends. Each backend type has its own
>  > configuration specifics (MDB being easier but still a "human guess"
>  > should be made to configure the expected size of it). Not rarely users
>  > put configuration into various locations which are not guessable
>  > automatically at all. Additionally, users should ideally see the
>  > possible errors during the process of conversion and verify the data
>  > has not broken.
>  >
>  > One relatively simple thing to provide would be a pre-run script that
>  > would disallow running the slapd's systemd service if the
>  > configuration contains BDB/HDB instances and the moduleload is not
>  > present, writing into journal clearly what happened and what needs to
>  > be done.
>  >
>
>  This would be highly desirable. It would go a long way towards helping
>  users understand the transition. Including a web link in the journal
>  message to a detailed conversion HOWTO would be ideal.
>
>  > This is not to give reasons not to automate the process, rather
>  > explain that understanding and thought needs to go into each
>  > instalment's conversion(s).
>
>  Sure, makes sense. Thank you for clarifying. 


-- 
Matúš Honěk
Software Engineer
Red Hat Czech

[Stephen Gallagher]

On Tue, Aug 6, 2019 at 8:58 AM Matus Honek <mhonek(a)redhat.com&gt; wrote:

>  Hello Stephen,
>
>  I've made changes according to your notes [1], please have a look. And
>  thanks for the feedback!
>
>  [1]
>
> https://fedoraproject.org/w/index.php?title=Changes%2FOpenLDAPwithBerkley...
>
>  
Looks good. Thank you.