F35 Change: Switching Cyrus Sasl from BerkeleyDB to GDBM (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Change/CyrusSaslBerkeleyDBtoGdbm

== Summary ==
cyrus-sasl package was built with libdb requirement, now it is replaced by gdbm.

== Owner ==
* Name: [[User:Dbelyavs| Dmitry Belyavskiy]]
* Email: dbelyavs(a)redhat.com



== Detailed Description ==
This change switches the default backend Key-Value DB used by sasldb
plugin from BerkeleyDB to GDBM and provides a migration tool for
automatic conversion from old to new format.


== Benefit to Fedora ==
According to more restrictive libdb licence policy exists effort to
remove libdb's dependencies.
cyrus-sasl package can now be built without libdb requirement.


== Scope ==
* Proposal owners:
* Other developers: The owners of the packages depending on cyrus-sasl
sasldb plugin should provide the documentation about the migration
procedure.

* Release engineering: [https://pagure.io/releng/issue/10084]
* Policies and guidelines: not needed for this Change
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:


== Upgrade/compatibility impact ==
The migration script should be used to upgrade the particular
databases used by specific applications via sasldb plugin


== How To Test ==
* Install the new version of the cyrus-sasl.
* Use the migration tool bdb2current provided by the package to
migrate your sasldb file
* update the configuration file to point on the new sasldb file
* restart the application if necessary
* Check that auth is still working


== User Experience ==
not provided

== Dependencies ==
A lot of application use cyrus-sasl sasldb plugins. Their maintainers
were notified via email and some of them have responded.


== Contingency Plan ==
* Contingency mechanism: Revert the shipped configuration
* Contingency deadline: Beta freeze
* Blocks release? Yes


== Documentation ==
Here is the notification sent to known developers of the depending packages:

New version of the cyrus-sasl is planned to use the gdbm database for
the sasldb plugins.

I've implemented the patch
(https://src.fedoraproject.org/rpms/cyrus-sasl/pull-request/3#request_diff)
changing the default DB and implementing the migration tool to make
the switching from BerkeleyDB to GDBM seamless.

I kindly ask you to check the information in the following spreadsheet:
https://docs.google.com/spreadsheets/d/1z5eTSm3rtlKtEKPCxhI_wE861Xzg8kbvI...:

whether your package is affected by the proposed change
whether the migration tool is suitable for your purposes

and let me know or mark the results in the table


== Release Notes ==
a new version of the cyrus-sasl package is landing in rawhide.

This version changes the database used to store saslauthdb data. This
is part of the move to deprecate use of Berkley DB. The new package
will use GDBM instead.

We provided a tool to perform migrations for database should that be
needed by a package:

The syntax of the migration tool is
bdb2current <sasldb path> <new_path>

Please check whether your packages use the sasldb plugin and provide a
relevant migration guideline.



-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis

[Zbigniew Jędrzejewski-Szmek]

On Wed, Apr 07, 2021 at 04:32:29PM -0400, Ben Cotton wrote:
>  == Detailed Description ==
>  This change switches the default backend Key-Value DB used by sasldb
>  plugin from BerkeleyDB to GDBM and provides a migration tool for
>  automatic conversion from old to new format. 
To clarify: will both backends be available for some time, and is only
the default being switched? Or will there be a "flag day" where suddenly
there's a new default, and after the update, old format cannot be read
anymore?

Also, the planned timeline is unclear. When will the default be switched?
By when do dependent packages need to be prepared?

>  == User Experience ==
>  not provided
>  
>  == Dependencies ==
>  A lot of application use cyrus-sasl sasldb plugins. Their maintainers
>  were notified via email and some of them have responded.
>  
>  
>  == Contingency Plan ==
>  * Contingency mechanism: Revert the shipped configuration
>  * Contingency deadline: Beta freeze
>  * Blocks release? Yes 
OK, let's say we decide will not be ready for F35, and decide to
activate the contingency mechanism. What happens with packages which
were already released with an update script and have converted to the
new format?

I think 'beta freeze' is too late. Even if it is just a matter of reverting
the default and disabling the migration script and rebuilding the dependent
packages, this should be done *before* the beta freeze.
What about moving the activation of the contingency mechanism to
"updates-testing activation"?

>  == Documentation ==
>  https://docs.google.com/spreadsheets/d/1z5eTSm3rtlKtEKPCxhI_wE861Xzg8kbvI...
Please consider copying the list of packages with affected=Yes to the
change page at some point. This will be easier for users to see and available
in perpetuity.

qemu, libvirt, postfix, exim, subversion, openldap...
Quite a few important packages, it would be unfortunate to lose user
data for them.

Zbyszek

[David Cantrell]

>  https://fedoraproject.org/wiki/Change/CyrusSaslBerkeleyDBtoGdbm
>  
>  == Summary ==
>  cyrus-sasl package was built with libdb requirement, now it is replaced by gdbm.
>  
>  == Owner ==
>  * Name: [[User:Dbelyavs| Dmitry Belyavskiy]]
>  * Email: dbelyavs(a)redhat.com
>  
>  
>  
>  == Detailed Description ==
>  This change switches the default backend Key-Value DB used by sasldb
>  plugin from BerkeleyDB to GDBM and provides a migration tool for
>  automatic conversion from old to new format.
>  
>  
>  == Benefit to Fedora ==
>  According to more restrictive libdb licence policy exists effort to
>  remove libdb's dependencies.
>  cyrus-sasl package can now be built without libdb requirement.
>  
>  
>  == Scope ==
>  * Proposal owners:
>  * Other developers: The owners of the packages depending on cyrus-sasl
>  sasldb plugin should provide the documentation about the migration
>  procedure.
>  
>  * Release engineering: [https://pagure.io/releng/issue/10084]
>  * Policies and guidelines: not needed for this Change
>  * Trademark approval: N/A (not needed for this Change)
>  * Alignment with Objectives:
>  
>  
>  == Upgrade/compatibility impact ==
>  The migration script should be used to upgrade the particular
>  databases used by specific applications via sasldb plugin
>  
>  
>  == How To Test ==
>  * Install the new version of the cyrus-sasl.
>  * Use the migration tool bdb2current provided by the package to
>  migrate your sasldb file
>  * update the configuration file to point on the new sasldb file
>  * restart the application if necessary
>  * Check that auth is still working
>  
>  
>  == User Experience ==
>  not provided
>  
>  == Dependencies ==
>  A lot of application use cyrus-sasl sasldb plugins. Their maintainers
>  were notified via email and some of them have responded.
>  
>  
>  == Contingency Plan ==
>  * Contingency mechanism: Revert the shipped configuration
>  * Contingency deadline: Beta freeze
>  * Blocks release? Yes
>  
>  
>  == Documentation ==
>  Here is the notification sent to known developers of the depending packages:
>  
>  New version of the cyrus-sasl is planned to use the gdbm database for
>  the sasldb plugins.
>  
>  I've implemented the patch
>  (https://src.fedoraproject.org/rpms/cyrus-sasl/pull-request/3#request_diff)
>  changing the default DB and implementing the migration tool to make
>  the switching from BerkeleyDB to GDBM seamless.
>  
>  I kindly ask you to check the information in the following spreadsheet:
>  https://docs.google.com/spreadsheets/d/1z5eTSm3rtlKtEKPCxhI_wE861Xzg8kbvI...:
>  
>  whether your package is affected by the proposed change
>  whether the migration tool is suitable for your purposes
>  
>  and let me know or mark the results in the table
>  
>  
>  == Release Notes ==
>  a new version of the cyrus-sasl package is landing in rawhide.
>  
>  This version changes the database used to store saslauthdb data. This
>  is part of the move to deprecate use of Berkley DB. The new package
>  will use GDBM instead.
>  
>  We provided a tool to perform migrations for database should that be
>  needed by a package:
>  
>  The syntax of the migration tool is
>  bdb2current <sasldb path> <new_path>
>  
>  Please check whether your packages use the sasldb plugin and provide a
>  relevant migration guideline. 
A couple of questions:

1) The contingency plan does not make it clear how users revert if the BDB version of
Cyrus SASL has to return.  Does the migration tool leave a copy of the previous db file or
files in place that can be used by the BDB version of Cyrus SASL?  Should the new file be
migrated back in case of changes post-migration?

2) I'm curious why GDBM was chosen instead of something like sqlite.

Thanks,

[Gary Buhrmaster]

On Fri, Apr 16, 2021 at 4:31 PM David Cantrell <dcantrell(a)redhat.com&gt; wrote:

>  2) I'm curious why GDBM was chosen instead of something like
> sqlite. 
I believe sasldb only supports gdbm and
ndbm as alternatives to bdb.

[David Cantrell]

On Fri, Apr 16, 2021 at 04:49:46PM +0000, Gary Buhrmaster wrote:
> On Fri, Apr 16, 2021 at 4:31 PM David Cantrell
> <dcantrell(a)redhat.com&gt; wrote:
>
> > 2) I'm curious why GDBM was chosen instead of something like sqlite.
>
> I believe sasldb only supports gdbm and
> ndbm as alternatives to bdb. 
Ooops, my mistake.  I see:

--with-dblib={berkeley|gdbm|lmdb|ndbm|none|auto_detect}

So yeah.  I was mistaking cyrus-sasl for another
project-with-db-backend-of-your-choice and sqlite was an option.

Nevermind.

Thanks,

-- 
David Cantrell <dcantrell(a)redhat.com&gt;
Red Hat, Inc. | Boston, MA | EST5EDT

[Simo Sorce]

On Fri, 2021-04-16 at 16:31 +0000, David Cantrell wrote:
>  > https://fedoraproject.org/wiki/Change/CyrusSaslBerkeleyDBtoGdbm
>  > 
>  > == Summary ==
>  > cyrus-sasl package was built with libdb requirement, now it is replaced by gdbm.
>  > 
>  > == Owner ==
>  > * Name: [[User:Dbelyavs| Dmitry Belyavskiy]]
>  > * Email: dbelyavs(a)redhat.com
>  > 
>  > 
>  > 
>  > == Detailed Description ==
>  > This change switches the default backend Key-Value DB used by sasldb
>  > plugin from BerkeleyDB to GDBM and provides a migration tool for
>  > automatic conversion from old to new format.
>  > 
>  > 
>  > == Benefit to Fedora ==
>  > According to more restrictive libdb licence policy exists effort to
>  > remove libdb's dependencies.
>  > cyrus-sasl package can now be built without libdb requirement.
>  > 
>  > 
>  > == Scope ==
>  > * Proposal owners:
>  > * Other developers: The owners of the packages depending on cyrus-sasl
>  > sasldb plugin should provide the documentation about the migration
>  > procedure.
>  > 
>  > * Release engineering: [https://pagure.io/releng/issue/10084]
>  > * Policies and guidelines: not needed for this Change
>  > * Trademark approval: N/A (not needed for this Change)
>  > * Alignment with Objectives:
>  > 
>  > 
>  > == Upgrade/compatibility impact ==
>  > The migration script should be used to upgrade the particular
>  > databases used by specific applications via sasldb plugin
>  > 
>  > 
>  > == How To Test ==
>  > * Install the new version of the cyrus-sasl.
>  > * Use the migration tool bdb2current provided by the package to
>  > migrate your sasldb file
>  > * update the configuration file to point on the new sasldb file
>  > * restart the application if necessary
>  > * Check that auth is still working
>  > 
>  > 
>  > == User Experience ==
>  > not provided
>  > 
>  > == Dependencies ==
>  > A lot of application use cyrus-sasl sasldb plugins. Their maintainers
>  > were notified via email and some of them have responded.
>  > 
>  > 
>  > == Contingency Plan ==
>  > * Contingency mechanism: Revert the shipped configuration
>  > * Contingency deadline: Beta freeze
>  > * Blocks release? Yes
>  > 
>  > 
>  > == Documentation ==
>  > Here is the notification sent to known developers of the depending packages:
>  > 
>  > New version of the cyrus-sasl is planned to use the gdbm database for
>  > the sasldb plugins.
>  > 
>  > I've implemented the patch
>  > (https://src.fedoraproject.org/rpms/cyrus-sasl/pull-request/3#request_diff)
>  > changing the default DB and implementing the migration tool to make
>  > the switching from BerkeleyDB to GDBM seamless.
>  > 
>  > I kindly ask you to check the information in the following spreadsheet:
>  > https://docs.google.com/spreadsheets/d/1z5eTSm3rtlKtEKPCxhI_wE861Xzg8kbvI...:
>  > 
>  > whether your package is affected by the proposed change
>  > whether the migration tool is suitable for your purposes
>  > 
>  > and let me know or mark the results in the table
>  > 
>  > 
>  > == Release Notes ==
>  > a new version of the cyrus-sasl package is landing in rawhide.
>  > 
>  > This version changes the database used to store saslauthdb data. This
>  > is part of the move to deprecate use of Berkley DB. The new package
>  > will use GDBM instead.
>  > 
>  > We provided a tool to perform migrations for database should that be
>  > needed by a package:
>  > 
>  > The syntax of the migration tool is
>  > bdb2current <sasldb path> <new_path>
>  > 
>  > Please check whether your packages use the sasldb plugin and provide a
>  > relevant migration guideline.
>  
>  A couple of questions:
>  
>  1) The contingency plan does not make it clear how users revert if
>  the BDB version of Cyrus SASL has to return.  Does the migration tool
>  leave a copy of the previous db file or files in place that can be
>  used by the BDB version of Cyrus SASL?  Should the new file be
>  migrated back in case of changes post-migration? 
The migration tool creates a new file, so the previous DB should remain
available. We do not really have a way to convert back though,
modifying the migrations script to go ther way around should be
possible, but unclear if that is really required as you have a backup
of th user's db at time of migration.

>  2) I'm curious why GDBM was chosen instead of something like
> sqlite.
>  
>  Thanks,
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[David Cantrell]

On Fri, Apr 16, 2021 at 03:48:04PM -0400, Simo Sorce wrote:
> On Fri, 2021-04-16 at 16:31 +0000, David Cantrell wrote:
> > > https://fedoraproject.org/wiki/Change/CyrusSaslBerkeleyDBtoGdbm
> > >
> > > == Summary ==
> > > cyrus-sasl package was built with libdb requirement, now it is replaced by
> gdbm.
> > >
> > > == Owner ==
> > > * Name: [[User:Dbelyavs| Dmitry Belyavskiy]]
> > > * Email: dbelyavs(a)redhat.com
> > >
> > >
> > >
> > > == Detailed Description ==
> > > This change switches the default backend Key-Value DB used by sasldb
> > > plugin from BerkeleyDB to GDBM and provides a migration tool for
> > > automatic conversion from old to new format.
> > >
> > >
> > > == Benefit to Fedora ==
> > > According to more restrictive libdb licence policy exists effort to
> > > remove libdb's dependencies.
> > > cyrus-sasl package can now be built without libdb requirement.
> > >
> > >
> > > == Scope ==
> > > * Proposal owners:
> > > * Other developers: The owners of the packages depending on cyrus-sasl
> > > sasldb plugin should provide the documentation about the migration
> > > procedure.
> > >
> > > * Release engineering: [https://pagure.io/releng/issue/10084]
> > > * Policies and guidelines: not needed for this Change
> > > * Trademark approval: N/A (not needed for this Change)
> > > * Alignment with Objectives:
> > >
> > >
> > > == Upgrade/compatibility impact ==
> > > The migration script should be used to upgrade the particular
> > > databases used by specific applications via sasldb plugin
> > >
> > >
> > > == How To Test ==
> > > * Install the new version of the cyrus-sasl.
> > > * Use the migration tool bdb2current provided by the package to
> > > migrate your sasldb file
> > > * update the configuration file to point on the new sasldb file
> > > * restart the application if necessary
> > > * Check that auth is still working
> > >
> > >
> > > == User Experience ==
> > > not provided
> > >
> > > == Dependencies ==
> > > A lot of application use cyrus-sasl sasldb plugins. Their maintainers
> > > were notified via email and some of them have responded.
> > >
> > >
> > > == Contingency Plan ==
> > > * Contingency mechanism: Revert the shipped configuration
> > > * Contingency deadline: Beta freeze
> > > * Blocks release? Yes
> > >
> > >
> > > == Documentation ==
> > > Here is the notification sent to known developers of the depending packages:
> > >
> > > New version of the cyrus-sasl is planned to use the gdbm database for
> > > the sasldb plugins.
> > >
> > > I've implemented the patch
> > > (https://src.fedoraproject.org/rpms/cyrus-sasl/pull-request/3#request_diff)
> > > changing the default DB and implementing the migration tool to make
> > > the switching from BerkeleyDB to GDBM seamless.
> > >
> > > I kindly ask you to check the information in the following spreadsheet:
> > > https://docs.google.com/spreadsheets/d/1z5eTSm3rtlKtEKPCxhI_wE861Xzg8kbvI...:
> > >
> > > whether your package is affected by the proposed change
> > > whether the migration tool is suitable for your purposes
> > >
> > > and let me know or mark the results in the table
> > >
> > >
> > > == Release Notes ==
> > > a new version of the cyrus-sasl package is landing in rawhide.
> > >
> > > This version changes the database used to store saslauthdb data. This
> > > is part of the move to deprecate use of Berkley DB. The new package
> > > will use GDBM instead.
> > >
> > > We provided a tool to perform migrations for database should that be
> > > needed by a package:
> > >
> > > The syntax of the migration tool is
> > > bdb2current <sasldb path> <new_path>
> > >
> > > Please check whether your packages use the sasldb plugin and provide a
> > > relevant migration guideline.
> >
> > A couple of questions:
> >
> > 1) The contingency plan does not make it clear how users revert if
> > the BDB version of Cyrus SASL has to return.  Does the migration tool
> > leave a copy of the previous db file or files in place that can be
> > used by the BDB version of Cyrus SASL?  Should the new file be
> > migrated back in case of changes post-migration?
>
> The migration tool creates a new file, so the previous DB should remain
> available. We do not really have a way to convert back though,
> modifying the migrations script to go ther way around should be
> possible, but unclear if that is really required as you have a backup
> of th user's db at time of migration. 
I think this should be acceptable.  Leaving the previous DB should a
revert of the software be necessary leaves users with the ability to
run what they had set up.  I'd like to see that noted in the proposal.

Thanks,

> > 2) I'm curious why GDBM was chosen instead of something like
> sqlite.
> >
> > Thanks,
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
David Cantrell <dcantrell(a)redhat.com&gt;
Red Hat, Inc. | Boston, MA | EST5EDT