[NSS Default File Format SQL in Fedora 28]

[Alexander Bokovoy]

Hi,

FYI.

We are going to affected by this change, so we should start looking into
possible breaks in Rawhide 'soon'.

[Petr Vobornik]

On Wed, Oct 25, 2017 at 1:05 PM, Alexander Bokovoy via FreeIPA-devel
 wrote:
> Hi,
>
> FYI.
>
> We are going to affected by this change, so we should start looking into
> possible breaks in Rawhide 'soon'.

If I get:

"""
Upgrade/compatibility impact

Data will be migrated automatically.
"""

In best case there won't be any breakage.  But we should be prepared
that some might appear.
> --
> / Alexander Bokovoy
>
> ---------- Forwarded message ----------
> From: Kai Engert 
> To: Development discussions related to Fedora
>
> Cc:
> Bcc:
> Date: Wed, 25 Oct 2017 11:45:09 +0200
> Subject: NSS Default File Format SQL in Fedora 28
> TL;DR: The change originally planned for Fedora 27 will now be done for
> Rawhide Fedora 28, probably tomorrow.
>
> We had previously announced to change the NSS crypto library to use the
> new sql file format by default. Please see the attached message for the
> technical details.
>
> Because of blocker bugs, it wasn't done for Fedora 27. During the past
> months the known bugs in NSS were fixed, and the Firefox/Mozilla code
> was improved. I think now we're ready to make the change for Rawhide.
>
> If I understand correctly, at the current time of the development phase
> of Rawhide Fedora 28, it isn't necessary to go through a formal process
> to make this change.
>
> The new tracking bug is:
>   https://bugzilla.redhat.com/show_bug.cgi?id=1496560
>
> The following is noteworthy, when using NSS with the sql file format:
>
> Performance is slightly reduced. The old dbm storage didn't use any
> locking mechanism, and therefore it was easy to get corrupted storage
> when accessing the files in parallel by more than one application.
> The new sqlite storage can safely be used by multiple applications
> in parallel, but this has a performance cost.
>
> NSS databases (old and new) can be protected with a password.
> Previously, some modification operations could be performed without
> unlocking the database.
> When using the new sql storage, NSS more strictly requires the
> user to log in to (unlock) the database prior to performing
> modifications.
> For example, when modifying the trust settings of a CA certificate
> with certutil, it will be necessary to provide the database password.
>
> With sql databases, NSS is more strict with half-initialized
> databases. In the early years of the Netscape/Mozilla era,
> in order to support some application/browser level functionality,
> it had been necessary to distinguish between
>   "half initialized state:
>    no password ever set on the database, not even an empty password"
> and
>   "fully initialized state:
>    password set on the database, even if it's just the empty string".
> We believe this state is no longer required,
> certutil and modutil no longer create half-initialized databases.
> Until recently, NSS databases created by Firefox/Thunderbird etc.
> were still in this half-initialized state.
> There might be other applications that did this.
> When using a database in the half-initialized state, some
> database operations could fail, with similar symptons as not having
> logged in to the database, even after databases are (automatically)
> migrated to the sql format (the half-initialized state is kept).
> When experiencing such issues, it will be necessary to explicitly
> set a password on the database, even if it's just the empty string.
>
> In order to adjust for these properties, Firefox has been changed to
> always initialize NSS databases with an empty password.
>
> In addition, Firefox has been changed to prompt the user for the NSS
> database password (Firefox calls it the master password) if necessary,
> prior to e.g. trust database modificiations.
>
> These recent changes will be contained in NSS 3.34, Firefox 58 and
> Thunderbird 59, which aren't released yet.
>
> In order to allow us testing the new sql database file format with
> Fedora Rawhide as soon as possible, the changes have been backported to
> the versions currently used in Rawhide: NSS 3.33, Firefox 57 beta,
> Thunderbird 52.4. The packages were built yesterday.
>
> If other applications based on Mozilla Gecko, e.g. like Iceowl or
> Icecat, would like to avoid that users run into these potential
> failures, they could pick up the same backported patches. Please look at
> the commits for Firefox and Thunderbird to find them, or see the
> dependency list of bug 1496560.
>
> Please let us know if you experience problems. (Ideally, please file a
> bug against the "nss" component and CC kengert@rh, hkario@rh,
> rrelyea@rh, dueno@rh.)
>
> Thanks
> Kai
>
> ---------- Forwarded message ----------
> From: Jaroslav Reznik 
> To: "discussions related to Fedora, Development"
> , devel-announce
>
> Cc:
> Bcc:
> Date: Thu, 6 Jul 2017 18:25:51 +0200
> Subject: F27 System Wide Change: NSS Default File Format SQL
> = System Wide Change: NSS Default File Format SQL =
> https://fedoraproject.org/wiki/Changes/NSSDefaultFileFormatSql
>
> Change owner(s):
> Kai Engert
>
> Change the NSS library default to use the sqlite based data storage,
> when applications don't specify their preferred storage file format.
>
> == Detailed Description ==
>
> Applications that use the NSS library often use a database for storage
> of keys, certificates and trust. NSS supports two different file
> formats, one called DBM (based on berkeley DB files) and another one
> called SQL (based on sqlite DB files).
>
> Today's default file format used by NSS, used when applications omit
> the type parameter, is the older DBM file format, which forbids
> parallel access to the storage. The suggestion is to change the
> default file format to SQL, which allows parallel access to the
> storage.
>
> Applications, or users using the NSS command line utilities, often
> provide the database storage location using a simple directory path
> parameter. Some might not be aware, or forget, that the parameter can
> be prefixed with a type modifier, either "dbm:" or "sql:".
>
> As a result, when not providing this parameter, the file format used
> will be the fragile DBM file format. This is particuarly problematic,
> if a user attempts to modify the NSS storage using command line tools,
> while another process, such as a daemon, is running concurrently,
> which also accesses the same database in the DBM file format. This
> often results in corrupted database storage, which cannot be
> recovered.
>
> By changing the default, all applications that currently use the DBM
> file format, will automatically be migrated to the SQL file format.
> NSS has the ability to discover if a storage location (a directory)
> contains the DBM file format. If configured to use the modern SQL
> format, NSS will automatically perform a one-time conversion from the
> DBM to the SQL format.
>
> The same applies to the NSS command line utilities. If the NSS library
> default is changed to SQL, the NSS tools will also trigger the
> one-time conversion, or access the already converted files.
>
> == Scope ==
> Proposal owners:
>
> A small downstream patch needs to be applied to the NSS library
> package, which changes the library default.
> Other developers:
>
> It's up to developers of NSS applications, if they accept the new
> default and an automatic conversion, or if they prefer to continue to
> use the classic DBM storage format. Although not recommended,
> developers can easily do so, by adding a "dbm:" prefix to the storage
> parameter they provide to NSS at NSS library initialization time.
> Release engineering: [1]
>
> No help should be necessary. No mass rebuild necessary.
>
> Policies and guidelines: N/A
>
> Trademark approval: N/A
>
> [1] https://pagure.io/releng/issue/6883
>
> Thanks,
> Jaroslav
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>
> FreeIPA-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Alexander Bokovoy]

On ma, 06 marras 2017, Petr Vobornik wrote:
> On Wed, Oct 25, 2017 at 1:05 PM, Alexander Bokovoy via FreeIPA-devel
>  wrote:
> > Hi,
> >
> > FYI.
> >
> > We are going to affected by this change, so we should start looking into
> > possible breaks in Rawhide 'soon'.
>
> If I get:
>
> """
> Upgrade/compatibility impact
>
> Data will be migrated automatically.
> """
>
> In best case there won't be any breakage.  But we should be prepared
> that some might appear.

At least, we need to look at various places where NSS databases are used
and their location is specified. If some of those configs include
explicit database format, we need to update them.
> > --
> > / Alexander Bokovoy
> >
> > ---------- Forwarded message ----------
> > From: Kai Engert 
> > To: Development discussions related to Fedora
> >
> > Cc:
> > Bcc:
> > Date: Wed, 25 Oct 2017 11:45:09 +0200
> > Subject: NSS Default File Format SQL in Fedora 28
> > TL;DR: The change originally planned for Fedora 27 will now be done for
> > Rawhide Fedora 28, probably tomorrow.
> >
> > We had previously announced to change the NSS crypto library to use the
> > new sql file format by default. Please see the attached message for the
> > technical details.
> >
> > Because of blocker bugs, it wasn't done for Fedora 27. During the past
> > months the known bugs in NSS were fixed, and the Firefox/Mozilla code
> > was improved. I think now we're ready to make the change for Rawhide.
> >
> > If I understand correctly, at the current time of the development phase
> > of Rawhide Fedora 28, it isn't necessary to go through a formal process
> > to make this change.
> >
> > The new tracking bug is:
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1496560
> >
> > The following is noteworthy, when using NSS with the sql file format:
> >
> > Performance is slightly reduced. The old dbm storage didn't use any
> > locking mechanism, and therefore it was easy to get corrupted storage
> > when accessing the files in parallel by more than one application.
> > The new sqlite storage can safely be used by multiple applications
> > in parallel, but this has a performance cost.
> >
> > NSS databases (old and new) can be protected with a password.
> > Previously, some modification operations could be performed without
> > unlocking the database.
> > When using the new sql storage, NSS more strictly requires the
> > user to log in to (unlock) the database prior to performing
> > modifications.
> > For example, when modifying the trust settings of a CA certificate
> > with certutil, it will be necessary to provide the database password.
> >
> > With sql databases, NSS is more strict with half-initialized
> > databases. In the early years of the Netscape/Mozilla era,
> > in order to support some application/browser level functionality,
> > it had been necessary to distinguish between
> >   "half initialized state:
> >    no password ever set on the database, not even an empty password"
> > and
> >   "fully initialized state:
> >    password set on the database, even if it's just the empty string".
> > We believe this state is no longer required,
> > certutil and modutil no longer create half-initialized databases.
> > Until recently, NSS databases created by Firefox/Thunderbird etc.
> > were still in this half-initialized state.
> > There might be other applications that did this.
> > When using a database in the half-initialized state, some
> > database operations could fail, with similar symptons as not having
> > logged in to the database, even after databases are (automatically)
> > migrated to the sql format (the half-initialized state is kept).
> > When experiencing such issues, it will be necessary to explicitly
> > set a password on the database, even if it's just the empty string.
> >
> > In order to adjust for these properties, Firefox has been changed to
> > always initialize NSS databases with an empty password.
> >
> > In addition, Firefox has been changed to prompt the user for the NSS
> > database password (Firefox calls it the master password) if necessary,
> > prior to e.g. trust database modificiations.
> >
> > These recent changes will be contained in NSS 3.34, Firefox 58 and
> > Thunderbird 59, which aren't released yet.
> >
> > In order to allow us testing the new sql database file format with
> > Fedora Rawhide as soon as possible, the changes have been backported to
> > the versions currently used in Rawhide: NSS 3.33, Firefox 57 beta,
> > Thunderbird 52.4. The packages were built yesterday.
> >
> > If other applications based on Mozilla Gecko, e.g. like Iceowl or
> > Icecat, would like to avoid that users run into these potential
> > failures, they could pick up the same backported patches. Please look at
> > the commits for Firefox and Thunderbird to find them, or see the
> > dependency list of bug 1496560.
> >
> > Please let us know if you experience problems. (Ideally, please file a
> > bug against the "nss" component and CC kengert@rh, hkario@rh,
> > rrelyea@rh, dueno@rh.)
> >
> > Thanks
> > Kai
> >
> > ---------- Forwarded message ----------
> > From: Jaroslav Reznik 
> > To: "discussions related to Fedora, Development"
> > , devel-announce
> >
> > Cc:
> > Bcc:
> > Date: Thu, 6 Jul 2017 18:25:51 +0200
> > Subject: F27 System Wide Change: NSS Default File Format SQL
> > = System Wide Change: NSS Default File Format SQL =
> > https://fedoraproject.org/wiki/Changes/NSSDefaultFileFormatSql
> >
> > Change owner(s):
> > Kai Engert
> >
> > Change the NSS library default to use the sqlite based data storage,
> > when applications don't specify their preferred storage file format.
> >
> > == Detailed Description ==
> >
> > Applications that use the NSS library often use a database for storage
> > of keys, certificates and trust. NSS supports two different file
> > formats, one called DBM (based on berkeley DB files) and another one
> > called SQL (based on sqlite DB files).
> >
> > Today's default file format used by NSS, used when applications omit
> > the type parameter, is the older DBM file format, which forbids
> > parallel access to the storage. The suggestion is to change the
> > default file format to SQL, which allows parallel access to the
> > storage.
> >
> > Applications, or users using the NSS command line utilities, often
> > provide the database storage location using a simple directory path
> > parameter. Some might not be aware, or forget, that the parameter can
> > be prefixed with a type modifier, either "dbm:" or "sql:".
> >
> > As a result, when not providing this parameter, the file format used
> > will be the fragile DBM file format. This is particuarly problematic,
> > if a user attempts to modify the NSS storage using command line tools,
> > while another process, such as a daemon, is running concurrently,
> > which also accesses the same database in the DBM file format. This
> > often results in corrupted database storage, which cannot be
> > recovered.
> >
> > By changing the default, all applications that currently use the DBM
> > file format, will automatically be migrated to the SQL file format.
> > NSS has the ability to discover if a storage location (a directory)
> > contains the DBM file format. If configured to use the modern SQL
> > format, NSS will automatically perform a one-time conversion from the
> > DBM to the SQL format.
> >
> > The same applies to the NSS command line utilities. If the NSS library
> > default is changed to SQL, the NSS tools will also trigger the
> > one-time conversion, or access the already converted files.
> >
> > == Scope ==
> > Proposal owners:
> >
> > A small downstream patch needs to be applied to the NSS library
> > package, which changes the library default.
> > Other developers:
> >
> > It's up to developers of NSS applications, if they accept the new
> > default and an automatic conversion, or if they prefer to continue to
> > use the classic DBM storage format. Although not recommended,
> > developers can easily do so, by adding a "dbm:" prefix to the storage
> > parameter they provide to NSS at NSS library initialization time.
> > Release engineering: [1]
> >
> > No help should be necessary. No mass rebuild necessary.
> >
> > Policies and guidelines: N/A
> >
> > Trademark approval: N/A
> >
> > [1] https://pagure.io/releng/issue/6883
> >
> > Thanks,
> > Jaroslav
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> >
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> >
> > FreeIPA-devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
>
> -- 
> Petr Vobornik
>
> Associate Manager, Engineering, Identity Management
> Red Hat

[Christian Heimes]

On 2017-11-06 11:44, Alexander Bokovoy via FreeIPA-devel wrote:
> On ma, 06 marras 2017, Petr Vobornik wrote:
> > On Wed, Oct 25, 2017 at 1:05 PM, Alexander Bokovoy via FreeIPA-devel
> >  wrote:
> > > Hi,
> > >
> > > FYI.
> > >
> > > We are going to affected by this change, so we should start looking into
> > > possible breaks in Rawhide 'soon'.
> >
> > If I get:
> >
> > """
> > Upgrade/compatibility impact
> >
> > Data will be migrated automatically.
> > """
> >
> > In best case there won't be any breakage.  But we should be prepared
> > that some might appear.
>
> At least, we need to look at various places where NSS databases are used
> and their location is specified. If some of those configs include
> explicit database format, we need to update them.

It might affect ipa-custodia. It uses pk12util to access NSS database.

Christian

[Petr Vobornik]

On Mon, Nov 6, 2017 at 1:59 PM, Christian Heimes via FreeIPA-devel
 wrote:
> On 2017-11-06 11:44, Alexander Bokovoy via FreeIPA-devel wrote:
> > On ma, 06 marras 2017, Petr Vobornik wrote:
> > > On Wed, Oct 25, 2017 at 1:05 PM, Alexander Bokovoy via FreeIPA-devel
> > >  wrote:
> > > > Hi,
> > > >
> > > > FYI.
> > > >
> > > > We are going to affected by this change, so we should start looking into
> > > > possible breaks in Rawhide 'soon'.
> > >
> > > If I get:
> > >
> > > """
> > > Upgrade/compatibility impact
> > >
> > > Data will be migrated automatically.
> > > """
> > >
> > > In best case there won't be any breakage.  But we should be prepared
> > > that some might appear.
> >
> > At least, we need to look at various places where NSS databases are used
> > and their location is specified. If some of those configs include
> > explicit database format, we need to update them.
>
> It might affect ipa-custodia. It uses pk12util to access NSS database.

This and Alexander's comment were reflected in new bug
https://pagure.io/freeipa/issue/7236

[Rob Crittenden]

Petr Vobornik via FreeIPA-devel wrote:
> On Mon, Nov 6, 2017 at 1:59 PM, Christian Heimes via FreeIPA-devel
>  wrote:
> > On 2017-11-06 11:44, Alexander Bokovoy via FreeIPA-devel wrote:
> > > On ma, 06 marras 2017, Petr Vobornik wrote:
> > > > On Wed, Oct 25, 2017 at 1:05 PM, Alexander Bokovoy via FreeIPA-devel
> > > >  wrote:
> > > > > Hi,
> > > > >
> > > > > FYI.
> > > > >
> > > > > We are going to affected by this change, so we should start looking into
> > > > > possible breaks in Rawhide 'soon'.
> > > >
> > > > If I get:
> > > >
> > > > """
> > > > Upgrade/compatibility impact
> > > >
> > > > Data will be migrated automatically.
> > > > """
> > > >
> > > > In best case there won't be any breakage.  But we should be prepared
> > > > that some might appear.
> > >
> > > At least, we need to look at various places where NSS databases are used
> > > and their location is specified. If some of those configs include
> > > explicit database format, we need to update them.
> >
> > It might affect ipa-custodia. It uses pk12util to access NSS database.
>
> This and Alexander's comment were reflected in new bug
> https://pagure.io/freeipa/issue/7236

Isn't this a duplicate of either https://pagure.io/freeipa/issue/7209 or
https://pagure.io/freeipa/issue/7049 ?

rob

[Petr Vobornik]

On Mon, Nov 6, 2017 at 4:34 PM, Rob Crittenden  wrote:
> Petr Vobornik via FreeIPA-devel wrote:
> > On Mon, Nov 6, 2017 at 1:59 PM, Christian Heimes via FreeIPA-devel
> >  wrote:
> > > On 2017-11-06 11:44, Alexander Bokovoy via FreeIPA-devel wrote:
> > > > On ma, 06 marras 2017, Petr Vobornik wrote:
> > > > > On Wed, Oct 25, 2017 at 1:05 PM, Alexander Bokovoy via FreeIPA-devel
> > > > >  wrote:
> > > > > > Hi,
> > > > > >
> > > > > > FYI.
> > > > > >
> > > > > > We are going to affected by this change, so we should start looking into
> > > > > > possible breaks in Rawhide 'soon'.
> > > > >
> > > > > If I get:
> > > > >
> > > > > """
> > > > > Upgrade/compatibility impact
> > > > >
> > > > > Data will be migrated automatically.
> > > > > """
> > > > >
> > > > > In best case there won't be any breakage.  But we should be prepared
> > > > > that some might appear.
> > > >
> > > > At least, we need to look at various places where NSS databases are used
> > > > and their location is specified. If some of those configs include
> > > > explicit database format, we need to update them.
> > >
> > > It might affect ipa-custodia. It uses pk12util to access NSS database.
> >
> > This and Alexander's comment were reflected in new bug
> > https://pagure.io/freeipa/issue/7236
>
> Isn't this a duplicate of either https://pagure.io/freeipa/issue/7209 or
> https://pagure.io/freeipa/issue/7049 ?
>
> rob

You are right. So #7236 was closed as dup.