Fedora rawhide (to be f41) and openssl engines

[Dmitry Belyavskiy]

Dear colleagues,

as the changes described in
https://fedoraproject.org/wiki/Changes/OpensslDeprecateEngine

were approved and implemented and a week or two has passed, we can
summarize the consequences.

Lack of openssl/engine.h file moved to a separate package is not
processed correctly by packages and requires changes in specs which
also comes with a cost. OpenSSL ABI is kept.

On the other hand, CentOS stream uses a different approach when
openssl keeps ABI, doesn't ship openssl/engine.h, and defines
OPENSSL_NO_ENGINE explicitly, so old applications keep working and at
the same time new application can mostly be rebuilt without
significant problems.
I understand that Fedora has much more packages but there are much
less complaints from CentOS/RHEL than from Fedora.

So I wonder if it's worth changing the engine deprecation mechanism in
Fedora to the one we have in CentOS and if yes, what is the mechanism
for such a change.

[Clemens Lang]

Hi Dima,
> On 22. Jul 2024, at 13:34, Dmitry Belyavskiy  wrote:
>
> Dear colleagues,
>
> as the changes described in
> https://fedoraproject.org/wiki/Changes/OpensslDeprecateEngine
>
> were approved and implemented and a week or two has passed, we can
> summarize the consequences.
>
> Lack of openssl/engine.h file moved to a separate package is not
> processed correctly by packages and requires changes in specs which
> also comes with a cost. OpenSSL ABI is kept.
>
> On the other hand, CentOS stream uses a different approach when
> openssl keeps ABI, doesn't ship openssl/engine.h, and defines
> OPENSSL_NO_ENGINE explicitly, so old applications keep working and at
> the same time new application can mostly be rebuilt without
> significant problems.
> I understand that Fedora has much more packages but there are much
> less complaints from CentOS/RHEL than from Fedora.
>
> So I wonder if it's worth changing the engine deprecation mechanism in
> Fedora to the one we have in CentOS and if yes, what is the mechanism
> for such a change.

I believe the answer is yes.

I’m working on making sure we catch all users of the ENGINE API and tell them about this change in https://bugzilla.redhat.com/show_bug.cgi?id=2296114.
I’ve recently been pointed to https://docs.fedoraproject.org/en-US/fesco/Mass_package_changes/, which explains the process to follow when mass-filing bugs.

I don’t know whether there would be any specific other Fedora process we would have to follow — maybe others can chime in on this.

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jul 22, 2024 at 01:34:39PM +0200, Dmitry Belyavskiy wrote:
> So I wonder if it's worth changing the engine deprecation mechanism in
> Fedora to the one we have in CentOS and if yes, what is the mechanism
> for such a change.

Does is make sense at this point? The mass rebuild is (almost?) finished
and I would expect that packages that needed to be updated for this
already were.

Do you have any statistics about how many packages still fail to
build because of the lacking header file?

Zbyszek

[Neal Gompa]

On Mon, Jul 22, 2024 at 8:57 AM Zbigniew Jędrzejewski-Szmek
 wrote:
> On Mon, Jul 22, 2024 at 01:34:39PM +0200, Dmitry Belyavskiy wrote:
> > So I wonder if it's worth changing the engine deprecation mechanism in
> > Fedora to the one we have in CentOS and if yes, what is the mechanism
> > for such a change.
>
> Does is make sense at this point? The mass rebuild is (almost?) finished
> and I would expect that packages that needed to be updated for this
> already were.
>
> Do you have any statistics about how many packages still fail to
> build because of the lacking header file?

The CentOS approach isn't a deprecation, it's flat out removal. It's a
completely different change.

Is anyone helping to migrate users of the engine API to newer APIs? If
that's not happening, then there's no way to support removing the
engine API from Fedora's OpenSSL.

--
真実はいつも一つ！/ Always, there's only one truth!

[Dmitry Belyavskiy]

Dear Neal,

On Mon, Jul 22, 2024 at 3:05 PM Neal Gompa  wrote:
> The CentOS approach isn't a deprecation, it's flat out removal. It's a
> completely different change.
>
> Is anyone helping to migrate users of the engine API to newer APIs? If
> that's not happening, then there's no way to support removing the
> engine API from Fedora's OpenSSL.

I didn't see such requests in Fedora, most people are just interested
in making their software compiling.
In CentOS there were 2-3 cases when people were interested in
replacing the functionality with the new one.

I'm ready to help if necessary, and Clemens made a great job
explaining people what's going on while I was on PTO

[Clemens Lang]

Hi Neal,
> On 22. Jul 2024, at 15:01, Neal Gompa  wrote:
>
> The CentOS approach isn't a deprecation, it's flat out removal. It's a
> completely different change.

This isn’t correct. The headers are removed, but the ABI is still present in CentOS Stream, so it is not flat out removal.

For Fedora, we could change this to keep the headers and the binary support present, but define `OPENSSL_NO_ENGINE` unless a special preprocessor define overrides it.
> Is anyone helping to migrate users of the engine API to newer APIs? If
> that's not happening, then there's no way to support removing the
> engine API from Fedora's OpenSSL.

We’re happy to help maintainers that want to do this. We may do it ourselves for selected components. We will not fix every single use ourselves.

[Fabio Valentini]

On Mon, Jul 22, 2024 at 4:28 PM Clemens Lang  wrote:
> Hi Neal,
> > On 22. Jul 2024, at 15:01, Neal Gompa  wrote:
> >
> > The CentOS approach isn't a deprecation, it's flat out removal. It's a
> > completely different change.
>
> This isn’t correct. The headers are removed, but the ABI is still present in CentOS Stream, so it is not flat out removal.

This is arguing about semantics, but probably the difference is that
packages in Fedora really MUST be kept in a state where they can be
rebuilt at any time, and removing the headers breaks that. It doesn't
break existing packages, but as soon as any changes need to be made to
any package that depends on those headers (or just a plain rebuild for
some other change in the distribution, or a mass rebuild), it *is*
equivalent to a removal.

Fabio

[Clemens Lang]

Hi,
> On 22. Jul 2024, at 16:32, Fabio Valentini  wrote:
>
> On Mon, Jul 22, 2024 at 4:28 PM Clemens Lang  wrote:
> > Hi Neal,
> > > On 22. Jul 2024, at 15:01, Neal Gompa  wrote:
> > >
> > > The CentOS approach isn't a deprecation, it's flat out removal. It's a
> > > completely different change.
> >
> > This isn’t correct. The headers are removed, but the ABI is still present in CentOS Stream, so it is not flat out removal.
>
> This is arguing about semantics, but probably the difference is that
> packages in Fedora really MUST be kept in a state where they can be
> rebuilt at any time, and removing the headers breaks that. It doesn't
> break existing packages, but as soon as any changes need to be made to
> any package that depends on those headers (or just a plain rebuild for
> some other change in the distribution, or a mass rebuild), it *is*
> equivalent to a removal.

There are three cases:

(1) packages that are broken now because they don’t yet depend on openssl-devel-engine and do not set OPENSSL_NO_ENGINE.
(2) packages that have been fixed by adding -DOPENSSL_NO_ENGINE to CPPFLAGS
(3) packages that have been fixed by adding a dependency on openssl-devel-engine

If we change OpenSSL to define OPENSSL_NO_ENGINE by default, with an override available, that affects these three cases as follows:

(1) now (hopefully, unless it’s an upstream bug) automatically don’t use ENGINEs, build should be fixed
(2) no change, continues to build
(3) continues to build, but stops using ENGINEs (but the maintainer would get a bug ticket about that from me, and then can set -DFEDORA_OPENSSL_STILL_USE_ENGINES)

At no point would a package move to a state where it doesn’t build.

(1) and (2) improve the situation for package maintainers. (3) is some extra work, but it’s also not fail-silent due to the ticket.

The alternative is doing nothing, which means packages in (1) stay broken and need to be fixed by somebody, and everybody else gets to keep the -DOPENSSL_NO_ENGINE define or dependency on openssl-devel-engine in their specfiles.

I think this would be a net improvement over what we currently have, but if others don’t agree, we can also just keep the current state and take it out on the backs of the maintainers that now have to deal with the -DOPENSSL_NO_ENGINE thing.

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jul 22, 2024 at 05:12:44PM +0200, Clemens Lang wrote:
> Hi,
> > On 22. Jul 2024, at 16:32, Fabio Valentini  wrote:
> >
> > On Mon, Jul 22, 2024 at 4:28 PM Clemens Lang  wrote:
> > > Hi Neal,
> > > > On 22. Jul 2024, at 15:01, Neal Gompa  wrote:
> > > >
> > > > The CentOS approach isn't a deprecation, it's flat out removal. It's a
> > > > completely different change.
> > >
> > > This isn’t correct. The headers are removed, but the ABI is still present in CentOS Stream, so it is not flat out removal.
> >
> > This is arguing about semantics, but probably the difference is that
> > packages in Fedora really MUST be kept in a state where they can be
> > rebuilt at any time, and removing the headers breaks that. It doesn't
> > break existing packages, but as soon as any changes need to be made to
> > any package that depends on those headers (or just a plain rebuild for
> > some other change in the distribution, or a mass rebuild), it *is*
> > equivalent to a removal.
>
> There are three cases:
>
> (1) packages that are broken now because they don’t yet depend on openssl-devel-engine and do not set OPENSSL_NO_ENGINE.
> (2) packages that have been fixed by adding -DOPENSSL_NO_ENGINE to CPPFLAGS
> (3) packages that have been fixed by adding a dependency on openssl-devel-engine
>
> If we change OpenSSL to define OPENSSL_NO_ENGINE by default, with an override available, that affects these three cases as follows:
>
> (1) now (hopefully, unless it’s an upstream bug) automatically don’t use ENGINEs, build should be fixed
> (2) no change, continues to build
> (3) continues to build, but stops using ENGINEs (but the maintainer would get a bug ticket about that from me, and then can set -DFEDORA_OPENSSL_STILL_USE_ENGINES)
>
> At no point would a package move to a state where it doesn’t build.
>
> (1) and (2) improve the situation for package maintainers. (3) is some extra work, but it’s also not fail-silent due to the ticket.
>
> The alternative is doing nothing, which means packages in (1) stay broken and need to be fixed by somebody, and everybody else gets to keep the -DOPENSSL_NO_ENGINE define or dependency on openssl-devel-engine in their specfiles.

At this point, this sounds like the best approach.
The problem is well understood and the build failures are trivially
resolved by adding a single BuildRequires line or a single define.

If we start changing things again, some packages will already adapted
will need to adapt again, and overall there'll much more confusion.

Zbyszek

[Clemens Lang]

Hi,
> On 22. Jul 2024, at 20:42, Zbigniew Jędrzejewski-Szmek  wrote:
>
> At this point, this sounds like the best approach.
> The problem is well understood and the build failures are trivially
> resolved by adding a single BuildRequires line or a single define.
>
> If we start changing things again, some packages will already adapted
> will need to adapt again, and overall there'll much more confusion.

I won’t object to this, it’s overall less work for Dmitry and me.

However, we should still consider the effect this will have on developers that build software on Fedora — they will also have to specify -DOPENSSL_NO_ENGINE now or see failing builds, and we don’t really see that impact until 41 releases.

[Gary Buhrmaster]

On Tue, Jul 23, 2024 at 8:55 AM Clemens Lang  wrote:
> However, we should still consider the effect this will have on developers that build software on Fedora — they will also have to specify -DOPENSSL_NO_ENGINE now or see failing builds, and we don’t really see that impact until 41 releases.

Or, perhaps, if they build locally, just
   `dnf install openssl-devel-engine`
once and not have to worry about
changing their build recipes until
OpenSSL 4.0 (or later).

That is certainly going to be a choice
some will make, and you should
always offer that as a possible
solution even if you wish it was not
used.

[Clemens Lang]

Hi,
> On 23. Jul 2024, at 16:36, Gary Buhrmaster  wrote:
>
> On Tue, Jul 23, 2024 at 8:55 AM Clemens Lang  wrote:
> > However, we should still consider the effect this will have on developers that build software on Fedora — they will also have to specify -DOPENSSL_NO_ENGINE now or see failing builds, and we don’t really see that impact until 41 releases.
>
> Or, perhaps, if they build locally, just
>   `dnf install openssl-devel-engine`
> once and not have to worry about
> changing their build recipes until
> OpenSSL 4.0 (or later).
>
> That is certainly going to be a choice
> some will make, and you should
> always offer that as a possible
> solution even if you wish it was not
> used.

Sure, people can do that, but they will just see an error saying that openssl/engine.h could not be found and start hunting for solutions. Let’s hope that until that happens, enough people have blogged about this (or Google has indexed this thread) to point them to potential solutions, because otherwise this will be a situation of “works on Ubuntu and everything else, but is just bad developer experience on Fedora”.

We already saw a number of threads on the mailing list and a ticket reporting this precise situation.

I guess from this thread it is pretty clear that there is no consensus on improving this for F41, so I’ll stop now.

[Stephen Smoogen]

On Tue, 23 Jul 2024 at 11:02, Clemens Lang  wrote:
> Hi,
> > On 23. Jul 2024, at 16:36, Gary Buhrmaster  wrote:
> >
> > On Tue, Jul 23, 2024 at 8:55 AM Clemens Lang  wrote:
> > > However, we should still consider the effect this will have on developers that build software on Fedora — they will also have to specify -DOPENSSL_NO_ENGINE now or see failing builds, and we don’t really see that impact until 41 releases.
> >
> > Or, perhaps, if they build locally, just
> >   `dnf install openssl-devel-engine`
> > once and not have to worry about
> > changing their build recipes until
> > OpenSSL 4.0 (or later).
> >
> > That is certainly going to be a choice
> > some will make, and you should
> > always offer that as a possible
> > solution even if you wish it was not
> > used.
>
> Sure, people can do that, but they will just see an error saying that openssl/engine.h could not be found and start hunting for solutions. Let’s hope that until that happens, enough people have blogged about this (or Google has indexed this thread) to point them to potential solutions, because otherwise this will be a situation of “works on Ubuntu and everything else, but is just bad developer experience on Fedora”.
>
> We already saw a number of threads on the mailing list and a ticket reporting this precise situation.
>
> I guess from this thread it is pretty clear that there is no consensus on improving this for F41, so I’ll stop now.

In order for it to be improved, it really needed to be done before the
mass rebuild. After that happens there really isn't much time for
medium to major changes. There are too many other groups all trying to
get their parts fixed and a lot of the Red Hat employees focus more on
getting EL10 beta and EL9.5 items than Fedora. I realize that this
isn't 'ideal' in any form but it is basically what happens every
couple of years when a new EL comes out.
> --
> Clemens Lang
> RHEL Crypto Team
> Red Hat
>
> --
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Joe Orton]

On Tue, Jul 23, 2024 at 10:54:12AM +0200, Clemens Lang wrote:
> However, we should still consider the effect this will have on 
> developers that build software on Fedora — they will also have to 
> specify -DOPENSSL_NO_ENGINE now or see failing builds, and we don’t 
> really see that impact until 41 releases.

Yup, exactly. This is only a bad outcome and will be seen (correctly) by 
upstream projects as Fedora shipping a broken OpenSSL when we start 
wasting their time with people filing "I couldn't build on Fedora" bug 
reports.

The time Fedora maintainers have spent on this is a sunk cost, doing 
nothing is clearly inferior to all other options. At minimum we should 
ship F41 with openssl-devel Requires: openssl-engine-devel again. 
(Having OpenSSL <configuration.h> #define OPENSSL_NO_ENGINE properly is 
by far my preferred option)

Regards, Joe

[Jonathan Wakely]

On Tue, 23 Jul 2024 at 16:32, Joe Orton wrote:
> On Tue, Jul 23, 2024 at 10:54:12AM +0200, Clemens Lang wrote:
> > However, we should still consider the effect this will have on
> > developers that build software on Fedora — they will also have to
> > specify -DOPENSSL_NO_ENGINE now or see failing builds, and we don’t
> > really see that impact until 41 releases.
>
> Yup, exactly. This is only a bad outcome and will be seen (correctly) by
> upstream projects as Fedora shipping a broken OpenSSL when we start
> wasting their time with people filing "I couldn't build on Fedora" bug
> reports.

Or those reports just come straight to us and we have to say "yes we
know, but we're not going to fix it", like this:
https://bugzilla.redhat.com/show_bug.cgi?id=2312672
> The time Fedora maintainers have spent on this is a sunk cost, doing
> nothing is clearly inferior to all other options. At minimum we should
> ship F41 with openssl-devel Requires: openssl-engine-devel again.
> (Having OpenSSL <configuration.h> #define OPENSSL_NO_ENGINE properly is
> by far my preferred option)

It's fixed in rawhide now, but are we still against fixing it for F41?
Affected Fedora packages might have been fixed, but people build other
software on Fedora that isn't already packaged and built in koji.

I'll add a workaround in boost-devel if the decision is still that
openssl shouldn't fix it in F41.

[Dmitry Belyavskiy]

Dear colleagues,

On Tue, Sep 17, 2024 at 10:05 PM Jonathan Wakely  wrote:
> It's fixed in rawhide now, but are we still against fixing it for F41?
> Affected Fedora packages might have been fixed, but people build other
> software on Fedora that isn't already packaged and built in koji.

I'm ok with porting this to F41

[Iñaki Ucar]

On Wed, 18 Sept 2024 at 15:36, Dmitry Belyavskiy 
wrote:
> Dear colleagues,
>
> On Tue, Sep 17, 2024 at 10:05 PM Jonathan Wakely 
> wrote:
> > It's fixed in rawhide now, but are we still against fixing it for F41?
> > Affected Fedora packages might have been fixed, but people build other
> > software on Fedora that isn't already packaged and built in koji.
>
> I'm ok with porting this to F41

It's just a matter of merging [1] into F41, right? Any blockers?

[1]
https://src.fedoraproject.org/rpms/openssl/c/13b583a535e62d12521cfeb5088a68e...

Iñaki
> --
> Dmitry Belyavskiy
>
> --
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Dmitry Belyavskiy]

It was recently merged to f41

On Mon, Sep 23, 2024 at 11:37 AM Iñaki Ucar  wrote:
> On Wed, 18 Sept 2024 at 15:36, Dmitry Belyavskiy  wrote:
> > Dear colleagues,
> >
> > On Tue, Sep 17, 2024 at 10:05 PM Jonathan Wakely  wrote:
> > > It's fixed in rawhide now, but are we still against fixing it for F41?
> > > Affected Fedora packages might have been fixed, but people build other
> > > software on Fedora that isn't already packaged and built in koji.
> >
> > I'm ok with porting this to F41
>
> It's just a matter of merging [1] into F41, right? Any blockers?
>
> [1] https://src.fedoraproject.org/rpms/openssl/c/13b583a535e62d12521cfeb5088a68e...
>
> Iñaki
> > --
> > Dmitry Belyavskiy
> >
> > --
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>
> --
> Iñaki Úcar
> --
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Dmitry Belyavskiy]

Dear Zbyszek,

On Mon, Jul 22, 2024 at 2:57 PM Zbigniew Jędrzejewski-Szmek
 wrote:
> On Mon, Jul 22, 2024 at 01:34:39PM +0200, Dmitry Belyavskiy wrote:
> > So I wonder if it's worth changing the engine deprecation mechanism in
> > Fedora to the one we have in CentOS and if yes, what is the mechanism
> > for such a change.
>
> Does is make sense at this point? The mass rebuild is (almost?) finished
> and I would expect that packages that needed to be updated for this
> already were.
>
> Do you have any statistics about how many packages still fail to
> build because of the lacking header file?

I've seen 10-15 email/Slack threads related to the change.
Most of them were resolved via explicit CFLAGS += -DOPENSSL_NO_ENGINE

[Gary Buhrmaster]

On Mon, Jul 22, 2024 at 11:35 AM Dmitry Belyavskiy  wrote:
> So I wonder if it's worth changing the engine deprecation mechanism in
> Fedora to the one we have in CentOS and if yes, what is the mechanism
> for such a change.

I think you are free to submit a (very) late change
request, but changing what was approved
(and what some people have already adopted
to) without a new FESCo approval seems
very very wrong to me, as it would seem to
make a mockery of the entire change request
process (i.e. lets get a change approved,
and then change the details afterwards).

It would seem to me that if the unexpected
fallout is considered to be too great, the
proper approach is to revert the entire
change, and re-propose in a future release
with a revised proposal (where all the
discussions can happen all over again).

[Vitaly Zaitsev]

On 22/07/2024 13:34, Dmitry Belyavskiy wrote:
> So I wonder if it's worth changing the engine deprecation mechanism in
> Fedora to the one we have in CentOS and if yes, what is the mechanism
> for such a change.

It's too late for F41. The mass rebuild is completed and the package 
maintainers have made changes to their packages by adding a new build 
requirement.

If you want to remove the engine.h completely, you need to create a new 
system-wide change, but for F42.