Heads up: OpenSSL-1.1.1e coming to Rawhide

[Tomas Mraz]

The new openssl-1.1.1e is coming to Rawhide.

It reports premature EOF/improper shutdown on TLS connections more
properly. However this might make some dependencies broken in build
tests (such as Ruby).

As I would like to eventually update the openssl also on stable
branches because it brings many bugfixes please consider bringing
eventual fixes/workarounds in depending packages also there.

-- 
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]

[Miro Hrončok]

On 19. 03. 20 17:31, Tomas Mraz wrote:
>  The new openssl-1.1.1e is coming to Rawhide.
>  
>  It reports premature EOF/improper shutdown on TLS connections more
>  properly. However this might make some dependencies broken in build
>  tests (such as Ruby).
>  
>  As I would like to eventually update the openssl also on stable
>  branches because it brings many bugfixes please consider bringing
>  eventual fixes/workarounds in depending packages also there. 
Packages failing to build:

https://koschei.fedoraproject.org/affected-by/openssl?epoch1=1&versio...

https://koschei.fedoraproject.org/affected-by/openssl-devel?epoch1=1&...

That includes Python interpreters.

We have Python tests defined in the CI:

https://src.fedoraproject.org/rpms/openssl/blob/master/f/tests/tests_pyth...

Why have this upgrade never been tested that way?

Please do not push this to older releases until we fix this.

Thanks,
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 22. 03. 20 17:29, Miro Hrončok wrote:
>  On 19. 03. 20 17:31, Tomas Mraz wrote:
> > The new openssl-1.1.1e is coming to Rawhide.
> >
> > It reports premature EOF/improper shutdown on TLS connections more
> > properly. However this might make some dependencies broken in build
> > tests (such as Ruby).
> >
> > As I would like to eventually update the openssl also on stable
> > branches because it brings many bugfixes please consider bringing
> > eventual fixes/workarounds in depending packages also there.
>  
>  That includes Python interpreters. 
https://bugs.python.org/issue40044

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Tomas Mraz]

On Sun, 2020-03-22 at 17:29 +0100, Miro Hrončok wrote:
>  On 19. 03. 20 17:31, Tomas Mraz wrote:
>  > The new openssl-1.1.1e is coming to Rawhide.
>  > 
>  > It reports premature EOF/improper shutdown on TLS connections more
>  > properly. However this might make some dependencies broken in build
>  > tests (such as Ruby).
>  > 
>  > As I would like to eventually update the openssl also on stable
>  > branches because it brings many bugfixes please consider bringing
>  > eventual fixes/workarounds in depending packages also there.
>  
>  Packages failing to build:
>  
>
> https://koschei.fedoraproject.org/affected-by/openssl?epoch1=1&versio...
>  
>
> https://koschei.fedoraproject.org/affected-by/openssl-devel?epoch1=1&...
>  
>  That includes Python interpreters.
>  
>  We have Python tests defined in the CI:
>  
>  https://src.fedoraproject.org/rpms/openssl/blob/master/f/tests/tests_pyth...
>  
>  Why have this upgrade never been tested that way? 
I knew there will be actual problems so that is the reason why I sent
the heads up. Next time I'll make sure the CI is run as well, not sure
if it would make any difference in this case apart from maybe I would
open bugs right away?

>  Please do not push this to older releases until we fix this. 
I will not push it to older releases. Most probably we will revert this
change in upstream 1.1.1 branch and I will update the rawhide build
with the revert patch as well. Anyway this change is going to stay in
the master branch of OpenSSL (for 3.0.0) so it is a good idea to be
able to handle it in the dependencies anyway.

-- 
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]

[Charalampos Stratakis]

----- Original Message -----
>  From: "Tomas Mraz" <tmraz(a)redhat.com&gt;
>  To: "Miro Hrončok" <mhroncok(a)redhat.com&gt;, "Development discussions
> related to Fedora" <devel(a)lists.fedoraproject.org&gt;
>  Cc: "python-maint" <python-maint(a)redhat.com&gt;
>  Sent: Tuesday, March 24, 2020 1:22:37 PM
>  Subject: Re: Heads up: OpenSSL-1.1.1e coming to Rawhide
>  
>  On Sun, 2020-03-22 at 17:29 +0100, Miro Hrončok wrote:
>  > On 19. 03. 20 17:31, Tomas Mraz wrote:
>  > > The new openssl-1.1.1e is coming to Rawhide.
>  > > 
>  > > It reports premature EOF/improper shutdown on TLS connections more
>  > > properly. However this might make some dependencies broken in build
>  > > tests (such as Ruby).
>  > > 
>  > > As I would like to eventually update the openssl also on stable
>  > > branches because it brings many bugfixes please consider bringing
>  > > eventual fixes/workarounds in depending packages also there.
>  > 
>  > Packages failing to build:
>  > 
>  >
> https://koschei.fedoraproject.org/affected-by/openssl?epoch1=1&versio...
>  > 
>  >
> https://koschei.fedoraproject.org/affected-by/openssl-devel?epoch1=1&...
>  > 
>  > That includes Python interpreters.
>  > 
>  > We have Python tests defined in the CI:
>  > 
>  > https://src.fedoraproject.org/rpms/openssl/blob/master/f/tests/tests_pyth...
>  > 
>  > Why have this upgrade never been tested that way?
>  
>  I knew there will be actual problems so that is the reason why I sent
>  the heads up. Next time I'll make sure the CI is run as well, not sure
>  if it would make any difference in this case apart from maybe I would
>  open bugs right away? 
With the PR workflow on pagure, the CI would be run and we can check out the issues that
might appear on the python side at least, as we have added the relevant python tests in
the openssl pagure repo. So indeed it would help a lot.

>  
>  > Please do not push this to older releases until we fix this.
>  
>  I will not push it to older releases. Most probably we will revert this
>  change in upstream 1.1.1 branch and I will update the rawhide build
>  with the revert patch as well. Anyway this change is going to stay in
>  the master branch of OpenSSL (for 3.0.0) so it is a good idea to be
>  able to handle it in the dependencies anyway.
>   
That would be great actually, thanks for considering it. Pushing this change for the 3.0.0
version of OpenSSL should provide us with enough time to iron out everything.

On a side note, is there some upstream CI of OpenSSL where we could maybe test its
integration with Python, or other projects? From the python upstream CI side, where we use
the buildbot software, we noticed that when the fedora servers running the builds got the
openssl package updated, the tests started failing. Maybe something similar could be
implemented for OpenSSL, depending of course if the infrastructure is in place.

>  --
>  Tomáš Mráz
>  No matter how far down the wrong road you've gone, turn back.
>                                                Turkish proverb
>  [You'll know whether the road is wrong if you carefully listen to your
>  conscience.]
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>   
-- 
Regards,

Charalampos Stratakis
Software Engineer
Python Maintenance Team, Red Hat

[Tomas Mraz]

On Tue, 2020-03-24 at 09:52 -0400, Charalampos Stratakis wrote:
>  
>  ----- Original Message -----
>  > From: "Tomas Mraz" <tmraz(a)redhat.com&gt;
>  > To: "Miro Hrončok" <mhroncok(a)redhat.com&gt;, "Development
> discussions
>  > related to Fedora" <devel(a)lists.fedoraproject.org&gt;
>  > Cc: "python-maint" <python-maint(a)redhat.com&gt;
>  > Sent: Tuesday, March 24, 2020 1:22:37 PM
>  > Subject: Re: Heads up: OpenSSL-1.1.1e coming to Rawhide
>  > 
>  > On Sun, 2020-03-22 at 17:29 +0100, Miro Hrončok wrote:
>  > > On 19. 03. 20 17:31, Tomas Mraz wrote:
>  > > > The new openssl-1.1.1e is coming to Rawhide.
>  > > > 
>  > > > It reports premature EOF/improper shutdown on TLS connections
>  > > > more
>  > > > properly. However this might make some dependencies broken in
>  > > > build
>  > > > tests (such as Ruby).
>  > > > 
>  > > > As I would like to eventually update the openssl also on stable
>  > > > branches because it brings many bugfixes please consider
>  > > > bringing
>  > > > eventual fixes/workarounds in depending packages also there.
>  > > 
>  > > Packages failing to build:
>  > > 
>  > >
> https://koschei.fedoraproject.org/affected-by/openssl?epoch1=1&versio...
>  > > 
>  > >
> https://koschei.fedoraproject.org/affected-by/openssl-devel?epoch1=1&...
>  > > 
>  > > That includes Python interpreters.
>  > > 
>  > > We have Python tests defined in the CI:
>  > > 
>  > >
> https://src.fedoraproject.org/rpms/openssl/blob/master/f/tests/tests_pyth...
>  > > 
>  > > Why have this upgrade never been tested that way?
>  > 
>  > I knew there will be actual problems so that is the reason why I
>  > sent
>  > the heads up. Next time I'll make sure the CI is run as well, not
>  > sure
>  > if it would make any difference in this case apart from maybe I
>  > would
>  > open bugs right away?
>  
>  With the PR workflow on pagure, the CI would be run and we can check
>  out the issues that might appear on the python side at least, as we
>  have added the relevant python tests in the openssl pagure repo. So
>  indeed it would help a lot.
>  
>  > > Please do not push this to older releases until we fix this.
>  > 
>  > I will not push it to older releases. Most probably we will revert
>  > this
>  > change in upstream 1.1.1 branch and I will update the rawhide build
>  > with the revert patch as well. Anyway this change is going to stay
>  > in
>  > the master branch of OpenSSL (for 3.0.0) so it is a good idea to be
>  > able to handle it in the dependencies anyway.
>  > 
>  
>  That would be great actually, thanks for considering it. Pushing this
>  change for the 3.0.0 version of OpenSSL should provide us with enough
>  time to iron out everything.
>  
>  On a side note, is there some upstream CI of OpenSSL where we could
>  maybe test its integration with Python, or other projects? From the
>  python upstream CI side, where we use the buildbot software, we
>  noticed that when the fedora servers running the builds got the
>  openssl package updated, the tests started failing. Maybe something
>  similar could be implemented for OpenSSL, depending of course if the
>  infrastructure is in place. 
There is already pyca-cryptography build and testsuite run in the
external tests. Perhaps some more python related stuff could be added
although I am not sure the way it is currently integrated would allow
much bigger testsuites being run.

-- 
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]

[Miro Hrončok]

On 24. 03. 20 13:22, Tomas Mraz wrote:
>  Most probably we will revert this
>  change in upstream 1.1.1 branch and I will update the rawhide build
>  with the revert patch as well. 
Can this please happen rather sooner than later?

The list of (likely) affected packages is growing:

https://koschei.fedoraproject.org/affected-by/openssl-libs?epoch1=1&v...

It shows that this does not only break Python's own test suite, but various 
Python packages are failing as well (incl. python-pip).

This blocks the testing rebuilds with Python 3.9.0a5 (reporting way too many 
unrelated FTBFSes).

Thanks,
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Jun Aruga]

>  The list of (likely) affected packages is growing:
>
>
> https://koschei.fedoraproject.org/affected-by/openssl-libs?epoch1=1&v...
It's a useful list! Thanks.
Note rubygem-puma is added to the list too.

-- 
Jun | He - His - Him

[Tomas Mraz]

On Wed, 2020-03-25 at 09:34 +0100, Miro Hrončok wrote:
>  On 24. 03. 20 13:22, Tomas Mraz wrote:
>  > Most probably we will revert this
>  > change in upstream 1.1.1 branch and I will update the rawhide build
>  > with the revert patch as well.
>  
>  Can this please happen rather sooner than later? 
I've built openssl-1.1.1e-2.fc33 with the EOF handling change reverted
today.

https://koji.fedoraproject.org/koji/buildinfo?buildID=1482948

>  The list of (likely) affected packages is growing:
>  
>
> https://koschei.fedoraproject.org/affected-by/openssl-libs?epoch1=1&v...
>  
>  It shows that this does not only break Python's own test suite, but
>  various 
>  Python packages are failing as well (incl. python-pip).
>  
>  This blocks the testing rebuilds with Python 3.9.0a5 (reporting way
>  too many 
>  unrelated FTBFSes). 
Unless OpenSSL upstream decides otherwise we will however have this
issue later in Fedora 33 development when the rebase to 3.0 happens. Of
course the rebase to 3.0 will be disruptive in more ways and so it
should be done in a side-tag first.

-- 
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]

[Miro Hrončok]

On 26. 03. 20 17:07, Tomas Mraz wrote:
>  On Wed, 2020-03-25 at 09:34 +0100, Miro Hrončok wrote:
> > On 24. 03. 20 13:22, Tomas Mraz wrote:
> >> Most probably we will revert this
> >> change in upstream 1.1.1 branch and I will update the rawhide build
> >> with the revert patch as well.
> >
> > Can this please happen rather sooner than later?
>  
>  I've built openssl-1.1.1e-2.fc33 with the EOF handling change reverted
>  today.
>  
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1482948 
Thanks a lot.

> > The list of (likely) affected packages is growing:
> >
> >
> https://koschei.fedoraproject.org/affected-by/openssl-libs?epoch1=1&v...
> >
> > It shows that this does not only break Python's own test suite, but
> > various
> > Python packages are failing as well (incl. python-pip).
> >
> > This blocks the testing rebuilds with Python 3.9.0a5 (reporting way
> > too many
> > unrelated FTBFSes).
>  
>  Unless OpenSSL upstream decides otherwise we will however have this
>  issue later in Fedora 33 development when the rebase to 3.0 happens. Of
>  course the rebase to 3.0 will be disruptive in more ways and so it
>  should be done in a side-tag first. 
Do you have at least a rough estimate for when should we expect this?

E.g. is it going to be after mid-May or before?

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Tomas Mraz]

On Thu, 2020-03-26 at 17:11 +0100, Miro Hrončok wrote:
>  On 26. 03. 20 17:07, Tomas Mraz wrote:
>  > On Wed, 2020-03-25 at 09:34 +0100, Miro Hrončok wrote:
>  > > On 24. 03. 20 13:22, Tomas Mraz wrote:
>  > > > Most probably we will revert this
>  > > > change in upstream 1.1.1 branch and I will update the rawhide
>  > > > build
>  > > > with the revert patch as well.
>  > > 
>  > > Can this please happen rather sooner than later?
>  > 
>  > I've built openssl-1.1.1e-2.fc33 with the EOF handling change
>  > reverted
>  > today.
>  > 
>  > https://koji.fedoraproject.org/koji/buildinfo?buildID=1482948
>  
>  Thanks a lot.
>  
>  > > The list of (likely) affected packages is growing:
>  > > 
>  > >
> https://koschei.fedoraproject.org/affected-by/openssl-libs?epoch1=1&v...
>  > > 
>  > > It shows that this does not only break Python's own test suite,
>  > > but
>  > > various
>  > > Python packages are failing as well (incl. python-pip).
>  > > 
>  > > This blocks the testing rebuilds with Python 3.9.0a5 (reporting
>  > > way
>  > > too many
>  > > unrelated FTBFSes).
>  > 
>  > Unless OpenSSL upstream decides otherwise we will however have this
>  > issue later in Fedora 33 development when the rebase to 3.0
>  > happens. Of
>  > course the rebase to 3.0 will be disruptive in more ways and so it
>  > should be done in a side-tag first.
>  
>  Do you have at least a rough estimate for when should we expect this?
>  
>  E.g. is it going to be after mid-May or before? 
I have to say that the schedule of the OpenSSL 3.0 release is a little
bit conflicting with the F33 schedule. However I should be able to have
a rebase to a beta version of OpenSSL 3.0 prepared at the end of June.

Of course depending on the problems with the dependent package rebuilds
in the side-tag we will see if the 3.0 would be viable for the Fedora
33 at all.

-- 
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]

[Miro Hrončok]

On 26. 03. 20 18:13, Tomas Mraz wrote:
> > E.g. is it going to be after mid-May or before?
>  I have to say that the schedule of the OpenSSL 3.0 release is a little
>  bit conflicting with the F33 schedule. However I should be able to have
>  a rebase to a beta version of OpenSSL 3.0 prepared at the end of June.
>  
>  Of course depending on the problems with the dependent package rebuilds
>  in the side-tag we will see if the 3.0 would be viable for the Fedora
>  33 at all. 
June is actually perfect for us, as it won't interfere with the Python 3.9 
rebuilds (if everything goes according to the plan).

https://fedoraproject.org/wiki/Changes/Python3.9#Important_dates_and_plan

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On Thu, Mar 26, 2020 at 06:15:13PM +0100, Miro Hrončok wrote:
>  On 26. 03. 20 18:13, Tomas Mraz wrote:
>  > > E.g. is it going to be after mid-May or before?
>  > I have to say that the schedule of the OpenSSL 3.0 release is a little
>  > bit conflicting with the F33 schedule. However I should be able to have
>  > a rebase to a beta version of OpenSSL 3.0 prepared at the end of June.
>  > 
>  > Of course depending on the problems with the dependent package rebuilds
>  > in the side-tag we will see if the 3.0 would be viable for the Fedora
>  > 33 at all.
>  
>  June is actually perfect for us, as it won't interfere with the Python 3.9
>  rebuilds (if everything goes according to the plan).
>  
>  https://fedoraproject.org/wiki/Changes/Python3.9#Important_dates_and_plan 
"2020-05-18: Expected side tag-merge (realistic)"

Is that supposed to be 06-18?

I'd like to note that our datacenter move is going to be happening in
late may/early june. This will mean koji builder capacity is much lower
than normal, so rebuilds of all python packages are not as likely to
finish quickly then. Just something to keep in mind. 

kevin

[Miro Hrončok]

On 26. 03. 20 18:46, Kevin Fenzi wrote:
>  On Thu, Mar 26, 2020 at 06:15:13PM +0100, Miro Hrončok wrote:
> > On 26. 03. 20 18:13, Tomas Mraz wrote:
> >>> E.g. is it going to be after mid-May or before?
> >> I have to say that the schedule of the OpenSSL 3.0 release is a little
> >> bit conflicting with the F33 schedule. However I should be able to have
> >> a rebase to a beta version of OpenSSL 3.0 prepared at the end of June.
> >>
> >> Of course depending on the problems with the dependent package rebuilds
> >> in the side-tag we will see if the 3.0 would be viable for the Fedora
> >> 33 at all.
> >
> > June is actually perfect for us, as it won't interfere with the Python 3.9
> > rebuilds (if everything goes according to the plan).
> >
> > https://fedoraproject.org/wiki/Changes/Python3.9#Important_dates_and_plan
>  
>  "2020-05-18: Expected side tag-merge (realistic)"
>  
>  Is that supposed to be 06-18? 
Good catch, thanks. Amended the wiki.

We are after:

2020-05-18: Python 3.9.0 beta 1, the ideal point when Koji builds start

>  I'd like to note that our datacenter move is going to be
> happening in
>  late may/early june. This will mean koji builder capacity is much lower
>  than normal, so rebuilds of all python packages are not as likely to
>  finish quickly then. Just something to keep in mind. 
We have a month allocated to finish this in the realistic scenario, 2 months in 
the pessimistic one, but in fact it's technically possible to do it within less 
than a week.

The month window was reserved for unforeseen blocking build failures.
The two moths window was planned in case the CPython beta release is delayed.

Do we need to plan a bigger window? How much will the capacity be reduced?

Thanks for the heads up.
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On Thu, Mar 26, 2020 at 07:01:22PM +0100, Miro Hrončok wrote:
>  On 26. 03. 20 18:46, Kevin Fenzi wrote:
>  > On Thu, Mar 26, 2020 at 06:15:13PM +0100, Miro Hrončok wrote:
>  > > On 26. 03. 20 18:13, Tomas Mraz wrote:
>  > > > > E.g. is it going to be after mid-May or before?
>  > > > I have to say that the schedule of the OpenSSL 3.0 release is a little
>  > > > bit conflicting with the F33 schedule. However I should be able to have
>  > > > a rebase to a beta version of OpenSSL 3.0 prepared at the end of June.
>  > > > 
>  > > > Of course depending on the problems with the dependent package rebuilds
>  > > > in the side-tag we will see if the 3.0 would be viable for the Fedora
>  > > > 33 at all.
>  > > 
>  > > June is actually perfect for us, as it won't interfere with the Python 3.9
>  > > rebuilds (if everything goes according to the plan).
>  > > 
>  > > https://fedoraproject.org/wiki/Changes/Python3.9#Important_dates_and_plan
>  > 
>  > "2020-05-18: Expected side tag-merge (realistic)"
>  > 
>  > Is that supposed to be 06-18?
>  
>  Good catch, thanks. Amended the wiki.
>  
>  We are after:
>  
>  2020-05-18: Python 3.9.0 beta 1, the ideal point when Koji builds start
>  
>  > I'd like to note that our datacenter move is going to be happening in
>  > late may/early june. This will mean koji builder capacity is much lower
>  > than normal, so rebuilds of all python packages are not as likely to
>  > finish quickly then. Just something to keep in mind.
>  
>  We have a month allocated to finish this in the realistic scenario, 2 months
>  in the pessimistic one, but in fact it's technically possible to do it
>  within less than a week.
>  
>  The month window was reserved for unforeseen blocking build failures.
>  The two moths window was planned in case the CPython beta release is delayed.
>  
>  Do we need to plan a bigger window? How much will the capacity be reduced? 
I don't think you need a bigger window. I just wanted to note that
builds may be slower during that time. 

I can't give you a specific number right now, as it depends on when
shipments go out and when we can bring things back online. I suspect
there will be a few weeks where we are down to 1/2 to 1/3 our normal
capacity, rising after that. 

kevin