Hello.
Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing farm fail.
The same tests work on Rawhide and worked on Fedora 37/36 previously -- unfortunately this could have been a while ago, maybe even couple weeks, due to the holidays.
See https://src.fedoraproject.org/rpms/python3.11/pull-request/99
The test failures look like this:
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
---------------------------------------------------------------------- Ran 101 tests in 1.792s
All teh failed tests are related to either multiprocessing or asyncio. All of them are either:
AssertionError: OSError not raised by bind
Or:
AssertionError: OSError not raised by Listener
The same tests pass during the Koji build when he package builds (unless they are skipped due to lack of network, which I need to investigate).
On 10. 01. 23 11:10, Miro Hrončok wrote:
The same tests pass during the Koji build when he package builds (unless they are skipped due to lack of network, which I need to investigate).
They indeed pass in Koji.
On 10. 01. 23 11:10, Miro Hrončok wrote:
Hello.
Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing farm fail.
The same tests work on Rawhide and worked on Fedora 37/36 previously -- unfortunately this could have been a while ago, maybe even couple weeks, due to the holidays.
See https://src.fedoraproject.org/rpms/python3.11/pull-request/99
The test failures look like this:
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
Ran 101 tests in 1.792s
All teh failed tests are related to either multiprocessing or asyncio. All of them are either:
AssertionError: OSError not raised by bind
Or:
AssertionError: OSError not raised by Listener
The same tests pass during the Koji build when he package builds.
I've checked and all the tests basically do:
- bind a socket - bind it again (and assert failure, which fails)
Petr (CC'ed) found out that SO_REUSEPORT socket option exists which when enabled might cause the assertions to fail.
But that should not be possible to set from outside of the process.
https://lwn.net/Articles/542629/
On 11. 01. 23 12:01, Miro Hrončok wrote:
On 10. 01. 23 11:10, Miro Hrončok wrote:
Hello.
Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing farm fail.
The same tests work on Rawhide and worked on Fedora 37/36 previously -- unfortunately this could have been a while ago, maybe even couple weeks, due to the holidays.
See https://src.fedoraproject.org/rpms/python3.11/pull-request/99
The test failures look like this:
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
====================================================================== FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
Traceback (most recent call last): File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind self.assertRaises(OSError, s2.bind, (self.addr[0], port)) AssertionError: OSError not raised by bind
Ran 101 tests in 1.792s
All teh failed tests are related to either multiprocessing or asyncio. All of them are either:
AssertionError: OSError not raised by bind
Or:
AssertionError: OSError not raised by Listener
The same tests pass during the Koji build when he package builds.
I've checked and all the tests basically do:
- bind a socket - bind it again (and assert failure, which fails)
Petr (CC'ed) found out that SO_REUSEPORT socket option exists which when enabled might cause the assertions to fail.
But that should not be possible to set from outside of the process.
Good news! I can reproduce this locally with the same kernel version:
Linux fedora 6.0.16-300.fc37.x86_64
Will try to update the kernel and see if it helps.
On 11. 01. 23 13:24, Miro Hrončok wrote:
Good news! I can reproduce this locally with the same kernel version:
Linux fedora 6.0.16-300.fc37.x86_64
Will try to update the kernel and see if it helps.
Same problem with 6.0.18-300.fc37.x86_64. No problem with 6.2.0-0.rc2.20230105git41c03ba9beea.20.fc38.x86_64.
On 11. 01. 23 13:34, Miro Hrončok wrote:
On 11. 01. 23 13:24, Miro Hrončok wrote:
Good news! I can reproduce this locally with the same kernel version:
Linux fedora 6.0.16-300.fc37.x86_64
Will try to update the kernel and see if it helps.
Same problem with 6.0.18-300.fc37.x86_64. No problem with 6.2.0-0.rc2.20230105git41c03ba9beea.20.fc38.x86_64.
I've decided to move this to devel list, as it is clearly not limited to the CI system.
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...