Has anything changed recently wrt networking or DNS on the Testing Farm?

[Miro Hrončok]

Hello.

Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing 
farm fail.

The same tests work on Rawhide and worked on Fedora 37/36 previously -- 
unfortunately this could have been a while ago, maybe even couple weeks, due to 
the holidays.

See https://src.fedoraproject.org/rpms/python3.11/pull-request/99

The test failures look like this:

======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
     self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind

======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
     self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind

======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
     self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind

======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
     self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind

----------------------------------------------------------------------
Ran 101 tests in 1.792s

All teh failed tests are related to either multiprocessing or asyncio.
All of them are either:

AssertionError: OSError not raised by bind

Or:

AssertionError: OSError not raised by Listener

The same tests pass during the Koji build when he package builds (unless they 
are skipped due to lack of network, which I need to investigate).

[Miro Hrončok]

On 10. 01. 23 11:10, Miro Hrončok wrote:
> The same tests pass during the Koji build when he package builds (unless they 
> are skipped due to lack of network, which I need to investigate).

They indeed pass in Koji.

[Miro Hrončok]

On 10. 01. 23 11:10, Miro Hrončok wrote:
> Hello.
>
> Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing 
> farm fail.
>
> The same tests work on Rawhide and worked on Fedora 37/36 previously -- 
> unfortunately this could have been a while ago, maybe even couple weeks, due to 
> the holidays.
>
> See https://src.fedoraproject.org/rpms/python3.11/pull-request/99
>
> The test failures look like this:
>
> ======================================================================
> FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
>
> Traceback (most recent call last):
>    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
>      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> AssertionError: OSError not raised by bind
>
> ======================================================================
> FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
>
> Traceback (most recent call last):
>    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
>      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> AssertionError: OSError not raised by bind
>
> ======================================================================
> FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
>
> Traceback (most recent call last):
>    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
>      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> AssertionError: OSError not raised by bind
>
> ======================================================================
> FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
>
> Traceback (most recent call last):
>    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
>      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> AssertionError: OSError not raised by bind
>
> Ran 101 tests in 1.792s
>
> All teh failed tests are related to either multiprocessing or asyncio.
> All of them are either:
>
> AssertionError: OSError not raised by bind
>
> Or:
>
> AssertionError: OSError not raised by Listener
>
> The same tests pass during the Koji build when he package builds.

I've checked and all the tests basically do:

- bind a socket
  - bind it again (and assert failure, which fails)

Petr (CC'ed) found out that SO_REUSEPORT socket option exists which when 
enabled might cause the assertions to fail.

But that should not be possible to set from outside of the process.

https://lwn.net/Articles/542629/

[Miro Hrončok]

On 11. 01. 23 12:01, Miro Hrončok wrote:
> On 10. 01. 23 11:10, Miro Hrončok wrote:
> > Hello.
> >
> > Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing 
> > farm fail.
> >
> > The same tests work on Rawhide and worked on Fedora 37/36 previously -- 
> > unfortunately this could have been a while ago, maybe even couple weeks, due 
> > to the holidays.
> >
> > See https://src.fedoraproject.org/rpms/python3.11/pull-request/99
> >
> > The test failures look like this:
> >
> > ======================================================================
> > FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
> >
> > Traceback (most recent call last):
> >    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
> >      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> > AssertionError: OSError not raised by bind
> >
> > ======================================================================
> > FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
> >
> > Traceback (most recent call last):
> >    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
> >      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> > AssertionError: OSError not raised by bind
> >
> > ======================================================================
> > FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
> >
> > Traceback (most recent call last):
> >    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
> >      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> > AssertionError: OSError not raised by bind
> >
> > ======================================================================
> > FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
> >
> > Traceback (most recent call last):
> >    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
> >      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> > AssertionError: OSError not raised by bind
> >
> > Ran 101 tests in 1.792s
> >
> > All teh failed tests are related to either multiprocessing or asyncio.
> > All of them are either:
> >
> > AssertionError: OSError not raised by bind
> >
> > Or:
> >
> > AssertionError: OSError not raised by Listener
> >
> > The same tests pass during the Koji build when he package builds.
>
> I've checked and all the tests basically do:
>
> - bind a socket
>   - bind it again (and assert failure, which fails)
>
> Petr (CC'ed) found out that SO_REUSEPORT socket option exists which when 
> enabled might cause the assertions to fail.
>
> But that should not be possible to set from outside of the process.
>
> https://lwn.net/Articles/542629/

Good news! I can reproduce this locally with the same kernel version:

Linux fedora 6.0.16-300.fc37.x86_64

Will try to update the kernel and see if it helps.

[Miro Hrončok]

On 11. 01. 23 13:24, Miro Hrončok wrote:
> Good news! I can reproduce this locally with the same kernel version:
>
> Linux fedora 6.0.16-300.fc37.x86_64
>
> Will try to update the kernel and see if it helps.

Same problem with 6.0.18-300.fc37.x86_64.
No problem with 6.2.0-0.rc2.20230105git41c03ba9beea.20.fc38.x86_64.

[Miro Hrončok]

On 11. 01. 23 13:34, Miro Hrončok wrote:
> On 11. 01. 23 13:24, Miro Hrončok wrote:
> > Good news! I can reproduce this locally with the same kernel version:
> >
> > Linux fedora 6.0.16-300.fc37.x86_64
> >
> > Will try to update the kernel and see if it helps.
>
> Same problem with 6.0.18-300.fc37.x86_64.
> No problem with 6.2.0-0.rc2.20230105git41c03ba9beea.20.fc38.x86_64.

I've decided to move this to devel list, as it is clearly not limited to the CI 
system.

https://lists.fedoraproject.org/archives/list/[e-mail]/...