Has anything changed recently wrt networking or DNS on the Testing Farm?

[Miro Hrončok]

Hello.

Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing 
farm fail.

The same tests work on Rawhide and worked on Fedora 37/36 previously -- 
unfortunately this could have been a while ago, maybe even couple weeks, due to 
the holidays.

See https://src.fedoraproject.org/rpms/python3.11/pull-request/99

The test failures look like this:

======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
     self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind

======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
     self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind

======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
     self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind

======================================================================
FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
     self.assertRaises(OSError, s2.bind, (self.addr[0], port))
AssertionError: OSError not raised by bind

----------------------------------------------------------------------
Ran 101 tests in 1.792s



All teh failed tests are related to either multiprocessing or asyncio.
All of them are either:

   AssertionError: OSError not raised by bind

Or:

   AssertionError: OSError not raised by Listener


The same tests pass during the Koji build when he package builds (unless they 
are skipped due to lack of network, which I need to investigate).

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 10. 01. 23 11:10, Miro Hrončok wrote:
>  The same tests pass during the Koji build when he package builds
> (unless they 
>  are skipped due to lack of network, which I need to investigate). 
They indeed pass in Koji.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 10. 01. 23 11:10, Miro Hrončok wrote:
>  Hello.
>  
>  Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing 
>  farm fail.
>  
>  The same tests work on Rawhide and worked on Fedora 37/36 previously -- 
>  unfortunately this could have been a while ago, maybe even couple weeks, due to 
>  the holidays.
>  
>  See https://src.fedoraproject.org/rpms/python3.11/pull-request/99
>  
>  The test failures look like this:
>  
>  ======================================================================
>  FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
>  ----------------------------------------------------------------------
>  Traceback (most recent call last):
>     File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
>       self.assertRaises(OSError, s2.bind, (self.addr[0], port))
>  AssertionError: OSError not raised by bind
>  
>  ======================================================================
>  FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
>  ----------------------------------------------------------------------
>  Traceback (most recent call last):
>     File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
>       self.assertRaises(OSError, s2.bind, (self.addr[0], port))
>  AssertionError: OSError not raised by bind
>  
>  ======================================================================
>  FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
>  ----------------------------------------------------------------------
>  Traceback (most recent call last):
>     File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
>       self.assertRaises(OSError, s2.bind, (self.addr[0], port))
>  AssertionError: OSError not raised by bind
>  
>  ======================================================================
>  FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
>  ----------------------------------------------------------------------
>  Traceback (most recent call last):
>     File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in test_bind
>       self.assertRaises(OSError, s2.bind, (self.addr[0], port))
>  AssertionError: OSError not raised by bind
>  
>  ----------------------------------------------------------------------
>  Ran 101 tests in 1.792s
>  
>  
>  
>  All teh failed tests are related to either multiprocessing or asyncio.
>  All of them are either:
>  
>     AssertionError: OSError not raised by bind
>  
>  Or:
>  
>     AssertionError: OSError not raised by Listener
>  
>  
>  The same tests pass during the Koji build when he package builds. 
I've checked and all the tests basically do:

  - bind a socket
  - bind it again (and assert failure, which fails)

Petr (CC'ed) found out that SO_REUSEPORT socket option exists which when 
enabled might cause the assertions to fail.

But that should not be possible to set from outside of the process.

https://lwn.net/Articles/542629/


-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 11. 01. 23 12:01, Miro Hrončok wrote:
>  On 10. 01. 23 11:10, Miro Hrončok wrote:
> > Hello.
> >
> > Yesterday I observed that the Fedora 37 and 36 Python CI tests on the Testing 
> > farm fail.
> >
> > The same tests work on Rawhide and worked on Fedora 37/36 previously -- 
> > unfortunately this could have been a while ago, maybe even couple weeks, due 
> > to the holidays.
> >
> > See https://src.fedoraproject.org/rpms/python3.11/pull-request/99
> >
> > The test failures look like this:
> >
> > ======================================================================
> > FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Poll.test_bind)
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in
> test_bind
> >      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> > AssertionError: OSError not raised by bind
> >
> > ======================================================================
> > FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv4Select.test_bind)
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in
> test_bind
> >      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> > AssertionError: OSError not raised by bind
> >
> > ======================================================================
> > FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Poll.test_bind)
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in
> test_bind
> >      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> > AssertionError: OSError not raised by bind
> >
> > ======================================================================
> > FAIL: test_bind (test.test_asyncore.TestAPI_UseIPv6Select.test_bind)
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >    File "/usr/lib64/python3.11/test/test_asyncore.py", line 752, in
> test_bind
> >      self.assertRaises(OSError, s2.bind, (self.addr[0], port))
> > AssertionError: OSError not raised by bind
> >
> > ----------------------------------------------------------------------
> > Ran 101 tests in 1.792s
> >
> >
> >
> > All teh failed tests are related to either multiprocessing or asyncio.
> > All of them are either:
> >
> >    AssertionError: OSError not raised by bind
> >
> > Or:
> >
> >    AssertionError: OSError not raised by Listener
> >
> >
> > The same tests pass during the Koji build when he package builds.
>  
>  I've checked and all the tests basically do:
>  
>    - bind a socket
>    - bind it again (and assert failure, which fails)
>  
>  Petr (CC'ed) found out that SO_REUSEPORT socket option exists which when 
>  enabled might cause the assertions to fail.
>  
>  But that should not be possible to set from outside of the process.
>  
>  https://lwn.net/Articles/542629/ 
Good news! I can reproduce this locally with the same kernel version:

Linux fedora 6.0.16-300.fc37.x86_64

Will try to update the kernel and see if it helps.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 11. 01. 23 13:24, Miro Hrončok wrote:
>  Good news! I can reproduce this locally with the same kernel
> version:
>  
>  Linux fedora 6.0.16-300.fc37.x86_64
>  
>  Will try to update the kernel and see if it helps. 
Same problem with 6.0.18-300.fc37.x86_64.
No problem with 6.2.0-0.rc2.20230105git41c03ba9beea.20.fc38.x86_64.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 11. 01. 23 13:34, Miro Hrončok wrote:
>  On 11. 01. 23 13:24, Miro Hrončok wrote:
> > Good news! I can reproduce this locally with the same kernel version:
> >
> > Linux fedora 6.0.16-300.fc37.x86_64
> >
> > Will try to update the kernel and see if it helps.
>  
>  Same problem with 6.0.18-300.fc37.x86_64.
>  No problem with 6.2.0-0.rc2.20230105git41c03ba9beea.20.fc38.x86_64. 
I've decided to move this to devel list, as it is clearly not limited to the CI 
system.

https://lists.fedoraproject.org/archives/list/[e-mail].o...

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok