rawhide - glibc/pthreads/... - broken pending mass rebuild?

[Alex Scheel]

Is Fedora Rawhide unstable at the moment, pending a mass rebuild?

I've seen a lot of random problems related to pthreads at the
moment, such as:

16/78 Test #12: JSS_DER_Encoding_of_Enumeration_regression_test ...Child
aborted***Exception:   0.99 sec
FINE: CryptoManager: loading JSS library
FINE: CryptoManager: loaded JSS library from java.library.path
java: ../nptl/pthread_mutex_lock.c:81: __pthread_mutex_lock: Assertion
`mutex->__data.__owner == 0' failed.


Another, different stack trace here (pthreads fails to lock,
triggering a bug in opencryptoki):

https://pagure.io/dogtagpki/issue/3181#comment-661911


And others. They don't reproduce consistently though.

Should I go ahead and file a bug or just wait and be patient? :)



This is blocking some rebuilds on rawhide at the moment for us.

- Alex

[Florian Weimer]

* Alex Scheel:

>  Is Fedora Rawhide unstable at the moment, pending a mass rebuild?
>
>  I've seen a lot of random problems related to pthreads at the
>  moment, such as:
>
>  16/78 Test #12: JSS_DER_Encoding_of_Enumeration_regression_test ...Child
> aborted***Exception:   0.99 sec
>  FINE: CryptoManager: loading JSS library
>  FINE: CryptoManager: loaded JSS library from java.library.path
>  java: ../nptl/pthread_mutex_lock.c:81: __pthread_mutex_lock: Assertion
> `mutex->__data.__owner == 0' failed.
>
>
>  Another, different stack trace here (pthreads fails to lock,
>  triggering a bug in opencryptoki):
>
>  https://pagure.io/dogtagpki/issue/3181#comment-661911 
I don't see why this would be fixed by a mass rebuild.

This is probably some sort of memory corruption: something has
overwritten the mutex data structure, and glibc happens to detect that.

Thanks,
Florian

[Vít Ondruch]

I just met something which might be of similar nature. Recent FF
78.0-1.fc33.x86_64 fails to start with older glibc:


~~~

$ firefox
XPCOMGlueLoad error for file /usr/lib64/firefox/libxul.so:
/usr/lib64/firefox/libxul.so: undefined symbol: pthread_getattr_np,
version GLIBC_2.32
Couldn't load XPCOM.

~~~


Update of glibc from glibc-2.31.9000-13.fc33.x86_64 to
glibc-common-2.31.9000-16.fc33.x86_64 resolved the issue. Nevertheless
issues like this are unexpected. There should be something, what would
force glibc update if FF requires more recent one.


Vít


Dne 01. 07. 20 v 6:57 Florian Weimer napsal(a):
>  * Alex Scheel:
>
> > Is Fedora Rawhide unstable at the moment, pending a mass rebuild?
> >
> > I've seen a lot of random problems related to pthreads at the
> > moment, such as:
> >
> > 16/78 Test #12: JSS_DER_Encoding_of_Enumeration_regression_test ...Child
> aborted***Exception:   0.99 sec
> > FINE: CryptoManager: loading JSS library
> > FINE: CryptoManager: loaded JSS library from java.library.path
> > java: ../nptl/pthread_mutex_lock.c:81: __pthread_mutex_lock: Assertion
> `mutex->__data.__owner == 0' failed.
> >
> >
> > Another, different stack trace here (pthreads fails to lock,
> > triggering a bug in opencryptoki):
> >
> > https://pagure.io/dogtagpki/issue/3181#comment-661911
>  I don't see why this would be fixed by a mass rebuild.
>
>  This is probably some sort of memory corruption: something has
>  overwritten the mutex data structure, and glibc happens to detect that.
>
>  Thanks,
>  Florian
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Florian Weimer]

* Vít Ondruch:

>  I just met something which might be of similar nature. Recent FF
>  78.0-1.fc33.x86_64 fails to start with older glibc:
>
>
>  ~~~
>
>  $ firefox
>  XPCOMGlueLoad error for file /usr/lib64/firefox/libxul.so:
>  /usr/lib64/firefox/libxul.so: undefined symbol: pthread_getattr_np,
>  version GLIBC_2.32
>  Couldn't load XPCOM.
>
>  ~~~
>
>
>  Update of glibc from glibc-2.31.9000-13.fc33.x86_64 to
>  glibc-common-2.31.9000-16.fc33.x86_64 resolved the issue. Nevertheless
>  issues like this are unexpected. There should be something, what would
>  force glibc update if FF requires more recent one. 
Partial rawhide updates used to be unsupported.  Has this changed?

Thanks,
Florian

[Petr Pisar]

On Thu, Jul 02, 2020 at 10:35:27AM +0200, Vít Ondruch wrote:
>  I just met something which might be of similar nature. Recent FF
>  78.0-1.fc33.x86_64 fails to start with older glibc:
>  
>  
>  ~~~
>  
>  $ firefox
>  XPCOMGlueLoad error for file /usr/lib64/firefox/libxul.so:
>  /usr/lib64/firefox/libxul.so: undefined symbol: pthread_getattr_np,
>  version GLIBC_2.32
>  Couldn't load XPCOM.
>  
>  ~~~
>  
>  
>  Update of glibc from glibc-2.31.9000-13.fc33.x86_64 to
>  glibc-common-2.31.9000-16.fc33.x86_64 resolved the issue. Nevertheless
>  issues like this are unexpected. There should be something, what would
>  force glibc update if FF requires more recent one.
>   
There is nothing like that. This issue is not specific to glibc. If a library
adds a symbol, it does not have to bump its soname. Thus on RPM level there
won't be any change. If another executable starts using the new symbol, there
won't be again no change on the RPM level. Therefore RPM finds a match at
installation, but the dynamic linker finds a dissonance at run time.

ELF RPM dependency generator could promote these linker symbol dependencies to
RPM level, but I worry that most people will complain about growing YUM
repository metadata.

I don't think it's realistic to expect from the packagers that they will check
and add these new dependencies manually. Teoretically, rpminpsect run in CI
could catch these changes and warn the packager. But I worry that many people
won't understand it and just keep ignoring it.

-- Petr

[Alex Scheel]

Just to revisit this, my issue turned out to be a bug in a p11-kit
component, opencryptoki, that failed to lock, causing its
deinitialization routine to trample some stuff. That's been fixed
now. Sorry for blaming glibc, Florian! :)


- Alex

----- Original Message -----
>  From: "Vít Ondruch" <vondruch(a)redhat.com&gt;
>  To: devel(a)lists.fedoraproject.org
>  Sent: Thursday, July 2, 2020 4:35:27 AM
>  Subject: Re: rawhide - glibc/pthreads/... - broken pending mass rebuild?
>  
>  I just met something which might be of similar nature. Recent FF
>  78.0-1.fc33.x86_64 fails to start with older glibc:
>  
>  
>  ~~~
>  
>  $ firefox
>  XPCOMGlueLoad error for file /usr/lib64/firefox/libxul.so:
>  /usr/lib64/firefox/libxul.so: undefined symbol: pthread_getattr_np,
>  version GLIBC_2.32
>  Couldn't load XPCOM.
>  
>  ~~~
>  
>  
>  Update of glibc from glibc-2.31.9000-13.fc33.x86_64 to
>  glibc-common-2.31.9000-16.fc33.x86_64 resolved the issue. Nevertheless
>  issues like this are unexpected. There should be something, what would
>  force glibc update if FF requires more recent one.
>  
>  
>  Vít
>  
>  
>  Dne 01. 07. 20 v 6:57 Florian Weimer napsal(a):
>  > * Alex Scheel:
>  >
>  >> Is Fedora Rawhide unstable at the moment, pending a mass rebuild?
>  >>
>  >> I've seen a lot of random problems related to pthreads at the
>  >> moment, such as:
>  >>
>  >> 16/78 Test #12: JSS_DER_Encoding_of_Enumeration_regression_test ...Child
>  >> aborted***Exception:   0.99 sec
>  >> FINE: CryptoManager: loading JSS library
>  >> FINE: CryptoManager: loaded JSS library from java.library.path
>  >> java: ../nptl/pthread_mutex_lock.c:81: __pthread_mutex_lock: Assertion
>  >> `mutex->__data.__owner == 0' failed.
>  >>
>  >>
>  >> Another, different stack trace here (pthreads fails to lock,
>  >> triggering a bug in opencryptoki):
>  >>
>  >> https://pagure.io/dogtagpki/issue/3181#comment-661911
>  > I don't see why this would be fixed by a mass rebuild.
>  >
>  > This is probably some sort of memory corruption: something has
>  > overwritten the mutex data structure, and glibc happens to detect that.
>  >
>  > Thanks,
>  > Florian
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
>  > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
>  > https://lists.fedoraproject.org/archives/list/[e-mail]
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>