Help needed with new segfaults in frame unwinding under gcc8
I could really use some help from the gcc experts.
A package I maintain, cyrus-imapd, contains two extensive test suites
which we run at package build time. After the big flag day where we
updated gcc and glibc and such in rawhide, one of the test suites now
shows failures and produces 22 core dumps, but only when run in mock
(not even fedpkg local on a rawhide container). Even in mock, if I
get into the chroot, duplicate the test environment and run the failing
program by hand (or under strace, or under gdb) then it doesn't
segfault.
After getting cores and all of the debugging stuff into mock
(instructions below) I found that all cores have substantially identical
backtraces:
(gdb) bt
#0 0x0000000000000120 in ?? ()
#1 0x00007f18a19d281e in _Unwind_ForcedUnwind_Phase2 (exc=0x7fffbc364c70,
context=0x7fffbc364990, frames_p=0x7fffbc364898) at ../../../libgcc/unwind.inc:170
#2 0x00007f18a19d3105 in _Unwind_Resume () at ../../../libgcc/unwind.inc:243
#3 0x00007f18a7dbbb90 in stem_version_set (version=<optimized out>,
database=<optimized out>) at /usr/include/c++/8/bits/char_traits.h:320
#4 xapian_dbw_open (paths=0x55aff951eb70, dbwp=0x55aff951f0f8) at
imap/xapian_wrap.cpp:327
#5 0x00007f18a7e2e2ef in begin_mailbox_update (rx=0x55aff951f060, mailbox=0x7f18a8c12018,
flags=0) at imap/search_xapian.c:1535
#6 0x00007f18a7e0f58b in search_update_mailbox (rx=0x55aff951f060,
mailbox=0x7f18a8c12018, flags=0) at imap/search_engines.c:211
#7 0x000055aff86b5105 in index_one (name=0x55aff951eb30 "user.cassandane",
blocking=1) at imap/squatter.c:292
#8 0x000055aff86b4774 in do_indexer (sa=0x7fffbc365370, sa=0x7fffbc365370) at
imap/squatter.c:352
#9 main (argc=3, argv=0x7fffbc3654c8) at imap/squatter.c:1004
Which as far as I've been able to gather from talking to folks on IRC,
its segfaulting in the stack unwinder trying to handle a C++ exception.
(The only C++ in the program is a wrapper for the Xapian search engine.)
I have done testing with older versions of Xapian (known to build the
package successfully) without any change in behavior, but I'm not sure I
have a reasonable way to roll back the gcc update.
The source is available from
https://github.com/cyrusimap/cyrus-imapd/tree/cyrus-imapd-3.0; the
specific function involved (stem_version_set) is
https://github.com/cyrusimap/cyrus-imapd/blob/cyrus-imapd-3.0/imap/xapian...
but it's only three lines.
I would appreciate any help from folks who can comprehend what's going
wrong here. Upstream is can't really offer much in the way of help;
given that the actual failure seems to happen in the libgcc code and
that this can't (so far) be reproduced outside of our buildsystem,
they're not sure what they can do.
Getting useful backtraces from a mock chroot:
If you turn off systemd coredump catching on the machine running mock
(sysctl -w 'kernel.core_pattern=core.%p') and delete the "%check" line
from the spec before building (so that the tests run as part of
%install), you can get the cores left in the mock chroot with the
complete debug information left unstripped. Then you can manually
install gdb and all of the debugfinfo packages:
mock -r fedora-rawhide-x86_64 --enablerepo fedora-debuginfo -i gdb
cyrus-sasl-lib-debuginfo glibc-debuginfo jansson-debuginfo
keyutils-libs-debuginfo krb5-libs-debuginfo libcom_err-debuginfo
libgcc-debuginfo libical-debuginfo libicu-debuginfo libnghttp2-debuginfo
libselinux-debuginfo libstdc++-debuginfo libuuid-debuginfo
libxcrypt-debuginfo libxml2-debuginfo nspr-debuginfo nss-debuginfo
nss-util-debuginfo openldap-debuginfo openssl-libs-debuginfo
pcre-debuginfo pcre2-debuginfo postgresql-libs-debuginfo
shapelib-debuginfo sqlite-libs-debuginfo xapian-core-libs-debuginfo
xz-libs-debuginfo zlib-debuginfo
Then run:
mock -r fedora-rawhide-x86_64 --shell
su - mockbuild
find . -name core.\*
and gdb as usual.
- J<