Koji ppc64le build failure

[Richard W.M. Jones]

https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509

On ppc64le:
FAIL: test-checkwrite.sh

There should be a core dump associated with this failure.  Is it
possible someone with Koji access could look to see if this is still
around and capture it?  (Even just a full stack trace would be useful)

We've been seeing this error occasionally (same test, only on ppc64le)
for a few weeks.  I reserved a ppc64le machine at Red Hat last week
and ran this test on basically exactly the same combination of
software thousands of times, and it didn't fail even once in that
time, so I'm out of ideas how to reproduce it for myself.

Rich.

[Richard W.M. Jones]

On Sun, Jul 17, 2022 at 08:34:30PM +0100, Richard W.M. Jones wrote:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
>
> On ppc64le:
> FAIL: test-checkwrite.sh
>
> There should be a core dump associated with this failure.  Is it
> possible someone with Koji access could look to see if this is still
> around and capture it?  (Even just a full stack trace would be useful)
>
> We've been seeing this error occasionally (same test, only on ppc64le)
> for a few weeks.  I reserved a ppc64le machine at Red Hat last week
> and ran this test on basically exactly the same combination of
> software thousands of times, and it didn't fail even once in that
> time, so I'm out of ideas how to reproduce it for myself.

I should say the error is intermittent.  Usually repeating the build
fixes it, which is what I'm going to do now.

Rich.

[Dan Horák]

On Sun, 17 Jul 2022 20:34:30 +0100
"Richard W.M. Jones"  wrote:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
>
> On ppc64le:
> FAIL: test-checkwrite.sh
>
> There should be a core dump associated with this failure.  Is it
> possible someone with Koji access could look to see if this is still
> around and capture it?  (Even just a full stack trace would be useful)
>
> We've been seeing this error occasionally (same test, only on ppc64le)
> for a few weeks.  I reserved a ppc64le machine at Red Hat last week
> and ran this test on basically exactly the same combination of
> software thousands of times, and it didn't fail even once in that
> time, so I'm out of ideas how to reproduce it for myself.

I don't have time now, but the failed chroot should be available for 24h
at least I believe, I can check that in the morning

Dan

[Kevin Fenzi]

On Sun, Jul 17, 2022 at 08:34:30PM +0100, Richard W.M. Jones wrote:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
>
> On ppc64le:
> FAIL: test-checkwrite.sh
>
> There should be a core dump associated with this failure.  Is it
> possible someone with Koji access could look to see if this is still
> around and capture it?  (Even just a full stack trace would be useful)
>
> We've been seeing this error occasionally (same test, only on ppc64le)
> for a few weeks.  I reserved a ppc64le machine at Red Hat last week
> and ran this test on basically exactly the same combination of
> software thousands of times, and it didn't fail even once in that
> time, so I'm out of ideas how to reproduce it for myself.

Core at: 
https://infrastructure.fedoraproject.org/infra/tmp/core.nbdkit.1000.9be14c79...

Module libtasn1.so.6 with build-id 460fc6bd3f93e57e7d593080f58952f61cdb344a
                Stack trace of thread 2744170:
                #0  0x00007fffa9b59b6c n/a (/usr/lib64/libc.so.6 + 0xb9b6c)
                ELF object binary architecture: PowerPC64

kevin

[Richard W.M. Jones]

On Sun, Jul 17, 2022 at 12:50:01PM -0700, Kevin Fenzi wrote:
> On Sun, Jul 17, 2022 at 08:34:30PM +0100, Richard W.M. Jones wrote:
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
> >
> > On ppc64le:
> > FAIL: test-checkwrite.sh
> >
> > There should be a core dump associated with this failure.  Is it
> > possible someone with Koji access could look to see if this is still
> > around and capture it?  (Even just a full stack trace would be useful)
> >
> > We've been seeing this error occasionally (same test, only on ppc64le)
> > for a few weeks.  I reserved a ppc64le machine at Red Hat last week
> > and ran this test on basically exactly the same combination of
> > software thousands of times, and it didn't fail even once in that
> > time, so I'm out of ideas how to reproduce it for myself.
>
> Core at: 
> https://infrastructure.fedoraproject.org/infra/tmp/core.nbdkit.1000.9be14c79...
>             Module libtasn1.so.6 with build-id 460fc6bd3f93e57e7d593080f58952f61cdb344a
>             Stack trace of thread 2744170:
>             #0  0x00007fffa9b59b6c n/a (/usr/lib64/libc.so.6 + 0xb9b6c)
>             ELF object binary architecture: PowerPC64

Thanks - I've grabbed that file.

Rich.

[Richard W.M. Jones]

On Sun, Jul 17, 2022 at 12:50:01PM -0700, Kevin Fenzi wrote:
> On Sun, Jul 17, 2022 at 08:34:30PM +0100, Richard W.M. Jones wrote:
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
> >
> > On ppc64le:
> > FAIL: test-checkwrite.sh
> >
> > There should be a core dump associated with this failure.  Is it
> > possible someone with Koji access could look to see if this is still
> > around and capture it?  (Even just a full stack trace would be useful)
> >
> > We've been seeing this error occasionally (same test, only on ppc64le)
> > for a few weeks.  I reserved a ppc64le machine at Red Hat last week
> > and ran this test on basically exactly the same combination of
> > software thousands of times, and it didn't fail even once in that
> > time, so I'm out of ideas how to reproduce it for myself.
>
> Core at: 
> https://infrastructure.fedoraproject.org/infra/tmp/core.nbdkit.1000.9be14c79...
>             Module libtasn1.so.6 with build-id 460fc6bd3f93e57e7d593080f58952f61cdb344a
>             Stack trace of thread 2744170:
>             #0  0x00007fffa9b59b6c n/a (/usr/lib64/libc.so.6 + 0xb9b6c)
>             ELF object binary architecture: PowerPC64

Sorry, I realised that the core dump won't work without the
executable.  Is <builddir>/server/nbdkit still around?  (Note there's
another "nbdkit" binary in the top build directory, which is _not_ the
server.)

Rich.

[Kevin Fenzi]

On Mon, Jul 18, 2022 at 11:59:49AM +0100, Richard W.M. Jones wrote:
> On Sun, Jul 17, 2022 at 12:50:01PM -0700, Kevin Fenzi wrote:
> > On Sun, Jul 17, 2022 at 08:34:30PM +0100, Richard W.M. Jones wrote:
> > > https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
> > >
> > > On ppc64le:
> > > FAIL: test-checkwrite.sh
> > >
> > > There should be a core dump associated with this failure.  Is it
> > > possible someone with Koji access could look to see if this is still
> > > around and capture it?  (Even just a full stack trace would be useful)
> > >
> > > We've been seeing this error occasionally (same test, only on ppc64le)
> > > for a few weeks.  I reserved a ppc64le machine at Red Hat last week
> > > and ran this test on basically exactly the same combination of
> > > software thousands of times, and it didn't fail even once in that
> > > time, so I'm out of ideas how to reproduce it for myself.
> >
> > Core at: 
> > https://infrastructure.fedoraproject.org/infra/tmp/core.nbdkit.1000.9be14c79...
> >             Module libtasn1.so.6 with build-id 460fc6bd3f93e57e7d593080f58952f61cdb344a
> >             Stack trace of thread 2744170:
> >             #0  0x00007fffa9b59b6c n/a (/usr/lib64/libc.so.6 + 0xb9b6c)
> >             ELF object binary architecture: PowerPC64
>
> Sorry, I realised that the core dump won't work without the
> executable.  Is <builddir>/server/nbdkit still around?  (Note there's
> another "nbdkit" binary in the top build directory, which is _not_ the
> server.)

To circle back to the list, it sounds like you got it to happen locally?

kevin

[Dan Horák]

On Mon, 18 Jul 2022 11:39:58 -0700
Kevin Fenzi  wrote:
> On Mon, Jul 18, 2022 at 11:59:49AM +0100, Richard W.M. Jones wrote:
> > On Sun, Jul 17, 2022 at 12:50:01PM -0700, Kevin Fenzi wrote:
> > > On Sun, Jul 17, 2022 at 08:34:30PM +0100, Richard W.M. Jones wrote:
> > > > https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
> > > >
> > > > On ppc64le:
> > > > FAIL: test-checkwrite.sh
> > > >
> > > > There should be a core dump associated with this failure.  Is it
> > > > possible someone with Koji access could look to see if this is still
> > > > around and capture it?  (Even just a full stack trace would be useful)
> > > >
> > > > We've been seeing this error occasionally (same test, only on ppc64le)
> > > > for a few weeks.  I reserved a ppc64le machine at Red Hat last week
> > > > and ran this test on basically exactly the same combination of
> > > > software thousands of times, and it didn't fail even once in that
> > > > time, so I'm out of ideas how to reproduce it for myself.
> > >
> > > Core at: 
> > > https://infrastructure.fedoraproject.org/infra/tmp/core.nbdkit.1000.9be14c79...
> > >             Module libtasn1.so.6 with build-id 460fc6bd3f93e57e7d593080f58952f61cdb344a
> > >             Stack trace of thread 2744170:
> > >             #0  0x00007fffa9b59b6c n/a (/usr/lib64/libc.so.6 + 0xb9b6c)
> > >             ELF object binary architecture: PowerPC64
> >
> > Sorry, I realised that the core dump won't work without the
> > executable.  Is <builddir>/server/nbdkit still around?  (Note there's
> > another "nbdkit" binary in the top build directory, which is _not_ the
> > server.)
>
> To circle back to the list, it sounds like you got it to happen locally?

I don't think so, I believe Rich needs the binary that produced the
coredump. I have grabbed it and it's now
http://fedora.danny.cz/tmp/nbdkit

Dan

[Richard W.M. Jones]

On Mon, Jul 18, 2022 at 09:28:42PM +0200, Dan Horák wrote:
> On Mon, 18 Jul 2022 11:39:58 -0700
> Kevin Fenzi  wrote:
> > On Mon, Jul 18, 2022 at 11:59:49AM +0100, Richard W.M. Jones wrote:
> > > On Sun, Jul 17, 2022 at 12:50:01PM -0700, Kevin Fenzi wrote:
> > > > On Sun, Jul 17, 2022 at 08:34:30PM +0100, Richard W.M. Jones wrote:
> > > > > https://koji.fedoraproject.org/koji/taskinfo?taskID=89618509
> > > > >
> > > > > On ppc64le:
> > > > > FAIL: test-checkwrite.sh
> > > > >
> > > > > There should be a core dump associated with this failure.  Is it
> > > > > possible someone with Koji access could look to see if this is still
> > > > > around and capture it?  (Even just a full stack trace would be useful)
> > > > >
> > > > > We've been seeing this error occasionally (same test, only on ppc64le)
> > > > > for a few weeks.  I reserved a ppc64le machine at Red Hat last week
> > > > > and ran this test on basically exactly the same combination of
> > > > > software thousands of times, and it didn't fail even once in that
> > > > > time, so I'm out of ideas how to reproduce it for myself.
> > > >
> > > > Core at: 
> > > > https://infrastructure.fedoraproject.org/infra/tmp/core.nbdkit.1000.9be14c79...
> > > >             Module libtasn1.so.6 with build-id 460fc6bd3f93e57e7d593080f58952f61cdb344a
> > > >             Stack trace of thread 2744170:
> > > >             #0  0x00007fffa9b59b6c n/a (/usr/lib64/libc.so.6 + 0xb9b6c)
> > > >             ELF object binary architecture: PowerPC64
> > >
> > > Sorry, I realised that the core dump won't work without the
> > > executable.  Is <builddir>/server/nbdkit still around?  (Note there's
> > > another "nbdkit" binary in the top build directory, which is _not_ the
> > > server.)
> >
> > To circle back to the list, it sounds like you got it to happen locally?
>
> I don't think so, I believe Rich needs the binary that produced the
> coredump. I have grabbed it and it's now
> http://fedora.danny.cz/tmp/nbdkit

Thanks for your help!  I did finally manage to reproduce it locally
(although not fix it so far ...).

Rich.