Issues with the retrace service?

[Peter Robinson]

Hi all,

It seems with F-31 on x86_64 the retrace service seems to fail every
single time I try to submit a crash report, is there known issues with
the service?

Peter

[Frantisek Zatloukal]

Hi,

I think retrace service is running on RHEL 7.7 server and since we have
zstd rpms in Fedora 31 [0], it won't work. I don't know what's the ETA for
fix, I can ask on Tuesday, if there is any update on this issue.

Until then, you'll probably have to generate stack traces locally :/

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1720729

[Peter Robinson]

> I think retrace service is running on RHEL 7.7 server and since we have zstd rpms in Fedora 31 [0], it won't work. I don't know what's the ETA for fix, I can ask on Tuesday, if there is any update on this issue.

It's a problem on RHEL-8 too then.
> Until then, you'll probably have to generate stack traces locally :/

Well we should probably disable the retrace and default to local on
the releases that it's a problem because it's not exactly fast and if
it's going to fail every time it's not exactly a fantastic experience
for the user, same goes if we don't support the architecture if it's
not supported for retrace.

Peter

[Neal Gompa]

On Sat, Oct 26, 2019 at 6:18 PM Peter Robinson  wrote:
> > I think retrace service is running on RHEL 7.7 server and since we have zstd rpms in Fedora 31 [0], it won't work. I don't know what's the ETA for fix, I can ask on Tuesday, if there is any update on this issue.
>
> It's a problem on RHEL-8 too then.

Maybe it'd be possible to swing a RHEL 8 system with the updated rpm
with zstd support? The RHEL bug[1] indicates that there's a build with
it enabled...

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1715799

[Miroslav Suchý]

Dne 26. 10. 19 v 22:16 Frantisek Zatloukal napsal(a):
> > I think retrace service is running on RHEL 7.7 server and since we have zstd rpms in Fedora 31 [0], it won't work. I
>
> don't know what's the ETA for fix, I can ask on Tuesday, if there is any update on this issue.

Right. The issue is zstd. Retrace-server uses Mock to install packages. Previously, Mock on RHEL7 was unable to install 
Fedora 31 because of zstd. This changed in Mock 1.4.20 [1], which hit stable in EPEL only 10 days ago [2].
We tested the retrace-server with the new Mock with retrace-server and we discovered another bug [3] in Mock which is 
not yet fixed. I want to fix it soon.
At the same time, Michal Fabik from ABRT team is working on using Podman directly instead of Mock for retracing. This 
should be done soon as well.

We need to finish one of those tasks. Until then, the retrace of F31+ will fail.

[1] https://github.com/rpm-software-management/mock/wiki/Release-Notes-1.4.20
[2] https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-ddf315f3b1
[3] https://github.com/rpm-software-management/mock/issues/381

[Peter Robinson]

> Dne 26. 10. 19 v 22:16 Frantisek Zatloukal napsal(a):
> > > I think retrace service is running on RHEL 7.7 server and since we have zstd rpms in Fedora 31 [0], it won't work. I
> >
> > don't know what's the ETA for fix, I can ask on Tuesday, if there is any update on this issue.
>
> Right. The issue is zstd. Retrace-server uses Mock to install packages. Previously, Mock on RHEL7 was unable to install
> Fedora 31 because of zstd. This changed in Mock 1.4.20 [1], which hit stable in EPEL only 10 days ago [2].
> We tested the retrace-server with the new Mock with retrace-server and we discovered another bug [3] in Mock which is
> not yet fixed. I want to fix it soon.
> At the same time, Michal Fabik from ABRT team is working on using Podman directly instead of Mock for retracing. This
> should be done soon as well.
>
> We need to finish one of those tasks. Until then, the retrace of F31+ will fail.

Which is a slow and terrible user experience, yes changes like this
take time but the sane thing to do would be to disable the retrace
option and default to local processing until it's fixed.
> [1] https://github.com/rpm-software-management/mock/wiki/Release-Notes-1.4.20
> [2] https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-ddf315f3b1
> [3] https://github.com/rpm-software-management/mock/issues/381
>
> --
> Miroslav Suchy, RHCA
> Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Chris Murphy]

I filed this awhile ago, but didn't pursue getting it disabled on the
Fedora side in the meantime. While I'm a fan of zstd, I think in hindsight
I'd have recommended postponing rolling out that change until a retrace
server solution was in place. I'd definitely say this should have been a
listed liability of the change to use zstd.

https://github.com/abrt/retrace-server/issues/258

--
Chris Murphy

[Tomasz Torcz]

On Sun, Oct 27, 2019 at 11:10:48AM +0100, Chris Murphy wrote:
> I filed this awhile ago, but didn't pursue getting it disabled on the
> Fedora side in the meantime. While I'm a fan of zstd, I think in hindsight
> I'd have recommended postponing rolling out that change until a retrace
> server solution was in place. I'd definitely say this should have been a
> listed liability of the change to use zstd.

This is a fallout of us not “dogfooding”, but running our infrastructure
on some other distributions.

[Neal Gompa]

On Sun, Oct 27, 2019 at 6:52 AM Tomasz Torcz  wrote:
> On Sun, Oct 27, 2019 at 11:10:48AM +0100, Chris Murphy wrote:
> > I filed this awhile ago, but didn't pursue getting it disabled on the
> > Fedora side in the meantime. While I'm a fan of zstd, I think in hindsight
> > I'd have recommended postponing rolling out that change until a retrace
> > server solution was in place. I'd definitely say this should have been a
> > listed liability of the change to use zstd.
>
> This is a fallout of us not “dogfooding”, but running our infrastructure
> on some other distributions.

I agree. While RHEL is part of the Fedora family, we can't really do
much for it. At least we'll eventually get zstd support in RHEL 8. I
suppose we could rebuild rpm with zstd support for the server before
then, since it's just a bcond away...