Testing / feedback request: DNF 3 crashes

[Adam Williamson]

Hi folks!

Around the time DNF 3 landed in Rawhide (hence F29), we had quite a few
folks on these lists reporting issues, including crasher bugs. Many of
these seemed somehow related to the DNF history database. They also
were not easy to isolate and fix.

We're now close to the F29 Beta release, but we don't have a great
sense of how many people are still having these problems with DNF 3.2
or 3.3.

Can anyone who is still struggling with DNF crashes on *basic*
operations on F29 or Rawhide please reply, and provide a few details on
what you're seeing and any workarounds or fixes you've found?

Thanks a lot!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Zbigniew Jędrzejewski-Szmek]

On Tue, Sep 11, 2018 at 12:26:09PM -0700, Adam Williamson wrote:
>  Hi folks!
>  
>  Around the time DNF 3 landed in Rawhide (hence F29), we had quite a few
>  folks on these lists reporting issues, including crasher bugs. Many of
>  these seemed somehow related to the DNF history database. They also
>  were not easy to isolate and fix.
>  
>  We're now close to the F29 Beta release, but we don't have a great
>  sense of how many people are still having these problems with DNF 3.2
>  or 3.3.
>  
>  Can anyone who is still struggling with DNF crashes on *basic*
>  operations on F29 or Rawhide please reply, and provide a few details on
>  what you're seeing and any workarounds or fixes you've found? 
I filed https://bugzilla.redhat.com/show_bug.cgi?id=1627694 yesterday.
The gist is that dnf stopped having locking around the rpm operation,
so it'll fail if two operations are invoked concurrently. Before it'd
very nicely do the downloads in parallel and then run the actual installations
sequentially. I think this is a huge regression in usability.

I also saw some "TransactionItem not found" messages, but this has already
been reported extensively.

Zbyszek

[Adam Williamson]

On Wed, 2018-09-12 at 06:27 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Tue, Sep 11, 2018 at 12:26:09PM -0700, Adam Williamson wrote:
>  > Hi folks!
>  > 
>  > Around the time DNF 3 landed in Rawhide (hence F29), we had quite a few
>  > folks on these lists reporting issues, including crasher bugs. Many of
>  > these seemed somehow related to the DNF history database. They also
>  > were not easy to isolate and fix.
>  > 
>  > We're now close to the F29 Beta release, but we don't have a great
>  > sense of how many people are still having these problems with DNF 3.2
>  > or 3.3.
>  > 
>  > Can anyone who is still struggling with DNF crashes on *basic*
>  > operations on F29 or Rawhide please reply, and provide a few details on
>  > what you're seeing and any workarounds or fixes you've found?
>  
>  I filed https://bugzilla.redhat.com/show_bug.cgi?id=1627694 yesterday.
>  The gist is that dnf stopped having locking around the rpm operation,
>  so it'll fail if two operations are invoked concurrently. Before it'd
>  very nicely do the downloads in parallel and then run the actual installations
>  sequentially. I think this is a huge regression in usability. 
Thanks. Doesn't really look like a Beta blocker, though.

>  I also saw some "TransactionItem not found" messages, but
> this has already
>  been reported extensively. 
On what operation? We have:
https://bugzilla.redhat.com/show_bug.cgi?id=1625259
for encountering them on downgrade or 'history undo' type operations.
If you hit one on upgrade or install, that'd be worse.

Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Zbigniew Jędrzejewski-Szmek]

On Wed, Sep 12, 2018 at 09:16:54AM -0700, Adam Williamson wrote:
>  On Wed, 2018-09-12 at 06:27 +0000, Zbigniew Jędrzejewski-Szmek
> wrote:
>  > On Tue, Sep 11, 2018 at 12:26:09PM -0700, Adam Williamson wrote:
>  > > Hi folks!
>  > > 
>  > > Around the time DNF 3 landed in Rawhide (hence F29), we had quite a few
>  > > folks on these lists reporting issues, including crasher bugs. Many of
>  > > these seemed somehow related to the DNF history database. They also
>  > > were not easy to isolate and fix.
>  > > 
>  > > We're now close to the F29 Beta release, but we don't have a great
>  > > sense of how many people are still having these problems with DNF 3.2
>  > > or 3.3.
>  > > 
>  > > Can anyone who is still struggling with DNF crashes on *basic*
>  > > operations on F29 or Rawhide please reply, and provide a few details on
>  > > what you're seeing and any workarounds or fixes you've found?
>  > 
>  > I filed https://bugzilla.redhat.com/show_bug.cgi?id=1627694 yesterday.
>  > The gist is that dnf stopped having locking around the rpm operation,
>  > so it'll fail if two operations are invoked concurrently. Before it'd
>  > very nicely do the downloads in parallel and then run the actual installations
>  > sequentially. I think this is a huge regression in usability.
>  
>  Thanks. Doesn't really look like a Beta blocker, though. 
It doesn't. I don't even think it'd be a Final blocker. Just unpleasant.

>  > I also saw some "TransactionItem not found" messages,
> but this has already
>  > been reported extensively.
>  
>  On what operation? We have:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1625259
>  for encountering them on downgrade or 'history undo' type operations.
>  If you hit one on upgrade or install, that'd be worse. 
https://bugzilla.redhat.com/show_bug.cgi?id=1627534

I marked it as duplicate, to avoid noise. Mikhail reported it during 'history
undo',
but I saw during 'dnf upgrade'.

Zbyszek

[Adam Williamson]

On Wed, 2018-09-12 at 16:33 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Wed, Sep 12, 2018 at 09:16:54AM -0700, Adam Williamson wrote:
>  > On Wed, 2018-09-12 at 06:27 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > > On Tue, Sep 11, 2018 at 12:26:09PM -0700, Adam Williamson wrote:
>  > > > Hi folks!
>  > > > 
>  > > > Around the time DNF 3 landed in Rawhide (hence F29), we had quite a few
>  > > > folks on these lists reporting issues, including crasher bugs. Many of
>  > > > these seemed somehow related to the DNF history database. They also
>  > > > were not easy to isolate and fix.
>  > > > 
>  > > > We're now close to the F29 Beta release, but we don't have a
> great
>  > > > sense of how many people are still having these problems with DNF 3.2
>  > > > or 3.3.
>  > > > 
>  > > > Can anyone who is still struggling with DNF crashes on *basic*
>  > > > operations on F29 or Rawhide please reply, and provide a few details on
>  > > > what you're seeing and any workarounds or fixes you've found?
>  > > 
>  > > I filed https://bugzilla.redhat.com/show_bug.cgi?id=1627694 yesterday.
>  > > The gist is that dnf stopped having locking around the rpm operation,
>  > > so it'll fail if two operations are invoked concurrently. Before it'd
>  > > very nicely do the downloads in parallel and then run the actual installations
>  > > sequentially. I think this is a huge regression in usability.
>  > 
>  > Thanks. Doesn't really look like a Beta blocker, though.
>  
>  It doesn't. I don't even think it'd be a Final blocker. Just unpleasant.
>  
>  > > I also saw some "TransactionItem not found" messages, but this has
> already
>  > > been reported extensively.
>  > 
>  > On what operation? We have:
>  > https://bugzilla.redhat.com/show_bug.cgi?id=1625259
>  > for encountering them on downgrade or 'history undo' type operations.
>  > If you hit one on upgrade or install, that'd be worse.
>  
>  https://bugzilla.redhat.com/show_bug.cgi?id=1627534
>  
>  I marked it as duplicate, to avoid noise. Mikhail reported it during 'history
> undo',
>  but I saw during 'dnf upgrade'. 
That seems significantly worse to me. I'm going to propose the bug as a
blocker and request input from dnf team. Thanks.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Alessio Ciregia]

On Tue, Sep 11, 2018 at 9:26 PM Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  Can anyone who is still struggling with DNF crashes on *basic*
>  operations on F29 or Rawhide please reply, and provide a few details on
>  what you're seeing and any workarounds or fixes you've found? 

FWIW, I never hit such problem.
The only way to get in troubles consists in hammering the history.sqlite file:
echo 1 > /var/lib/dnf/history.sqlite

Now almost all dnf operations fails (except dnf search):

dnf update
...
RuntimeError: C++ std::exception: Exec failed: file is not a database

And in the journal:
Sep 14 10:20:11 hostname python3[8830]: detected unhandled Python
exception in '/usr/bin/dnf'
Sep 14 10:20:11 hostname abrt-notification[8856]: Process 8830 (dnf)
of user 0 encountered an uncaught RuntimeError exception
-- Subject: ABRT has detected an uncaught RuntimeError exception in dnf
...

Deleting the "corrupted" history.sqlite, it restores the normal situation.

Ciao,
A.

[Samuel Rakitničan]

The following bug have not been updated since report. I would say it is a big issue since
an upgrade will break dnf,

https://bugzilla.redhat.com/show_bug.cgi?id=1598590

[Adam Williamson]

On Fri, 2018-09-14 at 14:41 +0000, Samuel Rakitničan wrote:
>  The following bug have not been updated since report. I would say it
> is a big issue since an upgrade will break dnf,
>  
>  https://bugzilla.redhat.com/show_bug.cgi?id=1598590 
I've closed it as a dupe of the bug we're using as the main report for
this. I agree it's a significant issue for those who hit it; it's
currently proposed as a Final blocker, I'll mark it CommonBugs so we
document the workaround (wipe history) for Beta, and if a fix comes
soon, we can consider a freeze exception to get it into Beta.

Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Peter Hutterer]

On Tue, Sep 11, 2018 at 12:26:09PM -0700, Adam Williamson wrote:
>  Hi folks!
>  
>  Around the time DNF 3 landed in Rawhide (hence F29), we had quite a few
>  folks on these lists reporting issues, including crasher bugs. Many of
>  these seemed somehow related to the DNF history database. They also
>  were not easy to isolate and fix.
>  
>  We're now close to the F29 Beta release, but we don't have a great
>  sense of how many people are still having these problems with DNF 3.2
>  or 3.3.
>  
>  Can anyone who is still struggling with DNF crashes on *basic*
>  operations on F29 or Rawhide please reply, and provide a few details on
>  what you're seeing and any workarounds or fixes you've found?
>  
>  Thanks a lot! 
fwiw, I just ran into https://bugzilla.redhat.com/show_bug.cgi?id=1632177
after a dnf system-upgrade and it crashed at least the dnf
history/update/remove commands (whenever it started dependency resolution):

Completion plugin: Generating completion cache...
--> Starting dependency resolution
zsh: segmentation fault  sudo dnf update --debuglevel 10

moving /var/lib/dnf out of the way fixed it but that's a less than polished
experience nonetheless ;)

Cheers,
   Peter

[Adam Williamson]

On Fri, 2018-10-12 at 09:17 +1000, Peter Hutterer wrote:
>  On Tue, Sep 11, 2018 at 12:26:09PM -0700, Adam Williamson wrote:
>  > Hi folks!
>  > 
>  > Around the time DNF 3 landed in Rawhide (hence F29), we had quite a few
>  > folks on these lists reporting issues, including crasher bugs. Many of
>  > these seemed somehow related to the DNF history database. They also
>  > were not easy to isolate and fix.
>  > 
>  > We're now close to the F29 Beta release, but we don't have a great
>  > sense of how many people are still having these problems with DNF 3.2
>  > or 3.3.
>  > 
>  > Can anyone who is still struggling with DNF crashes on *basic*
>  > operations on F29 or Rawhide please reply, and provide a few details on
>  > what you're seeing and any workarounds or fixes you've found?
>  > 
>  > Thanks a lot!
>  
>  fwiw, I just ran into https://bugzilla.redhat.com/show_bug.cgi?id=1632177
>  after a dnf system-upgrade and it crashed at least the dnf
>  history/update/remove commands (whenever it started dependency resolution):
>  
>  Completion plugin: Generating completion cache...
>  --> Starting dependency resolution
>  zsh: segmentation fault  sudo dnf update --debuglevel 10
>  
>  moving /var/lib/dnf out of the way fixed it but that's a less than polished
>  experience nonetheless ;) 
Thanks for the note, it seems 3.6 / 0.20 did not fix all these problems
:/ I have nominated that bug as a Final blocker.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Tomasz Torcz]

On Thu, Oct 11, 2018 at 04:23:03PM -0700, Adam Williamson wrote:
>  > 
>  > fwiw, I just ran into https://bugzilla.redhat.com/show_bug.cgi?id=1632177
>  > after a dnf system-upgrade and it crashed at least the dnf
>  > history/update/remove commands (whenever it started dependency resolution):
>  
>  Thanks for the note, it seems 3.6 / 0.20 did not fix all these problems
>  :/ I have nominated that bug as a Final blocker. 
  The https://bugzilla.redhat.com/show_bug.cgi?id=1600917 looks like a
 duplicate of above big. It also have some info and example of
 crash-inducing history database.

-- 
Tomasz Torcz               "Never underestimate the bandwidth of a station
xmpp: zdzichubg(a)chrome.pl    wagon filled with backup tapes." -- Jim Gray

[Adam Williamson]

On Fri, 2018-10-12 at 07:38 +0200, Tomasz Torcz wrote:
>  On Thu, Oct 11, 2018 at 04:23:03PM -0700, Adam Williamson wrote:
>  > > fwiw, I just ran into https://bugzilla.redhat.com/show_bug.cgi?id=1632177
>  > > after a dnf system-upgrade and it crashed at least the dnf
>  > > history/update/remove commands (whenever it started dependency resolution):
>  > 
>  > Thanks for the note, it seems 3.6 / 0.20 did not fix all these problems
>  > :/ I have nominated that bug as a Final blocker.
>  
>    The https://bugzilla.redhat.com/show_bug.cgi?id=1600917 looks like a
>   duplicate of above big. It also have some info and example of
>   crash-inducing history database. 
They both have that (see comment #2 of 1632177). I'll mark it as 'see
also' and let dmach decide if they're dupes. thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Miroslav Suchý]

Dne 11.9.2018 v 21:26 Adam Williamson napsal(a):
>  Can anyone who is still struggling with DNF crashes on *basic*
>  operations on F29 or Rawhide please reply, and provide a few details on
>  what you're seeing and any workarounds or fixes you've found? 

Obviously:
https://retrace.fedoraproject.org/faf/reports/?opsysreleases=20&opsys...

This is for last 14 days.

Miroslav

[Adam Williamson]

On Fri, 2018-10-12 at 09:07 +0200, Miroslav Suchý wrote:
>  Dne 11.9.2018 v 21:26 Adam Williamson napsal(a):
>  > Can anyone who is still struggling with DNF crashes on *basic*
>  > operations on F29 or Rawhide please reply, and provide a few details on
>  > what you're seeing and any workarounds or fixes you've found?
>  
>  Obviously:
>
> https://retrace.fedoraproject.org/faf/reports/?opsysreleases=20&opsys...
>  
>  This is for last 14 days. 
A lot of those look like they're basically:

https://bugzilla.redhat.com/show_bug.cgi?id=1631533

All the ones that wind up in line 729 of libdnf/transaction.py are
likely that.

Some of the others look interesting, but without a bug report it's hard
to know what they might be :/
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net