arm 32 bit fedora quit booting

[Timothy Krantz]

I have several 32 bit arm machines running fedora 35.  I update them via dnf
on an almost daily basis.

Yesterday afternoon I had a brief power failure and now none of them will
boot or boot into an unusable state.  At first I suspected the power blip
was the culprit but now I am not so sure.  I am getting similar behavior on
all of them.

Here is a capture of my wandboard quad :

https://pastebin.com/fVQYCXML

The first thing that I see that looks bad is about 26 seconds in :

[   26.070965] systemd-journald[234]: Received SIGTERM from PID 1 (systemd).

But maybe that is a red herring.  Things do go south from there.

Similar issues on a lime2 and a ras pi running 32 bit code.

Any suggestions would be appreciated.

Tim Krantz

[Jeffrey Walton]

On Wed, Dec 29, 2021 at 2:34 PM Timothy Krantz
 wrote:
> I have several 32 bit arm machines running fedora 35.  I update them via dnf on an almost daily basis.
>
> Yesterday afternoon I had a brief power failure and now none of them will boot or boot into an unusable state.  At first I suspected the power blip was the culprit but now I am not so sure.  I am getting similar behavior on all of them.
>
> Here is a capture of my wandboard quad :
>
> https://pastebin.com/fVQYCXML
>
> The first thing that I see that looks bad is about 26 seconds in :
>
> [   26.070965] systemd-journald[234]: Received SIGTERM from PID 1 (systemd).
>
> But maybe that is a red herring.  Things do go south from there.
>
> Similar issues on a lime2 and a ras pi running 32 bit code.
>
> Any suggestions would be appreciated.

It sounds like a hardware problem due to the earlier power problems.
There's no way a software update could be interrupted at the same
point on several machines, and produce the same exact error during
boot on those machines.

Check the power warts. They are cheap and go bad on their own. They
don't need a power surge or brownout to smoke them. That you had a
[significant enough] power event was probably more than enough to
break them.

Or, check your power distribution strip if the machines are plugged
into the same strip.

Jeff

[Kevin Fenzi]

On Wed, Dec 29, 2021 at 07:08:19PM -0500, Jeffrey Walton wrote:
> On Wed, Dec 29, 2021 at 2:34 PM Timothy Krantz
>  wrote:
> > I have several 32 bit arm machines running fedora 35.  I update them via dnf on an almost daily basis.
> >
> > Yesterday afternoon I had a brief power failure and now none of them will boot or boot into an unusable state.  At first I suspected the power blip was the culprit but now I am not so sure.  I am getting similar behavior on all of them.
> >
> > Here is a capture of my wandboard quad :
> >
> > https://pastebin.com/fVQYCXML
> >
> > The first thing that I see that looks bad is about 26 seconds in :
> >
> > [   26.070965] systemd-journald[234]: Received SIGTERM from PID 1 (systemd).
> >
> > But maybe that is a red herring.  Things do go south from there.
> >
> > Similar issues on a lime2 and a ras pi running 32 bit code.
> >
> > Any suggestions would be appreciated.
>
> It sounds like a hardware problem due to the earlier power problems.
> There's no way a software update could be interrupted at the same
> point on several machines, and produce the same exact error during
> boot on those machines.
>
> Check the power warts. They are cheap and go bad on their own. They
> don't need a power surge or brownout to smoke them. That you had a
> [significant enough] power event was probably more than enough to
> break them.
>
> Or, check your power distribution strip if the machines are plugged
> into the same strip.

Might be:

https://bugzilla.redhat.com/show_bug.cgi?id=2035802

(ie, libzstd breaking things, try downgrade/upgrading?)

kevin

[Jeffrey Walton]

On Wed, Dec 29, 2021 at 7:39 PM Kevin Fenzi  wrote:
> On Wed, Dec 29, 2021 at 07:08:19PM -0500, Jeffrey Walton wrote:
> > On Wed, Dec 29, 2021 at 2:34 PM Timothy Krantz
> >  wrote:
> > > ...
> > > Yesterday afternoon I had a brief power failure and now none of them will boot or boot into an unusable state.  At first I suspected the power blip was the culprit but now I am not so sure.  I am getting similar behavior on all of them.
> > > ...
>
> Might be:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=2035802
>
> (ie, libzstd breaking things, try downgrade/upgrading?)

Good find. I'm not sure why I thought the update process failed at the
same spot on different machines...

Jeff

[Michael Whapples]

Thanks for the answer about libzstd. On 27 Dec I also had a problem with 
booting my Odroid XU4. However I needed that system back up and running 
so I didn't dig too much into what was wrong, I just knew it was one of 
the recent updates. Probably was that libzstd bug as that was in the 
list of updates which did break things for me.

Michael Whapples

On 30/12/2021 00:39, Kevin Fenzi wrote:
> On Wed, Dec 29, 2021 at 07:08:19PM -0500, Jeffrey Walton wrote:
> > On Wed, Dec 29, 2021 at 2:34 PM Timothy Krantz
> >  wrote:
> > > I have several 32 bit arm machines running fedora 35.  I update them via dnf on an almost daily basis.
> > >
> > > Yesterday afternoon I had a brief power failure and now none of them will boot or boot into an unusable state.  At first I suspected the power blip was the culprit but now I am not so sure.  I am getting similar behavior on all of them.
> > >
> > > Here is a capture of my wandboard quad :
> > >
> > > https://pastebin.com/fVQYCXML
> > >
> > > The first thing that I see that looks bad is about 26 seconds in :
> > >
> > > [   26.070965] systemd-journald[234]: Received SIGTERM from PID 1 (systemd).
> > >
> > > But maybe that is a red herring.  Things do go south from there.
> > >
> > > Similar issues on a lime2 and a ras pi running 32 bit code.
> > >
> > > Any suggestions would be appreciated.
> >
> > It sounds like a hardware problem due to the earlier power problems.
> > There's no way a software update could be interrupted at the same
> > point on several machines, and produce the same exact error during
> > boot on those machines.
> >
> > Check the power warts. They are cheap and go bad on their own. They
> > don't need a power surge or brownout to smoke them. That you had a
> > [significant enough] power event was probably more than enough to
> > break them.
> >
> > Or, check your power distribution strip if the machines are plugged
> > into the same strip.
>
> Might be:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=2035802
>
> (ie, libzstd breaking things, try downgrade/upgrading?)
>
> kevin
>
> arm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Timothy Krantz]

Thank you!

I have resurrected my machine's!  That might teach me to not be such a
degenerate updater! (probably not)

Tim
> -----Original Message-----
> From: Kevin Fenzi [mailto:[e-mail]]
> Sent: Wednesday, December 29, 2021 7:39 PM
> To: Fedora ARM secondary architecture list 
> Subject: [fedora-arm] Re: arm 32 bit fedora quit booting
>
> On Wed, Dec 29, 2021 at 07:08:19PM -0500, Jeffrey Walton wrote:
> > On Wed, Dec 29, 2021 at 2:34 PM Timothy Krantz
> >  wrote:
> > > I have several 32 bit arm machines running fedora 35.  I update them

via
> dnf on an almost daily basis.
> > > Yesterday afternoon I had a brief power failure and now none of them
>
> will boot or boot into an unusable state.  At first I suspected the power

blip
> was the culprit but now I am not so sure.  I am getting similar behavior

on all
> of them.
> > > Here is a capture of my wandboard quad :
> > >
> > > https://pastebin.com/fVQYCXML
> > >
> > > The first thing that I see that looks bad is about 26 seconds in :
> > >
> > > [   26.070965] systemd-journald[234]: Received SIGTERM from PID 1
>
> (systemd).
> > > But maybe that is a red herring.  Things do go south from there.
> > >
> > > Similar issues on a lime2 and a ras pi running 32 bit code.
> > >
> > > Any suggestions would be appreciated.
> >
> > It sounds like a hardware problem due to the earlier power problems.
> > There's no way a software update could be interrupted at the same
> > point on several machines, and produce the same exact error during
> > boot on those machines.
> >
> > Check the power warts. They are cheap and go bad on their own. They
> > don't need a power surge or brownout to smoke them. That you had a
> > [significant enough] power event was probably more than enough to
> > break them.
> >
> > Or, check your power distribution strip if the machines are plugged
> > into the same strip.
>
> Might be:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=2035802
>
> (ie, libzstd breaking things, try downgrade/upgrading?)
>
> kevin

[Peter Robinson]

On Thu, Dec 30, 2021 at 3:26 PM Timothy Krantz
 wrote:
> Thank you!
>
> I have resurrected my machine's!  That might teach me to not be such a
> degenerate updater! (probably not)

I would like to think it would teach the maintainer to push to rawhide
and let it sit for a few days and then not to undermine the karma
setting on bodhi but then I suspect not.....
> Tim
> > -----Original Message-----
> > From: Kevin Fenzi [mailto:[e-mail]]
> > Sent: Wednesday, December 29, 2021 7:39 PM
> > To: Fedora ARM secondary architecture list 
> > Subject: [fedora-arm] Re: arm 32 bit fedora quit booting
> >
> > On Wed, Dec 29, 2021 at 07:08:19PM -0500, Jeffrey Walton wrote:
> > > On Wed, Dec 29, 2021 at 2:34 PM Timothy Krantz
> > >  wrote:
> > > > I have several 32 bit arm machines running fedora 35.  I update them
>
> via
> > dnf on an almost daily basis.
> > > > Yesterday afternoon I had a brief power failure and now none of them
> >
> > will boot or boot into an unusable state.  At first I suspected the power
>
> blip
> > was the culprit but now I am not so sure.  I am getting similar behavior
>
> on all
> > of them.
> > > > Here is a capture of my wandboard quad :
> > > >
> > > > https://pastebin.com/fVQYCXML
> > > >
> > > > The first thing that I see that looks bad is about 26 seconds in :
> > > >
> > > > [   26.070965] systemd-journald[234]: Received SIGTERM from PID 1
> >
> > (systemd).
> > > > But maybe that is a red herring.  Things do go south from there.
> > > >
> > > > Similar issues on a lime2 and a ras pi running 32 bit code.
> > > >
> > > > Any suggestions would be appreciated.
> > >
> > > It sounds like a hardware problem due to the earlier power problems.
> > > There's no way a software update could be interrupted at the same
> > > point on several machines, and produce the same exact error during
> > > boot on those machines.
> > >
> > > Check the power warts. They are cheap and go bad on their own. They
> > > don't need a power surge or brownout to smoke them. That you had a
> > > [significant enough] power event was probably more than enough to
> > > break them.
> > >
> > > Or, check your power distribution strip if the machines are plugged
> > > into the same strip.
> >
> > Might be:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=2035802
> >
> > (ie, libzstd breaking things, try downgrade/upgrading?)
> >
> > kevin
>
> arm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure