F32 upgrade fails (boots into Emergency mode)

[Ranjan Maitra]

Hi,

I was commenting yesterday afternoon on how painless the upgrade had been so far for me (4
laptops, 1 workstation). Got jinxed on a workstation at work! For me, F31 got upgraded to
F32 using the methods oulined in
https://fedoramagazine.org/upgrading-fedora-31-to-fedora-32/. All steps went smoothly,
until it was time to come up: I was thrown into emergency mode after the upgrade.

As instructed by the emergency mode, here is the output of journalctl and rdsosreport.txt

Note that I have changed the hostname to hostname.suppressed (I don't know what else
has the machine's identity even though it behind VPN).

Output of journalctl:

$ fpaste journal.txt
Uploading (115.3KiB)...
https://paste.centos.org/view/b759a587

rdsosreport.txt:

$ fpaste rdsosreport.txt
Uploading (136.7KiB)...
https://paste.centos.org/view/ec37f118

Any suggestions as to what is wrong? The machine is sort of hard to get to simply because
it is not online now, and I have to go in to get more information but I will try and
provide additional requests for information in as timely a manner as possible.

Many thanks again and best wishes,
Ranjan

[Sam Varshavchik]

Ranjan Maitra writes:

>  Hi,
>
>  I was commenting yesterday afternoon on how painless the upgrade had been so  
>  far for me (4 laptops, 1 workstation). Got jinxed on a workstation at work!  
>  For me, F31 got upgraded to F32 using the methods oulined in  
>  https://fedoramagazine.org/upgrading-fedora-31-to-fedora-32/. All steps went  
>  smoothly, until it was time to come up: I was thrown into emergency mode  
>  after the upgrade.
>
>  As instructed by the emergency mode, here is the output of journalctl and  
>  rdsosreport.txt
>
>  Note that I have changed the hostname to hostname.suppressed (I don't know  
>  what else has the machine's identity even though it behind VPN).
>
>  Output of journalctl:
>
>  $ fpaste journal.txt
>  Uploading (115.3KiB)...
>  https://paste.centos.org/view/b759a587
>
>  rdsosreport.txt:
>
>  $ fpaste rdsosreport.txt
>  Uploading (136.7KiB)...
>  https://paste.centos.org/view/ec37f118
>
>  Any suggestions as to what is wrong? The machine is sort of hard to get to  
>  simply because it is not online now, and I have to go in to get more  
>  information but I will try and provide additional requests for information  
>  in as timely a manner as possible. 
The apparent problem is this:

May 17 00:59:14 hostname.suppressed systemd[1]: Starting File System Check  
on /dev/disk/by-uuid/287a95e7-4b5d-420a-94e8-38c0d8161375…
May 17 00:59:14 hostname.suppressed systemd-fsck[470]: /dev/sda2 contains a  
file system with errors, check forced.
May 17 00:59:15 hostname.suppressed systemd-fsck[470]: /dev/sda2: Inodes  
that were part of a corrupted orphan linked list found.
May 17 00:59:15 hostname.suppressed systemd-fsck[470]: /dev/sda2: UNEXPECTED  
INCONSISTENCY; RUN fsck MANUALLY.
May 17 00:59:15 hostname.suppressed systemd-fsck[470]:         (i.e.,  
without -a or -p options)
May 17 00:59:15 hostname.suppressed systemd-fsck[468]: fsck failed with exit  
status 4.
May 17 00:59:15 hostname.suppressed systemd-fsck[468]: Running request  
emergency.target/start/replace

Looks like the root filesystem is damaged. The disclaimer is that even after  
this is fixed, you have very few guarantees as to the stability of the  
system, since the extent of the damage is clear. If your /bin/bash gets  
blown away, you have very few options. So, at some point this may become  
unsalvageable, and it will be faster to reinstall, but you can at least try  
fixing the filesystem and see what shakes out.

To do that you'll need to boot off a live image. You may or may not be able to  
do this from a rescue shell, with root mounted read-only. I wouldn't waste  
my time figuring it out, and just boot off a live image. Before upgrading to  
F32 I prepared and tested a USB drive with a live image. Always a good idea.

Boot off the live image, open a terminal prompt, sudo to root, then run

fsck /dev/sda2

The more errors it reports, asking for your permission to fix each one, the  
less likely is your workstation to survive.

That's only the root filesystem, no guarantees that other filesystem are  
healthy, the boot didn't get this far so you may end up repeating this  
several times with each partition. Or, just fsck every partition, for a good  
measure, as long as you booted in the live CD.

Presuming you manage to repair the workstation good enough so that boots,  
what I would do, immediately after booting, is revalidate the RPM database:

rpm --rebuilddb
rpm -qa | wc -l

If there are no errors when rebuildding the rpm database, and the package  
count looks reasonable, the next and the final step would be to run a script  
that runs rpm -V for each installed package, skips over the configuration  
files that are expected to be modified, and finds any corrupted binary or  
library, then reinstall the damaged package.

[Samuel Sieb]

On 5/17/20 8:05 AM, Ranjan Maitra wrote:
>  Any suggestions as to what is wrong? The machine is sort of hard to
> get to simply because it is not online now, and I have to go in to get more information
> but I will try and provide additional requests for information in as timely a manner as
> possible. 
 From the emergency prompt, run "e2fsck /dev/sda2" and then reboot.  You 
should be good after that.

F32 upgrade fails (boots into Emergency mode) [Solved]

[Ranjan Maitra]

On Sun, 17 May 2020 11:45:20 -0700 Samuel Sieb <samuel(a)sieb.net&gt; wrote:

>  On 5/17/20 8:05 AM, Ranjan Maitra wrote:
>  > Any suggestions as to what is wrong? The machine is sort of hard to get to simply
> because it is not online now, and I have to go in to get more information but I will try
> and provide additional requests for information in as timely a manner as possible.
>
>   From the emergency prompt, run "e2fsck /dev/sda2" and then reboot.  You
>  should be good after that. 
Thanks for this! I decided to try this first and and this has brought it up. Wonder why
there was an issue in the first place.

Ranjan

F32 upgrade fails (boots into Emergency mode) [Solved]

[Samuel Sieb]

On 5/17/20 2:06 PM, Ranjan Maitra wrote:
>  On Sun, 17 May 2020 11:45:20 -0700 Samuel Sieb
> <samuel(a)sieb.net&gt; wrote:
>  
> > On 5/17/20 8:05 AM, Ranjan Maitra wrote:
> >> Any suggestions as to what is wrong? The machine is sort of hard to get to simply
> because it is not online now, and I have to go in to get more information but I will try
> and provide additional requests for information in as timely a manner as possible.
> >
> >   From the emergency prompt, run "e2fsck /dev/sda2" and then reboot.  You
> > should be good after that.
>  
>  Thanks for this! I decided to try this first and and this has brought it up. Wonder why
> there was an issue in the first place. 
I've had it happen only a few times over many computers.  I expect it's 
caused by having the computer hard reset due to suspend failing or power 
outages.  Or just some random error.