Improper shutdown, now Kernel panic.

[murph nj]

Hi all:

I've got an odd problem that I was hoping for some help on.

The laptop was previously working fine, I am running F30, updated regularly.

I've got an Acer laptop that has been shutting down suddenly. (I
suspect a bad battery, working on that.)

After a sudden shutdown last night, I now get a kernel panic on boot
right after "Starting Switch Root..."

I was able to boot from a USB stick, and was able to read the journal,
but I didn't see anything obvious to help.

I was able to get all of my data off of the (encrypted) disks, so
that's not a problem, but I don't want to just give up, and wash and
reload too quickly.

Any suggestions?

Thanks,
  --murph

[stan]

On Tue, 3 Sep 2019 22:31:52 -0400
murph nj  wrote:
> I've got an Acer laptop that has been shutting down suddenly. (I
> suspect a bad battery, working on that.)
>
> After a sudden shutdown last night, I now get a kernel panic on boot
> right after "Starting Switch Root..."

The switch root is changing from the initramfs to the installed OS.  If
there is a power problem, it is possible that memory is not working
properly, and disk reads are erronious.  Fix the power supply before
you go any further.  Any testing will yield questionable results until
that is resolved.

[murph nj]

On Wed, Sep 4, 2019 at 11:45 AM stan via users
 wrote:
> The switch root is changing from the initramfs to the installed OS.  If
> there is a power problem, it is possible that memory is not working
> properly, and disk reads are erronious.  Fix the power supply before
> you go any further.  Any testing will yield questionable results until
> that is resolved.

Thanks for the response.

The power supply was not the problem, it was the battery.  All
subsequent testing has been done plugged into power.  It also boots
just fine from a USB stick with either Mint or Fedora.

There are three kernels available from the boot menu, along with
rescue mode.  All of them kernel panic.

--murph

[stan]

On Wed, 4 Sep 2019 12:33:59 -0400
murph nj  wrote:
> The power supply was not the problem, it was the battery.  All
> subsequent testing has been done plugged into power.  It also boots
> just fine from a USB stick with either Mint or Fedora.
>
> There are three kernels available from the boot menu, along with
> rescue mode.  All of them kernel panic.

Does it get far enough along to put messages in the journal?  You could
run a boot that fails, then boot from the USB, and mnt the installed
filesystem (usually under /mnt/sysimage), chroot to /mnt/sysimage, and
run journalctl -b to look at boot messages.

Before you do that, boot from the USB, and then run a file system check
on the drives in the system while they are not mounted.  For ext4, that
would be e2fsck -n -v /dev/sd??, answer no to any queries (make no
changes), and verbose messages.  If it wants to fix things, and the
changes look OK, run with -p instead of -n.  See man e2fsck; this is
your filesystem, you want to know what you are doing; a mistake can be
fatal.

[murph nj]

I was able to mount the drives, and check them, no problems.  I was
also able to move all of my important data off of the drives to an
external, so I'm not worried about any loss to the data on the drives.
No worries of fatal mistakes anymore.

I also took a look at the journals that were left, there was nothing
that seemed terribly indicative of the problem.    It was all from the
last time that it was running normally, nothing was getting written on
the failed boot attempts, unfortunately.

On Wed, Sep 4, 2019 at 2:56 PM stan via users
 wrote:
> On Wed, 4 Sep 2019 12:33:59 -0400
> murph nj  wrote:
> > The power supply was not the problem, it was the battery.  All
> > subsequent testing has been done plugged into power.  It also boots
> > just fine from a USB stick with either Mint or Fedora.
> >
> > There are three kernels available from the boot menu, along with
> > rescue mode.  All of them kernel panic.
>
> Does it get far enough along to put messages in the journal?  You could
> run a boot that fails, then boot from the USB, and mnt the installed
> filesystem (usually under /mnt/sysimage), chroot to /mnt/sysimage, and
> run journalctl -b to look at boot messages.
>
> Before you do that, boot from the USB, and then run a file system check
> on the drives in the system while they are not mounted.  For ext4, that
> would be e2fsck -n -v /dev/sd??, answer no to any queries (make no
> changes), and verbose messages.  If it wants to fix things, and the
> changes look OK, run with -p instead of -n.  See man e2fsck; this is
> your filesystem, you want to know what you are doing; a mistake can be
> fatal.
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[stan]

On Wed, 4 Sep 2019 15:05:36 -0400
murph nj  wrote:
> I was able to mount the drives, and check them, no problems.  I was
> also able to move all of my important data off of the drives to an
> external, so I'm not worried about any loss to the data on the drives.
> No worries of fatal mistakes anymore.

Did that include the boot partition, where the kernels and initramfs'
are?
> I also took a look at the journals that were left, there was nothing
> that seemed terribly indicative of the problem.    It was all from the
> last time that it was running normally, nothing was getting written on
> the failed boot attempts, unfortunately.

Did you remove rhgb and quiet from the kernel boot parameters?  That
should show you the stream of messages from systemd as the system
boots, and the error should be near the end of that stream, so it
should still be on screen when the crash happens.  I think you hit
space or escape during boot to get the boot menu, and then you can edit
the line.  I have a timeout, so I always see it.

Does it drop to a dracut or grub prompt?  If it does, it is also
possible to view the temporary boot logs from there (I think they are
under /tmp (/var/tmp?)). It's painful because the shell is very limited.
Use
ls /bin
to see the available commands.  I think less is there, as is vi.

[murph nj]

On Wed, Sep 4, 2019 at 4:21 PM stan via users
 wrote:
> On Wed, 4 Sep 2019 15:05:36 -0400
> murph nj  wrote:
> > I was able to mount the drives, and check them, no problems.
>
> Did that include the boot partition, where the kernels and initramfs'
> are?

I had not, but on your suggestion, I did.  And....
> > I also took a look at the journals that were left, there was nothing
> > that seemed terribly indicative of the problem.    It was all from the
> > last time that it was running normally, nothing was getting written on
> > the failed boot attempts, unfortunately.
>
> Did you remove rhgb and quiet from the kernel boot parameters?  That
> should show you the stream of messages from systemd as the system
> boots, and the error should be near the end of that stream, so it
> should still be on screen when the crash happens.  I think you hit
> space or escape during boot to get the boot menu, and then you can edit
> the line.  I have a timeout, so I always see it.

I had not, but I edited from the grub menu, and eliminated them.  I
was then able to see the messages. (I was able to see them previously
by hitting the escape key right after entering my password for the
disks.)

Everything left on the screen after the kernel panic is (too much to type)

https://imgur.com/a/kf81crl
> Does it drop to a dracut or grub prompt?  If it does, it is also
> possible to view the temporary boot logs from there (I think they are
> under /tmp (/var/tmp?)). It's painful because the shell is very limited.
> Use
> ls /bin
> to see the available commands.  I think less is there, as is vi.

Unfortunately, it goes right to a kernel panic, so no prompts at all
to work with.  (I'd have chewed on it more before asking for help if I
had more to work with)

[murph nj]

On Wed, Sep 4, 2019 at 5:42 PM murph nj  wrote:
> On Wed, Sep 4, 2019 at 4:21 PM stan via users
>  wrote:
> > On Wed, 4 Sep 2019 15:05:36 -0400
> > murph nj  wrote:
> > > I was able to mount the drives, and check them, no problems.
> >
> > Did that include the boot partition, where the kernels and initramfs'
> > are?
>
> I had not, but on your suggestion, I did.  And....

Sorry, responded prematurely.

I meant to say that both the volumes that are mounted as /boot and
/boot/efi check out OK.

thanks,
  --murph

[sixpack13]

e2fsck is for ext -partitions
AFAIK /boot/efi is fat !

[murph nj]

On Thu, Sep 5, 2019 at 11:08 AM sixpack13  wrote:
> e2fsck is for ext -partitions
> AFAIK /boot/efi is fat !

You are correct.  I already checked, all the partitions are fine.

[Tony Nelson]

On 19-09-04 17:42:40, murph nj wrote:
  ...
> Everything left on the screen after the kernel panic is (too much to  
> type)
>
> https://imgur.com/a/kf81crl

...
> Unfortunately, it goes right to a kernel panic, so no prompts at all
> to work with.  (I'd have chewed on it more before asking for help if I
> had more to work with)

It dies trying to pivot-root, so add to the kernel command line

rd.break=pre-pivot

(See `man dracut.cmdline`, from `man kernel-command-line`.)  It will
drop into a shell just before disaster strikes (if I read it right)
and you'll get a chance to look around.  The current root will be the
init ramdisk, and the new root (hard drive) should be /sysroot IIRC.

[murph nj]

On Wed, Sep 4, 2019 at 9:56 PM Tony Nelson  wrote:
>
> It dies trying to pivot-root, so add to the kernel command line
>  rd.break=pre-pivot

Interesting.  I was able to drop to a shell.  I took a look at the
journal, and it looked like a possible hibernation problem, so I took
that out of the kernel line.  Still panicked.  I'll have to pick this
up tomorrow afternoon.  I'll try to get a dump of the journal then and
send it along.

Thanks for the help so far!

[murph nj]

Tony:

Can you give me some insight as to what the boot process does next
AFTER that pivot point?  The output I'm getting isn't enlightening me,
and I'm not quite sure where to look next.

I can just reinstall, but I'm trying to learn more about how to
recover from errors, instead of taking the east way out.

Thanks,
  --murph

[Tony Nelson]

On 19-09-09 14:10:24, murph nj wrote:
> Tony:
>
> Can you give me some insight as to what the boot process does next
> AFTER that pivot point?  The output I'm getting isn't enlightening me,
> and I'm not quite sure where to look next.

...

Usually, when it all goes wrong just after pivot-root, there is a
problem with the new (real) root.  Exactly how it goes wrong is not
useful.  I would look hard at whatever is mounted at /sysroot.  It
should be what you expect / (root) to be, but mounted ro at that point
(`cat /proc/mounts` should show it mounted at /sysroot (see `man 5  
fstab`
for format); `ls -l /sysroot` should show the contents you expect).
(All off the top of my head; I haven't rebooted to test.)

[murph nj]

On Mon, Sep 9, 2019 at 9:07 PM Tony Nelson  wrote:
> Usually, when it all goes wrong just after pivot-root, there is a
> problem with the new (real) root.  Exactly how it goes wrong is not
> useful.  I would look hard at whatever is mounted at /sysroot.  It
> should be what you expect / (root) to be, but mounted ro at that point
> (`cat /proc/mounts` should show it mounted at /sysroot (see `man 5
> fstab`
> for format); `ls -l /sysroot` should show the contents you expect).
> (All off the top of my head; I haven't rebooted to test.)

Thanks.  That gives me something to examine.  I'm working tonight,
will probably dig back in to this tomorrow.

[Roger Heflin]

Symbolic links missing on / (/bin->/usr/bin and such).
/bin/bash missing or corrupted.
/lib64/ libraries missing or corrupted.
systemd itself missing, pivot root attempts to run systemd and that
may be what is failing, but you don't really know what the underlying
cause is for it to fail.

What exactly was the "improper shutdown"

The easiest way to debug is to livecd boot it mount up the old
filessystems on say /oldroot (rootfs) mount /oldroot/usr for usr and
so forth and then cd /oldroot and do a "chroot ."

I would expect the chroot . to fail with some sort of error that may
hint at what is broken.

On Tue, Sep 10, 2019 at 7:15 AM murph nj  wrote:
> On Mon, Sep 9, 2019 at 9:07 PM Tony Nelson  wrote:
> > Usually, when it all goes wrong just after pivot-root, there is a
> > problem with the new (real) root.  Exactly how it goes wrong is not
> > useful.  I would look hard at whatever is mounted at /sysroot.  It
> > should be what you expect / (root) to be, but mounted ro at that point
> > (`cat /proc/mounts` should show it mounted at /sysroot (see `man 5
> > fstab`
> > for format); `ls -l /sysroot` should show the contents you expect).
> > (All off the top of my head; I haven't rebooted to test.)
>
> Thanks.  That gives me something to examine.  I'm working tonight,
> will probably dig back in to this tomorrow.
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[murph nj]

On Tue, Sep 10, 2019 at 2:52 PM Roger Heflin  wrote:
> Symbolic links missing on / (/bin->/usr/bin and such).
> /bin/bash missing or corrupted.
> /lib64/ libraries missing or corrupted.
> systemd itself missing, pivot root attempts to run systemd and that
> may be what is failing, but you don't really know what the underlying
> cause is for it to fail.
>
> What exactly was the "improper shutdown"

A failing battery (I think) caused it to shut down when I wasn't
ready, and didn't shut down properly.  Then I accidentally opened the
lid, which started it again, and it again shut down cold.  I wouldn't
think that would corrupt things, but it's all I have for now.

I'll dig into this tomorrow probably.
> The easiest way to debug is to livecd boot it mount up the old
> filessystems on say /oldroot (rootfs) mount /oldroot/usr for usr and
> so forth and then cd /oldroot and do a "chroot ."
>
> I would expect the chroot . to fail with some sort of error that may
> hint at what is broken.

I'll try that.

Thanks.

[murph nj]

On Tue, Sep 10, 2019 at 2:52 PM Roger Heflin  wrote:
> Symbolic links missing on / (/bin->/usr/bin and such).
> /bin/bash missing or corrupted.
> /lib64/ libraries missing or corrupted.
> systemd itself missing, pivot root attempts to run systemd and that
> may be what is failing, but you don't really know what the underlying
> cause is for it to fail.
>
> What exactly was the "improper shutdown"
>
> The easiest way to debug is to livecd boot it mount up the old
> filessystems on say /oldroot (rootfs) mount /oldroot/usr for usr and
> so forth and then cd /oldroot and do a "chroot ."
>
> I would expect the chroot . to fail with some sort of error that may
> hint at what is broken.
>
> On Tue, Sep 10, 2019 at 7:15 AM murph nj  wrote:
> > On Mon, Sep 9, 2019 at 9:07 PM Tony Nelson  wrote:
> > > Usually, when it all goes wrong just after pivot-root, there is a
> > > problem with the new (real) root.  Exactly how it goes wrong is not
> > > useful.  I would look hard at whatever is mounted at /sysroot.  It
> > > should be what you expect / (root) to be, but mounted ro at that point
> > > (`cat /proc/mounts` should show it mounted at /sysroot (see `man 5
> > > fstab`
> > > for format); `ls -l /sysroot` should show the contents you expect).
> > > (All off the top of my head; I haven't rebooted to test.)

I was able to mount all of the boot drives, and it seemed OK, I
couldn't see anything that was out of order.

Unfortunately, I'm going to need this laptop next weekend, so my
journey of discovery on this is going to have to end, since I couldn't
fix it, I'll have to reload.  We gave it a shot though, and I have
some new things to try for other potential issues.

Again thanks for the help.

--murph

[stan]

On Wed, 4 Sep 2019 17:42:40 -0400
murph nj  wrote:
> I had not, but I edited from the grub menu, and eliminated them.  I
> was then able to see the messages. (I was able to see them previously
> by hitting the escape key right after entering my password for the
> disks.)
>
> Everything left on the screen after the kernel panic is (too much to
> type)
>
> https://imgur.com/a/kf81crl

What I pull from that is that the error causing the problem is a page
fault when it asks for some memory.  But I don't know where to go from
that insight, other than opening a bugzilla against the kernel,
describing your problem, and attaching your screenshot with its stack
trace. I'm stumped. If it boots from USB, then the memory must be
working properly, but here you are failing when accessing memory.
What's different?

Long shot, reseat the memory, and run a memory check, to be sure memory
is OK.
> Unfortunately, it goes right to a kernel panic, so no prompts at all
> to work with.  (I'd have chewed on it more before asking for help if I
> had more to work with)

Too bad.  I see that Tony gave you a workaround.  That will show
the state before the error, but not after the error.  I vaguely recall
that there is a way to tell the kernel or systemd to dump a core file
when this happens.  That would also be good to attach to a bugzilla, if
it is available.

[Ruggero Lot]

Two years later, I can say that this error might still be a thing.
I concluded that the most likely cause is my mother. Jokes aside, she unplugged the power chord before the pc switches off completely. Through the years, she managed to crash in the same way f33, f34, and now f35.
This time I had a look at the hard drive, and I can try to add something to the discussion.
The situation is the same as described above, kernel panic at switch-root. Three kernels were available (up to 5.12), and they all failed in the same way. Partitions look fine, fat for UEFI (not tested, but hey, I reach boot without problems), ext4 for boot, btrfs for data (2 sub-volumes with / and /home mount points) without encryption. All btrfs checks run from a live USB key were perfectly clean. 
I tried to switch root from the live key, and... error.
Some executables and libraries were completely deleted (zero sizes). I have a partial list, but I don't think it will be useful. Core software was all there except for python.
Unfortunately, I already reinstalled f35 by nuking the root subvolume and creating a new one. Next time I will be smarter and save it for further analysis.